Jobsub ID 108704.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108704.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:07:55
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-23 04:23:28
From worker node	Hostname	heplnc156.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:14:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_499_20231203T080505Z_gen_g4_detsim_hitreco__20240509T205953Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4110s)
	CPU time	1h (5284s = 128%)
Outputting started	2024-11-23 04:23:09
Output files
Finished	2024-11-23 04:23:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

un: 1 event: 50000
1st wire C:0 T:19 P:2 W:837 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:22:42 GMT run: 6416859 subRun: 1 event: 50000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:22:42 GMT run: 6416859 subRun: 1 event: 50000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:22:42 GMT run: 6416859 subRun: 1 event: 50000
1st wire C:0 T:19 P:2 W:575 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:22:42 GMT run: 6416859 subRun: 1 event: 50000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:22:42 GMT run: 6416859 subRun: 1 event: 50000
1st wire C:0 T:23 P:2 W:760 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:22:42 GMT run: 6416859 subRun: 1 event: 50000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 816, 878, 407
minwire 0: 1279
minwire 1: 512
minwire 2: 2082
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00980399, 0.00428763, 0.985908, 
Output 1: 0.155891, 0.380042, 0.287437, 0.17663, 
Output 2: 0.324916, 0.675084, 

693
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2473!
23-Nov-2024 04:23:07 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_499_20231203T080505Z_gen_g4_detsim_hitreco__20240509T205953Z_reco2_reco_data_2024-11-23T_031445Z.root"
23-Nov-2024 04:23:07 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/fe/45/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_499_20231203T080505Z_gen_g4_detsim_hitreco__20240509T205953Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.551661       39.4126       636.25        10.9934       86.1788        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0049035     0.0392923      0.6473       0.0166116     0.0870795       100    
reco:linecluster:LineCluster                           0.0442621     0.802718       45.8661      0.0776022      4.6137         100    
reco:trajcluster:TrajCluster                           0.0122662      1.39136       68.6386      0.0264229      7.11205        100    
reco:pandora:StandardPandora                           0.029816       2.76208       49.4814       1.51568       5.52541        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000165269   0.00784656     0.101136     0.00199852     0.0159431       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000232957    0.0145432     0.188037     0.00399308     0.0287903       100    
reco:pandoracalo:Calorimetry                          0.000139451    0.0360583      1.2318      0.00137837     0.142747        100    
reco:pandorapid:Chi2ParticleID                        5.0054e-05    0.000248104   0.00238931    0.00019739    0.000259359      100    
reco:trkshowersplit:TrackShowerHits                   0.000147056    0.0401793      1.4264      0.000926077     0.16037        100    
reco:pmtrack:PMAlgTrackMaker                          0.00127206      3.80156       104.413      0.190748       12.292         100    
reco:pmtrackcalo:Calorimetry                          7.3147e-05     0.0155708     0.276015     0.00104902     0.0450558       100    
reco:pmtrackpid:Chi2ParticleID                        2.7652e-05    0.000434458   0.00301073    0.000181916   0.000628223      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000178575     3.32582       47.3544      0.369577       8.94513        100    
reco:pmtrajfitcalo:Calorimetry                        6.7507e-05     0.0400823      1.15962     0.00187713     0.144475        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.178e-05    0.000274638   0.00097622    0.000214206   0.000196739      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000962544     4.70657       107.74       0.252287       14.3501        100    
reco:pmtracktccalo:Calorimetry                        7.1273e-05     0.0101246     0.199739     0.00114181     0.0269504       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7832e-05    0.000419862   0.00579609    0.00021062    0.000686109      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000151705     2.28948       48.4588      0.335097       6.40117        100    
reco:pmtrajfittccalo:Calorimetry                       6.412e-05     0.0250273     0.678135      0.0015233     0.0884358       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5608e-05    0.000557498   0.00658357    0.000214081   0.00100526       100    
reco:blurredcluster:BlurredClustering                 0.00011723      1.36809       8.81997      0.524435       1.91731        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0402444      14.4759       144.333       3.08454       28.3884        100    
reco:emshower:EMShower                                0.000340158    0.0383715     0.398649     0.00458248     0.0711168       100    
reco:cvnmap:CVNMapper                                 2.9465e-05     0.0149493     0.0489416     0.0167574     0.0116874       100    
reco:cvneva:CVNEvaluator                              2.3825e-05      1.19088       3.17294       1.36319      0.805273        100    
reco:energyrecnumu:EnergyReco                         0.000587441    0.0352171     0.996478     0.00572251     0.121526        100    
reco:energyrecnue:EnergyReco                          0.000153368   0.00266841     0.0343601    0.000602971   0.00541825       100    
reco:energyrecnc:EnergyReco                           0.00185001    0.00562066     0.0408166    0.00384366    0.00589659       100    
reco:energyrecnumurange:EnergyReco                    0.000157305    0.0321216     0.984054     0.00327042      0.12052        100    
reco:energyrecnumumcs:EnergyReco                      0.00182899     0.0338027     0.988615     0.00439235     0.120874        100    
reco:opdec:Deconvolution                               0.0406353      0.27848       4.01383      0.207634      0.401427        100    
reco:ophitspe:OpHitFinderDeco                           0.19132       2.29464       4.75839       2.22922       1.05862        100    
reco:opflash:OpFlashFinder                            0.000329518   0.00217527    0.00911971    0.00201717    0.00127571       100    
reco:rns:RandomNumberSaver                            2.0499e-05    7.56687e-05   0.00122437    5.42815e-05   0.000136678      100    
reco:anglereconue:NuAngularReco                       0.000172724   0.00218346     0.0193755    0.000625693   0.00398655       100    
reco:anglereconumu:NuAngularReco                      7.4489e-05    0.00165205     0.0168873    0.000335274   0.00356004       100    
reco:anglereconuepfps:NuAngularReco                   0.000157957   0.00194243     0.0177101    0.000524614   0.00373031       100    
reco:anglereconumupfps:NuAngularReco                  0.000115516   0.00181819     0.0177118    0.000430377   0.00385103       100    
[art]:TriggerResults:TriggerResultInserter             9.548e-06    2.04219e-05   8.6402e-05    1.8194e-05    1.07274e-05      100    
end_path:out1:RootOutput                               3.336e-06    1.1592e-05    6.3539e-05     9.878e-06    7.8377e-06       100    
end_path:out1:RootOutput(write)                        0.0219808     0.318588       1.23792      0.314406      0.231162        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39511.5 MB
  Peak resident set size usage (VmHWM): 2295.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_499_20231203T080505Z_gen_g4_detsim_hitreco__20240509T205953Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_499_20231203T080505Z_gen_g4_detsim_hitreco__20240509T205953Z_reco2_reco_2024-11-23T_031445Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_499_20231203T080505Z_gen_g4_detsim_hitreco__20240509T205953Z_reco2_reco_data_2024-11-23T_031445Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work