Jobsub ID 108648.45@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108648.45@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 00:56:54
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2024-11-23 02:19:55
From worker node	Hostname	iut2-c201.iu.edu
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:09:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4169s)
	CPU time	1h (3704s = 88%)
Outputting started	2024-11-23 02:19:30
Output files
Finished	2024-11-23 02:19:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

wer@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:19:20 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:19:22 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
247
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 569!
22-Nov-2024 21:19:26 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2_reco_data_2024-11-23T_011008Z.root"
22-Nov-2024 21:19:26 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/8c/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.861445       37.088        346.336       16.7295       56.7808        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0273097     0.0847178     0.189519      0.0841337     0.0291072       100    
reco:linecluster:LineCluster                           0.154872      0.357281       1.73996      0.305952      0.205517        100    
reco:trajcluster:TrajCluster                           0.0775202     0.695783       17.6878      0.0965032      2.1913         100    
reco:pandora:StandardPandora                           0.0958225      3.01469       19.3325       2.54589       2.22265        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00020555    0.00973043     0.172437      0.002913      0.022422        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000244751    0.0346138      1.00832     0.00695086     0.123687        100    
reco:pandoracalo:Calorimetry                          0.000141513    0.0218219     0.474726     0.00204139     0.0680664       100    
reco:pandorapid:Chi2ParticleID                        5.3163e-05    0.000341587   0.00283776    0.000266999   0.000392848      100    
reco:trkshowersplit:TrackShowerHits                   0.000121679    0.016897      0.308939     0.00127291     0.0504434       100    
reco:pmtrack:PMAlgTrackMaker                          0.00157948      3.12771       50.6371      0.407419       7.69158        100    
reco:pmtrackcalo:Calorimetry                          0.000119057    0.012355      0.162079     0.00162815     0.0290713       100    
reco:pmtrackpid:Chi2ParticleID                        4.6997e-05    0.000453535   0.00279097    0.000266726   0.000529678      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000259661     2.3835        22.2094      0.574924       4.19705        100    
reco:pmtrajfitcalo:Calorimetry                        0.000109889    0.0208886     0.349834     0.00267388     0.0572329       100    
reco:pmtrajfitpid:Chi2ParticleID                      4.7726e-05    0.00032217    0.00123323    0.000291678   0.000210409      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00137354      3.51086       64.935       0.463631       9.16781        100    
reco:pmtracktccalo:Calorimetry                        0.00010553     0.011109       0.33879      0.0016875     0.0358004       100    
reco:pmtracktcpid:Chi2ParticleID                      4.6146e-05    0.000443308    0.0037768    0.000269183   0.000506364      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000265197     2.34323       34.595       0.604872       4.61403        100    
reco:pmtrajfittccalo:Calorimetry                      0.000103638    0.0188461     0.356869     0.00217958     0.0542044       100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.27e-05     0.000536734   0.00348597    0.000326839   0.000591273      100    
reco:blurredcluster:BlurredClustering                 0.000158092     1.52939       8.73452      0.846219       1.87134        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000265348     15.2553       155.588       5.21209       25.5115        100    
reco:emshower:EMShower                                 0.0003083     0.0462193     0.430171     0.00603648     0.0764371       100    
reco:cvnmap:CVNMapper                                 3.5658e-05     0.0208085     0.0756681      0.02441      0.0150772       100    
reco:cvneva:CVNEvaluator                              2.8027e-05      1.05935       2.7988        1.28191       0.69765        100    
reco:energyrecnumu:EnergyReco                         0.000435835    0.0418325     0.847027     0.00781172     0.112334        100    
reco:energyrecnue:EnergyReco                          0.000211585   0.00298388     0.0510865    0.00122131     0.0059534       100    
reco:energyrecnc:EnergyReco                           0.00252572    0.00599145     0.0551094    0.00439456     0.0063767       100    
reco:energyrecnumurange:EnergyReco                    0.000296689    0.0405246     0.827371     0.00381493     0.118901        100    
reco:energyrecnumumcs:EnergyReco                      0.00162385     0.0403339     0.820352     0.00576091     0.109495        100    
reco:opdec:Deconvolution                               0.0844884     0.494955      0.966149      0.487589      0.202102        100    
reco:ophitspe:OpHitFinderDeco                          0.230654       2.43646       4.45941       2.58826      0.999894        100    
reco:opflash:OpFlashFinder                            0.000201484   0.00236992    0.00887085    0.00226892     0.0012189       100    
reco:rns:RandomNumberSaver                            2.5011e-05    6.90413e-05   0.000611857   6.2201e-05    5.85737e-05      100    
reco:anglereconue:NuAngularReco                       0.000206901   0.00200105     0.0278862    0.000994775   0.00364469       100    
reco:anglereconumu:NuAngularReco                      0.000125086   0.00170246     0.0280094    0.000548067   0.00393136       100    
reco:anglereconuepfps:NuAngularReco                   0.000187162   0.00199571     0.0287307    0.000795358   0.00400247       100    
reco:anglereconumupfps:NuAngularReco                  0.000172718   0.00186596     0.0281995    0.00070324    0.00397183       100    
[art]:TriggerResults:TriggerResultInserter             1.324e-05    2.83919e-05   0.00010958    2.73795e-05   1.13371e-05      100    
end_path:out1:RootOutput                               3.06e-06     9.11697e-06   3.2576e-05     8.802e-06    4.30516e-06      100    
end_path:out1:RootOutput(write)                        0.0186179     0.434558       1.44122      0.452231      0.277421        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9344.83 MB
  Peak resident set size usage (VmHWM): 2130.34 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2_reco_2024-11-23T_011008Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2_reco_data_2024-11-23T_011008Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work