Jobsub ID 108676.24@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108676.24@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:50:22
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce13-multicore
Last heartbeat	2024-11-23 03:00:46
From worker node	Hostname	tds218.pic.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:13:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_468_20231203T041416Z_gen_g4_detsim_hitreco__20240508T071928Z_reco2.root
Jobscript	Exit code	0
	Real time	47m (2834s)
	CPU time	40m (2406s = 84%)
Outputting started	2024-11-23 03:00:30
Output files
Finished	2024-11-23 03:00:46
List job events Wrapper job log
Jobscript log (last 10,000 characters)

mshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:21 CET run: 66006611 subRun: 1 event: 46900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 2032, 2038, 1454
minwire 0: 1447
minwire 1: 900
minwire 2: 1490
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0172772, 0.956165, 0.0265581, 
Output 1: 0.143655, 0.499381, 0.265161, 0.0918033, 
Output 2: 0.151949, 0.848051, 

772
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3002!
23-Nov-2024 04:00:29 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_468_20231203T041416Z_gen_g4_detsim_hitreco__20240508T071928Z_reco2_reco_data_2024-11-23T_021318Z.root"
23-Nov-2024 04:00:29 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/06/2f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_468_20231203T041416Z_gen_g4_detsim_hitreco__20240508T071928Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.87073       23.9026       134.207       12.4109       30.0596        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0342106     0.0517345     0.0704514     0.0360508     0.0169311       100    
reco:linecluster:LineCluster                            0.31129      0.397359       1.32602      0.343763      0.167886        100    
reco:trajcluster:TrajCluster                            0.10124       0.40389       5.10798      0.114752      0.843489        100    
reco:pandora:StandardPandora                            1.34251       2.11031       6.11079       2.05769      0.890604        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000140355   0.00571922     0.0458332    0.00186134    0.00989753       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000186727    0.0147998     0.211076     0.00396624     0.0312568       100    
reco:pandoracalo:Calorimetry                          0.000111237    0.0115074     0.169386     0.00135584     0.0280568       100    
reco:pandorapid:Chi2ParticleID                        4.2981e-05    0.000188813   0.000735744   0.000151942    0.0001415       100    
reco:trkshowersplit:TrackShowerHits                   0.000128741   0.00915612     0.162125     0.000869387    0.0235647       100    
reco:pmtrack:PMAlgTrackMaker                          0.00124707      1.68266       16.446       0.220574       3.30003        100    
reco:pmtrackcalo:Calorimetry                          8.7981e-05    0.00563882     0.0562648    0.00118318     0.0109838       100    
reco:pmtrackpid:Chi2ParticleID                        3.6991e-05    0.000249816   0.00117272    0.000166867   0.000249028      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00021894      1.60232       19.4956      0.304078       3.08923        100    
reco:pmtrajfitcalo:Calorimetry                        7.6454e-05     0.0119801     0.172987     0.00182663     0.0271242       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3499e-05    0.000226219   0.00117149    0.000170816   0.000192623      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00107915      1.77263       22.1992       0.30517       4.00249        100    
reco:pmtracktccalo:Calorimetry                        7.8936e-05    0.00494582     0.0842567    0.000981947    0.0108459       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2835e-05    0.000255859   0.00119985    0.000164979   0.000256026      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000208745     1.28489       11.0246      0.379747       2.14451        100    
reco:pmtrajfittccalo:Calorimetry                      8.0428e-05    0.00823048     0.0879339    0.00143126     0.0174199       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3347e-05    0.000317932   0.00204639    0.000189221   0.000372535      100    
reco:blurredcluster:BlurredClustering                 0.00528932     0.930014       6.30199      0.488075       1.13268        100    
reco:emtrkmichelid:EmTrackMichelId                     0.140924       10.084        60.4641       3.78009       15.1185        100    
reco:emshower:EMShower                                0.000264753    0.0381753     0.338997     0.00839178     0.0665937       100    
reco:cvnmap:CVNMapper                                 3.0099e-05     0.0127137     0.0349953     0.012504      0.0100575       100    
reco:cvneva:CVNEvaluator                               1.986e-05     0.645862       1.58028      0.790363      0.469713        100    
reco:energyrecnumu:EnergyReco                         0.000299038    0.050604       1.88684     0.00460641     0.211232        100    
reco:energyrecnue:EnergyReco                           0.0001689    0.00191786     0.0156869    0.000720723   0.00281373       100    
reco:energyrecnc:EnergyReco                           0.00226626    0.00414522     0.0191459    0.00299848    0.00310282       100    
reco:energyrecnumurange:EnergyReco                    0.000180016    0.0489403       1.877      0.00259593     0.210273        100    
reco:energyrecnumumcs:EnergyReco                      0.00125336     0.0491946      1.87812     0.00369186     0.208162        100    
reco:opdec:Deconvolution                               0.170745      0.433471      0.706014      0.432125      0.126408        100    
reco:ophitspe:OpHitFinderDeco                          0.525801       1.94136       3.96874       1.9109       0.884923        100    
reco:opflash:OpFlashFinder                            0.000441373   0.00166702    0.00407457    0.00161305    0.000824863      100    
reco:rns:RandomNumberSaver                             1.888e-05    3.66502e-05   0.000270302   3.14525e-05   2.6162e-05       100    
reco:anglereconue:NuAngularReco                       0.000153211   0.00132155     0.0131759    0.000434856   0.00226483       100    
reco:anglereconumu:NuAngularReco                      9.1102e-05    0.00115496     0.0130355    0.00027754    0.00222056       100    
reco:anglereconuepfps:NuAngularReco                   0.000144435   0.00129219     0.0132735    0.000440329   0.00220235       100    
reco:anglereconumupfps:NuAngularReco                  0.000132116    0.0012322     0.0133733    0.000343539   0.00220548       100    
[art]:TriggerResults:TriggerResultInserter            1.1134e-05    1.63841e-05   4.3506e-05    1.50695e-05   4.99208e-06      100    
end_path:out1:RootOutput                               2.799e-06    5.46325e-06   2.2079e-05    5.3285e-06    2.56998e-06      100    
end_path:out1:RootOutput(write)                        0.0277243     0.274422      0.747156      0.272251      0.182272        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8504.68 MB
  Peak resident set size usage (VmHWM): 2416.42 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_468_20231203T041416Z_gen_g4_detsim_hitreco__20240508T071928Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_468_20231203T041416Z_gen_g4_detsim_hitreco__20240508T071928Z_reco2_reco_2024-11-23T_021318Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_468_20231203T041416Z_gen_g4_detsim_hitreco__20240508T071928Z_reco2_reco_data_2024-11-23T_021318Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work