Jobsub ID 301669.112@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301669.112@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:17:34
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 04:35:13
From worker node	Hostname	n211.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:20:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_127_20231202T071006Z_gen_g4_detsim_hitreco__20240507T193035Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4480s)
	CPU time	1h (4093s = 91%)
Outputting started	2024-11-23 04:34:57
Output files
Finished	2024-11-23 04:35:13
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: 1 event: 12800
1st wire C:0 T:4 P:2 W:821 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:34:04 GMT run: 50546594 subRun: 1 event: 12800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:34:04 GMT run: 50546594 subRun: 1 event: 12800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:34:04 GMT run: 50546594 subRun: 1 event: 12800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:34:04 GMT run: 50546594 subRun: 1 event: 12800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:34:04 GMT run: 50546594 subRun: 1 event: 12800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:34:06 GMT run: 50546594 subRun: 1 event: 12800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 880, 1364, 1299
minwire 0: 693
minwire 1: 1558
minwire 2: 333
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00804971, 0.0143119, 0.977638, 
Output 1: 0.561809, 0.427059, 0.00999818, 0.00113388, 
Output 2: 0.217594, 0.782406, 

636
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2310!
23-Nov-2024 04:34:56 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_127_20231202T071006Z_gen_g4_detsim_hitreco__20240507T193035Z_reco2_reco_data_2024-11-23T_032021Z.root"
23-Nov-2024 04:34:56 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/a1/96/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_127_20231202T071006Z_gen_g4_detsim_hitreco__20240507T193035Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.25637       40.7993       684.331       13.728        93.4667        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0175079     0.0997567     0.231248      0.116998      0.0588131       100    
reco:linecluster:LineCluster                           0.173664      0.721333        24.65       0.199104       2.82082        100    
reco:trajcluster:TrajCluster                           0.049848       1.28917       40.8573      0.0607809      5.58242        100    
reco:pandora:StandardPandora                           0.0652156      3.18764       64.6716       1.69469       8.12362        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000171083   0.00737152     0.119824      0.0018474     0.0165584       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000222385    0.0249266     0.711812     0.00376536     0.0830463       100    
reco:pandoracalo:Calorimetry                          0.000124772    0.0310273      1.28926     0.00110464     0.141823        100    
reco:pandorapid:Chi2ParticleID                        4.7804e-05    0.000202054   0.00135829    0.000159105   0.000174252      100    
reco:trkshowersplit:TrackShowerHits                   0.000140881    0.0338757     0.995923     0.00075344     0.139282        100    
reco:pmtrack:PMAlgTrackMaker                          0.00166852      4.04552       120.813      0.258769       14.8252        100    
reco:pmtrackcalo:Calorimetry                          0.000100229    0.0129189     0.435011     0.000934804    0.0490368       100    
reco:pmtrackpid:Chi2ParticleID                        3.6746e-05    0.000356934   0.00437124    0.000170293   0.000614896      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00025609      2.94914       81.5752      0.311621       9.45345        100    
reco:pmtrajfitcalo:Calorimetry                        8.4432e-05     0.0321406      1.26075      0.0015314     0.141459        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.7363e-05    0.000243222   0.00148111    0.000177161   0.000235021      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00140395      4.23858       131.723      0.215847       15.6355        100    
reco:pmtracktccalo:Calorimetry                        9.1994e-05    0.00786839     0.109921     0.00104246     0.0191248       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5483e-05    0.000319101   0.00229354    0.000193311   0.000406508      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000221121     2.08852       43.4118      0.233536       5.9906         100    
reco:pmtrajfittccalo:Calorimetry                      8.1967e-05     0.0233813     0.757612     0.00122512     0.0923846       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4677e-05    0.000459565   0.00491527    0.000185336   0.000794982      100    
reco:blurredcluster:BlurredClustering                 7.6372e-05      1.1466        8.00416      0.538875       1.42968        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0448443      15.7485       215.115       4.18227       32.1986        100    
reco:emshower:EMShower                                0.000304404    0.0320988      0.38677      0.0056723     0.0554854       100    
reco:cvnmap:CVNMapper                                 2.5947e-05     0.0158711     0.0444022     0.0186546     0.0122069       100    
reco:cvneva:CVNEvaluator                               2.001e-05     0.849179       1.56834       1.16103      0.575249        100    
reco:energyrecnumu:EnergyReco                         0.00037828     0.0290737     0.846258     0.00570633     0.0909175       100    
reco:energyrecnue:EnergyReco                          0.000205444   0.00260038     0.0410981    0.000720686   0.00562378       100    
reco:energyrecnc:EnergyReco                           0.00259934    0.00510355     0.0431207    0.00333176    0.00554051       100    
reco:energyrecnumurange:EnergyReco                    0.000230275    0.0261569     0.846453     0.00297973     0.0908947       100    
reco:energyrecnumumcs:EnergyReco                      0.00114894     0.0278699     0.841542     0.00346741     0.090466        100    
reco:opdec:Deconvolution                               0.0872428     0.344069      0.623544      0.335915      0.118598        100    
reco:ophitspe:OpHitFinderDeco                          0.407304       3.48746       6.62518       3.34823       1.45292        100    
reco:opflash:OpFlashFinder                            0.000275019   0.00176027    0.00460199    0.00168962    0.000787131      100    
reco:rns:RandomNumberSaver                            2.0995e-05    4.52785e-05   0.00033517    4.4363e-05    3.07723e-05      100    
reco:anglereconue:NuAngularReco                       0.000175165   0.00187785     0.032887     0.00047183    0.00427745       100    
reco:anglereconumu:NuAngularReco                      9.3464e-05    0.00163884     0.0326317    0.000291922    0.0041879       100    
reco:anglereconuepfps:NuAngularReco                   0.000150678   0.00176725     0.0329909    0.000421477   0.00416079       100    
reco:anglereconumupfps:NuAngularReco                  0.000131335   0.00172111     0.0329207    0.000382826    0.0041894       100    
[art]:TriggerResults:TriggerResultInserter            1.1265e-05    1.53258e-05   5.0614e-05    1.45825e-05   4.23285e-06      100    
end_path:out1:RootOutput                               2.737e-06    6.94529e-06    1.882e-05     7.366e-06    2.30769e-06      100    
end_path:out1:RootOutput(write)                        0.0162454     0.278951       1.27944      0.286775      0.207116        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14093.7 MB
  Peak resident set size usage (VmHWM): 2644.9 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_127_20231202T071006Z_gen_g4_detsim_hitreco__20240507T193035Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_127_20231202T071006Z_gen_g4_detsim_hitreco__20240507T193035Z_reco2_reco_2024-11-23T_032021Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_127_20231202T071006Z_gen_g4_detsim_hitreco__20240507T193035Z_reco2_reco_data_2024-11-23T_032021Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work