Jobsub ID 108743.37@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108743.37@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:34:30
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 05:32:44
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:38:45
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_365_20231208T004237Z_gen_g4_detsim_hitreco__20240510T030650Z_reco2.root
Jobscript	Exit code	0
	Real time	53m (3212s)
	CPU time	47m (2838s = 88%)
Outputting started	2024-11-23 05:32:18
Output files
Finished	2024-11-23 05:32:44
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:32:09 GMT run: 6491894 subRun: 1 event: 36600
1st wire C:0 T:8 P:2 W:822 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:32:09 GMT run: 6491894 subRun: 1 event: 36600
1st wire C:0 T:8 P:2 W:841 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:32:09 GMT run: 6491894 subRun: 1 event: 36600
1st wire C:0 T:8 P:2 W:852 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:32:11 GMT run: 6491894 subRun: 1 event: 36600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:32:11 GMT run: 6491894 subRun: 1 event: 36600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:32:11 GMT run: 6491894 subRun: 1 event: 36600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 57, 63, 48
minwire 0: 920
minwire 1: 1806
minwire 2: 565
Used alternate method to get min and max wires due to vertex determination failure: 522, 721
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 474, 673
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.986124, 0.00342538, 0.0104511, 
Output 1: 0.297941, 0.692798, 0.00895458, 0.000306853, 
Output 2: 0.974213, 0.0257871, 

553
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1803!
23-Nov-2024 05:32:17 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_365_20231208T004237Z_gen_g4_detsim_hitreco__20240510T030650Z_reco2_reco_data_2024-11-23T_043850Z.root"
23-Nov-2024 05:32:17 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/35/d7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_365_20231208T004237Z_gen_g4_detsim_hitreco__20240510T030650Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.721098       28.1362       467.601       10.7393       59.7512        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0178311     0.0313166     0.0693432     0.0355538     0.0118154       100    
reco:linecluster:LineCluster                           0.103288      0.434633       16.633       0.190558       1.65762        100    
reco:trajcluster:TrajCluster                           0.0508524     0.715213       28.2857      0.0607577      3.17692        100    
reco:pandora:StandardPandora                           0.067523       2.42877       75.5946       1.45937       7.46429        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000141036   0.00565763     0.0882363    0.00137294     0.012164        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000167727    0.010678       0.15217      0.0029264     0.0207494       100    
reco:pandoracalo:Calorimetry                          9.7114e-05     0.0181774     0.749218     0.00091658     0.0796793       100    
reco:pandorapid:Chi2ParticleID                        3.8523e-05    0.000191458   0.00233485    0.000130461   0.000248307      100    
reco:trkshowersplit:TrackShowerHits                   0.000108083    0.0231805     0.760311     0.000850939    0.0864059       100    
reco:pmtrack:PMAlgTrackMaker                          0.00106083      2.21221       59.6555      0.123575       7.26692        100    
reco:pmtrackcalo:Calorimetry                           6.364e-05    0.00844784     0.163902      0.0010368     0.0243249       100    
reco:pmtrackpid:Chi2ParticleID                        2.7061e-05    0.000306896   0.00219307    0.000165297   0.000385127      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000168998     2.02431       43.3576      0.240648       5.85688        100    
reco:pmtrajfitcalo:Calorimetry                        6.2428e-05     0.0203137     0.770404     0.00135071     0.0855695       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.665e-05    0.000217086   0.000954259   0.000173212   0.000160073      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000841527     2.70642       59.5348       0.1605        8.65252        100    
reco:pmtracktccalo:Calorimetry                        6.4412e-05    0.00583255     0.122476     0.00107003     0.0158524       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4647e-05    0.000290147   0.00175299    0.000185865   0.000322144      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000152608     1.5991        35.3436      0.316934       4.42944        100    
reco:pmtrajfittccalo:Calorimetry                      5.7408e-05     0.0140705      0.39617     0.00128464     0.0490545       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3644e-05    0.000405263   0.00330962    0.00022522    0.000562884      100    
reco:blurredcluster:BlurredClustering                 0.000144592     1.25015       6.34189      0.704091       1.42168        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000191892     11.1832       137.01        3.25659       21.7421        100    
reco:emshower:EMShower                                0.000269047    0.0296007      0.24519      0.0122165     0.0424513       100    
reco:cvnmap:CVNMapper                                 2.8764e-05    0.00972626     0.0580501    0.00826355     0.0096583       100    
reco:cvneva:CVNEvaluator                              1.8245e-05      0.62111       1.35778      0.682552      0.420419        100    
reco:energyrecnumu:EnergyReco                         0.000385087    0.0355178     0.647222     0.00558263     0.104361        100    
reco:energyrecnue:EnergyReco                          0.00015939    0.00201682     0.0255475    0.000615585   0.00394835       100    
reco:energyrecnc:EnergyReco                           0.00201438     0.0046403     0.0277111    0.00372934    0.00416432       100    
reco:energyrecnumurange:EnergyReco                    0.000157356    0.0332128     0.646009     0.00358403     0.103575        100    
reco:energyrecnumumcs:EnergyReco                      0.00197094     0.0348522     0.636317     0.00411386     0.103916        100    
reco:opdec:Deconvolution                               0.0677615     0.309504      0.645986      0.317758      0.110589        100    
reco:ophitspe:OpHitFinderDeco                          0.265365       2.09269       6.0424        1.93145       1.07177        100    
reco:opflash:OpFlashFinder                            0.000237487   0.00156502     0.0055095    0.00152352    0.000785619      100    
reco:rns:RandomNumberSaver                            2.0708e-05    4.23093e-05   0.000417637   3.73805e-05   3.9523e-05       100    
reco:anglereconue:NuAngularReco                       0.000163969   0.00160481     0.0368497    0.000440976   0.00427297       100    
reco:anglereconumu:NuAngularReco                      7.3468e-05    0.00139109     0.0355478    0.000271331   0.00414987       100    
reco:anglereconuepfps:NuAngularReco                   0.000125096   0.00152179     0.0336019    0.000400234   0.00402447       100    
reco:anglereconumupfps:NuAngularReco                  0.00010082    0.00144781     0.0339466    0.000357103   0.00402422       100    
[art]:TriggerResults:TriggerResultInserter             9.288e-06    1.56131e-05   6.6656e-05    1.4988e-05    6.41826e-06      100    
end_path:out1:RootOutput                               3.096e-06    6.39749e-06   2.4176e-05      5.1e-06     3.43233e-06      100    
end_path:out1:RootOutput(write)                        0.0158421     0.260282        1.346       0.263196       0.20379        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38419.3 MB
  Peak resident set size usage (VmHWM): 2357.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_365_20231208T004237Z_gen_g4_detsim_hitreco__20240510T030650Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_365_20231208T004237Z_gen_g4_detsim_hitreco__20240510T030650Z_reco2_reco_2024-11-23T_043850Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_365_20231208T004237Z_gen_g4_detsim_hitreco__20240510T030650Z_reco2_reco_data_2024-11-23T_043850Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work