Jobsub ID 108744.151@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.151@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:53:47
From worker node	Hostname	n235.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7352 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:40:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_212_20231208T045012Z_gen_g4_detsim_hitreco__20240510T065425Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4392s)
	CPU time	1h (3628s = 82%)
Outputting started	2024-11-23 05:53:32
Output files
Finished	2024-11-23 05:53:47
List job events Wrapper job log
Jobscript log (last 10,000 characters)

tex determination failure: 0, 199
Classifier summary: 
Output 0: 0.111251, 0.0384661, 0.850283, 
Output 1: 0.448845, 0.436087, 0.0892512, 0.0258162, 
Output 2: 0.0336736, 0.966326, 

717
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2868!
Begin processing the 100th record. run: 50649927 subRun: 1 event: 21300 at 23-Nov-2024 05:53:26 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 05:53:28 GMT run: 50649927 subRun: 1 event: 21300
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:53:28 GMT run: 50649927 subRun: 1 event: 21300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:53:28 GMT run: 50649927 subRun: 1 event: 21300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:53:28 GMT run: 50649927 subRun: 1 event: 21300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:53:28 GMT run: 50649927 subRun: 1 event: 21300
Comparing two wires in the same plane: return failure
%MSG
175
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 413!
23-Nov-2024 05:53:31 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_212_20231208T045012Z_gen_g4_detsim_hitreco__20240510T065425Z_reco2_reco_data_2024-11-23T_044024Z.root"
23-Nov-2024 05:53:31 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/32/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_212_20231208T045012Z_gen_g4_detsim_hitreco__20240510T065425Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.12026       36.4632       951.861       9.26334       110.72         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0352881     0.0712149     0.127738      0.0701605     0.0200715       100    
reco:linecluster:LineCluster                           0.200818       1.57891       106.602      0.365217       10.5884        100    
reco:trajcluster:TrajCluster                           0.100668       2.8566        216.773      0.108691       21.7413        100    
reco:pandora:StandardPandora                           0.109203       2.66422       42.2852       1.58849       5.75482        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000124383   0.00689475     0.100035     0.00129605     0.0176367       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000172394    0.0114071     0.112863     0.00268649     0.0235854       100    
reco:pandoracalo:Calorimetry                          9.9276e-05     0.0235501      0.82287     0.000902357     0.10532        100    
reco:pandorapid:Chi2ParticleID                        3.6939e-05    0.000184497   0.00249892    0.00012237    0.000269262      100    
reco:trkshowersplit:TrackShowerHits                   4.9102e-05     0.029047       1.13205     0.000678367    0.136283        100    
reco:pmtrack:PMAlgTrackMaker                          0.00122118      3.50025       140.116      0.143838       15.3177        100    
reco:pmtrackcalo:Calorimetry                          6.2507e-05     0.0122821     0.270876     0.000723657    0.0426422       100    
reco:pmtrackpid:Chi2ParticleID                        2.5327e-05    0.000380403    0.0058555    0.000135109    0.0008124       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166392     2.66172       81.1378       0.21867       9.84983        100    
reco:pmtrajfitcalo:Calorimetry                        5.7338e-05     0.0242266     0.795771     0.00126045     0.105351        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3695e-05    0.000196282   0.00118554    0.000144235   0.000194341      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000985558     4.11608       177.733      0.178253       19.2568        100    
reco:pmtracktccalo:Calorimetry                        6.0423e-05    0.00814383     0.132554     0.000814112    0.0237968       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4596e-05    0.000338507   0.00543274    0.000140764   0.000686258      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000150342     1.88101       40.9482      0.230061       5.58289        100    
reco:pmtrajfittccalo:Calorimetry                      5.7869e-05     0.0185959     0.536978     0.000936757    0.0737863       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3204e-05    0.000385006   0.00568564    0.000137738   0.000796898      100    
reco:blurredcluster:BlurredClustering                 6.1275e-05      1.0775        9.17387      0.417792       1.76841        100    
reco:emtrkmichelid:EmTrackMichelId                    9.2614e-05      12.6555       169.291       2.34643       28.9921        100    
reco:emshower:EMShower                                0.000138069    0.032254      0.401835     0.00425478     0.0604394       100    
reco:cvnmap:CVNMapper                                  1.597e-05     0.0131125     0.0771405     0.0130117     0.0127187       100    
reco:cvneva:CVNEvaluator                              1.4598e-05     0.652427       1.31139      0.770744       0.4454         100    
reco:energyrecnumu:EnergyReco                         0.000377679    0.0486441      1.08185     0.00406391     0.151426        100    
reco:energyrecnue:EnergyReco                          0.000132709   0.00246626     0.0308433    0.000541666    0.0056242       100    
reco:energyrecnc:EnergyReco                           0.00182806    0.00468914     0.037526     0.00302716    0.00629784       100    
reco:energyrecnumurange:EnergyReco                    0.000118763    0.0462731      1.08402     0.00275108     0.149883        100    
reco:energyrecnumumcs:EnergyReco                      0.00150861     0.047352       1.08147     0.00339721     0.148582        100    
reco:opdec:Deconvolution                               0.108441      0.410574      0.808298      0.403341      0.138823        100    
reco:ophitspe:OpHitFinderDeco                          0.305852       1.74825       3.75279       1.69444      0.825627        100    
reco:opflash:OpFlashFinder                            0.000232706   0.00148018     0.0097526    0.00133619    0.00106599       100    
reco:rns:RandomNumberSaver                            1.8214e-05    5.02978e-05   0.000651351   4.23445e-05   6.28435e-05      100    
reco:anglereconue:NuAngularReco                       0.000153478   0.00184337     0.0277398    0.000414222   0.00475641       100    
reco:anglereconumu:NuAngularReco                      6.7366e-05    0.00154649     0.0256168    0.000185849    0.0045488       100    
reco:anglereconuepfps:NuAngularReco                   0.000117791   0.00177477     0.026015     0.000360225   0.00467317       100    
reco:anglereconumupfps:NuAngularReco                  9.7733e-05    0.00164952     0.0257304    0.000264386   0.00468077       100    
[art]:TriggerResults:TriggerResultInserter             8.727e-06    1.39647e-05   6.8368e-05    1.2754e-05    6.69378e-06      100    
end_path:out1:RootOutput                               2.956e-06    7.34361e-06   2.7912e-05     6.973e-06    4.17182e-06      100    
end_path:out1:RootOutput(write)                        0.010836      0.247935       1.1444       0.239378      0.204403        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16981 MB
  Peak resident set size usage (VmHWM): 2391.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_212_20231208T045012Z_gen_g4_detsim_hitreco__20240510T065425Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_212_20231208T045012Z_gen_g4_detsim_hitreco__20240510T065425Z_reco2_reco_2024-11-23T_044024Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_212_20231208T045012Z_gen_g4_detsim_hitreco__20240510T065425Z_reco2_reco_data_2024-11-23T_044024Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work