Jobsub ID 301697.77@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301697.77@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:11:36
Site	UK_Brunel
Entry	CMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat	2024-11-23 05:18:27
From worker node	Hostname	wn-b2-16-01.brunel.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:13:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_422_20231122T115925Z_gen_g4_detsim_hitreco__20240507T172507Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3902s)
	CPU time	59m (3550s = 90%)
Outputting started	2024-11-23 05:18:13
Output files
Finished	2024-11-23 05:18:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

dule  23-Nov-2024 05:18:02 GMT run: 74257901 subRun: 1 event: 42299
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:18:02 GMT run: 74257901 subRun: 1 event: 42299
Comparing two wires in the same plane: return failure
%MSG
502
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1520!
Begin processing the 100th record. run: 74257901 subRun: 1 event: 42300 at 23-Nov-2024 05:18:05 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 05:18:07 GMT run: 74257901 subRun: 1 event: 42300
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 05:18:07 GMT run: 74257901 subRun: 1 event: 42300
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:18:07 GMT run: 74257901 subRun: 1 event: 42300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:18:07 GMT run: 74257901 subRun: 1 event: 42300
Comparing two wires in the same plane: return failure
%MSG
375
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1097!
23-Nov-2024 05:18:12 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_422_20231122T115925Z_gen_g4_detsim_hitreco__20240507T172507Z_reco2_reco_data_2024-11-23T_041313Z.root"
23-Nov-2024 05:18:12 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f5/06/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_422_20231122T115925Z_gen_g4_detsim_hitreco__20240507T172507Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.42057       34.8603       624.112       9.24357       84.2186        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0218846     0.0462025     0.0782106     0.0499479     0.0126641       100    
reco:linecluster:LineCluster                           0.198666      0.791233       38.5037      0.225978       3.87255        100    
reco:trajcluster:TrajCluster                           0.0641101      1.82797       79.4333      0.0779745      8.74299        100    
reco:pandora:StandardPandora                            1.15316       2.53933       48.1921       1.24357       6.09764        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000121485   0.00826911     0.163305     0.00145253     0.0221746       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000165258    0.0110527     0.230292     0.00267829     0.027237        100    
reco:pandoracalo:Calorimetry                          9.5259e-05     0.0420015      1.5334      0.000990991    0.191377        100    
reco:pandorapid:Chi2ParticleID                        3.9075e-05    0.000146065   0.00117813    0.000111896   0.000142884      100    
reco:trkshowersplit:TrackShowerHits                   0.000109451    0.0315491      1.13094     0.000606802    0.130064        100    
reco:pmtrack:PMAlgTrackMaker                          0.00116114      3.27108       87.9933      0.177915       11.4047        100    
reco:pmtrackcalo:Calorimetry                          7.7602e-05     0.0190275     0.538002     0.000746584    0.0750832       100    
reco:pmtrackpid:Chi2ParticleID                        3.3383e-05    0.000296196   0.00328308    0.000126312   0.000492029      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000200106     2.66877       67.9947      0.269351       7.96905        100    
reco:pmtrajfitcalo:Calorimetry                        7.3215e-05     0.0432419      1.53719     0.00122299     0.193432        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2405e-05    0.000171845   0.00098812    0.000129748   0.000168543      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00101233      3.92678       92.4147      0.248617       12.9137        100    
reco:pmtracktccalo:Calorimetry                        7.7492e-05     0.0110159     0.267828     0.000738337    0.0336036       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3012e-05    0.000269146   0.00203367    0.000121135   0.000360728      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000191376     1.86477       31.4655      0.351474       4.69511        100    
reco:pmtrajfittccalo:Calorimetry                      7.4532e-05     0.0248774     0.609612      0.0011014     0.0898054       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.1914e-05    0.000374008    0.0033783    0.000142692   0.000588582      100    
reco:blurredcluster:BlurredClustering                 0.00780012      1.13624       8.21399       0.53212       1.5783         100    
reco:emtrkmichelid:EmTrackMichelId                     0.256544       13.9531       180.263       3.11393       28.5075        100    
reco:emshower:EMShower                                0.000240301    0.034885      0.251433     0.00629404     0.0573018       100    
reco:cvnmap:CVNMapper                                 2.6968e-05    0.00985213     0.0401985    0.00915918    0.00922517       100    
reco:cvneva:CVNEvaluator                              1.7038e-05     0.462984       1.08325       0.64756      0.313559        100    
reco:energyrecnumu:EnergyReco                         0.000263472    0.0639489      2.58721     0.00339155     0.271747        100    
reco:energyrecnue:EnergyReco                          0.000153205   0.00290111     0.0564645    0.000608046   0.00750109       100    
reco:energyrecnc:EnergyReco                           0.00200745    0.00479096     0.0602652    0.00240487    0.00779179       100    
reco:energyrecnumurange:EnergyReco                    0.000157812    0.0626676      2.58245     0.00216125     0.271179        100    
reco:energyrecnumumcs:EnergyReco                      0.00152351     0.0640399      2.58254     0.00300518      0.27108        100    
reco:opdec:Deconvolution                               0.120845      0.326299       0.54779      0.319626      0.0948991       100    
reco:ophitspe:OpHitFinderDeco                          0.313868       1.37915       2.35397       1.37142      0.503374        100    
reco:opflash:OpFlashFinder                            0.000248773   0.00120519    0.00444818    0.00109234    0.000626015      100    
reco:rns:RandomNumberSaver                             1.421e-05    2.21636e-05   0.000232266   1.7763e-05    2.17937e-05      100    
reco:anglereconue:NuAngularReco                       9.4763e-05    0.00219305     0.0496462    0.00027606    0.00652812       100    
reco:anglereconumu:NuAngularReco                      7.6492e-05    0.00210496     0.0491165    0.000236698   0.00644945       100    
reco:anglereconuepfps:NuAngularReco                   0.000113231    0.0021997     0.0492307    0.000315132   0.00645047       100    
reco:anglereconumupfps:NuAngularReco                  0.000113663   0.00217395     0.0493569    0.00031036    0.00646091       100    
[art]:TriggerResults:TriggerResultInserter             9.144e-06    1.05957e-05   3.5908e-05    1.0126e-05    2.75958e-06      100    
end_path:out1:RootOutput                               2.183e-06    3.64507e-06   1.4432e-05     2.725e-06     1.682e-06       100    
end_path:out1:RootOutput(write)                        0.0173479     0.219853       1.01759      0.209709      0.173889        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8639.65 MB
  Peak resident set size usage (VmHWM): 2541.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_422_20231122T115925Z_gen_g4_detsim_hitreco__20240507T172507Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_422_20231122T115925Z_gen_g4_detsim_hitreco__20240507T172507Z_reco2_reco_2024-11-23T_041313Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_422_20231122T115925Z_gen_g4_detsim_hitreco__20240507T172507Z_reco2_reco_data_2024-11-23T_041313Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work