Jobsub ID 301694.140@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301694.140@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:07:23
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 04:58:34
From worker node	Hostname	n259.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:08:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_38_20231207T204700Z_gen_g4_detsim_hitreco__20240510T030640Z_reco2.root
Jobscript	Exit code	0
	Real time	49m (2958s)
	CPU time	40m (2437s = 82%)
Outputting started	2024-11-23 04:58:14
Output files
Finished	2024-11-23 04:58:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

reIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:58:03 GMT run: 6491894 subRun: 1 event: 3899
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:58:04 GMT run: 6491894 subRun: 1 event: 3899
Comparing two wires in the same plane: return failure
%MSG
355
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 967!
Begin processing the 100th record. run: 6491894 subRun: 1 event: 3900 at 23-Nov-2024 04:58:06 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:58:10 GMT run: 6491894 subRun: 1 event: 3900
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:58:10 GMT run: 6491894 subRun: 1 event: 3900
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:58:10 GMT run: 6491894 subRun: 1 event: 3900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:58:10 GMT run: 6491894 subRun: 1 event: 3900
Comparing two wires in the same plane: return failure
%MSG
205
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 397!
23-Nov-2024 04:58:13 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_38_20231207T204700Z_gen_g4_detsim_hitreco__20240510T030640Z_reco2_reco_data_2024-11-23T_040922Z.root"
23-Nov-2024 04:58:13 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/02/88/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_38_20231207T204700Z_gen_g4_detsim_hitreco__20240510T030640Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.96179       25.3337       252.962       10.8078       43.9502        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0177227      0.19497      0.886695      0.129543      0.191877        100    
reco:linecluster:LineCluster                           0.0968833      0.62978       7.07996      0.224759       1.0486         100    
reco:trajcluster:TrajCluster                           0.0488662     0.515478       14.7711      0.0583862      2.12544        100    
reco:pandora:StandardPandora                           0.0600781      1.88172       13.676        1.42554       1.94689        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000123774   0.00507912     0.0654314    0.00156056     0.0100082       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000159162    0.0224718      0.33485     0.00293223     0.0607352       100    
reco:pandoracalo:Calorimetry                          9.4078e-05     0.0154305     0.325441     0.00101578     0.0543069       100    
reco:pandorapid:Chi2ParticleID                        3.7942e-05    0.000194355   0.00266793    0.000137932   0.000271377      100    
reco:trkshowersplit:TrackShowerHits                   9.6062e-05     0.0189037     0.564485     0.000814764    0.0775328       100    
reco:pmtrack:PMAlgTrackMaker                          0.00113416      2.00941       41.8892      0.214716       5.93475        100    
reco:pmtrackcalo:Calorimetry                          7.0895e-05    0.00853737     0.203527     0.000966176    0.0262218       100    
reco:pmtrackpid:Chi2ParticleID                        2.8023e-05    0.000264746   0.00213007    0.000167688   0.000352025      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000171665     1.46429       24.3787       0.34125       3.40199        100    
reco:pmtrajfitcalo:Calorimetry                        7.6135e-05     0.0134402      0.31547     0.00147926     0.0423405       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6691e-05    0.000198847   0.000689537   0.000178012   0.000129552      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000951053     2.38347       46.6047      0.194762       7.42078        100    
reco:pmtracktccalo:Calorimetry                        6.7638e-05    0.00684118     0.0873629    0.00104365     0.0162488       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7442e-05    0.000261973    0.0016822    0.000175688   0.00028571       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000251276     1.51154       21.0525      0.339922       3.43971        100    
reco:pmtrajfittccalo:Calorimetry                      9.4479e-05     0.0117067     0.282447      0.0012083     0.0384397       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9785e-05    0.000349835    0.0029705    0.000193677    0.0004953       100    
reco:blurredcluster:BlurredClustering                 0.000106401     1.08149       7.19649       0.57622       1.37907        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000171264     9.41325       90.1362        2.992        17.236         100    
reco:emshower:EMShower                                0.000227151    0.0210797     0.154758      0.0023382     0.0345897       100    
reco:cvnmap:CVNMapper                                  2.649e-05    0.00844702     0.037038     0.00687019    0.00821727       100    
reco:cvneva:CVNEvaluator                              1.7353e-05     0.632851       1.23053      0.844662      0.405538        100    
reco:energyrecnumu:EnergyReco                         0.000399147    0.0300402     0.682416     0.00466755     0.0822538       100    
reco:energyrecnue:EnergyReco                          0.000159783   0.00179343     0.0151298    0.00063791     0.0027613       100    
reco:energyrecnc:EnergyReco                           0.00191522    0.00449923     0.0191271    0.00358782    0.00289953       100    
reco:energyrecnumurange:EnergyReco                    0.000159402    0.0279101     0.687347     0.00346004     0.0824853       100    
reco:energyrecnumumcs:EnergyReco                      0.000820866    0.0296247     0.683827     0.00410905     0.0817861       100    
reco:opdec:Deconvolution                               0.111284      0.586575       3.12361      0.319317      0.578877        100    
reco:ophitspe:OpHitFinderDeco                          0.409721       2.52539       5.3239        2.41949        1.175         100    
reco:opflash:OpFlashFinder                            0.00032181    0.00154791    0.00637798    0.00148322    0.000788222      100    
reco:rns:RandomNumberSaver                            1.9326e-05    3.77872e-05   0.000374299   3.08535e-05   3.50802e-05      100    
reco:anglereconue:NuAngularReco                       0.00016365    0.00119614     0.0138125    0.000428973   0.00227282       100    
reco:anglereconumu:NuAngularReco                      9.1313e-05     0.0010157     0.0135142    0.000285877   0.00219053       100    
reco:anglereconuepfps:NuAngularReco                   0.000124676   0.00116954     0.0135307    0.000423152   0.00233205       100    
reco:anglereconumupfps:NuAngularReco                  0.000101953   0.00109607     0.0135811    0.000354422   0.00221878       100    
[art]:TriggerResults:TriggerResultInserter             9.007e-06    1.55671e-05    9.504e-05    1.45775e-05   8.68127e-06      100    
end_path:out1:RootOutput                               2.716e-06    6.19951e-06    3.152e-05    4.7085e-06    3.95504e-06      100    
end_path:out1:RootOutput(write)                        0.0247855     0.268055      0.928777      0.278359      0.177854        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38317.4 MB
  Peak resident set size usage (VmHWM): 2263.59 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_38_20231207T204700Z_gen_g4_detsim_hitreco__20240510T030640Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_38_20231207T204700Z_gen_g4_detsim_hitreco__20240510T030640Z_reco2_reco_2024-11-23T_040922Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_38_20231207T204700Z_gen_g4_detsim_hitreco__20240510T030640Z_reco2_reco_data_2024-11-23T_040922Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work