Jobsub ID 108677.42@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108677.42@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:52:31
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 06:00:58
From worker node	Hostname	n178.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:14:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_155_20231118T173351Z_gen_g4_detsim_hitreco__20240503T051634Z_reco2.root
Jobscript	Exit code	0
	Real time	3h (13589s)
	CPU time	3h (12766s = 93%)
Outputting started	2024-11-23 06:00:42
Output files
Finished	2024-11-23 06:00:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

oducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:34 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:00:36 GMT run: 50231162 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
298
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 761!
23-Nov-2024 06:00:41 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_155_20231118T173351Z_gen_g4_detsim_hitreco__20240503T051634Z_reco2_reco_data_2024-11-23T_021420Z.root"
23-Nov-2024 06:00:41 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/46/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_155_20231118T173351Z_gen_g4_detsim_hitreco__20240503T051634Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.39585       64.8048       1428.31       18.0687       167.411        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.017565      0.114656      0.528014      0.0904547     0.0793488       200    
reco:linecluster:LineCluster                           0.106237       1.0093        32.0998       0.36231       3.36051        200    
reco:trajcluster:TrajCluster                           0.0498725      3.21892       225.081      0.142384       17.9197        200    
reco:pandora:StandardPandora                           0.0805135      6.34437       331.026       2.15566       26.686         200    
reco:pandoraTrack:LArPandoraTrackCreation             0.00019357     0.0124238     0.257734     0.00252117     0.0341003       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000264125    0.0210541     0.450523     0.00467439      0.05128        200    
reco:pandoracalo:Calorimetry                          0.000148252    0.0572221      3.09331     0.00153797     0.283164        200    
reco:pandorapid:Chi2ParticleID                        5.5783e-05    0.000229409   0.00104544     0.0001823    0.000178891      200    
reco:trkshowersplit:TrackShowerHits                   9.9525e-05     0.0614936      2.01983     0.00106498     0.248213        200    
reco:pmtrack:PMAlgTrackMaker                          0.00204301      5.22777       133.673      0.327813       17.5589        200    
reco:pmtrackcalo:Calorimetry                          0.000123406    0.0280319     0.967451     0.00142071     0.113947        200    
reco:pmtrackpid:Chi2ParticleID                         4.775e-05    0.000509996   0.00632174    0.000213187   0.000969168      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000327138     5.23799       129.268      0.455486       18.2266        200    
reco:pmtrajfitcalo:Calorimetry                        0.000103767    0.0587577      3.06305     0.00227741     0.285938        200    
reco:pmtrajfitpid:Chi2ParticleID                      4.3699e-05    0.000285572   0.00172587    0.000211676   0.00026659       200    
reco:pmtracktc:PMAlgTrackMaker                        0.00195781      7.46807       233.612      0.346838       26.2943        200    
reco:pmtracktccalo:Calorimetry                        0.000116384    0.0157233     0.454841     0.00143651     0.0546679       200    
reco:pmtracktcpid:Chi2ParticleID                      4.4571e-05    0.000443336   0.00457219    0.00020978    0.00069315       200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000293577     3.60318       100.578       0.45519       10.7006        200    
reco:pmtrajfittccalo:Calorimetry                      0.000106362    0.0423122      1.56412     0.00166176     0.170806        200    
reco:pmtrajfittcpid:Chi2ParticleID                    4.5338e-05    0.000635275   0.00883051    0.000228723    0.0011865       200    
reco:blurredcluster:BlurredClustering                 8.9474e-05      1.7503        15.6958       0.87255       2.43833        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000161167     23.1477       352.708       5.14574       51.8891        200    
reco:emshower:EMShower                                0.000251105    0.0575934     0.798881      0.0118005     0.107366        200    
reco:cvnmap:CVNMapper                                 2.7264e-05     0.0149438     0.0635468     0.014871      0.0124161       200    
reco:cvneva:CVNEvaluator                              2.3193e-05      1.04715       2.09753       1.48573      0.704752        200    
reco:energyrecnumu:EnergyReco                         0.000430657     0.0838        3.52874     0.00805088     0.328202        200    
reco:energyrecnue:EnergyReco                          0.000234815   0.00425062     0.0847087    0.000961168    0.0112975       200    
reco:energyrecnc:EnergyReco                           0.00368074    0.00767943     0.090038      0.0044852     0.0111752       200    
reco:energyrecnumurange:EnergyReco                    0.000284027    0.0802403      3.51595     0.00402583     0.327211        200    
reco:energyrecnumumcs:EnergyReco                      0.00259335     0.0831167      3.5068      0.00531337     0.327986        200    
reco:opdec:Deconvolution                               0.112655      0.512011       1.13282      0.495335      0.180576        200    
reco:ophitspe:OpHitFinderDeco                           0.45682       5.07723       10.649        4.92473       2.0287         200    
reco:opflash:OpFlashFinder                            0.000355783   0.00209056    0.00536664    0.00189946    0.000964355      200    
reco:rns:RandomNumberSaver                            2.2192e-05    4.18106e-05   0.00038918    4.03005e-05   2.69654e-05      200    
reco:anglereconue:NuAngularReco                       0.000195237   0.00320879     0.0821619    0.000555758   0.00971775       200    
reco:anglereconumu:NuAngularReco                      0.000106008   0.00294078     0.0804361    0.000354307    0.0095311       200    
reco:anglereconuepfps:NuAngularReco                   0.000166911   0.00310568     0.0808247    0.000521277   0.00959948       200    
reco:anglereconumupfps:NuAngularReco                  0.000166687   0.00306422     0.0815646    0.000464068   0.00957321       200    
[art]:TriggerResults:TriggerResultInserter             1.251e-05    1.72032e-05   6.4327e-05    1.61265e-05   4.81238e-06      200    
end_path:out1:RootOutput                               2.737e-06    6.10163e-06   2.2099e-05    6.4065e-06    2.26936e-06      200    
end_path:out1:RootOutput(write)                        0.0180009     0.398627       2.12107      0.389649      0.326618        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 13996.9 MB
  Peak resident set size usage (VmHWM): 2540.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_155_20231118T173351Z_gen_g4_detsim_hitreco__20240503T051634Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_155_20231118T173351Z_gen_g4_detsim_hitreco__20240503T051634Z_reco2_reco_2024-11-23T_021420Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_155_20231118T173351Z_gen_g4_detsim_hitreco__20240503T051634Z_reco2_reco_data_2024-11-23T_021420Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work