Jobsub ID 108730.33@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108730.33@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:04:08
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-23 05:07:26
From worker node	Hostname	heplnc164.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:06:09
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_622_20231208T090704Z_gen_g4_detsim_hitreco__20240510T062701Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3648s)
	CPU time	1h (4041s = 110%)
Outputting started	2024-11-23 05:07:00
Output files
Finished	2024-11-23 05:07:26
List job events Wrapper job log
Jobscript log (last 10,000 characters)

reIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:06:50 GMT run: 6472771 subRun: 1 event: 62300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 412, 419, 382
minwire 0: 1797
minwire 1: 410
minwire 2: 1653
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.320768, 0.619354, 0.0598779, 
Output 1: 0.138558, 0.707075, 0.146572, 0.00779509, 
Output 2: 0.199661, 0.80034, 

601
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2203!
23-Nov-2024 05:06:57 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_622_20231208T090704Z_gen_g4_detsim_hitreco__20240510T062701Z_reco2_reco_data_2024-11-23T_040617Z.root"
23-Nov-2024 05:06:57 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/3f/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_622_20231208T090704Z_gen_g4_detsim_hitreco__20240510T062701Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.299502       33.7634       277.893       12.3618       56.8776        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00738733     0.114814      0.899915      0.0641551     0.144063        100    
reco:linecluster:LineCluster                           0.0253142      0.5736        11.8429      0.123997       1.44569        100    
reco:trajcluster:TrajCluster                           0.0122695     0.610803       13.9443      0.0274641      2.17119        100    
reco:pandora:StandardPandora                           0.0271568      2.39255       18.1261       1.61803       2.74344        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000201899   0.00812257     0.100895     0.00204175     0.0170122       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000336162    0.0234408     0.563685     0.00386502     0.069665        100    
reco:pandoracalo:Calorimetry                          0.000185198    0.0209373     0.335902     0.00141324     0.0590205       100    
reco:pandorapid:Chi2ParticleID                        5.1166e-05    0.000284331   0.00245405    0.000209023   0.000292585      100    
reco:trkshowersplit:TrackShowerHits                   0.000115026    0.0256707     0.460644     0.00081697     0.0812784       100    
reco:pmtrack:PMAlgTrackMaker                          0.00171106      2.55668       34.7634      0.244278       6.26053        100    
reco:pmtrackcalo:Calorimetry                          0.000107733    0.0115291     0.183624     0.00144776     0.0317799       100    
reco:pmtrackpid:Chi2ParticleID                        4.3311e-05    0.000419872   0.00332405    0.000249974   0.000548961      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000290716     2.21877       29.0277      0.284697       5.04504        100    
reco:pmtrajfitcalo:Calorimetry                        9.7674e-05     0.0217645     0.355396     0.00189885     0.0609099       100    
reco:pmtrajfitpid:Chi2ParticleID                       4.249e-05    0.000298444   0.00124054    0.000245116   0.000235443      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00147957      3.47798       47.3585      0.267698       8.77757        100    
reco:pmtracktccalo:Calorimetry                        9.9386e-05    0.00915315     0.140184     0.00144524     0.023009        100    
reco:pmtracktcpid:Chi2ParticleID                      4.2229e-05    0.000392215   0.00254792    0.00024215    0.000445036      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000253877     1.74123       17.003        0.3124        3.61774        100    
reco:pmtrajfittccalo:Calorimetry                      9.3466e-05     0.014641      0.206866     0.00146656     0.0392332       100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.1127e-05    0.000496041   0.00391445    0.000245376   0.00069174       100    
reco:blurredcluster:BlurredClustering                 0.000220595     1.49131       8.07018      0.751024       1.83952        100    
reco:emtrkmichelid:EmTrackMichelId                    0.00018079      13.7319       120.741       3.46931       25.1722        100    
reco:emshower:EMShower                                0.000308039    0.047595       0.47185     0.00821976     0.0800967       100    
reco:cvnmap:CVNMapper                                 2.9055e-05     0.0199884     0.0474154     0.0239705     0.0122998       100    
reco:cvneva:CVNEvaluator                              2.7752e-05      1.19318       2.00398       1.46516      0.668555        100    
reco:energyrecnumu:EnergyReco                         0.000529405    0.0471928      1.61427     0.00640392      0.18645        100    
reco:energyrecnue:EnergyReco                          0.000171322   0.00259196     0.0342617    0.000685728   0.00505049       100    
reco:energyrecnc:EnergyReco                           0.00200395    0.00561281     0.038619     0.00389254    0.00541735       100    
reco:energyrecnumurange:EnergyReco                    0.000203943    0.041079       1.34411     0.00349221     0.163633        100    
reco:energyrecnumumcs:EnergyReco                      0.00196903     0.0424242      1.18019     0.00513523     0.155266        100    
reco:opdec:Deconvolution                               0.0185593     0.521907       2.76118      0.277372      0.560268        100    
reco:ophitspe:OpHitFinderDeco                          0.0358066      2.4636        5.4909        2.3305        1.2348         100    
reco:opflash:OpFlashFinder                            0.000288542   0.00218779     0.0068229    0.00220737    0.000933889      100    
reco:rns:RandomNumberSaver                            2.8965e-05    7.54287e-05   0.000509388   6.90095e-05   4.72904e-05      100    
reco:anglereconue:NuAngularReco                       0.000237376   0.00218596     0.0322665    0.000685368   0.00442192       100    
reco:anglereconumu:NuAngularReco                      0.000104686   0.00170424     0.0307669    0.000323568    0.0042323       100    
reco:anglereconuepfps:NuAngularReco                   0.000192001   0.00196625     0.0311483    0.000576849   0.00424928       100    
reco:anglereconumupfps:NuAngularReco                  0.000124924   0.00179092     0.0307246    0.000373402   0.00424272       100    
[art]:TriggerResults:TriggerResultInserter             1.065e-05    2.06725e-05   7.2586e-05    1.9176e-05    8.71801e-06      100    
end_path:out1:RootOutput                               4.228e-06    1.24726e-05   3.9173e-05    1.1907e-05    4.53358e-06      100    
end_path:out1:RootOutput(write)                        0.0208471     0.317917       1.06001      0.313786      0.208729        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39350.3 MB
  Peak resident set size usage (VmHWM): 2177.89 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_622_20231208T090704Z_gen_g4_detsim_hitreco__20240510T062701Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_622_20231208T090704Z_gen_g4_detsim_hitreco__20240510T062701Z_reco2_reco_2024-11-23T_040617Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_622_20231208T090704Z_gen_g4_detsim_hitreco__20240510T062701Z_reco2_reco_data_2024-11-23T_040617Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work