Jobsub ID 108705.20@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108705.20@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:10:05
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-23 04:15:31
From worker node	Hostname	heplnc156.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:20:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_710_20231207T113218Z_gen_g4_detsim_hitreco__20240510T053813Z_reco2.root
Jobscript	Exit code	0
	Real time	54m (3257s)
	CPU time	1h (4399s = 135%)
Outputting started	2024-11-23 04:15:12
Output files
Finished	2024-11-23 04:15:31
List job events Wrapper job log
Jobscript log (last 10,000 characters)

itter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 74551443 subRun: 1 event: 71100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 74551443 subRun: 1 event: 71100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 74551443 subRun: 1 event: 71100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 74551443 subRun: 1 event: 71100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 74551443 subRun: 1 event: 71100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 74551443 subRun: 1 event: 71100
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:15:04 GMT run: 74551443 subRun: 1 event: 71100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:15:04 GMT run: 74551443 subRun: 1 event: 71100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 746, 761, 609
minwire 0: 888
minwire 1: 1255
minwire 2: 976
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0510713, 0.484775, 0.464153, 
Output 1: 0.164454, 0.533143, 0.257526, 0.0448769, 
Output 2: 0.0672821, 0.932718, 

589
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2215!
23-Nov-2024 04:15:11 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_710_20231207T113218Z_gen_g4_detsim_hitreco__20240510T053813Z_reco2_reco_data_2024-11-23T_032100Z.root"
23-Nov-2024 04:15:11 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/1f/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_710_20231207T113218Z_gen_g4_detsim_hitreco__20240510T053813Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.45496       30.4507       352.137       11.9449       55.1824        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00585475     0.117985      0.979275      0.0499044     0.157232        100    
reco:linecluster:LineCluster                           0.0430237     0.417877       7.56734      0.0710994     0.983383        100    
reco:trajcluster:TrajCluster                           0.0115027     0.809415       32.0919      0.0282088      3.65949        100    
reco:pandora:StandardPandora                           0.0244675      2.28323       19.4664       1.52607       2.84191        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000167815   0.00769842     0.097484     0.00201831     0.015718        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000230813    0.0188036     0.348544     0.00383054     0.0484306       100    
reco:pandoracalo:Calorimetry                          0.000130745    0.0171067     0.273607       0.00152      0.0476452       100    
reco:pandorapid:Chi2ParticleID                        4.7569e-05    0.000277536   0.00229022    0.000209763   0.000267376      100    
reco:trkshowersplit:TrackShowerHits                    7.978e-05     0.0249568     0.514369     0.000975493    0.0816759       100    
reco:pmtrack:PMAlgTrackMaker                          0.00155727      2.40563       37.6821      0.247713       6.42336        100    
reco:pmtrackcalo:Calorimetry                          7.8417e-05     0.010905      0.177009     0.00145473     0.0298484       100    
reco:pmtrackpid:Chi2ParticleID                        3.1609e-05    0.000420604    0.0030933    0.000230567   0.000553936      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000267642     2.2395        52.5467      0.369873       6.47806        100    
reco:pmtrajfitcalo:Calorimetry                        7.9649e-05     0.0201231     0.466308      0.0021669     0.0615997       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5907e-05    0.000308027   0.00143661    0.000243397   0.000260651      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00144808      2.82983       47.7989      0.239222       7.70214        100    
reco:pmtracktccalo:Calorimetry                        9.9587e-05    0.00887002     0.186718     0.00128607     0.0245887       100    
reco:pmtracktcpid:Chi2ParticleID                      4.0356e-05    0.000384283   0.00422108    0.000229675   0.000531242      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000231725     1.75862       25.1048      0.334386       3.99932        100    
reco:pmtrajfittccalo:Calorimetry                       9.573e-05     0.0150852     0.292289     0.00174729     0.0453399       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9594e-05    0.000488597   0.00429259    0.00026974    0.000737588      100    
reco:blurredcluster:BlurredClustering                 0.000177503     1.2086        9.33665      0.522383       1.68729        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000151945     11.959        119.223       3.49943       22.3333        100    
reco:emshower:EMShower                                0.000217828    0.0453109      0.66072     0.00603498     0.0890527       100    
reco:cvnmap:CVNMapper                                 2.4606e-05     0.0142728     0.0540688     0.0175154     0.0116418       100    
reco:cvneva:CVNEvaluator                              1.9436e-05      1.16313       3.13075       1.35615      0.838649        100    
reco:energyrecnumu:EnergyReco                         0.00038966     0.0544694      1.75071     0.00537755      0.20075        100    
reco:energyrecnue:EnergyReco                          0.000153387   0.00257551     0.0303097    0.000806528   0.00466426       100    
reco:energyrecnc:EnergyReco                           0.00182372    0.00522545     0.0341685    0.00383293    0.00499312       100    
reco:energyrecnumurange:EnergyReco                    0.000144992    0.0497703      1.75449     0.00329537     0.196937        100    
reco:energyrecnumumcs:EnergyReco                      0.00193762     0.0514221      1.71347     0.00434835     0.195379        100    
reco:opdec:Deconvolution                               0.0163476      0.36085       1.55296      0.209617       0.37921        100    
reco:ophitspe:OpHitFinderDeco                          0.0344633      2.25102       4.77249       2.06832       1.13101        100    
reco:opflash:OpFlashFinder                            0.000147787   0.00199224     0.0102796    0.00177567    0.00122231       100    
reco:rns:RandomNumberSaver                            2.2603e-05    7.06112e-05   0.000510477   6.2252e-05    5.67304e-05      100    
reco:anglereconue:NuAngularReco                       0.000215394   0.00203402     0.0219098    0.000592481   0.00390013       100    
reco:anglereconumu:NuAngularReco                      0.00010635    0.00156825     0.0202553    0.000313923   0.00351928       100    
reco:anglereconuepfps:NuAngularReco                   0.000157004   0.00181035     0.0211066    0.000516999   0.00360896       100    
reco:anglereconumupfps:NuAngularReco                  0.000132067   0.00165804     0.0203698    0.00041912    0.00359657       100    
[art]:TriggerResults:TriggerResultInserter            1.0689e-05    1.9609e-05    6.4761e-05    1.8109e-05    8.98941e-06      100    
end_path:out1:RootOutput                               3.266e-06    9.25184e-06   3.5377e-05    8.5515e-06     4.967e-06       100    
end_path:out1:RootOutput(write)                        0.016942      0.285404       0.94153      0.286849      0.196595        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39500.6 MB
  Peak resident set size usage (VmHWM): 2311.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_710_20231207T113218Z_gen_g4_detsim_hitreco__20240510T053813Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_710_20231207T113218Z_gen_g4_detsim_hitreco__20240510T053813Z_reco2_reco_2024-11-23T_032100Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_710_20231207T113218Z_gen_g4_detsim_hitreco__20240510T053813Z_reco2_reco_data_2024-11-23T_032100Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work