Jobsub ID 108730.116@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108730.116@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:04:08
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 05:30:03
From worker node	Hostname	heplnc153.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:06:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4974s)
	CPU time	1h (5526s = 111%)
Outputting started	2024-11-23 05:29:38
Output files
Finished	2024-11-23 05:30:03
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Maker:pmtrack@BeginModule  23-Nov-2024 05:29:27 GMT run: 50574832 subRun: 1 event: 92000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 05:29:27 GMT run: 50574832 subRun: 1 event: 92000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:29:27 GMT run: 50574832 subRun: 1 event: 92000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:29:28 GMT run: 50574832 subRun: 1 event: 92000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:29:28 GMT run: 50574832 subRun: 1 event: 92000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:29:28 GMT run: 50574832 subRun: 1 event: 92000
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:29:28 GMT run: 50574832 subRun: 1 event: 92000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 43, 81, 64
minwire 0: 2292
minwire 1: 310
minwire 2: 2601
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.695873, 0.0111433, 0.292984, 
Output 1: 0.0260079, 0.955048, 0.0186944, 0.000249644, 
Output 2: 0.643659, 0.356341, 

267
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 698!
23-Nov-2024 05:29:36 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2_reco_data_2024-11-23T_040650Z.root"
23-Nov-2024 05:29:36 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/7a/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.274174       47.1343       743.176       12.9454       107.027        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0059491     0.119646       0.66516      0.0467324     0.150643        100    
reco:linecluster:LineCluster                           0.0224861     0.554564       6.28057      0.099287       1.03366        100    
reco:trajcluster:TrajCluster                           0.0115722     0.994623       33.1848      0.0240287       3.88          100    
reco:pandora:StandardPandora                           0.0257564      4.37067       124.126       1.97174       12.8406        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000175098    0.0111308     0.251281     0.00173972     0.0316211       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000323366    0.0130593     0.177697     0.00319219     0.0262769       100    
reco:pandoracalo:Calorimetry                          0.000128571    0.0374928     0.628754     0.000998389    0.120802        100    
reco:pandorapid:Chi2ParticleID                        3.9884e-05    0.000317359   0.00615159    0.000180012   0.000625888      100    
reco:trkshowersplit:TrackShowerHits                   9.5199e-05     0.0367104     0.796571     0.000887877     0.12025        100    
reco:pmtrack:PMAlgTrackMaker                          0.00127092      3.64414       72.3739       0.11142       10.5895        100    
reco:pmtrackcalo:Calorimetry                          8.0211e-05     0.0172615     0.498281     0.00109962     0.0629988       100    
reco:pmtrackpid:Chi2ParticleID                        3.5246e-05    0.000500286   0.00557114    0.00020847    0.000912939      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000273683     3.43797       95.2813      0.191272       11.507         100    
reco:pmtrajfitcalo:Calorimetry                        8.4378e-05     0.0374157     0.812554     0.00142506     0.127021        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2431e-05    0.000320976   0.00210216    0.000222171   0.000314246      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113082      4.48008       100.64       0.118319       14.0991        100    
reco:pmtracktccalo:Calorimetry                        7.9679e-05     0.0110313     0.228815     0.00112579     0.0340064       100    
reco:pmtracktcpid:Chi2ParticleID                       3.178e-05    0.00046227    0.00420119    0.000235366   0.000713737      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000172062     2.35885       38.1405      0.237291       6.31925        100    
reco:pmtrajfittccalo:Calorimetry                      7.0192e-05     0.0221746     0.481171     0.00140026     0.0688946       100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.1338e-05    0.000610745   0.00516159    0.000250469   0.000979454      100    
reco:blurredcluster:BlurredClustering                 0.000201527     1.37821       10.6267      0.540912       2.13144        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000124203     21.1023       295.735       3.74081       48.0351        100    
reco:emshower:EMShower                                0.000186238    0.041369      0.451527      0.0107128     0.0739296       100    
reco:cvnmap:CVNMapper                                 2.0638e-05     0.0181916     0.0826366     0.019015      0.0167506       100    
reco:cvneva:CVNEvaluator                              1.6972e-05      1.23611       2.97719       1.5363       0.883288        100    
reco:energyrecnumu:EnergyReco                         0.00044883     0.0772524      1.66522     0.00660169      0.29547        100    
reco:energyrecnue:EnergyReco                          0.000143749   0.00292195     0.0590996    0.000665777   0.00728842       100    
reco:energyrecnc:EnergyReco                           0.00192255    0.00551052     0.0604439    0.00369492    0.00712264       100    
reco:energyrecnumurange:EnergyReco                    0.000200054    0.0714903      1.64564     0.00318975     0.286433        100    
reco:energyrecnumumcs:EnergyReco                       0.0018341     0.0749267      1.69895     0.00381344     0.297297        100    
reco:opdec:Deconvolution                               0.014414      0.388975       2.57697      0.248076      0.438008        100    
reco:ophitspe:OpHitFinderDeco                          0.0333708      2.25953       6.33534       2.21701       1.1878         100    
reco:opflash:OpFlashFinder                            0.000155962    0.0023447    0.00857045    0.00202354    0.00142655       100    
reco:rns:RandomNumberSaver                            2.0889e-05    7.8843e-05    0.000689521   6.9335e-05    6.88168e-05      100    
reco:anglereconue:NuAngularReco                       0.000207779   0.00261378     0.0511172    0.000675625   0.00636731       100    
reco:anglereconumu:NuAngularReco                      9.3004e-05    0.00208877     0.0483931    0.000343078   0.00598167       100    
reco:anglereconuepfps:NuAngularReco                   0.000147005   0.00232389     0.0488218    0.000532332   0.00600439       100    
reco:anglereconumupfps:NuAngularReco                  0.000150261   0.00215052     0.0480448    0.000408319   0.00594122       100    
[art]:TriggerResults:TriggerResultInserter             9.999e-06    2.29086e-05   9.9457e-05     1.862e-05    1.24902e-05      100    
end_path:out1:RootOutput                               2.986e-06    1.23276e-05   4.4704e-05    1.05145e-05   7.38916e-06      100    
end_path:out1:RootOutput(write)                        0.0144588     0.313539       1.42172      0.313485      0.256455        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39467.3 MB
  Peak resident set size usage (VmHWM): 2312.15 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2_reco_2024-11-23T_040650Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2_reco_data_2024-11-23T_040650Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work