Jobsub ID 108703.143@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108703.143@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:05:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 04:54:46
From worker node	Hostname	heplnc053.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:14:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_673_20231206T233004Z_gen_g4_detsim_hitreco__20240510T042920Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5794s)
	CPU time	1h (5343s = 92%)
Outputting started	2024-11-23 04:51:02
Output files
Finished	2024-11-23 04:54:46
List job events Wrapper job log
Jobscript log (last 10,000 characters)

04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:50:26 GMT run: 50602992 subRun: 1 event: 67400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 73, 67, 66
minwire 0: 1207
minwire 1: 1583
minwire 2: 1273
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.945484, 0.0153582, 0.0391584, 
Output 1: 0.00524893, 0.0413342, 0.76657, 0.186847, 
Output 2: 0.978656, 0.0213442, 

580
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2044!
23-Nov-2024 04:50:51 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_673_20231206T233004Z_gen_g4_detsim_hitreco__20240510T042920Z_reco2_reco_data_2024-11-23T_031435Z.root"
23-Nov-2024 04:50:51 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/02/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_673_20231206T233004Z_gen_g4_detsim_hitreco__20240510T042920Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.9939        54.5711       712.22        20.5945       101.632        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00511841     0.149248      0.894003      0.0722582     0.180965        100    
reco:linecluster:LineCluster                           0.0444554     0.578972       9.52111      0.152382       1.3202         100    
reco:trajcluster:TrajCluster                           0.0127249      1.46597       48.6418      0.046344       5.56898        100    
reco:pandora:StandardPandora                            2.07926       4.74356       28.2918       3.48064       4.03129        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000264501    0.0106169     0.090257      0.0037622     0.0192733       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000357912    0.0173313      0.32392     0.00548116     0.0388012       100    
reco:pandoracalo:Calorimetry                          0.000211413    0.0380816      0.70865      0.002762      0.114021        100    
reco:pandorapid:Chi2ParticleID                        8.1915e-05    0.000356064   0.00160346    0.000287397   0.000248006      100    
reco:trkshowersplit:TrackShowerHits                   0.000203088    0.0331912      0.57748     0.00165476     0.096657        100    
reco:pmtrack:PMAlgTrackMaker                          0.00273843      4.64748       87.4627      0.383947       12.3575        100    
reco:pmtrackcalo:Calorimetry                          0.000231031    0.0195994     0.269507      0.002263      0.0509769       100    
reco:pmtrackpid:Chi2ParticleID                        7.3958e-05    0.000720085   0.00518784    0.000337396   0.00100162       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000474632     4.63699       70.7142       0.6538        10.9422        100    
reco:pmtrajfitcalo:Calorimetry                        0.000169213    0.0431422      1.14489      0.0034723     0.145829        100    
reco:pmtrajfitpid:Chi2ParticleID                      7.3447e-05    0.00047575    0.00446762    0.000347437   0.000517756      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00264083      6.11948       151.517      0.460402       18.5709        100    
reco:pmtracktccalo:Calorimetry                        0.000241005    0.0135233     0.235899     0.00242578     0.0332907       100    
reco:pmtracktcpid:Chi2ParticleID                       8.251e-05    0.00064202     0.0032342    0.000363654   0.000755971      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000463168     3.20807       39.7409      0.585436       7.00502        100    
reco:pmtrajfittccalo:Calorimetry                      0.00017135     0.037182      0.941786     0.00376275      0.11656        100    
reco:pmtrajfittcpid:Chi2ParticleID                    7.0357e-05    0.000906953   0.00625925    0.000415341   0.00127463       100    
reco:blurredcluster:BlurredClustering                 0.000984448     1.35228       11.2477      0.662909       1.81558        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0024005      21.7097       254.177       6.50709       41.1931        100    
reco:emshower:EMShower                                0.000446315    0.0546653     0.447508      0.0188419     0.0759839       100    
reco:cvnmap:CVNMapper                                  5.168e-05     0.0289169     0.133698      0.0309336     0.0222581       100    
reco:cvneva:CVNEvaluator                              3.0668e-05      1.74522       4.04903       2.07422       1.05242        100    
reco:energyrecnumu:EnergyReco                         0.00054277     0.0651081      2.07344     0.00828484     0.255885        100    
reco:energyrecnue:EnergyReco                          0.000303756   0.00355272     0.0578061    0.00101538    0.00789535       100    
reco:energyrecnc:EnergyReco                           0.00274173    0.00804524     0.0648338    0.00492736    0.00937776       100    
reco:energyrecnumurange:EnergyReco                    0.000444514    0.0552814      1.91163     0.00413141     0.226605        100    
reco:energyrecnumumcs:EnergyReco                      0.00275609     0.0600135      1.84589     0.00567485     0.230475        100    
reco:opdec:Deconvolution                               0.0431448     0.594972       2.97757      0.394429      0.579008        100    
reco:ophitspe:OpHitFinderDeco                          0.118888       2.68126       5.47117       2.58217       1.24473        100    
reco:opflash:OpFlashFinder                            0.000531305   0.00292212     0.0153083    0.00254383    0.00195008       100    
reco:rns:RandomNumberSaver                            3.3976e-05    9.73193e-05   0.00113817    7.4397e-05    0.000131248      100    
reco:anglereconue:NuAngularReco                       0.000284998   0.00234222     0.0405843    0.00078772    0.00518678       100    
reco:anglereconumu:NuAngularReco                      0.000157804   0.00191745     0.0292228    0.000445262   0.00445868       100    
reco:anglereconuepfps:NuAngularReco                   0.000259838   0.00225989     0.0273772    0.000681994   0.00441262       100    
reco:anglereconumupfps:NuAngularReco                  0.000215527   0.00268364     0.0917297    0.000587267   0.00961719       100    
[art]:TriggerResults:TriggerResultInserter            1.5054e-05    7.27873e-05   0.00414676    2.9769e-05    0.000409642      100    
end_path:out1:RootOutput                               4.86e-06     5.1005e-05    0.00407299    9.5335e-06    0.000404255      100    
end_path:out1:RootOutput(write)                        0.0337116      0.42959       1.57912      0.409692      0.285801        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8347.16 MB
  Peak resident set size usage (VmHWM): 2250.22 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_673_20231206T233004Z_gen_g4_detsim_hitreco__20240510T042920Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_673_20231206T233004Z_gen_g4_detsim_hitreco__20240510T042920Z_reco2_reco_2024-11-23T_031435Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_673_20231206T233004Z_gen_g4_detsim_hitreco__20240510T042920Z_reco2_reco_data_2024-11-23T_031435Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work