Jobsub ID 108787.101@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.101@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:31:27
From worker node	Hostname	comp11-07
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_762_20231201T213002Z_gen_g4_detsim_hitreco__20240508T051307Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4252s)
	CPU time	1h (3876s = 91%)
Outputting started	2024-11-23 06:31:12
Output files
Finished	2024-11-23 06:31:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:51 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:51 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:51 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:51 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:51 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:51 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:31:05 GMT run: 74496296 subRun: 1 event: 76300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 425, 455, 457
minwire 0: 876
minwire 1: 1173
minwire 2: 750
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0200928, 0.979735, 0.000171999, 
Output 1: 0.993861, 0.00592892, 0.000167532, 4.2147e-05, 
Output 2: 0.998604, 0.00139584, 

579
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1961!
23-Nov-2024 06:31:10 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_762_20231201T213002Z_gen_g4_detsim_hitreco__20240508T051307Z_reco2_reco_data_2024-11-23T_052022Z.root"
23-Nov-2024 06:31:10 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/b6/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_762_20231201T213002Z_gen_g4_detsim_hitreco__20240508T051307Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.519106       39.0545       1353.99       14.1881       136.094        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0104427     0.0713043     0.387217      0.0374434     0.0813591       100    
reco:linecluster:LineCluster                           0.0572138      1.08574       65.1463      0.145113       6.50423        100    
reco:trajcluster:TrajCluster                           0.029018       1.97644       161.068      0.0515801      16.0256        100    
reco:pandora:StandardPandora                           0.0483807      6.06726       394.531       1.95403       39.0495        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000137073   0.00736135      0.28501     0.00207759     0.0286987       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000180147    0.0123288     0.222558     0.00377105     0.0286598       100    
reco:pandoracalo:Calorimetry                          9.7056e-05     0.0418649      3.30656     0.00149742     0.328842        100    
reco:pandorapid:Chi2ParticleID                        3.5545e-05    0.000156223   0.00066345    0.000120849   0.000111946      100    
reco:trkshowersplit:TrackShowerHits                   0.000129191    0.0221338      1.43512     0.00107919     0.143699        100    
reco:pmtrack:PMAlgTrackMaker                          0.00121376      2.7316        118.744      0.229198       12.6499        100    
reco:pmtrackcalo:Calorimetry                          8.1736e-05     0.0102429     0.601544     0.00112639     0.059997        100    
reco:pmtrackpid:Chi2ParticleID                        3.3206e-05    0.000245965   0.00387839    0.000152249   0.000408781      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000205166     2.05581       81.5903      0.363392       8.57143        100    
reco:pmtrajfitcalo:Calorimetry                        6.1592e-05     0.0421429      3.26315     0.00177755     0.324571        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7312e-05    0.000177863   0.00113003    0.000150086    0.000136        100    
reco:pmtracktc:PMAlgTrackMaker                        0.00105336      2.89228       140.089      0.282372       14.1587        100    
reco:pmtracktccalo:Calorimetry                        7.2994e-05    0.00718254     0.311952     0.00109532     0.0318895       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9842e-05    0.000228672   0.00267478    0.000155423   0.000296379      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000231489     1.35748       52.1253      0.313031       5.26512        100    
reco:pmtrajfittccalo:Calorimetry                       6.298e-05     0.0211008      1.44887     0.00123837      0.14414        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5532e-05    0.000302347   0.00480167    0.00016629    0.000517956      100    
reco:blurredcluster:BlurredClustering                 8.0618e-05      1.07756       8.37772      0.608881       1.40937        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000129305     15.8221       310.537       5.84279       34.5269        100    
reco:emshower:EMShower                                0.000194818    0.0244854     0.486707     0.00787339     0.0530286       100    
reco:cvnmap:CVNMapper                                 2.0766e-05     0.0157577     0.0446491     0.0165892    0.00940037       100    
reco:cvneva:CVNEvaluator                              1.6442e-05     0.644356      0.873864      0.797598      0.323182        100    
reco:energyrecnumu:EnergyReco                         0.000292998    0.0298434      1.23792     0.00607013     0.124917        100    
reco:energyrecnue:EnergyReco                          0.000120574   0.00222071     0.0884625    0.000579657   0.00888362       100    
reco:energyrecnc:EnergyReco                           0.00188855    0.00393013     0.089541     0.00228979    0.00882635       100    
reco:energyrecnumurange:EnergyReco                    0.000196495    0.0270075      1.22127     0.00254069      0.12349        100    
reco:energyrecnumumcs:EnergyReco                      0.00182971     0.0282967      1.23759     0.00354682      0.12495        100    
reco:opdec:Deconvolution                               0.0489056     0.425917       2.54169       0.27842      0.461936        100    
reco:ophitspe:OpHitFinderDeco                          0.230058       2.31763       5.41109       2.27367      0.953383        100    
reco:opflash:OpFlashFinder                            0.000298007    0.0013871    0.00276854    0.00136205    0.000507781      100    
reco:rns:RandomNumberSaver                            2.0105e-05    3.3752e-05    0.00024129    2.9325e-05    2.28269e-05      100    
reco:anglereconue:NuAngularReco                       0.00015671    0.00180966     0.0790859    0.000455467   0.00792391       100    
reco:anglereconumu:NuAngularReco                      7.1603e-05    0.00156291     0.076254     0.000220705   0.00765277       100    
reco:anglereconuepfps:NuAngularReco                   0.000124303   0.00169651     0.0765552    0.000343105    0.0076911       100    
reco:anglereconumupfps:NuAngularReco                   9.809e-05    0.00165268     0.078746     0.000294236    0.0078933       100    
[art]:TriggerResults:TriggerResultInserter             8.76e-06     1.1913e-05     5.236e-05    1.09345e-05   4.91271e-06      100    
end_path:out1:RootOutput                               2.783e-06    5.17924e-06    1.445e-05     4.507e-06    2.01055e-06      100    
end_path:out1:RootOutput(write)                        0.0129306     0.222061       1.24273      0.219883      0.146813        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9608.97 MB
  Peak resident set size usage (VmHWM): 2449.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_762_20231201T213002Z_gen_g4_detsim_hitreco__20240508T051307Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_762_20231201T213002Z_gen_g4_detsim_hitreco__20240508T051307Z_reco2_reco_2024-11-23T_052022Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_762_20231201T213002Z_gen_g4_detsim_hitreco__20240508T051307Z_reco2_reco_data_2024-11-23T_052022Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work