Jobsub ID 108740.95@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108740.95@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:25:45
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 06:16:45
From worker node	Hostname	n255.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:34:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_176_20231204T100357Z_gen_g4_detsim_hitreco__20240509T213818Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6104s)
	CPU time	1h (5115s = 83%)
Outputting started	2024-11-23 06:16:28
Output files
Finished	2024-11-23 06:16:45
List job events Wrapper job log
Jobscript log (last 10,000 characters)

itter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:16 GMT run: 74517726 subRun: 1 event: 17700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:16 GMT run: 74517726 subRun: 1 event: 17700
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:16:21 GMT run: 74517726 subRun: 1 event: 17700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:16:21 GMT run: 74517726 subRun: 1 event: 17700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:16:21 GMT run: 74517726 subRun: 1 event: 17700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:16:21 GMT run: 74517726 subRun: 1 event: 17700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:16:21 GMT run: 74517726 subRun: 1 event: 17700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 122, 105, 113
minwire 0: 424
minwire 1: 1864
minwire 2: 380
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.960072, 0.0318023, 0.00812586, 
Output 1: 0.469419, 0.517282, 0.0130279, 0.000271031, 
Output 2: 0.988142, 0.0118578, 

365
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1054!
23-Nov-2024 06:16:26 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_176_20231204T100357Z_gen_g4_detsim_hitreco__20240509T213818Z_reco2_reco_data_2024-11-23T_043447Z.root"
23-Nov-2024 06:16:26 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/10/10/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_176_20231204T100357Z_gen_g4_detsim_hitreco__20240509T213818Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.61535       51.804        818.47        13.9232       126.385        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0345879     0.0870241      1.24044      0.076024      0.125795        100    
reco:linecluster:LineCluster                           0.343774       1.05484       16.3864       0.38413       2.67854        100    
reco:trajcluster:TrajCluster                           0.101365       2.58343       88.9609      0.122003       10.9529        100    
reco:pandora:StandardPandora                            1.04369       3.78086       53.5059       1.51232        8.734         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00014225     0.0116647      0.18268     0.00253843     0.0293224       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187175    0.0181733     0.322921     0.00493777     0.0396319       100    
reco:pandoracalo:Calorimetry                          0.000102294    0.0520979      1.10708     0.00209885     0.185101        100    
reco:pandorapid:Chi2ParticleID                        3.8814e-05    0.000220443   0.00133549    0.000156306   0.000206721      100    
reco:trkshowersplit:TrackShowerHits                   0.000107083    0.0626519      1.95958     0.00131185     0.247198        100    
reco:pmtrack:PMAlgTrackMaker                          0.00115528      5.04917       96.4861      0.341017       15.3679        100    
reco:pmtrackcalo:Calorimetry                          7.0513e-05     0.026157      0.462564     0.00133836     0.0812243       100    
reco:pmtrackpid:Chi2ParticleID                        2.8754e-05    0.000471748   0.00421227    0.000184956   0.000861939      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000199819     4.64439       122.537      0.498728       15.9716        100    
reco:pmtrajfitcalo:Calorimetry                        7.1706e-05     0.0523784      1.14812     0.00248635     0.185726        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9937e-05    0.000262779   0.00157656    0.000182777   0.000277071      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000996048     7.38557       183.817      0.366695       25.2653        100    
reco:pmtracktccalo:Calorimetry                         6.826e-05     0.0139188     0.204526      0.0011303     0.0356155       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7623e-05    0.000404875   0.00340584    0.000191012   0.000639462      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000162117     2.98671       38.8783      0.387343       7.57665        100    
reco:pmtrajfittccalo:Calorimetry                      6.9472e-05     0.0366354     0.652653     0.00174223     0.116325        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8474e-05    0.000574853   0.00488809    0.000210364   0.00108263       100    
reco:blurredcluster:BlurredClustering                  0.0054578      1.27764       10.3617      0.427361       1.90741        100    
reco:emtrkmichelid:EmTrackMichelId                     0.220977       18.7329       201.709       4.47322       40.3678        100    
reco:emshower:EMShower                                0.000288267    0.0429538     0.548464     0.00294507     0.101832        100    
reco:cvnmap:CVNMapper                                 2.4226e-05     0.0121542     0.0499484    0.00975366     0.0115105       100    
reco:cvneva:CVNEvaluator                              1.8104e-05     0.673627       1.43138      0.809635      0.418251        100    
reco:energyrecnumu:EnergyReco                         0.000310038    0.0788273      1.98182      0.0071322     0.246146        100    
reco:energyrecnue:EnergyReco                          0.000146257   0.00356019     0.0520742    0.000721964   0.00878463       100    
reco:energyrecnc:EnergyReco                           0.00181801    0.00597751     0.0544556    0.00350224    0.00902524       100    
reco:energyrecnumurange:EnergyReco                    0.00014272      0.07562       1.98331     0.00349949     0.245342        100    
reco:energyrecnumumcs:EnergyReco                      0.000855232    0.0771741      1.91414     0.00421521     0.241437        100    
reco:opdec:Deconvolution                               0.153084      0.456666      0.818228      0.453051       0.14839        100    
reco:ophitspe:OpHitFinderDeco                          0.442594       2.20932       5.09178       2.08326       1.00404        100    
reco:opflash:OpFlashFinder                            0.000367617   0.00164473    0.00431713     0.0015003    0.000829704      100    
reco:rns:RandomNumberSaver                            1.8214e-05    3.84093e-05   0.000304467   3.35935e-05   2.96938e-05      100    
reco:anglereconue:NuAngularReco                       0.000158261   0.00272972     0.0449366    0.000484584   0.00721536       100    
reco:anglereconumu:NuAngularReco                      7.6284e-05    0.00248129     0.0440998    0.000310103   0.00705511       100    
reco:anglereconuepfps:NuAngularReco                   0.000123895   0.00260604     0.0446264    0.000449512   0.00702311       100    
reco:anglereconumupfps:NuAngularReco                  0.000104589   0.00254507     0.0437503    0.000372281   0.00697072       100    
[art]:TriggerResults:TriggerResultInserter             9.048e-06    1.47554e-05   4.4555e-05    1.4212e-05    4.28308e-06      100    
end_path:out1:RootOutput                               3.006e-06    7.08384e-06   1.9807e-05    6.5225e-06    3.6415e-06       100    
end_path:out1:RootOutput(write)                        0.0179532     0.295731       1.35487      0.270345      0.255736        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38514.4 MB
  Peak resident set size usage (VmHWM): 2446.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_176_20231204T100357Z_gen_g4_detsim_hitreco__20240509T213818Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_176_20231204T100357Z_gen_g4_detsim_hitreco__20240509T213818Z_reco2_reco_2024-11-23T_043447Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_176_20231204T100357Z_gen_g4_detsim_hitreco__20240509T213818Z_reco2_reco_data_2024-11-23T_043447Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work