Jobsub ID 301707.69@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301707.69@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:33:23
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 05:53:55
From worker node	Hostname	tdm004.pic.es
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:35:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_748_20231203T233254Z_gen_g4_detsim_hitreco__20240509T220218Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4696s)
	CPU time	1h (6135s = 130%)
Outputting started	2024-11-23 05:53:37
Output files
Finished	2024-11-23 05:53:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

onCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:26 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:53:31 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:53:31 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:53:31 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:53:31 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:53:32 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:53:32 CET run: 6419766 subRun: 1 event: 74900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 159, 159, 149
minwire 0: 1782
minwire 1: 832
minwire 2: 1827
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.2646, 0.702857, 0.0325426, 
Output 1: 0.183294, 0.137739, 0.242848, 0.436119, 
Output 2: 0.855714, 0.144286, 

523
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1725!
23-Nov-2024 06:53:36 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_748_20231203T233254Z_gen_g4_detsim_hitreco__20240509T220218Z_reco2_reco_data_2024-11-23T_043525Z.root"
23-Nov-2024 06:53:36 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/d1/d4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_748_20231203T233254Z_gen_g4_detsim_hitreco__20240509T220218Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.78667       46.1856       1941.6        8.0391        206.206        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00121485    0.00228284    0.00467294    0.00208312    0.000781501      100    
reco:linecluster:LineCluster                           0.0111113     0.654699       35.006       0.0178454      3.9244         100    
reco:trajcluster:TrajCluster                          0.00154222      2.84429       135.265     0.00953373      16.464         100    
reco:pandora:StandardPandora                            1.08629       11.5689       901.852       1.18746       89.8634        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000142872    0.0105643      0.39242     0.00135898     0.044098        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000198223    0.0158871     0.351765     0.00236758     0.0438122       100    
reco:pandoracalo:Calorimetry                          0.000103812    0.0463596      2.59025     0.00089183      0.27681        100    
reco:pandorapid:Chi2ParticleID                        4.0491e-05    0.000161763   0.00125534    0.000107242   0.000182553      100    
reco:trkshowersplit:TrackShowerHits                   0.000105782    0.0449742      1.70147     0.000691377    0.228353        100    
reco:pmtrack:PMAlgTrackMaker                           0.0013391      3.58438       125.584      0.134636       15.6503        100    
reco:pmtrackcalo:Calorimetry                          7.3261e-05     0.0217854      1.18747     0.000596091    0.124655        100    
reco:pmtrackpid:Chi2ParticleID                         2.992e-05    0.000304829    0.005465     0.000112167   0.000684376      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000212164     3.91539       216.945      0.186894       22.0839        100    
reco:pmtrajfitcalo:Calorimetry                        6.7742e-05     0.0470717      2.54165     0.00124141     0.274478        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.931e-05    0.000189636   0.00151168    0.000126167   0.000207242      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00125463      4.37973       162.982      0.148451       19.3712        100    
reco:pmtracktccalo:Calorimetry                        7.4112e-05     0.0148373     0.761115     0.000766564    0.0800328       100    
reco:pmtracktcpid:Chi2ParticleID                       2.972e-05    0.000293035   0.00468955    0.000123177   0.000606954      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000226973     2.51748       71.583       0.218241       9.68553        100    
reco:pmtrajfittccalo:Calorimetry                      7.0531e-05     0.032829       1.44836     0.000965002    0.167024        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.91e-05     0.000412294   0.00664972    0.000144443   0.000946809      100    
reco:blurredcluster:BlurredClustering                 0.00595897      1.10074       12.5632      0.457313       1.83379        100    
reco:emtrkmichelid:EmTrackMichelId                     0.243135       12.5933       280.355       2.53998       35.777         100    
reco:emshower:EMShower                                0.000281214     0.03242      0.562559     0.00696781     0.072813        100    
reco:cvnmap:CVNMapper                                  2.563e-05    0.00910193     0.046437     0.00740699    0.00961324       100    
reco:cvneva:CVNEvaluator                               1.767e-05     0.865063       2.23551       1.22423      0.661595        100    
reco:energyrecnumu:EnergyReco                         0.000287235    0.0376237      1.04461     0.00351361     0.129068        100    
reco:energyrecnue:EnergyReco                          0.000148742   0.00370691     0.153261     0.000453318    0.0167274       100    
reco:energyrecnc:EnergyReco                            0.0022146    0.00614997     0.160306     0.00277947     0.0174869       100    
reco:energyrecnumurange:EnergyReco                    0.000143983    0.0359693      1.02279     0.00254255     0.128404        100    
reco:energyrecnumumcs:EnergyReco                      0.00115189     0.0378302      1.02486     0.00315554     0.129672        100    
reco:opdec:Deconvolution                               0.0415387      0.1895       0.416513      0.184318      0.0816065       100    
reco:ophitspe:OpHitFinderDeco                          0.223617       1.29255       2.88073       1.24216      0.592563        100    
reco:opflash:OpFlashFinder                            0.000311596   0.00134868    0.00440836    0.00130106    0.000712343      100    
reco:rns:RandomNumberSaver                            1.7781e-05    3.24608e-05   0.000308915    2.418e-05    3.60357e-05      100    
reco:anglereconue:NuAngularReco                       0.000153162   0.00307501     0.141448     0.000308001    0.015211        100    
reco:anglereconumu:NuAngularReco                      7.4731e-05    0.00295125     0.145097     0.000183503    0.0155602       100    
reco:anglereconuepfps:NuAngularReco                   0.000124202   0.00308223     0.144942     0.00028513     0.0154881       100    
reco:anglereconumupfps:NuAngularReco                  0.000111482   0.00301749     0.144249     0.00026149     0.0154786       100    
[art]:TriggerResults:TriggerResultInserter             9.62e-06     1.27159e-05   4.9171e-05    1.10355e-05   6.57243e-06      100    
end_path:out1:RootOutput                               2.79e-06     4.64717e-06   1.8231e-05     3.485e-06    2.14202e-06      100    
end_path:out1:RootOutput(write)                        0.0356446     0.263458       2.02455      0.256581      0.263633        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39883.6 MB
  Peak resident set size usage (VmHWM): 2721.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_748_20231203T233254Z_gen_g4_detsim_hitreco__20240509T220218Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_748_20231203T233254Z_gen_g4_detsim_hitreco__20240509T220218Z_reco2_reco_2024-11-23T_043525Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_748_20231203T233254Z_gen_g4_detsim_hitreco__20240509T220218Z_reco2_reco_data_2024-11-23T_043525Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work