Jobsub ID 301713.18@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301713.18@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:45:14
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 05:31:48
From worker node	Hostname	b9p12p1803.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:46:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1100_20231203T235451Z_gen_g4_detsim_hitreco__20240509T203308Z_reco2.root
Jobscript	Exit code	0
	Real time	44m (2678s)
	CPU time	46m (2762s = 103%)
Outputting started	2024-11-23 05:31:29
Output files
Finished	2024-11-23 05:31:48
List job events Wrapper job log
Jobscript log (last 10,000 characters)

run: 66009532 subRun: 1 event: 110100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:31:22 CET run: 66009532 subRun: 1 event: 110100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:31:22 CET run: 66009532 subRun: 1 event: 110100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:31:22 CET run: 66009532 subRun: 1 event: 110100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:31:24 CET run: 66009532 subRun: 1 event: 110100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:31:24 CET run: 66009532 subRun: 1 event: 110100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 45, 29, 36
minwire 0: 2364
minwire 1: 386
minwire 2: 2581
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 14, 213
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.994353, 0.000513567, 0.00513344, 
Output 1: 0.149077, 0.8385, 0.0118694, 0.000553471, 
Output 2: 0.99781, 0.00219, 

350
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1028!
23-Nov-2024 06:31:28 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1100_20231203T235451Z_gen_g4_detsim_hitreco__20240509T203308Z_reco2_reco_data_2024-11-23T_044657Z.root"
23-Nov-2024 06:31:28 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/6a/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1100_20231203T235451Z_gen_g4_detsim_hitreco__20240509T203308Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.603178       24.6265       359.108       9.36838       52.1431        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0177834     0.0417834     0.0821516     0.0380055     0.0142257       100    
reco:linecluster:LineCluster                           0.120347      0.294196       3.73139      0.176243      0.504095        100    
reco:trajcluster:TrajCluster                           0.0515509     0.767452       26.659       0.0651362      3.13599        100    
reco:pandora:StandardPandora                           0.0720389      1.91092       15.8424       1.33807       2.18375        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000144012   0.00546826     0.0982615    0.00177033     0.013642        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000159452    0.0113601     0.267269     0.00319572     0.0313386       100    
reco:pandoracalo:Calorimetry                          9.4721e-05     0.0164207     0.354677     0.00125905     0.0542392       100    
reco:pandorapid:Chi2ParticleID                         3.833e-05    0.00017384    0.00081262    0.000146752   0.000116991      100    
reco:trkshowersplit:TrackShowerHits                   0.000100081    0.0208527     0.495094     0.000871796    0.073082        100    
reco:pmtrack:PMAlgTrackMaker                          0.00100495      2.20006       46.9094      0.222082       6.75413        100    
reco:pmtrackcalo:Calorimetry                          6.0971e-05    0.00925596     0.254938     0.000891931    0.0317166       100    
reco:pmtrackpid:Chi2ParticleID                         2.478e-05    0.000281686    0.0022768    0.000160976   0.000391164      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000153282     1.79144       30.9167      0.308887       4.85417        100    
reco:pmtrajfitcalo:Calorimetry                        5.5081e-05     0.0174404     0.415299     0.00163045     0.0613638       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3441e-05    0.000206541   0.000991362   0.000160246   0.000168277      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000784689     2.88129       64.3538      0.225341       9.25853        100    
reco:pmtracktccalo:Calorimetry                        5.7651e-05    0.00622656     0.162261     0.000956351    0.0196555       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5051e-05    0.000259448   0.00185749    0.000154647   0.000328858      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000144761     1.38875       25.5941       0.2692        3.78373        100    
reco:pmtrajfittccalo:Calorimetry                      5.8771e-05     0.0117793      0.22572     0.00118454     0.0374635       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.385e-05    0.000337293   0.00323747    0.000180512   0.000508554      100    
reco:blurredcluster:BlurredClustering                 0.000158772    0.901426       8.58174      0.513926       1.33255        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000154472     8.85185       136.263       2.25041       20.0709        100    
reco:emshower:EMShower                                0.000218412    0.0327338     0.293583     0.00724159     0.0548162       100    
reco:cvnmap:CVNMapper                                  2.231e-05     0.0151021     0.0544391     0.0168111     0.0118482       100    
reco:cvneva:CVNEvaluator                               1.596e-05     0.710882       1.19752      0.923896      0.456139        100    
reco:energyrecnumu:EnergyReco                         0.000425425    0.0220796     0.371318     0.00409873     0.0575707       100    
reco:energyrecnue:EnergyReco                          0.000133891   0.00194388     0.046061     0.000543746   0.00522131       100    
reco:energyrecnc:EnergyReco                            0.0014697    0.00428096     0.0483457    0.00327249    0.00528597       100    
reco:energyrecnumurange:EnergyReco                    0.000180582    0.0195577     0.362573     0.00301917     0.0547985       100    
reco:energyrecnumumcs:EnergyReco                      0.00114787     0.0211988     0.358607     0.00374413     0.0547501       100    
reco:opdec:Deconvolution                               0.0571927     0.277182      0.466726      0.274271      0.0885626       100    
reco:ophitspe:OpHitFinderDeco                          0.159681       2.11899       4.56119       2.03589      0.999767        100    
reco:opflash:OpFlashFinder                            0.000149021   0.00157356    0.00476404    0.00154216    0.000723974      100    
reco:rns:RandomNumberSaver                             2.04e-05     4.91551e-05   0.000361464   4.60705e-05   3.30033e-05      100    
reco:anglereconue:NuAngularReco                       0.000174772   0.00164367     0.0415721    0.00043967    0.00502937       100    
reco:anglereconumu:NuAngularReco                      8.3531e-05    0.00139385     0.0399385    0.000214443   0.00487849       100    
reco:anglereconuepfps:NuAngularReco                   0.000131431   0.00156495     0.0392104    0.000398444    0.0047944       100    
reco:anglereconumupfps:NuAngularReco                  0.000113931   0.00144846     0.039195     0.000285619   0.00480334       100    
[art]:TriggerResults:TriggerResultInserter             8.64e-06     1.45627e-05   4.5251e-05    1.3545e-05    5.27276e-06      100    
end_path:out1:RootOutput                               3.55e-06     7.59667e-06    2.44e-05      7.375e-06    2.69913e-06      100    
end_path:out1:RootOutput(write)                        0.0140559     0.263381       1.34558      0.270045      0.196778        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21793.3 MB
  Peak resident set size usage (VmHWM): 2363.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1100_20231203T235451Z_gen_g4_detsim_hitreco__20240509T203308Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1100_20231203T235451Z_gen_g4_detsim_hitreco__20240509T203308Z_reco2_reco_2024-11-23T_044657Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1100_20231203T235451Z_gen_g4_detsim_hitreco__20240509T203308Z_reco2_reco_data_2024-11-23T_044657Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work