Jobsub ID 301783.26@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301783.26@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:24:30
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2024-11-23 06:29:29
From worker node	Hostname	b9p05p7648.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:26:48
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_855_20231203T045429Z_gen_g4_detsim_hitreco__20240508T065106Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3741s)
	CPU time	1h (3701s = 98%)
Outputting started	2024-11-23 06:29:10
Output files
Finished	2024-11-23 06:29:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

er@BeginModule  23-Nov-2024 07:28:58 CET run: 6413332 subRun: 1 event: 85599
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:28:58 CET run: 6413332 subRun: 1 event: 85599
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1481, 1601, 593
minwire 0: 404
minwire 1: 1241
minwire 2: 411
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00408544, 0.988206, 0.00770878, 
Output 1: 0.212088, 0.357803, 0.259804, 0.170305, 
Output 2: 0.0134547, 0.986545, 

771
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3051!
Begin processing the 100th record. run: 6413332 subRun: 1 event: 85600 at 23-Nov-2024 07:29:03 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:29:05 CET run: 6413332 subRun: 1 event: 85600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:29:05 CET run: 6413332 subRun: 1 event: 85600
Comparing two wires in the same plane: return failure
%MSG
335
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 707!
23-Nov-2024 07:29:08 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_855_20231203T045429Z_gen_g4_detsim_hitreco__20240508T065106Z_reco2_reco_data_2024-11-23T_052655Z.root"
23-Nov-2024 07:29:08 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/b8/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_855_20231203T045429Z_gen_g4_detsim_hitreco__20240508T065106Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.40262       35.1897       541.292       11.7798       78.424         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0167931     0.0502934     0.859087      0.0416515     0.0826265       100    
reco:linecluster:LineCluster                            0.15076      0.546227       11.9141      0.176094       1.7369         100    
reco:trajcluster:TrajCluster                           0.0481486      1.1067        31.6193      0.0631078      4.22729        100    
reco:pandora:StandardPandora                           0.958986       2.79043       44.182        1.32966       6.10801        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000183683   0.00820455     0.105896     0.00171365     0.0191576       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000243783    0.0265647      1.13222     0.00406282     0.114995        100    
reco:pandoracalo:Calorimetry                          0.000134762    0.0381476      1.27678     0.00164823     0.161427        100    
reco:pandorapid:Chi2ParticleID                        4.8001e-05    0.000213603   0.00166693    0.000159307   0.000197635      100    
reco:trkshowersplit:TrackShowerHits                   0.000138301    0.039059      0.965094     0.00112589     0.138932        100    
reco:pmtrack:PMAlgTrackMaker                          0.00136519      3.7403        106.114      0.355909       12.909         100    
reco:pmtrackcalo:Calorimetry                          8.0531e-05     0.0141337     0.324207     0.00139168     0.0464008       100    
reco:pmtrackpid:Chi2ParticleID                        3.4741e-05    0.000362705   0.00365688    0.000171228   0.000602787      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000247964     2.71667       39.6208      0.405224       7.10804        100    
reco:pmtrajfitcalo:Calorimetry                        9.1191e-05     0.0371811      1.07855     0.00227516      0.14923        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2981e-05    0.000218683   0.000976684   0.000168748   0.000182264      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00116935      4.28882       113.43       0.382961       14.9067        100    
reco:pmtracktccalo:Calorimetry                        8.9121e-05     0.0111335     0.293462     0.00128666     0.0380529       100    
reco:pmtracktcpid:Chi2ParticleID                       3.482e-05    0.00032421    0.00283655    0.000186068   0.00045497       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000233133     2.2231        42.0424      0.385474       5.73679        100    
reco:pmtrajfittccalo:Calorimetry                      8.4541e-05     0.024303      0.600377     0.00185901     0.0886091       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.765e-05    0.000473399   0.00564232    0.000199403   0.000851537      100    
reco:blurredcluster:BlurredClustering                  0.0100353      1.18638       7.44391      0.520341       1.63961        100    
reco:emtrkmichelid:EmTrackMichelId                     0.271125       12.483        133.777       3.38528       25.055         100    
reco:emshower:EMShower                                0.000254084    0.0351406     0.304736      0.0058017     0.0620131       100    
reco:cvnmap:CVNMapper                                  2.434e-05     0.0120119     0.0339836     0.0140985    0.00889635       100    
reco:cvneva:CVNEvaluator                              1.6001e-05     0.795676       1.95052       1.01652      0.497215        100    
reco:energyrecnumu:EnergyReco                         0.000366385    0.0345432     0.827615     0.00524797     0.109569        100    
reco:energyrecnue:EnergyReco                          0.000130922   0.00267535     0.0329893    0.000693644    0.0058793       100    
reco:energyrecnc:EnergyReco                           0.00162974    0.00556492     0.0372207    0.00363107    0.00644315       100    
reco:energyrecnumurange:EnergyReco                    0.000152402    0.0315121     0.768276     0.00324241     0.104312        100    
reco:energyrecnumumcs:EnergyReco                      0.00110249     0.0334017     0.765496     0.00497236     0.104478        100    
reco:opdec:Deconvolution                               0.0893589     0.290066      0.490965       0.28652      0.0893172       100    
reco:ophitspe:OpHitFinderDeco                          0.358609       2.31129       5.13721       2.20714      0.982398        100    
reco:opflash:OpFlashFinder                            0.000373855   0.00167387     0.0047602    0.00156334    0.000756846      100    
reco:rns:RandomNumberSaver                            2.1831e-05    5.06543e-05   0.000368064   4.7525e-05    3.34812e-05      100    
reco:anglereconue:NuAngularReco                       0.000182143   0.00222396     0.0327311    0.000500676   0.00553089       100    
reco:anglereconumu:NuAngularReco                      8.0351e-05    0.00189353     0.0305055    0.000279308   0.00528742       100    
reco:anglereconuepfps:NuAngularReco                   0.000130952   0.00210266     0.0309428    0.000431641   0.00544126       100    
reco:anglereconumupfps:NuAngularReco                  0.000110141   0.00196803     0.0309758    0.000347514   0.00533182       100    
[art]:TriggerResults:TriggerResultInserter             9.23e-06     1.67707e-05   5.2361e-05    1.64255e-05   4.79921e-06      100    
end_path:out1:RootOutput                               3.84e-06     8.80169e-06   2.4351e-05     8.97e-06      2.228e-06       100    
end_path:out1:RootOutput(write)                        0.0253552     0.293165       1.26636      0.290259       0.22732        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21692.4 MB
  Peak resident set size usage (VmHWM): 2262.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_855_20231203T045429Z_gen_g4_detsim_hitreco__20240508T065106Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_855_20231203T045429Z_gen_g4_detsim_hitreco__20240508T065106Z_reco2_reco_2024-11-23T_052655Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_855_20231203T045429Z_gen_g4_detsim_hitreco__20240508T065106Z_reco2_reco_data_2024-11-23T_052655Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work