Jobsub ID 108735.61@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108735.61@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:18:14
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:03:59
From worker node	Hostname	b9p08p5554.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:35:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_560_20231205T102514Z_gen_g4_detsim_hitreco__20240510T040651Z_reco2.root
Jobscript	Exit code	0
	Real time	28m (1684s)
	CPU time	32m (1939s = 115%)
Outputting started	2024-11-23 05:03:43
Output files
Finished	2024-11-23 05:03:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

mination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.937536, 0.0140341, 0.0484295, 
Output 1: 0.167136, 0.603672, 0.156095, 0.0730964, 
Output 2: 0.213516, 0.786484, 

659
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2367!
Begin processing the 100th record. run: 50602933 subRun: 1 event: 56100 at 23-Nov-2024 06:03:38 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:03:40 CET run: 50602933 subRun: 1 event: 56100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:03:40 CET run: 50602933 subRun: 1 event: 56100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:03:40 CET run: 50602933 subRun: 1 event: 56100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 50602933 subRun: 1 event: 56100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 50602933 subRun: 1 event: 56100
Comparing two wires in the same plane: return failure
%MSG
159
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 387!
23-Nov-2024 06:03:42 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_560_20231205T102514Z_gen_g4_detsim_hitreco__20240510T040651Z_reco2_reco_data_2024-11-23T_043543Z.root"
23-Nov-2024 06:03:42 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/84/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_560_20231205T102514Z_gen_g4_detsim_hitreco__20240510T040651Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.521883       14.9881       110.207       7.65119       19.3914        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0164446     0.0466538     0.397162      0.0414622     0.0376414       100    
reco:linecluster:LineCluster                           0.0937497     0.183701      0.638429      0.163949      0.0722028       100    
reco:trajcluster:TrajCluster                           0.0470135     0.208694       2.33092      0.058133      0.435353        100    
reco:pandora:StandardPandora                           0.0545514      1.20354       4.38816       1.03067      0.592238        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000114781   0.00313772     0.0268408    0.00113389    0.00487388       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000151662   0.00764152     0.0964184    0.00258645     0.0146458       100    
reco:pandoracalo:Calorimetry                          8.6531e-05     0.0069904     0.157531     0.000772865    0.0191659       100    
reco:pandorapid:Chi2ParticleID                        3.2851e-05    0.000121542   0.000915972   0.000100846   9.92261e-05      100    
reco:trkshowersplit:TrackShowerHits                    5.439e-05    0.00791721     0.132668     0.000605628    0.020655        100    
reco:pmtrack:PMAlgTrackMaker                          0.000897081    0.981456       13.7877       0.14816       2.04511        100    
reco:pmtrackcalo:Calorimetry                          5.7051e-05    0.00365606     0.0623899    0.000737535   0.00903034       100    
reco:pmtrackpid:Chi2ParticleID                         2.434e-05    0.000176874   0.00122489    0.000109761   0.000198211      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000160342    0.826858       9.80457      0.238109       1.50596        100    
reco:pmtrajfitcalo:Calorimetry                         5.286e-05    0.00647489     0.103278     0.000852656    0.0150279       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.2631e-05    0.000133432   0.000523656   0.000108746   8.95428e-05      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00075481      1.28668       24.2273      0.185838       3.1741         100    
reco:pmtracktccalo:Calorimetry                        6.5061e-05    0.00300655     0.0431217    0.000742349    0.0063266       100    
reco:pmtracktcpid:Chi2ParticleID                       2.403e-05    0.00017244    0.00128582    0.000118757   0.00017466       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000130151    0.804924       7.37888      0.225963       1.44253        100    
reco:pmtrajfittccalo:Calorimetry                       5.344e-05    0.00566942     0.0774846    0.000973288    0.0138006       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1701e-05    0.000224729   0.00139302    0.000122217   0.000264837      100    
reco:blurredcluster:BlurredClustering                 9.3661e-05      0.88265       5.68638      0.540437       1.01519        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000104931     5.86509       45.1314       2.02108       9.13057        100    
reco:emshower:EMShower                                0.000169802     0.02144      0.308676     0.00318421     0.0431069       100    
reco:cvnmap:CVNMapper                                 1.9671e-05     0.007184      0.0285814    0.00526468    0.00666938       100    
reco:cvneva:CVNEvaluator                               1.416e-05     0.600357       1.03466      0.713213      0.328978        100    
reco:energyrecnumu:EnergyReco                         0.000267544    0.0157824     0.424886     0.00322285     0.049166        100    
reco:energyrecnue:EnergyReco                          0.000120081   0.00104627    0.00686055    0.000402771   0.00134512       100    
reco:energyrecnc:EnergyReco                           0.00156467    0.00287329    0.00979712    0.00214423    0.00154768       100    
reco:energyrecnumurange:EnergyReco                    0.000123741    0.0140445     0.415566     0.00182564     0.0482413       100    
reco:energyrecnumumcs:EnergyReco                      0.000647619    0.015663      0.430373     0.00290112     0.0497179       100    
reco:opdec:Deconvolution                               0.0493713     0.252526      0.416944      0.252584      0.0741209       100    
reco:ophitspe:OpHitFinderDeco                          0.140628       1.53458       3.39553       1.60259      0.647032        100    
reco:opflash:OpFlashFinder                            0.000132931   0.00110434    0.00436159     0.0010582    0.000543668      100    
reco:rns:RandomNumberSaver                             1.559e-05    2.9201e-05    0.000356835   2.5425e-05     3.376e-05       100    
reco:anglereconue:NuAngularReco                       0.000137162   0.000684612   0.00547632    0.000292969   0.00100442       100    
reco:anglereconumu:NuAngularReco                      5.8191e-05    0.000535569    0.0054703    0.000178072   0.000947155      100    
reco:anglereconuepfps:NuAngularReco                   0.000100552   0.00065419    0.00558793    0.000276148   0.000974831      100    
reco:anglereconumupfps:NuAngularReco                  8.1501e-05    0.000586362   0.00557102    0.000234713   0.000952864      100    
[art]:TriggerResults:TriggerResultInserter             7.56e-06     1.06395e-05    4.041e-05     9.43e-06     3.87008e-06      100    
end_path:out1:RootOutput                               2.73e-06     4.70456e-06    1.989e-05    4.3855e-06    1.99008e-06      100    
end_path:out1:RootOutput(write)                        0.0136677     0.181828      0.467679      0.180898       0.10121        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21582.5 MB
  Peak resident set size usage (VmHWM): 2158.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_560_20231205T102514Z_gen_g4_detsim_hitreco__20240510T040651Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_560_20231205T102514Z_gen_g4_detsim_hitreco__20240510T040651Z_reco2_reco_2024-11-23T_043543Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_560_20231205T102514Z_gen_g4_detsim_hitreco__20240510T040651Z_reco2_reco_data_2024-11-23T_043543Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work