Jobsub ID 108716.103@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108716.103@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:32:36
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 04:36:49
From worker node	Hostname	td830.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:40:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_720_20231203T052024Z_gen_g4_detsim_hitreco__20240508T051338Z_reco2.root
Jobscript	Exit code	0
	Real time	55m (3340s)
	CPU time	45m (2753s = 82%)
Outputting started	2024-11-23 04:36:23
Output files
Finished	2024-11-23 04:36:49
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:36:18 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:36:19 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:36:19 CET run: 6407968 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
307
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 864!
23-Nov-2024 05:36:22 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_720_20231203T052024Z_gen_g4_detsim_hitreco__20240508T051338Z_reco2_reco_data_2024-11-23T_034052Z.root"
23-Nov-2024 05:36:22 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/b1/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_720_20231203T052024Z_gen_g4_detsim_hitreco__20240508T051338Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.34956       28.5401       503.643       13.344        57.9688        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.033843      0.0516072     0.0744028     0.0435755     0.016734        100    
reco:linecluster:LineCluster                           0.308979      0.437634       8.01759      0.325566      0.773122        100    
reco:trajcluster:TrajCluster                           0.0997424     0.635193       24.7254      0.111158       2.69135        100    
reco:pandora:StandardPandora                            1.45102       3.11735       75.634        1.82416       8.00382        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000163662   0.00548285     0.0815646    0.00178278     0.0119588       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000243903    0.0115072     0.245837     0.00371316     0.0269107       100    
reco:pandoracalo:Calorimetry                          0.000131512    0.0135641     0.686159     0.00140615     0.0700417       100    
reco:pandorapid:Chi2ParticleID                         5.159e-05    0.000218518   0.00264579    0.000161382   0.000275824      100    
reco:trkshowersplit:TrackShowerHits                   0.000143652    0.0156842     0.516171     0.00084466     0.0636324       100    
reco:pmtrack:PMAlgTrackMaker                          0.00177263      2.20162       81.7874       0.23732       8.72587        100    
reco:pmtrackcalo:Calorimetry                          9.7911e-05    0.00756587     0.259381     0.00113807     0.0280814       100    
reco:pmtrackpid:Chi2ParticleID                        3.9391e-05    0.000340361   0.00432177    0.000175257   0.000544532      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000248043     1.89291       31.9706      0.426435       4.44688        100    
reco:pmtrajfitcalo:Calorimetry                        8.6271e-05     0.0151298     0.742263     0.00177091     0.075798        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6961e-05    0.000232966   0.00116953    0.000186647   0.000194098      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00150034      3.0956        130.553      0.269408       13.6341        100    
reco:pmtracktccalo:Calorimetry                        9.3531e-05    0.00558876     0.0840063    0.00135813     0.013151        100    
reco:pmtracktcpid:Chi2ParticleID                      3.9331e-05    0.000307675    0.0022996    0.000194522   0.000366494      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000242202     1.3469        25.3386      0.365432       3.12492        100    
reco:pmtrajfittccalo:Calorimetry                      8.6141e-05     0.0110211     0.403352     0.00136492     0.0432538       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.718e-05    0.000385617   0.00458132    0.000185627   0.000604341      100    
reco:blurredcluster:BlurredClustering                  0.0140944      1.08756        7.169       0.420223       1.49794        100    
reco:emtrkmichelid:EmTrackMichelId                     0.368724       10.3373       149.275       3.7445        20.7262        100    
reco:emshower:EMShower                                0.000332875    0.0313284     0.343444     0.00397056     0.0618961       100    
reco:cvnmap:CVNMapper                                  3.139e-05    0.00833862     0.0619531    0.00779848    0.00927508       100    
reco:cvneva:CVNEvaluator                               1.944e-05     0.859677       2.0255        1.17756      0.574974        100    
reco:energyrecnumu:EnergyReco                         0.000447235    0.0308831     0.543844     0.00532264     0.0862352       100    
reco:energyrecnue:EnergyReco                          0.000183982   0.00225943     0.0430875    0.000744495   0.00541022       100    
reco:energyrecnc:EnergyReco                           0.00257891    0.00570694     0.0511063    0.00408913    0.00599428       100    
reco:energyrecnumurange:EnergyReco                    0.000262974    0.027488      0.520003     0.00382376     0.0829695       100    
reco:energyrecnumumcs:EnergyReco                      0.00152712     0.0293451      0.51504     0.00449759     0.0819355       100    
reco:opdec:Deconvolution                               0.160102      0.449128      0.720213      0.452031      0.122325        100    
reco:ophitspe:OpHitFinderDeco                          0.513154       2.48142       4.5111        2.52036      0.962603        100    
reco:opflash:OpFlashFinder                            0.000327804   0.00170837     0.0100757     0.0015483    0.00116628       100    
reco:rns:RandomNumberSaver                            1.7581e-05    3.95284e-05   0.000756159   2.8016e-05    7.27567e-05      100    
reco:anglereconue:NuAngularReco                       0.000178022   0.00148139     0.0226043    0.000429035   0.00343405       100    
reco:anglereconumu:NuAngularReco                      8.8501e-05    0.00127285     0.0228096    0.000288374   0.00337982       100    
reco:anglereconuepfps:NuAngularReco                   0.000159082   0.00147379     0.0230469    0.000398075   0.00346237       100    
reco:anglereconumupfps:NuAngularReco                  0.000149511   0.00139504     0.023527     0.000337884   0.00350597       100    
[art]:TriggerResults:TriggerResultInserter             1.001e-05    1.65577e-05   0.000131432   1.43205e-05   1.27826e-05      100    
end_path:out1:RootOutput                               2.93e-06     6.67584e-06    3.934e-05      4.6e-06     5.61117e-06      100    
end_path:out1:RootOutput(write)                        0.0304051     0.313947       1.14289      0.337379      0.210434        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21196.2 MB
  Peak resident set size usage (VmHWM): 2048.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_720_20231203T052024Z_gen_g4_detsim_hitreco__20240508T051338Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_720_20231203T052024Z_gen_g4_detsim_hitreco__20240508T051338Z_reco2_reco_2024-11-23T_034052Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_720_20231203T052024Z_gen_g4_detsim_hitreco__20240508T051338Z_reco2_reco_data_2024-11-23T_034052Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work