Jobsub ID 301699.18@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301699.18@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:14:53
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2024-11-23 05:05:36
From worker node	Hostname	node2a02.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:16:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root
Jobscript	Exit code	0
	Real time	48m (2926s)
	CPU time	43m (2592s = 88%)
Outputting started	2024-11-23 05:05:23
Output files
Finished	2024-11-23 05:05:36
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: 0x00000000) ) from out-queu to in-queue.
[2024-11-23 05:05:21.387608 +0000][Debug  ][ExDbgMsg          ][ 1135] [msg: 0x29007580] Assigned MsgHandler: 0x204a0800.
[2024-11-23 05:05:21.387907 +0000][Debug  ][ExDbgMsg          ][ 1135] [handler: 0x204a0800] Removed MsgHandler: 0x204a0800 from the in-queue.
[2024-11-23 05:05:21.387985 +0000][Debug  ][ExDbgMsg          ][ 1135] [ds26.farm.particle.cz:22672] Calling MsgHandler: 0x204a0800 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-11-23 05:05:21.388091 +0000][Debug  ][File              ][ 1135] [0x1048ee10@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f2/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root?xrdcl.requuid=66d0a529-766a-4602-a48c-4fc31606f632] Close returned from ds26.farm.particle.cz:22672 with: [SUCCESS] 
[2024-11-23 05:05:21.388155 +0000][Debug  ][ExDbgMsg          ][ 1135] [ds26.farm.particle.cz:22672] Destroying MsgHandler: 0x204a0800.
23-Nov-2024 05:05:21 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f2/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.35544       26.2703       581.972       11.1744       64.2122        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0317012     0.0684112      0.26802      0.0696527     0.0353616       100    
reco:linecluster:LineCluster                           0.185438      0.622476       29.5067      0.296294       2.90907        100    
reco:trajcluster:TrajCluster                           0.0928774      1.11087       76.0431      0.102739       7.60921        100    
reco:pandora:StandardPandora                           0.103364       2.19751       45.9919       1.47556       4.61416        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136479   0.00478287      0.11608     0.00147489     0.0130054       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000190493    0.0074164     0.067663     0.00265339     0.0126458       100    
reco:pandoracalo:Calorimetry                          0.000103089    0.0162202     0.816623     0.000812718    0.0883296       100    
reco:pandorapid:Chi2ParticleID                        4.0246e-05     0.0001487    0.00148703    0.000118397   0.000158613      100    
reco:trkshowersplit:TrackShowerHits                   8.0822e-05     0.0171014      1.05136     0.000499281    0.106274        100    
reco:pmtrack:PMAlgTrackMaker                          0.00128609      1.6507        68.3949      0.208124       7.23408        100    
reco:pmtrackcalo:Calorimetry                          8.5334e-05    0.00710084     0.199617     0.000774892    0.0261574       100    
reco:pmtrackpid:Chi2ParticleID                        3.3676e-05    0.000230425   0.00276213    0.000130828   0.000358699      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000203071     1.8069        64.6912       0.26456       7.04994        100    
reco:pmtrajfitcalo:Calorimetry                        7.0526e-05     0.0166846     0.797536     0.00120474     0.0865428       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.1923e-05    0.000169712   0.00088821    0.000140642   0.000133007      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113047      2.07781       74.7222      0.203951       8.47739        100    
reco:pmtracktccalo:Calorimetry                        8.1061e-05    0.00451485     0.100317     0.000828732    0.0127557       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2557e-05    0.000214482   0.00191201    0.000139639   0.000261531      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000189273     1.08534       26.0873      0.295427       3.09349        100    
reco:pmtrajfittccalo:Calorimetry                      6.9437e-05     0.0116846     0.517007     0.000976562    0.0552517       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.004e-05    0.000280777   0.00388313    0.000147266   0.000479634      100    
reco:blurredcluster:BlurredClustering                 6.1067e-05     0.998457       5.76167      0.446243       1.34446        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000121238     10.7346       177.623       3.02203       23.168         100    
reco:emshower:EMShower                                0.000180036    0.0282958     0.388839     0.00710609     0.0514429       100    
reco:cvnmap:CVNMapper                                 2.1268e-05     0.0174665     0.0588637     0.0132077     0.0148843       100    
reco:cvneva:CVNEvaluator                              1.7747e-05     0.553658       1.48382      0.727219      0.336767        100    
reco:energyrecnumu:EnergyReco                         0.000299897    0.0260502     0.432778      0.0046903     0.0698813       100    
reco:energyrecnue:EnergyReco                          0.000152478   0.00177401     0.0407533    0.000494379   0.00501071       100    
reco:energyrecnc:EnergyReco                           0.00210427    0.00378133     0.0445566    0.00243485    0.00522643       100    
reco:energyrecnumurange:EnergyReco                    0.00021024     0.0238047     0.427789     0.00219869     0.069251        100    
reco:energyrecnumumcs:EnergyReco                      0.00161407     0.025344      0.435915     0.00305688     0.069479        100    
reco:opdec:Deconvolution                               0.123213      0.410667      0.653008      0.408504      0.105381        100    
reco:ophitspe:OpHitFinderDeco                          0.560831       2.50222       4.53248       2.44551      0.913522        100    
reco:opflash:OpFlashFinder                            0.000352452   0.00136199    0.00461261    0.00124776    0.000631096      100    
reco:rns:RandomNumberSaver                            1.9291e-05    3.33586e-05   0.000267204   2.70115e-05   2.61456e-05      100    
reco:anglereconue:NuAngularReco                       0.000159235   0.00128361     0.0312034    0.00040087    0.00382662       100    
reco:anglereconumu:NuAngularReco                      7.9278e-05    0.00108498     0.0301915    0.000227554   0.00370698       100    
reco:anglereconuepfps:NuAngularReco                   0.000128295   0.00121725     0.0305114    0.000349161    0.0037486       100    
reco:anglereconumupfps:NuAngularReco                  0.00011294    0.00115815     0.0305354    0.000305078   0.00373836       100    
[art]:TriggerResults:TriggerResultInserter            1.0181e-05    1.37717e-05   4.9837e-05    1.23655e-05   4.72655e-06      100    
end_path:out1:RootOutput                               2.306e-06    4.02508e-06   1.8041e-05    2.8405e-06    2.17761e-06      100    
end_path:out1:RootOutput(write)                        0.0196867     0.229814       1.04117      0.237027        0.15          100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8184.75 MB
  Peak resident set size usage (VmHWM): 2202.23 MB
====================================================================================================
Art has completed and will exit with status 0.
[2024-11-23 05:05:21.951287 +0000][Debug  ][JobMgr            ][ 1135] Stopping the job manager...
[2024-11-23 05:05:21.952049 +0000][Debug  ][JobMgr            ][ 1135] Job manager stopped
[2024-11-23 05:05:21.952073 +0000][Debug  ][TaskMgr           ][ 1135] Stopping the task manager...
[2024-11-23 05:05:21.952416 +0000][Debug  ][TaskMgr           ][ 1135] Task manager stopped
[2024-11-23 05:05:21.953048 +0000][Debug  ][Poller            ][ 1135] Stopping the poller...
[2024-11-23 05:05:21.953204 +0000][Debug  ][AsyncSock         ][ 1135] [ds26.farm.particle.cz:22672.0] Closing the socket
[2024-11-23 05:05:21.953372 +0000][Debug  ][Poller            ][ 1135] <[::ffff:192.41.105.3]:53222><--><[::ffff:147.231.25.164]:22672> Removing socket from the poller
[2024-11-23 05:05:21.953461 +0000][Debug  ][PostMaster        ][ 1135] [ds26.farm.particle.cz:22672] Destroying stream
[2024-11-23 05:05:21.953812 +0000][Debug  ][AsyncSock         ][ 1135] [ds26.farm.particle.cz:22672.0] Closing the socket
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2_reco_2024-11-23T_041638Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2_reco_data_2024-11-23T_041638Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work