Jobsub ID 301644.104@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301644.104@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:16:43
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2024-11-23 03:14:49
From worker node	Hostname	node2a02.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:25:47
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_66_20231202T195941Z_gen_g4_detsim_hitreco__20240508T062410Z_reco2.root
Jobscript	Exit code	0
	Real time	48m (2929s)
	CPU time	42m (2570s = 87%)
Outputting started	2024-11-23 03:14:36
Output files
Finished	2024-11-23 03:14:49
List job events Wrapper job log
Jobscript log (last 10,000 characters)

om out-queu to in-queue.
[2024-11-23 03:14:35.755295 +0000][Debug  ][ExDbgMsg          ][ 1136] [msg: 0x20d6a480] Assigned MsgHandler: 0x1c9bf1d0.
[2024-11-23 03:14:35.755574 +0000][Debug  ][ExDbgMsg          ][ 1136] [handler: 0x1c9bf1d0] Removed MsgHandler: 0x1c9bf1d0 from the in-queue.
[2024-11-23 03:14:35.755795 +0000][Debug  ][ExDbgMsg          ][ 1136] [dpmpool20.farm.particle.cz:24418] Calling MsgHandler: 0x1c9bf1d0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-11-23 03:14:35.756178 +0000][Debug  ][File              ][ 1136] [0xfb191a0@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/33/1b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_66_20231202T195941Z_gen_g4_detsim_hitreco__20240508T062410Z_reco2.root?xrdcl.requuid=ea6d23f4-ed97-4a98-8e40-5949d848ada8] Close returned from dpmpool20.farm.particle.cz:24418 with: [SUCCESS] 
[2024-11-23 03:14:35.756216 +0000][Debug  ][ExDbgMsg          ][ 1136] [dpmpool20.farm.particle.cz:24418] Destroying MsgHandler: 0x1c9bf1d0.
23-Nov-2024 03:14:35 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/33/1b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_66_20231202T195941Z_gen_g4_detsim_hitreco__20240508T062410Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.16763       26.1213       311.827       12.0531       44.7865        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0316489     0.0624783     0.254745      0.0664776     0.0273307       100    
reco:linecluster:LineCluster                           0.286434      0.484353       12.2498      0.312066       1.20851        100    
reco:trajcluster:TrajCluster                           0.0931187     0.727671       30.1971      0.104768       3.15924        100    
reco:pandora:StandardPandora                            1.37114       2.23312       24.7806       1.6322        2.71354        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139057   0.00498176     0.0685559    0.00152847      0.01083        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000186149    0.0102347     0.180098     0.00282882     0.0252974       100    
reco:pandoracalo:Calorimetry                          0.000109568    0.0118332     0.296074     0.000991864    0.0404175       100    
reco:pandorapid:Chi2ParticleID                        4.1264e-05    0.000160783   0.00160205    0.000122023   0.000173014      100    
reco:trkshowersplit:TrackShowerHits                   0.000116378    0.0139405     0.427712     0.000788224    0.0533524       100    
reco:pmtrack:PMAlgTrackMaker                          0.00125373      1.6317        55.4273      0.182657       5.98696        100    
reco:pmtrackcalo:Calorimetry                          8.5982e-05    0.00660257     0.135225     0.000911282    0.0197341       100    
reco:pmtrackpid:Chi2ParticleID                        3.3052e-05    0.000242765   0.00188954    0.00013732    0.000311328      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000216086     1.38273       18.3374      0.346887       3.25303        100    
reco:pmtrajfitcalo:Calorimetry                        7.2644e-05     0.0123605      0.2865       0.001278      0.0411262       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.1067e-05    0.000171213   0.000688907   0.000133715   0.000127051      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113545      1.84149       47.9403      0.242722       5.67092        100    
reco:pmtracktccalo:Calorimetry                        8.1055e-05    0.00518262     0.131458     0.00101917     0.015896        100    
reco:pmtracktcpid:Chi2ParticleID                      3.2275e-05    0.000230188   0.00159857    0.000150918   0.000264177      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000192566     1.02984       13.7657      0.253424       2.24123        100    
reco:pmtrajfittccalo:Calorimetry                      7.2066e-05    0.00989217     0.178572     0.00101373     0.0301009       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.0794e-05    0.000299949    0.0024891    0.000152315   0.000423198      100    
reco:blurredcluster:BlurredClustering                  0.0127272      1.19982       7.48613      0.520144       1.54343        100    
reco:emtrkmichelid:EmTrackMichelId                     0.275845       11.431        115.306       3.60833       20.864         100    
reco:emshower:EMShower                                0.000255268    0.0279237     0.157168      0.0135502     0.0388263       100    
reco:cvnmap:CVNMapper                                 2.2265e-05     0.0258923     0.0528869     0.0289133     0.0175891       100    
reco:cvneva:CVNEvaluator                              1.8346e-05     0.586325       1.02134      0.756049      0.332506        100    
reco:energyrecnumu:EnergyReco                         0.000268968    0.0499331      3.36003     0.00447194     0.334419        100    
reco:energyrecnue:EnergyReco                          0.000147368   0.00166276     0.0228438    0.000554448   0.00358608       100    
reco:energyrecnc:EnergyReco                           0.00210535    0.00371681     0.0257999    0.00252333    0.00378233       100    
reco:energyrecnumurange:EnergyReco                    0.000197648    0.0477227      3.35473     0.00221865     0.334059        100    
reco:energyrecnumumcs:EnergyReco                      0.00118447     0.0491066      3.3579      0.00312973     0.334242        100    
reco:opdec:Deconvolution                               0.205325       0.42649      0.689434      0.432286      0.102765        100    
reco:ophitspe:OpHitFinderDeco                          0.620497       2.55084       5.79672       2.54061      0.948303        100    
reco:opflash:OpFlashFinder                            0.000335584   0.00145047    0.00474114    0.00138622    0.000650538      100    
reco:rns:RandomNumberSaver                            2.0197e-05    3.9145e-05    0.000255143    3.475e-05    2.54871e-05      100    
reco:anglereconue:NuAngularReco                       0.000154778   0.00126727     0.0195493    0.000400573   0.00298781       100    
reco:anglereconumu:NuAngularReco                      7.7591e-05    0.00104599     0.0187742    0.000223514   0.00290557       100    
reco:anglereconuepfps:NuAngularReco                   0.000130971   0.00118429     0.0188855    0.000355563   0.00292869       100    
reco:anglereconumupfps:NuAngularReco                  0.000112323   0.00112361     0.0188249    0.000305519   0.00291145       100    
[art]:TriggerResults:TriggerResultInserter            1.0223e-05    1.46063e-05   4.2371e-05    1.4385e-05    3.97945e-06      100    
end_path:out1:RootOutput                               2.332e-06    5.3681e-06    1.6364e-05    5.3945e-06    2.40612e-06      100    
end_path:out1:RootOutput(write)                        0.022227      0.243434      0.805178      0.246749      0.145149        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8126.69 MB
  Peak resident set size usage (VmHWM): 2151.61 MB
====================================================================================================
Art has completed and will exit with status 0.
[2024-11-23 03:14:36.336488 +0000][Debug  ][JobMgr            ][ 1136] Stopping the job manager...
[2024-11-23 03:14:36.337183 +0000][Debug  ][JobMgr            ][ 1136] Job manager stopped
[2024-11-23 03:14:36.337207 +0000][Debug  ][TaskMgr           ][ 1136] Stopping the task manager...
[2024-11-23 03:14:36.338244 +0000][Debug  ][TaskMgr           ][ 1136] Task manager stopped
[2024-11-23 03:14:36.338307 +0000][Debug  ][Poller            ][ 1136] Stopping the poller...
[2024-11-23 03:14:36.338552 +0000][Debug  ][AsyncSock         ][ 1136] [dpmpool20.farm.particle.cz:24418.0] Closing the socket
[2024-11-23 03:14:36.338626 +0000][Debug  ][Poller            ][ 1136] <[::ffff:192.41.105.3]:58074><--><[::ffff:147.231.25.91]:24418> Removing socket from the poller
[2024-11-23 03:14:36.338741 +0000][Debug  ][PostMaster        ][ 1136] [dpmpool20.farm.particle.cz:24418] Destroying stream
[2024-11-23 03:14:36.338780 +0000][Debug  ][AsyncSock         ][ 1136] [dpmpool20.farm.particle.cz:24418.0] Closing the socket
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_66_20231202T195941Z_gen_g4_detsim_hitreco__20240508T062410Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_66_20231202T195941Z_gen_g4_detsim_hitreco__20240508T062410Z_reco2_reco_2024-11-23T_022550Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_66_20231202T195941Z_gen_g4_detsim_hitreco__20240508T062410Z_reco2_reco_data_2024-11-23T_022550Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work