Jobsub ID 301647.73@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301647.73@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:20:00
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2024-11-23 03:23:55
From worker node	Hostname	node2a02.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:34:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root
Jobscript	Exit code	0
	Real time	48m (2931s)
	CPU time	43m (2593s = 88%)
Outputting started	2024-11-23 03:23:43
Output files
Finished	2024-11-23 03:23:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

e: 0x00000000) ) from out-queu to in-queue.
[2024-11-23 03:23:40.532439 +0000][Debug  ][ExDbgMsg          ][ 1135] [msg: 0x132966c0] Assigned MsgHandler: 0x1a32b0d0.
[2024-11-23 03:23:40.532542 +0000][Debug  ][ExDbgMsg          ][ 1135] [handler: 0x1a32b0d0] Removed MsgHandler: 0x1a32b0d0 from the in-queue.
[2024-11-23 03:23:40.532974 +0000][Debug  ][ExDbgMsg          ][ 1135] [ds26.farm.particle.cz:22672] Calling MsgHandler: 0x1a32b0d0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-11-23 03:23:40.533034 +0000][Debug  ][File              ][ 1135] [0x10010120@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f2/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root?xrdcl.requuid=ca7d2eef-6506-4b26-a5e2-8eaf4fb64812] Close returned from ds26.farm.particle.cz:22672 with: [SUCCESS] 
[2024-11-23 03:23:40.533059 +0000][Debug  ][ExDbgMsg          ][ 1135] [ds26.farm.particle.cz:22672] Destroying MsgHandler: 0x1a32b0d0.
23-Nov-2024 03:23:40 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f2/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.29415       26.3071       580.904       11.2085        64.18         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0317619     0.0785452     0.615004      0.0701721     0.0769763       100    
reco:linecluster:LineCluster                           0.185121      0.628162       29.5045      0.303527       2.90886        100    
reco:trajcluster:TrajCluster                           0.0928856      1.1116        75.9831      0.102237       7.60329        100    
reco:pandora:StandardPandora                           0.103427       2.19278       45.4019       1.47897       4.55628        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000143294    0.0048032     0.118012     0.00150598     0.0131323       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000193354   0.00741772     0.0682931    0.00261283     0.0126622       100    
reco:pandoracalo:Calorimetry                          0.000105542    0.0161898     0.812983     0.000852686    0.0877489       100    
reco:pandorapid:Chi2ParticleID                        4.0684e-05    0.000150429   0.00158027    0.000120239   0.000167729      100    
reco:trkshowersplit:TrackShowerHits                   8.3403e-05     0.0170992      1.05146     0.000511533    0.106287        100    
reco:pmtrack:PMAlgTrackMaker                          0.00130564      1.65485       68.6349       0.20844       7.25853        100    
reco:pmtrackcalo:Calorimetry                          8.8066e-05    0.00710993     0.199337     0.000785065    0.0261399       100    
reco:pmtrackpid:Chi2ParticleID                        3.4014e-05    0.000226189   0.00239481    0.000134023   0.000331508      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000201301     1.80303       64.3115       0.26255       7.01635        100    
reco:pmtrajfitcalo:Calorimetry                        7.1906e-05     0.0163689     0.769176     0.00120594     0.0839173       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.1869e-05    0.000167112   0.000861902   0.000140372   0.00013242       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113354      2.08049       74.6641      0.204023       8.47687        100    
reco:pmtracktccalo:Calorimetry                        8.1667e-05    0.00454218     0.100234     0.000847684    0.0127565       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2875e-05    0.000212423   0.00188504    0.00013777    0.000258925      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000189331     1.09187       26.4023      0.294534       3.12761        100    
reco:pmtrajfittccalo:Calorimetry                      7.1508e-05      0.01168      0.516786     0.000938136    0.0552205       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.0219e-05    0.000279035   0.00366817    0.000146234   0.000463311      100    
reco:blurredcluster:BlurredClustering                 6.4697e-05      0.99849       5.73152      0.444791       1.3402         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000117542     10.7484       177.787       3.07068       23.235         100    
reco:emshower:EMShower                                0.00017634     0.0282731     0.393441     0.00529358     0.0522476       100    
reco:cvnmap:CVNMapper                                 1.9495e-05     0.0172059     0.0594526     0.013387      0.014724        100    
reco:cvneva:CVNEvaluator                              1.6915e-05     0.553993       1.19701      0.727071      0.333312        100    
reco:energyrecnumu:EnergyReco                         0.000293386    0.0257409      0.42822     0.00424496     0.069113        100    
reco:energyrecnue:EnergyReco                          0.000149614   0.00162079     0.0358947    0.000479025   0.00441884       100    
reco:energyrecnc:EnergyReco                           0.00211617    0.00362205     0.0392393     0.0024339     0.0046102       100    
reco:energyrecnumurange:EnergyReco                    0.000209059    0.0235263      0.42247     0.00218124     0.068257        100    
reco:energyrecnumumcs:EnergyReco                      0.00162006     0.0250807     0.422964     0.00299318     0.0685136       100    
reco:opdec:Deconvolution                               0.123208      0.413125      0.767206      0.409518      0.105842        100    
reco:ophitspe:OpHitFinderDeco                          0.560771       2.50332       4.53404       2.44542      0.916243        100    
reco:opflash:OpFlashFinder                            0.000336688   0.00134056    0.00472552    0.00130528    0.000622164      100    
reco:rns:RandomNumberSaver                             1.865e-05    3.37471e-05   0.000293832   2.99155e-05   2.81756e-05      100    
reco:anglereconue:NuAngularReco                       0.000156033   0.00127733     0.0311888    0.000386363   0.00385259       100    
reco:anglereconumu:NuAngularReco                      7.6213e-05    0.00105918     0.0296168    0.000221262   0.00363123       100    
reco:anglereconuepfps:NuAngularReco                   0.000128552    0.0011883     0.0298252    0.000334739   0.00366073       100    
reco:anglereconumupfps:NuAngularReco                  0.000114451   0.00113242     0.0299348    0.000304737   0.00365753       100    
[art]:TriggerResults:TriggerResultInserter             9.821e-06    1.4027e-05    5.7778e-05    1.21865e-05   5.57529e-06      100    
end_path:out1:RootOutput                               2.38e-06     4.21721e-06   1.8738e-05    2.9815e-06    2.31087e-06      100    
end_path:out1:RootOutput(write)                        0.0197247      0.22944       1.04067      0.239512      0.149485        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8184.8 MB
  Peak resident set size usage (VmHWM): 2208.01 MB
====================================================================================================
Art has completed and will exit with status 0.
[2024-11-23 03:23:41.098269 +0000][Debug  ][JobMgr            ][ 1135] Stopping the job manager...
[2024-11-23 03:23:41.098680 +0000][Debug  ][JobMgr            ][ 1135] Job manager stopped
[2024-11-23 03:23:41.098846 +0000][Debug  ][TaskMgr           ][ 1135] Stopping the task manager...
[2024-11-23 03:23:41.099139 +0000][Debug  ][TaskMgr           ][ 1135] Task manager stopped
[2024-11-23 03:23:41.099183 +0000][Debug  ][Poller            ][ 1135] Stopping the poller...
[2024-11-23 03:23:41.099637 +0000][Debug  ][AsyncSock         ][ 1135] [ds26.farm.particle.cz:22672.0] Closing the socket
[2024-11-23 03:23:41.099696 +0000][Debug  ][Poller            ][ 1135] <[::ffff:192.41.105.3]:53702><--><[::ffff:147.231.25.164]:22672> Removing socket from the poller
[2024-11-23 03:23:41.099801 +0000][Debug  ][PostMaster        ][ 1135] [ds26.farm.particle.cz:22672] Destroying stream
[2024-11-23 03:23:41.099939 +0000][Debug  ][AsyncSock         ][ 1135] [ds26.farm.particle.cz:22672.0] Closing the socket
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2_reco_2024-11-23T_023452Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_274_20231202T162321Z_gen_g4_detsim_hitreco__20240508T035637Z_reco2_reco_data_2024-11-23T_023452Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work