Jobsub ID 301682.66@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301682.66@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:44:28
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2024-11-23 04:24:55
From worker node	Hostname	node1g16.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:46:27
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_480_20231207T124608Z_gen_g4_detsim_hitreco__20240510T061511Z_reco2.root
Jobscript	Exit code	0
	Real time	38m (2292s)
	CPU time	32m (1976s = 86%)
Outputting started	2024-11-23 04:24:40
Output files
Finished	2024-11-23 04:24:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

o in-queue.
[2024-11-23 04:24:39.162303 +0000][Debug  ][ExDbgMsg          ][ 1134] [msg: 0x1e4bdf90] Assigned MsgHandler: 0x18aeaed0.
[2024-11-23 04:24:39.162382 +0000][Debug  ][ExDbgMsg          ][ 1134] [handler: 0x18aeaed0] Removed MsgHandler: 0x18aeaed0 from the in-queue.
[2024-11-23 04:24:39.162528 +0000][Debug  ][ExDbgMsg          ][ 1134] [dpmpool24.farm.particle.cz:20256] Calling MsgHandler: 0x18aeaed0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-11-23 04:24:39.162644 +0000][Debug  ][File              ][ 1134] [0x120ee990@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/85/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_480_20231207T124608Z_gen_g4_detsim_hitreco__20240510T061511Z_reco2.root?xrdcl.requuid=297748f5-77b3-4d37-ab8f-7e7bde313356] Close returned from dpmpool24.farm.particle.cz:20256 with: [SUCCESS] 
[2024-11-23 04:24:39.162700 +0000][Debug  ][ExDbgMsg          ][ 1134] [dpmpool24.farm.particle.cz:20256] Destroying MsgHandler: 0x18aeaed0.
23-Nov-2024 04:24:39 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/85/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_480_20231207T124608Z_gen_g4_detsim_hitreco__20240510T061511Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.11913       19.719        268.925        12.16        31.6962        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0334987      0.07354      0.572281      0.0702885     0.0654624       100    
reco:linecluster:LineCluster                           0.190819      0.436809       6.89084      0.337107       0.68255        100    
reco:trajcluster:TrajCluster                            0.09604      0.323807       13.2516      0.118463       1.32622        100    
reco:pandora:StandardPandora                            0.11383       2.33159       20.6476       2.0779        1.90172        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139971   0.00357738     0.0676282    0.00159997     0.008463        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00017498    0.00831048      0.18843     0.00256574     0.0214515       100    
reco:pandoracalo:Calorimetry                          0.000100399   0.00746351      0.35184     0.000991843    0.0360373       100    
reco:pandorapid:Chi2ParticleID                        3.9508e-05    0.000139755   0.00159064    0.000115056   0.000163733      100    
reco:trkshowersplit:TrackShowerHits                   9.3359e-05    0.00701133     0.372914     0.00066818     0.0378034       100    
reco:pmtrack:PMAlgTrackMaker                          0.00119071     0.942273       28.515       0.147852       3.31266        100    
reco:pmtrackcalo:Calorimetry                          8.1691e-05    0.00422084     0.0949919    0.000768623    0.0124375       100    
reco:pmtrackpid:Chi2ParticleID                         3.205e-05    0.000185149   0.00163342    0.000133587   0.000219951      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000189084    0.824814       15.4013      0.273012       2.07471        100    
reco:pmtrajfitcalo:Calorimetry                        6.6959e-05    0.00784283     0.334295     0.00124258     0.0346511       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9733e-05    0.000150137   0.000925106   0.000128391   0.000122188      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00106915      1.06157        33.75       0.158102       3.68218        100    
reco:pmtracktccalo:Calorimetry                        7.5175e-05    0.00367108     0.0837784    0.000748823    0.0102238       100    
reco:pmtracktcpid:Chi2ParticleID                      3.0605e-05    0.000178448   0.00135198    0.000125956   0.000186228      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000182688    0.713923       16.9704      0.215141       1.9303         100    
reco:pmtrajfittccalo:Calorimetry                      6.7176e-05     0.006587      0.254709     0.00126981     0.0274036       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8918e-05    0.000231513   0.00281697    0.000154911   0.000340494      100    
reco:blurredcluster:BlurredClustering                 6.8806e-05     0.902711       5.03264      0.467141       1.02171        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000118307     8.30462       120.774       3.79903       16.1342        100    
reco:emshower:EMShower                                0.000186325    0.0266164     0.253207     0.00594074     0.041297        100    
reco:cvnmap:CVNMapper                                 2.0578e-05     0.0229029     0.0438571     0.0276696      0.01376        100    
reco:cvneva:CVNEvaluator                              1.6466e-05     0.646446       1.1576        0.82701      0.355719        100    
reco:energyrecnumu:EnergyReco                         0.000282574    0.0267174     0.504499     0.00457443     0.0762316       100    
reco:energyrecnue:EnergyReco                          0.000144484   0.00123535     0.0217765    0.00050038    0.00266848       100    
reco:energyrecnc:EnergyReco                           0.00194802    0.00305364     0.0246235    0.00233108    0.00279925       100    
reco:energyrecnumurange:EnergyReco                    0.000190581    0.0238325      0.4909      0.00198908     0.0738333       100    
reco:energyrecnumumcs:EnergyReco                      0.00133991     0.0251672     0.492049      0.0029799     0.0737289       100    
reco:opdec:Deconvolution                               0.111246      0.423675      0.657746      0.439509      0.104319        100    
reco:ophitspe:OpHitFinderDeco                          0.353261       2.34013       4.09565       2.41938      0.822768        100    
reco:opflash:OpFlashFinder                            0.000178786   0.00126664    0.00401981    0.00129306    0.000550656      100    
reco:rns:RandomNumberSaver                            1.9243e-05    3.02526e-05   0.000276911   2.71805e-05   2.54816e-05      100    
reco:anglereconue:NuAngularReco                       0.000156562   0.000920782    0.018612     0.000373821   0.00224642       100    
reco:anglereconumu:NuAngularReco                      7.2079e-05    0.000740064    0.0179909    0.00021145    0.00218713       100    
reco:anglereconuepfps:NuAngularReco                   0.00011746    0.000857352    0.0184651    0.000304903   0.00223417       100    
reco:anglereconumupfps:NuAngularReco                  0.000101451   0.000809415    0.0182101    0.000270052   0.00222566       100    
[art]:TriggerResults:TriggerResultInserter            1.0124e-05    1.24975e-05    4.84e-05     1.16335e-05   3.95568e-06      100    
end_path:out1:RootOutput                               2.607e-06    4.33231e-06   1.4378e-05     3.616e-06    1.76724e-06      100    
end_path:out1:RootOutput(write)                       0.00861866     0.207606      0.676501      0.221888      0.113895        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12598.7 MB
  Peak resident set size usage (VmHWM): 2227.49 MB
====================================================================================================
Art has completed and will exit with status 0.
[2024-11-23 04:24:39.576567 +0000][Debug  ][JobMgr            ][ 1134] Stopping the job manager...
[2024-11-23 04:24:39.577568 +0000][Debug  ][JobMgr            ][ 1134] Job manager stopped
[2024-11-23 04:24:39.577595 +0000][Debug  ][TaskMgr           ][ 1134] Stopping the task manager...
[2024-11-23 04:24:39.578149 +0000][Debug  ][TaskMgr           ][ 1134] Task manager stopped
[2024-11-23 04:24:39.578616 +0000][Debug  ][Poller            ][ 1134] Stopping the poller...
[2024-11-23 04:24:39.578934 +0000][Debug  ][AsyncSock         ][ 1134] [dpmpool24.farm.particle.cz:20256.0] Closing the socket
[2024-11-23 04:24:39.578980 +0000][Debug  ][Poller            ][ 1134] <[::ffff:192.41.104.203]:39154><--><[::ffff:147.231.25.95]:20256> Removing socket from the poller
[2024-11-23 04:24:39.579142 +0000][Debug  ][PostMaster        ][ 1134] [dpmpool24.farm.particle.cz:20256] Destroying stream
[2024-11-23 04:24:39.579370 +0000][Debug  ][AsyncSock         ][ 1134] [dpmpool24.farm.particle.cz:20256.0] Closing the socket
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_480_20231207T124608Z_gen_g4_detsim_hitreco__20240510T061511Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_480_20231207T124608Z_gen_g4_detsim_hitreco__20240510T061511Z_reco2_reco_2024-11-23T_034631Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_480_20231207T124608Z_gen_g4_detsim_hitreco__20240510T061511Z_reco2_reco_data_2024-11-23T_034631Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work