Jobsub ID 301692.38@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301692.38@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:05:14
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2024-11-23 05:28:28
From worker node	Hostname	node2a02.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:08:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_663_20231204T140404Z_gen_g4_detsim_hitreco__20240509T202353Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4801s)
	CPU time	1h (4443s = 92%)
Outputting started	2024-11-23 05:28:15
Output files
Finished	2024-11-23 05:28:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

to in-queue.
[2024-11-23 05:28:14.351426 +0000][Debug  ][ExDbgMsg          ][ 1135] [msg: 0x1b3df110] Assigned MsgHandler: 0x1cbf1250.
[2024-11-23 05:28:14.351512 +0000][Debug  ][ExDbgMsg          ][ 1135] [handler: 0x1cbf1250] Removed MsgHandler: 0x1cbf1250 from the in-queue.
[2024-11-23 05:28:14.351737 +0000][Debug  ][ExDbgMsg          ][ 1135] [dpmpool20.farm.particle.cz:24418] Calling MsgHandler: 0x1cbf1250 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-11-23 05:28:14.351851 +0000][Debug  ][File              ][ 1135] [0x104e15d0@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/df/57/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_663_20231204T140404Z_gen_g4_detsim_hitreco__20240509T202353Z_reco2.root?xrdcl.requuid=c02e0359-d8ca-4098-9243-567174811de9] Close returned from dpmpool20.farm.particle.cz:24418 with: [SUCCESS] 
[2024-11-23 05:28:14.351888 +0000][Debug  ][ExDbgMsg          ][ 1135] [dpmpool20.farm.particle.cz:24418] Destroying MsgHandler: 0x1cbf1250.
23-Nov-2024 05:28:14 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/df/57/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_663_20231204T140404Z_gen_g4_detsim_hitreco__20240509T202353Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.00564       44.8469       1306.33       12.7553       137.598        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0318996     0.0711488     0.487347      0.0699015     0.0531224       100    
reco:linecluster:LineCluster                           0.185635       1.54271       112.097      0.313422       11.1224        100    
reco:trajcluster:TrajCluster                           0.0929206      2.31733       133.153      0.109658       13.5861        100    
reco:pandora:StandardPandora                           0.109901       3.37709       122.436       1.51492       12.1731        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000140321    0.0100503     0.301966     0.00172543     0.0336733       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187501    0.017186      0.293287     0.00342171     0.045007        100    
reco:pandoracalo:Calorimetry                          0.000107464    0.0534846      3.61028     0.00107944     0.361305        100    
reco:pandorapid:Chi2ParticleID                        4.2272e-05    0.000178013   0.00138315    0.000131228   0.000178206      100    
reco:trkshowersplit:TrackShowerHits                   5.9646e-05     0.0366536      1.69149     0.000846874    0.179334        100    
reco:pmtrack:PMAlgTrackMaker                          0.00123282      4.1743        221.678      0.208268       22.3452        100    
reco:pmtrackcalo:Calorimetry                          8.2942e-05     0.0164443     0.612971     0.00113714     0.0668216       100    
reco:pmtrackpid:Chi2ParticleID                        3.3343e-05    0.000330524   0.00441681    0.000154451   0.000563062      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000211277     3.29951       138.674      0.342267       14.2779        100    
reco:pmtrajfitcalo:Calorimetry                        6.8662e-05     0.0564532      3.8147      0.00160384     0.381518        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.0943e-05    0.000212904   0.00106839    0.000168466   0.000192669      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00114727      4.10307       157.916       0.2454        16.755         100    
reco:pmtracktccalo:Calorimetry                        7.6678e-05     0.0125104     0.372014     0.00107789     0.0429985       100    
reco:pmtracktcpid:Chi2ParticleID                      3.1997e-05    0.000293182   0.00330103    0.000151805   0.000437014      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000199621     2.27655       80.2517      0.377465       8.35953        100    
reco:pmtrajfittccalo:Calorimetry                      6.9471e-05     0.0314092      1.60406     0.00139593     0.164325        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.0634e-05    0.000420786    0.0059849    0.000171687   0.000765221      100    
reco:blurredcluster:BlurredClustering                  7.016e-05      1.45221       10.8002      0.727886       2.02765        100    
reco:emtrkmichelid:EmTrackMichelId                    0.00011706      17.9897       305.062       4.25199       39.6012        100    
reco:emshower:EMShower                                0.000183318    0.0329805     0.429397     0.00714314     0.0610134       100    
reco:cvnmap:CVNMapper                                 2.0821e-05     0.0220953     0.0678795     0.0200031     0.0182876       100    
reco:cvneva:CVNEvaluator                              1.7988e-05     0.545165       1.06453      0.755001      0.360444        100    
reco:energyrecnumu:EnergyReco                         0.000271895    0.0432843     0.998454     0.00486955      0.13694        100    
reco:energyrecnue:EnergyReco                          0.000158822   0.00335136     0.0922475    0.000734871    0.0104992       100    
reco:energyrecnc:EnergyReco                           0.00212365    0.00532321     0.0955782     0.0025824     0.0107471       100    
reco:energyrecnumurange:EnergyReco                    0.000201943    0.0404931     0.992125     0.00248558      0.13396        100    
reco:energyrecnumumcs:EnergyReco                      0.00210579     0.0419569     0.992262      0.0037207     0.133887        100    
reco:opdec:Deconvolution                               0.104996      0.428853      0.718196      0.436869      0.125193        100    
reco:ophitspe:OpHitFinderDeco                          0.278126       2.57156       5.35152       2.56741       1.0578         100    
reco:opflash:OpFlashFinder                            0.000145798   0.00145812    0.00429451    0.00141291     0.0007339       100    
reco:rns:RandomNumberSaver                            1.8315e-05    3.71866e-05   0.000267343   3.1104e-05    2.62873e-05      100    
reco:anglereconue:NuAngularReco                       0.000154184   0.00271903     0.0829656    0.00055189     0.0093502       100    
reco:anglereconumu:NuAngularReco                      7.3888e-05     0.0024349     0.0803299    0.000275238   0.00909902       100    
reco:anglereconuepfps:NuAngularReco                   0.000123845   0.00258197     0.0804354    0.00042215    0.00911031       100    
reco:anglereconumupfps:NuAngularReco                  0.000107465   0.00251475     0.0803642    0.00036226    0.00910836       100    
[art]:TriggerResults:TriggerResultInserter             9.811e-06    1.41981e-05   4.6474e-05    1.42305e-05   4.44197e-06      100    
end_path:out1:RootOutput                               2.396e-06    5.33413e-06   1.8303e-05    5.2385e-06    2.3442e-06       100    
end_path:out1:RootOutput(write)                       0.00955764     0.259111       1.57158      0.248177      0.216733        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8429.51 MB
  Peak resident set size usage (VmHWM): 2443.97 MB
====================================================================================================
Art has completed and will exit with status 0.
[2024-11-23 05:28:14.901012 +0000][Debug  ][JobMgr            ][ 1135] Stopping the job manager...
[2024-11-23 05:28:14.901265 +0000][Debug  ][JobMgr            ][ 1135] Job manager stopped
[2024-11-23 05:28:14.901733 +0000][Debug  ][TaskMgr           ][ 1135] Stopping the task manager...
[2024-11-23 05:28:14.902223 +0000][Debug  ][TaskMgr           ][ 1135] Task manager stopped
[2024-11-23 05:28:14.902270 +0000][Debug  ][Poller            ][ 1135] Stopping the poller...
[2024-11-23 05:28:14.902535 +0000][Debug  ][AsyncSock         ][ 1135] [dpmpool20.farm.particle.cz:24418.0] Closing the socket
[2024-11-23 05:28:14.902687 +0000][Debug  ][Poller            ][ 1135] <[::ffff:192.41.105.3]:59058><--><[::ffff:147.231.25.91]:24418> Removing socket from the poller
[2024-11-23 05:28:14.902730 +0000][Debug  ][PostMaster        ][ 1135] [dpmpool20.farm.particle.cz:24418] Destroying stream
[2024-11-23 05:28:14.902789 +0000][Debug  ][AsyncSock         ][ 1135] [dpmpool20.farm.particle.cz:24418.0] Closing the socket
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_663_20231204T140404Z_gen_g4_detsim_hitreco__20240509T202353Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_663_20231204T140404Z_gen_g4_detsim_hitreco__20240509T202353Z_reco2_reco_2024-11-23T_040816Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_663_20231204T140404Z_gen_g4_detsim_hitreco__20240509T202353Z_reco2_reco_data_2024-11-23T_040816Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work