justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108780.102@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108780.102@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 05:05:10
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2024-11-23 06:04:45
From worker nodeHostnamenode2a02.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 05:07:03
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_645_20231203T151709Z_gen_g4_detsim_hitreco__20240508T054755Z_reco2.root
JobscriptExit code0
Real time57m (3449s)
CPU time51m (3111s = 90%)
Outputting started2024-11-23 06:04:33
Output files
Finished2024-11-23 06:04:45
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

andle: 0x00000000) ) from out-queu to in-queue.
[2024-11-23 06:04:32.585558 +0000][Debug  ][ExDbgMsg          ][ 1134] [msg: 0x282a0180] Assigned MsgHandler: 0x11ebe2e0.
[2024-11-23 06:04:32.585704 +0000][Debug  ][ExDbgMsg          ][ 1134] [handler: 0x11ebe2e0] Removed MsgHandler: 0x11ebe2e0 from the in-queue.
[2024-11-23 06:04:32.585992 +0000][Debug  ][ExDbgMsg          ][ 1134] [ds28.farm.particle.cz:20862] Calling MsgHandler: 0x11ebe2e0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-11-23 06:04:32.586042 +0000][Debug  ][File              ][ 1134] [0x1070d700@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/60/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_645_20231203T151709Z_gen_g4_detsim_hitreco__20240508T054755Z_reco2.root?xrdcl.requuid=a4f46a4e-a94e-48c9-8744-9e7e369f6fd3] Close returned from ds28.farm.particle.cz:20862 with: [SUCCESS] 
[2024-11-23 06:04:32.586068 +0000][Debug  ][ExDbgMsg          ][ 1134] [ds28.farm.particle.cz:20862] Destroying MsgHandler: 0x11ebe2e0.
23-Nov-2024 06:04:32 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/60/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_645_20231203T151709Z_gen_g4_detsim_hitreco__20240508T054755Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.931662       31.4692       396.627       12.1225       66.8528        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0317606     0.0675332     0.187784      0.0729958     0.0223109       100    
reco:linecluster:LineCluster                           0.185399      0.546095       15.7505      0.305859       1.60286        100    
reco:trajcluster:TrajCluster                           0.0927682      1.10097       36.4175      0.106901       4.78492        100    
reco:pandora:StandardPandora                           0.102662       3.02777       74.3071       1.48427       8.31053        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136833   0.00666236     0.142411      0.0015738     0.0192657       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000179383    0.0101303     0.259859     0.00287271     0.0282019       100    
reco:pandoracalo:Calorimetry                          0.000100054    0.0188575     0.475046     0.00102192     0.0739637       100    
reco:pandorapid:Chi2ParticleID                        3.9622e-05    0.000153972   0.00121991    0.000114162   0.000153336      100    
reco:trkshowersplit:TrackShowerHits                   5.6036e-05     0.0214302     0.678861     0.000762978    0.0896592       100    
reco:pmtrack:PMAlgTrackMaker                           0.0012596      1.96966       40.4614      0.213984       6.42982        100    
reco:pmtrackcalo:Calorimetry                          8.2244e-05     0.0108553     0.291959     0.000972337    0.0386648       100    
reco:pmtrackpid:Chi2ParticleID                        3.2693e-05    0.000272463   0.00272855    0.000145755   0.000440191      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000201323     1.98903       46.0308      0.346851       6.25382        100    
reco:pmtrajfitcalo:Calorimetry                        6.9651e-05     0.0203913     0.523951     0.00143969     0.0785955       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9971e-05    0.000182763   0.000952421   0.000147629   0.000159633      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00111824      2.44914       48.5682      0.264127       8.23685        100    
reco:pmtracktccalo:Calorimetry                        7.6441e-05    0.00792243     0.155026     0.00106761     0.0240501       100    
reco:pmtracktcpid:Chi2ParticleID                      3.0637e-05    0.000256552   0.00212064    0.000151962   0.000366808      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000177513     1.38433       21.0786      0.324832       3.38783        100    
reco:pmtrajfittccalo:Calorimetry                      6.8323e-05     0.0161006     0.334187      0.0011836     0.0570607       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.9201e-05    0.000340119   0.00311073    0.000158072   0.000573917      100    
reco:blurredcluster:BlurredClustering                 6.2586e-05      1.25846       11.2198      0.629201       1.77723        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000108422     13.4957       151.529       3.82303       28.5954        100    
reco:emshower:EMShower                                0.000163985    0.0267385     0.196225     0.00750155     0.0396706       100    
reco:cvnmap:CVNMapper                                  1.921e-05     0.0157055     0.0567714     0.0137027     0.0120325       100    
reco:cvneva:CVNEvaluator                              1.5838e-05     0.589603      0.936265      0.727058      0.295907        100    
reco:energyrecnumu:EnergyReco                         0.000334727    0.0344047      1.27649     0.00436598     0.146667        100    
reco:energyrecnue:EnergyReco                          0.000154464   0.00223273     0.0460315    0.000504909   0.00646978       100    
reco:energyrecnc:EnergyReco                           0.00209466    0.00416366     0.0488281    0.00245972    0.00662307       100    
reco:energyrecnumurange:EnergyReco                    0.000208052    0.0321806      1.26663       0.00224      0.145902        100    
reco:energyrecnumumcs:EnergyReco                       0.0016123     0.0335489      1.26662     0.00289172      0.14598        100    
reco:opdec:Deconvolution                               0.0966672     0.445615      0.779732      0.452091       0.12677        100    
reco:ophitspe:OpHitFinderDeco                          0.278675       2.62679       4.78791       2.62146      0.975679        100    
reco:opflash:OpFlashFinder                            0.00012283     0.0013646    0.00349787    0.00131347    0.00062313       100    
reco:rns:RandomNumberSaver                            1.6583e-05    3.23068e-05   0.00025775    2.6856e-05    2.48598e-05      100    
reco:anglereconue:NuAngularReco                       0.000142679   0.00170864     0.0410532    0.000352391   0.00562302       100    
reco:anglereconumu:NuAngularReco                      7.3596e-05    0.00152572     0.0394767    0.00020484    0.00543999       100    
reco:anglereconuepfps:NuAngularReco                   0.000120039   0.00164452     0.0398044    0.000305578   0.00546606       100    
reco:anglereconumupfps:NuAngularReco                  0.000106621   0.00159523     0.0397574    0.000275058    0.0054645       100    
[art]:TriggerResults:TriggerResultInserter             9.586e-06    1.18628e-05   4.0352e-05    1.0793e-05    3.44271e-06      100    
end_path:out1:RootOutput                               2.329e-06    4.19499e-06   1.5972e-05    3.1635e-06    2.06526e-06      100    
end_path:out1:RootOutput(write)                       0.00831584     0.246476      0.935305      0.238431      0.160139        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8364.69 MB
  Peak resident set size usage (VmHWM): 2393.74 MB
====================================================================================================
Art has completed and will exit with status 0.
[2024-11-23 06:04:33.089780 +0000][Debug  ][JobMgr            ][ 1134] Stopping the job manager...
[2024-11-23 06:04:33.090531 +0000][Debug  ][JobMgr            ][ 1134] Job manager stopped
[2024-11-23 06:04:33.090574 +0000][Debug  ][TaskMgr           ][ 1134] Stopping the task manager...
[2024-11-23 06:04:33.090777 +0000][Debug  ][TaskMgr           ][ 1134] Task manager stopped
[2024-11-23 06:04:33.090894 +0000][Debug  ][Poller            ][ 1134] Stopping the poller...
[2024-11-23 06:04:33.090996 +0000][Debug  ][AsyncSock         ][ 1134] [ds28.farm.particle.cz:20862.0] Closing the socket
[2024-11-23 06:04:33.091147 +0000][Debug  ][Poller            ][ 1134] <[::ffff:192.41.105.3]:59910><--><[::ffff:147.231.25.166]:20862> Removing socket from the poller
[2024-11-23 06:04:33.091466 +0000][Debug  ][PostMaster        ][ 1134] [ds28.farm.particle.cz:20862] Destroying stream
[2024-11-23 06:04:33.091509 +0000][Debug  ][AsyncSock         ][ 1134] [ds28.farm.particle.cz:20862.0] Closing the socket
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_645_20231203T151709Z_gen_g4_detsim_hitreco__20240508T054755Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_645_20231203T151709Z_gen_g4_detsim_hitreco__20240508T054755Z_reco2_reco_2024-11-23T_050706Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_645_20231203T151709Z_gen_g4_detsim_hitreco__20240508T054755Z_reco2_reco_data_2024-11-23T_050706Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 12:08:15 UTC       justIN version: 01.01.09