justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.16@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.16@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:00:54
From worker nodeHostnametds420.pic.es
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:43:35
Input filesmonte-carlo-005746-000139
JobscriptExit code0
Real time16m (1018s)
CPU time16m (981s = 96%)
Max RSS bytes1720737792 (1641 MiB)
Outputting started2025-03-27 12:00:34
Output files
Finished2025-03-27 12:00:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

...
Begin: 2.44912e+07
0 2.31207e+07
1 2.30832e+07
2 2.30736e+07
Now with regularization...
Begin: 2.24377e+07
0 2.24307e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:53:31 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_16_1743075826_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:53:31 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358068_16_1743075826_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1641 traces, 1641 summary
Retagger: tagging trace set: gauss with 1126 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1126 traces tagged "gauss"
FrameSaver: q=148542 n=16519 tag=gauss
wclsFrameSaver: saving 1641 traces tagged "wiener"
FrameSaver: q=167937 n=14212 tag=wiener
27-Mar-2025 12:54:15 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_16_1743075826_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1677 traces, 1677 summary
Retagger: tagging trace set: gauss with 1232 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1232 traces tagged "gauss"
FrameSaver: q=138265 n=16639 tag=gauss
wclsFrameSaver: saving 1677 traces tagged "wiener"
FrameSaver: q=157078 n=14648 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1514 traces, 1514 summary
Retagger: tagging trace set: gauss with 1141 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1141 traces tagged "gauss"
FrameSaver: q=138720 n=15845 tag=gauss
wclsFrameSaver: saving 1514 traces tagged "wiener"
FrameSaver: q=154736 n=13904 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1520 traces, 1520 summary
Retagger: tagging trace set: gauss with 1061 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1061 traces tagged "gauss"
FrameSaver: q=135749 n=15220 tag=gauss
wclsFrameSaver: saving 1520 traces tagged "wiener"
FrameSaver: q=154413 n=13089 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1641 traces, 1641 summary
Retagger: tagging trace set: gauss with 1146 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1146 traces tagged "gauss"
FrameSaver: q=144063 n=16764 tag=gauss
wclsFrameSaver: saving 1641 traces tagged "wiener"
FrameSaver: q=161806 n=14267 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1614 traces, 1614 summary
Retagger: tagging trace set: gauss with 1165 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1165 traces tagged "gauss"
FrameSaver: q=140099 n=16441 tag=gauss
wclsFrameSaver: saving 1614 traces tagged "wiener"
FrameSaver: q=159588 n=14182 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1646 traces, 1646 summary
Retagger: tagging trace set: gauss with 1147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1147 traces tagged "gauss"
FrameSaver: q=137026 n=16157 tag=gauss
wclsFrameSaver: saving 1646 traces tagged "wiener"
FrameSaver: q=156818 n=13998 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1674 traces, 1674 summary
Retagger: tagging trace set: gauss with 1220 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1220 traces tagged "gauss"
FrameSaver: q=149885 n=16685 tag=gauss
wclsFrameSaver: saving 1674 traces tagged "wiener"
FrameSaver: q=170014 n=14512 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1582 traces, 1582 summary
Retagger: tagging trace set: gauss with 1127 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1127 traces tagged "gauss"
FrameSaver: q=137682 n=15801 tag=gauss
wclsFrameSaver: saving 1582 traces tagged "wiener"
FrameSaver: q=156053 n=13719 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1675 traces, 1675 summary
Retagger: tagging trace set: gauss with 1237 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1237 traces tagged "gauss"
FrameSaver: q=144386 n=17216 tag=gauss
wclsFrameSaver: saving 1675 traces tagged "wiener"
FrameSaver: q=162247 n=14984 tag=wiener
27-Mar-2025 13:00:33 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358068_16_1743075826_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:00:33 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358068_16_1743075826_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       36.5954       41.3891       46.636        41.6125       3.76494        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000321467   0.000533669   0.000987251   0.000493821   0.000186996      10     
reco:rns:RandomNumberSaver                     2.4661e-05    6.92296e-05   0.000346588   3.9576e-05    9.26564e-05      10     
reco:wclsdatavd:WireCellToolkit                  36.2152       40.7975       46.2619       40.099        3.72403        10     
reco:gaushit:GausHitFinder                      0.0149214     0.0192033     0.0259551     0.0192242    0.00377638       10     
reco:reco3d:SpacePointSolver                    0.0534127     0.0741258     0.129878      0.0677587     0.0211598       10     
reco:hitpdune:DisambigFromSpacePoints          0.00165176    0.00242734    0.00584818    0.00210728    0.00116875       10     
reco:pandora:StandardPandora                    0.189482      0.451338       2.46813      0.225553      0.672818        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00167467    0.00239282    0.00643484    0.00192055    0.00136912       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000158014   0.000273898   0.00093393    0.000211844   0.000221827      10     
reco:pandoraStdcalo:Calorimetry                0.00106058    0.00184296     0.0054331    0.00155872    0.00121914       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000794047   0.00134306    0.00419356    0.00105146    0.000974829      10     
[art]:TriggerResults:TriggerResultInserter      1.219e-05    2.50514e-05   7.9522e-05    1.5875e-05    2.14045e-05      10     
end_path:out1:RootOutput                         3.2e-06     5.9211e-06    2.3591e-05     4.09e-06     5.91214e-06      10     
end_path:out1:RootOutput(write)                 0.0294526     0.0348401     0.0403157     0.0347268    0.00452443       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3386.79 MB
  Peak resident set size usage (VmHWM): 1595.36 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_16_1743075826_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076372.4431577, 'end_time': 1743076833.8675969}
justIN time: 2025-04-03 08:22:33 UTC       justIN version: 01.03.00