justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.100@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.100@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:27:32
From worker nodeHostnametd809.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:10:45
Input filesmonte-carlo-005746-000012
JobscriptExit code0
Real time16m (986s)
CPU time15m (958s = 97%)
Max RSS bytes1726971904 (1646 MiB)
Outputting started2025-03-27 12:27:12
Output files
Finished2025-03-27 12:27:32
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rization...
Begin: 2.96099e+07
0 2.94157e+07
1 2.94072e+07
Now with regularization...
Begin: 2.89532e+07
0 2.89531e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:20:21 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_100_1743077456_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:20:21 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358070_100_1743077456_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1567 traces, 1567 summary
Retagger: tagging trace set: gauss with 1155 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1155 traces tagged "gauss"
FrameSaver: q=137772 n=15884 tag=gauss
wclsFrameSaver: saving 1567 traces tagged "wiener"
FrameSaver: q=156770 n=13794 tag=wiener
27-Mar-2025 13:20:58 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_100_1743077456_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1674 traces, 1674 summary
Retagger: tagging trace set: gauss with 1201 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1201 traces tagged "gauss"
FrameSaver: q=160071 n=17231 tag=gauss
wclsFrameSaver: saving 1674 traces tagged "wiener"
FrameSaver: q=177842 n=15069 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1555 traces, 1555 summary
Retagger: tagging trace set: gauss with 1105 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1105 traces tagged "gauss"
FrameSaver: q=135727 n=15128 tag=gauss
wclsFrameSaver: saving 1555 traces tagged "wiener"
FrameSaver: q=153890 n=13204 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1568 traces, 1568 summary
Retagger: tagging trace set: gauss with 1078 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1078 traces tagged "gauss"
FrameSaver: q=135856 n=15358 tag=gauss
wclsFrameSaver: saving 1568 traces tagged "wiener"
FrameSaver: q=154614 n=13314 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1605 traces, 1605 summary
Retagger: tagging trace set: gauss with 1138 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1138 traces tagged "gauss"
FrameSaver: q=144798 n=16242 tag=gauss
wclsFrameSaver: saving 1605 traces tagged "wiener"
FrameSaver: q=163662 n=14081 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1619 traces, 1619 summary
Retagger: tagging trace set: gauss with 1181 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1181 traces tagged "gauss"
FrameSaver: q=137803 n=16073 tag=gauss
wclsFrameSaver: saving 1619 traces tagged "wiener"
FrameSaver: q=155887 n=14040 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1674 traces, 1674 summary
Retagger: tagging trace set: gauss with 1219 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1219 traces tagged "gauss"
FrameSaver: q=148815 n=17131 tag=gauss
wclsFrameSaver: saving 1674 traces tagged "wiener"
FrameSaver: q=166854 n=15030 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1690 traces, 1690 summary
Retagger: tagging trace set: gauss with 1188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1188 traces tagged "gauss"
FrameSaver: q=143061 n=17043 tag=gauss
wclsFrameSaver: saving 1690 traces tagged "wiener"
FrameSaver: q=163312 n=14760 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1608 traces, 1608 summary
Retagger: tagging trace set: gauss with 1214 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1214 traces tagged "gauss"
FrameSaver: q=139780 n=16436 tag=gauss
wclsFrameSaver: saving 1608 traces tagged "wiener"
FrameSaver: q=156509 n=14344 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1614 traces, 1614 summary
Retagger: tagging trace set: gauss with 1267 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1267 traces tagged "gauss"
FrameSaver: q=157526 n=17724 tag=gauss
wclsFrameSaver: saving 1614 traces tagged "wiener"
FrameSaver: q=173324 n=15709 tag=wiener
27-Mar-2025 13:27:12 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358070_100_1743077456_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:27:12 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358070_100_1743077456_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       36.1964       40.6865       44.757        40.3279       2.52537        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000326462   0.000458886   0.000981216   0.000370273   0.000194084      10     
reco:rns:RandomNumberSaver                      3.278e-05    6.22025e-05   0.000282412   3.7575e-05    7.34647e-05      10     
reco:wclsdatavd:WireCellToolkit                  34.4085       40.2167       44.4632       40.0005       2.79598        10     
reco:gaushit:GausHitFinder                      0.014791      0.0169696     0.0239157     0.0163566    0.00250586       10     
reco:reco3d:SpacePointSolver                    0.0498542     0.0604951     0.0781374     0.0588954    0.00867081       10     
reco:hitpdune:DisambigFromSpacePoints          0.00144807    0.00203643    0.00358343    0.00187527    0.000604592      10     
reco:pandora:StandardPandora                    0.123929       0.35023       1.65672      0.209199       0.43712        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00151621    0.00205913    0.00384234     0.001908     0.000618721      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000159171   0.000242397   0.000538323   0.000180126   0.000127115      10     
reco:pandoraStdcalo:Calorimetry                0.00103403    0.00143675    0.00316919    0.00117053    0.000608967      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000642904   0.000990912   0.00210355    0.000848521   0.000411942      10     
[art]:TriggerResults:TriggerResultInserter      1.124e-05    1.61641e-05    4.143e-05     1.327e-05    8.64687e-06      10     
end_path:out1:RootOutput                        3.04e-06      5.763e-06     1.677e-05     4.665e-06    3.90593e-06      10     
end_path:out1:RootOutput(write)                 0.0285856     0.031851      0.0415136     0.0303673    0.00361463       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3397.59 MB
  Peak resident set size usage (VmHWM): 1586.33 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_100_1743077456_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077982.7584565, 'end_time': 1743078432.5070171}
justIN time: 2025-04-03 08:28:35 UTC       justIN version: 01.03.00