justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.70@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.70@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-03-27 12:32:37
From worker nodeHostnamemff2304.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:20:43
Input filesmonte-carlo-005746-000398
JobscriptExit code0
Real time11m (699s)
CPU time11m (687s = 98%)
Max RSS bytes1736388608 (1655 MiB)
Outputting started2025-03-27 12:32:23
Output files
Finished2025-03-27 12:32:37
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

82875e+06
0 4.13984e+06
1 4.09681e+06
2 4.09184e+06
3 4.09047e+06
Now with regularization...
Begin: 2.99439e+06
0 2.99348e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:27:09 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_70_1743078048_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:27:09 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_70_1743078048_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1575 traces, 1575 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=140524 n=15797 tag=gauss
wclsFrameSaver: saving 1575 traces tagged "wiener"
FrameSaver: q=159498 n=13708 tag=wiener
27-Mar-2025 13:27:37 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_70_1743078048_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1644 traces, 1644 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=137136 n=16214 tag=gauss
wclsFrameSaver: saving 1644 traces tagged "wiener"
FrameSaver: q=156625 n=14125 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1151 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1151 traces tagged "gauss"
FrameSaver: q=136582 n=16073 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=155429 n=13993 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=137356 n=16174 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=156257 n=13981 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1661 traces, 1661 summary
Retagger: tagging trace set: gauss with 1177 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1177 traces tagged "gauss"
FrameSaver: q=141724 n=16445 tag=gauss
wclsFrameSaver: saving 1661 traces tagged "wiener"
FrameSaver: q=161391 n=14251 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1554 traces, 1554 summary
Retagger: tagging trace set: gauss with 1095 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1095 traces tagged "gauss"
FrameSaver: q=137962 n=15493 tag=gauss
wclsFrameSaver: saving 1554 traces tagged "wiener"
FrameSaver: q=156813 n=13374 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1662 traces, 1662 summary
Retagger: tagging trace set: gauss with 1204 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1204 traces tagged "gauss"
FrameSaver: q=150488 n=16974 tag=gauss
wclsFrameSaver: saving 1662 traces tagged "wiener"
FrameSaver: q=168705 n=14758 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1655 traces, 1655 summary
Retagger: tagging trace set: gauss with 1162 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1162 traces tagged "gauss"
FrameSaver: q=146570 n=16915 tag=gauss
wclsFrameSaver: saving 1655 traces tagged "wiener"
FrameSaver: q=165441 n=14717 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1512 traces, 1512 summary
Retagger: tagging trace set: gauss with 1107 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1107 traces tagged "gauss"
FrameSaver: q=137860 n=15389 tag=gauss
wclsFrameSaver: saving 1512 traces tagged "wiener"
FrameSaver: q=155485 n=13278 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1586 traces, 1586 summary
Retagger: tagging trace set: gauss with 1182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1182 traces tagged "gauss"
FrameSaver: q=136174 n=15987 tag=gauss
wclsFrameSaver: saving 1586 traces tagged "wiener"
FrameSaver: q=154238 n=14030 tag=wiener
27-Mar-2025 13:32:22 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_70_1743078048_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:32:22 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_70_1743078048_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       27.9038       31.0713       32.7937       31.3577       1.56506        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000191777   0.000248793   0.000397894   0.000221092   6.82927e-05      10     
reco:rns:RandomNumberSaver                      1.924e-05    3.78032e-05   0.000182916   2.17605e-05   4.84026e-05      10     
reco:wclsdatavd:WireCellToolkit                  26.371        30.6986       32.5744       31.0988       1.84935        10     
reco:gaushit:GausHitFinder                      0.0112649     0.0121105     0.0128605     0.0120269    0.000544517      10     
reco:reco3d:SpacePointSolver                    0.0336563     0.0427843     0.0489714     0.0419263    0.00482366       10     
reco:hitpdune:DisambigFromSpacePoints          0.00117042    0.00151404    0.00277326     0.0013975    0.000428372      10     
reco:pandora:StandardPandora                    0.125549       0.28597       1.4314       0.164116      0.382036        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00118913    0.00167444    0.00286834    0.00155482    0.000440227      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000128824   0.000296708   0.00129784    0.00014257    0.000354454      10     
reco:pandoraStdcalo:Calorimetry                0.000818418   0.00111272    0.00234963    0.00102029    0.000422916      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000661262   0.000823104   0.00169363    0.00071895    0.000297157      10     
[art]:TriggerResults:TriggerResultInserter      8.721e-06    1.23195e-05   3.2071e-05    1.0395e-05    6.62043e-06      10     
end_path:out1:RootOutput                        2.57e-06     3.7052e-06    1.0271e-05     2.965e-06    2.2038e-06       10     
end_path:out1:RootOutput(write)                 0.0218853     0.0241163     0.026604      0.0242103     0.0012104       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3397.21 MB
  Peak resident set size usage (VmHWM): 1584.9 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_70_1743078048_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078407.7763362, 'end_time': 1743078742.9155667}
justIN time: 2025-04-03 08:21:43 UTC       justIN version: 01.03.00