justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358071.73@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358071.73@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:18:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-03-27 12:33:22
From worker nodeHostnamemff2304.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:21:18
Input filesmonte-carlo-005746-000079
JobscriptExit code0
Real time11m (712s)
CPU time11m (699s = 98%)
Max RSS bytes1741504512 (1660 MiB)
Outputting started2025-03-27 12:33:10
Output files
Finished2025-03-27 12:33:22
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

e+06
15 3.66585e+06
16 3.66136e+06
17 3.65689e+06
18 3.65222e+06
19 3.64777e+06
20 3.64352e+06
21 3.63959e+06
22 3.63624e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:27:47 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358071_73_1743078083_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:27:47 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358071_73_1743078083_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1632 traces, 1632 summary
Retagger: tagging trace set: gauss with 1154 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1154 traces tagged "gauss"
FrameSaver: q=138216 n=16295 tag=gauss
wclsFrameSaver: saving 1632 traces tagged "wiener"
FrameSaver: q=157233 n=14119 tag=wiener
27-Mar-2025 13:28:16 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358071_73_1743078083_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1558 traces, 1558 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=139088 n=15734 tag=gauss
wclsFrameSaver: saving 1558 traces tagged "wiener"
FrameSaver: q=157998 n=13646 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1635 traces, 1635 summary
Retagger: tagging trace set: gauss with 1144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1144 traces tagged "gauss"
FrameSaver: q=150278 n=16666 tag=gauss
wclsFrameSaver: saving 1635 traces tagged "wiener"
FrameSaver: q=169329 n=14487 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1545 traces, 1545 summary
Retagger: tagging trace set: gauss with 1106 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1106 traces tagged "gauss"
FrameSaver: q=137479 n=15648 tag=gauss
wclsFrameSaver: saving 1545 traces tagged "wiener"
FrameSaver: q=155092 n=13500 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1615 traces, 1615 summary
Retagger: tagging trace set: gauss with 1178 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1178 traces tagged "gauss"
FrameSaver: q=143238 n=16889 tag=gauss
wclsFrameSaver: saving 1615 traces tagged "wiener"
FrameSaver: q=160418 n=14774 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1649 traces, 1649 summary
Retagger: tagging trace set: gauss with 1199 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1199 traces tagged "gauss"
FrameSaver: q=151318 n=16730 tag=gauss
wclsFrameSaver: saving 1649 traces tagged "wiener"
FrameSaver: q=170476 n=14675 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1593 traces, 1593 summary
Retagger: tagging trace set: gauss with 1107 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1107 traces tagged "gauss"
FrameSaver: q=136752 n=15594 tag=gauss
wclsFrameSaver: saving 1593 traces tagged "wiener"
FrameSaver: q=154857 n=13441 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1731 traces, 1731 summary
Retagger: tagging trace set: gauss with 1223 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1223 traces tagged "gauss"
FrameSaver: q=154734 n=17482 tag=gauss
wclsFrameSaver: saving 1731 traces tagged "wiener"
FrameSaver: q=174778 n=15182 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1640 traces, 1640 summary
Retagger: tagging trace set: gauss with 1219 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1219 traces tagged "gauss"
FrameSaver: q=149348 n=17528 tag=gauss
wclsFrameSaver: saving 1640 traces tagged "wiener"
FrameSaver: q=165992 n=15379 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1600 traces, 1600 summary
Retagger: tagging trace set: gauss with 1130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1130 traces tagged "gauss"
FrameSaver: q=136169 n=15943 tag=gauss
wclsFrameSaver: saving 1600 traces tagged "wiener"
FrameSaver: q=154788 n=13706 tag=wiener
27-Mar-2025 13:33:10 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358071_73_1743078083_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:33:10 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358071_73_1743078083_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       28.1508       31.9754       33.7557       32.1092       1.56251        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000183547   0.000269618   0.000714155   0.000225573   0.000149238      10     
reco:rns:RandomNumberSaver                      2.064e-05    3.85953e-05   0.000190137   2.1896e-05    5.05182e-05      10     
reco:wclsdatavd:WireCellToolkit                  26.6417       31.5765       33.4877       31.8516       1.86989        10     
reco:gaushit:GausHitFinder                      0.0115288     0.0129363     0.0153354     0.0126938    0.00116069       10     
reco:reco3d:SpacePointSolver                    0.0397967     0.0630993     0.217955      0.0442649     0.0519726       10     
reco:hitpdune:DisambigFromSpacePoints          0.00106703    0.00155199    0.00304661    0.00138546    0.000535856      10     
reco:pandora:StandardPandora                    0.142039      0.290997       1.39313      0.173522      0.367556        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00107289    0.00157422    0.00278431    0.00149956    0.000452028      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000119994   0.000265272   0.000913511   0.00014345    0.000235555      10     
reco:pandoraStdcalo:Calorimetry                0.000752095   0.00110379    0.00233308    0.00100344    0.000417662      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000502417   0.000782085   0.00168329    0.000705889   0.000306887      10     
[art]:TriggerResults:TriggerResultInserter      9.23e-06     1.21653e-05   3.3341e-05     9.815e-06    7.07424e-06      10     
end_path:out1:RootOutput                        2.73e-06      3.781e-06     1.07e-05      3.015e-06    2.31453e-06      10     
end_path:out1:RootOutput(write)                 0.0230126     0.0242342     0.0269086     0.0237418    0.00136172       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3458.01 MB
  Peak resident set size usage (VmHWM): 1644.87 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358071_73_1743078083_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078445.7160456, 'end_time': 1743078790.3635547}
justIN time: 2025-04-03 08:23:25 UTC       justIN version: 01.03.00