justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.54@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.54@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-03-27 12:04:38
From worker nodeHostnametds420.pic.es
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:47:15
Input filesmonte-carlo-005746-000191
JobscriptExit code0
Real time17m (1021s)
CPU time16m (985s = 96%)
Max RSS bytes1717743616 (1638 MiB)
Outputting started2025-03-27 12:04:17
Output files
Finished2025-03-27 12:04:38
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

8699e+06
0 4.99024e+06
1 4.87786e+06
2 4.86343e+06
3 4.86008e+06
Now with regularization...
Begin: 3.71606e+06
0 3.71383e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 12:57:01 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_54_1743076046_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 12:57:01 CET  Opened input file "prod_muon_400MeV_protodunevd_1_358068_54_1743076046_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1693 traces, 1693 summary
Retagger: tagging trace set: gauss with 1252 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1252 traces tagged "gauss"
FrameSaver: q=142427 n=17351 tag=gauss
wclsFrameSaver: saving 1693 traces tagged "wiener"
FrameSaver: q=161518 n=15159 tag=wiener
27-Mar-2025 12:57:43 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_54_1743076046_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1658 traces, 1658 summary
Retagger: tagging trace set: gauss with 1213 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1213 traces tagged "gauss"
FrameSaver: q=144906 n=17139 tag=gauss
wclsFrameSaver: saving 1658 traces tagged "wiener"
FrameSaver: q=163372 n=14847 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1748 traces, 1748 summary
Retagger: tagging trace set: gauss with 1212 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1212 traces tagged "gauss"
FrameSaver: q=143305 n=17304 tag=gauss
wclsFrameSaver: saving 1748 traces tagged "wiener"
FrameSaver: q=163274 n=15023 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1592 traces, 1592 summary
Retagger: tagging trace set: gauss with 1167 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1167 traces tagged "gauss"
FrameSaver: q=136114 n=15799 tag=gauss
wclsFrameSaver: saving 1592 traces tagged "wiener"
FrameSaver: q=153907 n=13934 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1631 traces, 1631 summary
Retagger: tagging trace set: gauss with 1190 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1190 traces tagged "gauss"
FrameSaver: q=138022 n=16188 tag=gauss
wclsFrameSaver: saving 1631 traces tagged "wiener"
FrameSaver: q=156945 n=14158 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1595 traces, 1595 summary
Retagger: tagging trace set: gauss with 1225 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1225 traces tagged "gauss"
FrameSaver: q=152989 n=17152 tag=gauss
wclsFrameSaver: saving 1595 traces tagged "wiener"
FrameSaver: q=168983 n=15152 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1170 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1170 traces tagged "gauss"
FrameSaver: q=138316 n=15874 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=157670 n=13898 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1109 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1109 traces tagged "gauss"
FrameSaver: q=136557 n=15693 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=155796 n=13430 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1765 traces, 1765 summary
Retagger: tagging trace set: gauss with 1254 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1254 traces tagged "gauss"
FrameSaver: q=148939 n=17973 tag=gauss
wclsFrameSaver: saving 1765 traces tagged "wiener"
FrameSaver: q=168741 n=15650 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1645 traces, 1645 summary
Retagger: tagging trace set: gauss with 1229 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1229 traces tagged "gauss"
FrameSaver: q=137934 n=16406 tag=gauss
wclsFrameSaver: saving 1645 traces tagged "wiener"
FrameSaver: q=157678 n=14563 tag=wiener
27-Mar-2025 13:04:16 CET  Closed output file "prod_muon_400MeV_protodunevd_1_358068_54_1743076046_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:04:16 CET  Closed input file "prod_muon_400MeV_protodunevd_1_358068_54_1743076046_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       39.0639       42.8442       45.8585       42.9951       2.05171        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000399599   0.000593054   0.00106725    0.000513916   0.000194604      10     
reco:rns:RandomNumberSaver                     3.8571e-05    7.43466e-05   0.000343577   4.4851e-05     8.981e-05       10     
reco:wclsdatavd:WireCellToolkit                  38.6438       42.2315       45.4458        42.59        2.23749        10     
reco:gaushit:GausHitFinder                      0.0212413     0.023772      0.0280048     0.0233685    0.00206939       10     
reco:reco3d:SpacePointSolver                    0.0638081     0.0750717     0.0902427     0.0753847    0.00861214       10     
reco:hitpdune:DisambigFromSpacePoints           0.0020267     0.0027989    0.00541003    0.00257898    0.000911528      10     
reco:pandora:StandardPandora                    0.229686      0.455931       2.2539       0.262968      0.599543        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00172773    0.00313129    0.00631265     0.002491     0.00130083       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000209644   0.000314188   0.00101649    0.000233505   0.00023521       10     
reco:pandoraStdcalo:Calorimetry                0.00164503    0.00222103    0.00574724    0.00182002    0.00118699       10     
reco:pandoraGnocalo:GnocchiCalorimetry          0.0011702    0.00161697    0.00397722    0.00133292    0.000798334      10     
[art]:TriggerResults:TriggerResultInserter      1.501e-05    2.50887e-05   8.6232e-05    1.78355e-05   2.05898e-05      10     
end_path:out1:RootOutput                        3.57e-06      7.021e-06     2.833e-05     4.07e-06     7.32874e-06      10     
end_path:out1:RootOutput(write)                 0.0383207     0.0422524     0.0455066     0.0427196    0.00207453       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3413.65 MB
  Peak resident set size usage (VmHWM): 1601.96 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_54_1743076046_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076583.0141764, 'end_time': 1743077056.9584932}
justIN time: 2025-04-03 08:22:32 UTC       justIN version: 01.03.00