justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.4@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.4@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:23:48
From worker nodeHostnamee4037
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:06:21
Input filesmonte-carlo-005746-000023
JobscriptExit code0
Real time17m (1030s)
CPU time16m (964s = 93%)
Max RSS bytes1726337024 (1646 MiB)
Outputting started2025-03-27 12:23:32
Output files
Finished2025-03-27 12:23:48
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

n: 2.97224e+07
0 2.65683e+07
1 2.63189e+07
2 2.62854e+07
3 2.62763e+07
Now with regularization...
Begin: 2.53705e+07
0 2.53645e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:15:52 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_4_1743077201_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:15:52 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_4_1743077201_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1570 traces, 1570 summary
Retagger: tagging trace set: gauss with 1118 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1118 traces tagged "gauss"
FrameSaver: q=136297 n=15638 tag=gauss
wclsFrameSaver: saving 1570 traces tagged "wiener"
FrameSaver: q=155261 n=13541 tag=wiener
27-Mar-2025 07:16:46 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_4_1743077201_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1631 traces, 1631 summary
Retagger: tagging trace set: gauss with 1227 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1227 traces tagged "gauss"
FrameSaver: q=149068 n=17061 tag=gauss
wclsFrameSaver: saving 1631 traces tagged "wiener"
FrameSaver: q=166962 n=15129 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1810 traces, 1810 summary
Retagger: tagging trace set: gauss with 1294 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1294 traces tagged "gauss"
FrameSaver: q=153193 n=18270 tag=gauss
wclsFrameSaver: saving 1810 traces tagged "wiener"
FrameSaver: q=172895 n=15909 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1643 traces, 1643 summary
Retagger: tagging trace set: gauss with 1209 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1209 traces tagged "gauss"
FrameSaver: q=151848 n=16968 tag=gauss
wclsFrameSaver: saving 1643 traces tagged "wiener"
FrameSaver: q=168780 n=14865 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1686 traces, 1686 summary
Retagger: tagging trace set: gauss with 1183 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1183 traces tagged "gauss"
FrameSaver: q=139212 n=16623 tag=gauss
wclsFrameSaver: saving 1686 traces tagged "wiener"
FrameSaver: q=159388 n=14348 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1593 traces, 1593 summary
Retagger: tagging trace set: gauss with 1108 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1108 traces tagged "gauss"
FrameSaver: q=136217 n=15619 tag=gauss
wclsFrameSaver: saving 1593 traces tagged "wiener"
FrameSaver: q=155815 n=13529 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1531 traces, 1531 summary
Retagger: tagging trace set: gauss with 1142 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1142 traces tagged "gauss"
FrameSaver: q=137992 n=15727 tag=gauss
wclsFrameSaver: saving 1531 traces tagged "wiener"
FrameSaver: q=154948 n=13733 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1555 traces, 1555 summary
Retagger: tagging trace set: gauss with 1111 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1111 traces tagged "gauss"
FrameSaver: q=139833 n=15620 tag=gauss
wclsFrameSaver: saving 1555 traces tagged "wiener"
FrameSaver: q=156997 n=13504 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1714 traces, 1714 summary
Retagger: tagging trace set: gauss with 1258 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1258 traces tagged "gauss"
FrameSaver: q=152688 n=17400 tag=gauss
wclsFrameSaver: saving 1714 traces tagged "wiener"
FrameSaver: q=171302 n=15080 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1570 traces, 1570 summary
Retagger: tagging trace set: gauss with 1103 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1103 traces tagged "gauss"
FrameSaver: q=138624 n=15844 tag=gauss
wclsFrameSaver: saving 1570 traces tagged "wiener"
FrameSaver: q=156723 n=13728 tag=wiener
27-Mar-2025 07:23:31 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_4_1743077201_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:23:31 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_4_1743077201_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       41.3068       44.7432       53.064        44.6598       3.28772        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000489145   0.000862321   0.00163148    0.00078626    0.000317132      10     
reco:rns:RandomNumberSaver                     4.6286e-05    0.000107391   0.000548766   5.7292e-05    0.000147464      10     
reco:wclsdatavd:WireCellToolkit                  40.9226       44.099        50.047        44.3183       2.6483         10     
reco:gaushit:GausHitFinder                      0.0173786     0.0221325     0.0287317     0.0219519    0.00316941       10     
reco:reco3d:SpacePointSolver                    0.0539104     0.0777228      0.12325      0.0739726     0.0204258       10     
reco:hitpdune:DisambigFromSpacePoints          0.00148365    0.00267078    0.00650889    0.00229786    0.00135096       10     
reco:pandora:StandardPandora                     0.18017       0.47537       2.7318       0.240525      0.752882        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.0021561    0.00427328     0.0154427    0.00289587    0.00382184       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000169747   0.000734775   0.00281235    0.000255868   0.000973108      10     
reco:pandoraStdcalo:Calorimetry                0.00133117    0.00284521     0.0121828     0.001729     0.00314415       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000702444   0.00193492    0.00894649     0.0012049    0.00234799       10     
[art]:TriggerResults:TriggerResultInserter     1.5338e-05    4.37497e-05   0.000280304   1.79635e-05   7.88728e-05      10     
end_path:out1:RootOutput                        4.559e-06    1.51714e-05   7.9008e-05    8.4005e-06    2.13243e-05      10     
end_path:out1:RootOutput(write)                 0.0279856     0.0486457     0.0707231     0.0494217     0.0133681       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3437.82 MB
  Peak resident set size usage (VmHWM): 1637.83 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_4_1743077201_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743077714.9412496, 'end_time': 1743078211.9859111}
justIN time: 2025-04-03 08:14:15 UTC       justIN version: 01.03.00