justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358068.120@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358068.120@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:15:52
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:08:35
From worker nodeHostnamee4047
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:50:19
Input filesmonte-carlo-005746-000021
JobscriptExit code0
Real time17m (1077s)
CPU time17m (1041s = 96%)
Max RSS bytes1729695744 (1649 MiB)
Outputting started2025-03-27 12:08:17
Output files
Finished2025-03-27 12:08:35
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+07
1 1.05591e+07
2 1.0518e+07
3 1.05078e+07
Now with regularization...
Begin: 9.53406e+06
0 9.52381e+06
1 9.51613e+06
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:00:02 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358068_120_1743076229_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:00:02 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358068_120_1743076229_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1607 traces, 1607 summary
Retagger: tagging trace set: gauss with 1153 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1153 traces tagged "gauss"
FrameSaver: q=137675 n=15953 tag=gauss
wclsFrameSaver: saving 1607 traces tagged "wiener"
FrameSaver: q=156414 n=13910 tag=wiener
27-Mar-2025 07:00:56 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358068_120_1743076229_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1641 traces, 1641 summary
Retagger: tagging trace set: gauss with 1222 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1222 traces tagged "gauss"
FrameSaver: q=141048 n=17020 tag=gauss
wclsFrameSaver: saving 1641 traces tagged "wiener"
FrameSaver: q=159159 n=14815 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1687 traces, 1687 summary
Retagger: tagging trace set: gauss with 1197 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1197 traces tagged "gauss"
FrameSaver: q=143681 n=17063 tag=gauss
wclsFrameSaver: saving 1687 traces tagged "wiener"
FrameSaver: q=162548 n=14596 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1603 traces, 1603 summary
Retagger: tagging trace set: gauss with 1139 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1139 traces tagged "gauss"
FrameSaver: q=138273 n=16128 tag=gauss
wclsFrameSaver: saving 1603 traces tagged "wiener"
FrameSaver: q=156729 n=13945 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1548 traces, 1548 summary
Retagger: tagging trace set: gauss with 1115 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1115 traces tagged "gauss"
FrameSaver: q=136369 n=15356 tag=gauss
wclsFrameSaver: saving 1548 traces tagged "wiener"
FrameSaver: q=154345 n=13258 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1690 traces, 1690 summary
Retagger: tagging trace set: gauss with 1258 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1258 traces tagged "gauss"
FrameSaver: q=142335 n=17051 tag=gauss
wclsFrameSaver: saving 1690 traces tagged "wiener"
FrameSaver: q=161121 n=14878 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1635 traces, 1635 summary
Retagger: tagging trace set: gauss with 1195 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1195 traces tagged "gauss"
FrameSaver: q=137373 n=16331 tag=gauss
wclsFrameSaver: saving 1635 traces tagged "wiener"
FrameSaver: q=155840 n=14175 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1664 traces, 1664 summary
Retagger: tagging trace set: gauss with 1215 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1215 traces tagged "gauss"
FrameSaver: q=140685 n=16662 tag=gauss
wclsFrameSaver: saving 1664 traces tagged "wiener"
FrameSaver: q=160350 n=14528 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1575 traces, 1575 summary
Retagger: tagging trace set: gauss with 1153 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1153 traces tagged "gauss"
FrameSaver: q=137427 n=15760 tag=gauss
wclsFrameSaver: saving 1575 traces tagged "wiener"
FrameSaver: q=155039 n=13733 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1541 traces, 1541 summary
Retagger: tagging trace set: gauss with 1077 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1077 traces tagged "gauss"
FrameSaver: q=136316 n=15205 tag=gauss
wclsFrameSaver: saving 1541 traces tagged "wiener"
FrameSaver: q=154146 n=13094 tag=wiener
27-Mar-2025 07:08:16 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358068_120_1743076229_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:08:16 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358068_120_1743076229_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       44.6035       48.5876       53.3999       48.708        2.60897        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000697773   0.00106203    0.00211025    0.00078524    0.000517042      10     
reco:rns:RandomNumberSaver                     4.2639e-05    0.000101379   0.00051981    5.5694e-05    0.000139819      10     
reco:wclsdatavd:WireCellToolkit                  44.2501       48.0401       51.3212       48.3226       2.30066        10     
reco:gaushit:GausHitFinder                      0.017815      0.0224429     0.0268671     0.0232436    0.00274156       10     
reco:reco3d:SpacePointSolver                    0.052144      0.0711796     0.0916676     0.0719699     0.0113956       10     
reco:hitpdune:DisambigFromSpacePoints          0.00172732    0.00250527    0.00516305    0.00237259    0.000936188      10     
reco:pandora:StandardPandora                    0.188608      0.394756       1.90081      0.226774      0.502729        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00162436     0.0027015    0.00626237    0.00246425    0.00122758       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000168735   0.000297178   0.000858562   0.000240629   0.000189714      10     
reco:pandoraStdcalo:Calorimetry                0.00156902    0.00216727    0.00597288    0.00169902    0.00128117       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000912232   0.00141375    0.00363422    0.00115683    0.000749412      10     
[art]:TriggerResults:TriggerResultInserter     1.4056e-05    2.39538e-05   8.4858e-05    1.7032e-05    2.04056e-05      10     
end_path:out1:RootOutput                        6.663e-06    1.55901e-05   3.7139e-05    8.9115e-06    1.13554e-05      10     
end_path:out1:RootOutput(write)                 0.0358612     0.0448069     0.0556517     0.0450361    0.00632801       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3445.67 MB
  Peak resident set size usage (VmHWM): 1646.66 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358068_120_1743076229_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076765.3103848, 'end_time': 1743077297.2689536}
justIN time: 2025-04-03 08:30:41 UTC       justIN version: 01.03.00