justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358070.188@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358070.188@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:16:58
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-03-27 12:33:39
From worker nodeHostnamee4057
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:17:06
Input filesmonte-carlo-005746-000041
JobscriptExit code0
Real time16m (974s)
CPU time15m (933s = 95%)
Max RSS bytes1703448576 (1624 MiB)
Outputting started2025-03-27 12:33:22
Output files
Finished2025-03-27 12:33:39
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

egin: 2.48735e+07
0 2.23276e+07
1 2.22838e+07
2 2.22733e+07
Now with regularization...
Begin: 2.14209e+07
0 2.14174e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:25:53 CDT  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_358070_188_1743077836_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:25:54 CDT  Opened input file "prod_muon_400MeV_protodunevd_1_358070_188_1743077836_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1630 traces, 1630 summary
Retagger: tagging trace set: gauss with 1163 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1163 traces tagged "gauss"
FrameSaver: q=136538 n=15899 tag=gauss
wclsFrameSaver: saving 1630 traces tagged "wiener"
FrameSaver: q=155365 n=13851 tag=wiener
27-Mar-2025 07:26:36 CDT  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_358070_188_1743077836_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1612 traces, 1612 summary
Retagger: tagging trace set: gauss with 1183 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1183 traces tagged "gauss"
FrameSaver: q=141930 n=16642 tag=gauss
wclsFrameSaver: saving 1612 traces tagged "wiener"
FrameSaver: q=159921 n=14511 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1696 traces, 1696 summary
Retagger: tagging trace set: gauss with 1191 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1191 traces tagged "gauss"
FrameSaver: q=139850 n=17058 tag=gauss
wclsFrameSaver: saving 1696 traces tagged "wiener"
FrameSaver: q=160180 n=14732 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1594 traces, 1594 summary
Retagger: tagging trace set: gauss with 1245 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1245 traces tagged "gauss"
FrameSaver: q=142597 n=17246 tag=gauss
wclsFrameSaver: saving 1594 traces tagged "wiener"
FrameSaver: q=157838 n=15385 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1725 traces, 1725 summary
Retagger: tagging trace set: gauss with 1190 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1190 traces tagged "gauss"
FrameSaver: q=140310 n=17043 tag=gauss
wclsFrameSaver: saving 1725 traces tagged "wiener"
FrameSaver: q=160490 n=14762 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1172 traces tagged "gauss"
FrameSaver: q=137592 n=16100 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=155231 n=14127 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1680 traces, 1680 summary
Retagger: tagging trace set: gauss with 1225 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1225 traces tagged "gauss"
FrameSaver: q=146213 n=17349 tag=gauss
wclsFrameSaver: saving 1680 traces tagged "wiener"
FrameSaver: q=163907 n=14947 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1602 traces, 1602 summary
Retagger: tagging trace set: gauss with 1101 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1101 traces tagged "gauss"
FrameSaver: q=139455 n=15824 tag=gauss
wclsFrameSaver: saving 1602 traces tagged "wiener"
FrameSaver: q=158246 n=13568 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1529 traces, 1529 summary
Retagger: tagging trace set: gauss with 1137 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1137 traces tagged "gauss"
FrameSaver: q=137507 n=15700 tag=gauss
wclsFrameSaver: saving 1529 traces tagged "wiener"
FrameSaver: q=154742 n=13624 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1610 traces, 1610 summary
Retagger: tagging trace set: gauss with 1168 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1168 traces tagged "gauss"
FrameSaver: q=139669 n=16012 tag=gauss
wclsFrameSaver: saving 1610 traces tagged "wiener"
FrameSaver: q=157883 n=13825 tag=wiener
27-Mar-2025 07:33:21 CDT  Closed output file "prod_muon_400MeV_protodunevd_1_358070_188_1743077836_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 07:33:21 CDT  Closed input file "prod_muon_400MeV_protodunevd_1_358070_188_1743077836_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       41.7937       43.9079       47.0054       43.2597       2.0372         10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000722743   0.00105606    0.00201199    0.000955659   0.000371652      10     
reco:rns:RandomNumberSaver                     5.1727e-05    0.000115073   0.000518801   7.05565e-05   0.000135444      10     
reco:wclsdatavd:WireCellToolkit                  40.0441       43.3698       46.5834       42.8671       2.18596        10     
reco:gaushit:GausHitFinder                      0.0173251     0.022522      0.0265625     0.0227498    0.00296228       10     
reco:reco3d:SpacePointSolver                    0.0556527     0.0848443     0.115249      0.0785607     0.0184042       10     
reco:hitpdune:DisambigFromSpacePoints          0.00183441    0.00266894    0.00403695    0.00254886    0.000611914      10     
reco:pandora:StandardPandora                     0.16554      0.362136       1.60903       0.22939       0.41618        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00156503    0.00358332    0.00585245    0.00318235    0.00136435       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000199033   0.000368371   0.00111761    0.000263904   0.000264603      10     
reco:pandoraStdcalo:Calorimetry                0.00157434    0.00205768    0.00396787    0.00183206    0.000663746      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000805798   0.00132107    0.00252138    0.00125444    0.000430782      10     
[art]:TriggerResults:TriggerResultInserter     1.7212e-05    2.57614e-05   5.2749e-05    2.3419e-05    9.66277e-06      10     
end_path:out1:RootOutput                        7.304e-06    1.04847e-05   1.9517e-05     9.603e-06    3.18345e-06      10     
end_path:out1:RootOutput(write)                 0.0334064     0.0500126     0.0676556     0.0482091     0.0112194       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3463.39 MB
  Peak resident set size usage (VmHWM): 1645.92 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_358070_188_1743077836_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078316.0436592, 'end_time': 1743078801.688872}
justIN time: 2025-04-03 08:31:33 UTC       justIN version: 01.03.00