justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166655.178@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166655.178@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:13:42
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-03-27 12:10:23
From worker nodeHostnamecompute-1-3.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 11:45:40
Input filesmonte-carlo-005746-000176
JobscriptExit code0
Real time24m (1467s)
CPU time24m (1451s = 98%)
Max RSS bytes1597976576 (1523 MiB)
Outputting started2025-03-27 12:10:08
Output files
Finished2025-03-27 12:10:23
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

egin: 1.31216e+07
0 1.28174e+07
1 1.28017e+07
2 1.28008e+07
Now with regularization...
Begin: 1.16573e+07
0 1.16571e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 07:57:38 AST  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166655_178_1743075948_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 07:57:38 AST  Opened input file "prod_muon_400MeV_protodunevd_1_166655_178_1743075948_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1679 traces, 1679 summary
Retagger: tagging trace set: gauss with 1219 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1219 traces tagged "gauss"
FrameSaver: q=137152 n=16606 tag=gauss
wclsFrameSaver: saving 1679 traces tagged "wiener"
FrameSaver: q=156260 n=14483 tag=wiener
27-Mar-2025 07:58:54 AST  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166655_178_1743075948_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1714 traces, 1714 summary
Retagger: tagging trace set: gauss with 1210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1210 traces tagged "gauss"
FrameSaver: q=152861 n=17028 tag=gauss
wclsFrameSaver: saving 1714 traces tagged "wiener"
FrameSaver: q=172585 n=14697 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1578 traces, 1578 summary
Retagger: tagging trace set: gauss with 1099 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1099 traces tagged "gauss"
FrameSaver: q=136744 n=15638 tag=gauss
wclsFrameSaver: saving 1578 traces tagged "wiener"
FrameSaver: q=155870 n=13454 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1604 traces, 1604 summary
Retagger: tagging trace set: gauss with 1108 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1108 traces tagged "gauss"
FrameSaver: q=142026 n=16034 tag=gauss
wclsFrameSaver: saving 1604 traces tagged "wiener"
FrameSaver: q=159883 n=13791 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1640 traces, 1640 summary
Retagger: tagging trace set: gauss with 1269 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1269 traces tagged "gauss"
FrameSaver: q=154236 n=17365 tag=gauss
wclsFrameSaver: saving 1640 traces tagged "wiener"
FrameSaver: q=171883 n=15283 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1541 traces, 1541 summary
Retagger: tagging trace set: gauss with 1098 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1098 traces tagged "gauss"
FrameSaver: q=138274 n=15481 tag=gauss
wclsFrameSaver: saving 1541 traces tagged "wiener"
FrameSaver: q=156750 n=13406 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1751 traces, 1751 summary
Retagger: tagging trace set: gauss with 1255 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1255 traces tagged "gauss"
FrameSaver: q=152076 n=17534 tag=gauss
wclsFrameSaver: saving 1751 traces tagged "wiener"
FrameSaver: q=172645 n=15352 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1641 traces, 1641 summary
Retagger: tagging trace set: gauss with 1203 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1203 traces tagged "gauss"
FrameSaver: q=153372 n=17125 tag=gauss
wclsFrameSaver: saving 1641 traces tagged "wiener"
FrameSaver: q=171202 n=14843 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1600 traces, 1600 summary
Retagger: tagging trace set: gauss with 1145 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1145 traces tagged "gauss"
FrameSaver: q=137746 n=15715 tag=gauss
wclsFrameSaver: saving 1600 traces tagged "wiener"
FrameSaver: q=156685 n=13647 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1617 traces, 1617 summary
Retagger: tagging trace set: gauss with 1256 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1256 traces tagged "gauss"
FrameSaver: q=158990 n=17303 tag=gauss
wclsFrameSaver: saving 1617 traces tagged "wiener"
FrameSaver: q=175140 n=15308 tag=wiener
27-Mar-2025 08:10:06 AST  Closed output file "prod_muon_400MeV_protodunevd_1_166655_178_1743075948_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 08:10:06 AST  Closed input file "prod_muon_400MeV_protodunevd_1_166655_178_1743075948_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       69.8299       73.9121       90.0772       71.8449       5.55702        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000307593   0.000507707   0.000742379   0.000488286   0.000137299      10     
reco:rns:RandomNumberSaver                     3.2262e-05    6.19463e-05   0.000259652   3.8709e-05    6.62081e-05      10     
reco:wclsdatavd:WireCellToolkit                  69.4357       73.3421       89.6479       71.4693       5.54668        10     
reco:gaushit:GausHitFinder                      0.0183399     0.0204165     0.0223889     0.0202378    0.00119465       10     
reco:reco3d:SpacePointSolver                    0.0569389     0.0658948     0.0816377     0.0647305    0.00656773       10     
reco:hitpdune:DisambigFromSpacePoints          0.00204494    0.00247803    0.00388193    0.00236658    0.000500045      10     
reco:pandora:StandardPandora                    0.240871      0.438641       2.03089      0.263636      0.531025        10     
reco:pandoraTrack:LArPandoraTrackCreation      0.00164985    0.00261761    0.00424878    0.00264502    0.000689226      10     
reco:pandoraShower:LArPandoraShowerCreation    0.000210993   0.000403168   0.00141505    0.000245001   0.000360143      10     
reco:pandoraStdcalo:Calorimetry                0.00143264    0.00180978    0.00361422    0.00160861    0.000613722      10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000817992   0.00119655    0.00239743    0.00100917    0.000465412      10     
[art]:TriggerResults:TriggerResultInserter     1.8561e-05    2.20807e-05   4.4178e-05     1.966e-05    7.39967e-06      10     
end_path:out1:RootOutput                        4.144e-06    5.2829e-06    1.3872e-05     4.259e-06    2.86871e-06      10     
end_path:out1:RootOutput(write)                 0.0301619     0.0315059     0.034535      0.0312169    0.00118952       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3343.28 MB
  Peak resident set size usage (VmHWM): 1371.1 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166655_178_1743075948_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743076622.8274677, 'end_time': 1743077407.9833512}
justIN time: 2025-04-03 08:29:38 UTC       justIN version: 01.03.00