justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 166657.129@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID166657.129@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5746
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 11:14:47
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-03-27 12:35:02
From worker nodeHostnametd814.pic.es
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 12:18:20
Input filesmonte-carlo-005746-000340
JobscriptExit code0
Real time16m (980s)
CPU time15m (952s = 97%)
Max RSS bytes1712189440 (1632 MiB)
Outputting started2025-03-27 12:34:41
Output files
Finished2025-03-27 12:35:02
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rization...
Begin: 9.77949e+07
0 9.74136e+07
1 9.73716e+07
Now with regularization...
Begin: 9.68763e+07
0 9.68667e+07
Art has completed and will exit with status 0.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  pandoraWriter
  pandoracalipid
  pandorapid
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/8e14053062785f80c03da35c9e53597243e9b1b5/protodunevd_v4_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16499 nodes/ 2408 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Mar-2025 13:27:02 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_166657_129_1743077911_gen_g4_stage1_g4_stage2_detsim.root"
27-Mar-2025 13:27:02 CET  Opened input file "prod_muon_400MeV_protodunevd_1_166657_129_1743077911_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1589 traces, 1589 summary
Retagger: tagging trace set: gauss with 1137 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1137 traces tagged "gauss"
FrameSaver: q=142767 n=16059 tag=gauss
wclsFrameSaver: saving 1589 traces tagged "wiener"
FrameSaver: q=162133 n=13879 tag=wiener
27-Mar-2025 13:27:52 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_166657_129_1743077911_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1144 traces tagged "gauss"
FrameSaver: q=137935 n=15898 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=156313 n=13640 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1669 traces, 1669 summary
Retagger: tagging trace set: gauss with 1206 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1206 traces tagged "gauss"
FrameSaver: q=142073 n=16513 tag=gauss
wclsFrameSaver: saving 1669 traces tagged "wiener"
FrameSaver: q=160329 n=14347 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1615 traces, 1615 summary
Retagger: tagging trace set: gauss with 1186 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1186 traces tagged "gauss"
FrameSaver: q=137135 n=16155 tag=gauss
wclsFrameSaver: saving 1615 traces tagged "wiener"
FrameSaver: q=155948 n=14077 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1604 traces, 1604 summary
Retagger: tagging trace set: gauss with 1169 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1169 traces tagged "gauss"
FrameSaver: q=139318 n=16285 tag=gauss
wclsFrameSaver: saving 1604 traces tagged "wiener"
FrameSaver: q=157051 n=14111 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1619 traces, 1619 summary
Retagger: tagging trace set: gauss with 1236 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1236 traces tagged "gauss"
FrameSaver: q=154690 n=17015 tag=gauss
wclsFrameSaver: saving 1619 traces tagged "wiener"
FrameSaver: q=171902 n=15044 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1621 traces, 1621 summary
Retagger: tagging trace set: gauss with 1250 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1250 traces tagged "gauss"
FrameSaver: q=139913 n=16755 tag=gauss
wclsFrameSaver: saving 1621 traces tagged "wiener"
FrameSaver: q=156962 n=14801 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1693 traces, 1693 summary
Retagger: tagging trace set: gauss with 1195 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1195 traces tagged "gauss"
FrameSaver: q=142161 n=16605 tag=gauss
wclsFrameSaver: saving 1693 traces tagged "wiener"
FrameSaver: q=160290 n=14374 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1699 traces, 1699 summary
Retagger: tagging trace set: gauss with 1177 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1177 traces tagged "gauss"
FrameSaver: q=141394 n=16784 tag=gauss
wclsFrameSaver: saving 1699 traces tagged "wiener"
FrameSaver: q=161225 n=14453 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 1601 traces, 1601 summary
Retagger: tagging trace set: gauss with 1155 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 1155 traces tagged "gauss"
FrameSaver: q=138997 n=16231 tag=gauss
wclsFrameSaver: saving 1601 traces tagged "wiener"
FrameSaver: q=156579 n=14049 tag=wiener
27-Mar-2025 13:34:40 CET  Closed output file "prod_muon_400MeV_protodunevd_1_166657_129_1743077911_gen_g4_stage1_g4_stage2_detsim_reco.root"
27-Mar-2025 13:34:40 CET  Closed input file "prod_muon_400MeV_protodunevd_1_166657_129_1743077911_gen_g4_stage1_g4_stage2_detsim.root"

=================================================================================================================================
TimeTracker printout (sec)                         Min           Avg           Max         Median          RMS         nEvts   
=================================================================================================================================
Full event                                       43.3055       45.1792       49.1878       44.6086       1.82231        10     
---------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.000345073   0.000544681    0.0010205    0.000461413   0.000198959      10     
reco:rns:RandomNumberSaver                     2.6301e-05    6.36096e-05   0.000298522    4.009e-05    7.85879e-05      10     
reco:wclsdatavd:WireCellToolkit                  42.9787       44.6947       47.3709       44.2803       1.49521        10     
reco:gaushit:GausHitFinder                      0.0157614     0.0166615     0.0194779     0.0164797     0.0010413       10     
reco:reco3d:SpacePointSolver                    0.0486603     0.0580586     0.0706402     0.0565465    0.00588519       10     
reco:hitpdune:DisambigFromSpacePoints          0.00153474    0.00196385    0.00359278    0.00168027    0.000593268      10     
reco:pandora:StandardPandora                    0.140941      0.364998       1.77262      0.217762      0.470202        10     
reco:pandoraTrack:LArPandoraTrackCreation       0.001681     0.00216222    0.00352061    0.00203436    0.00048316       10     
reco:pandoraShower:LArPandoraShowerCreation    0.000168032   0.000317274   0.00120065    0.000187967   0.000306963      10     
reco:pandoraStdcalo:Calorimetry                0.00118909    0.00147975    0.00278911    0.00134613    0.00044662       10     
reco:pandoraGnocalo:GnocchiCalorimetry         0.000878278    0.0010219    0.00198076    0.000903273   0.000321883      10     
[art]:TriggerResults:TriggerResultInserter      1.329e-05    1.71732e-05    4.291e-05     1.39e-05     8.64188e-06      10     
end_path:out1:RootOutput                         2.9e-06     7.0281e-06     1.698e-05     6.21e-06     3.62604e-06      10     
end_path:out1:RootOutput(write)                 0.0288506     0.0334793     0.0456808     0.0325721    0.00435473       10     
=================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3414.98 MB
  Peak resident set size usage (VmHWM): 1574.13 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'prod_muon_400MeV_protodunevd_1_166657_129_1743077911_gen_g4_stage1_g4_stage2_detsim_reco.root', 'start_time': 1743078387.5008368, 'end_time': 1743078881.6043038}
justIN time: 2025-04-03 08:30:42 UTC       justIN version: 01.03.00