justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 78544.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID78544.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3451
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2024-09-26 10:24:11
SiteUK_Sheffield
EntryDUNE_UK_Sheffield_lcgce1
Last heartbeat2024-09-26 12:49:58
From worker nodeHostnamewn033.hep
cpuinfoIntel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-26 11:08:13
Input fileshd-protodune:np04hd_raw_run029107_2162_dataflow3_datawriter_0_20240908T023939.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-09-26 12:49:58
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

7.16706e+06
3 7.16304e+06
Now with regularization...
Begin: 6.68322e+06
0 6.68002e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Begin processing the 29th record. run: 29107 subRun: 1 event: 519131 at 26-Sep-2024 13:15:15 BST
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67976 traces, 67976 summary
Retagger: tagging trace set: gauss with 50344 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 50344 traces tagged "gauss"
FrameSaver: q=1.16004e+06 n=942223 tag=gauss
wclsFrameSaver: saving 67976 traces tagged "wiener"
FrameSaver: q=1.26762e+06 n=858439 tag=wiener
6783 hits in TPC 0
3863 hits in TPC 1
6872 hits in TPC 2
3867 hits in TPC 3
4032 hits in TPC 4
4365 hits in TPC 5
6667 hits in TPC 6
1957 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 6813 XUs and 4793 XVs -> 365 XUVs
C:0 T:2 50944 XUs and 64030 XVs -> 4600 XUVs
C:0 T:3 313 XUs and 395 XVs -> 22 XUVs
C:0 T:4 188 XUs and 242 XVs -> 19 XUVs
C:0 T:5 13186 XUs and 17370 XVs -> 519 XUVs
C:0 T:6 30874 XUs and 49444 XVs -> 4391 XUVs
C:0 T:7 61 XUs and 67 XVs -> 2 XUVs
9918 XUVs total
1823 collection wire objects
9918 potential space points
Neighbour search...
2022424 tests to find 865236 neighbours
Iterating with no regularization...
Begin: 1.50095e+07
0 1.39037e+07
1 1.38309e+07
2 1.38239e+07
Now with regularization...
Begin: 1.33941e+07
0 1.33908e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Begin processing the 30th record. run: 29107 subRun: 1 event: 519139 at 26-Sep-2024 13:16:48 BST
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 66596 traces, 66596 summary
Retagger: tagging trace set: gauss with 50166 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 50166 traces tagged "gauss"
FrameSaver: q=1.20009e+06 n=996732 tag=gauss
wclsFrameSaver: saving 66596 traces tagged "wiener"
FrameSaver: q=1.27432e+06 n=909065 tag=wiener
3666 hits in TPC 0
4100 hits in TPC 1
10536 hits in TPC 2
4855 hits in TPC 3
1563 hits in TPC 4
4408 hits in TPC 5
5329 hits in TPC 6
2949 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 295 XUs and 202 XVs -> 22 XUVs
C:0 T:2 28394 XUs and 34314 XVs -> 1430 XUVs
C:0 T:3 173 XUs and 238 XVs -> 12 XUVs
C:0 T:4 0 XUs and 0 XVs -> 0 XUVs
C:0 T:5 27022 XUs and 32198 XVs -> 1949 XUVs
C:0 T:6 9532 XUs and 9913 XVs -> 622 XUVs
C:0 T:7 20 XUs and 24 XVs -> 0 XUVs
4035 XUVs total
1574 collection wire objects
4035 potential space points
Neighbour search...
341225 tests to find 177756 neighbours
Iterating with no regularization...
Begin: 3.90759e+07
0 3.76034e+07
1 3.7461e+07
2 3.74536e+07
Now with regularization...
Begin: 3.56365e+07
0 3.56308e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

no of hits closeby  0   event 519139 TrkackID 0 startx, y, z 236.118 606.434 118.421  wireno, peakt tpcno 306 3006.89 2 dist 283.07min T, max_T24.5268 3047.07
no of hits closeby  5   event 519139 TrkackID 4 startx, y, z -23.4368 596.72 228.443  wireno, peakt tpcno 72 3956.6 6 dist 1.63773min T, max_T3900.63 5050.35
no of hits closeby  0   event 519139 TrkackID 0 startx, y, z 236.118 606.434 118.421  wireno, peakt tpcno 306 3006.89 2 dist 283.07min T, max_T24.5268 3047.07
no of hits closeby  5   event 519139 TrkackID 4 startx, y, z -23.4368 596.72 228.443  wireno, peakt tpcno 72 3956.6 6 dist 1.63773min T, max_T3900.63 5050.35
26-Sep-2024 13:18:18 BST  Closed output file "np04hd_raw_run029107_2162_dataflow3_datawriter_0_20240908T023939_reco_stage1_reco_stage2_20240926T121818_keepup.root"
26-Sep-2024 13:18:18 BST  Closed input file "np04hd_raw_run029107_2162_dataflow3_datawriter_0_20240908T023939_reco_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              64.3612       81.3631       111.412       79.4915       11.1164        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000361778   0.00135659    0.00394891    0.000908748   0.00108333       30     
produce:fembfilter:PDHDFEMBFilter                      0.598911      0.621745      0.739883      0.611985      0.0286231       30     
produce:wclsdatahd:WireCellToolkit                      42.8748       46.2356       50.6111       46.1703       1.82924        30     
produce:gaushit:GausHitFinder                          0.335944      0.606554      0.874398      0.610575      0.117342        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000250026   0.000391334   0.000577003   0.000396164   6.47242e-05      30     
produce:reco3d:SpacePointSolver                        0.192411      0.857111       2.34421      0.712685      0.496212        30     
produce:hitpdune:DisambigFromSpacePoints               0.146197      0.426447      0.892984      0.408296      0.156653        30     
produce:pandora:StandardPandora                         5.00993       10.2611       18.0971       9.31791       3.24582        30     
produce:pandoraWriter:StandardPandora                  0.0981184     0.149394      0.229452      0.146043      0.0291297       30     
produce:pandoraTrack:LArPandoraTrackCreation           0.707232       2.05519       4.90754       1.85585      0.952604        30     
produce:pandoraShower:LArPandoraShowerCreation         0.471865       2.32077       6.43786       2.13368       1.27999        30     
produce:pandoracalo:Calorimetry                         0.30336      0.817878       1.80824      0.747708      0.366305        30     
produce:pandoracalonosce:Calorimetry                   0.301872      0.762556        1.638       0.691742      0.336998        30     
produce:pandorapid:Chi2ParticleID                     0.00120127    0.00166945    0.00378353    0.00150619    0.000482176      30     
produce:pandoraShowercalo:ShowerCalorimetry            0.417134       1.86714       4.79338       1.62805      0.987804        30     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.417923       1.78737       4.54313       1.56574      0.884687        30     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0577911     0.107256      0.152102      0.106488      0.0221666       30     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0239966     0.0438973     0.0652163     0.0433664    0.00950652       30     
produce:emtrkmichelid:EmTrackMichelId                   6.83045       10.9542       16.6801       10.5363       2.31918        30     
[art]:TriggerResults:TriggerResultInserter            3.2525e-05    4.69599e-05   9.4599e-05    4.4276e-05    1.14086e-05      30     
end_path:out1:RootOutput                               9.518e-06    1.28116e-05   2.9451e-05    1.2174e-05    3.64183e-06      30     
end_path:michelremoving:michelremoving                 0.0233711     0.0605294      0.22509      0.0401334     0.0500903       30     
end_path:michelremovinggnocchi:michelremoving          0.0215786     0.0577107     0.217925      0.0362659     0.0476291       30     
end_path:calibanagnocchi:CalibAnaTree                  0.254247      0.529999       1.16655      0.463942      0.210689        30     
end_path:out1:RootOutput(write)                        0.391978      0.612296      0.877998      0.600828      0.123935        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5879.56 MB
  Peak resident set size usage (VmHWM): 2699.73 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_2162_dataflow3_datawriter_0_20240908T023939_reco_stage1_reco_stage2_20240926T121818_keepup.root
\tHists: np04hd_raw_run029107_2162_dataflow3_datawriter_0_20240908T023939_reco_stage1_reco_stage2_20240926T121818_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2024-11-17 06:00:30 UTC       justIN version: 01.01.09