Jobsub ID 78466.96@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78466.96@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3449 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:59:54 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:06:29 |
| From worker node | Hostname | wn098.hep |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:22:13 |
| Input files | hd-protodune:np04hd_raw_run029107_0825_dataflow3_datawriter_0_20240907T041642.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:06:29 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
set: wiener with 28656 traces, 28656 summary
Retagger: tagging trace set: gauss with 21380 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 21380 traces tagged "gauss"
FrameSaver: q=415912 n=366880 tag=gauss
wclsFrameSaver: saving 28656 traces tagged "wiener"
FrameSaver: q=444338 n=324203 tag=wiener
978 hits in TPC 0
664 hits in TPC 1
2972 hits in TPC 2
2188 hits in TPC 3
525 hits in TPC 4
1264 hits in TPC 5
2671 hits in TPC 6
2647 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 25 XUs and 18 XVs -> 1 XUVs
C:0 T:2 16106 XUs and 16693 XVs -> 537 XUVs
C:0 T:3 30 XUs and 14 XVs -> 2 XUVs
C:0 T:4 1 XUs and 1 XVs -> 0 XUVs
C:0 T:5 3919 XUs and 1427 XVs -> 53 XUVs
C:0 T:6 19621 XUs and 27770 XVs -> 1332 XUVs
C:0 T:7 222 XUs and 221 XVs -> 8 XUVs
1933 XUVs total
531 collection wire objects
1933 potential space points
Neighbour search...
194835 tests to find 92250 neighbours
Iterating with no regularization...
Begin: 1.42455e+06
0 1.29036e+06
1 1.27924e+06
2 1.27738e+06
3 1.27664e+06
Now with regularization...
Begin: 1.2e+06
0 1.19945e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29107 subRun: 1 event: 198251 at 26-Sep-2024 12:50:53 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 43987 traces, 43987 summary
Retagger: tagging trace set: gauss with 33782 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 33782 traces tagged "gauss"
FrameSaver: q=726866 n=679269 tag=gauss
wclsFrameSaver: saving 43987 traces tagged "wiener"
FrameSaver: q=763504 n=620213 tag=wiener
2802 hits in TPC 0
2919 hits in TPC 1
3210 hits in TPC 2
1825 hits in TPC 3
3027 hits in TPC 4
3972 hits in TPC 5
3344 hits in TPC 6
1517 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 5204 XUs and 5779 XVs -> 188 XUVs
C:0 T:3 36 XUs and 59 XVs -> 1 XUVs
C:0 T:4 81 XUs and 126 XVs -> 2 XUVs
C:0 T:5 8328 XUs and 11443 XVs -> 295 XUVs
C:0 T:6 4001 XUs and 8557 XVs -> 205 XUVs
C:0 T:7 25 XUs and 44 XVs -> 1 XUVs
692 XUVs total
543 collection wire objects
692 potential space points
Neighbour search...
5492 tests to find 2996 neighbours
Iterating with no regularization...
Begin: 2.40535e+06
0 2.32366e+06
1 2.32279e+06
Now with regularization...
Begin: 2.17557e+06
0 2.17551e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29107 subRun: 1 event: 198259 at 26-Sep-2024 12:52:50 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 58996 traces, 58996 summary
Retagger: tagging trace set: gauss with 46365 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 46365 traces tagged "gauss"
FrameSaver: q=1.0138e+06 n=940807 tag=gauss
wclsFrameSaver: saving 58996 traces tagged "wiener"
FrameSaver: q=1.08377e+06 n=871583 tag=wiener
2642 hits in TPC 0
3226 hits in TPC 1
8180 hits in TPC 2
2942 hits in TPC 3
4369 hits in TPC 4
7923 hits in TPC 5
3221 hits in TPC 6
1034 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 86 XUs and 91 XVs -> 8 XUVs
C:0 T:2 21569 XUs and 32053 XVs -> 1006 XUVs
C:0 T:3 52 XUs and 74 XVs -> 2 XUVs
C:0 T:4 45 XUs and 79 XVs -> 0 XUVs
C:0 T:5 59725 XUs and 50413 XVs -> 3019 XUVs
C:0 T:6 6444 XUs and 6926 XVs -> 263 XUVs
C:0 T:7 178 XUs and 168 XVs -> 0 XUVs
4298 XUVs total
1656 collection wire objects
4298 potential space points
Neighbour search...
189548 tests to find 104876 neighbours
Iterating with no regularization...
Begin: 8.41984e+06
0 7.27548e+06
1 7.18806e+06
2 7.17645e+06
3 7.17383e+06
Now with regularization...
Begin: 6.74203e+06
0 6.74051e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:54:46 BST Closed output file "np04hd_raw_run029107_0825_dataflow3_datawriter_0_20240907T041642_reco_stage1_reco_stage2_20240926T115446_keepup.root"
26-Sep-2024 12:54:46 BST Closed input file "np04hd_raw_run029107_0825_dataflow3_datawriter_0_20240907T041642_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 74.7135 121.245 162.642 120.724 19.8924 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000596628 0.00282021 0.00469637 0.00303672 0.00121876 30
produce:fembfilter:PDHDFEMBFilter 0.733209 0.767025 1.00802 0.742213 0.0609267 30
produce:wclsdatahd:WireCellToolkit 50.7725 55.3951 60.0899 55.4077 1.76699 30
produce:gaushit:GausHitFinder 0.302425 0.818694 1.19898 0.802032 0.2137 30
produce:nhitsfilter:NumberOfHitsFilter 0.000252844 0.000471288 0.000791413 0.000458223 0.000111443 30
produce:reco3d:SpacePointSolver 0.195855 1.19777 4.44381 0.944767 0.904357 30
produce:hitpdune:DisambigFromSpacePoints 0.104807 0.691525 1.77048 0.686699 0.346985 30
produce:pandora:StandardPandora 2.86484 15.7228 30.5728 15.4983 6.25028 30
produce:pandoraWriter:StandardPandora 0.0713235 0.177707 0.234582 0.184183 0.0359897 30
produce:pandoraTrack:LArPandoraTrackCreation 0.30374 2.11097 4.98631 2.29442 0.881263 30
produce:pandoraShower:LArPandoraShowerCreation 0.251244 2.53953 4.85399 2.28351 1.26466 30
produce:pandoracalo:Calorimetry 0.173089 0.993961 2.0916 0.972447 0.383309 30
produce:pandoracalonosce:Calorimetry 0.160823 0.920961 2.02093 0.899516 0.370799 30
produce:pandorapid:Chi2ParticleID 0.00103316 0.00207516 0.00307965 0.00207258 0.000419606 30
produce:pandoraShowercalo:ShowerCalorimetry 0.163894 2.27443 4.43117 2.22216 1.12835 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.160052 2.20841 4.26883 2.15573 1.07565 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0513899 0.141337 0.192315 0.143273 0.0330941 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.019896 0.053796 0.0739379 0.0546266 0.0118786 30
produce:emtrkmichelid:EmTrackMichelId 14.3556 33.4254 43.9974 33.8398 7.18455 30
[art]:TriggerResults:TriggerResultInserter 2.735e-05 4.81793e-05 7.7345e-05 4.49425e-05 1.39897e-05 30
end_path:out1:RootOutput 8.411e-06 1.13629e-05 2.3327e-05 9.6695e-06 4.03841e-06 30
end_path:michelremoving:michelremoving 0.0222463 0.0757637 0.364733 0.0492975 0.0698021 30
end_path:michelremovinggnocchi:michelremoving 0.0212748 0.0691873 0.291495 0.0414908 0.060729 30
end_path:calibanagnocchi:CalibAnaTree 0.133837 0.701407 1.6569 0.674393 0.301902 30
end_path:out1:RootOutput(write) 0.322947 0.812087 1.11172 0.80144 0.186865 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4930.65 MB
Peak resident set size usage (VmHWM): 2347.87 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_0825_dataflow3_datawriter_0_20240907T041642_reco_stage1_reco_stage2_20240926T115446_keepup.root
\tHists: np04hd_raw_run029107_0825_dataflow3_datawriter_0_20240907T041642_reco_stage1_reco_stage2_20240926T115446_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
