Jobsub ID 263306.117@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263306.117@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:01:13 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:19:37 |
| From worker node | Hostname | wn085.hep |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:15:24 |
| Input files | hd-protodune:np04hd_raw_run029081_0876_dataflow2_datawriter_0_20240905T063412.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:19:37 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
0 XUs and 279 XVs -> 4 XUVs
C:0 T:2 9270 XUs and 7253 XVs -> 223 XUVs
C:0 T:3 722 XUs and 1219 XVs -> 68 XUVs
C:0 T:4 273 XUs and 410 XVs -> 23 XUVs
C:0 T:5 29332 XUs and 50195 XVs -> 1979 XUVs
C:0 T:6 10102 XUs and 3253 XVs -> 94 XUVs
C:0 T:7 260 XUs and 257 XVs -> 59 XUVs
2450 XUVs total
912 collection wire objects
2450 potential space points
Neighbour search...
157166 tests to find 78066 neighbours
Iterating with no regularization...
Begin: 1.85149e+07
0 1.76181e+07
1 1.75775e+07
2 1.75744e+07
Now with regularization...
Begin: 1.66619e+07
0 1.6651e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 210474 at 26-Sep-2024 12:42:51 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 86341 traces, 86341 summary
Retagger: tagging trace set: gauss with 61199 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 61199 traces tagged "gauss"
FrameSaver: q=1.26172e+06 n=1037455 tag=gauss
wclsFrameSaver: saving 86341 traces tagged "wiener"
FrameSaver: q=1.37248e+06 n=915189 tag=wiener
2700 hits in TPC 0
2866 hits in TPC 1
8198 hits in TPC 2
4515 hits in TPC 3
3771 hits in TPC 4
6199 hits in TPC 5
9849 hits in TPC 6
5553 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 63 XUs and 39 XVs -> 1 XUVs
C:0 T:2 55595 XUs and 34130 XVs -> 554 XUVs
C:0 T:3 335 XUs and 279 XVs -> 7 XUVs
C:0 T:4 1092 XUs and 875 XVs -> 105 XUVs
C:0 T:5 34037 XUs and 17732 XVs -> 499 XUVs
C:0 T:6 59747 XUs and 58393 XVs -> 1500 XUVs
C:0 T:7 2959 XUs and 4392 XVs -> 108 XUVs
2774 XUVs total
1491 collection wire objects
2774 potential space points
Neighbour search...
53722 tests to find 30908 neighbours
Iterating with no regularization...
Begin: 9.33925e+06
0 8.14479e+06
1 8.05636e+06
2 8.0479e+06
3 8.04511e+06
Now with regularization...
Begin: 7.54465e+06
0 7.54212e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 210482 at 26-Sep-2024 12:45:19 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 78570 traces, 78570 summary
Retagger: tagging trace set: gauss with 56720 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 56720 traces tagged "gauss"
FrameSaver: q=1.23026e+06 n=1009257 tag=gauss
wclsFrameSaver: saving 78570 traces tagged "wiener"
FrameSaver: q=1.34197e+06 n=904159 tag=wiener
2420 hits in TPC 0
2897 hits in TPC 1
10232 hits in TPC 2
4897 hits in TPC 3
3101 hits in TPC 4
4375 hits in TPC 5
8650 hits in TPC 6
3417 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 9309 XUs and 20023 XVs -> 2637 XUVs
C:0 T:2 67773 XUs and 64289 XVs -> 1629 XUVs
C:0 T:3 69 XUs and 93 XVs -> 6 XUVs
C:0 T:4 79 XUs and 185 XVs -> 12 XUVs
C:0 T:5 19172 XUs and 9518 XVs -> 774 XUVs
C:0 T:6 84890 XUs and 86376 XVs -> 2643 XUVs
C:0 T:7 144 XUs and 146 XVs -> 5 XUVs
7706 XUVs total
1978 collection wire objects
7706 potential space points
Neighbour search...
556260 tests to find 300028 neighbours
Iterating with no regularization...
Begin: 9.92194e+06
0 8.91284e+06
1 8.85021e+06
2 8.8417e+06
Now with regularization...
Begin: 8.30019e+06
0 8.29703e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 1 event 210482 TrkackID 1 startx, y, z 162.214 371.832 458.378 wireno, peakt tpcno 1 2794.41 5 dist 0min T, max_T1822.62 3911
no of hits closeby 1 event 210482 TrkackID 1 startx, y, z 162.214 371.832 458.378 wireno, peakt tpcno 1 2794.41 5 dist 0min T, max_T1822.62 3911
26-Sep-2024 12:47:44 BST Closed output file "np04hd_raw_run029081_0876_dataflow2_datawriter_0_20240905T063412_reco_stage1_reco_stage2_20240926T114744_keepup.root"
26-Sep-2024 12:47:44 BST Closed input file "np04hd_raw_run029081_0876_dataflow2_datawriter_0_20240905T063412_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 92.9252 127.464 217.888 122.887 25.4077 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000556084 0.00231241 0.004255 0.00211839 0.000979175 30
produce:fembfilter:PDHDFEMBFilter 0.64871 0.666004 0.716254 0.660941 0.0142392 30
produce:wclsdatahd:WireCellToolkit 45.0706 46.6289 49.0677 46.4271 1.0476 30
produce:gaushit:GausHitFinder 0.532104 0.828446 1.31537 0.81568 0.190796 30
produce:nhitsfilter:NumberOfHitsFilter 0.000309524 0.000429062 0.000571631 0.000422325 7.28096e-05 30
produce:reco3d:SpacePointSolver 0.323916 1.11473 3.38129 0.880975 0.743795 30
produce:hitpdune:DisambigFromSpacePoints 0.247746 0.706861 1.34714 0.619481 0.303809 30
produce:pandora:StandardPandora 6.76546 15.6818 37.6738 14.1426 7.09707 30
produce:pandoraWriter:StandardPandora 0.113754 0.166894 0.299307 0.158991 0.0383338 30
produce:pandoraTrack:LArPandoraTrackCreation 0.719224 1.85873 4.65475 1.73907 0.848949 30
produce:pandoraShower:LArPandoraShowerCreation 0.863232 2.29916 8.81389 1.72418 1.58816 30
produce:pandoracalo:Calorimetry 0.389895 0.905866 2.28787 0.873018 0.373839 30
produce:pandoracalonosce:Calorimetry 0.345659 0.826563 2.12999 0.76603 0.355034 30
produce:pandorapid:Chi2ParticleID 0.00138289 0.00212343 0.00296479 0.00214632 0.000398888 30
produce:pandoraShowercalo:ShowerCalorimetry 0.729546 2.0363 7.99939 1.49991 1.38427 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.67681 1.95897 7.2935 1.56461 1.29271 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0884221 0.149199 0.201958 0.148118 0.0283919 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0297767 0.0497116 0.0770741 0.0487316 0.0104445 30
produce:emtrkmichelid:EmTrackMichelId 33.5707 49.8673 91.1509 47.6583 11.4567 30
[art]:TriggerResults:TriggerResultInserter 2.3965e-05 4.47965e-05 9.1381e-05 4.15905e-05 1.496e-05 30
end_path:out1:RootOutput 8.144e-06 1.07238e-05 2.4908e-05 9.572e-06 3.5437e-06 30
end_path:michelremoving:michelremoving 0.0228717 0.0816329 0.268032 0.0409951 0.0712026 30
end_path:michelremovinggnocchi:michelremoving 0.0197672 0.0772506 0.268215 0.0360348 0.0714329 30
end_path:calibanagnocchi:CalibAnaTree 0.255694 0.62906 1.67603 0.571439 0.290424 30
end_path:out1:RootOutput(write) 0.520915 0.813203 1.28507 0.79027 0.167694 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4949.83 MB
Peak resident set size usage (VmHWM): 2369.5 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0876_dataflow2_datawriter_0_20240905T063412_reco_stage1_reco_stage2_20240926T114744_keepup.root
\tHists: np04hd_raw_run029081_0876_dataflow2_datawriter_0_20240905T063412_reco_stage1_reco_stage2_20240926T114744_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
