Jobsub ID 263339.91@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263339.91@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:13:47 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:25:28 |
| From worker node | Hostname | wn035.hep |
| cpuinfo | AMD EPYC 7352 24-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:45:04 |
| Input files | hd-protodune:np04hd_raw_run029081_0781_dataflow7_datawriter_0_20240905T051406.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:25:28 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
eSaver: q=1.04226e+06 n=796777 tag=wiener
2658 hits in TPC 0
2660 hits in TPC 1
5408 hits in TPC 2
3875 hits in TPC 3
4250 hits in TPC 4
3956 hits in TPC 5
4985 hits in TPC 6
3552 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 10542 XUs and 20757 XVs -> 4249 XUVs
C:0 T:2 11226 XUs and 10366 XVs -> 595 XUVs
C:0 T:3 156 XUs and 149 XVs -> 7 XUVs
C:0 T:4 7 XUs and 13 XVs -> 0 XUVs
C:0 T:5 12094 XUs and 11466 XVs -> 390 XUVs
C:0 T:6 18381 XUs and 13180 XVs -> 319 XUVs
C:0 T:7 472 XUs and 841 XVs -> 25 XUVs
5585 XUVs total
1138 collection wire objects
5585 potential space points
Neighbour search...
1369739 tests to find 626098 neighbours
Iterating with no regularization...
Begin: 6.85887e+06
0 6.71777e+06
1 6.70679e+06
2 6.70585e+06
Now with regularization...
Begin: 6.28081e+06
0 6.28034e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 187663 TrkackID 1 startx, y, z -94.6514 274.309 447.158 wireno, peakt tpcno 435 4.24643 2 dist 0min T, max_T4.24643 2701.99
no of hits closeby 0 event 187663 TrkackID 1 startx, y, z -94.6514 274.309 447.158 wireno, peakt tpcno 435 4.24643 2 dist 0min T, max_T4.24643 2701.99
Begin processing the 29th record. run: 29081 subRun: 1 event: 187671 at 26-Sep-2024 12:53:33 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70937 traces, 70937 summary
Retagger: tagging trace set: gauss with 54846 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 54846 traces tagged "gauss"
FrameSaver: q=1.26492e+06 n=1059916 tag=gauss
wclsFrameSaver: saving 70937 traces tagged "wiener"
FrameSaver: q=1.34019e+06 n=967915 tag=wiener
3431 hits in TPC 0
2520 hits in TPC 1
7830 hits in TPC 2
6679 hits in TPC 3
3004 hits in TPC 4
5341 hits in TPC 5
5642 hits in TPC 6
4023 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 567 XUs and 333 XVs -> 9 XUVs
C:0 T:2 29072 XUs and 20095 XVs -> 563 XUVs
C:0 T:3 1163 XUs and 955 XVs -> 30 XUVs
C:0 T:4 15 XUs and 22 XVs -> 2 XUVs
C:0 T:5 21756 XUs and 21824 XVs -> 754 XUVs
C:0 T:6 16765 XUs and 16819 XVs -> 601 XUVs
C:0 T:7 96 XUs and 76 XVs -> 2 XUVs
1961 XUVs total
1151 collection wire objects
1961 potential space points
Neighbour search...
43299 tests to find 26320 neighbours
Iterating with no regularization...
Begin: 1.37338e+07
0 1.30919e+07
1 1.30049e+07
2 1.30012e+07
Now with regularization...
Begin: 1.25906e+07
0 1.25894e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 187679 at 26-Sep-2024 12:54:59 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 60373 traces, 60373 summary
Retagger: tagging trace set: gauss with 46825 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 46825 traces tagged "gauss"
FrameSaver: q=1.04083e+06 n=878003 tag=gauss
wclsFrameSaver: saving 60373 traces tagged "wiener"
FrameSaver: q=1.11721e+06 n=803006 tag=wiener
4050 hits in TPC 0
3717 hits in TPC 1
4807 hits in TPC 2
3342 hits in TPC 3
3806 hits in TPC 4
5111 hits in TPC 5
4464 hits in TPC 6
3333 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 16052 XUs and 13625 XVs -> 673 XUVs
C:0 T:2 10774 XUs and 10793 XVs -> 192 XUVs
C:0 T:3 12 XUs and 27 XVs -> 1 XUVs
C:0 T:4 996 XUs and 900 XVs -> 62 XUVs
C:0 T:5 10249 XUs and 8741 XVs -> 568 XUVs
C:0 T:6 13257 XUs and 8561 XVs -> 555 XUVs
C:0 T:7 12 XUs and 14 XVs -> 0 XUVs
2051 XUVs total
1007 collection wire objects
2051 potential space points
Neighbour search...
42021 tests to find 25570 neighbours
Iterating with no regularization...
Begin: 1.03376e+07
0 1.01447e+07
1 1.01377e+07
Now with regularization...
Begin: 9.5761e+06
0 9.57533e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:56:12 BST Closed output file "np04hd_raw_run029081_0781_dataflow7_datawriter_0_20240905T051406_reco_stage1_reco_stage2_20240926T115612_keepup.root"
26-Sep-2024 12:56:12 BST Closed input file "np04hd_raw_run029081_0781_dataflow7_datawriter_0_20240905T051406_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 69.0628 89.5989 142.75 84.5281 17.0287 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000432425 0.0030027 0.0144785 0.00237903 0.00267205 30
produce:fembfilter:PDHDFEMBFilter 0.515983 0.704182 0.903277 0.70163 0.0992931 30
produce:wclsdatahd:WireCellToolkit 35.9339 40.0605 43.9936 39.6171 2.11429 30
produce:gaushit:GausHitFinder 0.385484 0.603902 0.861056 0.589249 0.122131 30
produce:nhitsfilter:NumberOfHitsFilter 0.000220059 0.000368383 0.000730583 0.000334408 0.00011115 30
produce:reco3d:SpacePointSolver 0.152987 0.925667 4.33325 0.610192 0.989674 30
produce:hitpdune:DisambigFromSpacePoints 0.193965 0.416127 1.24455 0.370656 0.202427 30
produce:pandora:StandardPandora 4.49974 12.4399 40.5788 10.3411 8.56615 30
produce:pandoraWriter:StandardPandora 0.0825038 0.144879 0.249221 0.130381 0.0423268 30
produce:pandoraTrack:LArPandoraTrackCreation 0.42268 1.13547 2.40473 1.07674 0.506316 30
produce:pandoraShower:LArPandoraShowerCreation 0.410285 1.33377 3.2559 1.11714 0.680814 30
produce:pandoracalo:Calorimetry 0.229453 0.594142 1.5541 0.548977 0.269883 30
produce:pandoracalonosce:Calorimetry 0.194918 0.535719 1.18477 0.482077 0.217984 30
produce:pandorapid:Chi2ParticleID 0.000907669 0.00160625 0.00284674 0.00154241 0.000436895 30
produce:pandoraShowercalo:ShowerCalorimetry 0.324715 1.27134 3.36854 1.06942 0.665032 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.317452 1.25991 2.90953 1.11971 0.618782 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0632736 0.131943 0.208352 0.123089 0.037591 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.023198 0.0458376 0.0783698 0.0439099 0.011314 30
produce:emtrkmichelid:EmTrackMichelId 17.4327 26.64 40.6524 25.8905 6.05509 30
[art]:TriggerResults:TriggerResultInserter 1.4037e-05 2.5963e-05 6.2429e-05 2.063e-05 1.32605e-05 30
end_path:out1:RootOutput 3.256e-06 4.9112e-06 1.7353e-05 4.063e-06 2.63147e-06 30
end_path:michelremoving:michelremoving 0.0161788 0.0581058 0.187313 0.0305038 0.0537236 30
end_path:michelremovinggnocchi:michelremoving 0.0145148 0.0554797 0.183715 0.029254 0.0509695 30
end_path:calibanagnocchi:CalibAnaTree 0.176674 0.391936 0.925895 0.340285 0.159172 30
end_path:out1:RootOutput(write) 0.447143 0.706563 1.05632 0.673483 0.166938 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4915.34 MB
Peak resident set size usage (VmHWM): 2334.18 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0781_dataflow7_datawriter_0_20240905T051406_reco_stage1_reco_stage2_20240926T115612_keepup.root
\tHists: np04hd_raw_run029081_0781_dataflow7_datawriter_0_20240905T051406_reco_stage1_reco_stage2_20240926T115612_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
