Jobsub ID 78490.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78490.0@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3442 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:11:10 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:26:59 |
| From worker node | Hostname | wn032.hep |
| cpuinfo | Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:42:18 |
| Input files | hd-protodune:np04hd_raw_run029072_0020_dataflow6_datawriter_0_20240903T150453.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:26:59 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
10 startx, y, z 182.208 406.891 101.833 wireno, peakt tpcno 266 2288.7 2 dist 146.393min T, max_T5 2327.37
Begin processing the 29th record. run: 29072 subRun: 1 event: 5054 at 26-Sep-2024 12:56:18 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 92659 traces, 92659 summary
Retagger: tagging trace set: gauss with 68355 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 68355 traces tagged "gauss"
FrameSaver: q=1.54515e+06 n=1274504 tag=gauss
wclsFrameSaver: saving 92659 traces tagged "wiener"
FrameSaver: q=1.68059e+06 n=1159279 tag=wiener
6637 hits in TPC 0
5738 hits in TPC 1
6091 hits in TPC 2
3310 hits in TPC 3
7689 hits in TPC 4
11564 hits in TPC 5
5644 hits in TPC 6
3243 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 2872 XUs and 2445 XVs -> 71 XUVs
C:0 T:2 32799 XUs and 31316 XVs -> 824 XUVs
C:0 T:3 237 XUs and 294 XVs -> 18 XUVs
C:0 T:4 312 XUs and 426 XVs -> 17 XUVs
C:0 T:5 88911 XUs and 103954 XVs -> 5455 XUVs
C:0 T:6 44221 XUs and 36049 XVs -> 804 XUVs
C:0 T:7 0 XUs and 0 XVs -> 0 XUVs
7189 XUVs total
2494 collection wire objects
7189 potential space points
Neighbour search...
295823 tests to find 177822 neighbours
Iterating with no regularization...
Begin: 1.2053e+07
0 1.09942e+07
1 1.08929e+07
2 1.08737e+07
3 1.08672e+07
Now with regularization...
Begin: 1.01653e+07
0 1.01611e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 5 event 5054 TrkackID 0 startx, y, z 19.7071 561.746 1.06898 wireno, peakt tpcno 472 5664.05 6 dist 65.0157min T, max_T5525.68 5855.42
no of hits closeby 2 event 5054 TrkackID 10 startx, y, z 34.6986 292.333 441.397 wireno, peakt tpcno 27 3155.15 5 dist 0min T, max_T3154.42 3754.57
no of hits closeby 5 event 5054 TrkackID 0 startx, y, z 19.7071 561.746 1.06898 wireno, peakt tpcno 472 5664.05 6 dist 65.0157min T, max_T5525.68 5855.42
no of hits closeby 2 event 5054 TrkackID 10 startx, y, z 34.6986 292.333 441.397 wireno, peakt tpcno 27 3155.15 5 dist 0min T, max_T3154.42 3754.57
Begin processing the 30th record. run: 29072 subRun: 1 event: 5062 at 26-Sep-2024 12:58:34 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 91287 traces, 91287 summary
Retagger: tagging trace set: gauss with 67511 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 67511 traces tagged "gauss"
FrameSaver: q=1.537e+06 n=1230601 tag=gauss
wclsFrameSaver: saving 91287 traces tagged "wiener"
FrameSaver: q=1.70746e+06 n=1127006 tag=wiener
5051 hits in TPC 0
6831 hits in TPC 1
7168 hits in TPC 2
3626 hits in TPC 3
4857 hits in TPC 4
8016 hits in TPC 5
10402 hits in TPC 6
4546 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 2928 XUs and 1443 XVs -> 75 XUVs
C:0 T:2 26654 XUs and 81664 XVs -> 1417 XUVs
C:0 T:3 518 XUs and 779 XVs -> 12 XUVs
C:0 T:4 262 XUs and 545 XVs -> 3 XUVs
C:0 T:5 68363 XUs and 74314 XVs -> 2415 XUVs
C:0 T:6 73492 XUs and 161708 XVs -> 5418 XUVs
C:0 T:7 188 XUs and 296 XVs -> 4 XUVs
9344 XUVs total
2743 collection wire objects
9344 potential space points
Neighbour search...
749888 tests to find 313486 neighbours
Iterating with no regularization...
Begin: 3.082e+07
0 2.57778e+07
1 2.5316e+07
2 2.52376e+07
3 2.52087e+07
4 2.51946e+07
Now with regularization...
Begin: 2.44167e+07
0 2.4406e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 3 event 5062 TrkackID 16 startx, y, z -32.1254 586.986 268.437 wireno, peakt tpcno 378 3317.47 2 dist 0.67407min T, max_T3297.78 4149.24
no of hits closeby 3 event 5062 TrkackID 16 startx, y, z -32.1254 586.986 268.437 wireno, peakt tpcno 378 3317.47 2 dist 0.67407min T, max_T3297.78 4149.24
26-Sep-2024 13:00:43 BST Closed output file "np04hd_raw_run029072_0020_dataflow6_datawriter_0_20240903T150453_reco_stage1_reco_stage2_20240926T120043_keepup.root"
26-Sep-2024 13:00:43 BST Closed input file "np04hd_raw_run029072_0020_dataflow6_datawriter_0_20240903T150453_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 67.1387 96.7201 139.931 88.9377 21.7742 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000445358 0.00157994 0.0041326 0.00120489 0.00106613 30
produce:fembfilter:PDHDFEMBFilter 0.609709 0.675423 1.00266 0.64855 0.0882914 30
produce:wclsdatahd:WireCellToolkit 45.3121 49.8579 57.0712 50.1748 2.57589 30
produce:gaushit:GausHitFinder 0.405718 0.729364 1.0687 0.688733 0.196952 30
produce:nhitsfilter:NumberOfHitsFilter 0.000311204 0.000469365 0.000768354 0.00043225 0.000124513 30
produce:reco3d:SpacePointSolver 0.229493 1.29842 4.72064 0.818605 1.05186 30
produce:hitpdune:DisambigFromSpacePoints 0.199409 0.600689 1.37791 0.485217 0.322002 30
produce:pandora:StandardPandora 4.69643 13.9775 31.2584 11.6797 7.33774 30
produce:pandoraWriter:StandardPandora 0.108515 0.17385 0.30918 0.16104 0.0494277 30
produce:pandoraTrack:LArPandoraTrackCreation 0.871297 2.816 7.58408 2.63549 1.55869 30
produce:pandoraShower:LArPandoraShowerCreation 0.603149 3.38685 8.13084 2.80385 2.14335 30
produce:pandoracalo:Calorimetry 0.364005 1.11284 2.76369 1.02559 0.616039 30
produce:pandoracalonosce:Calorimetry 0.346515 1.10366 3.1316 0.958886 0.6494 30
produce:pandorapid:Chi2ParticleID 0.00116743 0.00189662 0.00559484 0.00178102 0.000810162 30
produce:pandoraShowercalo:ShowerCalorimetry 0.498261 3.03206 7.59507 2.48994 1.95197 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.42915 2.95424 7.72297 2.52428 1.80325 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0750473 0.123313 0.203446 0.117977 0.0340664 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0301992 0.0502157 0.0795867 0.0494743 0.0128684 30
produce:emtrkmichelid:EmTrackMichelId 7.86384 12.7788 22.2591 12.0233 3.36893 30
[art]:TriggerResults:TriggerResultInserter 3.0655e-05 5.05316e-05 7.9747e-05 5.0675e-05 1.06604e-05 30
end_path:out1:RootOutput 9.504e-06 1.64729e-05 4.7635e-05 1.57415e-05 6.74145e-06 30
end_path:michelremoving:michelremoving 0.0235386 0.0962519 0.31236 0.0673769 0.0743236 30
end_path:michelremovinggnocchi:michelremoving 0.0214058 0.0907681 0.315883 0.0590416 0.0754654 30
end_path:calibanagnocchi:CalibAnaTree 0.210392 0.811912 1.86481 0.694639 0.410643 30
end_path:out1:RootOutput(write) 0.419124 0.729888 1.27944 0.706617 0.204689 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5876.1 MB
Peak resident set size usage (VmHWM): 2713.53 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029072_0020_dataflow6_datawriter_0_20240903T150453_reco_stage1_reco_stage2_20240926T120043_keepup.root
\tHists: np04hd_raw_run029072_0020_dataflow6_datawriter_0_20240903T150453_reco_stage1_reco_stage2_20240926T120043_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
