Jobsub ID 78475.153@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78475.153@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3448 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:03:54 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:17:37 |
| From worker node | Hostname | wn027.hep |
| cpuinfo | Intel(R) Xeon(R) Gold 5215 CPU @ 2.50GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:33:54 |
| Input files | hd-protodune:np04hd_raw_run029094_0143_dataflow1_datawriter_0_20240906T094310.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:17:37 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
XUVs
C:0 T:3 34 XUs and 46 XVs -> 5 XUVs
C:0 T:4 312 XUs and 291 XVs -> 13 XUVs
C:0 T:5 29446 XUs and 22223 XVs -> 918 XUVs
C:0 T:6 33979 XUs and 12674 XVs -> 833 XUVs
C:0 T:7 463 XUs and 366 XVs -> 28 XUVs
4675 XUVs total
1524 collection wire objects
4675 potential space points
Neighbour search...
279331 tests to find 161150 neighbours
Iterating with no regularization...
Begin: 1.09653e+07
0 9.75841e+06
1 9.67612e+06
2 9.66427e+06
3 9.66024e+06
Now with regularization...
Begin: 9.03774e+06
0 9.03434e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29094 subRun: 1 event: 34545 at 26-Sep-2024 12:49:00 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 36569 traces, 36569 summary
Retagger: tagging trace set: gauss with 25457 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 25457 traces tagged "gauss"
FrameSaver: q=572008 n=456254 tag=gauss
wclsFrameSaver: saving 36569 traces tagged "wiener"
FrameSaver: q=620692 n=412995 tag=wiener
657 hits in TPC 0
345 hits in TPC 1
4578 hits in TPC 2
2946 hits in TPC 3
2509 hits in TPC 4
3440 hits in TPC 5
2588 hits in TPC 6
1200 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 21 XUs and 44 XVs -> 5 XUVs
C:0 T:2 7182 XUs and 9069 XVs -> 414 XUVs
C:0 T:3 119 XUs and 123 XVs -> 9 XUVs
C:0 T:4 227 XUs and 387 XVs -> 20 XUVs
C:0 T:5 7160 XUs and 12557 XVs -> 376 XUVs
C:0 T:6 1949 XUs and 4669 XVs -> 278 XUVs
C:0 T:7 56 XUs and 73 XVs -> 2 XUVs
1104 XUVs total
680 collection wire objects
1104 potential space points
Neighbour search...
16916 tests to find 10362 neighbours
Iterating with no regularization...
Begin: 4.36102e+06
0 4.13028e+06
1 4.10507e+06
2 4.10242e+06
Now with regularization...
Begin: 3.81505e+06
0 3.81457e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29094 subRun: 1 event: 34577 at 26-Sep-2024 12:50:07 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 93531 traces, 93531 summary
Retagger: tagging trace set: gauss with 69153 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69153 traces tagged "gauss"
FrameSaver: q=1.54328e+06 n=1277822 tag=gauss
wclsFrameSaver: saving 93531 traces tagged "wiener"
FrameSaver: q=1.66439e+06 n=1154727 tag=wiener
4709 hits in TPC 0
2517 hits in TPC 1
8255 hits in TPC 2
6393 hits in TPC 3
6272 hits in TPC 4
10757 hits in TPC 5
7404 hits in TPC 6
3459 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 300 XUs and 269 XVs -> 5 XUVs
C:0 T:2 31107 XUs and 37253 XVs -> 669 XUVs
C:0 T:3 918 XUs and 946 XVs -> 42 XUVs
C:0 T:4 173 XUs and 234 XVs -> 10 XUVs
C:0 T:5 54169 XUs and 69933 XVs -> 6054 XUVs
C:0 T:6 21256 XUs and 18466 XVs -> 1006 XUVs
C:0 T:7 758 XUs and 1130 XVs -> 84 XUVs
7870 XUVs total
2162 collection wire objects
7870 potential space points
Neighbour search...
838234 tests to find 445038 neighbours
Iterating with no regularization...
Begin: 1.00422e+07
0 8.81465e+06
1 8.71451e+06
2 8.69877e+06
3 8.693e+06
Now with regularization...
Begin: 8.02312e+06
0 8.01774e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 34577 TrkackID 12 startx, y, z -6.27112 299.597 15.1673 wireno, peakt tpcno 251 5 2 dist 0min T, max_T5 1894.12
no of hits closeby 0 event 34577 TrkackID 12 startx, y, z -6.27112 299.597 15.1673 wireno, peakt tpcno 251 5 2 dist 0min T, max_T5 1894.12
26-Sep-2024 12:52:30 BST Closed output file "np04hd_raw_run029094_0143_dataflow1_datawriter_0_20240906T094310_reco_stage1_reco_stage2_20240926T115230_keepup.root"
26-Sep-2024 12:52:30 BST Closed input file "np04hd_raw_run029094_0143_dataflow1_datawriter_0_20240906T094310_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 66.2171 99.4236 142.351 97.0043 16.1967 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000396321 0.00143353 0.00440535 0.00108077 0.00106348 30
produce:fembfilter:PDHDFEMBFilter 0.638997 0.667869 0.800339 0.658946 0.0349519 30
produce:wclsdatahd:WireCellToolkit 40.6379 46.1429 51.6012 45.9136 3.54626 30
produce:gaushit:GausHitFinder 0.43518 0.752089 1.06106 0.761272 0.149751 30
produce:nhitsfilter:NumberOfHitsFilter 0.000295125 0.000401934 0.000648687 0.000385636 7.03151e-05 30
produce:reco3d:SpacePointSolver 0.231293 0.749527 1.96313 0.671173 0.418495 30
produce:hitpdune:DisambigFromSpacePoints 0.198978 0.549698 1.16166 0.503001 0.217304 30
produce:pandora:StandardPandora 4.4719 12.7221 21.0064 12.0465 4.12999 30
produce:pandoraWriter:StandardPandora 0.112944 0.160835 0.229386 0.161063 0.0265655 30
produce:pandoraTrack:LArPandoraTrackCreation 0.903581 2.50102 5.20664 2.43028 0.906626 30
produce:pandoraShower:LArPandoraShowerCreation 0.689833 2.95784 7.36882 2.7418 1.38802 30
produce:pandoracalo:Calorimetry 0.34353 1.01642 2.70802 0.942276 0.421407 30
produce:pandoracalonosce:Calorimetry 0.298251 0.931798 2.44737 0.837875 0.383835 30
produce:pandorapid:Chi2ParticleID 0.0012922 0.00185728 0.00365249 0.00174838 0.00041709 30
produce:pandoraShowercalo:ShowerCalorimetry 0.53291 2.47544 6.97488 2.12038 1.26129 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.541167 2.45471 6.75155 2.23364 1.2671 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0809257 0.129517 0.182591 0.125175 0.027917 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0342581 0.0519573 0.0706657 0.0498351 0.00992355 30
produce:emtrkmichelid:EmTrackMichelId 12.2871 23.28 34.1759 22.298 4.75869 30
[art]:TriggerResults:TriggerResultInserter 2.6211e-05 4.60411e-05 8.8208e-05 4.33715e-05 1.16986e-05 30
end_path:out1:RootOutput 9.406e-06 1.388e-05 3.3002e-05 1.3368e-05 4.53535e-06 30
end_path:michelremoving:michelremoving 0.0282345 0.0864365 0.345771 0.0530437 0.0708618 30
end_path:michelremovinggnocchi:michelremoving 0.0237399 0.0779383 0.250006 0.0466964 0.0615246 30
end_path:calibanagnocchi:CalibAnaTree 0.249094 0.72607 2.04219 0.653845 0.333901 30
end_path:out1:RootOutput(write) 0.402933 0.763902 1.24787 0.743924 0.171644 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6075.24 MB
Peak resident set size usage (VmHWM): 2770.8 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029094_0143_dataflow1_datawriter_0_20240906T094310_reco_stage1_reco_stage2_20240926T115230_keepup.root
\tHists: np04hd_raw_run029094_0143_dataflow1_datawriter_0_20240906T094310_reco_stage1_reco_stage2_20240926T115230_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
