Jobsub ID 263685.137@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263685.137@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3448 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 13:08:36 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce4 |
| Last heartbeat | 2024-09-26 14:58:06 |
| From worker node | Hostname | n227.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 13:10:01 |
| Input files | hd-protodune:np04hd_raw_run029097_0168_dataflow1_datawriter_0_20240906T135943.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 14:58:06 |
| Saved logs | justin-logs:263685.137-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
s"
FrameSaver: q=1.24629e+06 n=1007806 tag=gauss
wclsFrameSaver: saving 73864 traces tagged "wiener"
FrameSaver: q=1.35899e+06 n=922517 tag=wiener
3687 hits in TPC 0
4186 hits in TPC 1
4906 hits in TPC 2
2687 hits in TPC 3
3555 hits in TPC 4
6045 hits in TPC 5
8006 hits in TPC 6
5774 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1102 XUs and 1299 XVs -> 112 XUVs
C:0 T:2 11282 XUs and 14362 XVs -> 1780 XUVs
C:0 T:3 38 XUs and 66 XVs -> 1 XUVs
C:0 T:4 29 XUs and 22 XVs -> 2 XUVs
C:0 T:5 14150 XUs and 25511 XVs -> 994 XUVs
C:0 T:6 33553 XUs and 29200 XVs -> 4030 XUVs
C:0 T:7 368 XUs and 345 XVs -> 9 XUVs
6928 XUVs total
1909 collection wire objects
6928 potential space points
Neighbour search...
464930 tests to find 278684 neighbours
Iterating with no regularization...
Begin: 9.05397e+06
0 7.88225e+06
1 7.77927e+06
2 7.76075e+06
3 7.75462e+06
Now with regularization...
Begin: 7.04943e+06
0 7.04495e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29097 subRun: 1 event: 40553 at 26-Sep-2024 15:53:49 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 83634 traces, 83634 summary
Retagger: tagging trace set: gauss with 64098 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 64098 traces tagged "gauss"
FrameSaver: q=1.54355e+06 n=1260313 tag=gauss
wclsFrameSaver: saving 83634 traces tagged "wiener"
FrameSaver: q=1.64411e+06 n=1150242 tag=wiener
4789 hits in TPC 0
4897 hits in TPC 1
6535 hits in TPC 2
3854 hits in TPC 3
3197 hits in TPC 4
4816 hits in TPC 5
10494 hits in TPC 6
7210 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1133 XUs and 1034 XVs -> 16 XUVs
C:0 T:2 26054 XUs and 69886 XVs -> 1568 XUVs
C:0 T:3 31 XUs and 39 XVs -> 3 XUVs
C:0 T:4 45 XUs and 60 XVs -> 4 XUVs
C:0 T:5 9325 XUs and 7676 XVs -> 289 XUVs
C:0 T:6 53419 XUs and 77744 XVs -> 2406 XUVs
C:0 T:7 135 XUs and 103 XVs -> 3 XUVs
4289 XUVs total
2075 collection wire objects
4289 potential space points
Neighbour search...
144733 tests to find 72066 neighbours
Iterating with no regularization...
Begin: 3.25946e+07
0 2.85367e+07
1 2.8252e+07
2 2.82343e+07
Now with regularization...
Begin: 2.75399e+07
0 2.75328e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29097 subRun: 1 event: 40561 at 26-Sep-2024 15:55:52 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 46329 traces, 46329 summary
Retagger: tagging trace set: gauss with 35372 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 35372 traces tagged "gauss"
FrameSaver: q=785975 n=663397 tag=gauss
wclsFrameSaver: saving 46329 traces tagged "wiener"
FrameSaver: q=833751 n=603072 tag=wiener
1520 hits in TPC 0
2374 hits in TPC 1
6043 hits in TPC 2
5095 hits in TPC 3
2180 hits in TPC 4
3154 hits in TPC 5
2472 hits in TPC 6
1485 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 19 XUs and 48 XVs -> 0 XUVs
C:0 T:2 16362 XUs and 14833 XVs -> 646 XUVs
C:0 T:3 235 XUs and 354 XVs -> 16 XUVs
C:0 T:4 54 XUs and 60 XVs -> 3 XUVs
C:0 T:5 5957 XUs and 4632 XVs -> 143 XUVs
C:0 T:6 2969 XUs and 2877 XVs -> 127 XUVs
C:0 T:7 85 XUs and 141 XVs -> 5 XUVs
940 XUVs total
583 collection wire objects
940 potential space points
Neighbour search...
9544 tests to find 6148 neighbours
Iterating with no regularization...
Begin: 4.09646e+06
0 3.93283e+06
1 3.92433e+06
2 3.92379e+06
Now with regularization...
Begin: 3.70512e+06
0 3.70501e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 15:57:35 BST Closed output file "np04hd_raw_run029097_0168_dataflow1_datawriter_0_20240906T135943_reco_stage1_reco_stage2_20240926T145735_keepup.root"
26-Sep-2024 15:57:35 BST Closed input file "np04hd_raw_run029097_0168_dataflow1_datawriter_0_20240906T135943_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 74.8572 114.81 153.972 118.261 19.8631 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000368363 0.00137142 0.00313089 0.00115485 0.000840525 30
produce:fembfilter:PDHDFEMBFilter 0.698884 0.985034 1.20419 1.04654 0.165742 30
produce:wclsdatahd:WireCellToolkit 45.0464 58.0218 65.7611 60.7328 6.56161 30
produce:gaushit:GausHitFinder 0.462693 1.0656 1.67345 0.995901 0.297052 30
produce:nhitsfilter:NumberOfHitsFilter 0.000237135 0.000434616 0.000708357 0.000418192 0.000106914 30
produce:reco3d:SpacePointSolver 0.187478 1.31997 3.33276 1.2344 0.858566 30
produce:hitpdune:DisambigFromSpacePoints 0.239183 0.855833 1.62898 0.781805 0.404657 30
produce:pandora:StandardPandora 6.37544 17.5203 34.8646 17.7454 6.47603 30
produce:pandoraWriter:StandardPandora 0.0929431 0.193901 0.297667 0.192282 0.0569792 30
produce:pandoraTrack:LArPandoraTrackCreation 0.540001 1.70709 4.30983 1.60042 0.774353 30
produce:pandoraShower:LArPandoraShowerCreation 0.622698 1.99598 3.76596 1.77231 0.813082 30
produce:pandoracalo:Calorimetry 0.286626 0.829169 1.47208 0.812613 0.317736 30
produce:pandoracalonosce:Calorimetry 0.256839 0.757223 1.67545 0.728257 0.316944 30
produce:pandorapid:Chi2ParticleID 0.0011456 0.00249103 0.0044178 0.00255628 0.000839801 30
produce:pandoraShowercalo:ShowerCalorimetry 0.483191 1.77013 3.59972 1.54312 0.8139 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.478755 1.73662 3.52487 1.49618 0.80365 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0979482 0.190299 0.304107 0.188388 0.0581048 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0317708 0.0744008 0.126094 0.0731168 0.0244025 30
produce:emtrkmichelid:EmTrackMichelId 13.1406 23.4831 33.6297 24.8252 5.07539 30
[art]:TriggerResults:TriggerResultInserter 2.3383e-05 7.01847e-05 0.000484538 3.6618e-05 0.000111811 30
end_path:out1:RootOutput 7.182e-06 1.73812e-05 0.000143569 1.08035e-05 2.42666e-05 30
end_path:michelremoving:michelremoving 0.022282 0.15642 0.97534 0.0601936 0.204612 30
end_path:michelremovinggnocchi:michelremoving 0.0183407 0.149413 0.951545 0.0509082 0.201847 30
end_path:calibanagnocchi:CalibAnaTree 0.186758 0.67238 1.34851 0.684126 0.272098 30
end_path:out1:RootOutput(write) 0.552409 1.1667 1.72581 1.11218 0.321202 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5836.31 MB
Peak resident set size usage (VmHWM): 2976.32 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029097_0168_dataflow1_datawriter_0_20240906T135943_reco_stage1_reco_stage2_20240926T145735_keepup.root
\tHists: np04hd_raw_run029097_0168_dataflow1_datawriter_0_20240906T135943_reco_stage1_reco_stage2_20240926T145735_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
