Jobsub ID 78463.20@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78463.20@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3455 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:58:39 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:20:55 |
| From worker node | Hostname | wn097.hep |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:17:27 |
| Input files | hd-protodune:np04hd_raw_run029109_0606_dataflow2_datawriter_0_20240910T031332.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:20:55 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
19 hits in TPC 2
5021 hits in TPC 3
3889 hits in TPC 4
6801 hits in TPC 5
6214 hits in TPC 6
2913 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 728 XUs and 299 XVs -> 36 XUVs
C:0 T:2 34819 XUs and 39877 XVs -> 705 XUVs
C:0 T:3 430 XUs and 536 XVs -> 6 XUVs
C:0 T:4 386 XUs and 393 XVs -> 9 XUVs
C:0 T:5 26696 XUs and 42529 XVs -> 2915 XUVs
C:0 T:6 38457 XUs and 54653 XVs -> 990 XUVs
C:0 T:7 183 XUs and 126 XVs -> 3 XUVs
4664 XUVs total
1267 collection wire objects
4664 potential space points
Neighbour search...
352896 tests to find 205296 neighbours
Iterating with no regularization...
Begin: 1.8751e+07
0 1.6091e+07
1 1.59663e+07
2 1.59399e+07
3 1.59306e+07
Now with regularization...
Begin: 1.53968e+07
0 1.5392e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29109 subRun: 1 event: 145706 at 26-Sep-2024 12:54:38 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 49341 traces, 49341 summary
Retagger: tagging trace set: gauss with 37083 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 37083 traces tagged "gauss"
FrameSaver: q=774904 n=691248 tag=gauss
wclsFrameSaver: saving 49341 traces tagged "wiener"
FrameSaver: q=844542 n=627888 tag=wiener
4396 hits in TPC 0
5190 hits in TPC 1
6124 hits in TPC 2
3713 hits in TPC 3
2016 hits in TPC 4
3311 hits in TPC 5
1619 hits in TPC 6
388 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 449 XUs and 515 XVs -> 91 XUVs
C:0 T:1 8469 XUs and 8674 XVs -> 768 XUVs
C:0 T:2 10843 XUs and 17045 XVs -> 886 XUVs
C:0 T:3 1493 XUs and 2846 XVs -> 229 XUVs
C:0 T:4 3 XUs and 0 XVs -> 0 XUVs
C:0 T:5 11123 XUs and 9765 XVs -> 257 XUVs
C:0 T:6 832 XUs and 2065 XVs -> 75 XUVs
C:0 T:7 1 XUs and 12 XVs -> 0 XUVs
2306 XUVs total
897 collection wire objects
2306 potential space points
Neighbour search...
73902 tests to find 41504 neighbours
Iterating with no regularization...
Begin: 3.4057e+06
0 3.12036e+06
1 3.1019e+06
2 3.10082e+06
Now with regularization...
Begin: 2.83661e+06
0 2.83628e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29109 subRun: 1 event: 145722 at 26-Sep-2024 12:56:25 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 49391 traces, 49391 summary
Retagger: tagging trace set: gauss with 34989 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 34989 traces tagged "gauss"
FrameSaver: q=731287 n=609521 tag=gauss
wclsFrameSaver: saving 49391 traces tagged "wiener"
FrameSaver: q=785436 n=540526 tag=wiener
2384 hits in TPC 0
1479 hits in TPC 1
2691 hits in TPC 2
2173 hits in TPC 3
2287 hits in TPC 4
4954 hits in TPC 5
4723 hits in TPC 6
3062 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 99 XUs and 44 XVs -> 1 XUVs
C:0 T:2 7766 XUs and 5737 XVs -> 189 XUVs
C:0 T:4 10 XUs and 10 XVs -> 0 XUVs
C:0 T:5 25161 XUs and 23343 XVs -> 611 XUVs
C:0 T:6 6905 XUs and 10250 XVs -> 195 XUVs
C:0 T:7 577 XUs and 799 XVs -> 23 XUVs
1019 XUVs total
462 collection wire objects
1019 potential space points
Neighbour search...
24445 tests to find 14954 neighbours
Iterating with no regularization...
Begin: 4.47481e+06
0 4.18374e+06
1 4.15425e+06
2 4.15053e+06
Now with regularization...
Begin: 3.951e+06
0 3.94967e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:58:03 BST Closed output file "np04hd_raw_run029109_0606_dataflow2_datawriter_0_20240910T031332_reco_stage1_reco_stage2_20240926T115803_keepup.root"
26-Sep-2024 12:58:03 BST Closed input file "np04hd_raw_run029109_0606_dataflow2_datawriter_0_20240910T031332_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 59.931 124.171 178.469 123.462 22.0773 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000521506 0.0030305 0.00458467 0.0031565 0.00102484 30
produce:fembfilter:PDHDFEMBFilter 0.719691 0.778505 0.955541 0.745917 0.0687199 30
produce:wclsdatahd:WireCellToolkit 54.205 57.766 61.1079 57.6882 1.49652 30
produce:gaushit:GausHitFinder 0.503207 0.874031 1.45688 0.832168 0.230036 30
produce:nhitsfilter:NumberOfHitsFilter 0.00027737 0.000477792 0.000751 0.000467777 9.87362e-05 30
produce:reco3d:SpacePointSolver 0.207618 2.00044 7.03137 0.943635 1.95148 29
produce:hitpdune:DisambigFromSpacePoints 0.172297 0.783488 1.93508 0.678425 0.397679 29
produce:pandora:StandardPandora 6.57292 16.2325 37.9437 14.7818 6.65207 29
produce:pandoraWriter:StandardPandora 0.113075 0.18517 0.272074 0.183262 0.0393395 29
produce:pandoraTrack:LArPandoraTrackCreation 0.840407 2.2571 5.1182 2.06595 0.957099 29
produce:pandoraShower:LArPandoraShowerCreation 0.666603 2.59168 5.14928 2.18991 1.24823 29
produce:pandoracalo:Calorimetry 0.380502 1.01037 2.06373 0.998316 0.381914 29
produce:pandoracalonosce:Calorimetry 0.351424 0.923314 1.7208 0.912386 0.3318 29
produce:pandorapid:Chi2ParticleID 0.00136621 0.00232248 0.00502225 0.00209198 0.000771111 29
produce:pandoraShowercalo:ShowerCalorimetry 0.511008 2.23272 4.83449 2.08593 1.02538 29
produce:pandoraShowercalonosce:ShowerCalorimetry 0.499684 2.14444 4.74708 1.96901 0.95008 29
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0956753 0.145616 0.204461 0.145508 0.0266527 29
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0368945 0.0543664 0.0740201 0.0550958 0.0104994 29
produce:emtrkmichelid:EmTrackMichelId 20.4913 34.5589 52.6023 34.671 6.88564 29
[art]:TriggerResults:TriggerResultInserter 2.4874e-05 5.21238e-05 0.000127493 4.7116e-05 2.18246e-05 30
end_path:out1:RootOutput 7.53e-06 1.13494e-05 2.8642e-05 9.31e-06 4.81441e-06 30
end_path:michelremoving:michelremoving 4.0302e-05 0.0621139 0.162951 0.0487709 0.0404382 30
end_path:michelremovinggnocchi:michelremoving 2.5099e-05 0.0551458 0.149658 0.0421296 0.0380935 30
end_path:calibanagnocchi:CalibAnaTree 3.7292e-05 0.691064 1.66876 0.647291 0.326498 30
end_path:out1:RootOutput(write) 0.444638 0.842019 1.15831 0.828722 0.178141 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4942.17 MB
Peak resident set size usage (VmHWM): 2346.69 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029109_0606_dataflow2_datawriter_0_20240910T031332_reco_stage1_reco_stage2_20240926T115803_keepup.root
\tHists: np04hd_raw_run029109_0606_dataflow2_datawriter_0_20240910T031332_reco_stage1_reco_stage2_20240926T115803_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
