Jobsub ID 263331.28@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263331.28@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3448 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:10:04 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 11:57:25 |
| From worker node | Hostname | wn034.hep |
| cpuinfo | AMD EPYC 75F3 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:43:38 |
| Input files | hd-protodune:np04hd_raw_run029094_0045_dataflow3_datawriter_0_20240906T085443.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 11:57:25 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
Begin: 2.00635e+07
0 1.63005e+07
1 1.58516e+07
2 1.58095e+07
3 1.57986e+07
Now with regularization...
Begin: 1.48477e+07
0 1.48391e+07
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29094 subRun: 1 event: 11083 at 26-Sep-2024 12:34:52 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 63658 traces, 63658 summary
Retagger: tagging trace set: gauss with 47656 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 47656 traces tagged "gauss"
FrameSaver: q=1.08492e+06 n=902383 tag=gauss
wclsFrameSaver: saving 63658 traces tagged "wiener"
FrameSaver: q=1.14934e+06 n=823647 tag=wiener
2479 hits in TPC 0
3202 hits in TPC 1
5843 hits in TPC 2
4385 hits in TPC 3
3410 hits in TPC 4
5224 hits in TPC 5
4400 hits in TPC 6
3334 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 918 XUs and 410 XVs -> 7 XUVs
C:0 T:2 12911 XUs and 14177 XVs -> 447 XUVs
C:0 T:3 147 XUs and 143 XVs -> 3 XUVs
C:0 T:4 20 XUs and 45 XVs -> 2 XUVs
C:0 T:5 13723 XUs and 15178 XVs -> 1451 XUVs
C:0 T:6 23744 XUs and 17684 XVs -> 410 XUVs
C:0 T:7 44 XUs and 128 XVs -> 5 XUVs
2325 XUVs total
1315 collection wire objects
2325 potential space points
Neighbour search...
37137 tests to find 22228 neighbours
Iterating with no regularization...
Begin: 8.64115e+06
0 8.39243e+06
1 8.36619e+06
2 8.36363e+06
Now with regularization...
Begin: 7.83738e+06
0 7.8369e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 2 event 11083 TrkackID 6 startx, y, z 59.6934 579.498 232.412 wireno, peakt tpcno 66 2036.53 5 dist 0min T, max_T2002.47 3898.5
no of hits closeby 0 event 11083 TrkackID 9 startx, y, z 211.2 289.074 279.537 wireno, peakt tpcno 137 2317.72 5 dist 0min T, max_T1.62018 2689.5
no of hits closeby 2 event 11083 TrkackID 6 startx, y, z 59.6934 579.498 232.412 wireno, peakt tpcno 66 2036.53 5 dist 0min T, max_T2002.47 3898.5
no of hits closeby 0 event 11083 TrkackID 9 startx, y, z 211.2 289.074 279.537 wireno, peakt tpcno 137 2317.72 5 dist 0min T, max_T1.62018 2689.5
Begin processing the 30th record. run: 29094 subRun: 1 event: 11091 at 26-Sep-2024 12:35:47 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68270 traces, 68270 summary
Retagger: tagging trace set: gauss with 48075 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48075 traces tagged "gauss"
FrameSaver: q=1.10097e+06 n=837791 tag=gauss
wclsFrameSaver: saving 68270 traces tagged "wiener"
FrameSaver: q=1.19631e+06 n=750384 tag=wiener
2522 hits in TPC 0
2113 hits in TPC 1
7809 hits in TPC 2
5813 hits in TPC 3
3105 hits in TPC 4
4798 hits in TPC 5
4058 hits in TPC 6
2340 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 237 XUs and 95 XVs -> 1 XUVs
C:0 T:2 30649 XUs and 42792 XVs -> 2512 XUVs
C:0 T:3 811 XUs and 1240 XVs -> 61 XUVs
C:0 T:4 40 XUs and 108 XVs -> 8 XUVs
C:0 T:5 16541 XUs and 14644 XVs -> 746 XUVs
C:0 T:6 8187 XUs and 11103 XVs -> 397 XUVs
C:0 T:7 34 XUs and 37 XVs -> 3 XUVs
3728 XUVs total
1669 collection wire objects
3728 potential space points
Neighbour search...
98510 tests to find 56956 neighbours
Iterating with no regularization...
Begin: 7.03424e+06
0 6.19829e+06
1 6.12939e+06
2 6.12166e+06
3 6.1197e+06
Now with regularization...
Begin: 5.50578e+06
0 5.50437e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:36:41 BST Closed output file "np04hd_raw_run029094_0045_dataflow3_datawriter_0_20240906T085443_reco_stage1_reco_stage2_20240926T113641_keepup.root"
26-Sep-2024 12:36:41 BST Closed input file "np04hd_raw_run029094_0045_dataflow3_datawriter_0_20240906T085443_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 38.0163 55.4264 96.2651 54.0066 10.7928 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000235196 0.000909265 0.0014808 0.000926182 0.000339206 30
produce:fembfilter:PDHDFEMBFilter 0.365903 0.382518 0.434287 0.381905 0.0132384 30
produce:wclsdatahd:WireCellToolkit 24.7213 25.6914 27.2485 25.6231 0.636766 30
produce:gaushit:GausHitFinder 0.194585 0.354563 0.518826 0.364943 0.0827802 30
produce:nhitsfilter:NumberOfHitsFilter 0.000117674 0.000204168 0.000288618 0.000203912 4.15594e-05 30
produce:reco3d:SpacePointSolver 0.0692809 0.3532 1.11572 0.282962 0.246887 30
produce:hitpdune:DisambigFromSpacePoints 0.057918 0.237711 0.798703 0.215309 0.143268 30
produce:pandora:StandardPandora 1.57004 6.2331 29.5219 5.20698 4.89659 30
produce:pandoraWriter:StandardPandora 0.0372595 0.0767004 0.135243 0.0766459 0.0229632 30
produce:pandoraTrack:LArPandoraTrackCreation 0.215489 0.602419 1.76435 0.564846 0.291139 30
produce:pandoraShower:LArPandoraShowerCreation 0.187415 0.704622 1.61535 0.640019 0.32411 30
produce:pandoracalo:Calorimetry 0.123804 0.332136 0.812243 0.324675 0.131972 30
produce:pandoracalonosce:Calorimetry 0.112236 0.302996 0.781083 0.298141 0.125571 30
produce:pandorapid:Chi2ParticleID 0.000669301 0.00106897 0.00198824 0.001027 0.0002593 30
produce:pandoraShowercalo:ShowerCalorimetry 0.172615 0.68055 1.49769 0.64891 0.310119 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.169444 0.663374 1.47433 0.631977 0.301284 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.038207 0.0664469 0.0982752 0.0673496 0.0151953 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0134778 0.0236086 0.038317 0.023773 0.00530745 30
produce:emtrkmichelid:EmTrackMichelId 8.95228 17.9222 29.2437 17.9768 4.26796 30
[art]:TriggerResults:TriggerResultInserter 1.1973e-05 1.68235e-05 5.4072e-05 1.49735e-05 7.66372e-06 30
end_path:out1:RootOutput 2.956e-06 4.04333e-06 1.5921e-05 3.5265e-06 2.24995e-06 30
end_path:michelremoving:michelremoving 0.00842164 0.0268758 0.171214 0.0165145 0.0322125 30
end_path:michelremovinggnocchi:michelremoving 0.00790701 0.0248648 0.139363 0.0163347 0.0273017 30
end_path:calibanagnocchi:CalibAnaTree 0.0998096 0.254367 0.653448 0.249245 0.109368 30
end_path:out1:RootOutput(write) 0.206672 0.404672 0.614331 0.40394 0.0940387 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4966.81 MB
Peak resident set size usage (VmHWM): 2361.3 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029094_0045_dataflow3_datawriter_0_20240906T085443_reco_stage1_reco_stage2_20240926T113641_keepup.root
\tHists: np04hd_raw_run029094_0045_dataflow3_datawriter_0_20240906T085443_reco_stage1_reco_stage2_20240926T113641_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
