Jobsub ID 263339.75@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263339.75@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:13:47 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:26:02 |
| From worker node | Hostname | wn035.hep |
| cpuinfo | AMD EPYC 7352 24-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:44:55 |
| Input files | hd-protodune:np04hd_raw_run029081_0777_dataflow2_datawriter_0_20240905T051105.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:26:02 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
meSaver: saving 56008 traces tagged "wiener"
FrameSaver: q=1.02402e+06 n=713185 tag=wiener
2696 hits in TPC 0
1926 hits in TPC 1
4509 hits in TPC 2
4762 hits in TPC 3
2083 hits in TPC 4
2128 hits in TPC 5
6833 hits in TPC 6
4071 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 281 XUs and 370 XVs -> 1 XUVs
C:0 T:2 12976 XUs and 14773 XVs -> 990 XUVs
C:0 T:3 128 XUs and 223 XVs -> 19 XUVs
C:0 T:4 120 XUs and 211 XVs -> 10 XUVs
C:0 T:5 2818 XUs and 6633 XVs -> 166 XUVs
C:0 T:6 20915 XUs and 21903 XVs -> 1193 XUVs
C:0 T:7 170 XUs and 206 XVs -> 1 XUVs
2380 XUVs total
1001 collection wire objects
2380 potential space points
Neighbour search...
87784 tests to find 46074 neighbours
Iterating with no regularization...
Begin: 8.99244e+06
0 7.88006e+06
1 7.78751e+06
2 7.77606e+06
3 7.77439e+06
Now with regularization...
Begin: 7.14721e+06
0 7.14415e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 186706 at 26-Sep-2024 12:53:35 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 45030 traces, 45030 summary
Retagger: tagging trace set: gauss with 33978 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 33978 traces tagged "gauss"
FrameSaver: q=714337 n=623329 tag=gauss
wclsFrameSaver: saving 45030 traces tagged "wiener"
FrameSaver: q=775545 n=571948 tag=wiener
2966 hits in TPC 0
2381 hits in TPC 1
5429 hits in TPC 2
3704 hits in TPC 3
2923 hits in TPC 4
4315 hits in TPC 5
1478 hits in TPC 6
1159 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 116 XUs and 380 XVs -> 0 XUVs
C:0 T:2 12104 XUs and 17133 XVs -> 526 XUVs
C:0 T:3 248 XUs and 399 XVs -> 14 XUVs
C:0 T:4 61 XUs and 128 XVs -> 3 XUVs
C:0 T:5 20509 XUs and 13380 XVs -> 529 XUVs
C:0 T:6 9137 XUs and 3172 XVs -> 187 XUVs
C:0 T:7 0 XUs and 0 XVs -> 0 XUVs
1259 XUVs total
717 collection wire objects
1259 potential space points
Neighbour search...
25559 tests to find 14390 neighbours
Iterating with no regularization...
Begin: 5.25445e+06
0 4.80085e+06
1 4.77094e+06
2 4.76843e+06
Now with regularization...
Begin: 4.58236e+06
0 4.58188e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 186722 at 26-Sep-2024 12:54:54 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70189 traces, 70189 summary
Retagger: tagging trace set: gauss with 53658 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53658 traces tagged "gauss"
FrameSaver: q=1.19928e+06 n=1090113 tag=gauss
wclsFrameSaver: saving 70189 traces tagged "wiener"
FrameSaver: q=1.27439e+06 n=1001785 tag=wiener
3996 hits in TPC 0
3378 hits in TPC 1
5797 hits in TPC 2
3808 hits in TPC 3
5068 hits in TPC 4
10307 hits in TPC 5
4885 hits in TPC 6
3936 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 98 XUs and 83 XVs -> 2 XUVs
C:0 T:2 12147 XUs and 16162 XVs -> 897 XUVs
C:0 T:3 150 XUs and 203 XVs -> 3 XUVs
C:0 T:4 600 XUs and 638 XVs -> 56 XUVs
C:0 T:5 90099 XUs and 92441 XVs -> 2842 XUVs
C:0 T:6 10387 XUs and 14560 XVs -> 607 XUVs
C:0 T:7 131 XUs and 223 XVs -> 6 XUVs
4413 XUVs total
2021 collection wire objects
4413 potential space points
Neighbour search...
214605 tests to find 123770 neighbours
Iterating with no regularization...
Begin: 1.14307e+07
0 1.04857e+07
1 1.04338e+07
2 1.04231e+07
3 1.04197e+07
Now with regularization...
Begin: 9.88021e+06
0 9.87838e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:56:20 BST Closed output file "np04hd_raw_run029081_0777_dataflow2_datawriter_0_20240905T051105_reco_stage1_reco_stage2_20240926T115620_keepup.root"
26-Sep-2024 12:56:20 BST Closed input file "np04hd_raw_run029081_0777_dataflow2_datawriter_0_20240905T051105_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 61.1136 89.7314 145.291 90.08 17.1905 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000466299 0.00198588 0.00423662 0.00184676 0.00108486 30
produce:fembfilter:PDHDFEMBFilter 0.513007 0.696071 0.99035 0.661764 0.119763 30
produce:wclsdatahd:WireCellToolkit 35.2761 39.2417 42.5702 39.3177 1.80793 30
produce:gaushit:GausHitFinder 0.384292 0.591566 0.919168 0.606717 0.111993 30
produce:nhitsfilter:NumberOfHitsFilter 0.000205251 0.000360322 0.000656191 0.000342624 9.7774e-05 30
produce:reco3d:SpacePointSolver 0.150106 0.909672 3.05182 0.755285 0.702186 30
produce:hitpdune:DisambigFromSpacePoints 0.117589 0.464692 0.831063 0.461753 0.193109 30
produce:pandora:StandardPandora 4.52655 12.6397 53.0833 11.3844 8.3861 30
produce:pandoraWriter:StandardPandora 0.0811497 0.147006 0.226137 0.147816 0.0366043 30
produce:pandoraTrack:LArPandoraTrackCreation 0.453651 1.17638 2.36699 1.12144 0.541496 30
produce:pandoraShower:LArPandoraShowerCreation 0.493046 1.42794 2.87436 1.32401 0.73112 30
produce:pandoracalo:Calorimetry 0.211509 0.57875 1.2948 0.513194 0.246229 30
produce:pandoracalonosce:Calorimetry 0.163101 0.531764 1.06753 0.472186 0.234003 30
produce:pandorapid:Chi2ParticleID 0.0010063 0.00171479 0.0030162 0.00152948 0.000516041 30
produce:pandoraShowercalo:ShowerCalorimetry 0.341037 1.3409 3.94588 1.21876 0.77445 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.341679 1.33301 4.79041 1.15881 0.886059 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0673399 0.13149 0.228834 0.122188 0.0399376 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0251503 0.0466033 0.0883389 0.0441224 0.014049 30
produce:emtrkmichelid:EmTrackMichelId 12.4316 27.0943 39.0321 27.8935 6.2503 30
[art]:TriggerResults:TriggerResultInserter 1.2935e-05 2.67851e-05 7.1116e-05 2.3224e-05 1.33613e-05 30
end_path:out1:RootOutput 3.777e-06 5.5326e-06 2.2994e-05 4.544e-06 3.53896e-06 30
end_path:michelremoving:michelremoving 0.0138763 0.0672324 0.228737 0.031586 0.0656095 30
end_path:michelremovinggnocchi:michelremoving 0.0148904 0.0619419 0.228511 0.0283565 0.0640308 30
end_path:calibanagnocchi:CalibAnaTree 0.136926 0.391248 0.701192 0.374448 0.153106 30
end_path:out1:RootOutput(write) 0.42186 0.716533 1.12787 0.716563 0.175205 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4938.27 MB
Peak resident set size usage (VmHWM): 2342.88 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0777_dataflow2_datawriter_0_20240905T051105_reco_stage1_reco_stage2_20240926T115620_keepup.root
\tHists: np04hd_raw_run029081_0777_dataflow2_datawriter_0_20240905T051105_reco_stage1_reco_stage2_20240926T115620_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
