Jobsub ID 78491.125@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78491.125@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:11:15 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:12:50 |
| From worker node | Hostname | wn033.hep |
| cpuinfo | Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:43:57 |
| Input files | hd-protodune:np04hd_raw_run029081_0590_dataflow7_datawriter_0_20240905T020052.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:12:50 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
d 510 XVs -> 20 XUVs
C:0 T:4 470 XUs and 461 XVs -> 15 XUVs
C:0 T:5 32973 XUs and 28712 XVs -> 813 XUVs
C:0 T:6 9072 XUs and 7410 XVs -> 486 XUVs
C:0 T:7 58 XUs and 52 XVs -> 0 XUVs
1858 XUVs total
857 collection wire objects
1858 potential space points
Neighbour search...
34242 tests to find 22808 neighbours
Iterating with no regularization...
Begin: 9.36723e+06
0 9.09448e+06
1 9.07599e+06
2 9.07381e+06
Now with regularization...
Begin: 8.62324e+06
0 8.62255e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 141831 at 26-Sep-2024 12:55:08 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 76954 traces, 76954 summary
Retagger: tagging trace set: gauss with 53800 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53800 traces tagged "gauss"
FrameSaver: q=1.42612e+06 n=949391 tag=gauss
wclsFrameSaver: saving 76954 traces tagged "wiener"
FrameSaver: q=1.54864e+06 n=845102 tag=wiener
1997 hits in TPC 0
2028 hits in TPC 1
7229 hits in TPC 2
4653 hits in TPC 3
5727 hits in TPC 4
7978 hits in TPC 5
6402 hits in TPC 6
3618 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 108 XUs and 57 XVs -> 2 XUVs
C:0 T:2 60668 XUs and 41626 XVs -> 2440 XUVs
C:0 T:3 205 XUs and 271 XVs -> 3 XUVs
C:0 T:4 111 XUs and 116 XVs -> 4 XUVs
C:0 T:5 18036 XUs and 25037 XVs -> 2628 XUVs
C:0 T:6 77320 XUs and 51651 XVs -> 2058 XUVs
C:0 T:7 10 XUs and 12 XVs -> 0 XUVs
7135 XUVs total
1753 collection wire objects
7135 potential space points
Neighbour search...
356465 tests to find 207370 neighbours
Iterating with no regularization...
Begin: 8.46163e+07
0 8.11611e+07
1 8.08094e+07
2 8.07529e+07
Now with regularization...
Begin: 7.94039e+07
0 7.93397e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 141855 at 26-Sep-2024 12:56:46 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 75100 traces, 75100 summary
Retagger: tagging trace set: gauss with 57447 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 57447 traces tagged "gauss"
FrameSaver: q=1.26162e+06 n=1137290 tag=gauss
wclsFrameSaver: saving 75100 traces tagged "wiener"
FrameSaver: q=1.33277e+06 n=1031951 tag=wiener
5328 hits in TPC 0
4655 hits in TPC 1
3775 hits in TPC 2
2550 hits in TPC 3
5133 hits in TPC 4
8875 hits in TPC 5
6018 hits in TPC 6
4314 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 118 XUs and 190 XVs -> 7 XUVs
C:0 T:2 8985 XUs and 11376 XVs -> 163 XUVs
C:0 T:3 420 XUs and 500 XVs -> 23 XUVs
C:0 T:4 723 XUs and 713 XVs -> 7 XUVs
C:0 T:5 35190 XUs and 45996 XVs -> 2431 XUVs
C:0 T:6 18064 XUs and 21755 XVs -> 782 XUVs
C:0 T:7 579 XUs and 588 XVs -> 27 XUVs
3440 XUVs total
1494 collection wire objects
3440 potential space points
Neighbour search...
188518 tests to find 112466 neighbours
Iterating with no regularization...
Begin: 1.88048e+07
0 1.81717e+07
1 1.81297e+07
2 1.81247e+07
Now with regularization...
Begin: 1.75183e+07
0 1.75165e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 141855 TrkackID 8 startx, y, z -121.724 567.313 242.796 wireno, peakt tpcno 104 5 6 dist 0min T, max_T5 1577.76
no of hits closeby 0 event 141855 TrkackID 8 startx, y, z -121.724 567.313 242.796 wireno, peakt tpcno 104 5 6 dist 0min T, max_T5 1577.76
26-Sep-2024 12:58:22 BST Closed output file "np04hd_raw_run029081_0590_dataflow7_datawriter_0_20240905T020052_reco_stage1_reco_stage2_20240926T115822_keepup.root"
26-Sep-2024 12:58:22 BST Closed input file "np04hd_raw_run029081_0590_dataflow7_datawriter_0_20240905T020052_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 66.3223 88.947 151.032 87.6951 16.7088 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000405995 0.00168973 0.00599149 0.00120097 0.00139483 30
produce:fembfilter:PDHDFEMBFilter 0.651557 1.05366 2.77174 0.818267 0.494656 30
produce:wclsdatahd:WireCellToolkit 45.3394 48.4822 51.437 48.7433 1.46583 30
produce:gaushit:GausHitFinder 0.414725 0.636648 1.03074 0.613643 0.151075 30
produce:nhitsfilter:NumberOfHitsFilter 0.00028962 0.00041503 0.000709914 0.000411526 9.37394e-05 30
produce:reco3d:SpacePointSolver 0.229028 1.01605 2.7165 0.765668 0.684164 30
produce:hitpdune:DisambigFromSpacePoints 0.17908 0.483033 1.14564 0.430688 0.2292 30
produce:pandora:StandardPandora 4.2969 11.1614 33.437 10.7987 5.40524 30
produce:pandoraWriter:StandardPandora 0.0870468 0.153742 0.279502 0.149285 0.0396895 30
produce:pandoraTrack:LArPandoraTrackCreation 0.8103 2.82528 12.5448 2.2797 2.06065 30
produce:pandoraShower:LArPandoraShowerCreation 0.593766 3.16379 10.3026 2.55942 2.13832 30
produce:pandoracalo:Calorimetry 0.337557 1.02126 3.91742 0.812821 0.664526 30
produce:pandoracalonosce:Calorimetry 0.320528 0.961809 3.78112 0.74074 0.643805 30
produce:pandorapid:Chi2ParticleID 0.0011516 0.00173699 0.00356703 0.00154684 0.000514546 30
produce:pandoraShowercalo:ShowerCalorimetry 0.375212 2.31338 6.24665 1.92577 1.41554 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.320382 2.18787 4.83991 1.92479 1.24314 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0639621 0.112038 0.163825 0.109546 0.0264473 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0264552 0.0463289 0.068537 0.0444702 0.0108684 30
produce:emtrkmichelid:EmTrackMichelId 6.77297 11.5913 17.5413 11.3555 2.67896 30
[art]:TriggerResults:TriggerResultInserter 3.7245e-05 5.011e-05 8.001e-05 4.8009e-05 1.04413e-05 30
end_path:out1:RootOutput 9.906e-06 1.61104e-05 3.6315e-05 1.4315e-05 6.56803e-06 30
end_path:michelremoving:michelremoving 0.0211466 0.0768879 0.175125 0.0590359 0.0477126 30
end_path:michelremovinggnocchi:michelremoving 0.0206632 0.074871 0.18926 0.0509917 0.0491473 30
end_path:calibanagnocchi:CalibAnaTree 0.152702 0.679611 2.71015 0.533732 0.455328 30
end_path:out1:RootOutput(write) 0.381363 0.650853 1.0888 0.628783 0.160071 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5858.75 MB
Peak resident set size usage (VmHWM): 2697.94 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0590_dataflow7_datawriter_0_20240905T020052_reco_stage1_reco_stage2_20240926T115822_keepup.root
\tHists: np04hd_raw_run029081_0590_dataflow7_datawriter_0_20240905T020052_reco_stage1_reco_stage2_20240926T115822_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
