Jobsub ID 78473.144@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78473.144@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3448 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:02:36 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:16:46 |
| From worker node | Hostname | wn033.hep |
| cpuinfo | Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:31:36 |
| Input files | hd-protodune:np04hd_raw_run029097_0070_dataflow0_datawriter_0_20240906T114652.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:16:46 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
in TPC 0
2129 hits in TPC 1
3205 hits in TPC 2
2493 hits in TPC 3
1667 hits in TPC 4
3328 hits in TPC 5
4361 hits in TPC 6
3072 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1172 XUs and 1398 XVs -> 39 XUVs
C:0 T:2 6839 XUs and 25723 XVs -> 453 XUVs
C:0 T:3 18 XUs and 19 XVs -> 1 XUVs
C:0 T:4 471 XUs and 127 XVs -> 21 XUVs
C:0 T:5 13230 XUs and 23191 XVs -> 2220 XUVs
C:0 T:6 13879 XUs and 8666 XVs -> 175 XUVs
C:0 T:7 44 XUs and 40 XVs -> 3 XUVs
2912 XUVs total
690 collection wire objects
2912 potential space points
Neighbour search...
246060 tests to find 149884 neighbours
Iterating with no regularization...
Begin: 4.19607e+06
0 3.56794e+06
1 3.50448e+06
2 3.49212e+06
3 3.48769e+06
4 3.48559e+06
Now with regularization...
Begin: 3.2467e+06
0 3.24519e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29097 subRun: 1 event: 17032 at 26-Sep-2024 12:43:27 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70773 traces, 70773 summary
Retagger: tagging trace set: gauss with 53363 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53363 traces tagged "gauss"
FrameSaver: q=1.16599e+06 n=1029340 tag=gauss
wclsFrameSaver: saving 70773 traces tagged "wiener"
FrameSaver: q=1.24631e+06 n=935415 tag=wiener
5501 hits in TPC 0
2620 hits in TPC 1
4818 hits in TPC 2
3184 hits in TPC 3
3097 hits in TPC 4
3981 hits in TPC 5
8447 hits in TPC 6
5159 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 2123 XUs and 2612 XVs -> 183 XUVs
C:0 T:2 12461 XUs and 13084 XVs -> 564 XUVs
C:0 T:3 177 XUs and 124 XVs -> 1 XUVs
C:0 T:4 101 XUs and 128 XVs -> 7 XUVs
C:0 T:5 9764 XUs and 7954 XVs -> 416 XUVs
C:0 T:6 22901 XUs and 36203 XVs -> 1255 XUVs
C:0 T:7 488 XUs and 731 XVs -> 25 XUVs
2451 XUVs total
1203 collection wire objects
2451 potential space points
Neighbour search...
46463 tests to find 28224 neighbours
Iterating with no regularization...
Begin: 6.26476e+06
0 5.8789e+06
1 5.85374e+06
2 5.85108e+06
Now with regularization...
Begin: 5.30769e+06
0 5.30698e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 7 event 17032 TrkackID 5 startx, y, z 81.4909 591.888 348.883 wireno, peakt tpcno 466 3549.7 5 dist 0min T, max_T3547.03 5009.65
no of hits closeby 7 event 17032 TrkackID 5 startx, y, z 81.4909 591.888 348.883 wireno, peakt tpcno 466 3549.7 5 dist 0min T, max_T3547.03 5009.65
Begin processing the 30th record. run: 29097 subRun: 1 event: 17040 at 26-Sep-2024 12:45:00 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 82215 traces, 82215 summary
Retagger: tagging trace set: gauss with 61406 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 61406 traces tagged "gauss"
FrameSaver: q=1.36132e+06 n=1144026 tag=gauss
wclsFrameSaver: saving 82215 traces tagged "wiener"
FrameSaver: q=1.49174e+06 n=1045813 tag=wiener
3020 hits in TPC 0
2430 hits in TPC 1
9661 hits in TPC 2
5493 hits in TPC 3
3089 hits in TPC 4
6279 hits in TPC 5
9526 hits in TPC 6
4714 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 22 XUs and 15 XVs -> 2 XUVs
C:0 T:2 77820 XUs and 91093 XVs -> 4237 XUVs
C:0 T:3 199 XUs and 325 XVs -> 13 XUVs
C:0 T:4 72 XUs and 116 XVs -> 2 XUVs
C:0 T:5 47421 XUs and 65558 XVs -> 6903 XUVs
C:0 T:6 63345 XUs and 63377 XVs -> 2795 XUVs
C:0 T:7 523 XUs and 626 XVs -> 4 XUVs
13956 XUVs total
3037 collection wire objects
13956 potential space points
Neighbour search...
2032136 tests to find 973634 neighbours
Iterating with no regularization...
Begin: 1.54431e+07
0 1.33441e+07
1 1.31301e+07
2 1.30908e+07
3 1.30795e+07
Now with regularization...
Begin: 1.21552e+07
0 1.21488e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:46:50 BST Closed output file "np04hd_raw_run029097_0070_dataflow0_datawriter_0_20240906T114652_reco_stage1_reco_stage2_20240926T114650_keepup.root"
26-Sep-2024 12:46:50 BST Closed input file "np04hd_raw_run029097_0070_dataflow0_datawriter_0_20240906T114652_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 63.1967 92.1217 126.79 92.4438 15.4675 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000402191 0.00144714 0.00407437 0.00144036 0.000873271 30
produce:fembfilter:PDHDFEMBFilter 0.599541 0.635852 1.07199 0.613026 0.0839508 30
produce:wclsdatahd:WireCellToolkit 44.878 47.5097 50.6601 47.6464 1.49321 30
produce:gaushit:GausHitFinder 0.356168 0.664659 0.978608 0.675351 0.151405 30
produce:nhitsfilter:NumberOfHitsFilter 0.000280234 0.0004152 0.000598299 0.000417807 7.63928e-05 30
produce:reco3d:SpacePointSolver 0.163325 1.09602 3.17026 0.814221 0.754245 30
produce:hitpdune:DisambigFromSpacePoints 0.178632 0.508108 1.01322 0.438114 0.227362 30
produce:pandora:StandardPandora 4.34092 11.4951 20.345 11.0017 4.08901 30
produce:pandoraWriter:StandardPandora 0.0795681 0.161593 0.284263 0.158884 0.0428372 30
produce:pandoraTrack:LArPandoraTrackCreation 0.778237 2.55071 5.44711 2.28683 1.24441 30
produce:pandoraShower:LArPandoraShowerCreation 0.585559 3.22182 8.5641 3.05329 1.81753 30
produce:pandoracalo:Calorimetry 0.342229 0.982531 1.91703 0.938161 0.418558 30
produce:pandoracalonosce:Calorimetry 0.305892 0.93558 2.07682 0.871276 0.434591 30
produce:pandorapid:Chi2ParticleID 0.001112 0.00175412 0.00543862 0.00166228 0.000746004 30
produce:pandoraShowercalo:ShowerCalorimetry 0.486204 2.58693 6.17744 2.51142 1.40682 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.463661 2.46635 5.52459 2.15246 1.36348 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0630394 0.118043 0.170674 0.114172 0.0284947 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.025915 0.0476217 0.0672123 0.0496212 0.0113735 30
produce:emtrkmichelid:EmTrackMichelId 8.05734 15.3843 22.0423 15.6012 3.53751 30
[art]:TriggerResults:TriggerResultInserter 2.8871e-05 4.74024e-05 7.7281e-05 4.72195e-05 1.02591e-05 30
end_path:out1:RootOutput 8.175e-06 1.28193e-05 2.3184e-05 1.2322e-05 3.219e-06 30
end_path:michelremoving:michelremoving 0.0224873 0.0794129 0.252391 0.0540864 0.0607756 30
end_path:michelremovinggnocchi:michelremoving 0.0211354 0.0747982 0.249694 0.0476535 0.0598898 30
end_path:calibanagnocchi:CalibAnaTree 0.221925 0.681772 1.38676 0.614361 0.341843 30
end_path:out1:RootOutput(write) 0.388229 0.678007 1.00456 0.683664 0.15603 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5793.69 MB
Peak resident set size usage (VmHWM): 2608.01 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029097_0070_dataflow0_datawriter_0_20240906T114652_reco_stage1_reco_stage2_20240926T114650_keepup.root
\tHists: np04hd_raw_run029097_0070_dataflow0_datawriter_0_20240906T114652_reco_stage1_reco_stage2_20240926T114650_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
