Jobsub ID 78519.37@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78519.37@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3451 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:17:47 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:22:39 |
| From worker node | Hostname | wn034.hep |
| cpuinfo | AMD EPYC 75F3 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:53:32 |
| Input files | hd-protodune:np04hd_raw_run029107_2220_dataflow6_datawriter_0_20240908T035331.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:22:39 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
-------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29107 subRun: 1 event: 533054 at 26-Sep-2024 12:45:27 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 93923 traces, 93923 summary
Retagger: tagging trace set: gauss with 69118 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69118 traces tagged "gauss"
FrameSaver: q=1.48613e+06 n=1246603 tag=gauss
wclsFrameSaver: saving 93923 traces tagged "wiener"
FrameSaver: q=1.59682e+06 n=1118594 tag=wiener
4067 hits in TPC 0
4473 hits in TPC 1
8310 hits in TPC 2
4789 hits in TPC 3
5286 hits in TPC 4
9520 hits in TPC 5
7486 hits in TPC 6
4525 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 900 XUs and 558 XVs -> 20 XUVs
C:0 T:2 26335 XUs and 36984 XVs -> 1247 XUVs
C:0 T:3 534 XUs and 307 XVs -> 4 XUVs
C:0 T:4 245 XUs and 394 XVs -> 12 XUVs
C:0 T:5 52342 XUs and 69137 XVs -> 1739 XUVs
C:0 T:6 29156 XUs and 25335 XVs -> 931 XUVs
C:0 T:7 184 XUs and 197 XVs -> 5 XUVs
3958 XUVs total
2255 collection wire objects
3958 potential space points
Neighbour search...
88468 tests to find 51866 neighbours
Iterating with no regularization...
Begin: 1.12205e+07
0 1.05151e+07
1 1.04784e+07
2 1.04742e+07
Now with regularization...
Begin: 9.89175e+06
0 9.88997e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 10 event 533054 TrkackID 16 startx, y, z 12.6633 589.48 279.927 wireno, peakt tpcno 85 3720.44 5 dist 2.11208min T, max_T3687.62 4430.44
no of hits closeby 0 event 533054 TrkackID 28 startx, y, z 3.83332 586.202 8.11854 wireno, peakt tpcno 116 3678.44 2 dist 75.6165min T, max_T3588.21 3704.83
no of hits closeby 10 event 533054 TrkackID 16 startx, y, z 12.6633 589.48 279.927 wireno, peakt tpcno 85 3720.44 5 dist 2.11208min T, max_T3687.62 4430.44
no of hits closeby 0 event 533054 TrkackID 28 startx, y, z 3.83332 586.202 8.11854 wireno, peakt tpcno 116 3678.44 2 dist 75.6165min T, max_T3588.21 3704.83
Begin processing the 30th record. run: 29107 subRun: 1 event: 533062 at 26-Sep-2024 12:46:39 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 53137 traces, 53137 summary
Retagger: tagging trace set: gauss with 40671 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 40671 traces tagged "gauss"
FrameSaver: q=875870 n=772458 tag=gauss
wclsFrameSaver: saving 53137 traces tagged "wiener"
FrameSaver: q=920424 n=698261 tag=wiener
2481 hits in TPC 0
2973 hits in TPC 1
3966 hits in TPC 2
1675 hits in TPC 3
3519 hits in TPC 4
4959 hits in TPC 5
4885 hits in TPC 6
2792 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 211 XUs and 198 XVs -> 11 XUVs
C:0 T:2 5919 XUs and 9535 XVs -> 348 XUVs
C:0 T:3 6 XUs and 9 XVs -> 0 XUVs
C:0 T:4 103 XUs and 66 XVs -> 2 XUVs
C:0 T:5 15249 XUs and 13514 XVs -> 337 XUVs
C:0 T:6 9653 XUs and 13121 XVs -> 250 XUVs
C:0 T:7 1192 XUs and 2108 XVs -> 47 XUVs
995 XUVs total
785 collection wire objects
995 potential space points
Neighbour search...
7583 tests to find 3870 neighbours
Iterating with no regularization...
Begin: 6.92931e+06
0 6.84591e+06
1 6.84432e+06
Now with regularization...
Begin: 6.42527e+06
0 6.42518e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 1 event 533062 TrkackID 0 startx, y, z 251.689 606.657 1.23642 wireno, peakt tpcno 457 1560.55 5 dist 0min T, max_T256.001 3473.42
no of hits closeby 1 event 533062 TrkackID 2 startx, y, z -178.388 546.14 409.766 wireno, peakt tpcno 117 307.149 6 dist 0min T, max_T5.64268 2275.28
no of hits closeby 1 event 533062 TrkackID 0 startx, y, z 251.689 606.657 1.23642 wireno, peakt tpcno 457 1560.55 5 dist 0min T, max_T256.001 3473.42
no of hits closeby 1 event 533062 TrkackID 2 startx, y, z -178.388 546.14 409.766 wireno, peakt tpcno 117 307.149 6 dist 0min T, max_T5.64268 2275.28
26-Sep-2024 12:47:28 BST Closed output file "np04hd_raw_run029107_2220_dataflow6_datawriter_0_20240908T035331_reco_stage1_reco_stage2_20240926T114728_keepup.root"
26-Sep-2024 12:47:28 BST Closed input file "np04hd_raw_run029107_2220_dataflow6_datawriter_0_20240908T035331_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 45.5578 59.2789 79.7491 57.9484 8.70734 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000211611 0.00102041 0.00155796 0.00106459 0.000330242 30
produce:fembfilter:PDHDFEMBFilter 0.367256 0.378895 0.461622 0.37405 0.0175357 30
produce:wclsdatahd:WireCellToolkit 23.7588 25.5293 28.1702 25.5956 0.792115 30
produce:gaushit:GausHitFinder 0.277422 0.404467 0.565668 0.39916 0.0807962 30
produce:nhitsfilter:NumberOfHitsFilter 0.000162629 0.000234094 0.000413253 0.000215864 6.19959e-05 30
produce:reco3d:SpacePointSolver 0.123615 0.535506 1.30468 0.427447 0.329629 30
produce:hitpdune:DisambigFromSpacePoints 0.129701 0.289355 0.523819 0.271125 0.106943 30
produce:pandora:StandardPandora 2.94833 6.8738 12.9397 6.67573 2.40358 30
produce:pandoraWriter:StandardPandora 0.0485201 0.0881296 0.179872 0.086345 0.0270908 30
produce:pandoraTrack:LArPandoraTrackCreation 0.345313 0.661824 1.05586 0.622106 0.189673 30
produce:pandoraShower:LArPandoraShowerCreation 0.368522 0.877262 1.97965 0.707422 0.383705 30
produce:pandoracalo:Calorimetry 0.208581 0.375982 0.692065 0.354442 0.108952 30
produce:pandoracalonosce:Calorimetry 0.185341 0.342831 0.588638 0.31875 0.0970951 30
produce:pandorapid:Chi2ParticleID 0.000720266 0.00107946 0.00242572 0.00103845 0.000296893 30
produce:pandoraShowercalo:ShowerCalorimetry 0.311081 0.854115 2.04313 0.737653 0.371416 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.305219 0.824947 1.9735 0.70715 0.354538 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0488946 0.0751485 0.106552 0.0720435 0.016039 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0163758 0.0260227 0.0370085 0.0253355 0.00554338 30
produce:emtrkmichelid:EmTrackMichelId 13.7984 20.1857 28.5546 19.5662 4.11122 30
[art]:TriggerResults:TriggerResultInserter 1.0921e-05 1.65103e-05 5.3081e-05 1.3976e-05 8.02901e-06 30
end_path:out1:RootOutput 2.956e-06 3.9559e-06 1.6141e-05 3.3665e-06 2.30898e-06 30
end_path:michelremoving:michelremoving 0.0113693 0.0652499 0.25415 0.0204414 0.068738 30
end_path:michelremovinggnocchi:michelremoving 0.0105425 0.0619927 0.239301 0.0192318 0.0672045 30
end_path:calibanagnocchi:CalibAnaTree 0.146627 0.287735 0.466885 0.279443 0.0812882 30
end_path:out1:RootOutput(write) 0.300334 0.45135 0.654237 0.442405 0.0940772 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4952.49 MB
Peak resident set size usage (VmHWM): 2349.47 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_2220_dataflow6_datawriter_0_20240908T035331_reco_stage1_reco_stage2_20240926T114728_keepup.root
\tHists: np04hd_raw_run029107_2220_dataflow6_datawriter_0_20240908T035331_reco_stage1_reco_stage2_20240926T114728_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
