Jobsub ID 263324.163@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263324.163@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:07:35 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:25:31 |
| From worker node | Hostname | wn094.hep |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:39:09 |
| Input files | hd-protodune:np04hd_raw_run029092_0352_dataflow0_datawriter_0_20240905T224711.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:25:31 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
4
5090 hits in TPC 5
6191 hits in TPC 6
2990 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1248 XUs and 514 XVs -> 2 XUVs
C:0 T:2 36454 XUs and 41949 XVs -> 786 XUVs
C:0 T:3 247 XUs and 320 XVs -> 10 XUVs
C:0 T:4 30 XUs and 34 XVs -> 2 XUVs
C:0 T:5 18618 XUs and 12401 XVs -> 286 XUVs
C:0 T:6 30270 XUs and 39467 XVs -> 2102 XUVs
C:0 T:7 40 XUs and 31 XVs -> 2 XUVs
3190 XUVs total
1204 collection wire objects
3190 potential space points
Neighbour search...
270226 tests to find 128064 neighbours
Iterating with no regularization...
Begin: 5.83966e+06
0 5.42346e+06
1 5.4085e+06
2 5.40625e+06
Now with regularization...
Begin: 5.02895e+06
0 5.02767e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29092 subRun: 1 event: 84712 at 26-Sep-2024 13:06:22 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68255 traces, 68255 summary
Retagger: tagging trace set: gauss with 50733 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 50733 traces tagged "gauss"
FrameSaver: q=1.36308e+06 n=970432 tag=gauss
wclsFrameSaver: saving 68255 traces tagged "wiener"
FrameSaver: q=1.43712e+06 n=876309 tag=wiener
2781 hits in TPC 0
2276 hits in TPC 1
6910 hits in TPC 2
5767 hits in TPC 3
5224 hits in TPC 4
7435 hits in TPC 5
3541 hits in TPC 6
2345 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 66 XUs and 29 XVs -> 1 XUVs
C:0 T:2 22804 XUs and 24853 XVs -> 1690 XUVs
C:0 T:3 108 XUs and 103 XVs -> 3 XUVs
C:0 T:4 970 XUs and 657 XVs -> 10 XUVs
C:0 T:5 25302 XUs and 29339 XVs -> 1077 XUVs
C:0 T:6 5110 XUs and 8190 XVs -> 186 XUVs
C:0 T:7 145 XUs and 211 XVs -> 10 XUVs
2977 XUVs total
1289 collection wire objects
2977 potential space points
Neighbour search...
54825 tests to find 32658 neighbours
Iterating with no regularization...
Begin: 1.83788e+07
0 1.68006e+07
1 1.67077e+07
2 1.66983e+07
Now with regularization...
Begin: 1.54569e+07
0 1.54452e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29092 subRun: 1 event: 84720 at 26-Sep-2024 13:08:27 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 58706 traces, 58706 summary
Retagger: tagging trace set: gauss with 43859 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 43859 traces tagged "gauss"
FrameSaver: q=926457 n=849084 tag=gauss
wclsFrameSaver: saving 58706 traces tagged "wiener"
FrameSaver: q=993093 n=772413 tag=wiener
3505 hits in TPC 0
2680 hits in TPC 1
4750 hits in TPC 2
3893 hits in TPC 3
3696 hits in TPC 4
6600 hits in TPC 5
5676 hits in TPC 6
3257 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 4 XUs and 24 XVs -> 0 XUVs
C:0 T:1 120 XUs and 187 XVs -> 0 XUVs
C:0 T:2 26994 XUs and 23965 XVs -> 919 XUVs
C:0 T:3 339 XUs and 263 XVs -> 7 XUVs
C:0 T:4 205 XUs and 195 XVs -> 9 XUVs
C:0 T:5 19122 XUs and 35503 XVs -> 1959 XUVs
C:0 T:6 32122 XUs and 39207 XVs -> 1358 XUVs
C:0 T:7 376 XUs and 258 XVs -> 23 XUVs
4275 XUVs total
1230 collection wire objects
4275 potential space points
Neighbour search...
465569 tests to find 212136 neighbours
Iterating with no regularization...
Begin: 6.78784e+06
0 6.11748e+06
1 6.03698e+06
2 6.02812e+06
3 6.02516e+06
Now with regularization...
Begin: 5.81143e+06
0 5.80941e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:10:49 BST Closed output file "np04hd_raw_run029092_0352_dataflow0_datawriter_0_20240905T224711_reco_stage1_reco_stage2_20240926T121049_keepup.root"
26-Sep-2024 13:10:49 BST Closed input file "np04hd_raw_run029092_0352_dataflow0_datawriter_0_20240905T224711_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 76.1214 122.474 189.917 118.108 24.1748 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000678922 0.00301208 0.00469443 0.00331159 0.00106872 30
produce:fembfilter:PDHDFEMBFilter 0.740436 0.836107 1.17988 0.815367 0.104283 30
produce:wclsdatahd:WireCellToolkit 49.4407 57.6341 74.9449 56.4787 4.34871 30
produce:gaushit:GausHitFinder 0.444922 0.808409 1.12757 0.808565 0.170458 30
produce:nhitsfilter:NumberOfHitsFilter 0.000196204 0.000410878 0.000679849 0.000417146 0.000102229 30
produce:reco3d:SpacePointSolver 0.0705105 1.12311 3.71213 0.944963 0.834491 30
produce:hitpdune:DisambigFromSpacePoints 0.108301 0.65942 1.44187 0.606179 0.299418 30
produce:pandora:StandardPandora 4.09415 15.7713 29.5677 14.0477 6.9499 30
produce:pandoraWriter:StandardPandora 0.0724139 0.18041 0.309081 0.176661 0.050646 30
produce:pandoraTrack:LArPandoraTrackCreation 0.455788 1.85241 3.49573 1.78696 0.741718 30
produce:pandoraShower:LArPandoraShowerCreation 0.36652 2.44705 6.90931 2.10864 1.42957 30
produce:pandoracalo:Calorimetry 0.275041 0.880714 1.71803 0.843874 0.34677 30
produce:pandoracalonosce:Calorimetry 0.213333 0.817857 1.71965 0.766273 0.333601 30
produce:pandorapid:Chi2ParticleID 0.00139478 0.00207114 0.00396344 0.00195105 0.000483877 30
produce:pandoraShowercalo:ShowerCalorimetry 0.313914 2.14117 5.65173 2.03619 1.20037 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.311715 2.11139 7.46497 1.9256 1.37974 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0595455 0.165296 0.392093 0.161734 0.0608937 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0255045 0.0656678 0.145051 0.0649344 0.0210885 30
produce:emtrkmichelid:EmTrackMichelId 13.6855 33.1995 61.9439 32.2229 9.99597 30
[art]:TriggerResults:TriggerResultInserter 2.2808e-05 4.02919e-05 8.1128e-05 3.56205e-05 1.32116e-05 30
end_path:out1:RootOutput 9.277e-06 1.18856e-05 2.5806e-05 1.1403e-05 3.37536e-06 30
end_path:michelremoving:michelremoving 0.0228971 0.0765273 0.248762 0.0478407 0.0595601 30
end_path:michelremovinggnocchi:michelremoving 0.0216873 0.072882 0.222715 0.0437235 0.0584593 30
end_path:calibanagnocchi:CalibAnaTree 0.203052 0.646059 1.30864 0.627509 0.258869 30
end_path:out1:RootOutput(write) 0.36924 0.850841 1.41809 0.817972 0.2309 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4932.36 MB
Peak resident set size usage (VmHWM): 2350.25 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029092_0352_dataflow0_datawriter_0_20240905T224711_reco_stage1_reco_stage2_20240926T121049_keepup.root
\tHists: np04hd_raw_run029092_0352_dataflow0_datawriter_0_20240905T224711_reco_stage1_reco_stage2_20240926T121049_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
