Jobsub ID 78462.113@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78462.113@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3454 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:58:37 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:16:25 |
| From worker node | Hostname | wn095.hep |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:16:10 |
| Input files | hd-protodune:np04hd_raw_run029108_0263_dataflow2_datawriter_0_20240909T142849.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:16:25 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
TPC 6
4737 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 5601 XUs and 6748 XVs -> 2141 XUVs
C:0 T:2 9909 XUs and 14602 XVs -> 540 XUVs
C:0 T:3 92 XUs and 171 XVs -> 5 XUVs
C:0 T:4 43 XUs and 74 XVs -> 2 XUVs
C:0 T:5 3970 XUs and 6309 XVs -> 752 XUVs
C:0 T:6 22533 XUs and 32624 XVs -> 1617 XUVs
C:0 T:7 61 XUs and 54 XVs -> 5 XUVs
5062 XUVs total
1305 collection wire objects
5062 potential space points
Neighbour search...
293832 tests to find 172940 neighbours
Iterating with no regularization...
Begin: 7.17854e+06
0 6.67456e+06
1 6.63371e+06
2 6.62737e+06
Now with regularization...
Begin: 6.10921e+06
0 6.10741e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29108 subRun: 1 event: 63370 at 26-Sep-2024 12:36:09 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 52015 traces, 52015 summary
Retagger: tagging trace set: gauss with 40498 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 40498 traces tagged "gauss"
FrameSaver: q=853985 n=769712 tag=gauss
wclsFrameSaver: saving 52015 traces tagged "wiener"
FrameSaver: q=905324 n=703419 tag=wiener
4515 hits in TPC 0
2801 hits in TPC 1
3320 hits in TPC 2
1819 hits in TPC 3
3539 hits in TPC 4
5974 hits in TPC 5
3766 hits in TPC 6
2132 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1324 XUs and 357 XVs -> 9 XUVs
C:0 T:2 4219 XUs and 6274 XVs -> 207 XUVs
C:0 T:3 81 XUs and 128 XVs -> 11 XUVs
C:0 T:4 61 XUs and 46 XVs -> 3 XUVs
C:0 T:5 18944 XUs and 13361 XVs -> 378 XUVs
C:0 T:6 13498 XUs and 8409 XVs -> 246 XUVs
C:0 T:7 112 XUs and 51 XVs -> 3 XUVs
857 XUVs total
645 collection wire objects
857 potential space points
Neighbour search...
6603 tests to find 3786 neighbours
Iterating with no regularization...
Begin: 4.032e+06
0 3.74903e+06
1 3.7333e+06
2 3.73182e+06
Now with regularization...
Begin: 3.54334e+06
0 3.54321e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29108 subRun: 1 event: 63402 at 26-Sep-2024 12:37:36 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 95962 traces, 95962 summary
Retagger: tagging trace set: gauss with 69910 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69910 traces tagged "gauss"
FrameSaver: q=1.44121e+06 n=1209705 tag=gauss
wclsFrameSaver: saving 95962 traces tagged "wiener"
FrameSaver: q=1.57756e+06 n=1084846 tag=wiener
4555 hits in TPC 0
4104 hits in TPC 1
9643 hits in TPC 2
6036 hits in TPC 3
3604 hits in TPC 4
5730 hits in TPC 5
8834 hits in TPC 6
5918 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 3004 XUs and 2749 XVs -> 78 XUVs
C:0 T:2 95822 XUs and 90927 XVs -> 5252 XUVs
C:0 T:3 157 XUs and 220 XVs -> 6 XUVs
C:0 T:4 167 XUs and 139 XVs -> 10 XUVs
C:0 T:5 34091 XUs and 26570 XVs -> 427 XUVs
C:0 T:6 58508 XUs and 83705 XVs -> 2037 XUVs
C:0 T:7 235 XUs and 315 XVs -> 11 XUVs
7821 XUVs total
2603 collection wire objects
7821 potential space points
Neighbour search...
783623 tests to find 387830 neighbours
Iterating with no regularization...
Begin: 1.32241e+07
0 1.1598e+07
1 1.14663e+07
2 1.14545e+07
3 1.1451e+07
Now with regularization...
Begin: 1.09213e+07
0 1.0919e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 63402 TrkackID 8 startx, y, z 58.7468 370.335 452.531 wireno, peakt tpcno 249 5854.87 6 dist 400.318min T, max_T4242.7 5856.25
no of hits closeby 0 event 63402 TrkackID 8 startx, y, z 58.7468 370.335 452.531 wireno, peakt tpcno 249 5854.87 6 dist 400.318min T, max_T4242.7 5856.25
26-Sep-2024 12:39:54 BST Closed output file "np04hd_raw_run029108_0263_dataflow2_datawriter_0_20240909T142849_reco_stage1_reco_stage2_20240926T113954_keepup.root"
26-Sep-2024 12:39:54 BST Closed input file "np04hd_raw_run029108_0263_dataflow2_datawriter_0_20240909T142849_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 76.8454 107.126 168.055 101.281 22.811 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000692159 0.00298109 0.00764081 0.00319103 0.00135834 30
produce:fembfilter:PDHDFEMBFilter 0.688072 0.757956 0.933262 0.744968 0.0497159 30
produce:wclsdatahd:WireCellToolkit 43.8923 47.566 58.6452 47.1978 2.76647 30
produce:gaushit:GausHitFinder 0.39546 0.802553 1.33318 0.786394 0.212219 30
produce:nhitsfilter:NumberOfHitsFilter 0.000230936 0.000463374 0.00112707 0.000424441 0.000186068 30
produce:reco3d:SpacePointSolver 0.110334 1.23695 5.10405 0.847982 1.11738 30
produce:hitpdune:DisambigFromSpacePoints 0.103235 0.64297 1.93089 0.566555 0.396131 30
produce:pandora:StandardPandora 6.53411 14.5554 33.9402 12.6816 6.58208 30
produce:pandoraWriter:StandardPandora 0.107838 0.177831 0.28149 0.171059 0.0431 30
produce:pandoraTrack:LArPandoraTrackCreation 0.666353 2.15733 5.21868 2.04518 1.05391 30
produce:pandoraShower:LArPandoraShowerCreation 0.758472 2.65703 8.84551 2.28661 1.76915 30
produce:pandoracalo:Calorimetry 0.373828 0.978658 2.43633 0.894226 0.441488 30
produce:pandoracalonosce:Calorimetry 0.301811 0.904757 2.22746 0.839236 0.417039 30
produce:pandorapid:Chi2ParticleID 0.00148648 0.00216168 0.00411685 0.00201072 0.000562327 30
produce:pandoraShowercalo:ShowerCalorimetry 0.634556 2.25732 7.84851 2.04376 1.50165 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.604016 2.15372 7.29512 1.86464 1.39266 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0897837 0.146227 0.247619 0.137999 0.0409703 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0357617 0.0543493 0.0954822 0.051264 0.0146405 30
produce:emtrkmichelid:EmTrackMichelId 17.4617 28.1618 47.0919 27.2396 7.00434 30
[art]:TriggerResults:TriggerResultInserter 2.4781e-05 4.94366e-05 0.000111941 4.9929e-05 1.773e-05 30
end_path:out1:RootOutput 8.343e-06 1.27869e-05 3.571e-05 1.0968e-05 6.10582e-06 30
end_path:michelremoving:michelremoving 0.0243392 0.114041 0.66552 0.069899 0.127176 30
end_path:michelremovinggnocchi:michelremoving 0.0231453 0.106366 0.653697 0.0638898 0.124043 30
end_path:calibanagnocchi:CalibAnaTree 0.246598 0.716342 1.71434 0.647377 0.329216 30
end_path:out1:RootOutput(write) 0.466575 0.827491 1.37094 0.770294 0.22233 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4928.61 MB
Peak resident set size usage (VmHWM): 2330.64 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029108_0263_dataflow2_datawriter_0_20240909T142849_reco_stage1_reco_stage2_20240926T113954_keepup.root
\tHists: np04hd_raw_run029108_0263_dataflow2_datawriter_0_20240909T142849_reco_stage1_reco_stage2_20240926T113954_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
