Jobsub ID 78530.104@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78530.104@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3451 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:19:09 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:20:56 |
| From worker node | Hostname | wn-200-04-07-04-a.cr.cnaf.infn.it |
| cpuinfo | AMD EPYC 7313 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 11:02:46 |
| Input files | hd-protodune:np04hd_raw_run029107_2078_dataflow0_datawriter_0_20240908T012540.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:20:56 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1291 XUs and 1662 XVs -> 59 XUVs
C:0 T:2 35989 XUs and 43701 XVs -> 2102 XUVs
C:0 T:3 44 XUs and 42 XVs -> 2 XUVs
C:0 T:4 258 XUs and 418 XVs -> 11 XUVs
C:0 T:5 11213 XUs and 11808 XVs -> 385 XUVs
C:0 T:6 58908 XUs and 54066 XVs -> 2338 XUVs
C:0 T:7 187 XUs and 356 XVs -> 4 XUVs
4901 XUVs total
1304 collection wire objects
4901 potential space points
Neighbour search...
623505 tests to find 261948 neighbours
Iterating with no regularization...
Begin: 2.13762e+07
0 1.84353e+07
1 1.80097e+07
2 1.79629e+07
3 1.7952e+07
Now with regularization...
Begin: 1.7188e+07
0 1.71801e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29107 subRun: 1 event: 498976 at 26-Sep-2024 13:56:35 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 60318 traces, 60318 summary
Retagger: tagging trace set: gauss with 44213 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 44213 traces tagged "gauss"
FrameSaver: q=955693 n=776759 tag=gauss
wclsFrameSaver: saving 60318 traces tagged "wiener"
FrameSaver: q=1.02574e+06 n=691942 tag=wiener
2053 hits in TPC 0
2129 hits in TPC 1
9542 hits in TPC 2
4894 hits in TPC 3
2709 hits in TPC 4
4132 hits in TPC 5
2523 hits in TPC 6
1207 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 209 XUs and 180 XVs -> 5 XUVs
C:0 T:2 52682 XUs and 61882 XVs -> 2862 XUVs
C:0 T:3 230 XUs and 328 XVs -> 8 XUVs
C:0 T:4 55 XUs and 25 XVs -> 0 XUVs
C:0 T:5 17848 XUs and 6695 XVs -> 222 XUVs
C:0 T:6 18092 XUs and 34219 XVs -> 1686 XUVs
C:0 T:7 12 XUs and 23 XVs -> 0 XUVs
4783 XUVs total
1549 collection wire objects
4783 potential space points
Neighbour search...
292099 tests to find 162190 neighbours
Iterating with no regularization...
Begin: 8.40712e+06
0 7.39789e+06
1 7.33833e+06
2 7.33137e+06
Now with regularization...
Begin: 6.89738e+06
0 6.89439e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29107 subRun: 1 event: 498984 at 26-Sep-2024 13:57:48 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 96838 traces, 96838 summary
Retagger: tagging trace set: gauss with 71152 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 71152 traces tagged "gauss"
FrameSaver: q=1.69188e+06 n=1327505 tag=gauss
wclsFrameSaver: saving 96838 traces tagged "wiener"
FrameSaver: q=1.85607e+06 n=1210481 tag=wiener
1508 hits in TPC 0
2070 hits in TPC 1
12218 hits in TPC 2
6618 hits in TPC 3
5029 hits in TPC 4
7552 hits in TPC 5
10440 hits in TPC 6
7155 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 132 XUs and 101 XVs -> 0 XUVs
C:0 T:2 57066 XUs and 72190 XVs -> 3118 XUVs
C:0 T:3 609 XUs and 578 XVs -> 19 XUVs
C:0 T:4 105 XUs and 90 XVs -> 5 XUVs
C:0 T:5 100042 XUs and 74510 XVs -> 5653 XUVs
C:0 T:6 44038 XUs and 72302 XVs -> 3565 XUVs
C:0 T:7 684 XUs and 909 XVs -> 20 XUVs
12380 XUVs total
3694 collection wire objects
12380 potential space points
Neighbour search...
899286 tests to find 392318 neighbours
Iterating with no regularization...
Begin: 4.00984e+07
0 3.16506e+07
1 3.0861e+07
2 3.07563e+07
3 3.07204e+07
4 3.07033e+07
Now with regularization...
Begin: 2.84454e+07
0 2.84259e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 14:00:05 CEST Closed output file "np04hd_raw_run029107_2078_dataflow0_datawriter_0_20240908T012540_reco_stage1_reco_stage2_20240926T120005_keepup.root"
26-Sep-2024 14:00:05 CEST Closed input file "np04hd_raw_run029107_2078_dataflow0_datawriter_0_20240908T012540_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 47.2141 79.5858 136.01 75.282 17.5393 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000339477 0.00207138 0.00691752 0.00186441 0.0013886 30
produce:fembfilter:PDHDFEMBFilter 0.419104 0.573492 0.801336 0.619748 0.107383 30
produce:wclsdatahd:WireCellToolkit 27.2667 33.555 37.6117 33.8466 2.96315 30
produce:gaushit:GausHitFinder 0.294651 0.628267 1.13859 0.617444 0.20877 30
produce:nhitsfilter:NumberOfHitsFilter 0.000143289 0.000308203 0.000683994 0.000290505 0.000100363 30
produce:reco3d:SpacePointSolver 0.193965 1.24419 6.52751 0.987119 1.16382 30
produce:hitpdune:DisambigFromSpacePoints 0.0879352 0.549529 1.74053 0.454099 0.350645 30
produce:pandora:StandardPandora 3.41645 11.6181 27.9303 9.36549 6.2003 30
produce:pandoraWriter:StandardPandora 0.0563219 0.121123 0.235866 0.12001 0.0432286 30
produce:pandoraTrack:LArPandoraTrackCreation 0.323549 1.02473 2.65907 0.874453 0.540826 30
produce:pandoraShower:LArPandoraShowerCreation 0.279579 1.19128 3.81749 1.04774 0.692541 30
produce:pandoracalo:Calorimetry 0.159123 0.544403 1.36848 0.50103 0.254473 30
produce:pandoracalonosce:Calorimetry 0.142387 0.476519 1.30016 0.447748 0.226061 30
produce:pandorapid:Chi2ParticleID 0.000835457 0.00165874 0.00404855 0.00158533 0.000608195 30
produce:pandoraShowercalo:ShowerCalorimetry 0.25366 1.13058 3.85629 1.0159 0.677506 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.250099 1.09022 3.64348 0.964221 0.636673 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0420648 0.117384 0.222702 0.11732 0.0440444 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0149815 0.0397102 0.0734198 0.041411 0.0138101 30
produce:emtrkmichelid:EmTrackMichelId 13.5065 24.4148 45.2687 23.1287 6.76698 30
[art]:TriggerResults:TriggerResultInserter 1.5809e-05 2.70664e-05 6.3789e-05 2.57035e-05 1.0164e-05 30
end_path:out1:RootOutput 5.761e-06 8.697e-06 2.1801e-05 8.09e-06 2.97817e-06 30
end_path:michelremoving:michelremoving 0.011543 0.0659034 0.193459 0.0436069 0.052534 30
end_path:michelremovinggnocchi:michelremoving 0.00975066 0.0637647 0.181808 0.0416936 0.0500151 30
end_path:calibanagnocchi:CalibAnaTree 0.104213 0.376673 1.07032 0.327105 0.200898 30
end_path:out1:RootOutput(write) 0.277568 0.643681 1.22724 0.640756 0.220376 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4933.62 MB
Peak resident set size usage (VmHWM): 2369.91 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_2078_dataflow0_datawriter_0_20240908T012540_reco_stage1_reco_stage2_20240926T120005_keepup.root
\tHists: np04hd_raw_run029107_2078_dataflow0_datawriter_0_20240908T012540_reco_stage1_reco_stage2_20240926T120005_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
