Jobsub ID 263324.145@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263324.145@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:07:35 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:12:22 |
| From worker node | Hostname | cn-leo-187.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:39:14 |
| Input files | hd-protodune:np04hd_raw_run029081_0499_dataflow5_datawriter_0_20240905T004535.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:12:22 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
potential space points
Neighbour search...
36708 tests to find 23362 neighbours
Iterating with no regularization...
Begin: 2.2985e+06
0 2.1537e+06
1 2.14451e+06
2 2.14375e+06
Now with regularization...
Begin: 1.91113e+06
0 1.91086e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 2 event 119981 TrkackID 7 startx, y, z 30.4648 578.81 347.495 wireno, peakt tpcno 85 5.07066 5 dist 0min T, max_T3.37685 401.976
no of hits closeby 2 event 119981 TrkackID 7 startx, y, z 30.4648 578.81 347.495 wireno, peakt tpcno 85 5.07066 5 dist 0min T, max_T3.37685 401.976
Begin processing the 29th record. run: 29081 subRun: 1 event: 119997 at 26-Sep-2024 13:43:58 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68665 traces, 68665 summary
Retagger: tagging trace set: gauss with 51766 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 51766 traces tagged "gauss"
FrameSaver: q=1.12178e+06 n=972953 tag=gauss
wclsFrameSaver: saving 68665 traces tagged "wiener"
FrameSaver: q=1.20592e+06 n=881284 tag=wiener
5540 hits in TPC 0
5684 hits in TPC 1
4642 hits in TPC 2
5581 hits in TPC 3
3760 hits in TPC 4
4997 hits in TPC 5
3873 hits in TPC 6
2566 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 58 XUs and 11 XVs -> 0 XUVs
C:0 T:1 4062 XUs and 3302 XVs -> 116 XUVs
C:0 T:2 20607 XUs and 14835 XVs -> 763 XUVs
C:0 T:3 671 XUs and 917 XVs -> 94 XUVs
C:0 T:4 5 XUs and 3 XVs -> 0 XUVs
C:0 T:5 18082 XUs and 17919 XVs -> 1433 XUVs
C:0 T:6 7122 XUs and 12532 XVs -> 782 XUVs
C:0 T:7 119 XUs and 169 XVs -> 7 XUVs
3195 XUVs total
1159 collection wire objects
3195 potential space points
Neighbour search...
153463 tests to find 85072 neighbours
Iterating with no regularization...
Begin: 9.86039e+06
0 9.18194e+06
1 9.1401e+06
2 9.13316e+06
Now with regularization...
Begin: 8.41507e+06
0 8.41109e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 119997 TrkackID 0 startx, y, z 191.754 443.133 459.83 wireno, peakt tpcno 75 2060.76 5 dist 0min T, max_T1995.88 4529.18
no of hits closeby 0 event 119997 TrkackID 0 startx, y, z 191.754 443.133 459.83 wireno, peakt tpcno 75 2060.76 5 dist 0min T, max_T1995.88 4529.18
Begin processing the 30th record. run: 29081 subRun: 1 event: 120005 at 26-Sep-2024 13:45:34 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 78902 traces, 78902 summary
Retagger: tagging trace set: gauss with 57940 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 57940 traces tagged "gauss"
FrameSaver: q=1.15489e+06 n=1011547 tag=gauss
wclsFrameSaver: saving 78902 traces tagged "wiener"
FrameSaver: q=1.24557e+06 n=903114 tag=wiener
3756 hits in TPC 0
4115 hits in TPC 1
7202 hits in TPC 2
2425 hits in TPC 3
4268 hits in TPC 4
7550 hits in TPC 5
7541 hits in TPC 6
2137 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 723 XUs and 351 XVs -> 8 XUVs
C:0 T:2 40976 XUs and 26018 XVs -> 1103 XUVs
C:0 T:3 140 XUs and 160 XVs -> 10 XUVs
C:0 T:4 33 XUs and 31 XVs -> 0 XUVs
C:0 T:5 38563 XUs and 55203 XVs -> 1304 XUVs
C:0 T:6 53873 XUs and 50942 XVs -> 2082 XUVs
C:0 T:7 84 XUs and 42 XVs -> 2 XUVs
4509 XUVs total
1778 collection wire objects
4509 potential space points
Neighbour search...
271689 tests to find 132764 neighbours
Iterating with no regularization...
Begin: 1.23462e+07
0 1.13995e+07
1 1.13387e+07
2 1.13304e+07
Now with regularization...
Begin: 1.09311e+07
0 1.09285e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:47:15 CEST Closed output file "np04hd_raw_run029081_0499_dataflow5_datawriter_0_20240905T004535_reco_stage1_reco_stage2_20240926T114715_keepup.root"
26-Sep-2024 13:47:15 CEST Closed input file "np04hd_raw_run029081_0499_dataflow5_datawriter_0_20240905T004535_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 73.345 97.1989 148.502 97.3432 14.4659 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000546485 0.0026144 0.00755484 0.00206969 0.00203177 30
produce:fembfilter:PDHDFEMBFilter 0.665004 0.697935 0.76508 0.696388 0.0183819 30
produce:wclsdatahd:WireCellToolkit 54.177 57.4111 61.7526 57.4016 2.18627 30
produce:gaushit:GausHitFinder 0.300029 0.553254 0.773627 0.57109 0.120685 30
produce:nhitsfilter:NumberOfHitsFilter 0.000320166 0.00049258 0.000642957 0.000491747 8.59931e-05 30
produce:reco3d:SpacePointSolver 0.13451 0.933086 3.24418 0.676068 0.701259 30
produce:hitpdune:DisambigFromSpacePoints 0.129186 0.350437 0.795399 0.335966 0.143529 30
produce:pandora:StandardPandora 3.38613 10.2235 25.4074 9.94461 4.71532 30
produce:pandoraWriter:StandardPandora 0.0980991 0.159376 0.258849 0.158314 0.0376119 30
produce:pandoraTrack:LArPandoraTrackCreation 0.841551 2.62153 9.15307 2.25342 1.67227 30
produce:pandoraShower:LArPandoraShowerCreation 0.999933 3.02301 8.09958 3.00546 1.50611 30
produce:pandoracalo:Calorimetry 0.343012 1.00756 3.28442 0.894983 0.565298 30
produce:pandoracalonosce:Calorimetry 0.315168 0.95118 3.32173 0.85932 0.578478 30
produce:pandorapid:Chi2ParticleID 0.00097657 0.001624 0.00290805 0.00153462 0.000452418 30
produce:pandoraShowercalo:ShowerCalorimetry 0.740255 2.37657 7.02302 2.27262 1.23516 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.717961 2.28681 6.19028 2.23325 1.11111 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0599838 0.0972132 0.142301 0.0988211 0.0206365 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0278292 0.0442069 0.0698681 0.0472993 0.0100402 30
produce:emtrkmichelid:EmTrackMichelId 7.29327 12.5692 20.3474 12.8561 2.70262 30
[art]:TriggerResults:TriggerResultInserter 4.1978e-05 6.35752e-05 8.4406e-05 6.15185e-05 1.11216e-05 30
end_path:out1:RootOutput 1.1481e-05 1.65405e-05 2.3402e-05 1.5534e-05 3.4793e-06 30
end_path:michelremoving:michelremoving 0.0269971 0.0916055 0.255362 0.0863658 0.0545549 30
end_path:michelremovinggnocchi:michelremoving 0.0226059 0.0894417 0.26052 0.0858733 0.0553844 30
end_path:calibanagnocchi:CalibAnaTree 0.224686 0.722887 2.85978 0.636886 0.480641 30
end_path:out1:RootOutput(write) 0.36163 0.615265 0.963045 0.635253 0.134414 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5802.36 MB
Peak resident set size usage (VmHWM): 2618.1 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0499_dataflow5_datawriter_0_20240905T004535_reco_stage1_reco_stage2_20240926T114715_keepup.root
\tHists: np04hd_raw_run029081_0499_dataflow5_datawriter_0_20240905T004535_reco_stage1_reco_stage2_20240926T114715_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
