Jobsub ID 78488.150@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78488.150@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:10:03 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:21:43 |
| From worker node | Hostname | cn-leo-044.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:42:24 |
| Input files | hd-protodune:np04hd_raw_run029081_0848_dataflow0_datawriter_0_20240905T061000.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:21:43 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
> 12 XUVs
C:0 T:4 884 XUs and 1981 XVs -> 57 XUVs
C:0 T:5 12503 XUs and 15049 XVs -> 608 XUVs
C:0 T:6 42549 XUs and 48263 XVs -> 1525 XUVs
C:0 T:7 305 XUs and 475 XVs -> 19 XUVs
4778 XUVs total
2017 collection wire objects
4778 potential space points
Neighbour search...
299984 tests to find 149800 neighbours
Iterating with no regularization...
Begin: 8.382e+06
0 7.84976e+06
1 7.8184e+06
2 7.81378e+06
Now with regularization...
Begin: 7.28935e+06
0 7.28753e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 203752 at 26-Sep-2024 13:48:48 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 29785 traces, 29785 summary
Retagger: tagging trace set: gauss with 23453 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 23453 traces tagged "gauss"
FrameSaver: q=490090 n=432851 tag=gauss
wclsFrameSaver: saving 29785 traces tagged "wiener"
FrameSaver: q=511077 n=386735 tag=wiener
1432 hits in TPC 0
1229 hits in TPC 1
2400 hits in TPC 2
1472 hits in TPC 3
1490 hits in TPC 4
2593 hits in TPC 5
2394 hits in TPC 6
1733 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 21 XUs and 13 XVs -> 0 XUVs
C:0 T:2 6075 XUs and 3290 XVs -> 146 XUVs
C:0 T:3 34 XUs and 32 XVs -> 4 XUVs
C:0 T:4 9 XUs and 9 XVs -> 0 XUVs
C:0 T:5 3567 XUs and 3143 XVs -> 287 XUVs
C:0 T:6 3569 XUs and 4399 XVs -> 60 XUVs
C:0 T:7 72 XUs and 77 XVs -> 3 XUVs
500 XUVs total
382 collection wire objects
500 potential space points
Neighbour search...
3288 tests to find 1704 neighbours
Iterating with no regularization...
Begin: 4.06779e+06
0 4.04833e+06
1 4.04812e+06
Now with regularization...
Begin: 3.7652e+06
0 3.76518e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 1 event 203752 TrkackID 2 startx, y, z -109.023 485.202 262.691 wireno, peakt tpcno 17 523.304 2 dist 0min T, max_T4.20884 1411.02
no of hits closeby 1 event 203752 TrkackID 2 startx, y, z -109.023 485.202 262.691 wireno, peakt tpcno 17 523.304 2 dist 0min T, max_T4.20884 1411.02
Begin processing the 30th record. run: 29081 subRun: 1 event: 203760 at 26-Sep-2024 13:50:00 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 69901 traces, 69901 summary
Retagger: tagging trace set: gauss with 53608 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53608 traces tagged "gauss"
FrameSaver: q=1.19935e+06 n=1094262 tag=gauss
wclsFrameSaver: saving 69901 traces tagged "wiener"
FrameSaver: q=1.26846e+06 n=1000312 tag=wiener
5604 hits in TPC 0
3524 hits in TPC 1
7864 hits in TPC 2
2981 hits in TPC 3
4436 hits in TPC 4
7961 hits in TPC 5
5996 hits in TPC 6
2895 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 363 XUs and 273 XVs -> 6 XUVs
C:0 T:2 11748 XUs and 17524 XVs -> 596 XUVs
C:0 T:3 431 XUs and 386 XVs -> 7 XUVs
C:0 T:4 130 XUs and 141 XVs -> 7 XUVs
C:0 T:5 26771 XUs and 24044 XVs -> 726 XUVs
C:0 T:6 12993 XUs and 30149 XVs -> 901 XUVs
C:0 T:7 8 XUs and 23 XVs -> 0 XUVs
2243 XUVs total
1371 collection wire objects
2243 potential space points
Neighbour search...
43553 tests to find 26048 neighbours
Iterating with no regularization...
Begin: 1.13352e+07
0 1.05132e+07
1 1.04797e+07
2 1.04778e+07
Now with regularization...
Begin: 9.75574e+06
0 9.75237e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:51:44 CEST Closed output file "np04hd_raw_run029081_0848_dataflow0_datawriter_0_20240905T061000_reco_stage1_reco_stage2_20240926T115144_keepup.root"
26-Sep-2024 13:51:44 CEST Closed input file "np04hd_raw_run029081_0848_dataflow0_datawriter_0_20240905T061000_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 69.5556 92.2454 125.668 92.7169 13.0748 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000458363 0.00254244 0.00635784 0.00211936 0.00174015 30
produce:fembfilter:PDHDFEMBFilter 0.663066 0.689868 0.850289 0.679606 0.0334769 30
produce:wclsdatahd:WireCellToolkit 54.2889 56.935 59.507 57.0277 1.40097 30
produce:gaushit:GausHitFinder 0.225013 0.526971 0.81971 0.527967 0.151098 30
produce:nhitsfilter:NumberOfHitsFilter 0.000306222 0.00050256 0.000769453 0.000499178 0.000111907 30
produce:reco3d:SpacePointSolver 0.0750945 0.918116 2.9961 0.647731 0.684931 30
produce:hitpdune:DisambigFromSpacePoints 0.065319 0.319502 0.721268 0.340959 0.139758 30
produce:pandora:StandardPandora 2.45832 8.80803 19.5031 8.15831 3.80057 30
produce:pandoraWriter:StandardPandora 0.0741517 0.14381 0.202457 0.150023 0.0357387 30
produce:pandoraTrack:LArPandoraTrackCreation 0.669213 1.93968 3.41196 1.88725 0.786073 30
produce:pandoraShower:LArPandoraShowerCreation 0.768299 2.51691 5.903 2.4302 1.29999 30
produce:pandoracalo:Calorimetry 0.288188 0.802826 1.39913 0.809554 0.295031 30
produce:pandoracalonosce:Calorimetry 0.265593 0.770321 1.52449 0.775078 0.309959 30
produce:pandorapid:Chi2ParticleID 0.000922836 0.00149243 0.00281988 0.00149763 0.000357147 30
produce:pandoraShowercalo:ShowerCalorimetry 0.536104 2.05454 5.43169 1.96249 1.13243 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.53226 1.98255 5.30546 1.92996 1.09288 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0514488 0.0948299 0.137845 0.101647 0.0238173 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0226763 0.0418195 0.0592538 0.0449078 0.0102016 30
produce:emtrkmichelid:EmTrackMichelId 6.52549 12.0263 17.902 11.7929 3.11642 30
[art]:TriggerResults:TriggerResultInserter 4.5822e-05 6.9841e-05 9.8491e-05 6.7783e-05 1.3235e-05 30
end_path:out1:RootOutput 1.6431e-05 2.37761e-05 3.866e-05 2.25135e-05 5.15456e-06 30
end_path:michelremoving:michelremoving 0.025225 0.0751187 0.193483 0.0540057 0.0484103 30
end_path:michelremovinggnocchi:michelremoving 0.0237231 0.0739941 0.192434 0.0513544 0.0489756 30
end_path:calibanagnocchi:CalibAnaTree 0.22873 0.591698 1.02498 0.574645 0.239047 30
end_path:out1:RootOutput(write) 0.305584 0.600887 0.947326 0.604913 0.16319 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5834.69 MB
Peak resident set size usage (VmHWM): 2678.21 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0848_dataflow0_datawriter_0_20240905T061000_reco_stage1_reco_stage2_20240926T115144_keepup.root
\tHists: np04hd_raw_run029081_0848_dataflow0_datawriter_0_20240905T061000_reco_stage1_reco_stage2_20240926T115144_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
