Jobsub ID 263351.9@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263351.9@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3449 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:16:27 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:25:10 |
| From worker node | Hostname | cn-leo-155.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:49:11 |
| Input files | hd-protodune:np04hd_raw_run029107_0865_dataflow2_datawriter_0_20240907T044823.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:25:10 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
: 1.04527e+07
0 1.04437e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29107 subRun: 1 event: 207866 at 26-Sep-2024 13:59:31 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 81374 traces, 81374 summary
Retagger: tagging trace set: gauss with 59680 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 59680 traces tagged "gauss"
FrameSaver: q=1.22352e+06 n=1083642 tag=gauss
wclsFrameSaver: saving 81374 traces tagged "wiener"
FrameSaver: q=1.32228e+06 n=978317 tag=wiener
5748 hits in TPC 0
5245 hits in TPC 1
7756 hits in TPC 2
5251 hits in TPC 3
1824 hits in TPC 4
5198 hits in TPC 5
6333 hits in TPC 6
4283 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 2845 XUs and 1677 XVs -> 46 XUVs
C:0 T:2 38098 XUs and 45657 XVs -> 2596 XUVs
C:0 T:3 1003 XUs and 1278 XVs -> 16 XUVs
C:0 T:4 28 XUs and 38 XVs -> 1 XUVs
C:0 T:5 18203 XUs and 44042 XVs -> 1901 XUVs
C:0 T:6 18878 XUs and 29332 XVs -> 1256 XUVs
C:0 T:7 225 XUs and 391 XVs -> 0 XUVs
5816 XUVs total
1708 collection wire objects
5816 potential space points
Neighbour search...
469260 tests to find 248566 neighbours
Iterating with no regularization...
Begin: 6.84591e+06
0 6.09768e+06
1 6.04038e+06
2 6.02795e+06
3 6.0239e+06
Now with regularization...
Begin: 5.48718e+06
0 5.48401e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 4 event 207866 TrkackID 1 startx, y, z -71.0391 484.838 10.6727 wireno, peakt tpcno 470 3556.64 6 dist 0min T, max_T3554.8 5688.29
no of hits closeby 4 event 207866 TrkackID 1 startx, y, z -71.0391 484.838 10.6727 wireno, peakt tpcno 470 3556.64 6 dist 0min T, max_T3554.8 5688.29
Begin processing the 30th record. run: 29107 subRun: 1 event: 207874 at 26-Sep-2024 14:01:37 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 54741 traces, 54741 summary
Retagger: tagging trace set: gauss with 41295 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 41295 traces tagged "gauss"
FrameSaver: q=855472 n=797060 tag=gauss
wclsFrameSaver: saving 54741 traces tagged "wiener"
FrameSaver: q=905366 n=723213 tag=wiener
3828 hits in TPC 0
2282 hits in TPC 1
6021 hits in TPC 2
3522 hits in TPC 3
3056 hits in TPC 4
3638 hits in TPC 5
4520 hits in TPC 6
3029 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 107 XUs and 157 XVs -> 1 XUVs
C:0 T:2 12043 XUs and 21900 XVs -> 884 XUVs
C:0 T:3 65 XUs and 62 XVs -> 6 XUVs
C:0 T:4 16 XUs and 13 XVs -> 1 XUVs
C:0 T:5 12346 XUs and 9556 XVs -> 169 XUVs
C:0 T:6 6289 XUs and 11018 XVs -> 270 XUVs
C:0 T:7 850 XUs and 1275 XVs -> 100 XUVs
1431 XUVs total
822 collection wire objects
1431 potential space points
Neighbour search...
56627 tests to find 30380 neighbours
Iterating with no regularization...
Begin: 4.35102e+06
0 3.91055e+06
1 3.85475e+06
2 3.84741e+06
3 3.84556e+06
Now with regularization...
Begin: 3.60952e+06
0 3.60817e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 207874 TrkackID 8 startx, y, z 37.2678 592.809 181.564 wireno, peakt tpcno 336 2184.01 2 dist 353.542min T, max_T6.01064 2206.59
no of hits closeby 0 event 207874 TrkackID 8 startx, y, z 37.2678 592.809 181.564 wireno, peakt tpcno 336 2184.01 2 dist 353.542min T, max_T6.01064 2206.59
26-Sep-2024 14:03:23 CEST Closed output file "np04hd_raw_run029107_0865_dataflow2_datawriter_0_20240907T044823_reco_stage1_reco_stage2_20240926T120323_keepup.root"
26-Sep-2024 14:03:23 CEST Closed input file "np04hd_raw_run029107_0865_dataflow2_datawriter_0_20240907T044823_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 82.433 107.372 150.296 102.386 18.7747 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000556022 0.00277909 0.00807514 0.00214604 0.00193303 30
produce:fembfilter:PDHDFEMBFilter 0.701771 0.759658 0.944771 0.747491 0.0516921 30
produce:wclsdatahd:WireCellToolkit 57.9986 62.7464 70.5219 61.4302 3.50528 30
produce:gaushit:GausHitFinder 0.260525 0.582947 0.930183 0.556867 0.162272 30
produce:nhitsfilter:NumberOfHitsFilter 0.000323943 0.000551631 0.000816711 0.000572863 0.000115529 30
produce:reco3d:SpacePointSolver 0.139023 1.12112 2.90772 0.9543 0.778068 30
produce:hitpdune:DisambigFromSpacePoints 0.103337 0.382054 0.921205 0.361864 0.204214 30
produce:pandora:StandardPandora 4.38195 10.4811 23.5764 8.41812 5.06458 30
produce:pandoraWriter:StandardPandora 0.0689601 0.165468 0.239091 0.161338 0.0466626 30
produce:pandoraTrack:LArPandoraTrackCreation 0.334433 2.41763 6.1794 2.0758 1.29805 30
produce:pandoraShower:LArPandoraShowerCreation 0.343226 3.17636 7.9037 2.88715 1.75492 30
produce:pandoracalo:Calorimetry 0.18631 0.98019 2.00709 0.912562 0.448653 30
produce:pandoracalonosce:Calorimetry 0.163988 0.920124 1.86852 0.835092 0.430744 30
produce:pandorapid:Chi2ParticleID 0.00118621 0.00171584 0.00264219 0.00161396 0.000392316 30
produce:pandoraShowercalo:ShowerCalorimetry 0.250217 2.59102 7.38598 2.22228 1.62942 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.245912 2.49121 7.58689 2.19153 1.48611 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.059699 0.106544 0.169458 0.103739 0.028271 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0223549 0.0458237 0.0749675 0.0451093 0.0126732 30
produce:emtrkmichelid:EmTrackMichelId 10.3603 16.4385 27.3217 15.507 4.39965 30
[art]:TriggerResults:TriggerResultInserter 5.8297e-05 8.09669e-05 0.000127959 7.3189e-05 1.71017e-05 30
end_path:out1:RootOutput 1.6099e-05 2.15332e-05 3.7588e-05 2.0309e-05 4.39238e-06 30
end_path:michelremoving:michelremoving 0.0219814 0.0949029 0.39714 0.0705723 0.0722006 30
end_path:michelremovinggnocchi:michelremoving 0.0155229 0.0851906 0.24891 0.0665155 0.0539228 30
end_path:calibanagnocchi:CalibAnaTree 0.0929511 0.734962 1.82712 0.693142 0.390709 30
end_path:out1:RootOutput(write) 0.391631 0.681508 1.05686 0.659094 0.190018 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5777.42 MB
Peak resident set size usage (VmHWM): 2630.12 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_0865_dataflow2_datawriter_0_20240907T044823_reco_stage1_reco_stage2_20240926T120323_keepup.root
\tHists: np04hd_raw_run029107_0865_dataflow2_datawriter_0_20240907T044823_reco_stage1_reco_stage2_20240926T120323_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
