Jobsub ID 263397.6@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263397.6@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3455 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:31:47 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:37:28 |
| From worker node | Hostname | cn-leo-007.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:57:53 |
| Input files | hd-protodune:np04hd_raw_run029109_0561_dataflow6_datawriter_0_20240910T023031.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:37:28 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
112 XVs -> 8 XUVs
4265 XUVs total
1661 collection wire objects
4265 potential space points
Neighbour search...
276693 tests to find 142594 neighbours
Iterating with no regularization...
Begin: 3.06078e+07
0 2.80241e+07
1 2.78415e+07
2 2.7832e+07
Now with regularization...
Begin: 2.73598e+07
0 2.73568e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29109 subRun: 1 event: 134902 at 26-Sep-2024 14:01:12 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 61413 traces, 61413 summary
Retagger: tagging trace set: gauss with 43866 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 43866 traces tagged "gauss"
FrameSaver: q=933607 n=777552 tag=gauss
wclsFrameSaver: saving 61413 traces tagged "wiener"
FrameSaver: q=1.01328e+06 n=693979 tag=wiener
2348 hits in TPC 0
3205 hits in TPC 1
5881 hits in TPC 2
3441 hits in TPC 3
3647 hits in TPC 4
5549 hits in TPC 5
3617 hits in TPC 6
2521 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 38350 XUs and 26869 XVs -> 921 XUVs
C:0 T:3 194 XUs and 251 XVs -> 3 XUVs
C:0 T:4 42 XUs and 53 XVs -> 1 XUVs
C:0 T:5 24986 XUs and 31011 XVs -> 1379 XUVs
C:0 T:6 5629 XUs and 5900 XVs -> 138 XUVs
C:0 T:7 227 XUs and 261 XVs -> 11 XUVs
2453 XUVs total
999 collection wire objects
2453 potential space points
Neighbour search...
98877 tests to find 53524 neighbours
Iterating with no regularization...
Begin: 5.31316e+06
0 4.90989e+06
1 4.88983e+06
2 4.88708e+06
Now with regularization...
Begin: 4.61328e+06
0 4.61219e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 134902 TrkackID 1 startx, y, z 318.861 335.92 461.671 wireno, peakt tpcno 205 4705.05 5 dist 0min T, max_T875.279 4962.7
no of hits closeby 0 event 134902 TrkackID 1 startx, y, z 318.861 335.92 461.671 wireno, peakt tpcno 205 4705.05 5 dist 0min T, max_T875.279 4962.7
Begin processing the 30th record. run: 29109 subRun: 1 event: 134918 at 26-Sep-2024 14:03:01 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 77167 traces, 77167 summary
Retagger: tagging trace set: gauss with 60196 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60196 traces tagged "gauss"
FrameSaver: q=1.27476e+06 n=1193128 tag=gauss
wclsFrameSaver: saving 77167 traces tagged "wiener"
FrameSaver: q=1.36977e+06 n=1101615 tag=wiener
5546 hits in TPC 0
6263 hits in TPC 1
6808 hits in TPC 2
4047 hits in TPC 3
2601 hits in TPC 4
7092 hits in TPC 5
7215 hits in TPC 6
2796 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 7851 XUs and 12397 XVs -> 3156 XUVs
C:0 T:2 44896 XUs and 20066 XVs -> 642 XUVs
C:0 T:3 217 XUs and 297 XVs -> 10 XUVs
C:0 T:4 83 XUs and 107 XVs -> 6 XUVs
C:0 T:5 26203 XUs and 28342 XVs -> 661 XUVs
C:0 T:6 40468 XUs and 33425 XVs -> 2228 XUVs
C:0 T:7 45 XUs and 83 XVs -> 3 XUVs
6706 XUVs total
1729 collection wire objects
6706 potential space points
Neighbour search...
783422 tests to find 413602 neighbours
Iterating with no regularization...
Begin: 7.70793e+06
0 7.21696e+06
1 7.18143e+06
2 7.1774e+06
Now with regularization...
Begin: 6.87798e+06
0 6.87661e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 134918 TrkackID 15 startx, y, z 109.686 92.5335 458.229 wireno, peakt tpcno 105 4139.82 5 dist 0min T, max_T4139.82 5631.49
no of hits closeby 0 event 134918 TrkackID 15 startx, y, z 109.686 92.5335 458.229 wireno, peakt tpcno 105 4139.82 5 dist 0min T, max_T4139.82 5631.49
26-Sep-2024 14:05:02 CEST Closed output file "np04hd_raw_run029109_0561_dataflow6_datawriter_0_20240910T023031_reco_stage1_reco_stage2_20240926T120502_keepup.root"
26-Sep-2024 14:05:02 CEST Closed input file "np04hd_raw_run029109_0561_dataflow6_datawriter_0_20240910T023031_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 74.186 96.452 122.341 95.3983 12.334 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000508834 0.00250306 0.00841402 0.0020223 0.00178833 30
produce:fembfilter:PDHDFEMBFilter 0.678624 0.717214 0.753398 0.720634 0.0240739 30
produce:wclsdatahd:WireCellToolkit 55.7289 59.5368 63.9453 59.8201 1.99091 30
produce:gaushit:GausHitFinder 0.36676 0.538031 0.755643 0.504932 0.105085 30
produce:nhitsfilter:NumberOfHitsFilter 0.000386692 0.000503429 0.000698535 0.000503723 7.32859e-05 30
produce:reco3d:SpacePointSolver 0.249791 0.935562 3.29696 0.779062 0.641155 30
produce:hitpdune:DisambigFromSpacePoints 0.124462 0.311149 0.578376 0.299252 0.10712 30
produce:pandora:StandardPandora 3.8479 9.21699 16.0857 9.07432 2.97741 30
produce:pandoraWriter:StandardPandora 0.092048 0.161669 0.252446 0.158242 0.0373271 30
produce:pandoraTrack:LArPandoraTrackCreation 0.698132 2.35077 5.18744 2.01885 1.17453 30
produce:pandoraShower:LArPandoraShowerCreation 1.00386 2.82421 6.95704 2.49087 1.50713 30
produce:pandoracalo:Calorimetry 0.330065 0.913182 1.82184 0.862639 0.387393 30
produce:pandoracalonosce:Calorimetry 0.274343 0.862166 1.68461 0.806212 0.36468 30
produce:pandorapid:Chi2ParticleID 0.000986177 0.00159106 0.00256222 0.00154093 0.00034663 30
produce:pandoraShowercalo:ShowerCalorimetry 0.699166 2.14398 4.95778 2.03236 1.06649 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.674102 2.10102 4.83772 2.02886 1.04669 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.067631 0.097272 0.143126 0.0916391 0.0213399 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.029083 0.0445494 0.0666185 0.0427644 0.0107743 30
produce:emtrkmichelid:EmTrackMichelId 8.23877 11.9405 17.3177 11.9708 2.1818 30
[art]:TriggerResults:TriggerResultInserter 3.9706e-05 6.9894e-05 9.4419e-05 7.3738e-05 1.42292e-05 30
end_path:out1:RootOutput 1.0496e-05 2.08399e-05 4.2897e-05 2.0851e-05 6.11794e-06 30
end_path:michelremoving:michelremoving 0.0275818 0.0779706 0.237877 0.0627236 0.0481787 30
end_path:michelremovinggnocchi:michelremoving 0.0237146 0.075667 0.238492 0.0589211 0.0478501 30
end_path:calibanagnocchi:CalibAnaTree 0.231118 0.63252 1.28446 0.605155 0.300727 30
end_path:out1:RootOutput(write) 0.421514 0.603047 0.845701 0.587409 0.113258 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5782.82 MB
Peak resident set size usage (VmHWM): 2609.82 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029109_0561_dataflow6_datawriter_0_20240910T023031_reco_stage1_reco_stage2_20240926T120502_keepup.root
\tHists: np04hd_raw_run029109_0561_dataflow6_datawriter_0_20240910T023031_reco_stage1_reco_stage2_20240926T120502_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
