Jobsub ID 263310.121@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263310.121@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:02:33 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:06:51 |
| From worker node | Hostname | cn-leo-064.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:20:32 |
| Input files | hd-protodune:np04hd_raw_run029081_1068_dataflow7_datawriter_0_20240905T100409.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:06:51 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
1 1.32162e+07
2 1.32124e+07
Now with regularization...
Begin: 1.27249e+07
0 1.27237e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 256551 at 26-Sep-2024 13:25:39 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 53419 traces, 53419 summary
Retagger: tagging trace set: gauss with 42004 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 42004 traces tagged "gauss"
FrameSaver: q=942054 n=864636 tag=gauss
wclsFrameSaver: saving 53419 traces tagged "wiener"
FrameSaver: q=994212 n=799013 tag=wiener
3102 hits in TPC 0
5141 hits in TPC 1
3797 hits in TPC 2
1910 hits in TPC 3
3538 hits in TPC 4
3955 hits in TPC 5
5161 hits in TPC 6
4400 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4523 XUs and 2281 XVs -> 36 XUVs
C:0 T:2 6487 XUs and 5820 XVs -> 165 XUVs
C:0 T:3 206 XUs and 239 XVs -> 9 XUVs
C:0 T:4 557 XUs and 1154 XVs -> 204 XUVs
C:0 T:5 10632 XUs and 7675 XVs -> 591 XUVs
C:0 T:6 16511 XUs and 10937 XVs -> 187 XUVs
C:0 T:7 808 XUs and 687 XVs -> 27 XUVs
1219 XUVs total
726 collection wire objects
1219 potential space points
Neighbour search...
32107 tests to find 17828 neighbours
Iterating with no regularization...
Begin: 4.32133e+06
0 4.09473e+06
1 4.07556e+06
2 4.07287e+06
Now with regularization...
Begin: 3.75722e+06
0 3.75637e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 256551 TrkackID 1 startx, y, z 358.474 411.213 431.387 wireno, peakt tpcno 0 4341.13 5 dist 0.824374min T, max_T684 5248.5
no of hits closeby 0 event 256551 TrkackID 1 startx, y, z 358.474 411.213 431.387 wireno, peakt tpcno 0 4341.13 5 dist 0.824374min T, max_T684 5248.5
Begin processing the 30th record. run: 29081 subRun: 1 event: 256559 at 26-Sep-2024 13:27:15 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 69050 traces, 69050 summary
Retagger: tagging trace set: gauss with 48760 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48760 traces tagged "gauss"
FrameSaver: q=1.01052e+06 n=842590 tag=gauss
wclsFrameSaver: saving 69050 traces tagged "wiener"
FrameSaver: q=1.09639e+06 n=743443 tag=wiener
2358 hits in TPC 0
2721 hits in TPC 1
7444 hits in TPC 2
5657 hits in TPC 3
2737 hits in TPC 4
4391 hits in TPC 5
5223 hits in TPC 6
3437 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 278 XUs and 166 XVs -> 2 XUVs
C:0 T:2 32494 XUs and 41746 XVs -> 1632 XUVs
C:0 T:3 212 XUs and 232 XVs -> 6 XUVs
C:0 T:4 192 XUs and 207 XVs -> 18 XUVs
C:0 T:5 21609 XUs and 18337 XVs -> 178 XUVs
C:0 T:6 16900 XUs and 16705 XVs -> 590 XUVs
C:0 T:7 669 XUs and 1044 XVs -> 43 XUVs
2469 XUVs total
1145 collection wire objects
2469 potential space points
Neighbour search...
66457 tests to find 37604 neighbours
Iterating with no regularization...
Begin: 5.32569e+06
0 4.67277e+06
1 4.63657e+06
2 4.63038e+06
3 4.62809e+06
Now with regularization...
Begin: 4.1919e+06
0 4.19018e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 5 event 256559 TrkackID 0 startx, y, z 88.1736 502.066 3.08379 wireno, peakt tpcno 463 870.943 5 dist 0min T, max_T829.054 3257.89
no of hits closeby 5 event 256559 TrkackID 0 startx, y, z 88.1736 502.066 3.08379 wireno, peakt tpcno 463 870.943 5 dist 0min T, max_T829.054 3257.89
26-Sep-2024 13:28:58 CEST Closed output file "np04hd_raw_run029081_1068_dataflow7_datawriter_0_20240905T100409_reco_stage1_reco_stage2_20240926T112858_keepup.root"
26-Sep-2024 13:28:59 CEST Closed input file "np04hd_raw_run029081_1068_dataflow7_datawriter_0_20240905T100409_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 76.5239 97.5739 123.178 96.4021 11.1792 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000595636 0.00233382 0.0127573 0.00129931 0.00248908 30
produce:fembfilter:PDHDFEMBFilter 0.694502 0.718428 0.758976 0.71714 0.0154054 30
produce:wclsdatahd:WireCellToolkit 58.7308 60.895 64.4674 61.0262 1.33426 30
produce:gaushit:GausHitFinder 0.32336 0.530442 0.769721 0.508928 0.116585 30
produce:nhitsfilter:NumberOfHitsFilter 0.000362537 0.000535872 0.000725239 0.000537405 8.77654e-05 30
produce:reco3d:SpacePointSolver 0.181495 0.835615 3.25945 0.570282 0.752271 30
produce:hitpdune:DisambigFromSpacePoints 0.110421 0.311914 0.578387 0.282602 0.129301 30
produce:pandora:StandardPandora 3.97094 8.50203 15.3563 7.62857 3.26563 30
produce:pandoraWriter:StandardPandora 0.0874372 0.155043 0.234458 0.156835 0.0339457 30
produce:pandoraTrack:LArPandoraTrackCreation 0.696622 2.35005 4.78742 2.20828 1.07725 30
produce:pandoraShower:LArPandoraShowerCreation 0.910837 2.81838 6.27601 2.81752 1.2921 30
produce:pandoracalo:Calorimetry 0.291592 0.909344 1.92587 0.868491 0.368852 30
produce:pandoracalonosce:Calorimetry 0.287043 0.851413 1.79388 0.821964 0.353488 30
produce:pandorapid:Chi2ParticleID 0.000995522 0.00162937 0.00278856 0.00164067 0.000359682 30
produce:pandoraShowercalo:ShowerCalorimetry 0.549772 2.13203 4.29917 2.09759 0.965451 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.546786 2.07535 4.12109 2.05498 0.926479 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0547657 0.0949309 0.139493 0.0952594 0.0229327 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0235521 0.0433732 0.0655521 0.0438244 0.0104343 30
produce:emtrkmichelid:EmTrackMichelId 7.35508 12.5713 17.2106 12.3971 2.52048 30
[art]:TriggerResults:TriggerResultInserter 3.5973e-05 7.63409e-05 0.000112622 7.5426e-05 1.55154e-05 30
end_path:out1:RootOutput 1.2689e-05 2.31471e-05 3.7501e-05 2.16295e-05 5.2665e-06 30
end_path:michelremoving:michelremoving 0.0247633 0.0733163 0.201873 0.0601971 0.0457075 30
end_path:michelremovinggnocchi:michelremoving 0.0229585 0.0693813 0.196935 0.0535156 0.045906 30
end_path:calibanagnocchi:CalibAnaTree 0.186344 0.658527 1.27875 0.681678 0.29396 30
end_path:out1:RootOutput(write) 0.339679 0.593528 0.90182 0.56879 0.136624 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5880.3 MB
Peak resident set size usage (VmHWM): 2747.59 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_1068_dataflow7_datawriter_0_20240905T100409_reco_stage1_reco_stage2_20240926T112858_keepup.root
\tHists: np04hd_raw_run029081_1068_dataflow7_datawriter_0_20240905T100409_reco_stage1_reco_stage2_20240926T112858_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
