Jobsub ID 78476.135@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78476.135@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3448 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:05:07 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:23:17 |
| From worker node | Hostname | cn-leo-134.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:36:34 |
| Input files | hd-protodune:np04hd_raw_run029094_0115_dataflow5_datawriter_0_20240906T092925.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:23:17 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
ireno, peakt tpcno 60 88.5 5 dist 0min T, max_T2.86624 3057
no of hits closeby 2 event 27821 TrkackID 2 startx, y, z 8.32856 350.463 446.847 wireno, peakt tpcno 372 4301.51 6 dist 498.417min T, max_T3261.7 4310.83
no of hits closeby 0 event 27821 TrkackID 0 startx, y, z 242.043 358.648 269.419 wireno, peakt tpcno 60 88.5 5 dist 0min T, max_T2.86624 3057
no of hits closeby 2 event 27821 TrkackID 2 startx, y, z 8.32856 350.463 446.847 wireno, peakt tpcno 372 4301.51 6 dist 498.417min T, max_T3261.7 4310.83
Begin processing the 29th record. run: 29094 subRun: 1 event: 27829 at 26-Sep-2024 13:42:10 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 58822 traces, 58822 summary
Retagger: tagging trace set: gauss with 44956 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 44956 traces tagged "gauss"
FrameSaver: q=1.02968e+06 n=879144 tag=gauss
wclsFrameSaver: saving 58822 traces tagged "wiener"
FrameSaver: q=1.10453e+06 n=806113 tag=wiener
4044 hits in TPC 0
2897 hits in TPC 1
3265 hits in TPC 2
2292 hits in TPC 3
3558 hits in TPC 4
5805 hits in TPC 5
5713 hits in TPC 6
3843 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 7660 XUs and 7812 XVs -> 139 XUVs
C:0 T:2 4258 XUs and 6786 XVs -> 238 XUVs
C:0 T:3 47 XUs and 67 XVs -> 2 XUVs
C:0 T:4 72 XUs and 78 XVs -> 4 XUVs
C:0 T:5 19536 XUs and 14722 XVs -> 782 XUVs
C:0 T:6 12433 XUs and 14547 XVs -> 392 XUVs
C:0 T:7 1659 XUs and 3987 XVs -> 211 XUVs
1768 XUVs total
872 collection wire objects
1768 potential space points
Neighbour search...
53088 tests to find 32320 neighbours
Iterating with no regularization...
Begin: 6.8757e+06
0 6.45686e+06
1 6.43582e+06
2 6.4319e+06
Now with regularization...
Begin: 6.04109e+06
0 6.0386e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 27829 TrkackID 7 startx, y, z 79.1655 534.084 3.71692 wireno, peakt tpcno 93 2676.74 2 dist 379.743min T, max_T7.27287 2708.51
no of hits closeby 0 event 27829 TrkackID 11 startx, y, z -28.7905 565.455 406.513 wireno, peakt tpcno 289 6.11997 6 dist 0.386686min T, max_T0.501159 379.925
no of hits closeby 0 event 27829 TrkackID 7 startx, y, z 79.1655 534.084 3.71692 wireno, peakt tpcno 93 2676.74 2 dist 379.743min T, max_T7.27287 2708.51
no of hits closeby 0 event 27829 TrkackID 11 startx, y, z -28.7905 565.455 406.513 wireno, peakt tpcno 289 6.11997 6 dist 0.386686min T, max_T0.501159 379.925
Begin processing the 30th record. run: 29094 subRun: 1 event: 27837 at 26-Sep-2024 13:43:41 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 47833 traces, 47833 summary
Retagger: tagging trace set: gauss with 33862 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 33862 traces tagged "gauss"
FrameSaver: q=679716 n=588083 tag=gauss
wclsFrameSaver: saving 47833 traces tagged "wiener"
FrameSaver: q=739957 n=528665 tag=wiener
2152 hits in TPC 0
1543 hits in TPC 1
4549 hits in TPC 2
2684 hits in TPC 3
2624 hits in TPC 4
4819 hits in TPC 5
2882 hits in TPC 6
1623 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 553 XUs and 278 XVs -> 12 XUVs
C:0 T:2 16956 XUs and 25456 XVs -> 3106 XUVs
C:0 T:3 131 XUs and 123 XVs -> 11 XUVs
C:0 T:4 30 XUs and 63 XVs -> 2 XUVs
C:0 T:5 16225 XUs and 15512 XVs -> 510 XUVs
C:0 T:6 7488 XUs and 11488 XVs -> 746 XUVs
C:0 T:7 6 XUs and 3 XVs -> 0 XUVs
4387 XUVs total
932 collection wire objects
4387 potential space points
Neighbour search...
316603 tests to find 198564 neighbours
Iterating with no regularization...
Begin: 3.96343e+06
0 3.40554e+06
1 3.35243e+06
2 3.34002e+06
3 3.33541e+06
4 3.33331e+06
Now with regularization...
Begin: 3.02426e+06
0 3.0221e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:45:00 CEST Closed output file "np04hd_raw_run029094_0115_dataflow5_datawriter_0_20240906T092925_reco_stage1_reco_stage2_20240926T114500_keepup.root"
26-Sep-2024 13:45:00 CEST Closed input file "np04hd_raw_run029094_0115_dataflow5_datawriter_0_20240906T092925_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 79.1514 94.3062 116.914 92.5832 9.7538 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000571931 0.00284817 0.00650354 0.00226267 0.00191134 30
produce:fembfilter:PDHDFEMBFilter 0.684293 0.708475 0.897109 0.702962 0.0369416 30
produce:wclsdatahd:WireCellToolkit 55.3113 59.6369 63.5124 59.9065 2.33973 30
produce:gaushit:GausHitFinder 0.311713 0.52466 0.787694 0.52533 0.119725 30
produce:nhitsfilter:NumberOfHitsFilter 0.000343859 0.000486165 0.000792868 0.000486839 9.5287e-05 30
produce:reco3d:SpacePointSolver 0.169839 0.631761 1.79799 0.496391 0.380001 30
produce:hitpdune:DisambigFromSpacePoints 0.108179 0.300054 0.534439 0.300977 0.124565 30
produce:pandora:StandardPandora 4.27343 8.75045 17.3455 8.55219 2.96652 30
produce:pandoraWriter:StandardPandora 0.100526 0.150653 0.19717 0.152751 0.0249271 30
produce:pandoraTrack:LArPandoraTrackCreation 1.00961 2.0076 3.9806 1.91562 0.697624 30
produce:pandoraShower:LArPandoraShowerCreation 1.02171 2.53945 4.71904 2.44584 0.97324 30
produce:pandoracalo:Calorimetry 0.459659 0.828543 1.69674 0.848162 0.269975 30
produce:pandoracalonosce:Calorimetry 0.414996 0.775299 1.62115 0.780291 0.259637 30
produce:pandorapid:Chi2ParticleID 0.00116304 0.00161698 0.00319545 0.00155039 0.000383143 30
produce:pandoraShowercalo:ShowerCalorimetry 0.761931 2.02815 4.07794 1.96906 0.80946 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.750273 1.978 3.96169 1.93061 0.791271 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0621861 0.0928193 0.125543 0.0914398 0.0168883 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0301668 0.0411027 0.0600683 0.0402121 0.00729937 30
produce:emtrkmichelid:EmTrackMichelId 7.8277 11.649 16.6427 11.5295 1.89899 30
[art]:TriggerResults:TriggerResultInserter 4.9749e-05 7.96004e-05 0.000103816 8.00165e-05 1.24005e-05 30
end_path:out1:RootOutput 1.4627e-05 2.4402e-05 3.1289e-05 2.4674e-05 3.91396e-06 30
end_path:michelremoving:michelremoving 0.0323376 0.0821778 0.410265 0.0519481 0.0718309 30
end_path:michelremovinggnocchi:michelremoving 0.0280136 0.0793403 0.415503 0.049409 0.0733307 30
end_path:calibanagnocchi:CalibAnaTree 0.256799 0.586936 1.26522 0.528502 0.216338 30
end_path:out1:RootOutput(write) 0.402788 0.581626 0.79029 0.592139 0.109094 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5747.98 MB
Peak resident set size usage (VmHWM): 2591.11 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029094_0115_dataflow5_datawriter_0_20240906T092925_reco_stage1_reco_stage2_20240926T114500_keepup.root
\tHists: np04hd_raw_run029094_0115_dataflow5_datawriter_0_20240906T092925_reco_stage1_reco_stage2_20240926T114500_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
