Jobsub ID 263349.116@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263349.116@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:16:24 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:15:24 |
| From worker node | Hostname | cn-leo-199.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:48:19 |
| Input files | hd-protodune:np04hd_raw_run029081_0763_dataflow1_datawriter_0_20240905T045913.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:15:24 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
.41972e+06
1 8.40151e+06
2 8.39858e+06
Now with regularization...
Begin: 7.99182e+06
0 7.99087e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 183345 at 26-Sep-2024 13:56:59 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 57204 traces, 57204 summary
Retagger: tagging trace set: gauss with 43140 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 43140 traces tagged "gauss"
FrameSaver: q=928181 n=843587 tag=gauss
wclsFrameSaver: saving 57204 traces tagged "wiener"
FrameSaver: q=985926 n=765692 tag=wiener
2100 hits in TPC 0
4240 hits in TPC 1
8193 hits in TPC 2
4272 hits in TPC 3
2759 hits in TPC 4
3804 hits in TPC 5
3843 hits in TPC 6
3063 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 744 XUs and 643 XVs -> 16 XUVs
C:0 T:2 38858 XUs and 51013 XVs -> 2543 XUVs
C:0 T:3 437 XUs and 593 XVs -> 21 XUVs
C:0 T:4 10 XUs and 9 XVs -> 0 XUVs
C:0 T:5 8205 XUs and 7035 XVs -> 337 XUVs
C:0 T:6 32045 XUs and 37163 XVs -> 619 XUVs
C:0 T:7 77 XUs and 101 XVs -> 7 XUVs
3543 XUVs total
1312 collection wire objects
3543 potential space points
Neighbour search...
144687 tests to find 78084 neighbours
Iterating with no regularization...
Begin: 4.84167e+06
0 4.25547e+06
1 4.18211e+06
2 4.1686e+06
3 4.16451e+06
Now with regularization...
Begin: 3.7803e+06
0 3.77801e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 183345 TrkackID 1 startx, y, z 101.381 318.114 7.11641 wireno, peakt tpcno 463 272.514 5 dist 0min T, max_T75.8482 1382.64
no of hits closeby 0 event 183345 TrkackID 1 startx, y, z 101.381 318.114 7.11641 wireno, peakt tpcno 463 272.514 5 dist 0min T, max_T75.8482 1382.64
Begin processing the 30th record. run: 29081 subRun: 1 event: 183353 at 26-Sep-2024 13:58:37 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 79651 traces, 79651 summary
Retagger: tagging trace set: gauss with 58774 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58774 traces tagged "gauss"
FrameSaver: q=1.34485e+06 n=1085477 tag=gauss
wclsFrameSaver: saving 79651 traces tagged "wiener"
FrameSaver: q=1.44272e+06 n=979038 tag=wiener
5012 hits in TPC 0
3821 hits in TPC 1
6462 hits in TPC 2
5261 hits in TPC 3
6119 hits in TPC 4
9075 hits in TPC 5
3004 hits in TPC 6
2376 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1285 XUs and 2028 XVs -> 20 XUVs
C:0 T:2 26886 XUs and 24949 XVs -> 1508 XUVs
C:0 T:3 234 XUs and 319 XVs -> 18 XUVs
C:0 T:4 143 XUs and 201 XVs -> 4 XUVs
C:0 T:5 51790 XUs and 44347 XVs -> 1970 XUVs
C:0 T:6 25991 XUs and 16511 XVs -> 1228 XUVs
C:0 T:7 9 XUs and 18 XVs -> 2 XUVs
4750 XUVs total
1757 collection wire objects
4750 potential space points
Neighbour search...
167850 tests to find 102658 neighbours
Iterating with no regularization...
Begin: 1.14961e+07
0 1.02206e+07
1 1.01419e+07
2 1.0129e+07
3 1.01248e+07
Now with regularization...
Begin: 9.4875e+06
0 9.48475e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 2 event 183353 TrkackID 2 startx, y, z -90.468 524.39 454.246 wireno, peakt tpcno 257 2929.8 2 dist 0min T, max_T2794 4041.05
no of hits closeby 2 event 183353 TrkackID 2 startx, y, z -90.468 524.39 454.246 wireno, peakt tpcno 257 2929.8 2 dist 0min T, max_T2794 4041.05
26-Sep-2024 14:00:27 CEST Closed output file "np04hd_raw_run029081_0763_dataflow1_datawriter_0_20240905T045913_reco_stage1_reco_stage2_20240926T120027_keepup.root"
26-Sep-2024 14:00:27 CEST Closed input file "np04hd_raw_run029081_0763_dataflow1_datawriter_0_20240905T045913_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 67.2308 100.575 255.118 94.4165 30.5493 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00057082 0.0023416 0.00590343 0.00183603 0.00150112 30
produce:fembfilter:PDHDFEMBFilter 0.690698 0.719797 0.863937 0.714295 0.0305415 30
produce:wclsdatahd:WireCellToolkit 50.2576 59.8234 63.8928 60.635 3.31752 30
produce:gaushit:GausHitFinder 0.279192 0.540062 0.733342 0.558698 0.108536 30
produce:nhitsfilter:NumberOfHitsFilter 0.00031625 0.000503233 0.000915984 0.000501728 0.00011346 30
produce:reco3d:SpacePointSolver 0.155122 1.29046 13.065 0.775115 2.26498 30
produce:hitpdune:DisambigFromSpacePoints 0.0907939 0.348832 1.06868 0.336834 0.180619 30
produce:pandora:StandardPandora 2.86534 12.8804 134.509 8.78822 22.7371 30
produce:pandoraWriter:StandardPandora 0.0788387 0.158059 0.226459 0.155241 0.0362187 30
produce:pandoraTrack:LArPandoraTrackCreation 0.511196 2.25972 6.09466 2.19161 1.09083 30
produce:pandoraShower:LArPandoraShowerCreation 0.595251 2.60959 5.11954 2.55747 1.14096 30
produce:pandoracalo:Calorimetry 0.229176 0.892845 2.09349 0.884506 0.375196 30
produce:pandoracalonosce:Calorimetry 0.197198 0.821066 1.90336 0.819649 0.341035 30
produce:pandorapid:Chi2ParticleID 0.00104933 0.00159128 0.00305671 0.00157705 0.000358389 30
produce:pandoraShowercalo:ShowerCalorimetry 0.391192 1.99846 3.58931 1.9741 0.821156 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.382616 1.93393 3.49036 1.92749 0.786635 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0483976 0.0949773 0.13581 0.0947748 0.02175 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0206175 0.0437074 0.0627945 0.0436728 0.0101815 30
produce:emtrkmichelid:EmTrackMichelId 6.21786 12.3566 17.4797 12.0205 2.73436 30
[art]:TriggerResults:TriggerResultInserter 4.7599e-05 7.44162e-05 9.883e-05 7.68515e-05 1.19831e-05 30
end_path:out1:RootOutput 1.3031e-05 2.20265e-05 3.6065e-05 2.13495e-05 5.31904e-06 30
end_path:michelremoving:michelremoving 0.0230476 0.076558 0.21423 0.056594 0.0472383 30
end_path:michelremovinggnocchi:michelremoving 0.0200601 0.070677 0.211922 0.0520137 0.0462303 30
end_path:calibanagnocchi:CalibAnaTree 0.133664 0.650522 1.5854 0.617192 0.302968 30
end_path:out1:RootOutput(write) 0.273666 0.605784 0.885985 0.57609 0.137494 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5759.12 MB
Peak resident set size usage (VmHWM): 2593.43 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0763_dataflow1_datawriter_0_20240905T045913_reco_stage1_reco_stage2_20240926T120027_keepup.root
\tHists: np04hd_raw_run029081_0763_dataflow1_datawriter_0_20240905T045913_reco_stage1_reco_stage2_20240926T120027_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
