Jobsub ID 263340.9@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263340.9@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3449 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:13:50 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:21:39 |
| From worker node | Hostname | cn-leo-169.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:45:57 |
| Input files | hd-protodune:np04hd_raw_run029107_0844_dataflow5_datawriter_0_20240907T043145.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:21:39 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
meSaver: saving 67991 traces tagged "wiener"
FrameSaver: q=1.08114e+06 n=864911 tag=wiener
5323 hits in TPC 0
5337 hits in TPC 1
5156 hits in TPC 2
2445 hits in TPC 3
4703 hits in TPC 4
6429 hits in TPC 5
4992 hits in TPC 6
2230 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 2373 XUs and 2810 XVs -> 14 XUVs
C:0 T:2 38364 XUs and 55328 XVs -> 2999 XUVs
C:0 T:3 168 XUs and 392 XVs -> 14 XUVs
C:0 T:4 176 XUs and 194 XVs -> 3 XUVs
C:0 T:5 28391 XUs and 30872 XVs -> 984 XUVs
C:0 T:6 23695 XUs and 47490 XVs -> 2090 XUVs
C:0 T:7 342 XUs and 412 XVs -> 6 XUVs
6110 XUVs total
1569 collection wire objects
6110 potential space points
Neighbour search...
625520 tests to find 295380 neighbours
Iterating with no regularization...
Begin: 4.10574e+06
0 3.71324e+06
1 3.68231e+06
2 3.67689e+06
3 3.67512e+06
Now with regularization...
Begin: 3.4339e+06
0 3.43265e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29107 subRun: 1 event: 202821 at 26-Sep-2024 13:52:35 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67424 traces, 67424 summary
Retagger: tagging trace set: gauss with 48718 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48718 traces tagged "gauss"
FrameSaver: q=948503 n=874990 tag=gauss
wclsFrameSaver: saving 67424 traces tagged "wiener"
FrameSaver: q=1.02838e+06 n=779672 tag=wiener
3431 hits in TPC 0
2477 hits in TPC 1
7311 hits in TPC 2
2970 hits in TPC 3
3194 hits in TPC 4
4579 hits in TPC 5
8360 hits in TPC 6
2805 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2 XUs and 2 XVs -> 0 XUVs
C:0 T:1 119 XUs and 53 XVs -> 0 XUVs
C:0 T:2 41519 XUs and 66229 XVs -> 1033 XUVs
C:0 T:3 448 XUs and 899 XVs -> 16 XUVs
C:0 T:4 935 XUs and 2395 XVs -> 131 XUVs
C:0 T:5 27507 XUs and 9264 XVs -> 263 XUVs
C:0 T:6 38392 XUs and 63571 XVs -> 1559 XUVs
C:0 T:7 393 XUs and 507 XVs -> 3 XUVs
3005 XUVs total
1255 collection wire objects
3005 potential space points
Neighbour search...
182983 tests to find 96138 neighbours
Iterating with no regularization...
Begin: 5.05284e+06
0 4.71278e+06
1 4.68907e+06
2 4.68541e+06
Now with regularization...
Begin: 4.48884e+06
0 4.4876e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29107 subRun: 1 event: 202829 at 26-Sep-2024 13:54:20 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 63065 traces, 63065 summary
Retagger: tagging trace set: gauss with 47905 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 47905 traces tagged "gauss"
FrameSaver: q=994636 n=931223 tag=gauss
wclsFrameSaver: saving 63065 traces tagged "wiener"
FrameSaver: q=1.05919e+06 n=848871 tag=wiener
4558 hits in TPC 0
3947 hits in TPC 1
7258 hits in TPC 2
4684 hits in TPC 3
2607 hits in TPC 4
3721 hits in TPC 5
3670 hits in TPC 6
2059 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13 XUs and 10 XVs -> 0 XUVs
C:0 T:1 490 XUs and 324 XVs -> 3 XUVs
C:0 T:2 23722 XUs and 22580 XVs -> 1079 XUVs
C:0 T:3 228 XUs and 212 XVs -> 1 XUVs
C:0 T:4 11 XUs and 21 XVs -> 0 XUVs
C:0 T:5 5599 XUs and 8835 XVs -> 128 XUVs
C:0 T:6 5822 XUs and 8945 XVs -> 1065 XUVs
C:0 T:7 16 XUs and 17 XVs -> 1 XUVs
2277 XUVs total
969 collection wire objects
2277 potential space points
Neighbour search...
63277 tests to find 37494 neighbours
Iterating with no regularization...
Begin: 3.16221e+06
0 2.86856e+06
1 2.82745e+06
2 2.82129e+06
3 2.81985e+06
Now with regularization...
Begin: 2.50873e+06
0 2.50806e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:55:56 CEST Closed output file "np04hd_raw_run029107_0844_dataflow5_datawriter_0_20240907T043145_reco_stage1_reco_stage2_20240926T115556_keepup.root"
26-Sep-2024 13:55:56 CEST Closed input file "np04hd_raw_run029107_0844_dataflow5_datawriter_0_20240907T043145_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 63.4512 100.51 145.202 98.7325 16.7069 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000432044 0.00260736 0.00722905 0.00190498 0.00194024 30
produce:fembfilter:PDHDFEMBFilter 0.665666 0.701075 0.7387 0.7019 0.0157138 30
produce:wclsdatahd:WireCellToolkit 53.668 60.0536 65.9094 60.623 2.82497 30
produce:gaushit:GausHitFinder 0.379465 0.579829 1.06541 0.538759 0.158722 30
produce:nhitsfilter:NumberOfHitsFilter 0.000332079 0.000515948 0.000975901 0.000506924 0.000132719 30
produce:reco3d:SpacePointSolver 0.124167 1.47656 4.16269 1.30318 1.13181 29
produce:hitpdune:DisambigFromSpacePoints 0.134064 0.374771 0.872858 0.317736 0.189186 29
produce:pandora:StandardPandora 4.9562 10.1052 18.3709 9.35367 3.72415 29
produce:pandoraWriter:StandardPandora 0.0968312 0.167438 0.255698 0.161914 0.0362536 29
produce:pandoraTrack:LArPandoraTrackCreation 0.878796 2.50042 6.4123 2.19373 1.25565 29
produce:pandoraShower:LArPandoraShowerCreation 0.824085 3.38924 9.00531 2.99115 2.01973 29
produce:pandoracalo:Calorimetry 0.386078 0.976202 2.3185 0.890287 0.441512 29
produce:pandoracalonosce:Calorimetry 0.351491 0.918908 2.21239 0.812636 0.429348 29
produce:pandorapid:Chi2ParticleID 0.000968765 0.00159729 0.00250779 0.00150639 0.000310341 29
produce:pandoraShowercalo:ShowerCalorimetry 0.640225 2.61269 6.72955 2.35042 1.43356 29
produce:pandoraShowercalonosce:ShowerCalorimetry 0.63372 2.56175 7.11458 2.34804 1.43551 29
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0562399 0.103469 0.14863 0.0992626 0.0226543 29
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0232592 0.0471168 0.0717688 0.0458388 0.011132 29
produce:emtrkmichelid:EmTrackMichelId 8.6142 13.4327 19.736 13.1031 2.60941 29
[art]:TriggerResults:TriggerResultInserter 3.8559e-05 6.65414e-05 9.3224e-05 6.84545e-05 1.20336e-05 30
end_path:out1:RootOutput 1.3668e-05 2.12231e-05 3.4102e-05 2.0548e-05 4.37664e-06 30
end_path:michelremoving:michelremoving 5.6199e-05 0.075376 0.219829 0.0511279 0.0568579 30
end_path:michelremovinggnocchi:michelremoving 4.0886e-05 0.07266 0.21235 0.0513388 0.0537343 30
end_path:calibanagnocchi:CalibAnaTree 3.8348e-05 0.652983 1.55388 0.625495 0.332487 30
end_path:out1:RootOutput(write) 0.405049 0.628323 0.950406 0.612699 0.147852 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5799.59 MB
Peak resident set size usage (VmHWM): 2649.32 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_0844_dataflow5_datawriter_0_20240907T043145_reco_stage1_reco_stage2_20240926T115556_keepup.root
\tHists: np04hd_raw_run029107_0844_dataflow5_datawriter_0_20240907T043145_reco_stage1_reco_stage2_20240926T115556_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
