Jobsub ID 263327.57@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263327.57@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:08:48 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:10:54 |
| From worker node | Hostname | cn-leo-134.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:41:09 |
| Input files | hd-protodune:np04hd_raw_run029081_0933_dataflow6_datawriter_0_20240905T072800.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:10:54 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
PC 2
7756 hits in TPC 3
1725 hits in TPC 4
3507 hits in TPC 5
6838 hits in TPC 6
3611 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 16 XUs and 13 XVs -> 0 XUVs
C:0 T:2 64867 XUs and 39152 XVs -> 1105 XUVs
C:0 T:3 291 XUs and 296 XVs -> 9 XUVs
C:0 T:4 49 XUs and 75 XVs -> 5 XUVs
C:0 T:5 15869 XUs and 11800 XVs -> 323 XUVs
C:0 T:6 33188 XUs and 30596 XVs -> 846 XUVs
C:0 T:7 387 XUs and 405 XVs -> 7 XUVs
2295 XUVs total
1158 collection wire objects
2295 potential space points
Neighbour search...
93201 tests to find 50178 neighbours
Iterating with no regularization...
Begin: 1.79421e+07
0 1.66238e+07
1 1.64818e+07
2 1.64793e+07
Now with regularization...
Begin: 1.61806e+07
0 1.61799e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 224182 at 26-Sep-2024 13:45:16 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 44748 traces, 44748 summary
Retagger: tagging trace set: gauss with 33096 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 33096 traces tagged "gauss"
FrameSaver: q=686066 n=605546 tag=gauss
wclsFrameSaver: saving 44748 traces tagged "wiener"
FrameSaver: q=766525 n=553434 tag=wiener
2989 hits in TPC 0
4350 hits in TPC 1
2303 hits in TPC 2
1645 hits in TPC 3
2975 hits in TPC 4
7740 hits in TPC 5
2487 hits in TPC 6
549 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4828 XUs and 3431 XVs -> 165 XUVs
C:0 T:2 5731 XUs and 4433 XVs -> 161 XUVs
C:0 T:3 559 XUs and 789 XVs -> 21 XUVs
C:0 T:4 60 XUs and 100 XVs -> 2 XUVs
C:0 T:5 26247 XUs and 39132 XVs -> 1667 XUVs
C:0 T:6 1473 XUs and 2252 XVs -> 62 XUVs
C:0 T:7 222 XUs and 450 XVs -> 53 XUVs
2131 XUVs total
837 collection wire objects
2131 potential space points
Neighbour search...
65461 tests to find 40440 neighbours
Iterating with no regularization...
Begin: 4.20806e+06
0 3.90167e+06
1 3.86731e+06
2 3.86131e+06
3 3.8599e+06
Now with regularization...
Begin: 3.60304e+06
0 3.60226e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 224190 at 26-Sep-2024 13:46:44 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64645 traces, 64645 summary
Retagger: tagging trace set: gauss with 46180 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 46180 traces tagged "gauss"
FrameSaver: q=959788 n=797896 tag=gauss
wclsFrameSaver: saving 64645 traces tagged "wiener"
FrameSaver: q=1.06471e+06 n=710114 tag=wiener
2280 hits in TPC 0
2066 hits in TPC 1
6235 hits in TPC 2
6066 hits in TPC 3
2785 hits in TPC 4
4101 hits in TPC 5
5128 hits in TPC 6
4414 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 45 XUs and 34 XVs -> 0 XUVs
C:0 T:2 69442 XUs and 99996 XVs -> 3282 XUVs
C:0 T:3 691 XUs and 1223 XVs -> 56 XUVs
C:0 T:4 22 XUs and 14 XVs -> 0 XUVs
C:0 T:5 12397 XUs and 10389 XVs -> 673 XUVs
C:0 T:6 32499 XUs and 49009 XVs -> 1441 XUVs
C:0 T:7 1391 XUs and 2780 XVs -> 395 XUVs
5847 XUVs total
1593 collection wire objects
5847 potential space points
Neighbour search...
596043 tests to find 275926 neighbours
Iterating with no regularization...
Begin: 2.30465e+08
0 2.24207e+08
1 2.23666e+08
2 2.23654e+08
Now with regularization...
Begin: 2.23303e+08
0 2.233e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:48:26 CEST Closed output file "np04hd_raw_run029081_0933_dataflow6_datawriter_0_20240905T072800_reco_stage1_reco_stage2_20240926T114826_keepup.root"
26-Sep-2024 13:48:26 CEST Closed input file "np04hd_raw_run029081_0933_dataflow6_datawriter_0_20240905T072800_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 64.2318 97.0259 133.641 96.045 18.1321 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000612119 0.00281466 0.00735733 0.00228376 0.00200722 30
produce:fembfilter:PDHDFEMBFilter 0.676598 0.708539 0.766996 0.707023 0.0173198 30
produce:wclsdatahd:WireCellToolkit 50.0622 58.5224 64.3455 59.7246 4.17651 30
produce:gaushit:GausHitFinder 0.249428 0.516751 0.74361 0.535444 0.128956 30
produce:nhitsfilter:NumberOfHitsFilter 0.000270299 0.000501762 0.000863071 0.00047152 0.000148093 30
produce:reco3d:SpacePointSolver 0.134125 1.02223 3.15032 0.663483 0.773812 30
produce:hitpdune:DisambigFromSpacePoints 0.104497 0.316351 0.587993 0.309816 0.129783 30
produce:pandora:StandardPandora 2.71483 9.48847 27.6576 9.32551 4.81854 30
produce:pandoraWriter:StandardPandora 0.0691062 0.156859 0.249356 0.16091 0.0483131 30
produce:pandoraTrack:LArPandoraTrackCreation 0.439244 2.38141 6.12051 2.14724 1.44435 30
produce:pandoraShower:LArPandoraShowerCreation 0.466316 3.09948 9.40833 2.60477 2.29071 30
produce:pandoracalo:Calorimetry 0.20984 0.918199 2.08534 0.832144 0.502917 30
produce:pandoracalonosce:Calorimetry 0.186567 0.857237 2.05581 0.762964 0.480901 30
produce:pandorapid:Chi2ParticleID 0.000884135 0.00159614 0.00256764 0.00152997 0.000382524 30
produce:pandoraShowercalo:ShowerCalorimetry 0.350966 2.30911 7.21406 2.01052 1.66388 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.33899 2.23975 6.64077 1.95458 1.56655 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0530055 0.0969347 0.144196 0.0951423 0.0264059 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.020931 0.0439731 0.0708576 0.0449422 0.0127867 30
produce:emtrkmichelid:EmTrackMichelId 6.93844 12.6352 19.3273 12.4623 3.28524 30
[art]:TriggerResults:TriggerResultInserter 4.6121e-05 8.32823e-05 0.000126342 7.93635e-05 1.85746e-05 30
end_path:out1:RootOutput 1.2019e-05 2.29968e-05 3.5754e-05 2.2711e-05 5.84937e-06 30
end_path:michelremoving:michelremoving 0.0182281 0.0623812 0.169531 0.052693 0.035667 30
end_path:michelremovinggnocchi:michelremoving 0.0155803 0.0593505 0.165655 0.051159 0.034633 30
end_path:calibanagnocchi:CalibAnaTree 0.115842 0.604615 1.456 0.559813 0.317163 30
end_path:out1:RootOutput(write) 0.331857 0.598091 0.912793 0.608128 0.154645 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5851.78 MB
Peak resident set size usage (VmHWM): 2689.39 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0933_dataflow6_datawriter_0_20240905T072800_reco_stage1_reco_stage2_20240926T114826_keepup.root
\tHists: np04hd_raw_run029081_0933_dataflow6_datawriter_0_20240905T072800_reco_stage1_reco_stage2_20240926T114826_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
