Jobsub ID 263562.14@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263562.14@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 11:39:06 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 13:49:39 |
| From worker node | Hostname | cn-leo-145.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 12:00:30 |
| Input files | hd-protodune:np04hd_raw_run029081_0935_dataflow0_datawriter_0_20240905T073025.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 13:49:39 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
1.09215e+07
0 9.13401e+06
1 8.98868e+06
2 8.97358e+06
3 8.96985e+06
Now with regularization...
Begin: 8.52324e+06
0 8.52077e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 224632 at 26-Sep-2024 15:07:43 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 88708 traces, 88708 summary
Retagger: tagging trace set: gauss with 63494 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63494 traces tagged "gauss"
FrameSaver: q=1.39109e+06 n=1168112 tag=gauss
wclsFrameSaver: saving 88708 traces tagged "wiener"
FrameSaver: q=1.52392e+06 n=1061446 tag=wiener
5040 hits in TPC 0
4315 hits in TPC 1
6913 hits in TPC 2
4219 hits in TPC 3
3825 hits in TPC 4
4755 hits in TPC 5
10579 hits in TPC 6
7739 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 356 XUs and 778 XVs -> 40 XUVs
C:0 T:2 46228 XUs and 58240 XVs -> 2306 XUVs
C:0 T:3 631 XUs and 1357 XVs -> 51 XUVs
C:0 T:4 69 XUs and 63 XVs -> 1 XUVs
C:0 T:5 27798 XUs and 61532 XVs -> 1604 XUVs
C:0 T:6 80851 XUs and 84200 XVs -> 3339 XUVs
C:0 T:7 966 XUs and 1485 XVs -> 67 XUVs
7408 XUVs total
2529 collection wire objects
7408 potential space points
Neighbour search...
556246 tests to find 288886 neighbours
Iterating with no regularization...
Begin: 1.15318e+07
0 1.00314e+07
1 9.93096e+06
2 9.91793e+06
3 9.91413e+06
Now with regularization...
Begin: 9.18936e+06
0 9.18689e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 224632 TrkackID 23 startx, y, z 44.1912 591.268 363.176 wireno, peakt tpcno 192 7.54343 5 dist 0min T, max_T5.97764 539.788
no of hits closeby 0 event 224632 TrkackID 23 startx, y, z 44.1912 591.268 363.176 wireno, peakt tpcno 192 7.54343 5 dist 0min T, max_T5.97764 539.788
Begin processing the 30th record. run: 29081 subRun: 1 event: 224640 at 26-Sep-2024 15:09:51 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 49997 traces, 49997 summary
Retagger: tagging trace set: gauss with 39090 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 39090 traces tagged "gauss"
FrameSaver: q=858243 n=764823 tag=gauss
wclsFrameSaver: saving 49997 traces tagged "wiener"
FrameSaver: q=905487 n=692257 tag=wiener
2609 hits in TPC 0
1775 hits in TPC 1
4546 hits in TPC 2
2596 hits in TPC 3
3269 hits in TPC 4
6106 hits in TPC 5
3604 hits in TPC 6
2240 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 703 XUs and 124 XVs -> 3 XUVs
C:0 T:2 8204 XUs and 5809 XVs -> 187 XUVs
C:0 T:3 74 XUs and 83 XVs -> 3 XUVs
C:0 T:4 22 XUs and 6 XVs -> 0 XUVs
C:0 T:5 52644 XUs and 20481 XVs -> 725 XUVs
C:0 T:6 5538 XUs and 6321 XVs -> 267 XUVs
C:0 T:7 164 XUs and 191 XVs -> 16 XUVs
1201 XUVs total
681 collection wire objects
1201 potential space points
Neighbour search...
24417 tests to find 14330 neighbours
Iterating with no regularization...
Begin: 6.62828e+06
0 6.07469e+06
1 6.05496e+06
2 6.05252e+06
Now with regularization...
Begin: 5.85207e+06
0 5.8512e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 15:11:23 CEST Closed output file "np04hd_raw_run029081_0935_dataflow0_datawriter_0_20240905T073025_reco_stage1_reco_stage2_20240926T131123_keepup.root"
26-Sep-2024 15:11:23 CEST Closed input file "np04hd_raw_run029081_0935_dataflow0_datawriter_0_20240905T073025_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 85.9408 103.486 153.065 100.145 14.3006 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000718448 0.00340166 0.0156488 0.00249552 0.00291065 30
produce:fembfilter:PDHDFEMBFilter 0.67886 0.735982 1.0385 0.705277 0.0729387 30
produce:wclsdatahd:WireCellToolkit 59.0899 62.2863 65.8647 62.3433 1.52915 30
produce:gaushit:GausHitFinder 0.362567 0.572675 1.02558 0.547398 0.137978 30
produce:nhitsfilter:NumberOfHitsFilter 0.000318544 0.000563232 0.000812884 0.000558514 0.000104409 30
produce:reco3d:SpacePointSolver 0.169891 1.0985 2.50643 0.959246 0.653383 30
produce:hitpdune:DisambigFromSpacePoints 0.132067 0.357509 0.821898 0.317831 0.15886 30
produce:pandora:StandardPandora 5.21329 10.539 28.4355 9.44062 4.77304 30
produce:pandoraWriter:StandardPandora 0.104714 0.162425 0.251611 0.158597 0.0350997 30
produce:pandoraTrack:LArPandoraTrackCreation 0.86546 2.34649 4.54453 2.24526 0.800332 30
produce:pandoraShower:LArPandoraShowerCreation 0.932327 3.1799 8.50228 2.87107 1.54991 30
produce:pandoracalo:Calorimetry 0.432843 0.976729 1.98965 0.954704 0.320999 30
produce:pandoracalonosce:Calorimetry 0.366701 0.920876 1.87783 0.904873 0.308459 30
produce:pandorapid:Chi2ParticleID 0.00137036 0.00171329 0.00282743 0.00165309 0.000307894 30
produce:pandoraShowercalo:ShowerCalorimetry 0.69777 2.56846 7.71525 2.27036 1.32596 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.666575 2.53277 9.00142 2.22473 1.49204 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0674599 0.101472 0.184855 0.0995477 0.0227078 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0286943 0.0451811 0.0872832 0.0438448 0.0108432 30
produce:emtrkmichelid:EmTrackMichelId 9.83011 13.1938 19.2159 12.87 2.26053 30
[art]:TriggerResults:TriggerResultInserter 4.5326e-05 7.95346e-05 0.000110816 7.9467e-05 1.66842e-05 30
end_path:out1:RootOutput 1.5833e-05 2.45375e-05 4.984e-05 2.36385e-05 6.37443e-06 30
end_path:michelremoving:michelremoving 0.0306259 0.070808 0.216816 0.0555182 0.0467903 30
end_path:michelremovinggnocchi:michelremoving 0.0218876 0.0667856 0.202739 0.0496585 0.046434 30
end_path:calibanagnocchi:CalibAnaTree 0.282718 0.693096 1.55002 0.622722 0.303297 30
end_path:out1:RootOutput(write) 0.471399 0.664085 1.03474 0.650307 0.138609 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5785.85 MB
Peak resident set size usage (VmHWM): 2638.6 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0935_dataflow0_datawriter_0_20240905T073025_reco_stage1_reco_stage2_20240926T131123_keepup.root
\tHists: np04hd_raw_run029081_0935_dataflow0_datawriter_0_20240905T073025_reco_stage1_reco_stage2_20240926T131123_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
