Jobsub ID 263269.131@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263269.131@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:51:00 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2024-09-26 11:03:49 |
| From worker node | Hostname | ccwcondor0621 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 09:53:06 |
| Input files | hd-protodune:np04hd_raw_run029092_0465_dataflow1_datawriter_0_20240906T004225.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 11:03:49 |
| Saved logs | justin-logs:263269.131-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
:1 1125 XUs and 1010 XVs -> 68 XUVs
C:0 T:2 14098 XUs and 24248 XVs -> 713 XUVs
C:0 T:3 344 XUs and 404 XVs -> 12 XUVs
C:0 T:4 90 XUs and 99 XVs -> 8 XUVs
C:0 T:5 20200 XUs and 33514 XVs -> 955 XUVs
C:0 T:6 30597 XUs and 48512 XVs -> 3520 XUVs
C:0 T:7 173 XUs and 152 XVs -> 4 XUVs
5280 XUVs total
1586 collection wire objects
5280 potential space points
Neighbour search...
420776 tests to find 244096 neighbours
Iterating with no regularization...
Begin: 1.17801e+07
0 1.10031e+07
1 1.09346e+07
2 1.0923e+07
3 1.09198e+07
Now with regularization...
Begin: 1.02564e+07
0 1.02536e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29092 subRun: 1 event: 111833 at 26-Sep-2024 12:51:54 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 78451 traces, 78451 summary
Retagger: tagging trace set: gauss with 58951 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58951 traces tagged "gauss"
FrameSaver: q=1.22663e+06 n=1074621 tag=gauss
wclsFrameSaver: saving 78451 traces tagged "wiener"
FrameSaver: q=1.29814e+06 n=959731 tag=wiener
3429 hits in TPC 0
3262 hits in TPC 1
7706 hits in TPC 2
4323 hits in TPC 3
2957 hits in TPC 4
5467 hits in TPC 5
8074 hits in TPC 6
4659 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 159 XUs and 82 XVs -> 21 XUVs
C:0 T:1 43 XUs and 31 XVs -> 0 XUVs
C:0 T:2 23396 XUs and 31856 XVs -> 974 XUVs
C:0 T:3 96 XUs and 176 XVs -> 8 XUVs
C:0 T:4 278 XUs and 202 XVs -> 21 XUVs
C:0 T:5 10897 XUs and 9899 XVs -> 244 XUVs
C:0 T:6 33698 XUs and 20961 XVs -> 674 XUVs
C:0 T:7 304 XUs and 385 XVs -> 17 XUVs
1959 XUVs total
1338 collection wire objects
1959 potential space points
Neighbour search...
25903 tests to find 16180 neighbours
Iterating with no regularization...
Begin: 5.13455e+06
0 4.8819e+06
1 4.87057e+06
2 4.86931e+06
Now with regularization...
Begin: 4.41976e+06
0 4.41933e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 111833 TrkackID 17 startx, y, z 288.704 383.953 315.63 wireno, peakt tpcno 366 5094.33 7 dist 4.11102min T, max_T5025 5855.59
no of hits closeby 0 event 111833 TrkackID 17 startx, y, z 288.704 383.953 315.63 wireno, peakt tpcno 366 5094.33 7 dist 4.11102min T, max_T5025 5855.59
Begin processing the 30th record. run: 29092 subRun: 1 event: 111841 at 26-Sep-2024 12:53:18 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 62214 traces, 62214 summary
Retagger: tagging trace set: gauss with 45658 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 45658 traces tagged "gauss"
FrameSaver: q=956053 n=812421 tag=gauss
wclsFrameSaver: saving 62214 traces tagged "wiener"
FrameSaver: q=1.03289e+06 n=732812 tag=wiener
3056 hits in TPC 0
3120 hits in TPC 1
3776 hits in TPC 2
2425 hits in TPC 3
3657 hits in TPC 4
5842 hits in TPC 5
6449 hits in TPC 6
3188 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1238 XUs and 2160 XVs -> 260 XUVs
C:0 T:2 9536 XUs and 15441 XVs -> 109 XUVs
C:0 T:3 148 XUs and 176 XVs -> 14 XUVs
C:0 T:4 37 XUs and 37 XVs -> 7 XUVs
C:0 T:5 29843 XUs and 33371 XVs -> 3749 XUVs
C:0 T:6 18469 XUs and 22059 XVs -> 418 XUVs
C:0 T:7 319 XUs and 369 XVs -> 2 XUVs
4559 XUVs total
948 collection wire objects
4559 potential space points
Neighbour search...
391435 tests to find 234072 neighbours
Iterating with no regularization...
Begin: 4.37671e+06
0 3.76851e+06
1 3.68674e+06
2 3.66703e+06
3 3.66067e+06
4 3.65761e+06
Now with regularization...
Begin: 3.30275e+06
0 3.29965e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:54:27 CEST Closed output file "np04hd_raw_run029092_0465_dataflow1_datawriter_0_20240906T004225_reco_stage1_reco_stage2_20240926T105427_keepup.root"
26-Sep-2024 12:54:27 CEST Closed input file "np04hd_raw_run029092_0465_dataflow1_datawriter_0_20240906T004225_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 53.0929 85.3337 161.413 80.9307 22.3199 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000622281 0.00289401 0.00453016 0.00320571 0.00108727 30
produce:fembfilter:PDHDFEMBFilter 0.492567 0.538941 0.668547 0.521903 0.0454318 30
produce:wclsdatahd:WireCellToolkit 31.2967 34.2159 38.2091 33.7133 1.82286 30
produce:gaushit:GausHitFinder 0.276786 0.548274 0.974824 0.526286 0.164451 30
produce:nhitsfilter:NumberOfHitsFilter 0.000198571 0.000314661 0.000497598 0.000298066 7.4166e-05 30
produce:reco3d:SpacePointSolver 0.147519 1.23696 6.72192 0.770042 1.42511 30
produce:hitpdune:DisambigFromSpacePoints 0.124875 0.469729 1.66918 0.38084 0.320829 30
produce:pandora:StandardPandora 3.00378 11.2752 28.6413 10.0351 5.99494 30
produce:pandoraWriter:StandardPandora 0.0605769 0.131339 0.211133 0.12876 0.0365376 30
produce:pandoraTrack:LArPandoraTrackCreation 0.401784 1.63458 5.32047 1.30771 1.06673 30
produce:pandoraShower:LArPandoraShowerCreation 0.313726 2.0361 8.0724 1.63021 1.60892 30
produce:pandoracalo:Calorimetry 0.218111 0.748142 3.15178 0.623273 0.530336 30
produce:pandoracalonosce:Calorimetry 0.187309 0.728957 3.06222 0.55505 0.520762 30
produce:pandorapid:Chi2ParticleID 0.00100001 0.00178417 0.00346769 0.00167381 0.000530908 30
produce:pandoraShowercalo:ShowerCalorimetry 0.269715 1.7717 8.34876 1.45667 1.48999 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.279851 1.62809 7.4457 1.4573 1.2888 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0542011 0.107473 0.217103 0.0956698 0.036544 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0230922 0.0416391 0.0904497 0.0382037 0.0135472 30
produce:emtrkmichelid:EmTrackMichelId 14.7005 26.6662 46.7683 26.3592 7.14779 30
[art]:TriggerResults:TriggerResultInserter 3.0858e-05 5.69238e-05 0.000189253 5.23475e-05 2.77128e-05 30
end_path:out1:RootOutput 1.1541e-05 1.91103e-05 5.5694e-05 1.62605e-05 8.63704e-06 30
end_path:michelremoving:michelremoving 0.014449 0.0912879 0.346858 0.0531005 0.0812456 30
end_path:michelremovinggnocchi:michelremoving 0.0153057 0.0856235 0.292585 0.048323 0.0759424 30
end_path:calibanagnocchi:CalibAnaTree 0.162216 0.541559 1.64169 0.485293 0.305885 30
end_path:out1:RootOutput(write) 0.334447 0.627882 1.06121 0.600279 0.173715 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4942.73 MB
Peak resident set size usage (VmHWM): 2364.19 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029092_0465_dataflow1_datawriter_0_20240906T004225_reco_stage1_reco_stage2_20240926T105427_keepup.root
\tHists: np04hd_raw_run029092_0465_dataflow1_datawriter_0_20240906T004225_reco_stage1_reco_stage2_20240926T105427_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
