Jobsub ID 263283.124@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263283.124@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:56:54 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce01 |
| Last heartbeat | 2024-09-26 11:20:13 |
| From worker node | Hostname | ccwcondor0563 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 09:59:10 |
| Input files | hd-protodune:np04hd_raw_run029081_0946_dataflow4_datawriter_0_20240905T074300.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 11:20:13 |
| Saved logs | justin-logs:263283.124-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
53041e+07
2 1.52811e+07
3 1.52735e+07
Now with regularization...
Begin: 1.46991e+07
0 1.46944e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 227276 at 26-Sep-2024 13:03:57 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 88632 traces, 88632 summary
Retagger: tagging trace set: gauss with 65683 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 65683 traces tagged "gauss"
FrameSaver: q=1.53812e+06 n=1269421 tag=gauss
wclsFrameSaver: saving 88632 traces tagged "wiener"
FrameSaver: q=1.66313e+06 n=1152480 tag=wiener
3147 hits in TPC 0
2579 hits in TPC 1
11714 hits in TPC 2
4543 hits in TPC 3
3801 hits in TPC 4
7840 hits in TPC 5
10712 hits in TPC 6
6479 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 323 XUs and 455 XVs -> 175 XUVs
C:0 T:2 90985 XUs and 102044 XVs -> 4719 XUVs
C:0 T:3 373 XUs and 545 XVs -> 12 XUVs
C:0 T:4 260 XUs and 254 XVs -> 17 XUVs
C:0 T:5 19152 XUs and 23303 XVs -> 1041 XUVs
C:0 T:6 103450 XUs and 80465 XVs -> 3029 XUVs
C:0 T:7 281 XUs and 271 XVs -> 9 XUVs
9002 XUVs total
3030 collection wire objects
9002 potential space points
Neighbour search...
590706 tests to find 256222 neighbours
Iterating with no regularization...
Begin: 1.46443e+07
0 1.24198e+07
1 1.22256e+07
2 1.21959e+07
3 1.21877e+07
Now with regularization...
Begin: 1.13308e+07
0 1.13254e+07
this->SetupTestBeamPfo(pNuPfo, pTestBeamPfo, testBeamStartVertex) return STATUS_CODE_NOT_FOUND
in function: Run
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_09_00-buildFW/src/larpandoracontent/LArThreeDReco/LArEventBuilding/TestBeamParticleCreationAlgorithm.cc line#: 43
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0087, LArTestBeamParticleCreation, STATUS_CODE_NOT_FOUND
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 227284 at 26-Sep-2024 13:06:07 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 82625 traces, 82625 summary
Retagger: tagging trace set: gauss with 60950 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60950 traces tagged "gauss"
FrameSaver: q=1.47253e+06 n=1197671 tag=gauss
wclsFrameSaver: saving 82625 traces tagged "wiener"
FrameSaver: q=1.59334e+06 n=1094431 tag=wiener
6221 hits in TPC 0
5725 hits in TPC 1
9317 hits in TPC 2
6491 hits in TPC 3
2710 hits in TPC 4
5571 hits in TPC 5
6810 hits in TPC 6
3354 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 5955 XUs and 8745 XVs -> 391 XUVs
C:0 T:2 85888 XUs and 89335 XVs -> 3859 XUVs
C:0 T:3 1065 XUs and 773 XVs -> 83 XUVs
C:0 T:4 489 XUs and 480 XVs -> 5 XUVs
C:0 T:5 19800 XUs and 43446 XVs -> 1806 XUVs
C:0 T:6 34979 XUs and 42597 XVs -> 945 XUVs
C:0 T:7 172 XUs and 228 XVs -> 20 XUVs
7109 XUVs total
2524 collection wire objects
7109 potential space points
Neighbour search...
389297 tests to find 178144 neighbours
Iterating with no regularization...
Begin: 3.85988e+07
0 3.1008e+07
1 3.0204e+07
2 3.01775e+07
Now with regularization...
Begin: 2.9056e+07
0 2.90478e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:08:19 CEST Closed output file "np04hd_raw_run029081_0946_dataflow4_datawriter_0_20240905T074300_reco_stage1_reco_stage2_20240926T110818_keepup.root"
26-Sep-2024 13:08:19 CEST Closed input file "np04hd_raw_run029081_0946_dataflow4_datawriter_0_20240905T074300_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 69.2466 105.175 138.762 105.153 17.3776 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000523511 0.00309219 0.00518461 0.00304896 0.00139566 30
produce:fembfilter:PDHDFEMBFilter 0.625967 0.978374 3.34389 0.787071 0.573497 30
produce:wclsdatahd:WireCellToolkit 35.5406 41.8484 48.775 42.3758 2.60888 30
produce:gaushit:GausHitFinder 0.469286 0.783197 1.27599 0.77865 0.181399 30
produce:nhitsfilter:NumberOfHitsFilter 0.000247464 0.000362459 0.00062423 0.000334086 8.40916e-05 30
produce:reco3d:SpacePointSolver 0.198202 1.38629 3.85903 1.27448 0.908304 30
produce:hitpdune:DisambigFromSpacePoints 0.160708 0.610104 1.49322 0.586845 0.281636 30
produce:pandora:StandardPandora 4.86015 14.4547 30.472 14.0947 5.60725 30
produce:pandoraWriter:StandardPandora 0.0835142 0.167119 0.246435 0.170039 0.0351375 30
produce:pandoraTrack:LArPandoraTrackCreation 0.591515 2.01535 4.56434 1.96291 0.896348 30
produce:pandoraShower:LArPandoraShowerCreation 0.43267 2.54385 5.68964 2.39525 1.37385 30
produce:pandoracalo:Calorimetry 0.354433 0.967506 1.94918 1.01532 0.373863 30
produce:pandoracalonosce:Calorimetry 0.278958 0.886492 2.01522 0.853105 0.379725 30
produce:pandorapid:Chi2ParticleID 0.00127224 0.00221862 0.00404781 0.00212164 0.000614889 30
produce:pandoraShowercalo:ShowerCalorimetry 0.479881 2.18831 5.87482 1.81739 1.20845 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.476931 1.98328 4.38395 1.69841 0.949088 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.052956 0.15258 0.21948 0.150172 0.0456135 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.020068 0.0545874 0.0821703 0.0563318 0.0157346 30
produce:emtrkmichelid:EmTrackMichelId 18.1084 32.2449 42.9079 32.6424 6.59096 30
[art]:TriggerResults:TriggerResultInserter 3.7791e-05 6.14217e-05 0.000110076 6.01125e-05 1.35192e-05 30
end_path:out1:RootOutput 1.1591e-05 1.87961e-05 3.7781e-05 1.7999e-05 5.05928e-06 30
end_path:michelremoving:michelremoving 0.0233501 0.0821522 0.259999 0.0524547 0.0689176 30
end_path:michelremovinggnocchi:michelremoving 0.0193688 0.0830979 0.324134 0.049676 0.0768358 30
end_path:calibanagnocchi:CalibAnaTree 0.246137 0.706481 1.44872 0.65644 0.306165 30
end_path:out1:RootOutput(write) 0.494118 0.807353 1.1665 0.844895 0.176395 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4948.32 MB
Peak resident set size usage (VmHWM): 2336.5 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0946_dataflow4_datawriter_0_20240905T074300_reco_stage1_reco_stage2_20240926T110818_keepup.root
\tHists: np04hd_raw_run029081_0946_dataflow4_datawriter_0_20240905T074300_reco_stage1_reco_stage2_20240926T110818_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata 