Jobsub ID 78788.24@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78788.24@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3453 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 12:38:51 |
| Site | ES_PIC |
| Entry | DUNE_T1_ES_PIC_ce14-multicore |
| Last heartbeat | 2024-09-26 14:10:50 |
| From worker node | Hostname | tds207.pic.es |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 12:39:42 |
| Input files | hd-protodune:np04hd_raw_run029107_3638_dataflow1_datawriter_0_20240909T043055.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 14:10:50 |
| Saved logs | justin-logs:78788.24-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
with no regularization...
Begin: 9.91827e+06
0 9.11893e+06
1 8.94339e+06
2 8.93868e+06
Now with regularization...
Begin: 8.4406e+06
0 8.44e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 873361 TrkackID 0 startx, y, z 130.969 599.323 128.673 wireno, peakt tpcno 457 1856.52 5 dist 0min T, max_T1856.52 3597
no of hits closeby 0 event 873361 TrkackID 0 startx, y, z 130.969 599.323 128.673 wireno, peakt tpcno 457 1856.52 5 dist 0min T, max_T1856.52 3597
Begin processing the 29th record. run: 29107 subRun: 1 event: 873369 at 26-Sep-2024 15:58:28 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 56244 traces, 56244 summary
Retagger: tagging trace set: gauss with 41725 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 41725 traces tagged "gauss"
FrameSaver: q=912161 n=779413 tag=gauss
wclsFrameSaver: saving 56244 traces tagged "wiener"
FrameSaver: q=969514 n=701219 tag=wiener
2917 hits in TPC 0
2822 hits in TPC 1
3856 hits in TPC 2
2678 hits in TPC 3
2112 hits in TPC 4
4641 hits in TPC 5
5759 hits in TPC 6
3356 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 101 XUs and 74 XVs -> 0 XUVs
C:0 T:2 9929 XUs and 16527 XVs -> 236 XUVs
C:0 T:3 124 XUs and 221 XVs -> 2 XUVs
C:0 T:4 19 XUs and 22 XVs -> 0 XUVs
C:0 T:5 6347 XUs and 7848 XVs -> 491 XUVs
C:0 T:6 26485 XUs and 23000 XVs -> 1389 XUVs
C:0 T:7 113 XUs and 210 XVs -> 3 XUVs
2121 XUVs total
1275 collection wire objects
2121 potential space points
Neighbour search...
41823 tests to find 25456 neighbours
Iterating with no regularization...
Begin: 1.17559e+07
0 1.05352e+07
1 1.04372e+07
2 1.04152e+07
3 1.0405e+07
Now with regularization...
Begin: 9.85474e+06
0 9.84921e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 873369 TrkackID 11 startx, y, z 44.0482 588.502 304.148 wireno, peakt tpcno 77 5 5 dist 0min T, max_T5 1562
no of hits closeby 0 event 873369 TrkackID 11 startx, y, z 44.0482 588.502 304.148 wireno, peakt tpcno 77 5 5 dist 0min T, max_T5 1562
Begin processing the 30th record. run: 29107 subRun: 1 event: 873377 at 26-Sep-2024 15:59:58 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 82507 traces, 82507 summary
Retagger: tagging trace set: gauss with 61603 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 61603 traces tagged "gauss"
FrameSaver: q=1.44236e+06 n=1152329 tag=gauss
wclsFrameSaver: saving 82507 traces tagged "wiener"
FrameSaver: q=1.55609e+06 n=1051974 tag=wiener
3926 hits in TPC 0
2718 hits in TPC 1
5164 hits in TPC 2
3514 hits in TPC 3
8971 hits in TPC 4
11392 hits in TPC 5
6133 hits in TPC 6
2352 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1274 XUs and 845 XVs -> 17 XUVs
C:0 T:2 20481 XUs and 15152 XVs -> 473 XUVs
C:0 T:3 72 XUs and 74 XVs -> 3 XUVs
C:0 T:4 335 XUs and 379 XVs -> 8 XUVs
C:0 T:5 54463 XUs and 59079 XVs -> 3377 XUVs
C:0 T:6 31436 XUs and 13677 XVs -> 604 XUVs
C:0 T:7 64 XUs and 93 XVs -> 2 XUVs
4484 XUVs total
1855 collection wire objects
4484 potential space points
Neighbour search...
128364 tests to find 78404 neighbours
Iterating with no regularization...
Begin: 1.2133e+07
0 1.13613e+07
1 1.12948e+07
2 1.12836e+07
Now with regularization...
Begin: 1.04432e+07
0 1.0439e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 1 event 873377 TrkackID 4 startx, y, z 12.6432 354.31 445.391 wireno, peakt tpcno 342 2800.91 6 dist 405.824min T, max_T1720.52 2813.63
no of hits closeby 1 event 873377 TrkackID 4 startx, y, z 12.6432 354.31 445.391 wireno, peakt tpcno 342 2800.91 6 dist 405.824min T, max_T1720.52 2813.63
26-Sep-2024 16:01:44 CEST Closed output file "np04hd_raw_run029107_3638_dataflow1_datawriter_0_20240909T043055_reco_stage1_reco_stage2_20240926T140144_keepup.root"
26-Sep-2024 16:01:44 CEST Closed input file "np04hd_raw_run029107_3638_dataflow1_datawriter_0_20240909T043055_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 43.6127 95.0995 257.742 88.1447 33.4767 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000327101 0.0019517 0.00374557 0.00195257 0.000905402 30
produce:fembfilter:PDHDFEMBFilter 0.600949 0.627116 0.650307 0.624916 0.0119731 30
produce:wclsdatahd:WireCellToolkit 39.9895 41.3875 43.7799 41.2014 0.923904 30
produce:gaushit:GausHitFinder 0.547083 0.810507 1.43748 0.766783 0.199357 30
produce:nhitsfilter:NumberOfHitsFilter 0.000208555 0.000337065 0.000525423 0.000311121 8.73271e-05 30
produce:reco3d:SpacePointSolver 0.206054 1.53428 19.1607 0.810865 3.38907 29
produce:hitpdune:DisambigFromSpacePoints 0.279351 0.732468 3.65903 0.498805 0.640538 29
produce:pandora:StandardPandora 5.85458 17.8499 139.974 12.2835 23.5025 29
produce:pandoraWriter:StandardPandora 0.0918022 0.148171 0.235618 0.142765 0.0311709 29
produce:pandoraTrack:LArPandoraTrackCreation 0.544163 1.29139 3.22493 1.16202 0.499554 29
produce:pandoraShower:LArPandoraShowerCreation 0.48112 1.52608 2.99236 1.39714 0.593016 29
produce:pandoracalo:Calorimetry 0.312435 0.667932 1.45913 0.607008 0.226422 29
produce:pandoracalonosce:Calorimetry 0.270158 0.607253 1.38569 0.549421 0.217292 29
produce:pandorapid:Chi2ParticleID 0.00113648 0.00180622 0.00320277 0.00175621 0.000404287 29
produce:pandoraShowercalo:ShowerCalorimetry 0.434549 1.44748 3.00968 1.34781 0.558686 29
produce:pandoraShowercalonosce:ShowerCalorimetry 0.428758 1.42016 2.96581 1.34267 0.561054 29
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.082325 0.139332 0.191988 0.140143 0.0275849 29
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0288629 0.0483267 0.0655314 0.0479376 0.00925233 29
produce:emtrkmichelid:EmTrackMichelId 16.8083 25.0222 37.8707 23.9466 4.70737 29
[art]:TriggerResults:TriggerResultInserter 1.7739e-05 2.91879e-05 7.0927e-05 2.62325e-05 9.41568e-06 30
end_path:out1:RootOutput 6.94e-06 9.94197e-06 2.3017e-05 9.287e-06 3.57839e-06 30
end_path:michelremoving:michelremoving 2.7173e-05 0.107881 0.493559 0.0401666 0.113603 30
end_path:michelremovinggnocchi:michelremoving 1.1933e-05 0.105524 0.502026 0.0347513 0.115515 30
end_path:calibanagnocchi:CalibAnaTree 1.1442e-05 0.507631 1.13969 0.52199 0.194919 30
end_path:out1:RootOutput(write) 0.33696 0.783095 1.2269 0.776673 0.173322 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4981.1 MB
Peak resident set size usage (VmHWM): 2628.91 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_3638_dataflow1_datawriter_0_20240909T043055_reco_stage1_reco_stage2_20240926T140144_keepup.root
\tHists: np04hd_raw_run029107_3638_dataflow1_datawriter_0_20240909T043055_reco_stage1_reco_stage2_20240926T140144_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
