Jobsub ID 272283.106@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272283.106@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3661 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-10-09 20:14:15 |
| Site | UK_Brunel |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_26 |
| Last heartbeat | 2024-10-09 23:26:51 |
| From worker node | Hostname | wn-a4-05 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-09 20:25:46 |
| Input files | hd-protodune:np04hd_raw_run029769_0219_dataflow1_datawriter_0_20241009T195623.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-10-09 23:26:51 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
q=951892 n=787074 tag=gauss
wclsFrameSaver: saving 54644 traces tagged "wiener"
FrameSaver: q=1.04319e+06 n=719185 tag=wiener
1081 hits in TPC 0
2112 hits in TPC 1
10534 hits in TPC 2
5706 hits in TPC 3
1352 hits in TPC 4
2851 hits in TPC 5
3922 hits in TPC 6
2948 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 201 XUs and 1375 XVs -> 1 XUVs
C:0 T:2 49911 XUs and 51839 XVs -> 3231 XUVs
C:0 T:3 93 XUs and 114 XVs -> 4 XUVs
C:0 T:4 34 XUs and 45 XVs -> 2 XUVs
C:0 T:5 3046 XUs and 2828 XVs -> 120 XUVs
C:0 T:6 9925 XUs and 12354 XVs -> 515 XUVs
C:0 T:7 53 XUs and 91 XVs -> 2 XUVs
3875 XUVs total
1294 collection wire objects
3875 potential space points
Neighbour search...
186913 tests to find 99378 neighbours
Iterating with no regularization...
Begin: 6.71453e+06
0 5.52659e+06
1 5.39725e+06
2 5.37409e+06
3 5.36721e+06
4 5.36456e+06
Now with regularization...
Begin: 4.90483e+06
0 4.90195e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 39th record. run: 29769 subRun: 1 event: 17167 at 09-Oct-2024 22:25:22 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 46676 traces, 46676 summary
Retagger: tagging trace set: gauss with 31692 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 31692 traces tagged "gauss"
FrameSaver: q=753467 n=550715 tag=gauss
wclsFrameSaver: saving 46676 traces tagged "wiener"
FrameSaver: q=826961 n=492386 tag=wiener
565 hits in TPC 0
445 hits in TPC 1
4871 hits in TPC 2
2486 hits in TPC 3
3163 hits in TPC 4
4112 hits in TPC 5
3508 hits in TPC 6
3246 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 28 XUs and 6 XVs -> 0 XUVs
C:0 T:2 8198 XUs and 14433 XVs -> 855 XUVs
C:0 T:3 107 XUs and 100 XVs -> 1 XUVs
C:0 T:4 75 XUs and 49 XVs -> 2 XUVs
C:0 T:5 15120 XUs and 26292 XVs -> 2182 XUVs
C:0 T:6 16698 XUs and 86203 XVs -> 4530 XUVs
C:0 T:7 66 XUs and 34 XVs -> 2 XUVs
7572 XUVs total
1648 collection wire objects
7572 potential space points
Neighbour search...
597486 tests to find 335798 neighbours
Iterating with no regularization...
Begin: 7.90278e+06
0 6.78e+06
1 6.6721e+06
2 6.6481e+06
3 6.64023e+06
4 6.63638e+06
Now with regularization...
Begin: 5.87976e+06
0 5.87523e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
09-Oct-2024 22:26:14 BST Closed output file "np04hd_raw_run029769_0219_dataflow1_datawriter_0_20241009T195623_reco_stage1_reco_stage2_20241009T212614_keepup.root"
09-Oct-2024 22:26:14 BST Closed input file "np04hd_raw_run029769_0219_dataflow1_datawriter_0_20241009T195623_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 46.2115 64.4439 93.4598 62.6649 10.8779 39
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000432457 0.00207298 0.00372117 0.00223526 0.00087331 39
produce:fembfilter:PDHDFEMBFilter 0.524581 0.548932 0.569519 0.548402 0.0109423 39
produce:wclsdatahd:WireCellToolkit 28.9767 30.1422 32.3325 29.9584 0.73945 39
produce:gaushit:GausHitFinder 0.269151 0.486804 0.8558 0.474598 0.112431 39
produce:nhitsfilter:NumberOfHitsFilter 0.000177462 0.000305573 0.000573859 0.000301094 6.76084e-05 39
produce:reco3d:SpacePointSolver 0.0876736 0.485373 1.7841 0.357956 0.379494 39
produce:hitpdune:DisambigFromSpacePoints 0.110229 0.313604 0.761765 0.293332 0.139974 39
produce:pandora:StandardPandora 2.80581 8.44905 27.0467 7.46734 4.5434 39
produce:pandoraWriter:StandardPandora 0.0688312 0.122444 0.207603 0.113141 0.0360177 39
produce:pandoraTrack:LArPandoraTrackCreation 0.27108 0.731105 1.6078 0.677364 0.298887 39
produce:pandoraShower:LArPandoraShowerCreation 0.243929 0.877428 2.67954 0.782487 0.437178 39
produce:pandoracalo:Calorimetry 0.153541 0.374815 0.722792 0.354271 0.12816 39
produce:pandoracalonosce:Calorimetry 0.128931 0.33235 0.643088 0.311936 0.118656 39
produce:pandorapid:Chi2ParticleID 0.000940985 0.00134601 0.00306346 0.0012942 0.000361644 39
produce:pandoraShowercalo:ShowerCalorimetry 0.207879 0.784794 2.13391 0.742157 0.365877 39
produce:pandoraShowercalonosce:ShowerCalorimetry 0.202396 0.765876 1.93016 0.761037 0.346004 39
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.063494 0.107686 0.151813 0.107954 0.0223047 39
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0217616 0.0372975 0.0537998 0.0371141 0.00787633 39
produce:emtrkmichelid:EmTrackMichelId 11.0437 18.7388 28.385 18.1311 4.15273 39
[art]:TriggerResults:TriggerResultInserter 1.604e-05 2.69198e-05 7.6501e-05 2.015e-05 1.44301e-05 39
end_path:out1:RootOutput 3.62e-06 9.45831e-06 2.502e-05 8.729e-06 3.36125e-06 39
end_path:michelremoving:michelremoving 0.0106331 0.0521141 0.240088 0.0233257 0.0541578 39
end_path:michelremovinggnocchi:michelremoving 0.00992153 0.0501243 0.239241 0.0207964 0.0532791 39
end_path:calibanagnocchi:CalibAnaTree 0.0883062 0.273554 0.451815 0.274876 0.0943298 39
end_path:out1:RootOutput(write) 0.378416 0.669906 0.994665 0.656495 0.141459 39
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4947.33 MB
Peak resident set size usage (VmHWM): 2575.38 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029769_0219_dataflow1_datawriter_0_20241009T195623_reco_stage1_reco_stage2_20241009T212614_keepup.root
\tHists: np04hd_raw_run029769_0219_dataflow1_datawriter_0_20241009T195623_reco_stage1_reco_stage2_20241009T212614_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run029769_0219_dataflow1_datawriter_0_20241009T195623_reco_stage1_reco_stage2_20241009T212614_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1234, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError
Error in reco metadata 