Jobsub ID 272283.113@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272283.113@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3661 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-10-09 20:14:15 |
| Site | UK_Brunel |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_26 |
| Last heartbeat | 2024-10-09 23:26:51 |
| From worker node | Hostname | wn-a4-05 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-09 20:25:55 |
| Input files | hd-protodune:np04hd_raw_run029769_0222_dataflow0_datawriter_0_20241009T200026.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-10-09 23:26:51 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
0 XUs and 391 XVs -> 14 XUVs
C:0 T:2 9915 XUs and 12908 XVs -> 828 XUVs
C:0 T:3 92 XUs and 186 XVs -> 6 XUVs
C:0 T:4 70 XUs and 116 XVs -> 9 XUVs
C:0 T:5 3403 XUs and 7730 XVs -> 107 XUVs
C:0 T:6 5619 XUs and 4389 XVs -> 182 XUVs
C:0 T:7 495 XUs and 459 XVs -> 20 XUVs
1166 XUVs total
557 collection wire objects
1166 potential space points
Neighbour search...
20118 tests to find 11910 neighbours
Iterating with no regularization...
Begin: 5.9061e+06
0 5.63598e+06
1 5.61796e+06
2 5.61384e+06
Now with regularization...
Begin: 5.26993e+06
0 5.2681e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 39th record. run: 29769 subRun: 1 event: 17398 at 09-Oct-2024 22:25:29 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 39559 traces, 39559 summary
Retagger: tagging trace set: gauss with 29512 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 29512 traces tagged "gauss"
FrameSaver: q=637238 n=567514 tag=gauss
wclsFrameSaver: saving 39559 traces tagged "wiener"
FrameSaver: q=683081 n=516006 tag=wiener
2555 hits in TPC 0
1638 hits in TPC 1
3794 hits in TPC 2
2364 hits in TPC 3
2160 hits in TPC 4
2928 hits in TPC 5
3788 hits in TPC 6
1825 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 318 XUs and 419 XVs -> 3 XUVs
C:0 T:2 10013 XUs and 24048 XVs -> 1165 XUVs
C:0 T:3 176 XUs and 184 XVs -> 11 XUVs
C:0 T:4 9 XUs and 11 XVs -> 2 XUVs
C:0 T:5 5390 XUs and 6127 XVs -> 148 XUVs
C:0 T:6 21331 XUs and 8120 XVs -> 527 XUVs
C:0 T:7 658 XUs and 186 XVs -> 11 XUVs
1867 XUVs total
885 collection wire objects
1867 potential space points
Neighbour search...
74991 tests to find 43154 neighbours
Iterating with no regularization...
Begin: 4.56805e+06
0 4.18023e+06
1 4.16699e+06
2 4.16531e+06
Now with regularization...
Begin: 3.93213e+06
0 3.93156e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 10 event 17398 TrkackID 11 startx, y, z 80.5622 593.19 334.199 wireno, peakt tpcno 201 4289.57 5 dist 4.76526min T, max_T3710.33 4712.1
no of hits closeby 10 event 17398 TrkackID 11 startx, y, z 80.5622 593.19 334.199 wireno, peakt tpcno 201 4289.57 5 dist 4.76526min T, max_T3710.33 4712.1
09-Oct-2024 22:26:17 BST Closed output file "np04hd_raw_run029769_0222_dataflow0_datawriter_0_20241009T200026_reco_stage1_reco_stage2_20241009T212617_keepup.root"
09-Oct-2024 22:26:17 BST Closed input file "np04hd_raw_run029769_0222_dataflow0_datawriter_0_20241009T200026_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 47.494 63.3821 83.8481 61.5898 9.07981 39
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000392396 0.00208933 0.00464436 0.00202055 0.0011661 39
produce:fembfilter:PDHDFEMBFilter 0.526906 0.55371 0.633919 0.551578 0.0184071 39
produce:wclsdatahd:WireCellToolkit 28.7735 30.4112 32.1975 30.4065 0.729484 39
produce:gaushit:GausHitFinder 0.317477 0.501395 0.730053 0.504997 0.115936 39
produce:nhitsfilter:NumberOfHitsFilter 0.000218764 0.000314573 0.000438587 0.000308596 5.48045e-05 39
produce:reco3d:SpacePointSolver 0.116779 0.423422 2.57403 0.32437 0.427028 39
produce:hitpdune:DisambigFromSpacePoints 0.12576 0.291396 0.535019 0.284454 0.111692 39
produce:pandora:StandardPandora 3.00427 7.91538 13.6565 7.63915 2.67341 39
produce:pandoraWriter:StandardPandora 0.0797129 0.125263 0.206835 0.117721 0.0358759 39
produce:pandoraTrack:LArPandoraTrackCreation 0.275785 0.694714 1.67586 0.568274 0.293027 39
produce:pandoraShower:LArPandoraShowerCreation 0.295286 0.807249 1.97443 0.710933 0.385278 39
produce:pandoracalo:Calorimetry 0.184716 0.367841 0.774043 0.333774 0.133763 39
produce:pandoracalonosce:Calorimetry 0.145646 0.324509 0.699075 0.28998 0.126318 39
produce:pandorapid:Chi2ParticleID 0.00102267 0.00142129 0.00226688 0.00132971 0.000304102 39
produce:pandoraShowercalo:ShowerCalorimetry 0.254025 0.744962 1.81465 0.679619 0.377611 39
produce:pandoraShowercalonosce:ShowerCalorimetry 0.247034 0.723162 1.73696 0.648282 0.36563 39
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0659481 0.106443 0.163113 0.103737 0.0252095 39
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0239587 0.0364837 0.0584084 0.0360182 0.00909205 39
produce:emtrkmichelid:EmTrackMichelId 11.8863 18.2341 26.7845 17.5637 3.88411 39
[art]:TriggerResults:TriggerResultInserter 1.66e-05 3.19708e-05 7.8781e-05 2.353e-05 1.69752e-05 39
end_path:out1:RootOutput 6.62e-06 1.18732e-05 2.596e-05 1.1211e-05 4.12527e-06 39
end_path:michelremoving:michelremoving 0.0117513 0.0445406 0.125111 0.0247396 0.0371906 39
end_path:michelremovinggnocchi:michelremoving 0.0110583 0.0432139 0.145701 0.0220564 0.038903 39
end_path:calibanagnocchi:CalibAnaTree 0.122789 0.270455 0.541852 0.263749 0.103222 39
end_path:out1:RootOutput(write) 0.415414 0.664486 1.01949 0.677202 0.153813 39
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4944.17 MB
Peak resident set size usage (VmHWM): 2589.54 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029769_0222_dataflow0_datawriter_0_20241009T200026_reco_stage1_reco_stage2_20241009T212617_keepup.root
\tHists: np04hd_raw_run029769_0222_dataflow0_datawriter_0_20241009T200026_reco_stage1_reco_stage2_20241009T212617_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run029769_0222_dataflow0_datawriter_0_20241009T200026_reco_stage1_reco_stage2_20241009T212617_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1234, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError
Error in reco metadata 