Jobsub ID 272476.5@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272476.5@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3661 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-10-10 00:11:22 |
| Site | UK_Brunel |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_26 |
| Last heartbeat | 2024-10-10 03:24:51 |
| From worker node | Hostname | wn-a4-16 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-10 00:13:03 |
| Input files | hd-protodune:np04hd_raw_run029769_0174_dataflow1_datawriter_0_20241009T185755.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-10-10 03:24:51 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
32103 tag=gauss
wclsFrameSaver: saving 74220 traces tagged "wiener"
FrameSaver: q=1.31515e+06 n=950450 tag=wiener
2893 hits in TPC 0
4456 hits in TPC 1
7495 hits in TPC 2
3355 hits in TPC 3
5265 hits in TPC 4
6813 hits in TPC 5
6753 hits in TPC 6
3557 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 173 XUs and 471 XVs -> 4 XUVs
C:0 T:2 20393 XUs and 27237 XVs -> 1048 XUVs
C:0 T:3 132 XUs and 254 XVs -> 9 XUVs
C:0 T:4 399 XUs and 508 XVs -> 16 XUVs
C:0 T:5 35578 XUs and 51987 XVs -> 1686 XUVs
C:0 T:6 26970 XUs and 65051 XVs -> 3414 XUVs
C:0 T:7 394 XUs and 471 XVs -> 23 XUVs
6200 XUVs total
1694 collection wire objects
6200 potential space points
Neighbour search...
486574 tests to find 255550 neighbours
Iterating with no regularization...
Begin: 2.07255e+07
0 1.87941e+07
1 1.86784e+07
2 1.86589e+07
3 1.86532e+07
Now with regularization...
Begin: 1.78107e+07
0 1.78061e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 39th record. run: 29769 subRun: 1 event: 13653 at 10-Oct-2024 02:23:38 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 39476 traces, 39476 summary
Retagger: tagging trace set: gauss with 31121 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 31121 traces tagged "gauss"
FrameSaver: q=649736 n=633353 tag=gauss
wclsFrameSaver: saving 39476 traces tagged "wiener"
FrameSaver: q=683546 n=580117 tag=wiener
2328 hits in TPC 0
2027 hits in TPC 1
2735 hits in TPC 2
1525 hits in TPC 3
1700 hits in TPC 4
3549 hits in TPC 5
4921 hits in TPC 6
2660 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 169 XUs and 464 XVs -> 61 XUVs
C:0 T:2 4992 XUs and 4832 XVs -> 78 XUVs
C:0 T:3 37 XUs and 29 XVs -> 1 XUVs
C:0 T:4 75 XUs and 197 XVs -> 8 XUVs
C:0 T:5 2977 XUs and 4074 XVs -> 131 XUVs
C:0 T:6 19154 XUs and 11310 XVs -> 331 XUVs
C:0 T:7 160 XUs and 118 XVs -> 4 XUVs
614 XUVs total
465 collection wire objects
614 potential space points
Neighbour search...
6288 tests to find 3672 neighbours
Iterating with no regularization...
Begin: 2.17852e+06
0 2.10135e+06
1 2.09646e+06
2 2.09609e+06
Now with regularization...
Begin: 1.95862e+06
0 1.95845e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
10-Oct-2024 02:24:25 BST Closed output file "np04hd_raw_run029769_0174_dataflow1_datawriter_0_20241009T185755_reco_stage1_reco_stage2_20241010T012425_keepup.root"
10-Oct-2024 02:24:25 BST Closed input file "np04hd_raw_run029769_0174_dataflow1_datawriter_0_20241009T185755_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 46.2193 65.9709 106.985 65.1462 10.616 39
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000558424 0.00212836 0.00409245 0.00216934 0.000876103 39
produce:fembfilter:PDHDFEMBFilter 0.530351 0.543482 0.563694 0.543875 0.00867276 39
produce:wclsdatahd:WireCellToolkit 27.5913 29.3348 31.0134 29.205 0.811528 39
produce:gaushit:GausHitFinder 0.316797 0.526649 0.862141 0.504015 0.117829 39
produce:nhitsfilter:NumberOfHitsFilter 0.000197394 0.000323189 0.000532553 0.000317288 6.76006e-05 39
produce:reco3d:SpacePointSolver 0.0781824 0.604635 1.89317 0.430187 0.455157 39
produce:hitpdune:DisambigFromSpacePoints 0.0997234 0.346033 0.901921 0.337345 0.143043 39
produce:pandora:StandardPandora 3.30439 9.08221 23.3134 8.23661 3.80249 39
produce:pandoraWriter:StandardPandora 0.0615842 0.133173 0.241044 0.130721 0.0351433 39
produce:pandoraTrack:LArPandoraTrackCreation 0.291352 0.811162 3.61 0.72508 0.499085 39
produce:pandoraShower:LArPandoraShowerCreation 0.267236 0.95724 2.89637 0.955509 0.42628 39
produce:pandoracalo:Calorimetry 0.16558 0.402548 0.966553 0.392457 0.13488 39
produce:pandoracalonosce:Calorimetry 0.142616 0.359512 0.86415 0.346164 0.123804 39
produce:pandorapid:Chi2ParticleID 0.000970294 0.00145329 0.00292613 0.00139555 0.000344495 39
produce:pandoraShowercalo:ShowerCalorimetry 0.236312 0.874456 2.66692 0.868425 0.391958 39
produce:pandoraShowercalonosce:ShowerCalorimetry 0.229944 0.856938 2.76164 0.83333 0.399761 39
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0532261 0.112812 0.194359 0.115734 0.0264645 39
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0194677 0.040442 0.0736578 0.0406646 0.0102027 39
produce:emtrkmichelid:EmTrackMichelId 11.0215 19.7585 32.1063 19.2608 4.0348 39
[art]:TriggerResults:TriggerResultInserter 1.399e-05 2.59251e-05 7.0052e-05 2.107e-05 1.2882e-05 39
end_path:out1:RootOutput 6.59e-06 1.10343e-05 3.1901e-05 9.92e-06 4.68048e-06 39
end_path:michelremoving:michelremoving 0.0140479 0.0548939 0.270996 0.0253991 0.0591701 39
end_path:michelremovinggnocchi:michelremoving 0.0120296 0.0524394 0.267333 0.0224635 0.0591609 39
end_path:calibanagnocchi:CalibAnaTree 0.126542 0.307226 0.751907 0.288446 0.111293 39
end_path:out1:RootOutput(write) 0.396494 0.706301 1.11966 0.688677 0.139268 39
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5030.13 MB
Peak resident set size usage (VmHWM): 2655.05 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029769_0174_dataflow1_datawriter_0_20241009T185755_reco_stage1_reco_stage2_20241010T012425_keepup.root
\tHists: np04hd_raw_run029769_0174_dataflow1_datawriter_0_20241009T185755_reco_stage1_reco_stage2_20241010T012425_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run029769_0174_dataflow1_datawriter_0_20241009T185755_reco_stage1_reco_stage2_20241010T012425_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1234, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError
Error in reco metadata 