Jobsub ID 272485.3@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272485.3@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3661 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-10-10 00:16:42 |
| Site | UK_Brunel |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_26 |
| Last heartbeat | 2024-10-10 03:22:38 |
| From worker node | Hostname | wn-a4-05 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-10 00:17:56 |
| Input files | hd-protodune:np04hd_raw_run029769_0224_dataflow1_datawriter_0_20241009T200252.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-10-10 03:22:38 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
0 XUVs
C:0 T:2 39562 XUs and 29703 XVs -> 1063 XUVs
C:0 T:3 585 XUs and 359 XVs -> 30 XUVs
C:0 T:4 16 XUs and 39 XVs -> 0 XUVs
C:0 T:5 7038 XUs and 18456 XVs -> 2021 XUVs
C:0 T:6 67435 XUs and 46004 XVs -> 1648 XUVs
C:0 T:7 178 XUs and 163 XVs -> 4 XUVs
4916 XUVs total
1372 collection wire objects
4916 potential space points
Neighbour search...
297584 tests to find 172532 neighbours
Iterating with no regularization...
Begin: 5.57349e+06
0 4.82364e+06
1 4.74827e+06
2 4.73754e+06
3 4.73405e+06
Now with regularization...
Begin: 4.47226e+06
0 4.47e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 39th record. run: 29769 subRun: 1 event: 17553 at 10-Oct-2024 02:21:19 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 50413 traces, 50413 summary
Retagger: tagging trace set: gauss with 37633 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 37633 traces tagged "gauss"
FrameSaver: q=796215 n=692184 tag=gauss
wclsFrameSaver: saving 50413 traces tagged "wiener"
FrameSaver: q=863943 n=626848 tag=wiener
1856 hits in TPC 0
2178 hits in TPC 1
4746 hits in TPC 2
2584 hits in TPC 3
1496 hits in TPC 4
2919 hits in TPC 5
5326 hits in TPC 6
4859 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 968 XUs and 1043 XVs -> 183 XUVs
C:0 T:2 11702 XUs and 8685 XVs -> 155 XUVs
C:0 T:3 206 XUs and 196 XVs -> 5 XUVs
C:0 T:4 33 XUs and 47 XVs -> 3 XUVs
C:0 T:5 3701 XUs and 4086 XVs -> 275 XUVs
C:0 T:6 35387 XUs and 59029 XVs -> 3016 XUVs
C:0 T:7 299 XUs and 478 XVs -> 31 XUVs
3668 XUVs total
980 collection wire objects
3668 potential space points
Neighbour search...
197530 tests to find 106586 neighbours
Iterating with no regularization...
Begin: 7.30381e+06
0 4.77089e+06
1 4.62496e+06
2 4.60072e+06
3 4.59242e+06
4 4.58959e+06
Now with regularization...
Begin: 4.29147e+06
0 4.28925e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 17553 TrkackID 4 startx, y, z 63.6427 454.943 7.80465 wireno, peakt tpcno 168 1314.44 2 dist 187.096min T, max_T6.12563 1352.22
no of hits closeby 0 event 17553 TrkackID 4 startx, y, z 63.6427 454.943 7.80465 wireno, peakt tpcno 168 1314.44 2 dist 187.096min T, max_T6.12563 1352.22
10-Oct-2024 02:22:12 BST Closed output file "np04hd_raw_run029769_0224_dataflow1_datawriter_0_20241009T200252_reco_stage1_reco_stage2_20241010T012212_keepup.root"
10-Oct-2024 02:22:12 BST Closed input file "np04hd_raw_run029769_0224_dataflow1_datawriter_0_20241009T200252_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 46.0679 62.6832 91.0361 62.9549 8.74566 39
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000553538 0.00169081 0.00329119 0.00163091 0.000689367 39
produce:fembfilter:PDHDFEMBFilter 0.51906 0.537039 0.565727 0.536391 0.0116585 39
produce:wclsdatahd:WireCellToolkit 26.8602 28.1992 29.6559 28.0862 0.723113 39
produce:gaushit:GausHitFinder 0.282278 0.49428 0.759185 0.508789 0.095289 39
produce:nhitsfilter:NumberOfHitsFilter 0.000196373 0.000344081 0.000995606 0.000321275 0.000160061 39
produce:reco3d:SpacePointSolver 0.157656 0.583145 1.80931 0.470441 0.405717 39
produce:hitpdune:DisambigFromSpacePoints 0.127229 0.346662 0.680238 0.339527 0.126091 39
produce:pandora:StandardPandora 3.82306 8.29359 17.7556 8.30021 3.10746 39
produce:pandoraWriter:StandardPandora 0.0634344 0.116776 0.195389 0.11707 0.0266611 39
produce:pandoraTrack:LArPandoraTrackCreation 0.276714 0.781056 1.63348 0.732754 0.304141 39
produce:pandoraShower:LArPandoraShowerCreation 0.291377 0.926365 2.55864 0.88053 0.420264 39
produce:pandoracalo:Calorimetry 0.158344 0.415116 0.822341 0.388589 0.138327 39
produce:pandoracalonosce:Calorimetry 0.145347 0.373134 0.766268 0.356413 0.129221 39
produce:pandorapid:Chi2ParticleID 0.000923055 0.00140439 0.00304446 0.0013216 0.000347222 39
produce:pandoraShowercalo:ShowerCalorimetry 0.267827 0.846386 2.03848 0.843656 0.358906 39
produce:pandoraShowercalonosce:ShowerCalorimetry 0.260515 0.815811 1.89131 0.804907 0.338871 39
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0580798 0.110377 0.167655 0.111035 0.0224178 39
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0199191 0.0376336 0.055239 0.0383584 0.00741399 39
produce:emtrkmichelid:EmTrackMichelId 11.723 18.6176 29.8332 18.7559 3.33846 39
[art]:TriggerResults:TriggerResultInserter 1.598e-05 2.38451e-05 6.8051e-05 2.0781e-05 1.07281e-05 39
end_path:out1:RootOutput 9.081e-06 1.11603e-05 2.299e-05 1.067e-05 2.30878e-06 39
end_path:michelremoving:michelremoving 0.010945 0.0657901 0.252596 0.0261989 0.0607756 39
end_path:michelremovinggnocchi:michelremoving 0.00996622 0.0632838 0.249347 0.0224035 0.0603308 39
end_path:calibanagnocchi:CalibAnaTree 0.100065 0.312409 0.708998 0.291432 0.114603 39
end_path:out1:RootOutput(write) 0.384096 0.661197 0.965026 0.671277 0.122507 39
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5013.09 MB
Peak resident set size usage (VmHWM): 2644.25 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029769_0224_dataflow1_datawriter_0_20241009T200252_reco_stage1_reco_stage2_20241010T012212_keepup.root
\tHists: np04hd_raw_run029769_0224_dataflow1_datawriter_0_20241009T200252_reco_stage1_reco_stage2_20241010T012212_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run029769_0224_dataflow1_datawriter_0_20241009T200252_reco_stage1_reco_stage2_20241010T012212_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1234, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError
Error in reco metadata 