Jobsub ID 272476.8@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272476.8@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3661 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-10-10 00:11:22 |
| Site | UK_Brunel |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_26 |
| Last heartbeat | 2024-10-10 03:24:51 |
| From worker node | Hostname | wn-a4-16 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-10 00:13:17 |
| Input files | hd-protodune:np04hd_raw_run029769_0224_dataflow0_datawriter_0_20241009T200256.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-10-10 03:24:51 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
2
3521 hits in TPC 3
4405 hits in TPC 4
6809 hits in TPC 5
1346 hits in TPC 6
969 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 22197 XUs and 17753 XVs -> 5088 XUVs
C:0 T:2 11781 XUs and 16024 XVs -> 1011 XUVs
C:0 T:3 416 XUs and 896 XVs -> 25 XUVs
C:0 T:4 216 XUs and 329 XVs -> 13 XUVs
C:0 T:5 27957 XUs and 28812 XVs -> 1327 XUVs
C:0 T:6 2169 XUs and 1428 XVs -> 93 XUVs
C:0 T:7 164 XUs and 148 XVs -> 10 XUVs
7567 XUVs total
1563 collection wire objects
7567 potential space points
Neighbour search...
824085 tests to find 483574 neighbours
Iterating with no regularization...
Begin: 8.06725e+06
0 7.42571e+06
1 7.37233e+06
2 7.3688e+06
Now with regularization...
Begin: 6.99375e+06
0 6.9928e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 39th record. run: 29769 subRun: 1 event: 17554 at 10-Oct-2024 02:23:30 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 53235 traces, 53235 summary
Retagger: tagging trace set: gauss with 40382 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 40382 traces tagged "gauss"
FrameSaver: q=872976 n=763215 tag=gauss
wclsFrameSaver: saving 53235 traces tagged "wiener"
FrameSaver: q=924395 n=684109 tag=wiener
3825 hits in TPC 0
4211 hits in TPC 1
4764 hits in TPC 2
2659 hits in TPC 3
2976 hits in TPC 4
4696 hits in TPC 5
2835 hits in TPC 6
1726 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 453 XUs and 383 XVs -> 7 XUVs
C:0 T:2 14452 XUs and 13231 XVs -> 279 XUVs
C:0 T:3 87 XUs and 102 XVs -> 2 XUVs
C:0 T:4 41 XUs and 52 XVs -> 2 XUVs
C:0 T:5 17364 XUs and 12379 XVs -> 981 XUVs
C:0 T:6 4290 XUs and 3942 XVs -> 308 XUVs
C:0 T:7 22 XUs and 61 XVs -> 1 XUVs
1580 XUVs total
677 collection wire objects
1580 potential space points
Neighbour search...
49648 tests to find 30810 neighbours
Iterating with no regularization...
Begin: 5.96583e+06
0 5.5976e+06
1 5.564e+06
2 5.55988e+06
Now with regularization...
Begin: 5.20098e+06
0 5.19963e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 2 event 17554 TrkackID 1 startx, y, z -86.3699 593.402 300.04 wireno, peakt tpcno 61 8 6 dist 0.371386min T, max_T3.65876 1999.86
no of hits closeby 2 event 17554 TrkackID 1 startx, y, z -86.3699 593.402 300.04 wireno, peakt tpcno 61 8 6 dist 0.371386min T, max_T3.65876 1999.86
10-Oct-2024 02:24:27 BST Closed output file "np04hd_raw_run029769_0224_dataflow0_datawriter_0_20241009T200256_reco_stage1_reco_stage2_20241010T012427_keepup.root"
10-Oct-2024 02:24:27 BST Closed input file "np04hd_raw_run029769_0224_dataflow0_datawriter_0_20241009T200256_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 45.6657 65.0133 84.0772 66.343 9.37044 39
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000467212 0.00202611 0.0031359 0.00214696 0.000746249 39
produce:fembfilter:PDHDFEMBFilter 0.527116 0.542415 0.565501 0.539133 0.00982655 39
produce:wclsdatahd:WireCellToolkit 29.2037 30.2176 31.6489 30.0683 0.651165 39
produce:gaushit:GausHitFinder 0.268982 0.50332 0.773011 0.523626 0.114538 39
produce:nhitsfilter:NumberOfHitsFilter 0.000189595 0.000319134 0.000541243 0.000329678 6.44812e-05 39
produce:reco3d:SpacePointSolver 0.0712182 0.44868 1.46291 0.381355 0.272604 39
produce:hitpdune:DisambigFromSpacePoints 0.0849977 0.333695 0.606164 0.324142 0.146486 39
produce:pandora:StandardPandora 2.57181 8.64488 18.8231 8.61689 3.34851 39
produce:pandoraWriter:StandardPandora 0.0542039 0.127203 0.187906 0.132429 0.0304294 39
produce:pandoraTrack:LArPandoraTrackCreation 0.136716 0.756022 1.2563 0.755614 0.246658 39
produce:pandoraShower:LArPandoraShowerCreation 0.169116 0.901844 1.77641 0.911457 0.355303 39
produce:pandoracalo:Calorimetry 0.0844499 0.393794 0.602825 0.40389 0.112784 39
produce:pandoracalonosce:Calorimetry 0.0710472 0.353138 0.563775 0.362119 0.105594 39
produce:pandorapid:Chi2ParticleID 0.000639296 0.00139274 0.00298253 0.00133743 0.000361366 39
produce:pandoraShowercalo:ShowerCalorimetry 0.131197 0.815052 1.67885 0.832413 0.326871 39
produce:pandoraShowercalonosce:ShowerCalorimetry 0.128123 0.792131 1.57998 0.812138 0.316249 39
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.042835 0.108068 0.139558 0.112057 0.0215821 39
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0151306 0.0383229 0.0499043 0.0401964 0.00768535 39
produce:emtrkmichelid:EmTrackMichelId 9.88778 18.8784 26.0147 19.2054 3.96264 39
[art]:TriggerResults:TriggerResultInserter 1.567e-05 2.53569e-05 7.9922e-05 2.022e-05 1.24157e-05 39
end_path:out1:RootOutput 4.08e-06 8.83485e-06 2.3471e-05 8.39e-06 2.755e-06 39
end_path:michelremoving:michelremoving 0.0142561 0.0451353 0.130992 0.0234329 0.0380186 39
end_path:michelremovinggnocchi:michelremoving 0.0130653 0.0431515 0.131386 0.0222707 0.0376057 39
end_path:calibanagnocchi:CalibAnaTree 0.0658857 0.294761 0.545271 0.294884 0.0908543 39
end_path:out1:RootOutput(write) 0.302633 0.670179 0.93664 0.684592 0.140445 39
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4943.7 MB
Peak resident set size usage (VmHWM): 2584.99 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029769_0224_dataflow0_datawriter_0_20241009T200256_reco_stage1_reco_stage2_20241010T012427_keepup.root
\tHists: np04hd_raw_run029769_0224_dataflow0_datawriter_0_20241009T200256_reco_stage1_reco_stage2_20241010T012427_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run029769_0224_dataflow0_datawriter_0_20241009T200256_reco_stage1_reco_stage2_20241010T012427_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1234, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError
Error in reco metadata 