Jobsub ID 272283.92@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272283.92@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3661 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-10-09 20:14:14 |
| Site | UK_Brunel |
| Entry | CMSHTPC_T2_UK_London_Brunel_dc2_28 |
| Last heartbeat | 2024-10-09 22:35:23 |
| From worker node | Hostname | wn-a4-31 |
| cpuinfo | AMD EPYC 7513 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-09 20:24:16 |
| Input files | hd-protodune:np04hd_raw_run029769_0220_dataflow0_datawriter_0_20241009T195745.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-10-09 21:44:50 |
| Saved logs | justin-logs:272283.92-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
1 XVs -> 417 XUVs
C:0 T:7 33 XUs and 45 XVs -> 1 XUVs
3845 XUVs total
700 collection wire objects
3845 potential space points
Neighbour search...
554963 tests to find 277254 neighbours
Iterating with no regularization...
Begin: 7.21978e+06
0 6.75541e+06
1 6.71601e+06
2 6.70899e+06
3 6.70669e+06
Now with regularization...
Begin: 6.35763e+06
0 6.35618e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 39th record. run: 29769 subRun: 1 event: 17242 at 09-Oct-2024 22:13:59 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 54010 traces, 54010 summary
Retagger: tagging trace set: gauss with 40590 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 40590 traces tagged "gauss"
FrameSaver: q=886993 n=741905 tag=gauss
wclsFrameSaver: saving 54010 traces tagged "wiener"
FrameSaver: q=941861 n=667345 tag=wiener
1887 hits in TPC 0
3332 hits in TPC 1
6099 hits in TPC 2
4396 hits in TPC 3
1782 hits in TPC 4
3909 hits in TPC 5
4305 hits in TPC 6
2011 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4051 XUs and 2408 XVs -> 27 XUVs
C:0 T:2 17561 XUs and 18741 XVs -> 727 XUVs
C:0 T:3 146 XUs and 193 XVs -> 4 XUVs
C:0 T:4 49 XUs and 70 XVs -> 1 XUVs
C:0 T:5 11299 XUs and 15373 XVs -> 799 XUVs
C:0 T:6 9585 XUs and 8917 XVs -> 223 XUVs
C:0 T:7 75 XUs and 101 XVs -> 4 XUVs
1785 XUVs total
991 collection wire objects
1785 potential space points
Neighbour search...
31673 tests to find 17674 neighbours
Iterating with no regularization...
Begin: 6.48221e+06
0 6.04827e+06
1 5.99979e+06
2 5.9959e+06
Now with regularization...
Begin: 5.63626e+06
0 5.63491e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 3 event 17242 TrkackID 5 startx, y, z -25.8798 603.739 186.016 wireno, peakt tpcno 45 2799.81 6 dist 0min T, max_T2794 5854.27
no of hits closeby 0 event 17242 TrkackID 16 startx, y, z -54.8607 91.1709 351.736 wireno, peakt tpcno 219 165.5 6 dist 0min T, max_T5 680.031
no of hits closeby 3 event 17242 TrkackID 5 startx, y, z -25.8798 603.739 186.016 wireno, peakt tpcno 45 2799.81 6 dist 0min T, max_T2794 5854.27
no of hits closeby 0 event 17242 TrkackID 16 startx, y, z -54.8607 91.1709 351.736 wireno, peakt tpcno 219 165.5 6 dist 0min T, max_T5 680.031
09-Oct-2024 22:14:45 BST Closed output file "np04hd_raw_run029769_0220_dataflow0_datawriter_0_20241009T195745_reco_stage1_reco_stage2_20241009T211445_keepup.root"
09-Oct-2024 22:14:45 BST Closed input file "np04hd_raw_run029769_0220_dataflow0_datawriter_0_20241009T195745_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 36.7332 50.3092 64.8871 50.713 7.02982 39
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000312419 0.00129036 0.00301489 0.00123255 0.000715455 39
produce:fembfilter:PDHDFEMBFilter 0.406998 0.423599 0.497316 0.421402 0.0136181 39
produce:wclsdatahd:WireCellToolkit 22.5159 23.4429 24.8314 23.3795 0.541746 39
produce:gaushit:GausHitFinder 0.185632 0.392381 0.553762 0.405972 0.0883672 39
produce:nhitsfilter:NumberOfHitsFilter 0.000110308 0.000221004 0.000362113 0.000210507 5.0793e-05 39
produce:reco3d:SpacePointSolver 0.0828232 0.324545 0.818626 0.285373 0.175264 39
produce:hitpdune:DisambigFromSpacePoints 0.0631228 0.234194 0.509381 0.247728 0.0984596 39
produce:pandora:StandardPandora 2.40759 5.89949 11.2283 5.74106 2.16193 39
produce:pandoraWriter:StandardPandora 0.0383181 0.0715166 0.123708 0.0692703 0.0184636 39
produce:pandoraTrack:LArPandoraTrackCreation 0.198628 0.548798 1.2637 0.523528 0.206871 39
produce:pandoraShower:LArPandoraShowerCreation 0.193545 0.65604 1.29064 0.651694 0.245738 39
produce:pandoracalo:Calorimetry 0.123114 0.312109 0.636172 0.305161 0.103859 39
produce:pandoracalonosce:Calorimetry 0.109043 0.282934 0.598976 0.278482 0.0981388 39
produce:pandorapid:Chi2ParticleID 0.000713134 0.00112272 0.00249498 0.00115384 0.000298206 39
produce:pandoraShowercalo:ShowerCalorimetry 0.162373 0.642871 1.26453 0.635385 0.250928 39
produce:pandoraShowercalonosce:ShowerCalorimetry 0.159002 0.625002 1.21517 0.617433 0.242551 39
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0384301 0.0750907 0.118107 0.0751188 0.0176845 39
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0139106 0.025634 0.0384148 0.0265535 0.00560442 39
produce:emtrkmichelid:EmTrackMichelId 8.99009 15.4578 21.9301 15.5992 3.1637 39
[art]:TriggerResults:TriggerResultInserter 1.1371e-05 2.05133e-05 6.7647e-05 1.8616e-05 8.94194e-06 39
end_path:out1:RootOutput 3.316e-06 1.03605e-05 2.4987e-05 9.368e-06 4.02407e-06 39
end_path:michelremoving:michelremoving 0.00776118 0.046689 0.172667 0.0172952 0.0473695 39
end_path:michelremovinggnocchi:michelremoving 0.00681147 0.0444332 0.170206 0.0149133 0.0469815 39
end_path:calibanagnocchi:CalibAnaTree 0.0681336 0.24153 0.529652 0.23381 0.0919338 39
end_path:out1:RootOutput(write) 0.24986 0.49126 0.709439 0.512813 0.111808 39
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4947.28 MB
Peak resident set size usage (VmHWM): 2565.76 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029769_0220_dataflow0_datawriter_0_20241009T195745_reco_stage1_reco_stage2_20241009T211445_keepup.root
\tHists: np04hd_raw_run029769_0220_dataflow0_datawriter_0_20241009T195745_reco_stage1_reco_stage2_20241009T211445_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run029769_0220_dataflow0_datawriter_0_20241009T195745_reco_stage1_reco_stage2_20241009T211445_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/71f643ddd59465043e3cd3712a1e24b9cd0fa631/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1234, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_42_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError
Error in reco metadata 