Jobsub ID 78491.71@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78491.71@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:11:15 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:19:04 |
| From worker node | Hostname | cn-leo-182.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:43:31 |
| Input files | hd-protodune:np04hd_raw_run029092_0337_dataflow3_datawriter_0_20240905T222319.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:19:04 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
nnels
Finding XUV coincidences...
C:0 T:1 66 XUs and 48 XVs -> 2 XUVs
C:0 T:2 29248 XUs and 44862 XVs -> 3631 XUVs
C:0 T:4 266 XUs and 371 XVs -> 10 XUVs
C:0 T:5 19260 XUs and 17311 XVs -> 696 XUVs
C:0 T:6 25981 XUs and 13995 XVs -> 457 XUVs
C:0 T:7 51 XUs and 34 XVs -> 1 XUVs
4797 XUVs total
1464 collection wire objects
4797 potential space points
Neighbour search...
268911 tests to find 165100 neighbours
Iterating with no regularization...
Begin: 1.51285e+07
0 1.37993e+07
1 1.36803e+07
2 1.36588e+07
3 1.3653e+07
Now with regularization...
Begin: 1.26737e+07
0 1.26688e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29092 subRun: 1 event: 81107 at 26-Sep-2024 13:51:53 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67079 traces, 67079 summary
Retagger: tagging trace set: gauss with 51291 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 51291 traces tagged "gauss"
FrameSaver: q=1.27222e+06 n=1036803 tag=gauss
wclsFrameSaver: saving 67079 traces tagged "wiener"
FrameSaver: q=1.37133e+06 n=949600 tag=wiener
2149 hits in TPC 0
3969 hits in TPC 1
7671 hits in TPC 2
3158 hits in TPC 3
6103 hits in TPC 4
7376 hits in TPC 5
3633 hits in TPC 6
4021 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 37 XUs and 15 XVs -> 0 XUVs
C:0 T:2 26570 XUs and 32198 XVs -> 1118 XUVs
C:0 T:3 1557 XUs and 1114 XVs -> 47 XUVs
C:0 T:4 72 XUs and 75 XVs -> 4 XUVs
C:0 T:5 26472 XUs and 23098 XVs -> 1119 XUVs
C:0 T:6 9157 XUs and 13195 XVs -> 1123 XUVs
C:0 T:7 406 XUs and 335 XVs -> 5 XUVs
3416 XUVs total
1550 collection wire objects
3416 potential space points
Neighbour search...
145232 tests to find 70888 neighbours
Iterating with no regularization...
Begin: 1.15573e+08
0 8.64075e+07
1 7.60136e+07
2 7.59552e+07
Now with regularization...
Begin: 7.44897e+07
0 7.44626e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0136321 0.0254461 0.015924 0.0134625 0.0152224 0.0150463 0.0143475 0.0124105 0.0219794 0.0149731 0.012032
----view 1: 0.0008834 0.000538707 0.000538647 0.000538707 0.000538647 0.000538707 0.000538647 0.000538707 0.000538647 0.000538707 0.000538647
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29092 subRun: 1 event: 81115 at 26-Sep-2024 13:53:59 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64536 traces, 64536 summary
Retagger: tagging trace set: gauss with 47239 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 47239 traces tagged "gauss"
FrameSaver: q=1.05101e+06 n=849822 tag=gauss
wclsFrameSaver: saving 64536 traces tagged "wiener"
FrameSaver: q=1.13384e+06 n=765313 tag=wiener
3677 hits in TPC 0
2203 hits in TPC 1
5024 hits in TPC 2
3405 hits in TPC 3
3367 hits in TPC 4
3496 hits in TPC 5
7088 hits in TPC 6
4480 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 787 XUs and 581 XVs -> 2 XUVs
C:0 T:2 19558 XUs and 9356 XVs -> 664 XUVs
C:0 T:3 79 XUs and 74 XVs -> 6 XUVs
C:0 T:4 54 XUs and 58 XVs -> 2 XUVs
C:0 T:5 17020 XUs and 16323 XVs -> 1011 XUVs
C:0 T:6 50066 XUs and 30125 XVs -> 1219 XUVs
C:0 T:7 169 XUs and 326 XVs -> 13 XUVs
2917 XUVs total
1071 collection wire objects
2917 potential space points
Neighbour search...
127717 tests to find 77658 neighbours
Iterating with no regularization...
Begin: 9.0417e+06
0 8.01417e+06
1 7.93709e+06
2 7.92659e+06
3 7.92463e+06
Now with regularization...
Begin: 7.51376e+06
0 7.51235e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:55:37 CEST Closed output file "np04hd_raw_run029092_0337_dataflow3_datawriter_0_20240905T222319_reco_stage1_reco_stage2_20240926T115537_keepup.root"
26-Sep-2024 13:55:37 CEST Closed input file "np04hd_raw_run029092_0337_dataflow3_datawriter_0_20240905T222319_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 81.7648 104.044 144.965 99.9702 13.9335 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000470086 0.00275622 0.00826907 0.00198332 0.00204223 30
produce:fembfilter:PDHDFEMBFilter 0.686249 0.715025 0.746915 0.713627 0.0168888 30
produce:wclsdatahd:WireCellToolkit 56.9254 61.1991 65.3929 61.3182 1.98399 30
produce:gaushit:GausHitFinder 0.37041 0.560384 0.822877 0.555568 0.118362 30
produce:nhitsfilter:NumberOfHitsFilter 0.000352357 0.000515186 0.000746295 0.000479782 9.53101e-05 30
produce:reco3d:SpacePointSolver 0.309071 0.857117 2.83044 0.664485 0.600736 30
produce:hitpdune:DisambigFromSpacePoints 0.139672 0.351434 0.839602 0.320137 0.156676 30
produce:pandora:StandardPandora 3.71533 9.62671 18.5477 8.85037 3.36718 30
produce:pandoraWriter:StandardPandora 0.0962286 0.156534 0.246377 0.150728 0.0357248 30
produce:pandoraTrack:LArPandoraTrackCreation 0.698043 2.32923 5.26752 2.18362 0.901814 30
produce:pandoraShower:LArPandoraShowerCreation 0.770649 3.03233 7.95662 2.61805 1.55457 30
produce:pandoracalo:Calorimetry 0.293904 0.95369 2.30614 0.896223 0.380346 30
produce:pandoracalonosce:Calorimetry 0.266 0.895743 2.11235 0.834521 0.364988 30
produce:pandorapid:Chi2ParticleID 0.000929337 0.00157515 0.00273446 0.00150019 0.000343583 30
produce:pandoraShowercalo:ShowerCalorimetry 0.551334 2.40822 6.37694 2.08246 1.29812 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.533849 2.30536 5.57662 2.08741 1.17139 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0553975 0.0998842 0.153346 0.0938713 0.0227995 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0232569 0.0449505 0.072371 0.0449082 0.0105353 30
produce:emtrkmichelid:EmTrackMichelId 10.8245 16.6981 25.8578 16.1459 3.52549 30
[art]:TriggerResults:TriggerResultInserter 4.624e-05 6.44303e-05 0.000101433 6.3413e-05 1.32818e-05 30
end_path:out1:RootOutput 1.1681e-05 1.72216e-05 2.8127e-05 1.63365e-05 4.39817e-06 30
end_path:michelremoving:michelremoving 0.0274925 0.0726629 0.18367 0.0559479 0.0380933 30
end_path:michelremovinggnocchi:michelremoving 0.0242929 0.0692541 0.176745 0.0512325 0.0377817 30
end_path:calibanagnocchi:CalibAnaTree 0.252789 0.684823 1.7942 0.639101 0.300133 30
end_path:out1:RootOutput(write) 0.372772 0.626861 1.02406 0.599176 0.15007 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5795.81 MB
Peak resident set size usage (VmHWM): 2648.29 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029092_0337_dataflow3_datawriter_0_20240905T222319_reco_stage1_reco_stage2_20240926T115537_keepup.root
\tHists: np04hd_raw_run029092_0337_dataflow3_datawriter_0_20240905T222319_reco_stage1_reco_stage2_20240926T115537_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
