Jobsub ID 263301.0@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263301.0@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3454 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:00:02 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:06:05 |
| From worker node | Hostname | cn-leo-031.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:10:10 |
| Input files | hd-protodune:np04hd_raw_run029109_0008_dataflow5_datawriter_0_20240909T163133.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:06:05 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
> 0 XUVs
637 XUVs total
504 collection wire objects
637 potential space points
Neighbour search...
4379 tests to find 2524 neighbours
Iterating with no regularization...
Begin: 1.60993e+06
0 1.55924e+06
1 1.55738e+06
2 1.55724e+06
Now with regularization...
Begin: 1.46027e+06
0 1.46023e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 2 event 2205 TrkackID 1 startx, y, z 167.519 592.326 0.985673 wireno, peakt tpcno 232 2190.73 2 dist 166.735min T, max_T159 2224.67
no of hits closeby 2 event 2205 TrkackID 1 startx, y, z 167.519 592.326 0.985673 wireno, peakt tpcno 232 2190.73 2 dist 166.735min T, max_T159 2224.67
Begin processing the 29th record. run: 29109 subRun: 1 event: 2213 at 26-Sep-2024 13:41:47 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67686 traces, 67686 summary
Retagger: tagging trace set: gauss with 48488 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48488 traces tagged "gauss"
FrameSaver: q=1.05192e+06 n=864871 tag=gauss
wclsFrameSaver: saving 67686 traces tagged "wiener"
FrameSaver: q=1.14547e+06 n=772750 tag=wiener
3105 hits in TPC 0
3027 hits in TPC 1
6406 hits in TPC 2
3346 hits in TPC 3
933 hits in TPC 4
3097 hits in TPC 5
8029 hits in TPC 6
6059 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 40 XUs and 14 XVs -> 0 XUVs
C:0 T:2 48826 XUs and 86405 XVs -> 5333 XUVs
C:0 T:3 278 XUs and 184 XVs -> 2 XUVs
C:0 T:4 140 XUs and 210 XVs -> 12 XUVs
C:0 T:5 30291 XUs and 18165 XVs -> 56 XUVs
C:0 T:6 48230 XUs and 56601 XVs -> 2924 XUVs
C:0 T:7 686 XUs and 838 XVs -> 57 XUVs
8384 XUVs total
1354 collection wire objects
8384 potential space points
Neighbour search...
1557240 tests to find 714652 neighbours
Iterating with no regularization...
Begin: 1.27794e+07
0 1.08181e+07
1 1.06736e+07
2 1.06499e+07
3 1.06426e+07
Now with regularization...
Begin: 1.00776e+07
0 1.00709e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29109 subRun: 1 event: 2221 at 26-Sep-2024 13:43:42 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 55778 traces, 55778 summary
Retagger: tagging trace set: gauss with 39118 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 39118 traces tagged "gauss"
FrameSaver: q=770905 n=647617 tag=gauss
wclsFrameSaver: saving 55778 traces tagged "wiener"
FrameSaver: q=854051 n=577082 tag=wiener
2813 hits in TPC 0
2640 hits in TPC 1
5636 hits in TPC 2
2455 hits in TPC 3
1631 hits in TPC 4
4254 hits in TPC 5
5363 hits in TPC 6
2428 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 383 XUs and 264 XVs -> 1 XUVs
C:0 T:2 71697 XUs and 78844 XVs -> 1898 XUVs
C:0 T:3 16 XUs and 0 XVs -> 0 XUVs
C:0 T:4 64 XUs and 64 XVs -> 2 XUVs
C:0 T:5 25484 XUs and 7863 XVs -> 209 XUVs
C:0 T:6 58995 XUs and 56388 XVs -> 1558 XUVs
C:0 T:7 33 XUs and 80 XVs -> 7 XUVs
3675 XUVs total
1272 collection wire objects
3675 potential space points
Neighbour search...
234619 tests to find 129702 neighbours
Iterating with no regularization...
Begin: 1.26835e+07
0 1.20898e+07
1 1.20365e+07
2 1.20293e+07
Now with regularization...
Begin: 1.18736e+07
0 1.18706e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:45:11 CEST Closed output file "np04hd_raw_run029109_0008_dataflow5_datawriter_0_20240909T163133_reco_stage1_reco_stage2_20240926T114511_keepup.root"
26-Sep-2024 13:45:11 CEST Closed input file "np04hd_raw_run029109_0008_dataflow5_datawriter_0_20240909T163133_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 85.2204 104.695 168.948 102.676 15.958 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000528533 0.00208971 0.00596342 0.00182447 0.00149938 30
produce:fembfilter:PDHDFEMBFilter 0.679322 0.733036 0.92064 0.715814 0.0500089 30
produce:wclsdatahd:WireCellToolkit 58.355 61.4638 64.8473 61.3261 1.75227 30
produce:gaushit:GausHitFinder 0.36806 0.554415 0.817806 0.551635 0.116661 30
produce:nhitsfilter:NumberOfHitsFilter 0.000380464 0.000511559 0.000723874 0.000479611 9.57811e-05 30
produce:reco3d:SpacePointSolver 0.165847 1.15745 4.72476 0.932742 0.949828 30
produce:hitpdune:DisambigFromSpacePoints 0.144222 0.335382 0.887153 0.304328 0.161473 30
produce:pandora:StandardPandora 4.5009 10.0232 45.5077 9.34441 7.09104 30
produce:pandoraWriter:StandardPandora 0.0994422 0.156868 0.235876 0.159432 0.0328026 30
produce:pandoraTrack:LArPandoraTrackCreation 0.792262 2.34502 4.9801 2.36661 0.899846 30
produce:pandoraShower:LArPandoraShowerCreation 1.02467 2.88909 7.07209 2.75991 1.34503 30
produce:pandoracalo:Calorimetry 0.421322 0.9596 2.33843 0.962808 0.377631 30
produce:pandoracalonosce:Calorimetry 0.359266 0.90189 2.20836 0.921264 0.362133 30
produce:pandorapid:Chi2ParticleID 0.00116924 0.00161637 0.0022913 0.00159314 0.00027569 30
produce:pandoraShowercalo:ShowerCalorimetry 0.83627 2.29558 6.31176 2.14871 1.14818 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.801109 2.24439 6.19579 2.10651 1.12805 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0648755 0.097757 0.146255 0.0978483 0.0196091 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0285274 0.0437214 0.0597991 0.0426213 0.00813043 30
produce:emtrkmichelid:EmTrackMichelId 11.139 16.6894 22.6692 15.7577 3.00404 30
[art]:TriggerResults:TriggerResultInserter 4.3642e-05 6.69358e-05 9.136e-05 6.62925e-05 9.31611e-06 30
end_path:out1:RootOutput 1.249e-05 2.04718e-05 3.6414e-05 1.95555e-05 5.094e-06 30
end_path:michelremoving:michelremoving 0.0239891 0.079527 0.278909 0.0588842 0.0532605 30
end_path:michelremovinggnocchi:michelremoving 0.0183833 0.0761186 0.273755 0.0534027 0.0538776 30
end_path:calibanagnocchi:CalibAnaTree 0.205028 0.666944 1.28006 0.70517 0.243651 30
end_path:out1:RootOutput(write) 0.422949 0.625816 1.04041 0.616201 0.133324 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5762.32 MB
Peak resident set size usage (VmHWM): 2599.26 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029109_0008_dataflow5_datawriter_0_20240909T163133_reco_stage1_reco_stage2_20240926T114511_keepup.root
\tHists: np04hd_raw_run029109_0008_dataflow5_datawriter_0_20240909T163133_reco_stage1_reco_stage2_20240926T114511_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
