Jobsub ID 263865.21@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263865.21@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3448 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 13:56:43 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 16:26:45 |
| From worker node | Hostname | wn-204-11-05-02-a |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 13:57:49 |
| Input files | hd-protodune:np04hd_raw_run029097_0038_dataflow2_datawriter_0_20240906T110849.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 16:26:45 |
| Saved logs | justin-logs:263865.21-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
3828 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 600 XUs and 149 XVs -> 4 XUVs
C:0 T:2 47813 XUs and 47723 XVs -> 3584 XUVs
C:0 T:3 371 XUs and 433 XVs -> 32 XUVs
C:0 T:4 54 XUs and 55 XVs -> 0 XUVs
C:0 T:5 10349 XUs and 12309 XVs -> 1171 XUVs
C:0 T:6 42528 XUs and 46664 XVs -> 3387 XUVs
C:0 T:7 251 XUs and 524 XVs -> 24 XUVs
8202 XUVs total
2525 collection wire objects
8202 potential space points
Neighbour search...
1083070 tests to find 409876 neighbours
Iterating with no regularization...
Begin: 7.22876e+07
0 5.7473e+07
1 5.62409e+07
2 5.61426e+07
3 5.6125e+07
Now with regularization...
Begin: 5.12216e+07
0 5.11708e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29097 subRun: 1 event: 9346 at 26-Sep-2024 18:18:01 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 65988 traces, 65988 summary
Retagger: tagging trace set: gauss with 48872 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48872 traces tagged "gauss"
FrameSaver: q=986174 n=887972 tag=gauss
wclsFrameSaver: saving 65988 traces tagged "wiener"
FrameSaver: q=1.06281e+06 n=804756 tag=wiener
3516 hits in TPC 0
4428 hits in TPC 1
5894 hits in TPC 2
3958 hits in TPC 3
3008 hits in TPC 4
5781 hits in TPC 5
4674 hits in TPC 6
2023 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 218 XUs and 133 XVs -> 6 XUVs
C:0 T:2 13891 XUs and 18344 XVs -> 1911 XUVs
C:0 T:3 539 XUs and 541 XVs -> 29 XUVs
C:0 T:4 128 XUs and 93 XVs -> 5 XUVs
C:0 T:5 39053 XUs and 64266 XVs -> 1175 XUVs
C:0 T:6 8590 XUs and 12557 XVs -> 333 XUVs
C:0 T:7 177 XUs and 266 XVs -> 4 XUVs
3463 XUVs total
1432 collection wire objects
3463 potential space points
Neighbour search...
129923 tests to find 79142 neighbours
Iterating with no regularization...
Begin: 4.73303e+06
0 4.26247e+06
1 4.21944e+06
2 4.21104e+06
3 4.20794e+06
Now with regularization...
Begin: 3.80029e+06
0 3.79869e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29097 subRun: 1 event: 9354 at 26-Sep-2024 18:21:37 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 78311 traces, 78311 summary
Retagger: tagging trace set: gauss with 57687 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 57687 traces tagged "gauss"
FrameSaver: q=1.32599e+06 n=1111739 tag=gauss
wclsFrameSaver: saving 78311 traces tagged "wiener"
FrameSaver: q=1.46144e+06 n=1013918 tag=wiener
4946 hits in TPC 0
4677 hits in TPC 1
5272 hits in TPC 2
3753 hits in TPC 3
5900 hits in TPC 4
8221 hits in TPC 5
5951 hits in TPC 6
4030 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 830 XUs and 653 XVs -> 9 XUVs
C:0 T:2 12062 XUs and 17841 XVs -> 471 XUVs
C:0 T:3 44 XUs and 62 XVs -> 1 XUVs
C:0 T:4 102 XUs and 146 XVs -> 1 XUVs
C:0 T:5 25622 XUs and 124889 XVs -> 3503 XUVs
C:0 T:6 23815 XUs and 29624 XVs -> 1735 XUVs
C:0 T:7 1116 XUs and 2598 XVs -> 69 XUVs
5789 XUVs total
1935 collection wire objects
5789 potential space points
Neighbour search...
816337 tests to find 351876 neighbours
Iterating with no regularization...
Begin: 2.13216e+07
0 1.80435e+07
1 1.78975e+07
2 1.78862e+07
Now with regularization...
Begin: 1.70284e+07
0 1.70243e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 18:26:11 CEST Closed output file "np04hd_raw_run029097_0038_dataflow2_datawriter_0_20240906T110849_reco_stage1_reco_stage2_20240926T162611_keepup.root"
26-Sep-2024 18:26:11 CEST Closed input file "np04hd_raw_run029097_0038_dataflow2_datawriter_0_20240906T110849_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 157.363 222.206 355.274 217.289 41.1404 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000730648 0.00380325 0.00644265 0.00371083 0.00157314 30
produce:fembfilter:PDHDFEMBFilter 1.25091 1.37726 1.5692 1.36174 0.0632572 30
produce:wclsdatahd:WireCellToolkit 87.8097 95.2448 106.283 95.2087 3.07321 30
produce:gaushit:GausHitFinder 0.920089 1.57198 2.54058 1.56774 0.374201 30
produce:nhitsfilter:NumberOfHitsFilter 0.000418547 0.000743087 0.00217244 0.000674047 0.000320103 30
produce:reco3d:SpacePointSolver 0.27189 2.47668 7.01763 2.39123 1.64256 30
produce:hitpdune:DisambigFromSpacePoints 0.418411 1.5226 4.4054 1.35957 0.84399 30
produce:pandora:StandardPandora 13.6333 28.7149 59.6114 25.8144 12.3869 30
produce:pandoraWriter:StandardPandora 0.207094 0.318796 0.584634 0.304309 0.0785913 30
produce:pandoraTrack:LArPandoraTrackCreation 2.24059 4.53166 11.6713 4.17053 1.75225 30
produce:pandoraShower:LArPandoraShowerCreation 1.81988 5.44716 15.506 5.19928 2.59561 30
produce:pandoracalo:Calorimetry 0.970266 2.07131 4.92065 1.96593 0.726543 30
produce:pandoracalonosce:Calorimetry 0.865351 1.90121 4.54828 1.71972 0.690857 30
produce:pandorapid:Chi2ParticleID 0.00297124 0.00411961 0.00660097 0.00393952 0.000758284 30
produce:pandoraShowercalo:ShowerCalorimetry 1.55613 4.8882 15.5811 4.36497 2.6754 30
produce:pandoraShowercalonosce:ShowerCalorimetry 1.39786 4.66097 14.6087 4.34959 2.47739 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.203957 0.313371 0.617986 0.298645 0.0822064 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0723889 0.120506 0.22979 0.113579 0.0290784 30
produce:emtrkmichelid:EmTrackMichelId 35.4671 63.1897 109.13 63.5752 15.4564 30
[art]:TriggerResults:TriggerResultInserter 6.5645e-05 9.21675e-05 0.000184548 9.12545e-05 1.96666e-05 30
end_path:out1:RootOutput 1.6897e-05 2.48764e-05 6.4477e-05 2.21235e-05 1.01963e-05 30
end_path:michelremoving:michelremoving 0.0489767 0.156936 0.533895 0.0967626 0.120359 30
end_path:michelremovinggnocchi:michelremoving 0.0471514 0.149916 0.509988 0.0904354 0.120138 30
end_path:calibanagnocchi:CalibAnaTree 0.644292 1.63088 3.50489 1.46838 0.619311 30
end_path:out1:RootOutput(write) 1.07896 1.66463 2.83903 1.63803 0.385712 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4940.86 MB
Peak resident set size usage (VmHWM): 2146.45 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029097_0038_dataflow2_datawriter_0_20240906T110849_reco_stage1_reco_stage2_20240926T162611_keepup.root
\tHists: np04hd_raw_run029097_0038_dataflow2_datawriter_0_20240906T110849_reco_stage1_reco_stage2_20240926T162611_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
