Jobsub ID 263854.5@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263854.5@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3454 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 13:52:08 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 16:24:49 |
| From worker node | Hostname | wn-204-13-01-02-a.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 13:53:37 |
| Input files | hd-protodune:np04hd_raw_run029108_0293_dataflow3_datawriter_0_20240909T152030.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 16:24:49 |
| Saved logs | justin-logs:263854.5-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
hits in TPC 6
1713 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 18169 XUs and 28213 XVs -> 2506 XUVs
C:0 T:2 19935 XUs and 22272 XVs -> 808 XUVs
C:0 T:3 200 XUs and 154 XVs -> 7 XUVs
C:0 T:4 180 XUs and 142 XVs -> 10 XUVs
C:0 T:5 27535 XUs and 22977 XVs -> 1047 XUVs
C:0 T:6 4152 XUs and 17326 XVs -> 352 XUVs
C:0 T:7 46 XUs and 50 XVs -> 3 XUVs
4733 XUVs total
1440 collection wire objects
4733 potential space points
Neighbour search...
342863 tests to find 196838 neighbours
Iterating with no regularization...
Begin: 7.72454e+06
0 7.27447e+06
1 7.22947e+06
2 7.22578e+06
Now with regularization...
Begin: 6.86912e+06
0 6.86837e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29108 subRun: 1 event: 70587 at 26-Sep-2024 18:14:01 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 94728 traces, 94728 summary
Retagger: tagging trace set: gauss with 69721 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69721 traces tagged "gauss"
FrameSaver: q=1.55673e+06 n=1298068 tag=gauss
wclsFrameSaver: saving 94728 traces tagged "wiener"
FrameSaver: q=1.70462e+06 n=1177229 tag=wiener
3683 hits in TPC 0
3611 hits in TPC 1
9467 hits in TPC 2
5345 hits in TPC 3
4445 hits in TPC 4
5989 hits in TPC 5
13430 hits in TPC 6
6034 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 5600 XUs and 6716 XVs -> 1917 XUVs
C:0 T:2 41458 XUs and 72243 XVs -> 1310 XUVs
C:0 T:3 178 XUs and 145 XVs -> 9 XUVs
C:0 T:4 141 XUs and 144 XVs -> 0 XUVs
C:0 T:5 30213 XUs and 32837 XVs -> 3237 XUVs
C:0 T:6 105556 XUs and 123919 XVs -> 13956 XUVs
C:0 T:7 441 XUs and 752 XVs -> 12 XUVs
20441 XUVs total
2912 collection wire objects
20441 potential space points
Neighbour search...
3267435 tests to find 1548122 neighbours
Iterating with no regularization...
Begin: 1.65299e+07
0 1.43448e+07
1 1.41275e+07
2 1.40869e+07
3 1.40733e+07
Now with regularization...
Begin: 1.3216e+07
0 1.32061e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 2 event 70587 TrkackID 17 startx, y, z 97.4637 596.845 229.08 wireno, peakt tpcno 389 1439.34 2 dist 287.625min T, max_T6.81242 1502.38
no of hits closeby 2 event 70587 TrkackID 17 startx, y, z 97.4637 596.845 229.08 wireno, peakt tpcno 389 1439.34 2 dist 287.625min T, max_T6.81242 1502.38
Begin processing the 30th record. run: 29108 subRun: 1 event: 70595 at 26-Sep-2024 18:19:42 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 73609 traces, 73609 summary
Retagger: tagging trace set: gauss with 53184 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53184 traces tagged "gauss"
FrameSaver: q=1.12897e+06 n=965740 tag=gauss
wclsFrameSaver: saving 73609 traces tagged "wiener"
FrameSaver: q=1.22228e+06 n=863674 tag=wiener
1842 hits in TPC 0
2808 hits in TPC 1
8670 hits in TPC 2
4161 hits in TPC 3
2946 hits in TPC 4
5353 hits in TPC 5
8250 hits in TPC 6
4155 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 120 XUs and 213 XVs -> 0 XUVs
C:0 T:2 67557 XUs and 119587 XVs -> 6371 XUVs
C:0 T:3 3121 XUs and 3143 XVs -> 96 XUVs
C:0 T:4 35 XUs and 22 XVs -> 1 XUVs
C:0 T:5 18690 XUs and 9144 XVs -> 323 XUVs
C:0 T:6 50242 XUs and 77524 XVs -> 2459 XUVs
C:0 T:7 966 XUs and 1346 XVs -> 24 XUVs
9274 XUVs total
1852 collection wire objects
9274 potential space points
Neighbour search...
1087554 tests to find 516192 neighbours
Iterating with no regularization...
Begin: 9.95218e+06
0 8.52258e+06
1 8.40484e+06
2 8.38241e+06
3 8.37541e+06
Now with regularization...
Begin: 7.95289e+06
0 7.94836e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 18:24:14 CEST Closed output file "np04hd_raw_run029108_0293_dataflow3_datawriter_0_20240909T152030_reco_stage1_reco_stage2_20240926T162414_keepup.root"
26-Sep-2024 18:24:14 CEST Closed input file "np04hd_raw_run029108_0293_dataflow3_datawriter_0_20240909T152030_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 131.375 227.307 340.852 220.635 44.648 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000670926 0.00415604 0.00679002 0.0043862 0.00143251 30
produce:fembfilter:PDHDFEMBFilter 1.2831 1.3566 1.48807 1.34756 0.0410158 30
produce:wclsdatahd:WireCellToolkit 86.9566 93.0383 98.4971 92.9817 2.19064 30
produce:gaushit:GausHitFinder 0.656669 1.57035 2.28624 1.58286 0.365949 30
produce:nhitsfilter:NumberOfHitsFilter 0.000322581 0.000648137 0.00185967 0.000599733 0.000263551 30
produce:reco3d:SpacePointSolver 0.17664 2.52454 8.30507 1.70063 1.9641 30
produce:hitpdune:DisambigFromSpacePoints 0.279905 1.5935 3.98188 1.52946 0.801905 30
produce:pandora:StandardPandora 6.49941 32.5883 76.5544 30.4224 15.9388 30
produce:pandoraWriter:StandardPandora 0.148568 0.320914 0.488859 0.317387 0.0748609 30
produce:pandoraTrack:LArPandoraTrackCreation 1.06056 4.24667 9.70663 4.4004 1.74803 30
produce:pandoraShower:LArPandoraShowerCreation 0.841896 5.51885 13.6849 5.28504 2.63475 30
produce:pandoracalo:Calorimetry 0.452445 2.11292 5.56805 1.94112 1.09256 30
produce:pandoracalonosce:Calorimetry 0.407106 1.88421 4.50323 1.90022 0.781064 30
produce:pandorapid:Chi2ParticleID 0.00190177 0.00397197 0.00569302 0.00391977 0.000844018 30
produce:pandoraShowercalo:ShowerCalorimetry 0.741891 4.96197 12.861 4.49742 2.66015 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.745501 4.62187 13.4182 4.26136 2.51494 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.156707 0.331023 0.519242 0.322579 0.0824455 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0611413 0.132176 0.208711 0.127276 0.0329123 30
produce:emtrkmichelid:EmTrackMichelId 28.9264 66.6712 97.0632 64.9264 14.8885 30
[art]:TriggerResults:TriggerResultInserter 5.581e-05 8.45955e-05 0.000149037 8.24325e-05 1.88785e-05 30
end_path:out1:RootOutput 1.0964e-05 1.93452e-05 5.6329e-05 1.73915e-05 8.74833e-06 30
end_path:michelremoving:michelremoving 0.0527362 0.159329 0.538166 0.101248 0.138417 30
end_path:michelremovinggnocchi:michelremoving 0.0506154 0.154807 0.533059 0.0941558 0.142184 30
end_path:calibanagnocchi:CalibAnaTree 0.523577 1.54711 3.60936 1.54247 0.674856 30
end_path:out1:RootOutput(write) 0.869794 1.75296 2.68789 1.74787 0.419703 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4943.07 MB
Peak resident set size usage (VmHWM): 2144.58 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029108_0293_dataflow3_datawriter_0_20240909T152030_reco_stage1_reco_stage2_20240926T162414_keepup.root
\tHists: np04hd_raw_run029108_0293_dataflow3_datawriter_0_20240909T152030_reco_stage1_reco_stage2_20240926T162414_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
