Jobsub ID 263327.89@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263327.89@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:08:48 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:16:52 |
| From worker node | Hostname | cn-leo-157.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:40:56 |
| Input files | hd-protodune:np04hd_raw_run029081_1086_dataflow5_datawriter_0_20240905T102525.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:16:52 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
---------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 260869 at 26-Sep-2024 13:46:17 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64851 traces, 64851 summary
Retagger: tagging trace set: gauss with 50744 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 50744 traces tagged "gauss"
FrameSaver: q=1.20561e+06 n=1036705 tag=gauss
wclsFrameSaver: saving 64851 traces tagged "wiener"
FrameSaver: q=1.2638e+06 n=949548 tag=wiener
3249 hits in TPC 0
3482 hits in TPC 1
3815 hits in TPC 2
2425 hits in TPC 3
4713 hits in TPC 4
6969 hits in TPC 5
6926 hits in TPC 6
4169 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 222 XUs and 239 XVs -> 4 XUVs
C:0 T:2 5315 XUs and 6986 XVs -> 224 XUVs
C:0 T:3 521 XUs and 636 XVs -> 24 XUVs
C:0 T:4 19 XUs and 23 XVs -> 0 XUVs
C:0 T:5 18680 XUs and 13223 XVs -> 377 XUVs
C:0 T:6 17451 XUs and 19890 XVs -> 628 XUVs
C:0 T:7 16 XUs and 31 XVs -> 0 XUVs
1257 XUVs total
999 collection wire objects
1257 potential space points
Neighbour search...
14529 tests to find 7096 neighbours
Iterating with no regularization...
Begin: 5.8742e+06
0 5.60357e+06
1 5.59856e+06
Now with regularization...
Begin: 5.26918e+06
0 5.26861e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 260885 at 26-Sep-2024 13:48:02 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 63339 traces, 63339 summary
Retagger: tagging trace set: gauss with 45546 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 45546 traces tagged "gauss"
FrameSaver: q=981454 n=824474 tag=gauss
wclsFrameSaver: saving 63339 traces tagged "wiener"
FrameSaver: q=1.08568e+06 n=752156 tag=wiener
3983 hits in TPC 0
4203 hits in TPC 1
7413 hits in TPC 2
4039 hits in TPC 3
1107 hits in TPC 4
2538 hits in TPC 5
6475 hits in TPC 6
2585 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 1519 XUs and 4623 XVs -> 265 XUVs
C:0 T:2 58001 XUs and 75563 XVs -> 3429 XUVs
C:0 T:3 376 XUs and 613 XVs -> 12 XUVs
C:0 T:4 68 XUs and 26 XVs -> 0 XUVs
C:0 T:5 9348 XUs and 3424 XVs -> 56 XUVs
C:0 T:6 31673 XUs and 97542 XVs -> 3880 XUVs
C:0 T:7 58 XUs and 132 XVs -> 1 XUVs
7643 XUVs total
1729 collection wire objects
7643 potential space points
Neighbour search...
524817 tests to find 301234 neighbours
Iterating with no regularization...
Begin: 1.50443e+07
0 1.29158e+07
1 1.27828e+07
2 1.27607e+07
3 1.27536e+07
Now with regularization...
Begin: 1.19756e+07
0 1.19704e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000156462 0.000156403 0.000156462 0.000156403 0.000156462 0.000156403 0.000156462 0.000156403 0.000156462 0.000156403 0.000156462 0.000156403 0.000156462
----view 1: 0.0111609 0.0107485 0.0121042 0.024473 0.0227772 0.0220402 0.0139482 0.0189817 0.0126069 0.0278105 0.0171546 0.029563 0.0342675
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 260885 TrkackID 1 startx, y, z 150.117 602.636 59.511 wireno, peakt tpcno 250 3606.6 2 dist 391.417min T, max_T19.7772 3681.02
no of hits closeby 5 event 260885 TrkackID 9 startx, y, z 23.4588 589.738 317.38 wireno, peakt tpcno 425 4996.53 5 dist 0min T, max_T4951.9 5659.15
no of hits closeby 0 event 260885 TrkackID 1 startx, y, z 150.117 602.636 59.511 wireno, peakt tpcno 250 3606.6 2 dist 391.417min T, max_T19.7772 3681.02
no of hits closeby 5 event 260885 TrkackID 9 startx, y, z 23.4588 589.738 317.38 wireno, peakt tpcno 425 4996.53 5 dist 0min T, max_T4951.9 5659.15
26-Sep-2024 13:49:45 CEST Closed output file "np04hd_raw_run029081_1086_dataflow5_datawriter_0_20240905T102525_reco_stage1_reco_stage2_20240926T114945_keepup.root"
26-Sep-2024 13:49:45 CEST Closed input file "np04hd_raw_run029081_1086_dataflow5_datawriter_0_20240905T102525_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 82.0439 96.9104 122.208 96.3214 10.3794 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000552746 0.00277231 0.00941283 0.0022133 0.00195007 30
produce:fembfilter:PDHDFEMBFilter 0.684595 0.72116 0.925191 0.704024 0.0531563 30
produce:wclsdatahd:WireCellToolkit 54.8245 60.6272 63.6136 60.9834 1.90504 30
produce:gaushit:GausHitFinder 0.307461 0.531805 0.684677 0.537074 0.0960861 30
produce:nhitsfilter:NumberOfHitsFilter 0.000397452 0.000511299 0.000672958 0.000520313 6.76271e-05 30
produce:reco3d:SpacePointSolver 0.162565 1.0054 3.35451 0.835394 0.770844 30
produce:hitpdune:DisambigFromSpacePoints 0.131738 0.321612 0.611956 0.329492 0.114788 30
produce:pandora:StandardPandora 3.80674 8.42389 14.4968 8.20973 2.38276 30
produce:pandoraWriter:StandardPandora 0.0940653 0.153162 0.231994 0.152785 0.031976 30
produce:pandoraTrack:LArPandoraTrackCreation 0.704436 2.32229 6.10111 2.22284 1.14389 30
produce:pandoraShower:LArPandoraShowerCreation 0.653986 2.68144 5.58182 2.60677 1.15902 30
produce:pandoracalo:Calorimetry 0.428353 0.921441 2.21848 0.874001 0.39503 30
produce:pandoracalonosce:Calorimetry 0.320245 0.865382 2.11903 0.805001 0.389955 30
produce:pandorapid:Chi2ParticleID 0.00116567 0.0016413 0.00288294 0.00155163 0.000374606 30
produce:pandoraShowercalo:ShowerCalorimetry 0.547055 2.10792 4.53711 1.87863 0.965469 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.496783 2.03701 4.44605 1.85514 0.941586 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0580099 0.0932479 0.125641 0.0957041 0.0173363 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0247259 0.0418267 0.0584936 0.0416376 0.0079605 30
produce:emtrkmichelid:EmTrackMichelId 8.65573 12.2663 18.2036 12.2386 2.22227 30
[art]:TriggerResults:TriggerResultInserter 5.1443e-05 7.998e-05 9.8934e-05 8.0786e-05 1.17766e-05 30
end_path:out1:RootOutput 1.4217e-05 2.2059e-05 3.7955e-05 2.1712e-05 5.31797e-06 30
end_path:michelremoving:michelremoving 0.030326 0.0792309 0.168933 0.0636636 0.0413815 30
end_path:michelremovinggnocchi:michelremoving 0.0252692 0.0758946 0.171171 0.0591065 0.0425924 30
end_path:calibanagnocchi:CalibAnaTree 0.221707 0.672494 2.00703 0.588155 0.355883 30
end_path:out1:RootOutput(write) 0.398238 0.594004 0.884561 0.597747 0.113245 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5741.64 MB
Peak resident set size usage (VmHWM): 2606.17 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_1086_dataflow5_datawriter_0_20240905T102525_reco_stage1_reco_stage2_20240926T114945_keepup.root
\tHists: np04hd_raw_run029081_1086_dataflow5_datawriter_0_20240905T102525_reco_stage1_reco_stage2_20240926T114945_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
