Jobsub ID 78469.141@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78469.141@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3449 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:01:18 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:14:09 |
| From worker node | Hostname | cn-leo-116.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:24:58 |
| Input files | hd-protodune:np04hd_raw_run029107_0473_dataflow5_datawriter_0_20240906T233440.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:14:09 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
XUVs
C:0 T:2 39632 XUs and 53176 XVs -> 1137 XUVs
C:0 T:3 124 XUs and 188 XVs -> 0 XUVs
C:0 T:4 163 XUs and 304 XVs -> 4 XUVs
C:0 T:5 15854 XUs and 16266 XVs -> 346 XUVs
C:0 T:6 31959 XUs and 59438 XVs -> 1814 XUVs
C:0 T:7 961 XUs and 2128 XVs -> 31 XUVs
3343 XUVs total
1494 collection wire objects
3343 potential space points
Neighbour search...
166831 tests to find 88784 neighbours
Iterating with no regularization...
Begin: 8.62868e+06
0 7.97767e+06
1 7.93255e+06
2 7.92749e+06
Now with regularization...
Begin: 7.33343e+06
0 7.32472e+06
1 7.32361e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
no of hits closeby 0 event 113765 TrkackID 25 startx, y, z 28.0933 590.685 214.981 wireno, peakt tpcno 439 5253 2 dist 66.0429min T, max_T4906.34 5253
no of hits closeby 0 event 113765 TrkackID 25 startx, y, z 28.0933 590.685 214.981 wireno, peakt tpcno 439 5253 2 dist 66.0429min T, max_T4906.34 5253
Begin processing the 29th record. run: 29107 subRun: 1 event: 113773 at 26-Sep-2024 13:33:31 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 80930 traces, 80930 summary
Retagger: tagging trace set: gauss with 61849 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 61849 traces tagged "gauss"
FrameSaver: q=1.44875e+06 n=1209141 tag=gauss
wclsFrameSaver: saving 80930 traces tagged "wiener"
FrameSaver: q=1.549e+06 n=1112558 tag=wiener
3212 hits in TPC 0
2454 hits in TPC 1
9949 hits in TPC 2
6645 hits in TPC 3
5024 hits in TPC 4
5533 hits in TPC 5
6856 hits in TPC 6
4262 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 151 XUs and 222 XVs -> 2 XUVs
C:0 T:2 61359 XUs and 34763 XVs -> 975 XUVs
C:0 T:3 328 XUs and 841 XVs -> 7 XUVs
C:0 T:4 449 XUs and 609 XVs -> 48 XUVs
C:0 T:5 14464 XUs and 13854 XVs -> 586 XUVs
C:0 T:6 41263 XUs and 36701 XVs -> 3392 XUVs
C:0 T:7 1245 XUs and 666 XVs -> 35 XUVs
5045 XUVs total
1645 collection wire objects
5045 potential space points
Neighbour search...
367791 tests to find 221632 neighbours
Iterating with no regularization...
Begin: 1.19978e+07
0 1.08832e+07
1 1.08154e+07
2 1.08025e+07
3 1.07972e+07
Now with regularization...
Begin: 1.02444e+07
0 1.02411e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29107 subRun: 1 event: 113821 at 26-Sep-2024 13:35:27 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 82676 traces, 82676 summary
Retagger: tagging trace set: gauss with 60913 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60913 traces tagged "gauss"
FrameSaver: q=1.36904e+06 n=1115045 tag=gauss
wclsFrameSaver: saving 82676 traces tagged "wiener"
FrameSaver: q=1.48744e+06 n=1009683 tag=wiener
3808 hits in TPC 0
3116 hits in TPC 1
6034 hits in TPC 2
3103 hits in TPC 3
6541 hits in TPC 4
8868 hits in TPC 5
8053 hits in TPC 6
4272 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4731 XUs and 3550 XVs -> 86 XUVs
C:0 T:2 15943 XUs and 39675 XVs -> 1140 XUVs
C:0 T:3 381 XUs and 704 XVs -> 29 XUVs
C:0 T:4 3421 XUs and 3856 XVs -> 450 XUVs
C:0 T:5 63311 XUs and 40919 XVs -> 2425 XUVs
C:0 T:6 30176 XUs and 33020 XVs -> 565 XUVs
C:0 T:7 483 XUs and 624 XVs -> 17 XUVs
4712 XUVs total
1679 collection wire objects
4712 potential space points
Neighbour search...
192410 tests to find 107776 neighbours
Iterating with no regularization...
Begin: 1.16525e+07
0 1.00673e+07
1 9.94569e+06
2 9.92332e+06
3 9.91686e+06
Now with regularization...
Begin: 9.14377e+06
0 9.13754e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:37:23 CEST Closed output file "np04hd_raw_run029107_0473_dataflow5_datawriter_0_20240906T233440_reco_stage1_reco_stage2_20240926T113723_keepup.root"
26-Sep-2024 13:37:23 CEST Closed input file "np04hd_raw_run029107_0473_dataflow5_datawriter_0_20240906T233440_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 75.7907 103.551 157.877 100.967 18.1483 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000638834 0.00279926 0.00697149 0.00199609 0.00203273 30
produce:fembfilter:PDHDFEMBFilter 0.699576 0.740024 0.958541 0.731491 0.044452 30
produce:wclsdatahd:WireCellToolkit 58.5786 62.5311 67.2749 61.8212 2.13021 30
produce:gaushit:GausHitFinder 0.281756 0.587629 0.948056 0.568882 0.160878 30
produce:nhitsfilter:NumberOfHitsFilter 0.000315492 0.000554019 0.000807041 0.000560961 0.000122368 30
produce:reco3d:SpacePointSolver 0.150702 1.09057 5.30635 0.730732 0.960108 30
produce:hitpdune:DisambigFromSpacePoints 0.0907348 0.369247 0.818441 0.371344 0.177606 30
produce:pandora:StandardPandora 4.29277 9.71718 25.7379 8.89109 4.30992 30
produce:pandoraWriter:StandardPandora 0.0752695 0.163536 0.265207 0.160389 0.0472106 30
produce:pandoraTrack:LArPandoraTrackCreation 0.291174 2.64628 7.87413 2.6343 1.57613 30
produce:pandoraShower:LArPandoraShowerCreation 0.2165 3.37682 11.0246 2.91979 2.32392 30
produce:pandoracalo:Calorimetry 0.167203 1.04087 3.20546 0.991488 0.620959 30
produce:pandoracalonosce:Calorimetry 0.13268 1.01118 3.21987 0.962444 0.659104 30
produce:pandorapid:Chi2ParticleID 0.00109919 0.00171145 0.0026492 0.00169989 0.000378868 30
produce:pandoraShowercalo:ShowerCalorimetry 0.173286 2.63198 9.24803 2.32521 1.89638 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.159552 2.54147 8.22829 2.29499 1.75336 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0527865 0.102557 0.171356 0.105415 0.0314822 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0182207 0.0456204 0.0756013 0.0461635 0.0140125 30
produce:emtrkmichelid:EmTrackMichelId 6.88697 13.0519 20.9362 13.1443 3.26667 30
[art]:TriggerResults:TriggerResultInserter 5.2836e-05 8.39809e-05 0.000118665 8.02725e-05 1.47318e-05 30
end_path:out1:RootOutput 1.5495e-05 2.53528e-05 3.7714e-05 2.5764e-05 4.47458e-06 30
end_path:michelremoving:michelremoving 0.0223444 0.0803397 0.314513 0.0587411 0.0631434 30
end_path:michelremovinggnocchi:michelremoving 0.0160285 0.0769179 0.298724 0.0550549 0.061273 30
end_path:calibanagnocchi:CalibAnaTree 0.146336 0.724117 2.04346 0.755229 0.393383 30
end_path:out1:RootOutput(write) 0.323983 0.638073 1.11184 0.626842 0.182443 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5904.47 MB
Peak resident set size usage (VmHWM): 2750.11 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_0473_dataflow5_datawriter_0_20240906T233440_reco_stage1_reco_stage2_20240926T113723_keepup.root
\tHists: np04hd_raw_run029107_0473_dataflow5_datawriter_0_20240906T233440_reco_stage1_reco_stage2_20240926T113723_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
