Jobsub ID 263351.15@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263351.15@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3449 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:16:27 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2024-09-26 12:26:02 |
| From worker node | Hostname | cn-leo-169.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:49:16 |
| Input files | hd-protodune:np04hd_raw_run029107_0871_dataflow6_datawriter_0_20240907T045252.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:26:02 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
wclsFrameSaver: saving 61587 traces tagged "wiener"
FrameSaver: q=989483 n=697396 tag=wiener
3672 hits in TPC 0
2120 hits in TPC 1
5149 hits in TPC 2
3374 hits in TPC 3
4479 hits in TPC 4
4731 hits in TPC 5
4123 hits in TPC 6
3277 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 751 XUs and 922 XVs -> 39 XUVs
C:0 T:2 17443 XUs and 21879 XVs -> 1225 XUVs
C:0 T:3 120 XUs and 175 XVs -> 9 XUVs
C:0 T:4 184 XUs and 120 XVs -> 5 XUVs
C:0 T:5 20003 XUs and 13006 XVs -> 292 XUVs
C:0 T:6 13231 XUs and 23211 XVs -> 429 XUVs
C:0 T:7 14 XUs and 28 XVs -> 0 XUVs
1999 XUVs total
772 collection wire objects
1999 potential space points
Neighbour search...
169993 tests to find 91046 neighbours
Iterating with no regularization...
Begin: 2.5819e+06
0 2.38007e+06
1 2.36644e+06
2 2.36425e+06
Now with regularization...
Begin: 2.15712e+06
0 2.15632e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29107 subRun: 1 event: 209294 at 26-Sep-2024 13:54:20 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 59044 traces, 59044 summary
Retagger: tagging trace set: gauss with 45121 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 45121 traces tagged "gauss"
FrameSaver: q=948807 n=850912 tag=gauss
wclsFrameSaver: saving 59044 traces tagged "wiener"
FrameSaver: q=1.00495e+06 n=767007 tag=wiener
3081 hits in TPC 0
2719 hits in TPC 1
5144 hits in TPC 2
3671 hits in TPC 3
4685 hits in TPC 4
7297 hits in TPC 5
2829 hits in TPC 6
1794 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 71 XUs and 48 XVs -> 2 XUVs
C:0 T:2 17457 XUs and 12197 XVs -> 265 XUVs
C:0 T:3 120 XUs and 101 XVs -> 6 XUVs
C:0 T:4 120 XUs and 122 XVs -> 0 XUVs
C:0 T:5 26559 XUs and 23766 XVs -> 595 XUVs
C:0 T:6 5729 XUs and 8085 XVs -> 77 XUVs
C:0 T:7 81 XUs and 109 XVs -> 4 XUVs
949 XUVs total
734 collection wire objects
949 potential space points
Neighbour search...
7141 tests to find 3542 neighbours
Iterating with no regularization...
Begin: 3.09632e+06
0 2.992e+06
1 2.98883e+06
2 2.98857e+06
Now with regularization...
Begin: 2.78583e+06
0 2.78567e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29107 subRun: 1 event: 209302 at 26-Sep-2024 13:55:51 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 57392 traces, 57392 summary
Retagger: tagging trace set: gauss with 42165 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 42165 traces tagged "gauss"
FrameSaver: q=927850 n=765127 tag=gauss
wclsFrameSaver: saving 57392 traces tagged "wiener"
FrameSaver: q=1.00239e+06 n=689954 tag=wiener
2616 hits in TPC 0
2831 hits in TPC 1
5352 hits in TPC 2
3606 hits in TPC 3
2267 hits in TPC 4
3418 hits in TPC 5
5864 hits in TPC 6
4019 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 19 XUs and 35 XVs -> 1 XUVs
C:0 T:2 28214 XUs and 18838 XVs -> 446 XUVs
C:0 T:3 440 XUs and 537 XVs -> 34 XUVs
C:0 T:4 70 XUs and 85 XVs -> 6 XUVs
C:0 T:5 3702 XUs and 4000 XVs -> 229 XUVs
C:0 T:6 79180 XUs and 38689 XVs -> 982 XUVs
C:0 T:7 85 XUs and 72 XVs -> 1 XUVs
1699 XUVs total
909 collection wire objects
1699 potential space points
Neighbour search...
30443 tests to find 19806 neighbours
Iterating with no regularization...
Begin: 1.11812e+07
0 1.08163e+07
1 1.07895e+07
2 1.07847e+07
Now with regularization...
Begin: 1.04991e+07
0 1.04977e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:57:19 CEST Closed output file "np04hd_raw_run029107_0871_dataflow6_datawriter_0_20240907T045252_reco_stage1_reco_stage2_20240926T115719_keepup.root"
26-Sep-2024 13:57:19 CEST Closed input file "np04hd_raw_run029107_0871_dataflow6_datawriter_0_20240907T045252_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 78.8825 97.1918 131.43 93.8239 11.5996 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000554931 0.00234839 0.00650071 0.00132265 0.00193299 30
produce:fembfilter:PDHDFEMBFilter 0.668571 0.708289 0.919833 0.693685 0.0554799 30
produce:wclsdatahd:WireCellToolkit 55.0375 58.424 61.527 58.8367 1.84894 30
produce:gaushit:GausHitFinder 0.322499 0.568046 0.847197 0.56822 0.124325 30
produce:nhitsfilter:NumberOfHitsFilter 0.000383534 0.000533729 0.000715977 0.000530098 9.3715e-05 30
produce:reco3d:SpacePointSolver 0.146297 0.987793 3.18217 0.8035 0.687727 30
produce:hitpdune:DisambigFromSpacePoints 0.150866 0.363685 0.896477 0.32777 0.148975 30
produce:pandora:StandardPandora 3.77594 9.87189 18.6212 9.2458 3.78363 30
produce:pandoraWriter:StandardPandora 0.106468 0.159424 0.231509 0.159736 0.0285011 30
produce:pandoraTrack:LArPandoraTrackCreation 1.05191 2.31342 4.262 2.17063 0.788125 30
produce:pandoraShower:LArPandoraShowerCreation 1.03639 2.83694 7.17884 2.64281 1.25271 30
produce:pandoracalo:Calorimetry 0.453037 0.916459 1.85517 0.877126 0.288655 30
produce:pandoracalonosce:Calorimetry 0.424297 0.862422 1.73914 0.824375 0.2798 30
produce:pandorapid:Chi2ParticleID 0.00109822 0.00162124 0.00266436 0.00158269 0.000331533 30
produce:pandoraShowercalo:ShowerCalorimetry 0.73899 2.26904 6.93727 2.10191 1.15319 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.724678 2.2209 6.58343 2.02949 1.11021 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0594628 0.101158 0.144554 0.0952767 0.0209415 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0273263 0.045149 0.0718198 0.0419111 0.00958576 30
produce:emtrkmichelid:EmTrackMichelId 8.02144 12.716 16.6969 12.6844 2.1966 30
[art]:TriggerResults:TriggerResultInserter 4.6808e-05 6.55186e-05 0.000105968 6.2249e-05 1.29786e-05 30
end_path:out1:RootOutput 1.146e-05 1.78972e-05 3.2867e-05 1.7353e-05 4.49092e-06 30
end_path:michelremoving:michelremoving 0.0303849 0.087719 0.260045 0.051857 0.0590234 30
end_path:michelremovinggnocchi:michelremoving 0.0299789 0.0844372 0.261717 0.0535505 0.0562276 30
end_path:calibanagnocchi:CalibAnaTree 0.326282 0.669276 1.12822 0.649777 0.196061 30
end_path:out1:RootOutput(write) 0.385364 0.627871 0.881206 0.627749 0.122045 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5834.4 MB
Peak resident set size usage (VmHWM): 2717.3 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029107_0871_dataflow6_datawriter_0_20240907T045252_reco_stage1_reco_stage2_20240926T115719_keepup.root
\tHists: np04hd_raw_run029107_0871_dataflow6_datawriter_0_20240907T045252_reco_stage1_reco_stage2_20240926T115719_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
