justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 78687.162@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID78687.162@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3447
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2024-09-26 11:27:14
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2024-09-26 14:21:13
From worker nodeHostnamewn1907241.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-26 12:18:16
Input fileshd-protodune:np04hd_raw_run029081_0686_dataflow7_datawriter_0_20240905T035613.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-09-26 14:21:13
Saved logsjustin-logs:78687.162-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00186217 0.00135905 0.00571203 0.00179982 0.00325704 0.00523376 0.00822663 0.000653446 5.6684e-05 0.000352144 0.00118291 
----view 1: 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00135905 0.00571203 0.00179982 0.00325704 0.00523376 0.00822663 0.000653446 5.6684e-05 0.000352144 0.00118291 0.00316447 
----view 1: 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00571203 0.00179982 0.00325704 0.00523376 0.00822663 0.000653446 5.6684e-05 0.000352144 0.00118291 0.00316447 0.00267297 
----view 1: 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00179982 0.00325704 0.00523376 0.00822663 0.000653446 5.6684e-05 0.000352144 0.00118291 0.00316447 0.00267297 0.00132722 
----view 1: 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00325704 0.00523376 0.00822663 0.000653446 5.6684e-05 0.000352144 0.00118291 0.00316447 0.00267297 0.00132722 0.00126821 
----view 1: 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.72612e-05 8.73208e-05 8.72612e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0111724 0.00122511 0.00122511 0.00122505 0.00122511 0.00122511 0.00122505 0.00122511 0.00122511 0.00122511 0.00122505 
----view 1: 0 0 0 0 0 0 0 0 0 0 0 
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Begin processing the 30th record. run: 29081 subRun: 1 event: 164887 at 26-Sep-2024 15:16:12 BST
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 75115 traces, 75115 summary
Retagger: tagging trace set: gauss with 55462 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 55462 traces tagged "gauss"
FrameSaver: q=1.43769e+06 n=1075562 tag=gauss
wclsFrameSaver: saving 75115 traces tagged "wiener"
FrameSaver: q=1.52961e+06 n=974890 tag=wiener
3259 hits in TPC 0
3564 hits in TPC 1
6534 hits in TPC 2
3407 hits in TPC 3
5170 hits in TPC 4
9011 hits in TPC 5
7268 hits in TPC 6
4429 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 16 XUs and 13 XVs -> 0 XUVs
C:0 T:2 29594 XUs and 42055 XVs -> 1194 XUVs
C:0 T:3 265 XUs and 470 XVs -> 13 XUVs
C:0 T:4 1998 XUs and 4682 XVs -> 153 XUVs
C:0 T:5 40358 XUs and 33119 XVs -> 2127 XUVs
C:0 T:6 43272 XUs and 53852 XVs -> 2706 XUVs
C:0 T:7 1926 XUs and 1964 XVs -> 135 XUVs
6328 XUVs total
2156 collection wire objects
6328 potential space points
Neighbour search...
594662 tests to find 264794 neighbours
Iterating with no regularization...
Begin: 2.9602e+07
0 2.42419e+07
1 2.3749e+07
2 2.36968e+07
3 2.369e+07
Now with regularization...
Begin: 2.25446e+07
0 2.25377e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

26-Sep-2024 15:20:37 BST  Closed output file "np04hd_raw_run029081_0686_dataflow7_datawriter_0_20240905T035613_reco_stage1_reco_stage2_20240926T142037_keepup.root"
26-Sep-2024 15:20:37 BST  Closed input file "np04hd_raw_run029081_0686_dataflow7_datawriter_0_20240905T035613_reco_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              126.468       178.498       344.705       161.734       48.3646        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000566443   0.00362415     0.0107326    0.00344708    0.00193155       30     
produce:fembfilter:PDHDFEMBFilter                       0.82926       1.08059       1.46627       1.07396      0.132755        30     
produce:wclsdatahd:WireCellToolkit                      58.935        74.1878       86.2645       74.2412       6.11091        30     
produce:gaushit:GausHitFinder                          0.860038       1.27235       1.81996       1.22434      0.244597        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000463577   0.000915298   0.00837664    0.000610269   0.00139824       30     
produce:reco3d:SpacePointSolver                        0.388149       2.97918       34.328        1.41875       5.9695         30     
produce:hitpdune:DisambigFromSpacePoints                0.59261       1.12127       3.28473      0.979092      0.519975        30     
produce:pandora:StandardPandora                         11.3499       26.6083       136.255       21.7263       22.7057        30     
produce:pandoraWriter:StandardPandora                  0.165488      0.284867      0.464448      0.289071      0.0712627       30     
produce:pandoraTrack:LArPandoraTrackCreation            1.74767       3.61924       8.29072       3.4986        1.65337        30     
produce:pandoraShower:LArPandoraShowerCreation          1.58069       4.14961       12.672        4.06651       2.29682        30     
produce:pandoracalo:Calorimetry                        0.644587       1.49577       3.55177       1.38623      0.677472        30     
produce:pandoracalonosce:Calorimetry                   0.524441       1.40429       3.36107       1.29249      0.665566        30     
produce:pandorapid:Chi2ParticleID                     0.00150398    0.00307422    0.00504251    0.00303244    0.00077722       30     
produce:pandoraShowercalo:ShowerCalorimetry             1.18857       3.45934       10.9309       3.23137       2.0494         30     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.08682       3.35891       10.7213       3.13912       1.99588        30     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.127092      0.245404      0.396675       0.25978      0.056848        30     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0469411     0.0956804     0.165391      0.0968775     0.0248181       30     
produce:emtrkmichelid:EmTrackMichelId                   27.6046       49.899        104.433       46.9949       15.859         30     
[art]:TriggerResults:TriggerResultInserter            4.5807e-05    7.65224e-05   0.000124365   7.3709e-05    1.65173e-05      30     
end_path:out1:RootOutput                              1.2213e-05    2.32035e-05   4.6143e-05    2.1928e-05    8.06558e-06      30     
end_path:michelremoving:michelremoving                 0.0316076     0.174946      0.588456      0.0984408     0.146235        30     
end_path:michelremovinggnocchi:michelremoving          0.0296273     0.174349      0.576472      0.096741       0.14818        30     
end_path:calibanagnocchi:CalibAnaTree                  0.437898       1.23433       3.04398       1.11706      0.669017        30     
end_path:out1:RootOutput(write)                        0.788187       1.44313       2.17645       1.39888      0.378844        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4928.94 MB
  Peak resident set size usage (VmHWM): 2598.18 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0686_dataflow7_datawriter_0_20240905T035613_reco_stage1_reco_stage2_20240926T142037_keepup.root
\tHists: np04hd_raw_run029081_0686_dataflow7_datawriter_0_20240905T035613_reco_stage1_reco_stage2_20240926T142037_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2024-11-17 07:30:10 UTC       justIN version: 01.01.09