justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 88240.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID88240.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3850
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2024-10-28 20:32:46
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2024-10-30 04:45:37
From worker nodeHostnamer25-n11.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit259200 (72 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-28 20:34:04
Input fileshd-protodune:np04hd_raw_run032177_1364_dataflow1_datawriter_0_20241027T065311.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-30 04:45:37
Saved logsjustin-logs:88240.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ired window for channel 10215
wclsFrameSaver: no samples within desired window for channel 10215
wclsFrameSaver: no samples within desired window for channel 10216
wclsFrameSaver: no samples within desired window for channel 10216
wclsFrameSaver: no samples within desired window for channel 10217
wclsFrameSaver: no samples within desired window for channel 10217
wclsFrameSaver: no samples within desired window for channel 10218
wclsFrameSaver: no samples within desired window for channel 10218
wclsFrameSaver: no samples within desired window for channel 10219
wclsFrameSaver: no samples within desired window for channel 10219
wclsFrameSaver: no samples within desired window for channel 10220
wclsFrameSaver: no samples within desired window for channel 10220
wclsFrameSaver: no samples within desired window for channel 10221
wclsFrameSaver: no samples within desired window for channel 10221
wclsFrameSaver: no samples within desired window for channel 10222
wclsFrameSaver: no samples within desired window for channel 10222
wclsFrameSaver: no samples within desired window for channel 10223
wclsFrameSaver: no samples within desired window for channel 10223
wclsFrameSaver: no samples within desired window for channel 10224
wclsFrameSaver: no samples within desired window for channel 10224
wclsFrameSaver: no samples within desired window for channel 10225
wclsFrameSaver: no samples within desired window for channel 10225
wclsFrameSaver: no samples within desired window for channel 10226
wclsFrameSaver: no samples within desired window for channel 10226
wclsFrameSaver: no samples within desired window for channel 10227
wclsFrameSaver: no samples within desired window for channel 10227
wclsFrameSaver: no samples within desired window for channel 10228
wclsFrameSaver: no samples within desired window for channel 10228
wclsFrameSaver: no samples within desired window for channel 10229
wclsFrameSaver: no samples within desired window for channel 10229
wclsFrameSaver: no samples within desired window for channel 10230
wclsFrameSaver: no samples within desired window for channel 10230
wclsFrameSaver: no samples within desired window for channel 10231
wclsFrameSaver: no samples within desired window for channel 10231
wclsFrameSaver: no samples within desired window for channel 10232
wclsFrameSaver: no samples within desired window for channel 10232
wclsFrameSaver: no samples within desired window for channel 10233
wclsFrameSaver: no samples within desired window for channel 10233
wclsFrameSaver: no samples within desired window for channel 10234
wclsFrameSaver: no samples within desired window for channel 10234
wclsFrameSaver: no samples within desired window for channel 10235
wclsFrameSaver: no samples within desired window for channel 10235
wclsFrameSaver: no samples within desired window for channel 10236
wclsFrameSaver: no samples within desired window for channel 10236
wclsFrameSaver: no samples within desired window for channel 10237
wclsFrameSaver: no samples within desired window for channel 10237
wclsFrameSaver: no samples within desired window for channel 10238
wclsFrameSaver: no samples within desired window for channel 10238
wclsFrameSaver: no samples within desired window for channel 10239
wclsFrameSaver: no samples within desired window for channel 10239
FrameSaver: q=2.25061e+06 n=1283824 tag=wiener
1107 hits in TPC 0
1294 hits in TPC 1
8496 hits in TPC 2
4553 hits in TPC 3
4224 hits in TPC 4
15620 hits in TPC 5
19386 hits in TPC 6
6262 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 38742 XUs and 34905 XVs -> 1935 XUVs
C:0 T:3 330 XUs and 409 XVs -> 18 XUVs
C:0 T:4 72 XUs and 94 XVs -> 3 XUVs
C:0 T:5 87301 XUs and 105062 XVs -> 6016 XUVs
C:0 T:6 285178 XUs and 409043 XVs -> 37958 XUVs
C:0 T:7 99 XUs and 125 XVs -> 3 XUVs
45933 XUVs total
5988 collection wire objects
45933 potential space points
Neighbour search...
21991447 tests to find 4557436 neighbours
Iterating with no regularization...
Begin: 7.83967e+07
0 3.22227e+07
1 2.83206e+07
2 2.7606e+07
3 2.73496e+07
4 2.72301e+07
5 2.71636e+07
6 2.71213e+07
7 2.70925e+07
8 2.70725e+07
Now with regularization...
Begin: 2.18119e+07
0 2.1616e+07
1 2.15363e+07
2 2.14901e+07
3 2.14555e+07
4 2.14287e+07
5 2.14072e+07
6 2.13897e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

30-Oct-2024 04:45:00 GMT  Closed output file "np04hd_raw_run032177_1364_dataflow1_datawriter_0_20241027T065311_reco_stage1_reco_stage2_20241030T044500_keepup.root"
30-Oct-2024 04:45:00 GMT  Closed input file "np04hd_raw_run032177_1364_dataflow1_datawriter_0_20241027T065311_reco_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              187.847       5967.34       106603        347.503       23721.5        19     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0006307     0.0190795     0.186444     0.00474306     0.0439711       19     
produce:fembfilter:PDHDFEMBFilter                       1.19475       2.86772       20.703        1.28491       4.57163        19     
produce:wclsdatahd:WireCellToolkit                      173.627       192.14        278.129       188.032       22.8452        19     
produce:gaushit:GausHitFinder                          0.692988       2.4517        5.63663       1.66698       1.62826        19     
produce:nhitsfilter:NumberOfHitsFilter                0.000326768   0.00127517    0.00572188    0.000921993   0.00117013       19     
[art]:TriggerResults:TriggerResultInserter            2.7524e-05    0.000383674   0.00630596    5.1162e-05    0.00139614       19     
end_path:out1:RootOutput                               8.282e-06    0.000470583   0.00645625     9.322e-06    0.00149927       19     
end_path:michelremoving:michelremoving                2.7136e-05     0.101229      0.832722      0.0405554     0.203135        19     
end_path:michelremovinggnocchi:michelremoving          1.283e-05     0.0832938     0.583632      0.0337786     0.153562        19     
end_path:calibanagnocchi:CalibAnaTree                 1.1492e-05      1.24079       6.04557       0.71121       1.60882        19     
end_path:out1:RootOutput(write)                        0.523104       1.6796        6.78327       1.27251       1.55307        19     
produce:reco3d:SpacePointSolver                         2.26439       38.3984       299.487       9.89246       87.1618        10     
produce:hitpdune:DisambigFromSpacePoints                2.02392       3.04762       5.13941       2.53764       1.12152        10     
produce:pandora:StandardPandora                         50.7713       10777.7       105916        143.964       31713.6        10     
produce:pandoraWriter:StandardPandora                   0.18334      0.385265       1.01425      0.310024      0.230474        10     
produce:pandoraTrack:LArPandoraTrackCreation            2.0181        6.36608       12.9894       4.92648       3.53748        10     
produce:pandoraShower:LArPandoraShowerCreation          1.80032       7.07809       17.0972       5.24141       4.31575        10     
produce:pandoracalo:Calorimetry                        0.893304       2.96934       5.83162       2.3566        1.60371        10     
produce:pandoracalonosce:Calorimetry                    0.84669       2.67289       5.2967        2.26602       1.35181        10     
produce:pandorapid:Chi2ParticleID                     0.00252481     0.0118292     0.0540466    0.00345128     0.0172137       10     
produce:pandoraShowercalo:ShowerCalorimetry             1.51673       6.12446       12.8576       5.22843        3.028         10     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.50079       6.02439       13.2701       5.09474       3.07278        10     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0969469     0.283613      0.742876      0.201778      0.196586        10     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0383424     0.0723545     0.104163      0.0689645     0.0223598       10     
produce:emtrkmichelid:EmTrackMichelId                   60.874        105.26        180.314       98.8599       32.6125        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6907.89 MB
  Peak resident set size usage (VmHWM): 3817.69 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run032177_1364_dataflow1_datawriter_0_20241027T065311_reco_stage1_reco_stage2_20241030T044500_keepup.root
\tHists: np04hd_raw_run032177_1364_dataflow1_datawriter_0_20241027T065311_reco_stage1_reco_stage2_20241030T044500_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2024-11-24 08:23:40 UTC       justIN version: 01.01.09