justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108774.26@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108774.26@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 04:54:30
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod00
Last heartbeat2024-11-23 05:04:28
From worker nodeHostnamewm04.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 04:56:39
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_104_20231203T130451Z_gen_g4_detsim_hitreco__20240508T072241Z_reco2.root
JobscriptExit code0
Real time7m (448s)
CPU time1m (93s = 20%)
Outputting started2024-11-23 05:04:08
Output files
Finished2024-11-23 05:04:28
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:03:27 GMT run: 6414162 subRun: 1 event: 10402
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:03:27 GMT run: 6414162 subRun: 1 event: 10402
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:03:27 GMT run: 6414162 subRun: 1 event: 10402
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:03:27 GMT run: 6414162 subRun: 1 event: 10402
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 233, 253, 169
minwire 0: 823
minwire 1: 2450
minwire 2: 414
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:04:06 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ca/08/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_104_20231203T130451Z_gen_g4_detsim_hitreco__20240508T072241Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00378792      133.106       266.207       133.106       133.102         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00378792    0.00880563     0.0138233    0.00880563    0.00501771        2     
reco:linecluster:LineCluster                           0.0752582     0.187396      0.299533      0.187396      0.112137         2     
reco:trajcluster:TrajCluster                           0.0316464     0.0367734     0.0419005     0.0367734    0.00512701        2     
reco:pandora:StandardPandora                           0.229407       131.749       263.269       131.749       131.52          2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000197874   0.00173695    0.00327602    0.00173695    0.00153907        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000273388   0.00246942    0.00466545    0.00246942    0.00219603        2     
reco:pandoracalo:Calorimetry                          0.000218483   0.00116778    0.00211707    0.00116778    0.000949296       2     
reco:pandorapid:Chi2ParticleID                        5.8992e-05    0.000599173   0.00113936    0.000599173   0.000540181       2     
reco:trkshowersplit:TrackShowerHits                   0.00217254     0.0029167    0.00366087     0.0029167    0.000744162       2     
reco:pmtrack:PMAlgTrackMaker                           0.0287706      0.12961       0.23045       0.12961       0.10084         2     
reco:pmtrackcalo:Calorimetry                           0.0020883    0.00301091    0.00393352    0.00301091    0.00092261        2     
reco:pmtrackpid:Chi2ParticleID                        0.000255844   0.000777095   0.00129835    0.000777095   0.000521251       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000372225   0.00153689    0.00270155    0.00153689    0.00116466        2     
reco:pmtrajfitcalo:Calorimetry                        0.000129465   0.000156351   0.000183237   0.000156351   2.6886e-05        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.6718e-05    7.86835e-05   0.000110649   7.86835e-05   3.19655e-05       2     
reco:pmtracktc:PMAlgTrackMaker                         0.0183523     0.198258      0.378163      0.198258      0.179905         2     
reco:pmtracktccalo:Calorimetry                        0.00040159    0.00131134    0.00222109    0.00131134    0.000909751       2     
reco:pmtracktcpid:Chi2ParticleID                      0.000128884   0.000252232   0.000375581   0.000252232   0.000123348       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.243276      0.572123      0.900969      0.572123      0.328846         2     
reco:pmtrajfittccalo:Calorimetry                      0.000545692   0.00187052    0.00319535    0.00187052    0.00132483        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000115499   0.000235726   0.000355953   0.000235726   0.000120227       2     
reco:blurredcluster:BlurredClustering                  0.0565691     0.149223      0.241877      0.149223      0.092654         2     
reco:emtrkmichelid:EmTrackMichelId                     0.816891       3.41371       6.01054       3.41371       2.59682         2     
reco:emshower:EMShower                                0.000585287   0.00297855    0.00537181    0.00297855    0.00239326        2     
reco:cvnmap:CVNMapper                                 0.00033853     0.0182594     0.0361803     0.0182594     0.0179209        2     
reco:cvneva:CVNEvaluator                              0.000409495   0.000409495   0.000409495   0.000409495        0            1     
reco:energyrecnumu:EnergyReco                         0.00761292    0.00761292    0.00761292    0.00761292         0            1     
reco:energyrecnue:EnergyReco                          0.00375506    0.00375506    0.00375506    0.00375506         0            1     
reco:energyrecnc:EnergyReco                           0.00381549    0.00381549    0.00381549    0.00381549         0            1     
reco:energyrecnumurange:EnergyReco                    0.00341669    0.00341669    0.00341669    0.00341669         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00356056    0.00356056    0.00356056    0.00356056         0            1     
reco:opdec:Deconvolution                                0.17582       0.17582       0.17582       0.17582          0            1     
reco:ophitspe:OpHitFinderDeco                           1.33242       1.33242       1.33242       1.33242          0            1     
reco:opflash:OpFlashFinder                            0.00657667    0.00657667    0.00657667    0.00657667         0            1     
reco:rns:RandomNumberSaver                            0.000553678   0.000553678   0.000553678   0.000553678        0            1     
reco:anglereconue:NuAngularReco                       0.00133892    0.00133892    0.00133892    0.00133892         0            1     
reco:anglereconumu:NuAngularReco                      0.000280311   0.000280311   0.000280311   0.000280311        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00216287    0.00216287    0.00216287    0.00216287         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000222982   0.000222982   0.000222982   0.000222982        0            1     
[art]:TriggerResults:TriggerResultInserter            5.9403e-05    5.9403e-05    5.9403e-05    5.9403e-05         0            1     
end_path:out1:RootOutput                              2.6681e-05    2.6681e-05    2.6681e-05    2.6681e-05         0            1     
end_path:out1:RootOutput(write)                        0.0783167     0.0783167     0.0783167     0.0783167         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37136.3 MB
  Peak resident set size usage (VmHWM): 1145.6 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:04:07 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6414162 subRun: 1 event: 10402
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-8c67-7a77-8d34-05bf.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_104_20231203T130451Z_gen_g4_detsim_hitreco__20240508T072241Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_104_20231203T130451Z_gen_g4_detsim_hitreco__20240508T072241Z_reco2_reco_2024-11-23T_045644Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-27 04:03:55 UTC       justIN version: 01.01.09