Jobsub ID 108779.103@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.103@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod02
Last heartbeat	2024-11-23 05:14:54
From worker node	Hostname	wg36.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:05:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2.root
Jobscript	Exit code	0
	Real time	9m (546s)
	CPU time	2m (159s = 29%)
Outputting started	2024-11-23 05:14:33
Output files
Finished	2024-11-23 05:14:54
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:pmtrajfittc@BeginModule  23-Nov-2024 05:13:31 GMT run: 6472771 subRun: 1 event: 10502
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:13:31 GMT run: 6472771 subRun: 1 event: 10502
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:13:31 GMT run: 6472771 subRun: 1 event: 10502
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:13:31 GMT run: 6472771 subRun: 1 event: 10502
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:13:31 GMT run: 6472771 subRun: 1 event: 10502
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 762, 790, 370
minwire 0: 566
minwire 1: 2189
minwire 2: 73
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:14:30 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/59/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0115943      115.598       231.184       115.598       115.586         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0115943     0.0483764     0.0851586     0.0483764     0.0367821        2     
reco:linecluster:LineCluster                           0.337131      0.349759      0.362388      0.349759      0.0126285        2     
reco:trajcluster:TrajCluster                           0.0903451     0.180512      0.270678      0.180512      0.0901667        2     
reco:pandora:StandardPandora                           0.418533       113.93        227.442       113.93        113.512         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000234692   0.00184872    0.00346276    0.00184872    0.00161403        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00025772    0.00258643    0.00491515    0.00258643    0.00232871        2     
reco:pandoracalo:Calorimetry                           0.000166     0.00107358    0.00198117    0.00107358    0.000907583       2     
reco:pandorapid:Chi2ParticleID                        7.9582e-05    0.000561078   0.00104257    0.000561078   0.000481496       2     
reco:trkshowersplit:TrackShowerHits                   0.00330646    0.00549743     0.0076884    0.00549743    0.00219097        2     
reco:pmtrack:PMAlgTrackMaker                          0.00888528      1.20543       2.40198       1.20543       1.19655         2     
reco:pmtrackcalo:Calorimetry                          0.000189176   0.00647706     0.0127649    0.00647706    0.00628788        2     
reco:pmtrackpid:Chi2ParticleID                        7.3451e-05    0.000278509   0.000483568   0.000278509   0.000205058       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000516517   0.00155861    0.00260071    0.00155861     0.0010421        2     
reco:pmtrajfitcalo:Calorimetry                        0.000129283   0.000138454   0.000147626   0.000138454   9.1715e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      5.1235e-05    5.70435e-05   6.2852e-05    5.70435e-05   5.8085e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.020868       1.17016       2.31946       1.17016       1.1493          2     
reco:pmtracktccalo:Calorimetry                         0.0039968    0.00789999     0.0118032    0.00789999    0.00390319        2     
reco:pmtracktcpid:Chi2ParticleID                      0.00047756    0.000968399   0.00145924    0.000968399   0.000490839       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.171677      0.962911       1.75414      0.962911      0.791234         2     
reco:pmtrajfittccalo:Calorimetry                      0.00047503    0.00662729     0.0127795    0.00662729    0.00615226        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000139753   0.000365543   0.000591333   0.000365543   0.00022579        2     
reco:blurredcluster:BlurredClustering                  0.0209294      1.26066       2.50039       1.26066       1.23973         2     
reco:emtrkmichelid:EmTrackMichelId                     0.663287       11.505        22.3468       11.505        10.8418         2     
reco:emshower:EMShower                                0.00117955    0.00320424    0.00522893    0.00320424    0.00202469        2     
reco:cvnmap:CVNMapper                                 0.000343846    0.0164263     0.0325087     0.0164263     0.0160824        2     
reco:cvneva:CVNEvaluator                              0.000351306   0.000351306   0.000351306   0.000351306        0            1     
reco:energyrecnumu:EnergyReco                         0.00828964    0.00828964    0.00828964    0.00828964         0            1     
reco:energyrecnue:EnergyReco                          0.00425588    0.00425588    0.00425588    0.00425588         0            1     
reco:energyrecnc:EnergyReco                           0.00421494    0.00421494    0.00421494    0.00421494         0            1     
reco:energyrecnumurange:EnergyReco                    0.00403271    0.00403271    0.00403271    0.00403271         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00402381    0.00402381    0.00402381    0.00402381         0            1     
reco:opdec:Deconvolution                               0.531167      0.531167      0.531167      0.531167          0            1     
reco:ophitspe:OpHitFinderDeco                           1.68304       1.68304       1.68304       1.68304          0            1     
reco:opflash:OpFlashFinder                            0.00524623    0.00524623    0.00524623    0.00524623         0            1     
reco:rns:RandomNumberSaver                            0.000434879   0.000434879   0.000434879   0.000434879        0            1     
reco:anglereconue:NuAngularReco                       0.00131268    0.00131268    0.00131268    0.00131268         0            1     
reco:anglereconumu:NuAngularReco                      0.000237672   0.000237672   0.000237672   0.000237672        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00193248    0.00193248    0.00193248    0.00193248         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000311811   0.000311811   0.000311811   0.000311811        0            1     
[art]:TriggerResults:TriggerResultInserter            7.2444e-05    7.2444e-05    7.2444e-05    7.2444e-05         0            1     
end_path:out1:RootOutput                              2.8582e-05    2.8582e-05    2.8582e-05    2.8582e-05         0            1     
end_path:out1:RootOutput(write)                        0.0622619     0.0622619     0.0622619     0.0622619         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6239.44 MB
  Peak resident set size usage (VmHWM): 1390.52 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:14:31 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6472771 subRun: 1 event: 10502
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-4fff-3850-8e85-688e.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2_reco_2024-11-23T_050530Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work