Jobsub ID 301715.65@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301715.65@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:48:29
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:02:40
From worker node	Hostname	wj47.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:50:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_453_20231203T132811Z_gen_g4_detsim_hitreco__20240508T075948Z_reco2.root
Jobscript	Exit code	0
	Real time	11m (702s)
	CPU time	2m (134s = 19%)
Outputting started	2024-11-23 05:02:18
Output files
Finished	2024-11-23 05:02:40
List job events Wrapper job log
Jobscript log (last 10,000 characters)

tionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:01:40 GMT run: 74511302 subRun: 1 event: 45302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:01:40 GMT run: 74511302 subRun: 1 event: 45302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:01:41 GMT run: 74511302 subRun: 1 event: 45302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:01:41 GMT run: 74511302 subRun: 1 event: 45302
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 313, 197, 207
minwire 0: 1749
minwire 1: 486
minwire 2: 2027
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:02:17 GMT  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d1/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_453_20231203T132811Z_gen_g4_detsim_hitreco__20240508T075948Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0205063      121.237       242.454       121.237       121.217         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0205063     0.0212667     0.0220271     0.0212667    0.000760403       2     
reco:linecluster:LineCluster                           0.233089      0.256551      0.280012      0.256551      0.0234612        2     
reco:trajcluster:TrajCluster                           0.0798107     0.111894      0.143977      0.111894      0.0320831        2     
reco:pandora:StandardPandora                           0.235991       118.987       237.739       118.987       118.751         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000268565   0.00162639    0.00298421    0.00162639    0.00135782        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000542631   0.00226361     0.0039846    0.00226361    0.00172098        2     
reco:pandoracalo:Calorimetry                          0.000203973   0.000939298   0.00167462    0.000939298   0.000735325       2     
reco:pandorapid:Chi2ParticleID                        0.000117739   0.000531895   0.00094605    0.000531895   0.000414155       2     
reco:trkshowersplit:TrackShowerHits                   0.00264714    0.00335852     0.0040699    0.00335852    0.000711381       2     
reco:pmtrack:PMAlgTrackMaker                           0.0790655     0.974559       1.87005      0.974559      0.895494         2     
reco:pmtrackcalo:Calorimetry                          0.00320326    0.00378213    0.00436099    0.00378213    0.000578862       2     
reco:pmtrackpid:Chi2ParticleID                        0.000490021   0.000793862    0.0010977    0.000793862   0.000303841       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000692811    0.0015899    0.00248699     0.0015899    0.000897091       2     
reco:pmtrajfitcalo:Calorimetry                        0.000162368   0.000169237   0.000176107   0.000169237   6.8695e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      6.8412e-05    6.91625e-05   6.9913e-05    6.91625e-05    7.505e-07        2     
reco:pmtracktc:PMAlgTrackMaker                         0.0323459     0.727535       1.42272      0.727535      0.695189         2     
reco:pmtracktccalo:Calorimetry                        0.000715787   0.00184792    0.00298005    0.00184792    0.00113213        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000143978   0.000266503   0.000389029   0.000266503   0.000122525       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.139638      0.416903      0.694168      0.416903      0.277265         2     
reco:pmtrajfittccalo:Calorimetry                      0.000618511   0.00219097    0.00376343    0.00219097    0.00157246        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000150517   0.000291113   0.000431708   0.000291113   0.000140596       2     
reco:blurredcluster:BlurredClustering                  0.153004      0.931589       1.71017      0.931589      0.778585         2     
reco:emtrkmichelid:EmTrackMichelId                     0.751145       5.49194       10.2327       5.49194       4.7408          2     
reco:emshower:EMShower                                0.000808827   0.00256258    0.00431634    0.00256258    0.00175376        2     
reco:cvnmap:CVNMapper                                 0.000297898    0.0224387     0.0445794     0.0224387     0.0221408        2     
reco:cvneva:CVNEvaluator                              0.000297911   0.000297911   0.000297911   0.000297911        0            1     
reco:energyrecnumu:EnergyReco                         0.00751299    0.00751299    0.00751299    0.00751299         0            1     
reco:energyrecnue:EnergyReco                          0.00412796    0.00412796    0.00412796    0.00412796         0            1     
reco:energyrecnc:EnergyReco                           0.00407331    0.00407331    0.00407331    0.00407331         0            1     
reco:energyrecnumurange:EnergyReco                    0.00390411    0.00390411    0.00390411    0.00390411         0            1     
reco:energyrecnumumcs:EnergyReco                       0.003926      0.003926      0.003926      0.003926          0            1     
reco:opdec:Deconvolution                               0.408395      0.408395      0.408395      0.408395          0            1     
reco:ophitspe:OpHitFinderDeco                           2.55531       2.55531       2.55531       2.55531          0            1     
reco:opflash:OpFlashFinder                             0.0178486     0.0178486     0.0178486     0.0178486         0            1     
reco:rns:RandomNumberSaver                            0.000398841   0.000398841   0.000398841   0.000398841        0            1     
reco:anglereconue:NuAngularReco                       0.00118486    0.00118486    0.00118486    0.00118486         0            1     
reco:anglereconumu:NuAngularReco                      0.000250152   0.000250152   0.000250152   0.000250152        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00168052    0.00168052    0.00168052    0.00168052         0            1     
reco:anglereconumupfps:NuAngularReco                  0.00033475    0.00033475    0.00033475    0.00033475         0            1     
[art]:TriggerResults:TriggerResultInserter            6.8442e-05    6.8442e-05    6.8442e-05    6.8442e-05         0            1     
end_path:out1:RootOutput                              2.1391e-05    2.1391e-05    2.1391e-05    2.1391e-05         0            1     
end_path:out1:RootOutput(write)                        0.186801      0.186801      0.186801      0.186801          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7144.1 MB
  Peak resident set size usage (VmHWM): 1262.84 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:02:17 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 74511302 subRun: 1 event: 45302
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1c3b-d564-e95a-af0c.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_453_20231203T132811Z_gen_g4_detsim_hitreco__20240508T075948Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_453_20231203T132811Z_gen_g4_detsim_hitreco__20240508T075948Z_reco2_reco_2024-11-23T_045040Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work