Jobsub ID 301870.5@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301870.5@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:27:52
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2024-11-23 07:34:11
From worker node	Hostname	wn-choc-036.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:29:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1070_20231203T053032Z_gen_g4_detsim_hitreco__20240508T063605Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (238s)
	CPU time	2m (146s = 61%)
Outputting started	2024-11-23 07:33:57
Output files
Finished	2024-11-23 07:34:11
List job events Wrapper job log
Jobscript log (last 10,000 characters)

%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:33:56 CET run: 6413332 subRun: 1 event: 107004
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:33:57 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cd/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1070_20231203T053032Z_gen_g4_detsim_hitreco__20240508T063605Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0615597      12.1473       26.4441       11.0419       9.4082          4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0154616     0.0341976     0.0615597     0.0298846     0.017217         4     
reco:linecluster:LineCluster                           0.144734      0.206275      0.340804      0.169781      0.0786809        4     
reco:trajcluster:TrajCluster                           0.0562949     0.572769       1.88497      0.174906      0.762644         4     
reco:pandora:StandardPandora                            1.84429       2.54422       3.17993       2.57632      0.540988         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.00120318     0.0118466     0.0306128    0.00778525     0.0112794        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00361551     0.0183141     0.0329216     0.0183597     0.0105679        4     
reco:pandoracalo:Calorimetry                          0.00100127     0.0152102     0.0484583    0.00569064     0.0193258        4     
reco:pandorapid:Chi2ParticleID                        0.000163341   0.000616671   0.00165955    0.000321898   0.000605665       4     
reco:trkshowersplit:TrackShowerHits                   0.000904138    0.0209149     0.0721259    0.00531474     0.0296814        4     
reco:pmtrack:PMAlgTrackMaker                           0.187753       1.45829       3.93336      0.856026       1.52106         4     
reco:pmtrackcalo:Calorimetry                          0.00112605    0.00899305     0.0244299     0.0052081    0.00949743        4     
reco:pmtrackpid:Chi2ParticleID                        0.000178849   0.000413028   0.000905784   0.000283739   0.000295151       4     
reco:pmtrajfit:PMAlgTrajFitter                         0.0550011      1.43186       3.66791       1.00226       1.37191         4     
reco:pmtrajfitcalo:Calorimetry                        0.000566678    0.0147957     0.046558     0.00602914     0.0187694        4     
reco:pmtrajfitpid:Chi2ParticleID                      0.000115968   0.000318272   0.000528609   0.000314255   0.000175455       4     
reco:pmtracktc:PMAlgTrackMaker                         0.102589       1.98787       5.88739      0.980761       2.32819         4     
reco:pmtracktccalo:Calorimetry                        0.000815349   0.00628264     0.0144018    0.00495672    0.00565631        4     
reco:pmtracktcpid:Chi2ParticleID                      0.000150941   0.000361882   0.000779102   0.000258742   0.000248939       4     
reco:pmtrajfittc:PMAlgTrajFitter                       0.385834       1.86465       5.03554       1.01862       1.88971         4     
reco:pmtrajfittccalo:Calorimetry                       0.0011481     0.0161012     0.0523665    0.00544512     0.0211944        4     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000175005   0.000520107   0.00131488    0.000295272   0.000464623       4     
reco:blurredcluster:BlurredClustering                  0.485573      0.779862       1.26513      0.588886      0.345717         3     
reco:emtrkmichelid:EmTrackMichelId                      2.28256       6.64446       14.0391       3.61172       5.25687         3     
reco:emshower:EMShower                                0.00443107     0.0220975     0.0370438     0.0248175     0.0134523        3     
reco:cvnmap:CVNMapper                                  0.0111743     0.0147395     0.020829      0.0122152    0.00432682        3     
reco:cvneva:CVNEvaluator                                1.17209       1.19003       1.2214        1.17658      0.0222614        3     
reco:energyrecnumu:EnergyReco                         0.00261901    0.00739931     0.011092      0.0084869    0.00354355        3     
reco:energyrecnue:EnergyReco                          0.000491976   0.00144289    0.00331464    0.000522061   0.00132358        3     
reco:energyrecnc:EnergyReco                            0.0028651    0.00389675    0.00573383    0.00309131    0.00130229        3     
reco:energyrecnumurange:EnergyReco                     0.0012891    0.00264607    0.00464447    0.00200464    0.00144296        3     
reco:energyrecnumumcs:EnergyReco                      0.00397469    0.00448988    0.00495767    0.00453728    0.000402696       3     
reco:opdec:Deconvolution                               0.245565      0.269425      0.286793      0.275918      0.0174464        3     
reco:ophitspe:OpHitFinderDeco                           1.17568       1.34991       1.45134       1.42272      0.123755         3     
reco:opflash:OpFlashFinder                            0.00105731    0.00228018    0.00453855    0.00124467    0.00159874        3     
reco:rns:RandomNumberSaver                            3.0189e-05    0.000136953   0.000348991   3.1678e-05    0.000149935       3     
reco:anglereconue:NuAngularReco                       0.000426141   0.00143757    0.00252351    0.00136307    0.000857867       3     
reco:anglereconumu:NuAngularReco                      0.000298602   0.00102911    0.00246035    0.00032838    0.00101211        3     
reco:anglereconuepfps:NuAngularReco                   0.000560731   0.00205955    0.00303143    0.00258649    0.00107528        3     
reco:anglereconumupfps:NuAngularReco                  0.000394133   0.00115076    0.00261904    0.000439108   0.00103839        3     
[art]:TriggerResults:TriggerResultInserter            1.3257e-05    2.8411e-05    5.8607e-05    1.3369e-05    2.13518e-05       3     
end_path:out1:RootOutput                               4.529e-06    1.03797e-05   2.0158e-05     6.452e-06    6.95875e-06       3     
end_path:out1:RootOutput(write)                        0.247165      0.279587      0.317926      0.273669      0.0291896        3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6418.86 MB
  Peak resident set size usage (VmHWM): 1671.26 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:33:57 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6413332 subRun: 1 event: 107004
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-6d17-bb25-a54a-28d8.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1070_20231203T053032Z_gen_g4_detsim_hitreco__20240508T063605Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1070_20231203T053032Z_gen_g4_detsim_hitreco__20240508T063605Z_reco2_reco_2024-11-23T_073002Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work