Jobsub ID 108883.6@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108883.6@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:26:49
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:30:08
From worker node	Hostname	wn-choc-032.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:28:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_607_20231201T210347Z_gen_g4_detsim_hitreco__20240507T222001Z_reco2.root
Jobscript	Exit code	0
	Real time	1m (101s)
	CPU time	1m (95s = 94%)
Outputting started	2024-11-23 07:29:54
Output files
Finished	2024-11-23 07:30:08
List job events Wrapper job log
Jobscript log (last 10,000 characters)

eginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
1st wire C:0 T:18 P:2 W:903 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
1st wire C:0 T:18 P:2 W:519 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:53 CET run: 6390196 subRun: 1 event: 60704
1st wire C:0 T:18 P:2 W:919 does not exist (max wire number: 480)
%MSG
23-Nov-2024 08:29:54 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/77/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_607_20231201T210347Z_gen_g4_detsim_hitreco__20240507T222001Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.000921998     4.0306        9.35366       3.38391       3.42159         4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000921998   0.00113688    0.00134039    0.00114257    0.000191206       4     
reco:linecluster:LineCluster                           0.0104462     0.0195616     0.0338243     0.016988     0.00866545        4     
reco:trajcluster:TrajCluster                           0.0011915     0.0111831     0.0190116     0.0122647    0.00645271        4     
reco:pandora:StandardPandora                            1.40375       2.06511       3.10587       1.8754       0.673242         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000143954   0.00193281    0.00368657    0.00195036    0.00125266        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000209073   0.00478559     0.0128523    0.00304052    0.00480017        4     
reco:pandoracalo:Calorimetry                          0.000128181   0.00135016    0.00290256    0.00118494    0.00099772        4     
reco:pandorapid:Chi2ParticleID                        4.7842e-05    0.000448615   0.000892162   0.000427227   0.000348556       4     
reco:trkshowersplit:TrackShowerHits                   0.00011411    0.000875063   0.00207589    0.000655129   0.000734063       4     
reco:pmtrack:PMAlgTrackMaker                          0.00140646     0.0680828     0.171463      0.049731      0.0702456        4     
reco:pmtrackcalo:Calorimetry                          9.7384e-05    0.000532146   0.00100672    0.000512238   0.000428038       4     
reco:pmtrackpid:Chi2ParticleID                        4.0905e-05    0.000101512   0.000162492   0.000101325   5.80127e-05       4     
reco:pmtrajfit:PMAlgTrajFitter                        0.000224877    0.135301      0.299869      0.120555      0.136087         4     
reco:pmtrajfitcalo:Calorimetry                         8.742e-05    0.000938576   0.00217547    0.000745707   0.000890199       4     
reco:pmtrajfitpid:Chi2ParticleID                       3.957e-05    0.00012532    0.000241683   0.000110013   8.73672e-05       4     
reco:pmtracktc:PMAlgTrackMaker                        0.00123668     0.105814      0.278229      0.071896      0.114995         4     
reco:pmtracktccalo:Calorimetry                        9.1654e-05    0.000499794   0.00109549    0.000406016   0.000428111       4     
reco:pmtracktcpid:Chi2ParticleID                      3.9718e-05    9.95368e-05   0.000175101   9.1664e-05    5.98121e-05       4     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000214311    0.119479      0.238988      0.119356      0.119221         4     
reco:pmtrajfittccalo:Calorimetry                      8.7078e-05    0.000649312   0.00129404    0.000608066   0.000563105       4     
reco:pmtrajfittcpid:Chi2ParticleID                    3.9393e-05    0.000115278   0.000197952   0.000111883   7.51408e-05       4     
reco:blurredcluster:BlurredClustering                 0.00864825     0.192548      0.533519      0.0354766     0.241351         3     
reco:emtrkmichelid:EmTrackMichelId                     0.238795       1.10635       2.73859      0.341672       1.15493         3     
reco:emshower:EMShower                                0.000270487   0.00883492     0.0227239    0.00351039    0.00990964        3     
reco:cvnmap:CVNMapper                                 2.8786e-05    0.00911355     0.0270813    0.00023051     0.0127054        3     
reco:cvneva:CVNEvaluator                              2.1127e-05     0.396148       1.18819     0.000235165    0.560057         3     
reco:energyrecnumu:EnergyReco                          0.0025965    0.00436972    0.00555057    0.00496209    0.00127666        3     
reco:energyrecnue:EnergyReco                          0.00124596     0.0021949    0.00267127    0.00266748    0.000671006       3     
reco:energyrecnc:EnergyReco                           0.00264934    0.00272702    0.00277575    0.00275596    5.55168e-05       3     
reco:energyrecnumurange:EnergyReco                    0.00265965    0.00270745    0.00278522    0.00267748    5.54685e-05       3     
reco:energyrecnumumcs:EnergyReco                      0.00256885    0.00266793    0.00277314     0.0026618    8.35133e-05       3     
reco:opdec:Deconvolution                               0.0856834     0.117615      0.176576      0.0905865     0.0417397        3     
reco:ophitspe:OpHitFinderDeco                          0.516226      0.811227       1.32832      0.589131      0.366852         3     
reco:opflash:OpFlashFinder                             0.000488     0.00191335    0.00395926    0.00129279    0.00148352        3     
reco:rns:RandomNumberSaver                            2.0699e-05    0.000118196   0.000312847   2.1041e-05    0.000137639       3     
reco:anglereconue:NuAngularReco                       0.000154328   0.000601032   0.00089501    0.000753758   0.000321088       3     
reco:anglereconumu:NuAngularReco                       9.577e-05    0.00016831    0.000249663   0.000159496   6.31349e-05       3     
reco:anglereconuepfps:NuAngularReco                   0.000139985   0.00107229    0.00179789     0.001279     0.000692439       3     
reco:anglereconumupfps:NuAngularReco                  0.000137506   0.000215036   0.000307828   0.000199773   7.03662e-05       3     
[art]:TriggerResults:TriggerResultInserter            1.2596e-05    2.48473e-05   4.9147e-05    1.2799e-05    1.71827e-05       3     
end_path:out1:RootOutput                               2.867e-06    8.04067e-06   1.8342e-05     2.913e-06    7.28417e-06       3     
end_path:out1:RootOutput(write)                        0.042716      0.119536       0.27204      0.0438507     0.107838         3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6425.33 MB
  Peak resident set size usage (VmHWM): 1677.48 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:29:54 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6390196 subRun: 1 event: 60704
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-435d-9303-8722-20c6.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_607_20231201T210347Z_gen_g4_detsim_hitreco__20240507T222001Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_607_20231201T210347Z_gen_g4_detsim_hitreco__20240507T222001Z_reco2_reco_2024-11-23T_072816Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work