justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108523.108@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108523.108@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-22 22:49:41
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2024-11-22 23:11:14
From worker nodeHostnamewn-choc-034.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-22 23:05:10
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root
JobscriptExit code0
Real time3m (196s)
CPU time3m (191s = 97%)
Outputting started2024-11-22 23:08:27
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root
Finished2024-11-22 23:11:14
Saved logsjustin-logs:108523.108-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

gTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:08:26 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/53/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.000878664     7.63993       19.8559       5.35146       7.41664         4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000878664   0.00112385    0.00128904    0.00116386    0.000152395       4     
reco:linecluster:LineCluster                           0.0109955     0.168325       0.59252      0.034893      0.245588         4     
reco:trajcluster:TrajCluster                          0.00223252      3.75481       14.9338      0.041597       6.45428         4     
reco:pandora:StandardPandora                            1.58822       3.32144       6.31449       2.69153       1.8267          4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000281982    0.0129981     0.0431544    0.00427804     0.0176774        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000660387    0.020296      0.0620978    0.00921288     0.0249802        4     
reco:pandoracalo:Calorimetry                          0.000215069    0.0281142     0.103896     0.00417279     0.0438629        4     
reco:pandorapid:Chi2ParticleID                        8.2947e-05    0.00054977    0.00164587    0.00023513    0.000642653       4     
reco:trkshowersplit:TrackShowerHits                   0.000176166    0.0530208      0.20636     0.00277327     0.0885552        4     
reco:pmtrack:PMAlgTrackMaker                          0.00145376      5.36694       20.2021      0.632112       8.57626         4     
reco:pmtrackcalo:Calorimetry                          9.9334e-05     0.0192272     0.0730235    0.00189292     0.0310831        4     
reco:pmtrackpid:Chi2ParticleID                        4.3105e-05    0.000500034    0.0015523    0.000202364   0.00061442        4     
reco:pmtrajfit:PMAlgTrajFitter                        0.00334019      2.91882       10.1782      0.746854       4.23204         4     
reco:pmtrajfitcalo:Calorimetry                        0.000197092    0.0290507     0.110638     0.00268404     0.0471401        4     
reco:pmtrajfitpid:Chi2ParticleID                      6.9106e-05    0.000190364   0.000460529   0.000115911   0.000157196       4     
reco:pmtracktc:PMAlgTrackMaker                        0.00265352      6.60947       25.6649      0.385176       11.0052         4     
reco:pmtracktccalo:Calorimetry                        0.000182511    0.0122782     0.0462258    0.00135228     0.0196146        4     
reco:pmtracktcpid:Chi2ParticleID                      6.5614e-05    0.000408804   0.00119976    0.000184921   0.000462545       4     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00434149      2.47554       8.94698      0.475419       3.74664         4     
reco:pmtrajfittccalo:Calorimetry                      0.000254494    0.028259      0.107804     0.00248886     0.0459556        4     
reco:pmtrajfittcpid:Chi2ParticleID                    7.7284e-05    0.000575814   0.00173596    0.000245007   0.000679767       4     
reco:blurredcluster:BlurredClustering                  0.153075      0.803988       1.9227       0.336189      0.794574         3     
reco:emtrkmichelid:EmTrackMichelId                     0.710515       3.39768       8.54541      0.937127       3.64117         3     
reco:emshower:EMShower                                0.000434979    0.0158457     0.0437009    0.00340125     0.0197338        3     
reco:cvnmap:CVNMapper                                 3.5274e-05    0.00531233     0.0158603    4.1441e-05    0.00745852        3     
reco:cvneva:CVNEvaluator                              2.2181e-05     0.404484       1.2134      2.9834e-05     0.571989         3     
reco:energyrecnumu:EnergyReco                          0.0026756    0.00436043    0.00757435    0.00283136    0.00227347        3     
reco:energyrecnue:EnergyReco                          0.000254362   0.000695021   0.00154654    0.000284158   0.00060224        3     
reco:energyrecnc:EnergyReco                           0.00256457    0.00277354    0.00311465    0.00264141    0.000243231       3     
reco:energyrecnumurange:EnergyReco                    0.000833592    0.0021839    0.00309855    0.00261957    0.000974633       3     
reco:energyrecnumumcs:EnergyReco                      0.00262281    0.00306867    0.00337711    0.00320608    0.000322904       3     
reco:opdec:Deconvolution                               0.141048       0.17647       0.22422      0.164142      0.0350559        3     
reco:ophitspe:OpHitFinderDeco                           1.01464       1.30735       1.63867       1.26873       0.25622         3     
reco:opflash:OpFlashFinder                            0.000997139   0.00219331    0.00448243    0.00110035    0.00161921        3     
reco:rns:RandomNumberSaver                            2.7829e-05    0.000119227   0.000300056   2.9797e-05    0.000127868       3     
reco:anglereconue:NuAngularReco                       0.000188364   0.00062745    0.00134006    0.000353923   0.000508407       3     
reco:anglereconumu:NuAngularReco                      0.000115717   0.000263248   0.000407384   0.000266642   0.000119097       3     
reco:anglereconuepfps:NuAngularReco                   0.000216227   0.00116418    0.00235171    0.000924613   0.000888111       3     
reco:anglereconumupfps:NuAngularReco                  0.000166197   0.000308344   0.000439487   0.000319347   0.000111841       3     
[art]:TriggerResults:TriggerResultInserter            1.1949e-05    2.42363e-05   4.7345e-05    1.3415e-05    1.63513e-05       3     
end_path:out1:RootOutput                               2.949e-06     7.516e-06    1.5626e-05     3.973e-06    5.74985e-06       3     
end_path:out1:RootOutput(write)                        0.0727976     0.157194      0.315635      0.0831497     0.112114         3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6423.31 MB
  Peak resident set size usage (VmHWM): 1658.36 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 00:08:27 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74481118 subRun: 1 event: 41204
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-3f37-d0bb-3228-74f8.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2_reco_2024-11-22T_230514Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-25 08:26:30 UTC       justIN version: 01.01.09