justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108885.34@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108885.34@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 07:33:13
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2024-11-23 07:38:13
From worker nodeHostnamewn-choc-032.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 07:35:08
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_473_20231203T111345Z_gen_g4_detsim_hitreco__20240508T044205Z_reco2.root
JobscriptExit code0
Real time2m (170s)
CPU time2m (120s = 70%)
Outputting started2024-11-23 07:37:59
Output files
Finished2024-11-23 07:38:13
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

8365 subRun: 1 event: 47303
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:37:58 CET run: 6418365 subRun: 1 event: 47303
TuneFullTree failed.
%MSG
23-Nov-2024 08:37:58 CET  Closed input file "root://penguin12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/29/17/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_473_20231203T111345Z_gen_g4_detsim_hitreco__20240508T044205Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00180073      15.401        36.7306       9.47073       15.5698         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00127637     0.001531     0.00180073    0.00151588    0.000214336       3     
reco:linecluster:LineCluster                           0.012472      0.032059      0.0530054     0.0306995     0.0165756        3     
reco:trajcluster:TrajCluster                          0.00357441     0.133412      0.370651      0.0260098     0.168003         3     
reco:pandora:StandardPandora                            1.5559        2.40365       3.1371        2.51795       0.65056         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000166333   0.00530491    0.00945139    0.00629702    0.00385498        3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000603473    0.0148382     0.0257151     0.0181962     0.0105232        3     
reco:pandoracalo:Calorimetry                          0.000117199   0.00428455    0.00767951    0.00505694    0.00313524        3     
reco:pandorapid:Chi2ParticleID                        4.9804e-05    0.000690016   0.00174987    0.000270376   0.000754819       3     
reco:trkshowersplit:TrackShowerHits                   0.000311656   0.00502869     0.0121956    0.00257881    0.00515159        3     
reco:pmtrack:PMAlgTrackMaker                          0.00735893     0.671064       1.90987      0.0959581     0.876718         3     
reco:pmtrackcalo:Calorimetry                          0.000309157   0.00479597     0.0131592    0.000919522   0.00591897        3     
reco:pmtrackpid:Chi2ParticleID                        9.2037e-05    0.00024023    0.00050802    0.000120633   0.000189716       3     
reco:pmtrajfit:PMAlgTrajFitter                         0.0241663     0.626951       1.43387      0.422814      0.593337         3     
reco:pmtrajfitcalo:Calorimetry                        0.000292207   0.00375004    0.00972226    0.00123564    0.00424053        3     
reco:pmtrajfitpid:Chi2ParticleID                      8.3916e-05    0.000195108   0.000362245   0.000139162   0.000120317       3     
reco:pmtracktc:PMAlgTrackMaker                         0.0197463     0.673943       1.89407      0.108015      0.863511         3     
reco:pmtracktccalo:Calorimetry                        0.000528123   0.00501309     0.0134048    0.00110632    0.00593855        3     
reco:pmtracktcpid:Chi2ParticleID                      0.000122106   0.000283983   0.000602559   0.000127283   0.000225277       3     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0745702     0.782666       2.13189      0.141533      0.954441         3     
reco:pmtrajfittccalo:Calorimetry                      0.000507675   0.00586758     0.0160866     0.0010085    0.00722881        3     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000120904   0.00030414    0.000668177   0.000123339   0.000257415       3     
reco:blurredcluster:BlurredClustering                  0.281987       1.25562       2.22925       1.25562      0.973631         2     
reco:emtrkmichelid:EmTrackMichelId                      2.5324        11.1514       19.7703       11.1514       8.61897         2     
reco:emshower:EMShower                                0.00503467     0.0350095     0.0649843     0.0350095     0.0299748        2     
reco:cvnmap:CVNMapper                                  0.0120042     0.0238108     0.0356175     0.0238108     0.0118066        2     
reco:cvneva:CVNEvaluator                                1.1729        1.19994       1.22697       1.19994      0.0270387        2     
reco:energyrecnumu:EnergyReco                          0.0170039     0.0633792     0.109754      0.0633792     0.0463753        2     
reco:energyrecnue:EnergyReco                          0.000652673   0.00156437    0.00247607    0.00156437    0.000911701       2     
reco:energyrecnc:EnergyReco                           0.00319198    0.00407431    0.00495665    0.00407431    0.000882336       2     
reco:energyrecnumurange:EnergyReco                    0.00880645     0.0550626     0.101319      0.0550626     0.0462561        2     
reco:energyrecnumumcs:EnergyReco                       0.0111734     0.0568884     0.102603      0.0568884     0.045715         2     
reco:opdec:Deconvolution                               0.146858      0.211507      0.276156      0.211507      0.0646485        2     
reco:ophitspe:OpHitFinderDeco                          0.962579       1.5059        2.04921       1.5059       0.543317         2     
reco:opflash:OpFlashFinder                            0.00198615    0.00323243    0.00447871    0.00323243    0.00124628        2     
reco:rns:RandomNumberSaver                            2.9014e-05    0.000175701   0.000322388   0.000175701   0.000146687       2     
reco:anglereconue:NuAngularReco                       0.00137212    0.00155455    0.00173697    0.00155455    0.000182424       2     
reco:anglereconumu:NuAngularReco                      0.000377224   0.00101368    0.00165014    0.00101368    0.000636459       2     
reco:anglereconuepfps:NuAngularReco                   0.00179133    0.00233134    0.00287135    0.00233134    0.000540011       2     
reco:anglereconumupfps:NuAngularReco                  0.000450178   0.00111457    0.00177896    0.00111457    0.000664389       2     
[art]:TriggerResults:TriggerResultInserter            1.2047e-05    3.4223e-05    5.6399e-05    3.4223e-05    2.2176e-05        2     
end_path:out1:RootOutput                               5.161e-06    1.17425e-05   1.8324e-05    1.17425e-05   6.5815e-06        2     
end_path:out1:RootOutput(write)                        0.247685      0.312713      0.377741      0.312713      0.0650283        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6422.01 MB
  Peak resident set size usage (VmHWM): 1609.12 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:37:59 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6418365 subRun: 1 event: 47303
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-ac06-723a-6685-6738.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_473_20231203T111345Z_gen_g4_detsim_hitreco__20240508T044205Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_473_20231203T111345Z_gen_g4_detsim_hitreco__20240508T044205Z_reco2_reco_2024-11-23T_073511Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:26:01 UTC       justIN version: 01.01.09