Jobsub ID 301553.136@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301553.136@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:36:34 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-23 00:03:35 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:51:24 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_40_20231201T120912Z_gen_g4_detsim_hitreco__20240507T194724Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 7m (442s) |
| CPU time | 7m (433s = 97%) |
| Outputting started | 2024-11-22 23:58:47 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_40_20231201T120912Z_gen_g4_detsim_hitreco__20240507T194724Z_reco2.root
|
| Finished | 2024-11-23 00:03:35 |
| Saved logs | justin-logs:301553.136-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:58:45 CET run: 74481120 subRun: 1 event: 4005
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:58:46 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/38/3c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_40_20231201T120912Z_gen_g4_detsim_hitreco__20240507T194724Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00131344 9.86448 27.8309 7.2476 9.78789 5
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000883674 0.00118469 0.00142508 0.00131344 0.000233441 5
reco:linecluster:LineCluster 0.0109397 1.22836 6.00547 0.0398993 2.38864 5
reco:trajcluster:TrajCluster 0.0016512 9.24824 46.0865 0.0292379 18.4192 5
reco:pandora:StandardPandora 1.37432 9.01289 36.4365 2.56406 13.7269 5
reco:pandoraTrack:LArPandoraTrackCreation 0.000193845 0.0197325 0.0844616 0.00607353 0.0324762 5
reco:pandoraShower:LArPandoraModularShowerCreation 0.000688683 0.0176564 0.0601978 0.00833293 0.0219804 5
reco:pandoracalo:Calorimetry 0.000138365 0.188598 0.930148 0.00530922 0.370784 5
reco:pandorapid:Chi2ParticleID 5.9887e-05 0.000506669 0.00166372 0.000163249 0.000598009 5
reco:trkshowersplit:TrackShowerHits 0.000190797 0.109457 0.539637 0.00269694 0.215096 5
reco:pmtrack:PMAlgTrackMaker 0.00176927 11.6076 56.4592 0.237272 22.4294 5
reco:pmtrackcalo:Calorimetry 0.000105259 0.100089 0.492276 0.00129065 0.196104 5
reco:pmtrackpid:Chi2ParticleID 4.6914e-05 0.00139018 0.00615302 0.000191691 0.00238405 5
reco:pmtrajfit:PMAlgTrajFitter 0.0579115 7.16067 33.2997 0.447103 13.0861 5
reco:pmtrajfitcalo:Calorimetry 0.000308398 0.189599 0.93623 0.00235258 0.373326 5
reco:pmtrajfitpid:Chi2ParticleID 8.7494e-05 0.000286363 0.000750273 0.000175245 0.000241991 5
reco:pmtracktc:PMAlgTrackMaker 0.0144977 18.6512 90.844 0.241883 36.1033 5
reco:pmtracktccalo:Calorimetry 0.000221823 0.0507694 0.245648 0.000813233 0.0974654 5
reco:pmtracktcpid:Chi2ParticleID 8.0978e-05 0.000718665 0.0027769 0.000146599 0.00103684 5
reco:pmtrajfittc:PMAlgTrajFitter 0.110414 8.35825 39.1777 0.470761 15.4206 5
reco:pmtrajfittccalo:Calorimetry 0.000285935 0.128768 0.633506 0.00106497 0.252386 5
reco:pmtrajfittcpid:Chi2ParticleID 8.4983e-05 0.000991125 0.0040339 0.000166972 0.00152881 5
reco:blurredcluster:BlurredClustering 0.0261565 0.626296 1.48166 0.498685 0.550105 4
reco:emtrkmichelid:EmTrackMichelId 0.427629 4.55034 12.7921 2.49082 4.87346 4
reco:emshower:EMShower 0.000501617 0.047462 0.158753 0.0152967 0.0649268 4
reco:cvnmap:CVNMapper 5.0536e-05 0.0142864 0.0342046 0.0114451 0.0124201 4
reco:cvneva:CVNEvaluator 2.3648e-05 0.888481 1.20312 1.17539 0.513124 4
reco:energyrecnumu:EnergyReco 0.00276691 0.00505413 0.00720732 0.00512114 0.00205683 4
reco:energyrecnue:EnergyReco 0.000207466 0.000764687 0.00186777 0.000491757 0.000657387 4
reco:energyrecnc:EnergyReco 0.00254684 0.00314397 0.00420551 0.00291176 0.000645837 4
reco:energyrecnumurange:EnergyReco 0.00148473 0.00298432 0.00583736 0.0023076 0.00169172 4
reco:energyrecnumumcs:EnergyReco 0.00271407 0.00509847 0.00874343 0.00446819 0.00222905 4
reco:opdec:Deconvolution 0.0674446 0.169188 0.26639 0.171458 0.0704334 4
reco:ophitspe:OpHitFinderDeco 0.454823 1.25209 1.99719 1.27818 0.546528 4
reco:opflash:OpFlashFinder 0.000461165 0.00197908 0.00430175 0.00157671 0.00143985 4
reco:rns:RandomNumberSaver 2.1178e-05 9.72988e-05 0.000305133 3.1442e-05 0.000120148 4
reco:anglereconue:NuAngularReco 0.000190218 0.000783984 0.00135751 0.000794104 0.000557292 4
reco:anglereconumu:NuAngularReco 0.000146725 0.000488215 0.00125055 0.00027779 0.000448274 4
reco:anglereconuepfps:NuAngularReco 0.000176209 0.0011377 0.00269932 0.000837629 0.00101832 4
reco:anglereconumupfps:NuAngularReco 0.000163381 0.000557752 0.0013408 0.000363414 0.000463528 4
[art]:TriggerResults:TriggerResultInserter 1.2583e-05 2.2235e-05 4.7087e-05 1.4635e-05 1.44086e-05 4
end_path:out1:RootOutput 2.848e-06 7.05525e-06 1.6119e-05 4.627e-06 5.40231e-06 4
end_path:out1:RootOutput(write) 0.0379831 0.223592 0.344779 0.255804 0.113468 4
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6425.33 MB
Peak resident set size usage (VmHWM): 1670.37 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:58:46 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74481120 subRun: 1 event: 4005
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-04a0-629e-70a0-3c3c.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_40_20231201T120912Z_gen_g4_detsim_hitreco__20240507T194724Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_40_20231201T120912Z_gen_g4_detsim_hitreco__20240507T194724Z_reco2_reco_2024-11-22T_235127Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 