Jobsub ID 108519.159@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108519.159@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:43:09 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 22:56:57 |
| From worker node | Hostname | wn-choc-026.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 22:53:22 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_241_20231204T033517Z_gen_g4_detsim_hitreco__20240509T204411Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 3m (181s) |
| CPU time | 2m (175s = 96%) |
| Outputting started | 2024-11-22 22:56:24 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_241_20231204T033517Z_gen_g4_detsim_hitreco__20240509T204411Z_reco2.root
|
| Finished | 2024-11-22 22:56:57 |
| Saved logs | justin-logs:108519.159-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
G
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
1st wire C:0 T:17 P:2 W:729 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:56:23 CET run: 6420853 subRun: 1 event: 24103
TuneFullTree failed.
%MSG
22-Nov-2024 23:56:23 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ab/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_241_20231204T033517Z_gen_g4_detsim_hitreco__20240509T204411Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00144298 32.1873 88.2855 8.27497 39.811 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00144298 0.00162345 0.00182942 0.00159796 0.000158791 3
reco:linecluster:LineCluster 0.0119107 0.0623597 0.135087 0.0400819 0.052696 3
reco:trajcluster:TrajCluster 0.00333213 0.285792 0.761336 0.0927091 0.338234 3
reco:pandora:StandardPandora 1.76258 2.86834 4.66266 2.17978 1.28016 3
reco:pandoraTrack:LArPandoraTrackCreation 0.00037996 0.0125483 0.0334349 0.00382994 0.0148361 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.000849477 0.0401303 0.107161 0.0123803 0.0476312 3
reco:pandoracalo:Calorimetry 0.000231975 0.0155265 0.0431443 0.00320318 0.0195664 3
reco:pandorapid:Chi2ParticleID 7.8202e-05 0.000702839 0.00186277 0.000167546 0.000821005 3
reco:trkshowersplit:TrackShowerHits 0.000372601 0.0106978 0.0276549 0.00406592 0.0120849 3
reco:pmtrack:PMAlgTrackMaker 0.064873 2.75306 6.34508 1.84923 2.64234 3
reco:pmtrackcalo:Calorimetry 0.00042812 0.0110514 0.030311 0.00241514 0.0136427 3
reco:pmtrackpid:Chi2ParticleID 0.000109019 0.000338674 0.000673968 0.000233035 0.000242434 3
reco:pmtrajfit:PMAlgTrajFitter 0.041008 1.90977 5.08283 0.605464 2.2555 3
reco:pmtrajfitcalo:Calorimetry 0.000354868 0.0124898 0.0337442 0.00337031 0.0150795 3
reco:pmtrajfitpid:Chi2ParticleID 9.4587e-05 0.000217884 0.000375867 0.000183197 0.000117422 3
reco:pmtracktc:PMAlgTrackMaker 0.0390493 2.92725 8.3506 0.392094 3.8376 3
reco:pmtracktccalo:Calorimetry 0.000469251 0.0162712 0.0432369 0.00510754 0.0191614 3
reco:pmtracktcpid:Chi2ParticleID 0.00011584 0.000384864 0.000753336 0.000285417 0.000269589 3
reco:pmtrajfittc:PMAlgTrajFitter 0.086189 1.98592 5.0915 0.780078 2.21417 3
reco:pmtrajfittccalo:Calorimetry 0.00042802 0.0152752 0.0407815 0.00461618 0.0181165 3
reco:pmtrajfittcpid:Chi2ParticleID 0.000106199 0.000380286 0.000762774 0.000271884 0.00027879 3
reco:blurredcluster:BlurredClustering 1.06064 2.55517 4.0497 2.55517 1.49453 2
reco:emtrkmichelid:EmTrackMichelId 1.50712 23.2156 44.924 23.2156 21.7084 2
reco:emshower:EMShower 0.0100704 0.0170677 0.0240651 0.0170677 0.00699734 2
reco:cvnmap:CVNMapper 0.0105362 0.0242876 0.038039 0.0242876 0.0137514 2
reco:cvneva:CVNEvaluator 1.16789 1.29815 1.42841 1.29815 0.130259 2
reco:energyrecnumu:EnergyReco 0.00150157 0.612333 1.22316 0.612333 0.610831 2
reco:energyrecnue:EnergyReco 0.000224142 0.00511456 0.010005 0.00511456 0.00489041 2
reco:energyrecnc:EnergyReco 0.00258399 0.00720006 0.0118161 0.00720006 0.00461606 2
reco:energyrecnumurange:EnergyReco 0.000391319 0.610015 1.21964 0.610015 0.609624 2
reco:energyrecnumumcs:EnergyReco 0.00305021 0.613563 1.22408 0.613563 0.610513 2
reco:opdec:Deconvolution 0.205059 0.275767 0.346474 0.275767 0.0707077 2
reco:ophitspe:OpHitFinderDeco 1.57686 2.00987 2.44288 2.00987 0.433009 2
reco:opflash:OpFlashFinder 0.00151849 0.00345381 0.00538914 0.00345381 0.00193532 2
reco:rns:RandomNumberSaver 2.0902e-05 0.00014847 0.000276037 0.00014847 0.000127567 2
reco:anglereconue:NuAngularReco 0.000248335 0.00425708 0.00826582 0.00425708 0.00400874 2
reco:anglereconumu:NuAngularReco 0.000139864 0.00365157 0.00716328 0.00365157 0.00351171 2
reco:anglereconuepfps:NuAngularReco 0.000243553 0.00490731 0.00957107 0.00490731 0.00466376 2
reco:anglereconumupfps:NuAngularReco 0.000229323 0.00384933 0.00746934 0.00384933 0.00362001 2
[art]:TriggerResults:TriggerResultInserter 1.3812e-05 3.5306e-05 5.68e-05 3.5306e-05 2.1494e-05 2
end_path:out1:RootOutput 2.989e-06 1.17655e-05 2.0542e-05 1.17655e-05 8.7765e-06 2
end_path:out1:RootOutput(write) 0.279284 0.400509 0.521734 0.400509 0.121225 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6416.51 MB
Peak resident set size usage (VmHWM): 1600.35 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 22-Nov-2024 23:56:23 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6420853 subRun: 1 event: 24103
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-c87e-430f-f7e1-f53a.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_241_20231204T033517Z_gen_g4_detsim_hitreco__20240509T204411Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_241_20231204T033517Z_gen_g4_detsim_hitreco__20240509T204411Z_reco2_reco_2024-11-22T_225325Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 