Jobsub ID 301451.186@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301451.186@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:37:24 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:13:45 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:06:35 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_911_20231202T085434Z_gen_g4_detsim_hitreco__20240507T205846Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (137s) |
| CPU time | 2m (132s = 96%) |
| Outputting started | 2024-11-22 23:08:53 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_911_20231202T085434Z_gen_g4_detsim_hitreco__20240507T205846Z_reco2.root
|
| Finished | 2024-11-22 23:13:45 |
| Saved logs | justin-logs:301451.186-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:50 CET run: 6400837 subRun: 1 event: 91102
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:08:52 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/42/9e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_911_20231202T085434Z_gen_g4_detsim_hitreco__20240507T205846Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00118157 3.09005 6.17892 3.09005 3.08887 2
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00118157 0.0012989 0.00141623 0.0012989 0.000117329 2
reco:linecluster:LineCluster 0.0227644 0.183047 0.343329 0.183047 0.160282 2
reco:trajcluster:TrajCluster 0.0168583 2.38616 4.75546 2.38616 2.3693 2
reco:pandora:StandardPandora 3.10131 3.92858 4.75584 3.92858 0.827264 2
reco:pandoraTrack:LArPandoraTrackCreation 0.00586265 0.0164401 0.0270176 0.0164401 0.0105775 2
reco:pandoraShower:LArPandoraModularShowerCreation 0.0164251 0.0377215 0.0590178 0.0377215 0.0212964 2
reco:pandoracalo:Calorimetry 0.00449583 0.0385921 0.0726883 0.0385921 0.0340963 2
reco:pandorapid:Chi2ParticleID 0.000338753 0.00105314 0.00176753 0.00105314 0.000714389 2
reco:trkshowersplit:TrackShowerHits 0.00242338 0.0425408 0.0826583 0.0425408 0.0401175 2
reco:pmtrack:PMAlgTrackMaker 0.0548139 6.69092 13.327 6.69092 6.63611 2
reco:pmtrackcalo:Calorimetry 0.000575479 0.0400147 0.0794539 0.0400147 0.0394392 2
reco:pmtrackpid:Chi2ParticleID 0.000112588 0.000724752 0.00133692 0.000724752 0.000612164 2
reco:pmtrajfit:PMAlgTrajFitter 0.0987277 2.44834 4.79795 2.44834 2.34961 2
reco:pmtrajfitcalo:Calorimetry 0.000546479 0.0383256 0.0761047 0.0383256 0.0377791 2
reco:pmtrajfitpid:Chi2ParticleID 0.000110148 0.000282015 0.000453883 0.000282015 0.000171868 2
reco:pmtracktc:PMAlgTrackMaker 0.0174758 9.55286 19.0882 9.55286 9.53538 2
reco:pmtracktccalo:Calorimetry 0.00041445 0.0123016 0.0241888 0.0123016 0.0118872 2
reco:pmtracktcpid:Chi2ParticleID 9.9521e-05 0.000520021 0.000940521 0.000520021 0.0004205 2
reco:pmtrajfittc:PMAlgTrajFitter 0.0984476 3.12559 6.15272 3.12559 3.02714 2
reco:pmtrajfittccalo:Calorimetry 0.000422358 0.0372521 0.0740818 0.0372521 0.0368297 2
reco:pmtrajfittcpid:Chi2ParticleID 0.000101057 0.00073435 0.00136764 0.00073435 0.000633293 2
reco:blurredcluster:BlurredClustering 0.0355555 0.0355555 0.0355555 0.0355555 0 1
reco:emtrkmichelid:EmTrackMichelId 0.758741 0.758741 0.758741 0.758741 0 1
reco:emshower:EMShower 0.00498977 0.00498977 0.00498977 0.00498977 0 1
reco:cvnmap:CVNMapper 0.000273535 0.000273535 0.000273535 0.000273535 0 1
reco:cvneva:CVNEvaluator 0.000266986 0.000266986 0.000266986 0.000266986 0 1
reco:energyrecnumu:EnergyReco 0.00666428 0.00666428 0.00666428 0.00666428 0 1
reco:energyrecnue:EnergyReco 0.000508014 0.000508014 0.000508014 0.000508014 0 1
reco:energyrecnc:EnergyReco 0.00301068 0.00301068 0.00301068 0.00301068 0 1
reco:energyrecnumurange:EnergyReco 0.000673747 0.000673747 0.000673747 0.000673747 0 1
reco:energyrecnumumcs:EnergyReco 0.00331712 0.00331712 0.00331712 0.00331712 0 1
reco:opdec:Deconvolution 0.2128 0.2128 0.2128 0.2128 0 1
reco:ophitspe:OpHitFinderDeco 1.58767 1.58767 1.58767 1.58767 0 1
reco:opflash:OpFlashFinder 0.0048675 0.0048675 0.0048675 0.0048675 0 1
reco:rns:RandomNumberSaver 0.000368853 0.000368853 0.000368853 0.000368853 0 1
reco:anglereconue:NuAngularReco 0.00146174 0.00146174 0.00146174 0.00146174 0 1
reco:anglereconumu:NuAngularReco 0.000342794 0.000342794 0.000342794 0.000342794 0 1
reco:anglereconuepfps:NuAngularReco 0.00314684 0.00314684 0.00314684 0.00314684 0 1
reco:anglereconumupfps:NuAngularReco 0.000375158 0.000375158 0.000375158 0.000375158 0 1
[art]:TriggerResults:TriggerResultInserter 6.0254e-05 6.0254e-05 6.0254e-05 6.0254e-05 0 1
end_path:out1:RootOutput 1.902e-05 1.902e-05 1.902e-05 1.902e-05 0 1
end_path:out1:RootOutput(write) 0.106351 0.106351 0.106351 0.106351 0 1
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6406.29 MB
Peak resident set size usage (VmHWM): 1581.41 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:08:53 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6400837 subRun: 1 event: 91102
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-f7c6-2cdd-1423-b911.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_911_20231202T085434Z_gen_g4_detsim_hitreco__20240507T205846Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_911_20231202T085434Z_gen_g4_detsim_hitreco__20240507T205846Z_reco2_reco_2024-11-22T_230638Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 