Jobsub ID 108605.26@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108605.26@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:35:25 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:44:15 |
| From worker node | Hostname | wn-choc-026.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:39:31 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 1m (104s) |
| CPU time | 1m (98s = 94%) |
| Outputting started | 2024-11-22 23:41:16 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2.root
|
| Finished | 2024-11-22 23:44:15 |
| Saved logs | justin-logs:108605.26-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
AlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:41:15 CET run: 74517882 subRun: 1 event: 73502
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:41:15 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/0e/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00112843 7.95068 15.9002 7.95068 7.94956 2
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00112843 0.00120043 0.00127243 0.00120043 7.2004e-05 2
reco:linecluster:LineCluster 0.0819742 0.104362 0.126749 0.104362 0.0223875 2
reco:trajcluster:TrajCluster 0.0329403 0.221295 0.40965 0.221295 0.188355 2
reco:pandora:StandardPandora 3.1293 3.18263 3.23596 3.18263 0.0533301 2
reco:pandoraTrack:LArPandoraTrackCreation 0.00666926 0.00854497 0.0104207 0.00854497 0.00187571 2
reco:pandoraShower:LArPandoraModularShowerCreation 0.0146408 0.0159763 0.0173118 0.0159763 0.00133549 2
reco:pandoracalo:Calorimetry 0.00453694 0.00980213 0.0150673 0.00980213 0.00526519 2
reco:pandorapid:Chi2ParticleID 0.000244179 0.000841602 0.00143903 0.000841602 0.000597423 2
reco:trkshowersplit:TrackShowerHits 0.0031034 0.0148812 0.026659 0.0148812 0.0117778 2
reco:pmtrack:PMAlgTrackMaker 0.155931 1.99369 3.83145 1.99369 1.83776 2
reco:pmtrackcalo:Calorimetry 0.000901523 0.00501046 0.0091194 0.00501046 0.00410894 2
reco:pmtrackpid:Chi2ParticleID 0.000169655 0.000319777 0.000469899 0.000319777 0.000150122 2
reco:pmtrajfit:PMAlgTrajFitter 0.303671 1.02994 1.75622 1.02994 0.726273 2
reco:pmtrajfitcalo:Calorimetry 0.0017366 0.00920694 0.0166773 0.00920694 0.00747034 2
reco:pmtrajfitpid:Chi2ParticleID 0.000185695 0.000279209 0.000372723 0.000279209 9.3514e-05 2
reco:pmtracktc:PMAlgTrackMaker 0.495549 1.55794 2.62034 1.55794 1.0624 2
reco:pmtracktccalo:Calorimetry 0.00160204 0.00684914 0.0120962 0.00684914 0.0052471 2
reco:pmtracktcpid:Chi2ParticleID 0.000215286 0.000367965 0.000520644 0.000367965 0.000152679 2
reco:pmtrajfittc:PMAlgTrajFitter 0.484856 0.807355 1.12985 0.807355 0.322499 2
reco:pmtrajfittccalo:Calorimetry 0.00188528 0.00538162 0.00887796 0.00538162 0.00349634 2
reco:pmtrajfittcpid:Chi2ParticleID 0.00022373 0.000337383 0.000451035 0.000337383 0.000113652 2
reco:blurredcluster:BlurredClustering 1.38173 1.38173 1.38173 1.38173 0 1
reco:emtrkmichelid:EmTrackMichelId 5.52132 5.52132 5.52132 5.52132 0 1
reco:emshower:EMShower 0.101848 0.101848 0.101848 0.101848 0 1
reco:cvnmap:CVNMapper 0.0223923 0.0223923 0.0223923 0.0223923 0 1
reco:cvneva:CVNEvaluator 1.23778 1.23778 1.23778 1.23778 0 1
reco:energyrecnumu:EnergyReco 0.00842525 0.00842525 0.00842525 0.00842525 0 1
reco:energyrecnue:EnergyReco 0.000896487 0.000896487 0.000896487 0.000896487 0 1
reco:energyrecnc:EnergyReco 0.00340409 0.00340409 0.00340409 0.00340409 0 1
reco:energyrecnumurange:EnergyReco 0.00189051 0.00189051 0.00189051 0.00189051 0 1
reco:energyrecnumumcs:EnergyReco 0.00471073 0.00471073 0.00471073 0.00471073 0 1
reco:opdec:Deconvolution 0.28125 0.28125 0.28125 0.28125 0 1
reco:ophitspe:OpHitFinderDeco 2.13447 2.13447 2.13447 2.13447 0 1
reco:opflash:OpFlashFinder 0.00473206 0.00473206 0.00473206 0.00473206 0 1
reco:rns:RandomNumberSaver 0.000270655 0.000270655 0.000270655 0.000270655 0 1
reco:anglereconue:NuAngularReco 0.00132704 0.00132704 0.00132704 0.00132704 0 1
reco:anglereconumu:NuAngularReco 0.000520554 0.000520554 0.000520554 0.000520554 0 1
reco:anglereconuepfps:NuAngularReco 0.00260246 0.00260246 0.00260246 0.00260246 0 1
reco:anglereconumupfps:NuAngularReco 0.000580795 0.000580795 0.000580795 0.000580795 0 1
[art]:TriggerResults:TriggerResultInserter 4.3655e-05 4.3655e-05 4.3655e-05 4.3655e-05 0 1
end_path:out1:RootOutput 1.342e-05 1.342e-05 1.342e-05 1.342e-05 0 1
end_path:out1:RootOutput(write) 0.310948 0.310948 0.310948 0.310948 0 1
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6425.2 MB
Peak resident set size usage (VmHWM): 1644.1 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:41:16 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74517882 subRun: 1 event: 73502
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-31fe-519f-7270-dc90.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2_reco_2024-11-22T_233934Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 