Jobsub ID 301547.86@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301547.86@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:29:37 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:53:34 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:43:03 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (143s) |
| CPU time | 2m (137s = 95%) |
| Outputting started | 2024-11-22 23:45:27 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2.root
|
| Finished | 2024-11-22 23:53:34 |
| Saved logs | justin-logs:301547.86-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
17882 subRun: 1 event: 10606
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:45:26 CET run: 74517882 subRun: 1 event: 10606
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:45:26 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/75/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.000966608 9.95409 20.8129 7.45521 8.19268 6
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000878171 0.00105986 0.00130552 0.00100165 0.000155276 6
reco:linecluster:LineCluster 0.011867 0.0370909 0.0739476 0.0306734 0.0220666 6
reco:trajcluster:TrajCluster 0.0025576 0.073417 0.218312 0.0236873 0.0842157 6
reco:pandora:StandardPandora 1.39332 1.98088 3.01394 1.82402 0.509697 6
reco:pandoraTrack:LArPandoraTrackCreation 0.000156675 0.00463205 0.0125145 0.00409595 0.00394497 6
reco:pandoraShower:LArPandoraModularShowerCreation 0.000216385 0.00905361 0.0278148 0.00683226 0.0089561 6
reco:pandoracalo:Calorimetry 0.000118953 0.00697217 0.0236454 0.00397084 0.00800364 6
reco:pandorapid:Chi2ParticleID 4.9016e-05 0.000322754 0.0013082 0.000142536 0.000442181 6
reco:trkshowersplit:TrackShowerHits 0.000168016 0.00397642 0.0158155 0.0020874 0.00537912 6
reco:pmtrack:PMAlgTrackMaker 0.00473851 0.699036 2.2906 0.523087 0.786152 6
reco:pmtrackcalo:Calorimetry 0.000108868 0.00130755 0.00349244 0.000856247 0.00121929 6
reco:pmtrackpid:Chi2ParticleID 4.3904e-05 0.000167156 0.000298974 0.000158652 9.42636e-05 6
reco:pmtrajfit:PMAlgTrajFitter 0.000263921 1.05412 3.39725 0.716313 1.16009 6
reco:pmtrajfitcalo:Calorimetry 9.045e-05 0.00466462 0.0102297 0.00418642 0.00422084 6
reco:pmtrajfitpid:Chi2ParticleID 3.8938e-05 0.000140065 0.000285518 0.00013685 7.68098e-05 6
reco:pmtracktc:PMAlgTrackMaker 0.00256166 0.509841 1.43167 0.419475 0.508276 6
reco:pmtracktccalo:Calorimetry 0.000150932 0.00145076 0.00334252 0.000920757 0.00133141 6
reco:pmtracktcpid:Chi2ParticleID 4.9804e-05 0.00018295 0.000347629 0.000167962 0.000112353 6
reco:pmtrajfittc:PMAlgTrajFitter 0.00483123 0.495757 1.21641 0.273317 0.515161 6
reco:pmtrajfittccalo:Calorimetry 0.000189857 0.00256753 0.00869662 0.000870509 0.00312827 6
reco:pmtrajfittcpid:Chi2ParticleID 6.5099e-05 0.0002254 0.000556961 0.000161762 0.000173416 6
reco:blurredcluster:BlurredClustering 0.211077 1.07547 1.99231 1.13057 0.582296 5
reco:emtrkmichelid:EmTrackMichelId 0.329354 4.35823 9.38848 3.03109 3.89401 5
reco:emshower:EMShower 0.000288433 0.0374731 0.116816 0.0141316 0.0443261 5
reco:cvnmap:CVNMapper 2.9926e-05 0.00891674 0.0197442 0.0106745 0.00774412 5
reco:cvneva:CVNEvaluator 3.2405e-05 0.699634 1.18255 1.13912 0.571344 5
reco:energyrecnumu:EnergyReco 0.00147099 0.0196813 0.0859618 0.00405242 0.0331557 5
reco:energyrecnue:EnergyReco 0.000359994 0.00114629 0.00265734 0.00104272 0.000829178 5
reco:energyrecnc:EnergyReco 0.00265712 0.0030982 0.00365256 0.00290892 0.000386683 5
reco:energyrecnumurange:EnergyReco 0.000262606 0.01765 0.0812957 0.00266309 0.0318414 5
reco:energyrecnumumcs:EnergyReco 0.00266323 0.0193292 0.0816604 0.00356355 0.0311854 5
reco:opdec:Deconvolution 0.0412648 0.139892 0.238662 0.123065 0.0763006 5
reco:ophitspe:OpHitFinderDeco 0.251944 1.17488 2.36631 0.883199 0.748707 5
reco:opflash:OpFlashFinder 0.000323118 0.00404546 0.0158328 0.00143428 0.00591104 5
reco:rns:RandomNumberSaver 2.0506e-05 8.60082e-05 0.000326644 2.7237e-05 0.000120353 5
reco:anglereconue:NuAngularReco 0.000181749 0.000608116 0.00117018 0.000663179 0.000336506 5
reco:anglereconumu:NuAngularReco 9.6171e-05 0.000343429 0.000588154 0.000242513 0.000195435 5
reco:anglereconuepfps:NuAngularReco 0.000145681 0.00213831 0.00854418 0.00067653 0.00321577 5
reco:anglereconumupfps:NuAngularReco 0.000136526 0.000425805 0.000686027 0.000324409 0.000213866 5
[art]:TriggerResults:TriggerResultInserter 1.2673e-05 2.30116e-05 5.8855e-05 1.5099e-05 1.79574e-05 5
end_path:out1:RootOutput 2.877e-06 7.4468e-06 2.2219e-05 3.345e-06 7.4632e-06 5
end_path:out1:RootOutput(write) 0.0271334 0.18455 0.307437 0.233975 0.123536 5
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6425.21 MB
Peak resident set size usage (VmHWM): 1644.2 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:45:26 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74517882 subRun: 1 event: 10606
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-00e0-4030-eb2c-a905.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2_reco_2024-11-22T_234306Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 