Jobsub ID 301451.106@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301451.106@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:37:24 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:05:57 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:00:50 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_565_20231202T114117Z_gen_g4_detsim_hitreco__20240507T215905Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 4m (256s) |
| CPU time | 4m (249s = 97%) |
| Outputting started | 2024-11-22 23:05:07 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_565_20231202T114117Z_gen_g4_detsim_hitreco__20240507T215905Z_reco2.root
|
| Finished | 2024-11-22 23:05:57 |
| Saved logs | justin-logs:301451.106-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
Check: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:05:04 CET run: 6405486 subRun: 1 event: 56509
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:05:06 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/25/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_565_20231202T114117Z_gen_g4_detsim_hitreco__20240507T215905Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00132287 12.9174 30.8344 11.0921 8.74755 9
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00107201 0.00126824 0.00151216 0.0012846 0.000128509 9
reco:linecluster:LineCluster 0.0108907 0.0586405 0.277891 0.0233088 0.0800121 9
reco:trajcluster:TrajCluster 0.00205937 0.3816 2.85873 0.0291591 0.880868 9
reco:pandora:StandardPandora 1.41149 2.27307 5.7608 1.78292 1.34193 9
reco:pandoraTrack:LArPandoraTrackCreation 0.000166658 0.00641447 0.0277499 0.00235275 0.00799063 9
reco:pandoraShower:LArPandoraModularShowerCreation 0.000502372 0.0149864 0.0398294 0.0110192 0.0121399 9
reco:pandoracalo:Calorimetry 0.000119072 0.0248702 0.194032 0.00239186 0.0598682 9
reco:pandorapid:Chi2ParticleID 4.7959e-05 0.00033073 0.00169819 0.000165798 0.000486998 9
reco:trkshowersplit:TrackShowerHits 0.00019137 0.0132924 0.096176 0.00168035 0.0294938 9
reco:pmtrack:PMAlgTrackMaker 0.00240521 2.00547 13.3424 0.343278 4.05248 9
reco:pmtrackcalo:Calorimetry 0.000185498 0.012158 0.0842502 0.00188828 0.0256423 9
reco:pmtrackpid:Chi2ParticleID 7.0304e-05 0.000348223 0.00131373 0.000214994 0.000358946 9
reco:pmtrajfit:PMAlgTrajFitter 0.00581216 2.36055 13.5802 0.463403 4.10668 9
reco:pmtrajfitcalo:Calorimetry 0.000181959 0.024732 0.194062 0.00230837 0.0599316 9
reco:pmtrajfitpid:Chi2ParticleID 6.4458e-05 0.00019483 0.000344493 0.000178197 7.73087e-05 9
reco:pmtracktc:PMAlgTrackMaker 0.0359758 2.39459 16.6826 0.450061 5.07905 9
reco:pmtracktccalo:Calorimetry 0.000323307 0.0084823 0.0540667 0.00186588 0.016378 9
reco:pmtracktcpid:Chi2ParticleID 8.7004e-05 0.000289516 0.00109113 0.000159217 0.000300861 9
reco:pmtrajfittc:PMAlgTrajFitter 0.00916589 1.25792 6.61009 0.518562 1.94681 9
reco:pmtrajfittccalo:Calorimetry 0.000183718 0.0140326 0.0998413 0.00197955 0.0305147 9
reco:pmtrajfittcpid:Chi2ParticleID 6.4344e-05 0.000375039 0.00160745 0.0002109 0.000452399 9
reco:blurredcluster:BlurredClustering 0.0945488 0.691712 1.21631 0.687766 0.393877 8
reco:emtrkmichelid:EmTrackMichelId 0.890116 6.36096 16.7452 4.64523 4.84376 8
reco:emshower:EMShower 0.000830761 0.00709124 0.0207025 0.00536416 0.00657465 8
reco:cvnmap:CVNMapper 3.6084e-05 0.0135388 0.0253893 0.0125886 0.00713717 8
reco:cvneva:CVNEvaluator 2.1748e-05 1.05834 1.31827 1.16912 0.403442 8
reco:energyrecnumu:EnergyReco 0.00210129 0.0116518 0.0551315 0.0053922 0.016693 8
reco:energyrecnue:EnergyReco 0.000208872 0.000879542 0.00212388 0.000733054 0.000544528 8
reco:energyrecnc:EnergyReco 0.00268688 0.00328417 0.00457329 0.00306084 0.000537028 8
reco:energyrecnumurange:EnergyReco 0.000940522 0.00862151 0.0550256 0.00194793 0.0175579 8
reco:energyrecnumumcs:EnergyReco 0.00212385 0.00992073 0.0538812 0.0035912 0.0166657 8
reco:opdec:Deconvolution 0.134476 0.167631 0.220438 0.172266 0.0273957 8
reco:ophitspe:OpHitFinderDeco 0.968883 1.21884 1.60305 1.22249 0.202698 8
reco:opflash:OpFlashFinder 0.00090411 0.00159045 0.00461021 0.00122849 0.00115719 8
reco:rns:RandomNumberSaver 2.0451e-05 6.5925e-05 0.000324876 2.8867e-05 9.80111e-05 8
reco:anglereconue:NuAngularReco 0.000245456 0.000719266 0.00143491 0.000563598 0.000421308 8
reco:anglereconumu:NuAngularReco 0.000153538 0.000493703 0.0012781 0.000408728 0.000321163 8
reco:anglereconuepfps:NuAngularReco 0.000205688 0.000986465 0.00371948 0.000522501 0.00108442 8
reco:anglereconumupfps:NuAngularReco 0.000165091 0.000585948 0.00138782 0.000501527 0.000339871 8
[art]:TriggerResults:TriggerResultInserter 1.3543e-05 2.0939e-05 6.537e-05 1.4297e-05 1.68392e-05 8
end_path:out1:RootOutput 2.969e-06 6.89438e-06 1.8759e-05 5.754e-06 4.91324e-06 8
end_path:out1:RootOutput(write) 0.0899413 0.249235 0.337696 0.260591 0.067711 8
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6402.7 MB
Peak resident set size usage (VmHWM): 1621.64 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:05:06 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6405486 subRun: 1 event: 56509
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1b7c-54a1-b5ef-8d36.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_565_20231202T114117Z_gen_g4_detsim_hitreco__20240507T215905Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_565_20231202T114117Z_gen_g4_detsim_hitreco__20240507T215905Z_reco2_reco_2024-11-22T_230053Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 