Jobsub ID 301489.100@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301489.100@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:05:17 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:28:21 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:20:58 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (161s) |
| CPU time | 2m (152s = 94%) |
| Outputting started | 2024-11-22 23:23:40 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2.root
|
| Finished | 2024-11-22 23:28:21 |
| Saved logs | justin-logs:301489.100-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
gTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:23:36 CET run: 66009532 subRun: 1 event: 72703
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:23:39 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/15/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00154706 18.865 31.6579 24.9354 13.6179 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00111386 0.00131284 0.00154706 0.00127759 0.0001786 3
reco:linecluster:LineCluster 0.0569185 0.0984819 0.135872 0.102655 0.0323675 3
reco:trajcluster:TrajCluster 0.183718 0.4271 0.866186 0.231395 0.311091 3
reco:pandora:StandardPandora 2.40376 3.08453 3.63339 3.21644 0.510588 3
reco:pandoraTrack:LArPandoraTrackCreation 0.00923116 0.0168895 0.031743 0.00969425 0.0105047 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.00993841 0.0521585 0.126695 0.0198422 0.0528601 3
reco:pandoracalo:Calorimetry 0.00780198 0.0460959 0.117592 0.0128937 0.0505981 3
reco:pandorapid:Chi2ParticleID 0.000241138 0.00286564 0.00661112 0.00174465 0.00271865 3
reco:trkshowersplit:TrackShowerHits 0.00765577 0.0209242 0.045443 0.00967374 0.017357 3
reco:pmtrack:PMAlgTrackMaker 0.695922 2.54823 5.57971 1.36905 2.16113 3
reco:pmtrackcalo:Calorimetry 0.00356099 0.0208666 0.0539632 0.0050758 0.0234109 3
reco:pmtrackpid:Chi2ParticleID 0.000339652 0.000521142 0.00085928 0.000364495 0.000239314 3
reco:pmtrajfit:PMAlgTrajFitter 2.8481 3.42469 4.37191 3.05405 0.675044 3
reco:pmtrajfitcalo:Calorimetry 0.00960353 0.0263582 0.0593309 0.0101401 0.0233163 3
reco:pmtrajfitpid:Chi2ParticleID 0.000221716 0.000298759 0.000390023 0.000284539 6.94429e-05 3
reco:pmtracktc:PMAlgTrackMaker 0.826099 2.87206 5.82258 1.96751 2.13773 3
reco:pmtracktccalo:Calorimetry 0.00388925 0.0185653 0.0471243 0.00468226 0.0201969 3
reco:pmtracktcpid:Chi2ParticleID 0.000302358 0.00048515 0.000787456 0.000365636 0.000215318 3
reco:pmtrajfittc:PMAlgTrajFitter 1.19376 2.543 5.22906 1.20617 1.89934 3
reco:pmtrajfittccalo:Calorimetry 0.00743428 0.0244418 0.055258 0.0106332 0.0218294 3
reco:pmtrajfittcpid:Chi2ParticleID 0.000442576 0.000668887 0.00104931 0.000514775 0.00027061 3
reco:blurredcluster:BlurredClustering 0.915934 1.16684 1.41774 1.16684 0.250901 2
reco:emtrkmichelid:EmTrackMichelId 11.2312 12.8998 14.5683 12.8998 1.66851 2
reco:emshower:EMShower 0.0685175 0.0885709 0.108624 0.0885709 0.0200535 2
reco:cvnmap:CVNMapper 0.025613 0.0297758 0.0339386 0.0297758 0.0041628 2
reco:cvneva:CVNEvaluator 1.15799 1.1805 1.203 1.1805 0.0225043 2
reco:energyrecnumu:EnergyReco 0.0114314 0.0139612 0.016491 0.0139612 0.00252979 2
reco:energyrecnue:EnergyReco 0.00211427 0.00226815 0.00242203 0.00226815 0.000153878 2
reco:energyrecnc:EnergyReco 0.00397615 0.00439929 0.00482242 0.00439929 0.000423134 2
reco:energyrecnumurange:EnergyReco 0.00212473 0.0082497 0.0143747 0.0082497 0.00612497 2
reco:energyrecnumumcs:EnergyReco 0.00485485 0.0112213 0.0175877 0.0112213 0.00636644 2
reco:opdec:Deconvolution 0.223006 0.245426 0.267847 0.245426 0.0224204 2
reco:ophitspe:OpHitFinderDeco 1.75538 1.8832 2.01102 1.8832 0.127819 2
reco:opflash:OpFlashFinder 0.00208512 0.00335854 0.00463196 0.00335854 0.00127342 2
reco:rns:RandomNumberSaver 3.0946e-05 0.00016373 0.000296515 0.00016373 0.000132785 2
reco:anglereconue:NuAngularReco 0.00173668 0.00190148 0.00206627 0.00190148 0.000164795 2
reco:anglereconumu:NuAngularReco 0.000942666 0.0013061 0.00166953 0.0013061 0.000363433 2
reco:anglereconuepfps:NuAngularReco 0.00177596 0.00251296 0.00324996 0.00251296 0.000737002 2
reco:anglereconumupfps:NuAngularReco 0.00103 0.00139217 0.00175434 0.00139217 0.00036217 2
[art]:TriggerResults:TriggerResultInserter 1.4482e-05 3.1089e-05 4.7696e-05 3.1089e-05 1.6607e-05 2
end_path:out1:RootOutput 6.744e-06 1.1847e-05 1.695e-05 1.1847e-05 5.103e-06 2
end_path:out1:RootOutput(write) 0.327664 0.338562 0.34946 0.338562 0.0108984 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6378.74 MB
Peak resident set size usage (VmHWM): 1606.96 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:23:39 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 66009532 subRun: 1 event: 72703
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1933-a630-b3a8-4a4a.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2_reco_2024-11-22T_232102Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 