Jobsub ID 301489.157@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301489.157@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:05:17 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:39:07 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:23:47 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_390_20231118T092720Z_gen_g4_detsim_hitreco__20240503T051959Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 10m (602s) |
| CPU time | 9m (589s = 97%) |
| Outputting started | 2024-11-22 23:33:49 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_390_20231118T092720Z_gen_g4_detsim_hitreco__20240503T051959Z_reco2.root
|
| Finished | 2024-11-22 23:39:07 |
| Saved logs | justin-logs:301489.157-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
heck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:33:47 CET run: 6171714 subRun: 1 event: 78003
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:33:49 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/94/10/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_390_20231118T092720Z_gen_g4_detsim_hitreco__20240503T051959Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00135464 17.7179 43.0204 10.1319 18.3633 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00120987 0.00130436 0.00135464 0.00134857 6.68584e-05 3
reco:linecluster:LineCluster 0.0152091 4.19884 12.4853 0.0959865 5.85952 3
reco:trajcluster:TrajCluster 0.0119661 42.0311 125.25 0.830983 58.8459 3
reco:pandora:StandardPandora 2.18845 15.7089 41.4349 3.50339 18.1989 3
reco:pandoraTrack:LArPandoraTrackCreation 0.00123774 0.107858 0.30666 0.015677 0.140698 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.00200854 0.0752768 0.183399 0.0404233 0.0780455 3
reco:pandoracalo:Calorimetry 0.000964778 0.431341 1.27825 0.0148073 0.598882 3
reco:pandorapid:Chi2ParticleID 0.00012031 0.000896476 0.0015855 0.000983614 0.000601328 3
reco:trkshowersplit:TrackShowerHits 0.000755572 0.356485 1.05271 0.0159832 0.492348 3
reco:pmtrack:PMAlgTrackMaker 0.0617476 31.2575 89.8179 3.89292 41.438 3
reco:pmtrackcalo:Calorimetry 0.000925861 0.297002 0.874784 0.0152946 0.408596 3
reco:pmtrackpid:Chi2ParticleID 0.000161436 0.00167939 0.00439843 0.000478303 0.001927 3
reco:pmtrajfit:PMAlgTrajFitter 0.491802 14.0851 39.8623 1.90127 18.2363 3
reco:pmtrajfitcalo:Calorimetry 0.00159678 0.453152 1.34525 0.0126142 0.630822 3
reco:pmtrajfitpid:Chi2ParticleID 0.000152309 0.000743189 0.00180931 0.000267952 0.000755335 3
reco:pmtracktc:PMAlgTrackMaker 0.158786 40.0241 115.594 4.31975 53.4628 3
reco:pmtracktccalo:Calorimetry 0.00110565 0.123295 0.364768 0.00401125 0.170751 3
reco:pmtracktcpid:Chi2ParticleID 0.000191632 0.00120874 0.00307918 0.000355416 0.00132429 3
reco:pmtrajfittc:PMAlgTrajFitter 0.616096 13.4176 37.5151 2.12162 17.0506 3
reco:pmtrajfittccalo:Calorimetry 0.00148238 0.302257 0.891111 0.0141774 0.416415 3
reco:pmtrajfittcpid:Chi2ParticleID 0.000216033 0.0017451 0.00458489 0.000434373 0.00201001 3
reco:blurredcluster:BlurredClustering 0.196749 0.851135 1.50552 0.851135 0.654386 2
reco:emtrkmichelid:EmTrackMichelId 2.80506 11.3789 19.9527 11.3789 8.57382 2
reco:emshower:EMShower 0.00281951 0.00656482 0.0103101 0.00656482 0.00374531 2
reco:cvnmap:CVNMapper 0.0124701 0.0197902 0.0271103 0.0197902 0.00732009 2
reco:cvneva:CVNEvaluator 1.16385 1.18524 1.20663 1.18524 0.0213893 2
reco:energyrecnumu:EnergyReco 0.000920496 0.113743 0.226565 0.113743 0.112822 2
reco:energyrecnue:EnergyReco 0.000413388 0.00198129 0.00354919 0.00198129 0.0015679 2
reco:energyrecnc:EnergyReco 0.00279789 0.00435997 0.00592206 0.00435997 0.00156209 2
reco:energyrecnumurange:EnergyReco 0.00053807 0.110088 0.219638 0.110088 0.10955 2
reco:energyrecnumumcs:EnergyReco 0.00337249 0.111573 0.219773 0.111573 0.1082 2
reco:opdec:Deconvolution 0.286737 0.293772 0.300806 0.293772 0.00703457 2
reco:ophitspe:OpHitFinderDeco 1.80961 1.96781 2.12602 1.96781 0.158203 2
reco:opflash:OpFlashFinder 0.001674 0.00353893 0.00540386 0.00353893 0.00186493 2
reco:rns:RandomNumberSaver 2.854e-05 0.000172491 0.000316442 0.000172491 0.000143951 2
reco:anglereconue:NuAngularReco 0.000416198 0.00215472 0.00389325 0.00215472 0.00173853 2
reco:anglereconumu:NuAngularReco 0.000252112 0.00135486 0.00245762 0.00135486 0.00110275 2
reco:anglereconuepfps:NuAngularReco 0.00036774 0.00271973 0.00507172 0.00271973 0.00235199 2
reco:anglereconumupfps:NuAngularReco 0.000334221 0.00150585 0.00267747 0.00150585 0.00117163 2
[art]:TriggerResults:TriggerResultInserter 1.3713e-05 3.57335e-05 5.7754e-05 3.57335e-05 2.20205e-05 2
end_path:out1:RootOutput 3.258e-06 1.2077e-05 2.0896e-05 1.2077e-05 8.819e-06 2
end_path:out1:RootOutput(write) 0.285554 0.337639 0.389725 0.337639 0.0520857 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6425.39 MB
Peak resident set size usage (VmHWM): 1668.96 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:33:49 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6171714 subRun: 1 event: 78003
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-f58b-2967-222c-762d.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_390_20231118T092720Z_gen_g4_detsim_hitreco__20240503T051959Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_390_20231118T092720Z_gen_g4_detsim_hitreco__20240503T051959Z_reco2_reco_2024-11-22T_232350Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 