Jobsub ID 108523.108@justin-prod-sched02.dune.hep.ac.uk
Jobsub ID | 108523.108@justin-prod-sched02.dune.hep.ac.uk |
Workflow ID | 4213 |
Stage ID | 1 |
User name | ismerio@fnal.gov |
HTCondor Group | group_dune |
Requested | Processors | 1 |
RSS bytes | 4194304000 (4000 MiB) |
Wall seconds limit | 80000 (22 hours) |
Submitted time | 2024-11-22 22:49:41 |
Site | NL_NIKHEF |
Entry | VIRGO_NL_NIKHEF_klomp |
Last heartbeat | 2024-11-22 23:11:14 |
From worker node | Hostname | wn-choc-034.farm.nikhef.nl |
cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
OS release | Scientific Linux release 7.9 (Nitrogen) |
Processors | 1 |
RSS bytes | 4194304000 (4000 MiB) |
Wall seconds limit | 129600 (36 hours) |
Inner Apptainer? | True |
Job state | finished |
Allocator name | justin-allocator-pro.dune.hep.ac.uk |
Started | 2024-11-22 23:05:10 |
Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root
|
Jobscript | Exit code | 0 |
Real time | 3m (196s) |
CPU time | 3m (191s = 97%) |
Outputting started | 2024-11-22 23:08:27 |
Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root
|
Finished | 2024-11-22 23:11:14 |
Saved logs | justin-logs:108523.108-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
gTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:08:25 CET run: 74481118 subRun: 1 event: 41204
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:08:26 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/53/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.000878664 7.63993 19.8559 5.35146 7.41664 4
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000878664 0.00112385 0.00128904 0.00116386 0.000152395 4
reco:linecluster:LineCluster 0.0109955 0.168325 0.59252 0.034893 0.245588 4
reco:trajcluster:TrajCluster 0.00223252 3.75481 14.9338 0.041597 6.45428 4
reco:pandora:StandardPandora 1.58822 3.32144 6.31449 2.69153 1.8267 4
reco:pandoraTrack:LArPandoraTrackCreation 0.000281982 0.0129981 0.0431544 0.00427804 0.0176774 4
reco:pandoraShower:LArPandoraModularShowerCreation 0.000660387 0.020296 0.0620978 0.00921288 0.0249802 4
reco:pandoracalo:Calorimetry 0.000215069 0.0281142 0.103896 0.00417279 0.0438629 4
reco:pandorapid:Chi2ParticleID 8.2947e-05 0.00054977 0.00164587 0.00023513 0.000642653 4
reco:trkshowersplit:TrackShowerHits 0.000176166 0.0530208 0.20636 0.00277327 0.0885552 4
reco:pmtrack:PMAlgTrackMaker 0.00145376 5.36694 20.2021 0.632112 8.57626 4
reco:pmtrackcalo:Calorimetry 9.9334e-05 0.0192272 0.0730235 0.00189292 0.0310831 4
reco:pmtrackpid:Chi2ParticleID 4.3105e-05 0.000500034 0.0015523 0.000202364 0.00061442 4
reco:pmtrajfit:PMAlgTrajFitter 0.00334019 2.91882 10.1782 0.746854 4.23204 4
reco:pmtrajfitcalo:Calorimetry 0.000197092 0.0290507 0.110638 0.00268404 0.0471401 4
reco:pmtrajfitpid:Chi2ParticleID 6.9106e-05 0.000190364 0.000460529 0.000115911 0.000157196 4
reco:pmtracktc:PMAlgTrackMaker 0.00265352 6.60947 25.6649 0.385176 11.0052 4
reco:pmtracktccalo:Calorimetry 0.000182511 0.0122782 0.0462258 0.00135228 0.0196146 4
reco:pmtracktcpid:Chi2ParticleID 6.5614e-05 0.000408804 0.00119976 0.000184921 0.000462545 4
reco:pmtrajfittc:PMAlgTrajFitter 0.00434149 2.47554 8.94698 0.475419 3.74664 4
reco:pmtrajfittccalo:Calorimetry 0.000254494 0.028259 0.107804 0.00248886 0.0459556 4
reco:pmtrajfittcpid:Chi2ParticleID 7.7284e-05 0.000575814 0.00173596 0.000245007 0.000679767 4
reco:blurredcluster:BlurredClustering 0.153075 0.803988 1.9227 0.336189 0.794574 3
reco:emtrkmichelid:EmTrackMichelId 0.710515 3.39768 8.54541 0.937127 3.64117 3
reco:emshower:EMShower 0.000434979 0.0158457 0.0437009 0.00340125 0.0197338 3
reco:cvnmap:CVNMapper 3.5274e-05 0.00531233 0.0158603 4.1441e-05 0.00745852 3
reco:cvneva:CVNEvaluator 2.2181e-05 0.404484 1.2134 2.9834e-05 0.571989 3
reco:energyrecnumu:EnergyReco 0.0026756 0.00436043 0.00757435 0.00283136 0.00227347 3
reco:energyrecnue:EnergyReco 0.000254362 0.000695021 0.00154654 0.000284158 0.00060224 3
reco:energyrecnc:EnergyReco 0.00256457 0.00277354 0.00311465 0.00264141 0.000243231 3
reco:energyrecnumurange:EnergyReco 0.000833592 0.0021839 0.00309855 0.00261957 0.000974633 3
reco:energyrecnumumcs:EnergyReco 0.00262281 0.00306867 0.00337711 0.00320608 0.000322904 3
reco:opdec:Deconvolution 0.141048 0.17647 0.22422 0.164142 0.0350559 3
reco:ophitspe:OpHitFinderDeco 1.01464 1.30735 1.63867 1.26873 0.25622 3
reco:opflash:OpFlashFinder 0.000997139 0.00219331 0.00448243 0.00110035 0.00161921 3
reco:rns:RandomNumberSaver 2.7829e-05 0.000119227 0.000300056 2.9797e-05 0.000127868 3
reco:anglereconue:NuAngularReco 0.000188364 0.00062745 0.00134006 0.000353923 0.000508407 3
reco:anglereconumu:NuAngularReco 0.000115717 0.000263248 0.000407384 0.000266642 0.000119097 3
reco:anglereconuepfps:NuAngularReco 0.000216227 0.00116418 0.00235171 0.000924613 0.000888111 3
reco:anglereconumupfps:NuAngularReco 0.000166197 0.000308344 0.000439487 0.000319347 0.000111841 3
[art]:TriggerResults:TriggerResultInserter 1.1949e-05 2.42363e-05 4.7345e-05 1.3415e-05 1.63513e-05 3
end_path:out1:RootOutput 2.949e-06 7.516e-06 1.5626e-05 3.973e-06 5.74985e-06 3
end_path:out1:RootOutput(write) 0.0727976 0.157194 0.315635 0.0831497 0.112114 3
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6423.31 MB
Peak resident set size usage (VmHWM): 1658.36 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:08:27 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74481118 subRun: 1 event: 41204
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-3f37-d0bb-3228-74f8.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_412_20231201T112850Z_gen_g4_detsim_hitreco__20240507T193910Z_reco2_reco_2024-11-22T_230514Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work