Jobsub ID 301452.85@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301452.85@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:40:16 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:20:05 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:09:37 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_5_20231203T090737Z_gen_g4_detsim_hitreco__20240509T190654Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 6m (384s) |
| CPU time | 6m (375s = 97%) |
| Outputting started | 2024-11-22 23:16:01 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_5_20231203T090737Z_gen_g4_detsim_hitreco__20240509T190654Z_reco2.root
|
| Finished | 2024-11-22 23:20:05 |
| Saved logs | justin-logs:301452.85-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
: 74510902 subRun: 1 event: 502
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:15:59 CET run: 74510902 subRun: 1 event: 502
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:15:59 CET run: 74510902 subRun: 1 event: 502
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:16:00 CET run: 74510902 subRun: 1 event: 502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:16:00 CET run: 74510902 subRun: 1 event: 502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:16:00 CET run: 74510902 subRun: 1 event: 502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:16:00 CET run: 74510902 subRun: 1 event: 502
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:16:00 CET run: 74510902 subRun: 1 event: 502
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:16:01 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/6e/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_5_20231203T090737Z_gen_g4_detsim_hitreco__20240509T190654Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.000902166 4.80415 9.6074 4.80415 4.80325 2
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000902166 0.00117594 0.00144971 0.00117594 0.000273772 2
reco:linecluster:LineCluster 0.021689 12.4224 24.8231 12.4224 12.4007 2
reco:trajcluster:TrajCluster 0.0164266 37.1734 74.3303 37.1734 37.1569 2
reco:pandora:StandardPandora 3.03077 13.3226 23.6145 13.3226 10.2919 2
reco:pandoraTrack:LArPandoraTrackCreation 0.000462586 0.00232859 0.00419459 0.00232859 0.001866 2
reco:pandoraShower:LArPandoraModularShowerCreation 0.000356065 0.00638143 0.0124068 0.00638143 0.00602537 2
reco:pandoracalo:Calorimetry 0.000405969 0.00195909 0.00351222 0.00195909 0.00155313 2
reco:pandorapid:Chi2ParticleID 8.4451e-05 0.000779736 0.00147502 0.000779736 0.000695285 2
reco:trkshowersplit:TrackShowerHits 0.0021401 0.343656 0.685171 0.343656 0.341515 2
reco:pmtrack:PMAlgTrackMaker 0.017657 32.1595 64.3013 32.1595 32.1418 2
reco:pmtrackcalo:Calorimetry 0.000611862 0.201739 0.402866 0.201739 0.201127 2
reco:pmtrackpid:Chi2ParticleID 0.000122176 0.00161027 0.00309837 0.00161027 0.0014881 2
reco:pmtrajfit:PMAlgTrajFitter 0.00167025 0.0201026 0.0385349 0.0201026 0.0184323 2
reco:pmtrajfitcalo:Calorimetry 0.000121849 0.0002271 0.00033235 0.0002271 0.000105251 2
reco:pmtrajfitpid:Chi2ParticleID 4.7937e-05 6.9243e-05 9.0549e-05 6.9243e-05 2.1306e-05 2
reco:pmtracktc:PMAlgTrackMaker 0.00879651 42.3258 84.6428 42.3258 42.317 2
reco:pmtracktccalo:Calorimetry 0.000479119 0.0855847 0.17069 0.0855847 0.0851056 2
reco:pmtracktcpid:Chi2ParticleID 0.000114183 0.00118201 0.00224983 0.00118201 0.00106783 2
reco:pmtrajfittc:PMAlgTrajFitter 0.0967839 13.2342 26.3716 13.2342 13.1374 2
reco:pmtrajfittccalo:Calorimetry 0.000394349 0.301135 0.601876 0.301135 0.300741 2
reco:pmtrajfittcpid:Chi2ParticleID 9.5907e-05 0.00201212 0.00392833 0.00201212 0.00191621 2
reco:blurredcluster:BlurredClustering 2.13864 2.13864 2.13864 2.13864 0 1
reco:emtrkmichelid:EmTrackMichelId 1.06891 1.06891 1.06891 1.06891 0 1
reco:emshower:EMShower 0.00683499 0.00683499 0.00683499 0.00683499 0 1
reco:cvnmap:CVNMapper 0.0353488 0.0353488 0.0353488 0.0353488 0 1
reco:cvneva:CVNEvaluator 1.18092 1.18092 1.18092 1.18092 0 1
reco:energyrecnumu:EnergyReco 0.00704589 0.00704589 0.00704589 0.00704589 0 1
reco:energyrecnue:EnergyReco 0.000411431 0.000411431 0.000411431 0.000411431 0 1
reco:energyrecnc:EnergyReco 0.00297367 0.00297367 0.00297367 0.00297367 0 1
reco:energyrecnumurange:EnergyReco 0.000436825 0.000436825 0.000436825 0.000436825 0 1
reco:energyrecnumumcs:EnergyReco 0.00315556 0.00315556 0.00315556 0.00315556 0 1
reco:opdec:Deconvolution 0.204358 0.204358 0.204358 0.204358 0 1
reco:ophitspe:OpHitFinderDeco 1.41732 1.41732 1.41732 1.41732 0 1
reco:opflash:OpFlashFinder 0.00456791 0.00456791 0.00456791 0.00456791 0 1
reco:rns:RandomNumberSaver 0.000333978 0.000333978 0.000333978 0.000333978 0 1
reco:anglereconue:NuAngularReco 0.00124363 0.00124363 0.00124363 0.00124363 0 1
reco:anglereconumu:NuAngularReco 0.000259951 0.000259951 0.000259951 0.000259951 0 1
reco:anglereconuepfps:NuAngularReco 0.00258946 0.00258946 0.00258946 0.00258946 0 1
reco:anglereconumupfps:NuAngularReco 0.000299368 0.000299368 0.000299368 0.000299368 0 1
[art]:TriggerResults:TriggerResultInserter 5.1776e-05 5.1776e-05 5.1776e-05 5.1776e-05 0 1
end_path:out1:RootOutput 1.6668e-05 1.6668e-05 1.6668e-05 1.6668e-05 0 1
end_path:out1:RootOutput(write) 0.271439 0.271439 0.271439 0.271439 0 1
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6422.31 MB
Peak resident set size usage (VmHWM): 1676.03 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:16:01 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74510902 subRun: 1 event: 502
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-890f-ce1a-4fae-f724.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_5_20231203T090737Z_gen_g4_detsim_hitreco__20240509T190654Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_5_20231203T090737Z_gen_g4_detsim_hitreco__20240509T190654Z_reco2_reco_2024-11-22T_230940Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 