Jobsub ID 301560.39@justin-prod-sched01.dune.hep.ac.uk
Jobsub ID | 301560.39@justin-prod-sched01.dune.hep.ac.uk |
Workflow ID | 4213 |
Stage ID | 1 |
User name | ismerio@fnal.gov |
HTCondor Group | group_dune |
Requested | Processors | 1 |
RSS bytes | 4194304000 (4000 MiB) |
Wall seconds limit | 80000 (22 hours) |
Submitted time | 2024-11-22 23:53:54 |
Site | NL_NIKHEF |
Entry | VIRGO_NL_NIKHEF_klomp |
Last heartbeat | 2024-11-23 00:16:02 |
From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
OS release | Scientific Linux release 7.9 (Nitrogen) |
Processors | 1 |
RSS bytes | 4194304000 (4000 MiB) |
Wall seconds limit | 129600 (36 hours) |
Inner Apptainer? | True |
Job state | finished |
Allocator name | justin-allocator-pro.dune.hep.ac.uk |
Started | 2024-11-23 00:04:22 |
Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root
|
Jobscript | Exit code | 0 |
Real time | 5m (302s) |
CPU time | 3m (202s = 66%) |
Outputting started | 2024-11-23 00:09:25 |
Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root
|
Finished | 2024-11-23 00:16:02 |
Saved logs | justin-logs:301560.39-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
led hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:46 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 01:07:47 CET run: 6464269 subRun: 1 event: 16101
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 01:07:48 CET Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2_reco_data_2024-11-23T_000425Z.root"
23-Nov-2024 01:09:24 CET Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/e6/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6387.03 MB
Peak resident set size usage (VmHWM): 1513.22 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 01:09:24 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6464269 subRun: 1 event: 16101
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-0ac7-5656-c232-e2b2.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2_reco_2024-11-23T_000425Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work