21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 426354.132@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 426354.132@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 8058 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-07-02 17:44:32 | |
| Site | US_FNAL-FermiGrid | |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly | |
| Last heartbeat | 2025-07-03 01:16:41 | |
| From worker node | Hostname | dunegli-5871304-0-fnpc22015.fnal.gov |
| cpuinfo | AMD EPYC 7543 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-07-03 00:12:18 | |
| Input files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T052605Z_gen_000974_supernova_g4stage1_g4stage2_detsim_reco.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-07-03 01:16:41 | |
| Saved logs | justin-logs:426354.132-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
ne: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 03-Jul-2025 01:14:04 UTC run: 5031 subRun: 0 event: 9735
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 6th record. run: 5031 subRun: 0 event: 9736 at 03-Jul-2025 01:16:12 UTC
03-Jul-2025 01:16:15 UTC Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5d/be/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T052605Z_gen_000974_supernova_g4stage1_g4stage2_detsim_reco.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.0277745 629.001 794.811 739.956 282.301 6
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00290025 0.00991776 0.0277745 0.00678724 0.00832719 6
reco:pandora:StandardPandora 622.992 643.341 663.902 642.224 13.4811 5
reco:pandoraTrack:LArPandoraTrackCreation 0.000167827 0.0124338 0.0566506 0.00160965 0.0221349 5
reco:pandoraShower:LArPandoraModularShowerCreation 0.000183868 0.0017004 0.00454216 0.000762151 0.00157921 5
reco:pandoracalo:Calorimetry 0.000285901 0.000774969 0.00119253 0.000931972 0.000399337 5
reco:pandorapid:Chi2ParticleID 3.6079e-05 0.000184958 0.000414063 6.4572e-05 0.000161996 5
reco:linecluster:LineCluster 0.115318 0.12765 0.140848 0.128171 0.010242 5
reco:trajcluster:TrajCluster 0.175797 0.188752 0.224701 0.183575 0.0183194 5
reco:pmtracktc:PMAlgTrackMaker 3.26018 3.58316 3.85097 3.60862 0.238911 5
reco:emtrkmichelid:EmTrackMichelId 88.5246 107.082 128.634 96.3093 17.0432 5
reco:solarflash:SolarOpFlash 0.291119 0.309506 0.335235 0.310334 0.0151439 5
[art]:TriggerResults:TriggerResultInserter 1.5058e-05 2.77684e-05 6.3531e-05 1.7603e-05 1.82462e-05 5
end_path:out1:RootOutput 7.685e-06 1.154e-05 2.4366e-05 8.346e-06 6.44343e-06 5
end_path:out1:RootOutput(write) 0.136205 0.14145 0.145249 0.142031 0.00310999 5
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4026.61 MB
Peak resident set size usage (VmHWM): 2563.88 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Jul-2025 01:16:15 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- ProductNotFound BEGIN
A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 3415784363
cannot be satisfied because the product cannot be found.
Probably the branch containing the product is not stored in the input file.
The above exception was thrown while processing module StandardPandora/pandora run: 5031 subRun: 0 event: 9736
---- ProductNotFound END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.