21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 426221.33@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 426221.33@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 8052 | |
| Stage ID | 1 | |
| User name | higuera@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-07-02 12:38:34 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc02 | |
| Last heartbeat | 2025-07-02 19:43:28 | |
| From worker node | Hostname | wn-da-16.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-07-02 13:50:16 | |
| Input files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T023856Z_gen_004342_supernova_g4stage1_g4stage2_detsim_reco.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-07-02 19:43:28 | |
| Saved logs | justin-logs:426221.33-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrackMaker:pmtracktc@BeginModule 02-Jul-2025 21:28:41 CEST run: 4923 subRun: 0 event: 43418
Flip, endpoint closer to vStart.
%MSG
Begin processing the 9th record. run: 4923 subRun: 0 event: 43419 at 02-Jul-2025 21:31:29 CEST
02-Jul-2025 21:31:38 CEST Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/18/6f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T023856Z_gen_004342_supernova_g4stage1_g4stage2_detsim_reco.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.0169127 2236.66 2690.25 2519.68 800.19 9
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00329007 0.0355985 0.130664 0.0140943 0.0462063 9
reco:pandora:StandardPandora 2139.61 2325.65 2494.11 2347.41 121.771 8
reco:pandoraTrack:LArPandoraTrackCreation 0.000255351 0.00849233 0.0558359 0.00155945 0.0179678 8
reco:pandoraShower:LArPandoraModularShowerCreation 0.000232487 0.00221275 0.00946048 0.000899403 0.00295185 8
reco:pandoracalo:Calorimetry 0.000232515 0.00159682 0.00644392 0.000431489 0.00201208 8
reco:pandorapid:Chi2ParticleID 4.9865e-05 0.000194177 0.000975915 8.6421e-05 0.000295888 8
reco:linecluster:LineCluster 0.970174 1.1263 1.64707 0.993765 0.238128 8
reco:trajcluster:TrajCluster 0.256322 0.294756 0.313059 0.300809 0.0194945 8
reco:pmtracktc:PMAlgTrackMaker 4.8311 5.17894 5.68929 5.06473 0.309737 8
reco:emtrkmichelid:EmTrackMichelId 164.776 182.475 196.476 183.964 10.2564 8
reco:solarflash:SolarOpFlash 1.19998 1.25526 1.40387 1.23426 0.0645978 8
[art]:TriggerResults:TriggerResultInserter 2.1682e-05 3.00066e-05 5.7431e-05 2.35155e-05 1.23805e-05 8
end_path:out1:RootOutput 4.449e-06 7.22512e-06 2.098e-05 4.995e-06 5.24841e-06 8
end_path:out1:RootOutput(write) 0.20277 0.211613 0.219482 0.212745 0.00468178 8
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4035.55 MB
Peak resident set size usage (VmHWM): 2567.11 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 02-Jul-2025 21:31:38 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- ProductNotFound BEGIN
A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 2625675690
cannot be satisfied because the product cannot be found.
Probably the branch containing the product is not stored in the input file.
The above exception was thrown while processing module StandardPandora/pandora run: 4923 subRun: 0 event: 43419
---- ProductNotFound END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.