21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 425319.2@justin-prod-sched01.dune.hep.ac.uk
Jobsub ID | 425319.2@justin-prod-sched01.dune.hep.ac.uk | |
Workflow ID | 8036 | |
Stage ID | 1 | |
User name | epennacc@fnal.gov | |
HTCondor Group | group_dune.prod_mcsim | |
Requested | Processors | 1 |
GPU | No | |
RSS bytes | 4194304000 (4000 MiB) | |
Wall seconds limit | 80000 (22 hours) | |
Submitted time | 2025-07-01 03:08:30 | |
Site | UK_Durham | |
Entry | DUNE_UK_SGridDurham_ce3 | |
Last heartbeat | 2025-07-01 07:57:18 | |
From worker node | Hostname | n230.dur.scotgrid.ac.uk |
cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz | |
OS release | Scientific Linux release 7.9 (Nitrogen) | |
Processors | 1 | |
RSS bytes | 4194304000 (4000 MiB) | |
Wall seconds limit | 171000 (47 hours) | |
GPU | ||
Inner Apptainer? | True | |
Job state | jobscript_error | |
Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
Started | 2025-07-01 06:32:46 | |
Input files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T201950Z_gen_003332_supernova_g4stage1_g4stage2_detsim_reco.root | |
Jobscript | Exit code | 139 |
Real time | 0m (0s) | |
CPU time | 0m (0s = 0%) | |
Max RSS bytes | 0 (0 MiB) | |
Outputting started | ||
Output files | ||
Finished | 2025-07-01 07:57:18 | |
Saved logs | justin-logs:425319.2-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
01:03 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:03 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:03 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:03 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:03 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:04 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:05 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:05 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 08:01:05 BST run: 4992 subRun: 0 event: 33311 Comparing two wires in the same plane: return failure %MSG 01-Jul-2025 08:06:19 BST Opened output file with pattern "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T201950Z_gen_003332_supernova_g4stage1_g4stage2_detsim_reco_20250701T063250Z_reco2.root" Begin processing the 2nd record. run: 4992 subRun: 0 event: 33312 at 01-Jul-2025 08:25:04 BST ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.00269518 950.729 1901.45 950.729 950.726 2 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00269518 0.060897 0.119099 0.060897 0.0582018 2 reco:pandora:StandardPandora 1573.9 1679.15 1784.4 1679.15 105.247 2 reco:pandoraTrack:LArPandoraTrackCreation 0.000991967 0.0300654 0.0591388 0.0300654 0.0290734 2 reco:pandoraShower:LArPandoraModularShowerCreation 0.0011306 0.00818247 0.0152343 0.00818247 0.00705187 2 reco:pandoracalo:Calorimetry 0.000358945 0.00245124 0.00454354 0.00245124 0.0020923 2 reco:pandorapid:Chi2ParticleID 8.4148e-05 0.00084824 0.00161233 0.00084824 0.000764092 2 reco:linecluster:LineCluster 3.0172 3.0172 3.0172 3.0172 0 1 reco:trajcluster:TrajCluster 0.492892 0.492892 0.492892 0.492892 0 1 reco:pmtracktc:PMAlgTrackMaker 10.6417 10.6417 10.6417 10.6417 0 1 reco:emtrkmichelid:EmTrackMichelId 310.097 310.097 310.097 310.097 0 1 reco:solarflash:SolarOpFlash 2.69191 2.69191 2.69191 2.69191 0 1 [art]:TriggerResults:TriggerResultInserter 6.3025e-05 6.3025e-05 6.3025e-05 6.3025e-05 0 1 end_path:out1:RootOutput 2.4682e-05 2.4682e-05 2.4682e-05 2.4682e-05 0 1 end_path:out1:RootOutput(write) 0.411413 0.411413 0.411413 0.411413 0 1 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 4005.24 MB Peak resident set size usage (VmHWM): 2538.5 MB ==================================================================================================== %MSG-s ArtException: PostEndJob 01-Jul-2025 08:56:50 BST ModuleEndJob ---- EventProcessorFailure BEGIN EventProcessor: an exception occurred during current event processing ---- ScheduleExecutionFailure BEGIN Path: ProcessingStopped. ---- FileReadError BEGIN ---- FatalRootError BEGIN Fatal Root Error: TNetXNGFile::ReadBuffer [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010]) ROOT severity: 3000 ---- FatalRootError END The above exception was thrown while processing module LineCluster/linecluster run: 4992 subRun: 0 event: 33312 ---- FileReadError END Exception going through path reco ---- ScheduleExecutionFailure END ---- EventProcessorFailure END ---- FatalRootError BEGIN Fatal Root Error: TTree::SetEntries Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries. ROOT severity: 2000 ---- FatalRootError END ---- FatalRootError BEGIN Fatal Root Error: TNetXNGFile::Close [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010]) ROOT severity: 3000 ---- FatalRootError END %MSG ../justin-jobscript: line 74: 1139 Segmentation fault (core dumped) lar -c reco2_supernova_dunevd10kt_1x8x14_3view_30deg_prod2024.fcl $FILE -o ${reco2_name}.root -n -1