21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 232696.28@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 232696.28@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 8030 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-30 17:06:10 | |
| Site | US_FNAL-T1 | |
| Entry | CMSHTPC_T1_US_FNAL_condce_opp1_whole | |
| Last heartbeat | 2025-07-01 07:51:31 | |
| From worker node | Hostname | dunegli-43424-0-cmswn6026.fnal.gov |
| cpuinfo | AMD EPYC 9354 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-07-01 02:06:05 | |
| Input files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T111946Z_gen_002012_supernova_g4stage1_g4stage2_detsim_reco.root | |
| Jobscript | Exit code | 139 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-07-01 07:51:31 | |
| Saved logs | justin-logs:232696.28-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
r:pmtracktc@BeginModule 01-Jul-2025 07:11:28 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:28 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:28 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:28 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 07:11:29 UTC run: 5043 subRun: 0 event: 20118
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 9th record. run: 5043 subRun: 0 event: 20119 at 01-Jul-2025 07:15:57 UTC
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.0386453 2048.07 2457.82 2344.85 736.508 9
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00464952 0.0251182 0.0386453 0.026302 0.0113112 9
reco:pandora:StandardPandora 1781.75 2047.48 2286.55 2084.16 159.479 9
reco:pandoraTrack:LArPandoraTrackCreation 0.000540492 0.0311506 0.194663 0.0151261 0.0581909 9
reco:pandoraShower:LArPandoraModularShowerCreation 0.000351375 0.00510692 0.0329279 0.00124025 0.00988444 9
reco:pandoracalo:Calorimetry 0.000298374 0.00493529 0.0300303 0.000634105 0.00938235 9
reco:pandorapid:Chi2ParticleID 5.921e-05 0.000587662 0.00401245 0.000180084 0.0012124 9
reco:linecluster:LineCluster 0.0976159 0.209804 0.295945 0.231967 0.060654 8
reco:trajcluster:TrajCluster 0.222952 0.359897 0.48274 0.357762 0.065802 8
reco:pmtracktc:PMAlgTrackMaker 3.9056 6.46264 8.09466 6.6425 1.11683 8
reco:emtrkmichelid:EmTrackMichelId 166.25 239.881 281.551 254.482 37.7706 8
reco:solarflash:SolarOpFlash 0.310587 0.45435 0.54784 0.467505 0.0643228 8
[art]:TriggerResults:TriggerResultInserter 5.4674e-05 0.000111735 0.000303461 9.65015e-05 7.45111e-05 8
end_path:out1:RootOutput 1.0265e-05 2.64377e-05 7.8971e-05 1.95295e-05 2.05496e-05 8
end_path:out1:RootOutput(write) 0.167392 0.248508 0.312923 0.2548 0.0380413 8
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4037.56 MB
Peak resident set size usage (VmHWM): 2537.9 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 01-Jul-2025 07:51:07 UTC ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- FileReadError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::ReadBuffer
[ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
ROOT severity: 3000
---- FatalRootError END
The above exception was thrown while processing module LineCluster/linecluster run: 5043 subRun: 0 event: 20119
---- FileReadError END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 74: 1138 Segmentation fault (core dumped) lar -c reco2_supernova_dunevd10kt_1x8x14_3view_30deg_prod2024.fcl $FILE -o ${reco2_name}.root -n -1