21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 425356.10@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 425356.10@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 8046 | |
| Stage ID | 1 | |
| User name | higuera@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-07-01 04:12:46 | |
| Site | US_Wisconsin | |
| Entry | HCCHTPC_US_Wisconsin_osg01_rhel7 | |
| Last heartbeat | 2025-07-01 08:12:10 | |
| From worker node | Hostname | e2614 |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 82800 (23 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-07-01 04:14:00 | |
| Input files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T104130Z_gen_001436_supernova_g4stage1_g4stage2_detsim_reco.root | |
| Jobscript | Exit code | 139 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-07-01 08:12:10 | |
| Saved logs | justin-logs:425356.10-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 01-Jul-2025 02:17:57 CDT run: 5059 subRun: 0 event: 14354
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 5th record. run: 5059 subRun: 0 event: 14355 at 01-Jul-2025 02:21:01 CDT
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00785744 2140.72 3034.67 2775.38 1112.81 5
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00785744 0.0145018 0.0349792 0.00997461 0.0103199 5
reco:pandora:StandardPandora 1797.71 2419.96 2820.32 2587.2 354.634 5
reco:pandoraTrack:LArPandoraTrackCreation 0.000619835 0.0133492 0.0440066 0.00605072 0.0160292 5
reco:pandoraShower:LArPandoraModularShowerCreation 0.00102105 0.00327055 0.0103445 0.00168365 0.00355305 5
reco:pandoracalo:Calorimetry 0.000350939 0.0010099 0.00274306 0.000639601 0.000880613 5
reco:pandorapid:Chi2ParticleID 0.000113653 0.000328245 0.000984149 0.000150723 0.000331852 5
reco:linecluster:LineCluster 0.308319 0.347403 0.413105 0.334094 0.0422904 4
reco:trajcluster:TrajCluster 0.242498 0.278238 0.324031 0.273212 0.0298835 4
reco:pmtracktc:PMAlgTrackMaker 4.66537 6.53877 8.50033 6.4947 1.39613 4
reco:emtrkmichelid:EmTrackMichelId 182.22 294.947 483.842 256.862 118.649 4
reco:solarflash:SolarOpFlash 0.487249 0.633373 0.835712 0.605266 0.132475 4
[art]:TriggerResults:TriggerResultInserter 2.4787e-05 7.45175e-05 0.000194015 3.9634e-05 6.94921e-05 4
end_path:out1:RootOutput 9.237e-06 2.66298e-05 6.9911e-05 1.36855e-05 2.51437e-05 4
end_path:out1:RootOutput(write) 0.170079 0.248379 0.394355 0.214541 0.0901184 4
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4046.14 MB
Peak resident set size usage (VmHWM): 2566.18 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 01-Jul-2025 03:05:42 CDT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- FileReadError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::ReadBuffer
[ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
ROOT severity: 3000
---- FatalRootError END
The above exception was thrown while processing module LineCluster/linecluster run: 5059 subRun: 0 event: 14355
---- FileReadError END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 74: 1144 Segmentation fault (core dumped) lar -c reco2_supernova_dunevd10kt_1x8x14_3view_30deg_prod2024.fcl $FILE -o ${reco2_name}.root -n -1