justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 425396.73@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID425396.73@justin-prod-sched01.dune.hep.ac.uk
Workflow ID8045
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-01 05:12:55
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-07-01 07:50:51
From worker nodeHostnamee4075
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-07-01 05:23:44
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T234630Z_gen_000780_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code139
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-07-01 07:50:51
Saved logsjustin-logs:425396.73-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:40 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 02:33:41 CDT run: 5055 subRun: 0 event: 7797
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 8th record. run: 5055 subRun: 0 event: 7798 at 01-Jul-2025 02:35:46 CDT
Error: A CaloHitList is empty
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00856481      972.288       1477.53       1043.29        396.7          8     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00271233     0.0135406     0.0301887    0.00876116    0.00933242        8     
reco:pandora:StandardPandora                            819.952       958.13        1358.05       900.891       160.86          8     
reco:pandoraTrack:LArPandoraTrackCreation             0.000210692   0.00592604     0.0334066    0.000998426    0.0107558        8     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000197107   0.00186661    0.00835636    0.000605925   0.00266303        8     
reco:pandoracalo:Calorimetry                          0.00016566     0.001487     0.00669888    0.000372168   0.00208709        8     
reco:pandorapid:Chi2ParticleID                        4.0676e-05    0.000172926   0.000756069   8.76385e-05   0.00022338        8     
reco:linecluster:LineCluster                           0.267663      0.318098      0.426569      0.305398      0.0474196        7     
reco:trajcluster:TrajCluster                            0.21248      0.234514      0.254518      0.229567      0.0150302        7     
reco:pmtracktc:PMAlgTrackMaker                           4.264        4.6833        5.16934       4.65004      0.257486         7     
reco:emtrkmichelid:EmTrackMichelId                      108.977       127.368       169.759       120.092        20.72          7     
reco:solarflash:SolarOpFlash                           0.464725       0.4868       0.527276      0.484906      0.0198797        7     
[art]:TriggerResults:TriggerResultInserter            2.3474e-05    3.76687e-05   6.0152e-05    2.6659e-05    1.46783e-05       7     
end_path:out1:RootOutput                               7.234e-06    1.43794e-05   2.9876e-05    1.1871e-05    7.43714e-06       7     
end_path:out1:RootOutput(write)                        0.164818      0.198184       0.25385      0.187554      0.0311755        7     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4038.21 MB
  Peak resident set size usage (VmHWM): 2557.2 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 01-Jul-2025 02:50:39 CDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TNetXNGFile::ReadBuffer
        [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module LineCluster/linecluster run: 5055 subRun: 0 event: 7798
    ---- FileReadError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 7 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
  ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 74:  1145 Segmentation fault      (core dumped) lar -c reco2_supernova_dunevd10kt_1x8x14_3view_30deg_prod2024.fcl $FILE -o ${reco2_name}.root -n -1
justIN time: 2025-08-14 16:36:47 UTC       justIN version: 01.03.02