justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 232889.182@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID232889.182@justin-prod-sched02.dune.hep.ac.uk
Workflow ID8046
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-01 03:52:06
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-07-01 08:01:27
From worker nodeHostnamewn-sate-038.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-07-01 04:45:50
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T161942Z_gen_002155_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code139
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-07-01 08:01:27
Saved logsjustin-logs:232889.182-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

2025 09:27:44 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:44 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:44 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:44 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Jul-2025 09:27:45 CEST run: 5048 subRun: 0 event: 21545
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 6th record. run: 5048 subRun: 0 event: 21546 at 01-Jul-2025 09:30:29 CEST

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.133248       1591.73       1948.93       1903.82       712.109         6     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00331222     0.0479054     0.133248     0.00810818     0.058875         6     
reco:pandora:StandardPandora                            1690.38       1723.64       1760.89       1721.08       22.767          6     
reco:pandoraTrack:LArPandoraTrackCreation             0.000388201    0.0134496     0.0572783    0.00535437     0.0197238        6     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000301217   0.00358987     0.0157865    0.00125306    0.00547327        6     
reco:pandoracalo:Calorimetry                          0.000219864   0.00105441    0.00435011    0.000397028    0.0014777        6     
reco:pandorapid:Chi2ParticleID                        5.7308e-05    0.000355204   0.00164176    0.000107197   0.000575685       6     
reco:linecluster:LineCluster                            1.15838       1.46889       2.08117       1.2088       0.362417         5     
reco:trajcluster:TrajCluster                           0.369379      0.394215      0.423196      0.394811      0.0209991        5     
reco:pmtracktc:PMAlgTrackMaker                          6.36041       6.79279       7.34183       6.57215      0.398859         5     
reco:emtrkmichelid:EmTrackMichelId                      160.346       169.282       176.369       169.526       5.87469         5     
reco:solarflash:SolarOpFlash                            1.43156       1.49567       1.70302       1.45173      0.103955         5     
[art]:TriggerResults:TriggerResultInserter            2.3073e-05    3.51622e-05   6.6555e-05    2.8614e-05    1.59238e-05       5     
end_path:out1:RootOutput                               6.873e-06    1.1628e-05    2.7372e-05     8.366e-06    7.90435e-06       5     
end_path:out1:RootOutput(write)                        0.266001      0.278864      0.295291      0.273195      0.0105886        5     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4044.53 MB
  Peak resident set size usage (VmHWM): 2584.78 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 01-Jul-2025 10:00:00 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TNetXNGFile::ReadBuffer
        [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module LineCluster/linecluster run: 5048 subRun: 0 event: 21546
    ---- FileReadError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
  ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 74:  1137 Segmentation fault      (core dumped) lar -c reco2_supernova_dunevd10kt_1x8x14_3view_30deg_prod2024.fcl $FILE -o ${reco2_name}.root -n -1
justIN time: 2025-08-14 20:15:22 UTC       justIN version: 01.03.02