justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 425876.9@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID425876.9@justin-prod-sched01.dune.hep.ac.uk
Workflow ID8045
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-01 23:25:35
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-07-02 04:45:04
From worker nodeHostnamewn-db-04.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-07-01 23:27:55
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T193129Z_gen_000233_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-07-02 04:45:04
Saved logsjustin-logs:425876.9-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:53 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 06:40:54 CEST run: 4810 subRun: 0 event: 2328
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 9th record. run: 4810 subRun: 0 event: 2329 at 02-Jul-2025 06:44:06 CEST
02-Jul-2025 06:44:14 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/84/e6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T193129Z_gen_000233_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.015011       2079.27       2576.88       2303.07       750.127         9     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00408388     0.0315308     0.114258      0.0104827     0.041047         9     
reco:pandora:StandardPandora                            1938.14       2153.1        2382.31       2178.12       152.874         8     
reco:pandoraTrack:LArPandoraTrackCreation             0.000224327   0.00857141     0.0532203    0.00211457     0.0169975        8     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000229387    0.0018626    0.00804095    0.00109707    0.00244821        8     
reco:pandoracalo:Calorimetry                          0.000192978   0.00153949    0.00625547    0.000815532   0.00187347        8     
reco:pandorapid:Chi2ParticleID                         4.185e-05    0.000192458   0.000590072   8.0974e-05    0.000216371       8     
reco:linecluster:LineCluster                           0.902641      0.966888       1.33573      0.911616      0.139839         8     
reco:trajcluster:TrajCluster                           0.285455      0.304438      0.336632      0.303976      0.0160385        8     
reco:pmtracktc:PMAlgTrackMaker                          5.02441       5.4518        5.72032       5.51813      0.243683         8     
reco:emtrkmichelid:EmTrackMichelId                      160.119       177.874       189.099       182.867       10.962          8     
reco:solarflash:SolarOpFlash                            1.16612       1.21159       1.35632       1.19684      0.0570458        8     
[art]:TriggerResults:TriggerResultInserter            1.9988e-05    3.05783e-05    7.855e-05    2.2423e-05    1.88109e-05       8     
end_path:out1:RootOutput                               4.759e-06    8.14762e-06   2.4707e-05     5.63e-06     6.31509e-06       8     
end_path:out1:RootOutput(write)                        0.215448      0.220609      0.228694      0.220337     0.00397883        8     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4034.01 MB
  Peak resident set size usage (VmHWM): 2569.95 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Jul-2025 06:44:15 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 1028231978
      cannot be satisfied because the product cannot be found.
      Probably the branch containing the product is not stored in the input file.
      The above exception was thrown while processing module StandardPandora/pandora run: 4810 subRun: 0 event: 2329
    ---- ProductNotFound END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
justIN time: 2025-08-14 16:39:16 UTC       justIN version: 01.03.02