justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 148497.3@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID148497.3@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5274
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2025-02-24 18:22:56
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-02-24 20:12:54
From worker nodeHostnamecompute-1-4.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-02-24 19:49:39
Input filesfardet-hd:nutau_dune10kt_1x2x6_1121_345_20230828T053343Z_gen_g4_detsim_hitreco__20240219T190339Z_reco2.root
JobscriptExit code0
Real time22m (1371s)
CPU time10m (646s = 47%)
Max RSS bytes1070829568 (1021 MiB)
Outputting started2025-02-24 20:12:31
Output files
Finished2025-02-24 20:12:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

-----------------
---------------------------------------------------------
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/a0/3c/nutau_dune10kt_1x2x6_1121_345_20230828T053343Z_gen_g4_detsim_hitreco__20240219T190339Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 43rd record. run: 1121 subRun: 1 event: 34543 at 24-Feb-2025 16:05:43 AST
Begin processing the 44th record. run: 1121 subRun: 1 event: 34544 at 24-Feb-2025 16:05:47 AST
Begin processing the 45th record. run: 1121 subRun: 1 event: 34545 at 24-Feb-2025 16:05:53 AST
Begin processing the 46th record. run: 1121 subRun: 1 event: 34546 at 24-Feb-2025 16:05:57 AST
Begin processing the 47th record. run: 1121 subRun: 1 event: 34547 at 24-Feb-2025 16:06:02 AST
Begin processing the 48th record. run: 1121 subRun: 1 event: 34548 at 24-Feb-2025 16:06:06 AST
Begin processing the 49th record. run: 1121 subRun: 1 event: 34549 at 24-Feb-2025 16:06:10 AST
Begin processing the 50th record. run: 1121 subRun: 1 event: 34550 at 24-Feb-2025 16:06:22 AST
Begin processing the 51st record. run: 1121 subRun: 1 event: 34551 at 24-Feb-2025 16:06:27 AST
Begin processing the 52nd record. run: 1121 subRun: 1 event: 34552 at 24-Feb-2025 16:06:32 AST
Begin processing the 53rd record. run: 1121 subRun: 1 event: 34553 at 24-Feb-2025 16:06:36 AST
Begin processing the 54th record. run: 1121 subRun: 1 event: 34554 at 24-Feb-2025 16:06:40 AST
Begin processing the 55th record. run: 1121 subRun: 1 event: 34555 at 24-Feb-2025 16:06:57 AST
Begin processing the 56th record. run: 1121 subRun: 1 event: 34556 at 24-Feb-2025 16:07:01 AST
Begin processing the 57th record. run: 1121 subRun: 1 event: 34557 at 24-Feb-2025 16:07:07 AST
Begin processing the 58th record. run: 1121 subRun: 1 event: 34558 at 24-Feb-2025 16:07:11 AST
Begin processing the 59th record. run: 1121 subRun: 1 event: 34559 at 24-Feb-2025 16:07:15 AST
Begin processing the 60th record. run: 1121 subRun: 1 event: 34560 at 24-Feb-2025 16:07:20 AST
Begin processing the 61st record. run: 1121 subRun: 1 event: 34561 at 24-Feb-2025 16:07:24 AST
Begin processing the 62nd record. run: 1121 subRun: 1 event: 34562 at 24-Feb-2025 16:07:28 AST
Begin processing the 63rd record. run: 1121 subRun: 1 event: 34563 at 24-Feb-2025 16:07:39 AST
Begin processing the 64th record. run: 1121 subRun: 1 event: 34564 at 24-Feb-2025 16:07:43 AST
Begin processing the 65th record. run: 1121 subRun: 1 event: 34565 at 24-Feb-2025 16:07:49 AST
Begin processing the 66th record. run: 1121 subRun: 1 event: 34566 at 24-Feb-2025 16:07:53 AST
Begin processing the 67th record. run: 1121 subRun: 1 event: 34567 at 24-Feb-2025 16:07:57 AST
Begin processing the 68th record. run: 1121 subRun: 1 event: 34568 at 24-Feb-2025 16:08:01 AST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_pandoraBugFix/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_pandoraBugFix/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 69th record. run: 1121 subRun: 1 event: 34569 at 24-Feb-2025 16:08:04 AST
Begin processing the 70th record. run: 1121 subRun: 1 event: 34570 at 24-Feb-2025 16:08:10 AST
Begin processing the 71st record. run: 1121 subRun: 1 event: 34571 at 24-Feb-2025 16:08:18 AST
Begin processing the 72nd record. run: 1121 subRun: 1 event: 34572 at 24-Feb-2025 16:08:22 AST
Begin processing the 73rd record. run: 1121 subRun: 1 event: 34573 at 24-Feb-2025 16:08:29 AST
Begin processing the 74th record. run: 1121 subRun: 1 event: 34574 at 24-Feb-2025 16:08:33 AST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_pandoraBugFix/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_pandoraBugFix/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 75th record. run: 1121 subRun: 1 event: 34575 at 24-Feb-2025 16:08:35 AST
Begin processing the 76th record. run: 1121 subRun: 1 event: 34576 at 24-Feb-2025 16:08:41 AST
Begin processing the 77th record. run: 1121 subRun: 1 event: 34577 at 24-Feb-2025 16:08:46 AST
Begin processing the 78th record. run: 1121 subRun: 1 event: 34578 at 24-Feb-2025 16:08:53 AST
Begin processing the 79th record. run: 1121 subRun: 1 event: 34579 at 24-Feb-2025 16:08:57 AST
Begin processing the 80th record. run: 1121 subRun: 1 event: 34580 at 24-Feb-2025 16:10:28 AST
Begin processing the 81st record. run: 1121 subRun: 1 event: 34581 at 24-Feb-2025 16:10:35 AST
Begin processing the 82nd record. run: 1121 subRun: 1 event: 34582 at 24-Feb-2025 16:10:39 AST
Begin processing the 83rd record. run: 1121 subRun: 1 event: 34583 at 24-Feb-2025 16:10:53 AST
Begin processing the 84th record. run: 1121 subRun: 1 event: 34584 at 24-Feb-2025 16:10:58 AST
Begin processing the 85th record. run: 1121 subRun: 1 event: 34585 at 24-Feb-2025 16:11:02 AST
Begin processing the 86th record. run: 1121 subRun: 1 event: 34586 at 24-Feb-2025 16:11:12 AST
Begin processing the 87th record. run: 1121 subRun: 1 event: 34587 at 24-Feb-2025 16:11:16 AST
Begin processing the 88th record. run: 1121 subRun: 1 event: 34588 at 24-Feb-2025 16:11:20 AST
Begin processing the 89th record. run: 1121 subRun: 1 event: 34589 at 24-Feb-2025 16:11:24 AST
Begin processing the 90th record. run: 1121 subRun: 1 event: 34590 at 24-Feb-2025 16:11:29 AST
Begin processing the 91st record. run: 1121 subRun: 1 event: 34591 at 24-Feb-2025 16:11:39 AST
Begin processing the 92nd record. run: 1121 subRun: 1 event: 34592 at 24-Feb-2025 16:11:44 AST
Begin processing the 93rd record. run: 1121 subRun: 1 event: 34593 at 24-Feb-2025 16:11:54 AST
Begin processing the 94th record. run: 1121 subRun: 1 event: 34594 at 24-Feb-2025 16:11:58 AST
Begin processing the 95th record. run: 1121 subRun: 1 event: 34595 at 24-Feb-2025 16:12:02 AST
Begin processing the 96th record. run: 1121 subRun: 1 event: 34596 at 24-Feb-2025 16:12:06 AST
Begin processing the 97th record. run: 1121 subRun: 1 event: 34597 at 24-Feb-2025 16:12:10 AST
Begin processing the 98th record. run: 1121 subRun: 1 event: 34598 at 24-Feb-2025 16:12:14 AST
Begin processing the 99th record. run: 1121 subRun: 1 event: 34599 at 24-Feb-2025 16:12:19 AST
Begin processing the 100th record. run: 1121 subRun: 1 event: 34600 at 24-Feb-2025 16:12:25 AST
24-Feb-2025 16:12:30 AST  Closed output file "nutau_dune10kt_1x2x6_1121_345_20230828T053343Z_gen_g4_detsim_hitreco__20240219T190339Z_reco2_reco2.root"
24-Feb-2025 16:12:30 AST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/a0/3c/nutau_dune10kt_1x2x6_1121_345_20230828T053343Z_gen_g4_detsim_hitreco__20240219T190339Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.64233       6.32995       87.5754       3.08648       12.7405        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0851921     0.179888      0.405602      0.175785      0.031714        100    
reco:pandora:StandardPandora                   0.464757       6.13593       87.2812       2.91421       12.7225        100    
[art]:TriggerResults:TriggerResultInserter    1.4086e-05    2.39408e-05   7.4005e-05    2.10735e-05   1.13008e-05      100    
end_path:out1:RootOutput                       2.841e-06    3.39297e-06   1.3662e-05    3.1675e-06    1.08405e-06      100    
end_path:out1:RootOutput(write)               0.00157181     0.0139321     0.106012     0.00528464     0.0191141       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2259.87 MB
  Peak resident set size usage (VmHWM): 1070.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'MLPHierarchyValidationFile.root': No such file or directory
justIN time: 2025-04-03 08:14:19 UTC       justIN version: 01.03.00