Jobsub ID 296845.0@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 296845.0@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4129 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 43200 (12 hours) |
| Submitted time | 2024-11-19 11:52:06 |
| Site | CA_SFU |
| Entry | DUNE_CA_SFU_lcg-ce3 |
| Last heartbeat | 2024-11-19 11:55:14 |
| From worker node | Hostname | cdr821.int.cedar.computecanada.ca |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 84598 (23 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-19 11:54:14 |
| Input files | fardet-hd:nutau_dune10kt_1x2x6_1105_335_20230826T143334Z_gen_g4_detsim_hitreco__20240220T113538Z_reco2.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-11-19 11:55:14 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
gov/data/dune/RSE/fardet-hd/1c/b9/nutau_dune10kt_1x2x6_1105_335_20230826T143334Z_gen_g4_detsim_hitreco__20240220T113538Z_reco2.root
lar exit code 20
=== Start last 100 lines of lar log file ===
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 3831 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
GeoApaChannelGroupService::ctor: Group 0 (apa00) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 1 (apa01) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 2 (apa02) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 3 (apa03) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 4 (apa04) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 5 (apa05) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 6 (apa06) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 7 (apa07) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 8 (apa08) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 9 (apa09) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 10 (apa10) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 11 (apa11) has 2560 channels from 4/4 readout planes.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
: Booking "BDTG" of type "BDT" from /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/MCC11_FHC_Pandizzle_NewProduction.weights.xml.
: Reading weight file: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/MCC11_FHC_Pandizzle_NewProduction.weights.xml
<HEADER> DataSetInfo : [Default] : Added class "Signal"
<HEADER> DataSetInfo : [Default] : Added class "Background"
: Booked classifier "BDTG" of type: "BDT"
: Booking "BDTG" of type "BDT" from /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/MCC11_FHC_Pandrizzle_Backup_NewProduction.weights.xml.
: Reading weight file: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/MCC11_FHC_Pandrizzle_Backup_NewProduction.weights.xml
<HEADER> DataSetInfo : [Default] : Added class "Signal"
<HEADER> DataSetInfo : [Default] : Added class "Background"
: Booked classifier "BDTG" of type: "BDT"
: Booking "BDTG" of type "BDT" from /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/MCC11_FHC_Pandrizzle_Enhanced_NewProduction.weights.xml.
: Reading weight file: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/MCC11_FHC_Pandrizzle_Enhanced_NewProduction.weights.xml
<HEADER> DataSetInfo : [Default] : Added class "Signal"
<HEADER> DataSetInfo : [Default] : Added class "Background"
: Booked classifier "BDTG" of type: "BDT"
HELLLLO
Directory path: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/my_model_calo_VGG_BN_VARS/
GOOOODBYE
*************************************************************************************************************************************
Unique Ptrs that are added to the event
*************************************************************************************************************************************
* Data Product Name: InitialTrack * Instance Name: * Type: std::vector<recob::Track, std::allocator<recob::Track> >* *
* Data Product Name: ShowerPCA * Instance Name: * Type: std::vector<recob::PCAxis, std::allocator<recob::PCAxis> >* *
* Data Product Name: shower * Instance Name: * Type: std::vector<recob::Shower, std::allocator<recob::Shower> >* *
* Association Name: PFParticlePCAxisAssn * Instance Name: * Type: art::Assns<recob::PFParticle, recob::PCAxis, void>* *
* Association Name: ShowerPCAxisAssn * Instance Name: * Type: art::Assns<recob::Shower, recob::PCAxis, void>* *
* Association Name: ShowerTrackAssn * Instance Name: * Type: art::Assns<recob::Shower, recob::Track, void>* *
* Association Name: ShowerTrackHitAssn * Instance Name: * Type: art::Assns<recob::Track, recob::Hit, void>* *
* Association Name: clusterAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::Cluster, void>* *
* Association Name: hitAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::Hit, void>* *
* Association Name: pfShowerAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::PFParticle, void>* *
* Association Name: spShowerAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::SpacePoint, void>* *
*************************************************************************************************************************************
: Booking "electron" of type "BDT" from /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/electron_all_BDT.weights.xml.
: Reading weight file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/electron_all_BDT.weights.xml
<HEADER> DataSetInfo : [Default] : Added class "Signal"
<HEADER> DataSetInfo : [Default] : Added class "Background"
: Booked classifier "BDT" of type: "BDT"
: Booking "muon" of type "BDT" from /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/muon_all_BDT.weights.xml.
: Reading weight file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/muon_all_BDT.weights.xml
: Booked classifier "BDT" of type: "BDT"
: Booking "photon" of type "BDT" from /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/photon_all_BDT.weights.xml.
: Reading weight file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/photon_all_BDT.weights.xml
: Booked classifier "BDT" of type: "BDT"
: Booking "pich" of type "BDT" from /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/pich_all_BDT.weights.xml.
: Reading weight file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/pich_all_BDT.weights.xml
: Booked classifier "BDT" of type: "BDT"
: Booking "proton" of type "BDT" from /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/proton_all_BDT.weights.xml.
: Reading weight file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/mvapid_weights/proton_all_BDT.weights.xml
: Booked classifier "BDT" of type: "BDT"
19-Nov-2024 03:54:56 PST Initiating request to open input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/1c/b9/nutau_dune10kt_1x2x6_1105_335_20230826T143334Z_gen_g4_detsim_hitreco__20240220T113538Z_reco2.root"
19-Nov-2024 03:54:56 PST Initiating request to open input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/1c/b9/nutau_dune10kt_1x2x6_1105_335_20230826T143334Z_gen_g4_detsim_hitreco__20240220T113538Z_reco2.root"
MultiPandoraApiImpl::DeletePandoraInstances - unable to find daughter instances associated with primary 0
%MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 19-Nov-2024 03:54:59 PST ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Open
[FATAL] Auth failed: No protocols left to try
ROOT severity: 3000
---- FatalRootError END
RootInputFileSequence::initFile(): Input file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/1c/b9/nutau_dune10kt_1x2x6_1105_335_20230826T143334Z_gen_g4_detsim_hitreco__20240220T113538Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
%MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 19-Nov-2024 03:54:59 PST ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Open
[FATAL] Auth failed: No protocols left to try
ROOT severity: 3000
---- FatalRootError END
RootInputFileSequence::initFile(): Input file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/1c/b9/nutau_dune10kt_1x2x6_1105_335_20230826T143334Z_gen_g4_detsim_hitreco__20240220T113538Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
Art has completed and will exit with status 20.
=== End last 100 lines of lar log file === 