justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Workflow 6449, Stage 1

Priority50
Processors1
Wall seconds50000
Image/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest
RSS bytes4193255424 (3999 MiB)
Max distance for inputs100.0
Enabled input RSEs CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, MONTECARLO, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC
Enabled output RSEs CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC
Enabled sites BR_CBPF, CA_SFU, CA_Victoria, CERN, CH_UNIBE-LHEP, ES_CIEMAT, ES_PIC, FR_CCIN2P3, IN_TIFR, IT_CNAF, NL_SURFsara, UK_Bristol, UK_Brunel, UK_Durham, UK_Edinburgh, UK_Glasgow, UK_Imperial, UK_Lancaster, UK_Liverpool, UK_Manchester, UK_Oxford, UK_QMUL, UK_RAL-PPD, UK_RAL-Tier1, UK_Sheffield, US_Caltech, US_Colorado, US_FNAL-FermiGrid, US_FNAL-T1, US_Michigan, US_MIT, US_Nebraska, US_NotreDame, US_PuertoRico, US_SU-ITS, US_Swan, US_UChicago, US_UConn-HPC, US_UCSD, US_Wisconsin
Scopeusertests
Events for this stage

Output patterns

 DestinationPatternLifetimeFor next stageRSE expression
1Rucio usertests:calcuttj_g4bl_prod_full_neg1_042425-w6449s1p1*root2592000False

Environment variables

NameValue
G4BL_DIR/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/023b964308029fd1755f5b4a2ae2fc05d5107859/
G4DATA_DIR/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/754ab2932db4c5a30549905327154f5c7a9f083c/
INPUT_DIR/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/40a9891e2a7c1f62d618824f8f8588fe2ebec5fe/
NPART100000
PACK_DIR/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/3851f3036b8fdf366c8f0dd3fe8fea81c1d30f87/
POLARITY-

File states

Total filesFindingUnallocatedAllocatedOutputtingProcessedNot foundFailed
10000000098170183

Job states

TotalSubmittedStartedProcessingOutputtingFinishedNotusedAbortedStalledJobscript errorOutputting failedNone processed
2990600001077901187950229102
Files processed002002004004006006008008001000100012001200140014001600160018001800Apr-28 21:00Apr-29 01:00Apr-29 05:00Apr-29 09:00Apr-29 13:00Apr-29 17:00Apr-29 21:00Apr-30 01:00Apr-30 05:00Apr-30 09:00Apr-30 13:00Apr-30 17:00Apr-30 21:00May-01 01:00May-01 05:00May-01 09:00May-01 13:00May-01 17:00May-01 21:00May-02 01:00May-02 05:00May-02 09:00Files processedBin start timesNumber per binUS_FNAL-FermiG…US_FNAL-FermiGridCERNFR_CCIN2P3US_WisconsinUK_OxfordUK_QMULUK_DurhamUS_ColoradoUK_RAL-PPDUS_UChicagoUK_EdinburghUK_GlasgowUK_ImperialUK_LancasterUK_RAL-Tier1US_FNAL-T1UK_LiverpoolUK_BrunelES_PICUK_SheffieldUS_UConn-HPCUK_ManchesterBR_CBPFUK_Bristol

RSEs used

NameInputsOutputs
MONTECARLO258770
DUNE_CERN_EOS04524
DUNE_US_FNAL_DISK_STAGE02058
RAL-PP01063
RAL_ECHO0841
DUNE_FR_CCIN2P3_DISK0346
DUNE_IT_INFN_CNAF0309
DUNE_UK_LANCASTER_CEPH0239
QMUL0190
DUNE_UK_GLASGOW098
DUNE_UK_MANCHESTER_CEPH083
DUNE_US_BNL_SDCC058
DUNE_CA_SFU02

Stats of processed input files as CSV or JSON, and of uploaded output files as CSV or JSON (up to 10000 files included)

File reset events, by site

SiteAllocatedOutputting
CERN5358438
UK_RAL-Tier1202158
US_FNAL-FermiGrid1683495
UK_RAL-PPD72017
ES_PIC60629
FR_CCIN2P348064
UK_QMUL47141
US_NotreDame4480
UK_Durham38136
US_UChicago36532
UK_Imperial33537
UK_Lancaster23414
US_Colorado22325
UK_Glasgow21616
UK_Liverpool1640
US_FNAL-T11603
UK_Oxford15511
US_Wisconsin15216
UK_Edinburgh12114
UK_Manchester731
BR_CBPF620
UK_Brunel361
UK_Sheffield291
US_UConn-HPC210
US_PuertoRico140

Jobscript

#!/bin/bash

SECONDS=0

## Set the poolarity
export POLARITY="${POLARITY:-+}"
if [ $POLARITY != "+" ] && [ ${POLARITY} != "-" ]; then
  echo "ERROR MUST SUPPLY + OR - TO POLARITY"
  exit 1
fi

##Set the upstream momentum
export PMOMENTUM="${POLARITY:-+}80000"
echo "PMOMENTUM: ${PMOMENTUM}"

export BEAMLINE="${BEAMLINE:-H4}"
export INFILE=${BEAMLINE}.in
export CENTRALP="${CENTRALP:-1}" #Set the momentum going into protodune
export MOMENTUMVLE="${POLARITY}${CENTRALP}" #"3"
echo "MOMENTUMVLE: ${MOMENTUMVLE}"

##Number of POT to run
export PARTPERJOB=${NPART:-100}
export ADDPARAM="momentumVLE=$MOMENTUMVLE pMomentum=$PMOMENTUM"
export ADDFILES=""


if [ -z ${JUSTIN_PROCESSORS} ]; then
  JUSTIN_PROCESSORS=1
fi

echo "Justin processors: ${JUSTIN_PROCESSORS}"

export TF_NUM_THREADS=${JUSTIN_PROCESSORS}   
export OPENBLAS_NUM_THREADS=${JUSTIN_PROCESSORS} 
export JULIA_NUM_THREADS=${JUSTIN_PROCESSORS} 
export MKL_NUM_THREADS=${JUSTIN_PROCESSORS} 
export NUMEXPR_NUM_THREADS=${JUSTIN_PROCESSORS} 
export OMP_NUM_THREADS=${JUSTIN_PROCESSORS}  

JOBID=1
#JOBID=`echo "${JUSTIN_JOBSUB_ID:-1}" | cut -f1 -d'@' | sed -e "s/\./_/"`
#echo "Jobid: ${JOBID}" 
#export JOBID=$(($st+1)) ##TODO -- REPLACE WITH JUSTIN JOB

##Get the MC number from this to bookkeep for justin
DID_PFN_RSE=`$JUSTIN_PATH/justin-get-file`
pfn_exit=$?
if [ $pfn_exit -ne 0 ]; then
  echo "Error in justin-get-file. Exiting safely"
  exit 0
fi
echo "did_pfn_rse $DID_PFN_RSE"
pfn=`echo $DID_PFN_RSE | cut -f2 -d' '`

echo $INPUT_DIR
ls $INPUT_DIR

echo $G4DATA_DIR
ls $G4DATA_DIR
#cp -rs $G4DATA_DIR ./Geant4Data/
ln -s $G4DATA_DIR/Geant4Data ./Geant4Data

echo $G4BL_DIR
ls $G4BL_DIR
cp -rs $G4BL_DIR/g4bl ./g4bl
export G4BL_DIR=$PWD/g4bl

echo $PACK_DIR
ls $PACK_DIR
for i in $PACK_DIR/*; do
  ln -s $i .
done

#Unpack all the tars -- TODO: put on cvmfs as a single tar
#echo "Unpacking g4bl"
#tar -xzf $INPUT_DIR/g4bl.tar.gz --checkpoint=1000
#if [ $? -ne 0 ]
#then
#  echo "Exiting with error"
#  exit 1
#fi
#
#echo "Unpacking Geant4Data"
#tar -xzf $INPUT_DIR/Geant4Data.tar.gz --checkpoint=1000
#if [ $? -ne 0 ]
#then
#  echo "Exiting with error"
#  exit 1
#fi
CURDIR=$(pwd)
echo $CURDIR/Geant4Data > g4bl/.data


#echo "Unpacking Inputfiles Pack"
#tar -xzf $INPUT_DIR/pack.tar.gz --checkpoint=1000
#if [ $? -ne 0 ]
#then
#  echo "Exiting with error"
#  exit 1
#fi

#Run
echo "running"
#$CURDIR
./g4bl/bin/g4bl $INFILE jobID=$JOBID totNumEv=$PARTPERJOB $ADDPARAM 2>&1 | tee g4bloutput.txt
g4bl_res=$?
if [ $g4bl_res -ne 0 ]
then
  echo "Failed running g4bl"
  exit $g4bl_res
fi
echo "ran"

#Clean up
unlink Geant4Data
rm -rf g4bl
for i in *.in *.map; do
  unlink ${i}
done



#Add timestamp to the output
now=$(date -u +"%Y%m%dT%H%M%SZ")
oldname=`ls ${BEAMLINE}*.root`
newname=`echo ${oldname} | sed -e "s/.root/_${now}_${pfn}.root/"`
mv ${oldname} ${newname}
if [ $? -ne 0 ]
then
  echo "Failed renaming ${oldname} ${newname}"
  exit 1
fi


if [ $POLARITY != "+" ]; then
  polar_str="neg"
else
  polar_str="pos"
fi

subrun=`echo $JUSTIN_JOBSUB_ID  | cut -f1 -d@ | cut -f2 -d.`
run=`echo $JUSTIN_JOBSUB_ID  | cut -f1 -d@ | cut -f1 -d.`

python $INPUT_DIR/make_g4bl_metadata.py \
  --bl "${BEAMLINE}" --polarity $polar_str --momentum $CENTRALP \
  --run $run --subrun $subrun --name $newname \
  --namespace ${JUSTIN_SCOPE:-dummy}

if [ $? -ne 0 ]
then
  echo "Exiting with error"
  exit 1
else
  echo "$pfn" > justin-processed-pfns.txt
fi

#errorsSaving=$((`cat g4bloutput.txt | grep "Error in <T" | wc -l`))
#if [ $errorsSaving -ne 0 ]
#then
#  echo "Exiting with error"
#  exit 1
#fi
#
#echo "RUNTIME: $SECONDS seconds elapsed."
justIN time: 2025-08-15 02:59:06 UTC       justIN version: 01.03.02