Every time I login to our company cluster, I first get a Kerberos ticket and then try to submit a job, but it fails to find spark-ubmit:
-bash-4.1$ ./pipeline.runtime/scripts/vpspark-submit.sh crow-env/ /user/gsamaras/crow_spm-1.0.2.3.tar.gz --properties-file $SPARK_CONF_PROPERTIES --num-executors 1 foo.py
./pipeline.runtime/scripts/vpspark-submit.sh: line 51: spark-submit: command not found
If however, I source the .bashrc, it works fine.
>> cat .bashrc
[ -f /etc/bashrc ] && . /etc/bashrc
export PATH=${SPARK_HOME}/bin:$PATH
export SPARK_CONF_DIR=~/tripodrt-spark/conf/
export SPARK_CONF_PROPERTIES=~/tripodrt-spark/conf/spark-overrides-tt.conf
Why? I would expect that .bashrc runs on login. I am on Red Hat Enterprise Linux.
Best Answer
From the
bash
manual:So if your shell is a login shell,
~/.bashrc
is not read.I would put additions to
PATH
in~/.bash_profile
instead, and/or possibly source~/.bashrc
from there.Some shells makes a distinction between two or three types of shell initialization files:
For
bash
:~/.bash_profile
~/.bashrc
$BASH_ENV
points to.The point of separating the profile startup script from the script read by interactive shells is to let you do (potentially heavy) set-up tasks for all your future shell sessions once and for all. This usually involves setting the
PATH
at the minimum, but also any other exported variables (i.e. environment variables).The interactive shell initialization script (
~/.bashrc
in the case ofbash
) then only needs to do thing specifically needed for interactive shells. This includes setting shell variables likePS1
that shouldn't be exported, and defining aliases and functions etc. You should also set up environment variables that need to be set for this session and any started process from this session, such asGPG_TTY
(GPG_TTY="$(tty)"
) if you're using GnuPG.I tend to
source $HOME/.bashrc
from my~/.bash_profile
after setting myPATH
and adjusting myumask
the way I want them, and let~/.bashrc
set up history file variables, thePS1
prompt,PAGER
andEDITOR
and a few aliases and functions and whatnot. (That's half a lie; I actuallyexec
theksh93
shell from my profile if I can find it, but if I can't, I continue withbash
). I also renice my shell in the interactive shell initialization file (withrenice -n 4 $$
), because I often work on a shared machine with loads of other people that needs the CPU more than I do (but that's just me).The
ksh93
shell uses~/.profile
~.kshrc
(or whatever$ENV
points to)ksh93
is a bit saner thanbash
in that it knows there is such a thing as an interactive login shell, so it reads$ENV
(if it is set) for those shell sessions too, after reading~/.profile
.The
pdksh
shell also reads~/.kshrc
for non-interactive shells, which annoys me (and breaks a few important scripts that I use), so I haveat the top of my
~/.kshrc
file.