bash and shell scripting seems to be a problem for many
people, including capable programmers in other languages.
This
guide is not a list if tips and tricks. Its a description of the syntax
and a list of things I wish someone had explained to me before I
started.
I've had a fair bit of experience with bash
over the years, hacked at the C code added features and extensions.
None of those extensions I will recommend. This post is about writing
vanilla bash scripts.
When to use bash
From Google's bash style guide: "
Shell should only be used for small utilities or simple wrapper scripts."
There is no loss in starting a util with bash and, if it grows up, porting it to a "proper" language.
That said, it is difficult to have to deal with little utils and applications written in esoteric languages or compiled C that could have been a shell script.
One on the main reasons to use shell is that it is decomposable.
You run it, it works: and vim will tell you how. Copy paste will let you run parts of it. This is a pretty neat feature and not to be scoffed at.
The main rule for bash code is to keep it simple, that does not mean you cannot achieve great things. I have written a replacement for Jenkins on a raspberrypi as a few lines of bash and, famously, git was written as shell scripts.
ToC
- Syntax / whitespace rules
- How quotes work
- Why builtins exist
- pwd
Syntax
Syntax seems
deceptively simple, many folk tear their hair out with quotes, $var, and whitespace, because they have not yet assimilated the rules.
A shell script is generally a text file made up of command lines to execute,
tokenized by space (seems obvious right) but it is not always.
myprog arg1 arg2
e.g.
mkdir -p target/
N.B. this is split by bash into an array
[ 'mkdir' ,'-p', 'target/' ]
There are times when bash's behavior seems erratic but it
is quite consistent.
Assignments
The statement
myvar=foo
is tokenized to a _single_ token
[ 'myvar=foo' ] which is a command that bash knows how to interpret, the string
foo is assigned to a variable called
myvar.
If we change the whitespace
myvar = foo
is tokenized to
[ 'myvar', '=', 'foo'] bash tries to execute a command called
myvar passing the parameters
= and
foo.
If you are lucky
myvar does _not_ exist and you get an error, if you are unlucky
myvar exists and is executed.
This is tends to be confusing because other languages parse both
a=b and
a = b as 3 tokens.
bash rules are quite simple, but different,
the command line is tokenized with whitespace.
[
Take, for example, the following test
[ -z "$myvar" ] && echo missing variable: myvar
-z means zero length string, this line checks to see if
myvar is empty and if so, it is prints a message
missing variable: myvar.
There are two things that confuse people here...
The following does not work
[-z $myvar] && echo missing variable: myvar
[ is a command called
[ it _requires_ whitespace to tokenize.
assuming
myvar is missing
[-z $myvar] is tokenized to
[ '[-z' ,']' ] i.e. bash looks for a program called
[-z and passes it the argument
] and you get a cryptic message
[-z: command not found
Remember that bash tokenizes lines with whitespace and the error messages will not seem so confusing.
echo
echo concatenates all its arguments with 1 space.
echo missing variable: myvar
Is tokenized to
[ 'echo', 'missing', 'variable:', 'myvar' ] and echo prints it as
missing variable: myvar
N.B. without the extra whitespace before
myvar, seems a bit confusing at first, but your script is not "stripping whitespace", bash is tokenizing and echo is printing what it receives.
If you quote it
echo 'missing variable: myvar'
Cli is tokenized to
[ 'echo', 'missing variable: myvar' ] and now echo prints with the correct whitespace.
missing variable: myvar
Concatenating arguments is
only a feature of echo, its not a general practice.
git commit -m documentation changes
Will submit just "documentation" as the comment (the parameter to -m).
git commit -m "documentation changes"
Passes the single argument with the text.
Quotes
The rules for quoting strings are
different from other languages. The basic rules for quoting are simple but can lead to confusion, and there are confusing exceptions.
The basic rules are
1. everything is a string
2. spaces require quotes
3. single quotes do no escaping or replacements
4. double quotes support escaping and replacements
1 Everything is a string
There are no number data types as with other languages.
"1" and
'1' and
1 are all the string "1".
Programs may interpret the strings as numbers, but bash does not.
bc is a command line calculator
echo 1 + 1 | bc
prints
2
bash tokenized the command to
[ 'echo', '1', '+', '1', '|', 'bc' ]
bc converted the string "1 + 1" to number.
2 Spaces require quotes
If you want a space in a string you must quote it.
"hello world"
or single quotes
'hello world'
Unlike JavaScript and python,
" and
' are very different.
3 Single quotes do no replacements or escaping
Single quotes do
no changes at all to the string.
singe quotes...
myvar=foo
echo 'myvar: $myvar'
prints
myvar: $myvar
double quotes...
myvar=foo
echo "myvar: $myvar"
prints
myvar: foo
In single quotes there is
no escaping supported not even
', there is no way to escape
' inside single quotes.
4 Double quotes support escaping and replacements
Double quotes do
replacement magic and
escaping magic.
"Replacement" replaces
$var` with the value from an environment variable.
"Escaping" is prefixing some characters with
\
- \" escapes to just "
- \$ escapes to just $
so
myvar=cruel
echo "hello \"$myvar\" world"
does escaping and replacing to print
hello "cruel" world
N.B.
${} syntax has a lot more magic than you might expect. Details follow.
Using the simple rules
When
no quotes are supplied strings are tokenized to multiple strings, escaping and replacements work without quotes.
myvar=foo
echo $myvar
Its good practice to quote strings with spaces even it does not really matter as with echo.
echo "out:failed:missing_file"
Any string can be quoted or un-quoted, its possible but bad practice to quote programs and arguments that don't have spaces.
The following is valid but looks a bit weird
"mkdir" "logs"
"rm" '-rf'
N.B. there is nothing special about args this is just a convention
grep "-red"
Is the same as
grep -red
grep will interpret the first parameter as argument"
-", it will not search for the string "-red".
echo newlines
Newlines terminate the command line and, as with JavaScript semicolons
; are optional.
Unlike most other languages a new line
is valid inside a string.
echo '
one
two
three
'
prints the new lines.
one
two
three
You may see
here docs in existing scripts
echo <<EOF
one
two
three
EOF
Single and double quotes are often more convenient.
Empty strings
Empty string can be a bit complicated
myprog $myvar
is subtly different to
myprog "$myvar"
"" is an empty string, it
is a token. If
$myvar is missing it is
not an empty string.
The former will pass a single argument, the empty string, to
myprog the later will not send any arguments.
This can be important if arguments are expected
myprog -v -m $foo -b $baa
Is better written, as follows if the variables may be empty.
myprog -v -m "$foo" -b "$baa"
Breaking the rules
Now you know the rules, and recognize how simple they are, you will be disheartened to know that there are couple of constructs that break these simple rules. Fortunately they behave a bit more "normally".
((
In later versions of bash there is the
(( construct, this is not a program called
((, its proper syntax and its parsing rules are different.
Inside double parens, the simple whitespace tokenizing rules are broken and things that look like numbers are numbers. The rules are much more complicated, they are undocumented C-style constructs, but they are familiar if you write and C based languages like JavaScript.
Basically, you can do math, if
myvar looks like a number
((myvar++))
works as "expected".
((myvar+=2))
also works as "expected" provided you were not expecting the simpler whitespace token rules.
[[
[[ is a more modern version of
[ its rules are subtly different,
[[ is designed to be more natural and its use is recommended. Again [[ is
not a command it is proper syntax.
- missing vars $nothere generate empty strings.
- && and || mean "and" and "or" (they do not terminate the command)
- < and > mean greater and less than (not redirects)
- ~ is not $HOME (its used for regexp)
- globs *.* and regexp don't need (so much) escaping
The result is generally code that look more like other languages, despite breaking the simple rules.
"\ "
Rule 2 "spaces require quotes" is not 100% true, you
can escape space if it is outside quotes.
It is usually best to ignore this fact in scripts.
Escaping spaces looks pretty odd in scripts.
echo hello\ world
Escaping is more useful on the cli with tab completion.
Understanding Builtins
aka command vs programs.
Typically a the first token in a command line is a single "command" and all the others are "arguments".
echo hello world
The "command" is "echo" and the arguments are "hello" and "world". Bash has a few "builtins" i.e. commands that do not need an external program to run. Builtins are checked first, if one does not exist, bash looks for a program.
There
is a builtin called
echo, were it not there, bash would look for a program called
echo and would probably find
/bin/echo.
The builtin
echo and the program
/bin/echo are similar but its important to know they are not the same thing.
Builtin rules
Builtins run
inside the bash process so they
can change bash's behavior and state.
Programs run in a
separate process so then
cannot change bash's state.
This is why commands like
set and
export must be builtins, they change bash's state.
echo does not change bash's state, it could be a builtin or a program.
echo exists as a builtin as an optimization, it saves forking a process to print a line of output.
So next time you ask yourself how can I run a "program" that changes a variable in the current shell, you know the answer, you cant! You have to use a builtin.
There are various builtins and ways to fork processes, the following list servers to explain
why builtins exist, its not comprehensive.
cd
cd changes the current directory of the current bash shell, therefor it must be a builtin.
source
source runs a whole script (excluding the shebang) inside the current process without forking, therefor it must be a builtin. The script it runs must be a bash script since it runs inside the current bash process, you cant source a python script. Source is like a dynamic include.
Because source is a builtin it
can change bash's state. So if you want to write a script that changes lots of variables in the current bash instance you should use
source.
e.g. you could create a file
all-my-vars.sh that has lots of handy variables like
LOG=/var/log/mylog
BIN=/usr/lib/myprog/bin
etc. To use the variables in the
current script or the
current shell use
source.
source ./all-my-vars.sh
There is a shortcut for this the period ".".
. ./all-my-vars.sh
Careful with the difference between
./all-my-vars.sh
and
. all-my-vars.sh
The former "runs the script", it will not affect the current instance, the latter "sources the script" and
will affect the current instance.
export
myvar=foo
Changes the environment of the current bash shell, therefor it must be a builtin.
export myvar=foo
Changes the environment of the current bash shell, it also tells bash to set the myvar variable in any subshell it creates from now on.
myvar=foo myprog
sets the myvar variable just for the single execution of the myprog command.
exec
exec transmogrifies the bash process into a different program. Its pretty magic, since it very much changes the state of bash it must be a builtin.
If at the end of a script you write
exec myprog arg1 arg2
When you run the script, the bash instance will "disappear" and will be replaced by
myprog, when you look in the
ps list you will not see bash.
This is handy for launch scripts. For example if you want to write a script
./run-myprog.sh to run a program called
myprog that creates a needed directory first, you could write
#!/bin/bash
mkdir -p /tmp/myprog
myprog $1
When you run this and look in the
ps list you will see.
me 123 44 bash ./run-myprog.sh arg1
me 124 123 myprog arg1
Using exec builtin the bash process will magically disappear
#!/bin/bash
mkdir -p /tmp/myprog
exec myprog arg1
When you run this and look in the
ps list you will just see.
me 124 123 myprog arg1
If you are using lots of little scripts to setup launching a daemon, judicious use of
source and
exec can keep the number of bash instances in the
ps list during and after the boot to a minimum.
Subshells
Subshells are bash instances started by bash, there are a few syntax options.
bash ./myscript
(./myscript)
Subshells are new processes, not builtins, they cannot change the current process.
You can capture the output of a subshells, you have two syntax options, backticks or the more modern dollar parens.
output=`./myscript`
output=$(myscript)
String manipulation
Linux provides lots of tools for processing strings in pipes. e.g.
sed and
awk it is tempting to use subshells and pipes to change strings
new_string=`echo $oldstring | cut 5-10 | tr -d '\n'`
For example, if you have a variable
my_file containing
myprog.dat and you want to create a filename called
myprog.bak.
You could solve this with subshells and Linux tools e.g.
new_name=$(echo $my_file | sed -e 's/.dat$//')
new_name=${new_name}.bak
There is nothing wrong with this approach except perhaps performance. Bash ninjas would prefer not to fork processes and will tend to use the builtin string manipulation features of bash. This is a big subject, the important thing to know is that...
- bash has very flexible string manipulation
- the syntax is screwy
The basic syntax is
${VAR [insert magic] } where magic has operators such as
%,
#,
/, and
:
Regexp is the easiest to remember since it uses
/ / similar to sed.
new_name=${my_file/.dat/.bak}
Escaping regexp is complicated. In this case trim is more reliable.
new_name=${my_file%.dat}.bak
I would recommend commenting such magic.
% is not a common trim operator.
The performance benefits are often negligible, kudos on StackOverflow is priceless.
If you find yourself using string manipulation a lot in one particular script, it is an indicator you should port to a proper language.
PWD concenrs
Every unix process has a current working directory PWD, bash included, it changes it more often than most programs.
Usually the same command works on the command line, as in a script.
mkdir -p target/
typed on the cli and in a script has the same effect.
Where the target dir is created is dependent on your current directory, so its best to always use absolute paths in scripts.
Or add
cd $(dirname $0)
at the start so that all commands run from the directory where the script is located.
e.g. if you have a script called
/home/me/workspace/init.sh
#!/bin/bash
cd $(dirname $0)
mkdir -p target/
and you run it as
./init.sh or from home as
workspace/init.sh it will have the same result.
Subshells can have a different PWD and do not affect the parent shell. This can be useful to temporarily change directory.
You could save the PWD and cd back to it after your change...
OLDPWD=$PWD
cd /var/log/$myprogname
cat $myprogname.log
cd $OLDPWD
unset OLDPWD
This is neater with subshells
( cd /var/log/$myprogname; cat $myprogname.log )
Handy for executing prgrams the are dependent on the PWD.
( cd src/c; make )
Or running a series of commands in a different directory
(
cd /tmp
touch foo
touch baa
touch quxx
)
Now what?
That was not really too much to digest was it. I get called over to help with tricky bash issues quite a lot and 9 times out of 10 the problems is caused by one of these issues. Hence writing this doc.
Once you know the basics, I highly recommend you read
man bash through once. It is dull, but there you will find a ton of tricks.
For sane coding conventions I would recommend (
Google's shell style guide)
Google come down hard on the the tabs vs spaces issue. It makes sense because you really want to be able to copy paste and tab is introduces a serious risk if you do that. You really should avoid tabs in new code.
For sane design guide, KISS cannot be overstated.
