Interesting Bash Programming Tricks
These tricks were described in Google’s internal project “Testing on the Toilet” (handing out flyers in restrooms to remind developers about testing).
This post reviews and expands on them.
Security
I start every script with the following lines
#!/bin/bash
set -o nounset
set -o errexit
This protects against two common mistakes
- Trying to use undeclared variables
- Ignoring command failures
If a command can fail and that’s fine with us, we can use this code:
if ! ; then
echo "failure ignored"
fi
Keep in mind that some commands don’t return a failure exit code, for example “mkdir -p” and “rm -f”.
| There are also complications with calling chains of subcommands (command1 | command2 | command3) from a script; to work around this limitation you can use the following construct: |
(./failing_command && echo A)
In this case the && operator won’t let the next command run — more on this here: Bash: Error_handling
Functions
Bash lets you use functions like regular commands, which greatly improves your code’s readability:
Example 1:
ExtractBashComments() {
egrep "^#"
}
cat myscript.sh | ExtractBashComments | wc
comments=$(ExtractBashComments < myscript.sh)
Example 2:
SumLines() { # iterating over stdin - similar to awk
local sum=0
local line=””
while read line ; do
sum=$((${sum} + ${line}))
done
echo ${sum}
}
SumLines < data_one_number_per_line.txt
Example 3:
log() {
# classic logger
local prefix="[$(date +%Y/%m/%d\ %H:%M:%S)]: "
echo "${prefix} $@" >&2
}
log "INFO" "a message"
Try moving all of your code into functions, leaving only global variables/constants and a call to a main function that holds all the high-level logic.
Variable declarations
Bash lets you declare several types of variables, the most important being:
- local (for variables used only inside functions)
- readonly (variables where reassignment attempts trigger an error)
## If DEFAULT_VAL is already declared, use its value, otherwise use '-7'
readonly DEFAULT_VAL=${DEFAULT_VAL:-7}
myfunc() {
# Using a local variable set to the global variable's value
local some_var=${DEFAULT_VAL}
...
}
You can also turn an already-declared variable into a readonly one:
x=5
x=6
readonly x
x=7 # failure
Aim to make all your variables either local or readonly — it improves readability and cuts down on bugs.
Use $() instead of backticks ` `` `
Backticks are hard to read and in some fonts can easily be mistaken for single quotes.
The $() construct also lets you nest calls without the headache of escaping:
# both commands print: A-B-C-D
echo "A-`echo B-\`echo C-\\\`echo D\\\`\``"
echo "A-$(echo B-$(echo C-$(echo D)))"
Use double square brackets [[]] instead of single []
Double square brackets help you avoid accidentally using paths instead of variables:
$ [ a < b ]
-bash: b: No such file or directory
$ [[ a < b ]]
In some cases they simplify the syntax:
[ "${name}" \> "a" -o ${name} \< "m" ] [[ "${name}" > "a" && "${name}" < "m" ]]
They also provide extra functionality:
New operators:
-
** ** Logical OR (logical or) — double brackets only. - && Logical AND (logical and) — double brackets only.
- < String comparison (string comparison) — no escaping needed with double brackets.
- == String matching with globbing (string matching with globbing) — double brackets only.
- =~ String matching with regular expressions (string matching with regular expressions) — double brackets only.
Extended/changed operators:
- -lt Numerical comparison (numerical comparison)
- -n String is non-empty (string is non-empty)
- -z String is empty (string is empty)
- -eq Numerical equality (numerical equality)
- -ne Numerical inequality (numerical inequality)
Examples:
t="abc123"
[[ "$t" == abc* ]] # true (globbing)
[[ "$t" == "abc*" ]] # false (literal matching)
[[ "$t" =~ [abc]+[123]+ ]] # true (regular expression)
[[ "$t" =~ "abc*" ]] # false (literal matching)
Starting with bash 3.2, regular expressions or globbing expressions don’t need to be quoted; if your expression contains spaces, you can put it into a variable:
r="a b+"
[[ "a bbb" =~ $r ]] # true
String matching with globbing is also available in the case statement:
case $t in
abc*) ;;
esac
Working with string variables:
Bash has several (underrated) built-in ways to work with string variables:
Basics:
f="path1/path2/file.ext"
len="${#f}" # = 20 (length of the string variable)
# extracting a slice from a variable: ${<variable>:<start>} or ${<variable>:<start>:<length>}
slice1="${f:6}" # = "path2/file.ext"
slice2="${f:6:5}" # = "path2"
slice3="${f: -8}" # = "file.ext" #(note the space before the '-' sign)
pos=6
len=5
slice4="${f:${pos}:${len}}" # = "path2"
Substitution replace:
f="path1/path2/file.ext"
single_subst="${f/path?/x}" # = "x/path2/file.ext" #(replaces the first match)
global_subst="${f//path?/x}" # = "x/x/file.ext" #(replaces all matches)
Splitting variables:
f="path1/path2/file.ext"
readonly DIR_SEP="/"
array=(${f//${DIR_SEP}/ })
second_dir="${arrray[1]}" # = path2
Removal via substitution:
# Remove from the start of the string up to the first match
f="path1/path2/file.ext"
extension="${f#*.}" # = "ext"
Remove from the start of the string up to the last match
f="path1/path2/file.ext"
filename="${f##*/}" # = "file.ext"
Remove from the end of the string up to the first match
f="path1/path2/file.ext"
dirname="${f%/*}" # = "path1/path2"
Remove from the end of the string up to the last match
f="path1/path2/file.ext"
root="${f%%/*}" # = "path1"
Getting rid of temp files
Some commands expect a filename as input; the <() operator helps here — it takes a command and turns it into something you can use as a filename:
# download two URLs and pass them to diff
diff <(wget -O - url1) <(wget -O - url2)
Using a marker to pass multiline variables:
# MARKER — any word.
command << MARKER
...
${var}
$(cmd)
...
MARKER
If you need to avoid substitution, you can quote the marker:
# this construct returns the literal '$var', not the variable's value
var="text"
cat << 'MARKER'
...
$var
...
MARKER
Built-in variables
- $0 Name of the script (name of the script)
- $1 $2… $n Parameters passed to the script/function (positional parameters to script/function)
- $$ PID of the script (PID of the script)
- $! PID of the last command run in the background (PID of the last command executed (and run in the background))
- $? Exit status returned by the last command (exit status of the last command (${PIPESTATUS} for pipelined commands))
- $# Number of parameters passed to the script/function (number of parameters to script/function)
- $@ All parameters passed to the script/function, as separate words (sees arguments as separate word)
- $* All parameters passed to the script/function, as a single word (sees arguments as single word)
As a rule:
- $* Rarely useful
- $@ Correctly handles empty parameters and parameters with spaces
- $@ Usually wrapped in double quotes when used —
$@
Example:
for i in "$@"; do echo '$@ param:' $i; done
for i in "$*"; do echo '$! param:' $i; done
output:
bash ./parameters.sh arg1 arg2
$@ param: arg1
$@ param: arg2
$! param: arg1 arg2
Debugging
Syntax check (saves time if your script takes more than 15 seconds to run):
bash -n myscript.sh
Tracing:
bash -v myscripts.sh
Tracing with expansion of complex commands:
bash -x myscript.sh
The -v and -x options can also be set directly in the code, which can be useful if your script runs on one machine while logging happens on another:
set -o verbose
set -o xtrace
Signs that you shouldn’t be using shell scripts:
- Your script is more than a few hundred lines long.
- You need data structures more complex than plain arrays.
- You’re sick of wrestling with quoting and escaping.
- You need to process/modify a lot of string variables.
- You don’t need to call external programs and don’t need folders.
- Speed/performance matters to you.
If your project matches items from this list, consider using Python or Ruby instead.
Links: