My bash scripting cheatsheet after maintaining a couple tools for 4+ years: use system Ruby / Python or anything else instead, really. Once over a hundred lines of code it becomes a massive pain to work with.
The biggest benefit to Bash scripting over a friendlier language is that Bash is guaranteed to just work on pretty much every machine. Bash is on virtually every *nix machine (except maybe embedded devices), and it's extremely resilient to time. Looking through the change history (https://wiki.bash-hackers.org/scripting/bashchanges), I'd wager that 99% of scripts don't use anything that wasn't in Bash 2.05b, which came out in Jul 2002. Anecdotally, the "newest" feature that I've ever used is regex support in conditional expressions, which was added in 3.0-alpha (Aug 2004).
Ruby and Python, meanwhile, are not quite so universally installed as Bash, and they have had all sorts of major breaking changes since 2004. And even if they are installed and version-compatible, they still might not work. Countless times have I seen a misconfiguration with Python 2 vs 3 or RVM or the PATH variable or whatever else cause errors, but I have never run into a scenario where a Bash script didn't just work.
All that said, I'm totally with you that Bash scripts should be short and sweet. If you're doing something with over 100 lines, Bash's downsides overtake its benefits.
It's true that bash can be installed on any Unix-like OS, but so can Python, Perl, etc, but it's not available by default, on OpenBSD there's pdksh, FreeBSD tcsh iirc, and on MacOS I believe the version of bash is really old due to some previous license change.
Edit: I read your post quickly, you addressed the old bash scenario, but still in my opinion I would always, and do target POSIX shell for maximum cross-platform compatibility and that is definitely available on Unix-like OSs by default.
That’s true, and it’s the reason the tools I mentioned were written in Bash in the first place.
Python 2/3 was a pain, Ruby had very disparate versions available. But now, a few years later, you can assume at least Ruby 1.9 nearly everywhere, and OTOH still can’t use any of the Bash 4 features. There also quite decent cross-platform packaging solutions for python, ruby, node, go, crystal, mruby that make it really hard to choose bash for any serious project.
Python is fantastic if it requires functions or anything over several lines. I'm not a Python dev but I can very quickly find standard library examples or well-documented packages.
Shell scripting has a place and script languages aren't always the answer for improvement. For many tasks it can even be more verbose but logic crafting can be easier.
You should rather look for languages like fish, which makes shell scripting feel better than drop it altogether.
For Python, I have used plumbum https://plumbum.readthedocs.io/en/latest/ which basically makes Python as terse as bash for common use cases and greatly simplifies the complicated stuff.
> Note that [[ is actually a command/program that returns either 0 (true) or 1 (false).
This is true for the [ command, which is the same as `test`, but the [[ ]] syntax is parsed directly by bash, as is (( )). In particular, [[ ]] is not subject to whitespace splitting, so you don't need to quote most variables.
They mention $(( )), but don't go into (( )), which gives you very nice integer arithmetic and logic:
for (( x = 0; x < 5; x += 2 )); do
echo $x
(( y = x * 10, z = y + 2 ))
echo $y $z
done
Here are two lesser-know tricks that can be quite handy:
Using () instead of {} for the functions body allows to isolate it:
foo=5
bar() (
cd /
foo=7
)
bar
echo $PWD # not / but the current directory
echo $foo # still 5
Bash 4.0 can splat array of arguments back into a valid bash string with @Q. This is super useful when you need to stop using arrays, like with `bash -c`
My newly forming pet peeve are cheatsheets that are long scrolling documents with way too much whitespace.
Cheatsheets are dense, well formatted items that either demonstrated a thing or reminded you of a thing. Ideally they would be printable in either letter or A4, possibly double-sided.
We need a different word for what this is. Tips/hints blog post?
mapfile -t arrayname < filename
-t # remove trailing newline for each element
-u FD # read from file descriptor FD
-s N # skip first N lines
-n N # read at most N lines
-O N # start populating array at index N
array quoting
"${array[@]}" # expand all elements, individually quoted
"${array[*]}" # expand all elements, group quoted
translate number from base to decimal for base in (2,64)
.. looking at >1000 LOC bash project this week, with autotools install to make them go.. it uses 'SOURCE' as an include mechanism.. runs FAST and uses lots of whitespace, but no one on earth but the original author wants to touch it
Start by moving code into bash functions if it's not already doing that.
Declare variables with `local` to reduce the amount of global state.
Once you have code in functions, you can factor code into independent scripts. Anything that runs in a subshell can move into its own script, so anything:
1. in a pipe | line
2. running in the background&
3. in a ( subshell )
4. that doesn't modify global variables
You can then work on those scripts piece by piece.
Shellcheck is awesome! Been using it for about a month and it’s great to understand certain behaviour in the script, even if I don’t end up fixing every single warning.
Which is surprising since "research" here means pasting it into a terminal...
Maybe they meant `cat file.txt | while ...`? Even then, you never want to do that when you're reading because elements of a pipeline are run in subshells. The usual idiom is:
while cond; do
thing
done < input > output
It's also the problem with cheatsheets: the underlying idea in bash is that compound statements are like mini programs, so they have file handles that can be redirected. That doesn't come through a cheatsheet very well.
Looks like this is an artifact of the built-in (old) bash on osx. I see that it works in the newer versions of bash on my freebsd and linux hosts. The mac bash man page says:
Only backslash (\) and single quotes can quote the history expansion character.
where the newer man page says:
Only backslash (\) and single quotes can quote the history expansion character,
but the history expansion character is also treated as quoted if it
immediately precedes the closing double quote in a double-quoted string.
Ruby and Python, meanwhile, are not quite so universally installed as Bash, and they have had all sorts of major breaking changes since 2004. And even if they are installed and version-compatible, they still might not work. Countless times have I seen a misconfiguration with Python 2 vs 3 or RVM or the PATH variable or whatever else cause errors, but I have never run into a scenario where a Bash script didn't just work.
All that said, I'm totally with you that Bash scripts should be short and sweet. If you're doing something with over 100 lines, Bash's downsides overtake its benefits.
Edit: I read your post quickly, you addressed the old bash scenario, but still in my opinion I would always, and do target POSIX shell for maximum cross-platform compatibility and that is definitely available on Unix-like OSs by default.
Python 2/3 was a pain, Ruby had very disparate versions available. But now, a few years later, you can assume at least Ruby 1.9 nearly everywhere, and OTOH still can’t use any of the Bash 4 features. There also quite decent cross-platform packaging solutions for python, ruby, node, go, crystal, mruby that make it really hard to choose bash for any serious project.
Perl/Ruby are too clever for me.
You should rather look for languages like fish, which makes shell scripting feel better than drop it altogether.
This cheatsheet is still a nice to have though.
This is true for the [ command, which is the same as `test`, but the [[ ]] syntax is parsed directly by bash, as is (( )). In particular, [[ ]] is not subject to whitespace splitting, so you don't need to quote most variables.
They mention $(( )), but don't go into (( )), which gives you very nice integer arithmetic and logic:
[ Is a Builtin, But [[ Is Part of the Language
http://www.oilshell.org/blog/2016/10/12.html
I didn't know that before I started implementing a shell :)
I knew that [ could be a builtin or /usr/bin/[, but not about the difference between [ and [[.
Using () instead of {} for the functions body allows to isolate it:
Bash 4.0 can splat array of arguments back into a valid bash string with @Q. This is super useful when you need to stop using arrays, like with `bash -c`Cheatsheets are dense, well formatted items that either demonstrated a thing or reminded you of a thing. Ideally they would be printable in either letter or A4, possibly double-sided.
We need a different word for what this is. Tips/hints blog post?
brace expansions
mapfile to read lines from file to array array quoting translate number from base to decimal for base in (2,64)Start by moving code into bash functions if it's not already doing that.
Declare variables with `local` to reduce the amount of global state.
Once you have code in functions, you can factor code into independent scripts. Anything that runs in a subshell can move into its own script, so anything:
1. in a pipe | line
2. running in the background&
3. in a ( subshell )
4. that doesn't modify global variables
You can then work on those scripts piece by piece.
Using shellcheck will give the developer more confidence in their code, and also learn through the feedback it gives
Maybe they meant `cat file.txt | while ...`? Even then, you never want to do that when you're reading because elements of a pipeline are run in subshells. The usual idiom is:
It's also the problem with cheatsheets: the underlying idea in bash is that compound statements are like mini programs, so they have file handles that can be redirected. That doesn't come through a cheatsheet very well.The thing is I actually find it quicker to simply use my own notes today and/or google anything I need to know when I need to know it.