Friday, December 19, 2014

Shell script "libraries"

If you are a security profesional (or an IT profesional) probably you -like us- are constantly writing shell scripts, so that you can automate certain tasks in your linux (or unix) environment.

We don't usually use shell scripting to write complex applications (although some shell scripts become quite big), but we do use it extensively to create some "utilities" or little tools to quickly fulfill certain needs that arise along the way.

This happens to us all the time when doing pentesting. Very often, we have to write a shell script very quickly just to solve a particular problem, so we write it as fast as possible, without regard to any software design aspect. When you do this, you know that that is not right way to write programs, but you accept it because you think the extra work that would entail doing it well is not worth it, and you prefer to have a quick working result over a well designed code.

An obvious consequence is that you end up writing the same piece of code again and again. One of the most infamous examples that applies to our case is the argument parsing function: we cannot count the number of times we have written a function to handle script options and arguments and display usage help in a way that is reasonably comfortable for us.

During the latest few months, we have been working on a job that has required us to write (and use) many shell scripts, and this time, since we suspected in advance that that would be the case, we decided to take a -let's say- cleaner approach: we decided to write what we call "shell script libraries", which turned out to be a big help for us with the aforementioned situation.

These "shell script libraries" are sets of shell functions that you can import and use from within your shell scripting code, and some of the functions can be useful even if invoked directly from the shell command line.

In this article we present the following shell libraries:


We started out by writing an option parser library. If your shell script needs to be able to behave in different ways depending on its invocation or if you need to pass information to it, you usually achieve this through the use of options and/or arguments. We liked the way this is handled in libraries that you can find in C or python languages, so we tried to write something similar. The library that we have written is intented to be generic and easy to use.

Note: Perhaps there is something similar out there, but none of the code we found and tested happened to match exactly what we were looking for

To use the library you have to download it and put it in a directory that is in your PATH environmental variable (or in the same directory as the invoking shell script).

Then, source it from within your code, for example as follows:

. || exit 1

Then, call add_program_option as many times as options you have to handle, in this way:

Note: In this context we use option and argument as synonyms; see considerations below

add_program_option "-h" "--help" "Shows this help." "NO" "NO"
-h is the short flag of the option
--help is the long flag of the option
"Shows this help" is the explanation that will appear when the usage is shown
"NO" means that this is not a mandatory option
"NO" means that this option doesn't have an associated value

After you have all your options added, you just call:

parse_program_options $@

And then you may call:

show_program_usage "-h" && exit 0

Which will test if "-h" (or "--help") is present and, in that case, will show program usage and then exit. You can also specify no arguments to show_program_usage in which case no test will be performed.

If latter in your code you want to know if an option is present, you can do it like this:

if is_option_present "-h"

And if you want to get the value for a specific option, you can do it in this way:

_myvar=`get_option_value "-h"`

_myvar will take the value associated to the option. A value is everything between the option and the next short or long option, or the end of the command line. Obviously in this example _myvar will simply be assigned an empty string.

That's _almost_ everything you need to know to use the library! In the code comments you have deeper explanation of the functions, although you probably won't need it.

Let us add just a couple of considerations we think you should be aware of if you are considering using the library:

  • The library is written for bash, because that is the shell interpreter that we use, and we haven't tested it on other interpreters. Perhaps it could be re-written in a more universal way, but we have no plans to move in that direction because, at least for now, bash is enough for us.
  • We know there is much discussion about the right terminology regarding arguments, options and parameters. Please note that, arbitrarily, we decided to use the terms "option", "argument" and "parameter" as synonyms in the context of our shell scripting libraries, and we, also arbitrarily, decided that all options would always include an explicit switch (e.g: "-h", "--help"), some of them with an associated value (e.g: "-i INTERFACE") and some without (e.g: "-h" for help or "-v" for verbose), and finally, we also decided that each option will be either mandatory (its presence will be required) or optional. Please note that therefore, in this context, "option" does not mean "optional" :-)
The library worked so well for us that we decided to take the same approach to tackle other problems, and so we started two more libraries that are described in the following sections. They are far from being complete, but our idea is to continue expanding them, and any new libraries we may find interesting to create, with ever growing functionality. is a library that will contain mathematical utilities. At the present moment, it just includes the following functions:

  • get_random_uint
  • get_random_hex_digits
  • hex2dec

The following is an example of use:

jl:~ root # .
jl:~ root # get_random_uint 0 -1
jl:~ root # get_random_uint 0 10
jl:~ root # get_random_uint 0 10
jl:~ root # get_random_uint 200 100000
jl:~ root # get_random_uint 200 100000
jl:~ root # get_random_uint 200 1000000
jl:~ root # get_random_uint 200 1000000
jl:~ root #
jl:~ root # get_random_hex_digits
jl:~ root # get_random_hex_digits 20
jl:~ root # get_random_hex_digits 20
jl:~ root #
jl:~ root # hex2dec x
jl:~ root # hex2dec
jl:~ root # hex2dec FA
jl:~ root # hex2dec 10
16 is a library that will contain networking related utilities. At this moment it just includes the following functions:

  • is_mac_address
  • generate_rand_mac

Here are some usage examples:

jl:~ root # .
jl:~ root #
jl:~ root # is_mac_address "This is not a MAC"; echo $?
jl:~ root # is_mac_address "XX:XX:XX:XX:XX:XX"; echo $?
jl:~ root # is_mac_address "0A:1B:2C:3D:4E:5X"; echo $?
jl:~ root # is_mac_address "0A:1B:2C:3D:4E:5F"; echo $?
jl:~ root #
jl:~ root # generate_rand_mac
jl:~ root # generate_rand_mac FULL

Conclusion and future work

We found these small shell libraries to be really useful for us, and so we thought we would share them. We hope you find it useful. You are free to use them in almost any way you see fit, since we are publishing them under the GPLv3 license.

Obviously, the code can be improved and expanded, and while we will certainly do so, we would also be more than happy to get your comments and contributions, which we would study and eventually include in the code, giving you the appropriate credit, of course.