Pages

Friday, December 19, 2014

Shell script "libraries"

If you are a security profesional (or an IT profesional) probably you -like us- are constantly writing shell scripts, so that you can automate certain tasks in your linux (or unix) environment.

We don't usually use shell scripting to write complex applications (although some shell scripts become quite big), but we do use it extensively to create some "utilities" or little tools to quickly fulfill certain needs that arise along the way.

This happens to us all the time when doing pentesting. Very often, we have to write a shell script very quickly just to solve a particular problem, so we write it as fast as possible, without regard to any software design aspect. When you do this, you know that that is not right way to write programs, but you accept it because you think the extra work that would entail doing it well is not worth it, and you prefer to have a quick working result over a well designed code.

An obvious consequence is that you end up writing the same piece of code again and again. One of the most infamous examples that applies to our case is the argument parsing function: we cannot count the number of times we have written a function to handle script options and arguments and display usage help in a way that is reasonably comfortable for us.

During the latest few months, we have been working on a job that has required us to write (and use) many shell scripts, and this time, since we suspected in advance that that would be the case, we decided to take a -let's say- cleaner approach: we decided to write what we call "shell script libraries", which turned out to be a big help for us with the aforementioned situation.

These "shell script libraries" are sets of shell functions that you can import and use from within your shell scripting code, and some of the functions can be useful even if invoked directly from the shell command line.

In this article we present the following shell libraries:

  • lk_option_parser.sh
  • lk_math.sh
  • lk_net.sh

lk_option_parser.sh

We started out by writing an option parser library. If your shell script needs to be able to behave in different ways depending on its invocation or if you need to pass information to it, you usually achieve this through the use of options and/or arguments. We liked the way this is handled in libraries that you can find in C or python languages, so we tried to write something similar. The library that we have written is intented to be generic and easy to use.

Note: Perhaps there is something similar out there, but none of the code we found and tested happened to match exactly what we were looking for

To use the library you have to download it and put it in a directory that is in your PATH environmental variable (or in the same directory as the invoking shell script).

Then, source it from within your code, for example as follows:

. lk_option_parser.sh || exit 1

Then, call add_program_option as many times as options you have to handle, in this way:

Note: In this context we use option and argument as synonyms; see considerations below

add_program_option "-h" "--help" "Shows this help." "NO" "NO"
where:
-h is the short flag of the option
--help is the long flag of the option
"Shows this help" is the explanation that will appear when the usage is shown
"NO" means that this is not a mandatory option
"NO" means that this option doesn't have an associated value

After you have all your options added, you just call:

parse_program_options $@

And then you may call:

show_program_usage "-h" && exit 0

Which will test if "-h" (or "--help") is present and, in that case, will show program usage and then exit. You can also specify no arguments to show_program_usage in which case no test will be performed.

If latter in your code you want to know if an option is present, you can do it like this:

if is_option_present "-h"
then
 ...
fi

And if you want to get the value for a specific option, you can do it in this way:

_myvar=`get_option_value "-h"`

_myvar will take the value associated to the option. A value is everything between the option and the next short or long option, or the end of the command line. Obviously in this example _myvar will simply be assigned an empty string.

That's _almost_ everything you need to know to use the library! In the code comments you have deeper explanation of the functions, although you probably won't need it.

Let us add just a couple of considerations we think you should be aware of if you are considering using the library:

  • The library is written for bash, because that is the shell interpreter that we use, and we haven't tested it on other interpreters. Perhaps it could be re-written in a more universal way, but we have no plans to move in that direction because, at least for now, bash is enough for us.
  • We know there is much discussion about the right terminology regarding arguments, options and parameters. Please note that, arbitrarily, we decided to use the terms "option", "argument" and "parameter" as synonyms in the context of our shell scripting libraries, and we, also arbitrarily, decided that all options would always include an explicit switch (e.g: "-h", "--help"), some of them with an associated value (e.g: "-i INTERFACE") and some without (e.g: "-h" for help or "-v" for verbose), and finally, we also decided that each option will be either mandatory (its presence will be required) or optional. Please note that therefore, in this context, "option" does not mean "optional" :-)
The lk_option_parser.sh library worked so well for us that we decided to take the same approach to tackle other problems, and so we started two more libraries that are described in the following sections. They are far from being complete, but our idea is to continue expanding them, and any new libraries we may find interesting to create, with ever growing functionality.


lk_math.sh

lk_math.sh is a library that will contain mathematical utilities. At the present moment, it just includes the following functions:

  • get_random_uint
  • get_random_hex_digits
  • hex2dec

The following is an example of use:

jl:~ root # . lk_math.sh
jl:~ root # get_random_uint 0 -1
jl:~ root # get_random_uint 0 10
2
jl:~ root # get_random_uint 0 10
8
jl:~ root # get_random_uint 200 100000
94970
jl:~ root # get_random_uint 200 100000
46624
jl:~ root # get_random_uint 200 1000000
394239
jl:~ root # get_random_uint 200 1000000
525972
jl:~ root #
jl:~ root # get_random_hex_digits
4
jl:~ root # get_random_hex_digits 20
2BAB96D82D9D7BBE0429
jl:~ root # get_random_hex_digits 20
2E7F41F8F6EB098A078E
jl:~ root #
jl:~ root # hex2dec x
0
jl:~ root # hex2dec
jl:~ root # hex2dec FA
250
jl:~ root # hex2dec 10
16


lk_net.sh

lk_net.sh is a library that will contain networking related utilities. At this moment it just includes the following functions:

  • is_mac_address
  • generate_rand_mac

Here are some usage examples:

jl:~ root # . lk_net.sh
jl:~ root #
jl:~ root # is_mac_address "This is not a MAC"; echo $?
1
jl:~ root # is_mac_address "XX:XX:XX:XX:XX:XX"; echo $?
1
jl:~ root # is_mac_address "0A:1B:2C:3D:4E:5X"; echo $?
1
jl:~ root # is_mac_address "0A:1B:2C:3D:4E:5F"; echo $?
0
jl:~ root #
jl:~ root # generate_rand_mac
BE:9B:FD
jl:~ root # generate_rand_mac FULL
60:AD:CA:70:C5:D4


Conclusion and future work

We found these small shell libraries to be really useful for us, and so we thought we would share them. We hope you find it useful. You are free to use them in almost any way you see fit, since we are publishing them under the GPLv3 license.

Obviously, the code can be improved and expanded, and while we will certainly do so, we would also be more than happy to get your comments and contributions, which we would study and eventually include in the code, giving you the appropriate credit, of course.

68 comments:

  1. Hi friends, This is Christy from Chennai. I did Unix certification course in Chennai at Fita academy. This is really useful for me to make a bright career. Suppose if anyone interested to learn Unix Training in Chennai please visit Fita academy located at Chennai.

    ReplyDelete
  2. Thankful for sharing hadoop training in chennai instructive article on Salesforce development.informatica training in chennai Your article helped me a ton to appreciate the calling oracle training in chennai prospects in appropriated figuring development.

    ReplyDelete
  3. Hi friends, This is Jamuna from Chennai. Your technical information is really useful for me. Keep update your blog.
    Regards..
    Oracle Training Institutes in Chennai

    ReplyDelete
  4. Hi, I have read your blog and I got a useful information from this blog. Thanks for sharing, keep posting..Salesforce is a cloud based CRM software. Today's most of the IT industry use this software for customer relationship management.
    Regards
    Salesforce Course in Chennai



    ReplyDelete
  5. SAP is one of the customer relationship management software and it support all end to end customer related process. To know more details about SAP modules please refer the following site.
    Regards..
    SAP Training Chennai

    ReplyDelete
  6. Your blog is really awesome and I got some useful information from your blog. This is really useful for me. Thanks for sharing such a informative blog. Keep posting.

    Regards..
    Cloud Training in Chennai



    ReplyDelete
  7. Java is one of the popular technologies with improved job opportunity for hopeful professionals. Java Training in Chennai helps you to study this technology in details.If you are looking for best Java Training Institutes in Chennai reach Fita academy.

    ReplyDelete
  8. Really awesome blog. Software testing is a method of executing the application or program with the intent of searching the software errors. Testing Training in Chennai offering this course at reasonable cost.



    ReplyDelete
  9. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this. FITA chennai reviews

    ReplyDelete
  10. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
    Regards,
    Informatica training in chennai|Best Informatica Training In Chennai|Informatica training center in Chennai

    ReplyDelete
  11. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
    Regards,
    Regards

    Informatica Training in Chennai | iOS Training in Chennai | QTP Training in Chennai | SAP Training in Chennai | CCNA Training in Chennai

    ReplyDelete
  12. Thanks for sharing this niche useful informative post to our knowledge, Actually SAP is ERP software that can be used in many companies for their day to day business activities it has great scope in future.
    Regards,
    SAP Training in Chennai|SAP Course in Chennai|sap training in Chennai

    ReplyDelete
  13. Awesome Post! I like writing style, how you describing the topics throughout the post. I hope many web reader will keep reading your post at the end, Thanks for sharing your view.
    Regards,
    Angularjs training in chennai|Angularjs training chennai|Angularjs course in chennai|Angularjs training center in Chennai

    ReplyDelete
  14. Thank you so much for sharing this unique and useful information with us. I gained more knowledge by reading your post. keep on blogging.!!
    Loadrunner Training in Chennai

    ReplyDelete
  15. It is very interesting to read this information.Thanks for this valuable topic.
    Java Training in Chennai

    ReplyDelete
  16. All of these tips are great,that’s very interesting. I’m so tempted to try that myself, it's very helpful for my carrier.
    iOS Training in Chennai

    ReplyDelete
  17. Thanks for your great article post.You shared more informative and interesting article.We are providing best Salesforce training course in our institute at reasonble price with placement support.The people who are interested to learn salesforce,can join in our institute.For more details reach us @Salesforce developer training in Chennai

    ReplyDelete
  18. very great explanation of the shell script libraries. all that are explained clearly with coded examples. that are provided and more informative blog.
    Informatica Training in Chennai

    ReplyDelete
  19. Thanks for providing this information. this information is very useful. This program helps a lot for the beginnersManual Testing Training in Chennai

    ReplyDelete
  20. You have explained clearly about shell script libraries.I have learned new thing today,It's easy to understand about it.Thank you for sharing this post.
    Android Training in Chennai

    ReplyDelete
  21. thank you for this blog. now only learn about shell script and their libraries.VMWare Training in Chennai

    ReplyDelete
  22. ur way of explaantion is good in this blog.thanks for sharing this valuable information with us and thanks for spend ur time with us.

    CCNA Training in Chennai

    ReplyDelete

  23. i got the information about the blog articles which is useful and informative thanks for sharing, it is nice .



    sharepoint-developer-training in chennai

    ReplyDelete
  24. Thanks for sharing your blog.Have learned new about shell script.Its very easy and understanding.
    Linux training in chennai

    ReplyDelete
  25. Your blog is informative. It clearly explains the concept of shell script. very easy and simple in understanding ssis training in chennai

    ReplyDelete
  26. This shell script libraries are very nice and is easy to understand that.Thanks for your information.
    Sharepoint admin training in chennai

    ReplyDelete
  27. well defined expalanation about linux commands of its shell scripts. helpful to our candidates to understand about the shell scriopts information. keep updating more about shell scripts. thanks.
    veritas volume manager training in chennai

    ReplyDelete
  28. those are not needed for all kind of peoples. peoples they can work deeply they only want to understand this.CCNA Training in Chennai

    ReplyDelete
  29. well defined explanation about shell scripts. and also explained step by step procedure it is very useful to all basics students.
    Datawarehousing Training in Chennai

    ReplyDelete
  30. Your post is awesome and keep on me wondering how you did this post , thanks for sharing.


    websphere training in chennai

    ReplyDelete
  31. very informative blog. It explained about the shell scripts in a step by step process. thanks for posting this information ios training in chennai

    ReplyDelete
  32. Have gained new knowledge about shell script.Your explanation is very easy to understand..Thanks for sharing.
    weblogic training in chennai

    ReplyDelete

  33. too good piece of information, I had come to know about your site from my friend sajid, bangalore,i have read atleast 7 posts of yours by now, and let me tell you, your web-page gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new post, once again hats off to you! Thanks a lot once again, Regards, informatica mdm training in hyderabad,informatica training in hyderabad

    ReplyDelete
  34. I got the complete picture on Shell scripting. I am planning to teach Shellscripting Training in Chennai for students does my syllabus covers everything my IICT Chromepet Covers all.

    ReplyDelete


  35. These provided information was really so nice,thanks for giving that post and the more skills to develop after refer that post. Your articles really impressed for me,because of all information so nice.

    Peridot Systems Adyar Chennai Reviews

    ReplyDelete
  36. Thanks for the good words! Really appreciated. Great post. I’ve been commenting a lot on a few blogs recently, but I hadn’t thought about my approach until you brought it up.

    SEO training in Adyar

    ReplyDelete
  37. This wordpress module is made for shippers who need to offer items from their 3dcart Online Store inside their WordPress blog .
    wordpress shop

    ReplyDelete
  38. Grateful to check out your website, I seem to be ahead to more excellent content and I believe we all really like to thank for so many excellent content, weblog to discuss with us seo packages prices

    ReplyDelete
  39. nice about library..
    Best SEO training in hyderabad all modules are clearly solved and practice with guide.seo training in hyderabadAnd clearly getting knowledge of seo.

    ReplyDelete
  40. I am not sure the place you are getting your information, however good topic. I needs to spend some time studying more or understanding more. Thank you for wonderful information I was in search of this info for my mission.

    Digital marketing company in Chennai

    ReplyDelete
  41. nice posts...
    SAS Institute introduced the SAS Certified Professional Program,training proper understanding of how the SAS software works. Among the five certification programs that SAS Institute has come up with, SAS training can be considered as the entry point into the big data and the data analytics industry.
    SAS online training in hyderabad

    ReplyDelete
  42. I feel thanks to you for posting such a good blog, keep updates regularly
    sharing with us that awesome article you have amazing blog.....salesforce training in hyderabad

    ReplyDelete
  43. wow great,nowadays this type of blog is more important and informative technology,it was more impressive to read ,which helps to design more in effective ways





    DotNet Training in Chennai Adyar

    ReplyDelete
  44. It's like you read my mind! You seem to know a lot about this, like you wrote the book in it or something. I think that you can do with some pics to drive the message home a little bit, but instead of that, this is fantastic blog. A great read. I will definitely be back.

    Corporate Training in Chennai

    ReplyDelete
  45. Thank you for having taken your time to provide us with your valuable information relating to your stay with us.we are sincerely concerned.., Most importantly, you Keepit the major...
    seo company in chennai

    ReplyDelete
  46. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    Digital Marketing Company in India


    ReplyDelete
  47. Very Nice Blog I like the way you explained these things. I’ve been looking for ways to improve my website and overall rankings.I hope your future article will help me further.Take SEO Training in Chennai to mould yourself.

    ReplyDelete
  48. very very amazing explaintion....many things gather about yourself...yes realy i enjoy it
    Digital Marketing company in Chennai

    ReplyDelete
  49. Excellent post. You have post an very useful and informative information. Keep sharing. thank you..
    Software Testing Training in Chennai | Big data Analytics Training in Chennai

    ReplyDelete

  50. Certification test center in Marathahalli
    Signetsoft is the authorised Pearson vue test center for all global certifications in bangalore
    Certification test center in Bangalore
    Pearson vue test center in Bangalore
    Pearson vue in Marathahalli . Pearson Vue Exam Center in Bangalore |
    Pearson Vue Exam Centers in Bangalore |

    ReplyDelete
  51. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
    Regards,
    Ezhilarasu

    Java Training Institute in Chennai

    ReplyDelete