Automation & Shell

Scripting
With Linux
Agenda
UNIX/LINUX and Shell
Introduction UNIX Commands and Utilities
Basic Shell Scripting Structure

Variable
Shell Programming Operators
Logic Structures

Examples of Application in Research Computing

Hands-on Exercises
Why Shell Scripting ?

Shell scripts can be used

to prepare input files, job Useful to create own Save lots of time on file
monitoring, and output commands. processing.
processing.

System Administration
To automate some task of
part can be also
day-to-day life.
automated.
What is a “Shell”?
• The “Shell” is simply another program on top of the kernel which
provides a basic human-OS interface.
• It is a command interpreter
• Built on top of the kernel
• Enables users to run services provided by the UNIX OS
• In its simplest form, a series of commands in a file is a shell program that
saves having to retype commands to perform common tasks.
use
Shel r
l

use OS
r

use
r
UNIX Shells

• sh Bourne Shell (Original Shell) (Steven Bourne of AT&T)

• bash Bourne Again Shell (GNU Improved Bourne Shell)
• csh C-Shell (C-like Syntax)(Bill Joy of Univ. of California)
• ksh Korn-Shell (Bourne+some C-shell)(David Korn of AT&T)
• tcsh Turbo C-Shell (More User Friendly C-Shell).
• To check shell:
• $ echo $SHELL (shell is a pre-defined variable)
• To switch shell:
• $ exec shellname (e.g., $ exec bash or simply type $ bash)
• You can switch from one shell to another by just typing the name of the shell. exit return you back
to previous shell.
Which Shell to Use?

• sh ( Bourne shell) was considered better for programming

• csh (C-Shell ) was considered better for interactive work.
• tcsh and korn were improvements on c-shell and bourne shell respectively.
• bash is largely compatible with sh and also has many of the nice features of the other shells
• On many systems such as our LINUX clusters sh is symbolically linked to bash, /bin/sh -> /bin/bash
• We recommend that you use sh/bash for writing new shell scripts but learn csh/tcsh to understand existing scripts.
• Many, if not all, scientific applications require csh/tcsh environment (GUI, Graphics Utility Interface)
• All Linux versions use the Bash shell (Bourne Again Shell) as the default shell
• Bash/Bourn/ksh/sh prompt: $
• All UNIX system include C shell and its predecessor Bourne shell.
• Csh/tcsh prompt: %
What is Shell Script?

• A shell script is a script written for the shell

• Two key ingredients
• UNIX/LINUX commands
• Shell programming syntax
UNIX/LINUX Commands

• File Management and Viewing

• Filesystem Mangement
• Help,Job/Process Management
• Network Management
• System Management
• User Management
• Printing and Programming
• Document Preparation
• Miscellaneous
 File and Directory Management

• cd Change the current directory. With no arguments "cd" changes to the users home directory. (cd <directory path>)
• chmod Change the file permissions.
• Ex: chmod 751 myfile : change the file permissions to rwx for owner, rx for group and x for others (x=1,r=4,w=2)
• Ex: chmod go=+r myfile : Add read permission for the group and others (character meanings u-user, g-group, o-other, + add permission,-remove,r-
read,w-write,x-exe)

Ex: chmod +s myfile - Setuid bit on the file which allows the program to run with user or group privileges of the file.
• chown Change owner.
• Ex: chown <owner1> <filename> : Change ownership of a file to owner1.
• chgrp Change group.
• Ex: chgrp <group1> <filename> : Change group of a file to group1.
• cp Copy a file from one location to another.
• Ex: cp file1 file2 : Copy file1 to file2; Ex: cp –R dir1 dir2 : Copy dir1 to dir2
File and Directory Management

• ls List contents of a directory.

• Ex: ls, ls –l , ls –al, ls –ld, ls –R
• mkdir Make a directory.
• Ex: mkdir <directory name> : Makes a directory
• Ex mkdir –p /www/chache/var/log will create all the directories starting from www.
• mv Move or rename a file or directory.
• Ex: mv <source> <destination>
• find Find files (find <start directory> -name <file name> -print)
• Ex: find /home –name readme -print
• Search for readme starting at home and output full path, “/home" = Search starting at the home directory and proceed through all its
subdirectories; "-name readme" = Search for a file named readme "-print" = Output the full path to that file
• locate File locating program that uses the slocate database.
• Ex: locate –u to create the database,
• locate <file/directory> to find file/directory
File and Directory Management

• pwd Print or list the present working directory with full path.
• rm Delete files (Remove files). (rm –rf <directory/file>)
• rmdir Remove a directory. The directory must be empty. (rmdir <directory>)
• touch . Make the file if it doesn't exist. (touch <filename>)
• whereis Locate the binary and man page files for a command. (whereis
<program/command>)
• which Show full path of commands where given commands reside. (which <command>)
File viewing and editing

• emacs Full screen editor.

• pico Simple text editor.
• vi Editor with a command mode and text mode. Starts in command mode.
• gedit GUI Text Editor
• tail Look at the last 10 lines of a file.
• Ex: tail –f <filename> ; Ex: tail -100 <filename>
• head Look at the first 10 lines of a file. (head <filename>)
File compression, backing up and
restoring

• compress Compress data.

• uncompress Expand data.
• cpio Can store files on tapes. to/from archives.
• gzip - zip a file to a gz file.
• gunzip - unzip a gz file.
• tar Archives files and directories. Can store files and directories on tapes.
• Ex: tar -zcvf <destination> <files/directories> - Archive copy groups of files. tar –zxvf <compressed file> to uncompress
• zip – Compresses a file to a .zip file.
• unzip – Uncompresses a file with .zip extension.
• cat View a file
• Ex: cat filename
• cmp Compare two files.
File and Directory Management

• diff Show the differences between files.

• Ex: diff file1 file2 : Find differences between file1 & file2.
• echo Display a line of text.
• grep List all files with the specified expression.
(grep pattern <filename/directorypath>)

Ex: ls –l |grep sidbi : List all lines with a sidbi in them.

• Ex: grep " R " : Search for R with a space on each side
• sleep Delay for a specified amount of time.
• sort Sort a file alphabetically.
• uniq Remove duplicate lines from a sorted file.
• wc Count lines, words, characters in a file. (wc –c/w/l <filename>).
• sed stream editor, extremely powerful!
• awk an extremely versatile programming language for working on files
Useful Commands in Scripting

• grep
• Pattern searching
• Example: grep ‘boo’ filename
• sed
• Text editing
• Example: sed 's/XYZ/xyz/g' filename
• awk
• Pattern scanning and processing
• Example: awk ‘{print $4, $7}’ filename
Shell Scripting

• Start vi scriptfilename.sh with the line

• #!/bin/bash
• All other lines starting with # are comments.
• make code readable by including comments
• Tell Unix that the script file is executable
• $ chmod u+x scriptfilename.sh
• $ chmod +x scriptfilename.sh
• Execute the shell-script
• $ ./scriptfilename.sh
My First Shell Script

$ vi myfirstscript.sh
#! /bin/sh
# The first example of a shell script
directory=‘pwd’
echo Hello World!
echo “The date today is `date`”
echo “The current directory is $directory”

$ chmod +x myfirstscript.sh
$ ./myfirstscript.sh
Hello World!
The date today is Mon Mar 8 15:20:09 EST 2010
The current directory is /netscr/shubin/test
Commenting

• Lines starting with # are comments except the very first line where #! indicates the location
of the shell that will be run to execute the script.
• On any line characters following an unquoted # are considered to be comments and
ignored.
• Comments are used to;
• Identify who wrote it and when
• Identify input variables
• Make code easy to read
• Explain complex code sections
• Version control tracking
• Record modifications
Quote Characters

• There are three different quote characters with different behaviour. These are:
• : “double quote” weak quote. If a string is enclosed in “ ” the references to variables
(i.e $variable ) are replaced by their values. Also back-quote and escape \ characters are
treated specially.
• ‘single quote’, strong quote. Everything inside single quotes are taken literally;
nothing is treated as special.
• `back quote` A string enclosed as such is treated as a command and the shell
attempts to execute it. If the execution is successful the primary output from the command
replaces the string.
• Example: echo “Today is:” `date`
• Echo ‘ls’
Echo

• Echo command is well appreciated when trying to debug scripts.

• Syntax : echo {options} string
User Input During Shell Script Execution

• As shown on the hello script input from the standard input location is done
via the read command.
• Example
echo "Please enter three filenames:”
read filea fileb filec
echo “These files are used:$filea $fileb $filec”
• Each read statement reads an entire line. In the above example if there are
less than 3 items in the response the trailing variables will be set to blank ‘ ‘.
• Three items are separated by one space.
Hello script exercise continued…

• The following script asks the user to enter his name and displays a
personalised hello.
• #!/bin/sh
• echo “Who am I talking to?”
• read user_name
• echo “Hello $user_name”
• Try replacing “ with ‘ in the last line to see what happens.
Shell Programming

• Shell variables:
• Your scripts often need to keep values in memory for later use. Shell variables are symbolic
names that can access values stored in memory
• Operators:
• Shell scripts support many operators, including those for performing mathematical
operations
• Logic structures:
• Shell scripts support
• sequential logic (for performing a series of commands),
• decision logic (for branching from one point in a script to another),
• looping logic (for repeating a command several times), and
• case logic (for choosing an action from several possible alternatives)
Variables

• Variables are symbolic names that represent values stored in memory

• Three different types of variables
• Global Variables:
• Environment and configuration variables, capitalized, such as HOME, PATH, SHELL,
USERNAME, and PWD.
• When you login, there will be many global System variables that are already defined. These can
be freely referenced and used in your shell scripts.
• Local Variables
• Within a shell script, you can create as many new variables as needed. Any variable created in
this manner remains in existence only within that shell.
• E,g directory=`pwd`
• Special Variables
• Reversed for OS, shell programming, etc. such as positional parameters $0, $1 …
A few global (environment)
variables

SHELL Current shell

DISPLAY Used by X-Windows system to identify the display

HOME Fully qualified name of your login directory

PATH Search path for commands

MANPATH Search path for <man> pages

PS1 & PS2 Primary and Secondary prompt strings

USER Your login name

TERM terminal type

PWD Current working directory

Referencing Variables

• Variable contents are accessed using ‘$’:

• e.g.
echo $HOME
echo $SHELL
• To see a list of your environment variables:
$ printenv
• or:
$ env
 Defining Local Variables

• As in any other programming language, variables can be defined and used in shell scripts.
• Unlike other programming languages, variables in Shell Scripts are not typed.
• Examples :
a=1234 # a is NOT an integer, a string instead
b=$a+1 # will not perform arithmetic but be the string b= “1234+1”
b=`expr $a + 1 ` will perform arithmetic so b is 1235 now.
Note : +,-,/,*,**, % operators are available.
b=abcde # b is string
b=‘abcde’ # same as above but much safer.
b=abc def # will not work unless ‘quoted’
b=‘abc def’# this will work.
IMPORTANT NOTE: DO NOT LEAVE SPACES AROUND THE =
Referencing variables --curly bracket

• Having defined a variable, its contents can be referenced by the $ symbol. E.g., $
{variable} or simply $variable.
• When ambiguity exists $variable will not work. Use ${ } the rigorous form to be
on the safe side.
• Example:
a=“abc”
b=${a}def # abcdef
• this would not have worked without the{ } as
• #it would try to access a variable named adef
Positional Parameters

• When a shell script is invoked with a set of command line parameters each of these parameters are copied into special variables that can be
accessed.
• $0 This variable that contains the name of the script
• $1, $2, ….. $n 1st, 2nd 3rd command line parameter
• $# Number of command line parameters
• $$ process ID of the shell
• $@ same as $* but as a list one at a time (see for loops later )
• $? Return code ‘exit code’ of the last command

• Example:
• Invoke : ./myscript one two buckle my shoe
• During the execution of myscript variables $1 $2 $3 $4 and $5 will contain the values one, two, buckle, my, shoe
respectively.
Variables
• vi myinputs.sh
#! /bin/csh
echo Total number of inputs: $#
echo First input: $1
echo Second input: $2

chmod u+x myinputs.sh

myinputs.sh Introduction to Internet Programming CSE

Total number of inputs: 5
First input: Introduction
Second input: to
Shell Programming

• programming features of the UNIX shell:

• Shell variables
• Operators
• Logic structures
• The Bash/Bourne/ksh shell operators are divided into three groups: defining and
evaluating operators, arithmetic operators, and redirecting and piping operators
Defining and Evaluating

• A shell variable take on the generalized form variable=value (except in the C shell).
$ set x=37; echo $x
37
$ unset x; echo $x
x: Undefined variable.
• You can set a pathname or a command to a variable or substitute to set the variable.
$ mydir=`pwd`; echo ${mydir}
Pipes & Redirecting

• Piping: An important early development in Unix , a way to pass the output of one tool to
the input of another.
$ ls -l | grep “IS” | command2 | command3
• By combining these two tools, giving the grep command the output of ls –l
• $ sudo apt-cache pkgnames | less
• In the above command, the standard output of the command left to the pipe “|”
sign is the standard input of the command right to the pipe “|” sign.
Pipes & Redirecting
• Redirecting via angle brackets.
• Redirecting input and output follows a similar principle to that of piping except that redirects work with files, not
commands.
• Standard output redirection “>”
• $ echo Hello World > file.txt
• $ ls –l > FileNames
• Standard Input redirection “<”
• less < /etc/passwd
• input from the file /etc/passwd instead of a keyboard
• Standard error redirection “2>”
• the standard errors can be redirected and written to a file
• sudo apt-get update 2> error
• If any error occurs, it will not show on the terminal window; rather, it will be stored in an error file. If the error file already exists, then it will be
overwritten.
• Standard output and error redirection “&>”
• a more efficient way to redirect standard output and standard error simultaneously
• Standard output redirection “>>”
• redirection method redirects the standard output of a command or a file to another file
• if the file already exists, the data will be appended to the file
• Standard input redirection “<<”
• This redirection method reads the user input from the terminal and then appends it to the file
 Arithmetic Operators
• expr supports the following vi math.sh
#!/bin/sh
operators:
count=5
• arithmetic operators: +,-,*,/,%
count=`expr $count + 1`
• comparison operators: <, <=, ==, !=,
>=, > count= $[$count + 1]
• boolean/logical operators: & | echo $count
• parentheses: (, )
• precedence is the same as C, Java
./math.sh
6
Arithmetic operations in shell scripts

var++ ,var-- , ++var , --var post/pre increment/decrement

** power of for example 2**3

!,~ logical/bitwise negation

&,| bitwise AND, OR

&& || logical AND, OR

Shell Logic Structures

• The four basic logic structures needed for program development are:
• Sequential logic: to execute commands in the order in which they appear in the
program
• Decision logic: to execute commands only if a certain condition is satisfied
• Looping logic: to repeat a series of commands for a given number of times
• Case logic: to replace “if then/else if/else” statements when making numerous
comparisons
Conditional Statements (if constructs )

The most general form of the if construct is; #!/bin/bash

echo "Enter password"
if command executes successfully read pass
if [ $pass=="password" ]
then then
execute command echo "The password is correct."
elif this command executes successfully else
echo "The password is incorrect, try
again."
then fi
execute this command

and execute this command

else
execute default command
fi
 String and numeric comparisons used with test or [[ ]] which is an alias for test and also [ ] which is
Tests another acceptable syntax

#! /bin/sh
 int1 –eq int2 Test # number is positive, zero or negative
identity
 int1 –ne int2 Test echo –e "enter a number:\c"
inequality read number
 int1 –lt int2 Less than if [ “$number” -lt 0 ]
 int1 –gt int2 Greater
than then
 int1 –int2 Less than echo "negative"
or equal elif [ “$number” -eq 0 ]
 int1 –lege int2 Greater
#!/bin/sh then
than or equal
if [ “$#” -ne 2 ] then echo zero
echo $0 needs two parameters!
else
echo You are inputting $#
fi echo positive
fi

Additional if – examples
https://tldp.org/LDP/Bash-Beginners-Guide/html/chap_07.html
File enquiry operations

-d file Test if file is a directory

-f file Test if file is not a directory
-s file Test if the file has non zero length
-r file Test if the file is readable
-w file Test if the file is writable
-x file Test if the file is executable
-o file Test if the file is owned by the user
-e file Test if the file exists
-z file Test if the file has zero length

All these conditions return true if satisfied and false otherwise.

Safely Modifying Files

#!/bin/bash
a=12234
B=${xyz}a = 12234a
CFGDIR=/etc/openldap
CFGFILE=ldap.conf
if [ ! -d $CFGDIR ]; then
`mkdir -p $CFGDIR`
fi
if [ -f $CFGFILE ]; then
`mv $CFGFILE $CFGFILE.orig`
fi
# edit config file once directory created and original saved
Combining tests with logical operators || (or) and && (and)

• Syntax: if cond1m && cond2 || cond3 …

An alternative form is to use a compound statement using the –a and –o keywords, i.e.
if cond1 –a cond22 –o cond3 …
• Where cond1,2,3 .. Are either commands returning a value or test conditions of the form
[ ] or test …
• Examples:
if [ “$a” –lt 0 –o “$a” –gt 100 ]
then
echo “ limits exceeded”
fi
Loops

• Loop is a block of code that is repeated a number of times.

• The repeating is performed either a pre-determined number of times determined by a list
of items in the loop count ( for loops ) or until a particular condition is satisfied ( while
and until loops)
• To provide flexibility to the loop constructs there are also two statements namely break
and continue are provided.
for loops
Syntax:
for arg in list
do
#!/bin/bash
command(s) # remove all files with extension html
... LIST="$(ls *.html)"
for i in "$LIST"; do
done `rm $i’
Where the value of the variable arg is set to the done
values provided in the list one at a time and the
block of statements executed. This is repeated
until the list is exhausted.
 while loops #!/bin/bash
Syntax:
while this_command_execute_successfully # This script copies files from my homedirectory into
the webserver directory.
do # (use scp and SSH keys for a remote directory)
this command # A new directory is created every hour.
and this command
PICSDIR=/home/carol/pics
done
WEBDIR=/var/www/carol/webcam

while true; do
DATE=`date +%Y%m%d`
HOUR=`date +%H`
#!/bin/bash mkdir $WEBDIR/"$DATE"
# This script opens 4 terminal
while [ $HOUR -ne "00" ]; do
windows. DESTDIR=$WEBDIR/"$DATE"/"$HOUR"
i="0" mkdir "$DESTDIR"
while [ $i -lt 4 ] mv $PICDIR/*.jpg "$DESTDIR"/
do sleep 3600
HOUR=`date +%H`
xterm & done
i=$[$i+1] done
done

https://linuxhint.com/date-command-bash/
Switch/Case Logic

• The switch logic structure simplifies the selection of a match when you have
a list of choices
• It allows your program to perform one of many actions, depending upon the
value of a variable
Case statements

The case structure compares a string ‘usually contained in a variable’ to one or more patterns
and executes a block of code associated with the matching pattern. Matching-tests start
with the first pattern and the subsequent patterns are tested only if no match is not
found so far.
case argument in
pattern 1) execute this command
and this
and this;;
pattern 2) execute this command
and this
and this;;
esac
 Functions

• Functions are a way of grouping together commands so that they can later be executed via a single reference to their
name.
#!/bin/sh
myfunc()
{ The script, when called as
echo "I was called as : $@“
echo $1 function.sh a b c,
echo $2
echo $3 gives the following output:
x=2
}
Script was called with a b c
myfunc John Jane Mary x is 1
echo “this is the end of the scrip” I was called as : 1 2 3
x is 2
Exercises

1. Script to find out the biggest number in 3 numbers

2. to reverse a given number – e.g., input 1234 and output 4321
3. Sort the given five numbers in ascending order (using array)
4. Calculating average of given numbers on command line arguments
5. Write a script to see current date, username and current directory
6. Write a script to determine if a given file exists on the computer