Unix and

Scripting 101

Rosalind Archer
April 17th, 2002
 Disclaimer
• I am not a scripting “guru”. I find ways to
get scripts to work but I can guarantee
they are not the most efficient ways
possible!
 Web Directory
• All files related to this presentation are
available at:

http://pumpjack.tamu.edu/~archer/UNIX/
 Example Files
• All the files presented in the scripting
examples at the end of the presentation
are contained in a “zipped” file called
scripts.tar.

• Download this and place it on your Unix

account (blackgold, Z:\).

• To unzip the files type tar -xvf scripts.tar

at the command line.
 Introduction
• Unix scripting tools can be very powerful
tools for:
– running jobs consecutively
– processing output
– and a host of other things
• This presentation will introduce the use of
shell scripts, sed, and awk. No
knowledge of Unix will be assumed …
just a willingness to try things out.
 The Unix Environment
• Working in Unix often lacks the graphical
user interfaces that most of us have
become used to.
• Commands are frequently typed at the
command line.
– This may be perceived as a weakness of
working in Unix but I see actually as one of
the chief advantages!
 Case Sensitivity
• Unix is case sensitive so “file.dat”” is
different to “File.dat” and “file.DAT”

• This takes some getting used to if you

are a Windows user!
 Unix Computers
• The department has several Unix
machines running the Solaris operating
system.

• Wildcat is the has the most memory

and CPU capability. It’s use should be
reserved for large jobs. Wildcat does not
have a monitor/keyboard that you can sit
in front of.
•w
 Unix Computers
• Sparc309-1 is currently on loan to me
and resides in my office (I need its
graphics capabilities for a Department of
Energy project).

• Sparc309-2 resides in the 3rd floor

Unix lab.

• Iril-5 also resides in the 3rd floor Unix

lab.
 Unix Computers
• If you are doing text based work your can
connect to the Unix computers using a
“telnet” session.
• All PC’s on the network usually have
telnet capability. You can use a telnet
program or just go the Start menu in
windows and choose Run. In the box
that opens type telnet wildcat.
Telnet
 Xwin32
• If you need the graphical capabilities of a
Unix machine most packages can be run
remotely by connecting to the Unix
machine using a program called xwin32.
It should be available on all lab PCs.
 Editing Files - Windows
• You edit Unix files from Windows. Make
sure you save them as “text only”.
Before you use the files in Unix you may
have to remove control-M characters
from them that denotes carriage returns
in Windows.

• One way to do this is by typing the

command dos2unix filename
filename.
 Editing Files - vi
• My favourite Unix editor is called “vi”. It
is not especially user friendly (but in
return is very powerful).

• If you know vi and want to use it to

remove control-M’s use :%s/^V^M//g
(if that means nothing to you don’t worry.
The ^ signs mean control.)
 Editing Files - pico
• A more friendly Unix editor is called pico.
To start to it just type pico filename.
It has a menu at the base of the screen
that is reasonably easy to follow.

• If the command pico is not found you

may have to type /usr/local/bin/pico
instead.
Editing Files - pico
 Useful Commands
• Sometimes when you connect to a Unix
computer the backspace key on your
keyboard does not work. To fix this using
the following command:
stty erase [BS]
where [BS] means you hit your
backspace key.
• Sometimes control-h will also work as a
backspace.
 Disk Usage
• Disk usage is a perennial problem. To
see the overall status of the disk space
available on any machine type df -k.

• The following output shows the data and

data2 disks on wildcat are both
completely full!
Disk Usage
 Disk Usage
• To see how much disk space you
personally are using use the command
du -sk. The output will be in kilobytes so
divide by 1000 to get megabytes.
 Disk Usage

1.78 Gigagbytes … I was a

disk hog that day! Though
this is a faculty only drive.
 CPU Usage
• You can see who is using the CPU on
any machine with the command “top”.
Typing top at the command line may not
work so type:
/usr/local/bin/top
 CPU Usage

Hit control-C to get out of this

 CPU Usage
• The output shows user y0l5579 running
one Eclipse 100 job and usr ymd5001
running three Eclipse 300 jobs.

• Wildcat has 4 CPUs so 4 jobs can run

simultaneously without any degradation
in performance.
 Killing Processes
• If you have a process running that needs
to be stopped using the kill command
e.g.

kill - 9 15059

would kill the Eclipse 300 job (if I were its

owner).

USE WITH CARE!

 Unix Filenames
• In Unix it is not generally a good idea to
put spaces or strange characters e.g. “#”
in filenames (even windows will allow it).

• Directories can be created using the

mkdir command. They can be removed
using the rmdir command.

• To move into a directory us the cd

command.
 Unix File Structure
• To move up a level use cd ..

• To copy a file use cp file1 file2.

• To delete a file use rm file

• To rename a file use mv file1 file2

 Unix File Structure
• To show the file in a directory use the ls
command.

• To get more information size, permission,

modification time etc us ls -la.

• The * wildcard character is very useful.

 Unix File Structure

File permissions
 File Permissions
• There are 3 sets of file permssions on
any Unix file: read, write, execute access
for you, members of your “group” and the
rest of the world.

• Usually you only need to be concerned

with the first set of “rwx” values. If you
are placing files in your public_html
directory then they need to be readable
by the rest of the world as well.
 File Permissions
• File permissions are changed using the
chmod command. It is followed by 3
numbers indicating the desired
permissions for you/group/rest of the
world.
• The numbers are determined by
summing the required permissions:
read = 4 read + write = 6
write = 2 read + write
execute = 1 + execute =7
File Permissions
 Further Reading
• For more detailed information about Unix
see “Unix.pdf” posted at
http://pumpjack.tamu.edu/~archer/Unix/
 Scripting
• Unix scripts are collections of Unix
commands that can be used to automate
process such as running simulation jobs
and processing output.

• We’ll discuss shell scripts, sed and awk.

 Shell Scripts
• Shell scripts are executable files (refer
back to the discussion on file
permissions) that contain Unix
commands that might otherwise be typed
in at the command line.

• Unix knows a file is a shell script if it

starts with #!/bin/sh
Shell Scripts
 Shell Scripts
• To use this script just type EclClean at
the command prompt. (The script file
would have to be in the same directory
as the Eclipse files you are trying to
remove.)

• Any Eclipse files with names ending in

.DBG, .EGRID etc would be deleted.
 ppw Script
• ppw is an existing script that you may
use. It takes arguments:

• ppw “pattern” lines repetition filename

ppw Script

“more” displays contents of file

on screen
 ppw Script
• If you want to redirect the output from
ppw to a file not to the screen use:

ppw “pattern” lines repetition file1 > file2

 Example: Repeated Eclipse
Runs

< ecl_input means that script takes input from this file
Example: Repeated Eclipse
Runs
Example: Changing Eclipse
Data Files
 sed - stream editor
• Sed is a command line based text editor
that can be very useful as part of shell
scripts.

sed -n ‘1,3p’ myfile

prints the first 3 lines of myfile
sed -n ‘2,4p’ myfile
prints lines 2 to 4
 sed - stream editor
sed -n ‘1,2d’ myfile
deletes lines 1 to 2 and prints the rest of
the file

sed ‘s/abc/xyz/’ myfile > myNewFile

replaces all instances of “abc” in myfile
with “xyz” and writes the result to the file
called myNewFile.
Example - Editing Multiple Files
• The next script example edits multiple
files at once.

• You supply a pattern, e.g. *.DATA, for the

files names. sed commands to make the
changes to the files.
Example - Editing Multiple Files
 awk
• If you think of sed as a text editor, think of
awk as more like a spreadsheet. It reads
files line by line and divides the results
into fields.

• The first value in any row is referred to as

$1, the next is $2 etc.
awk - Example 1
awk - Example 2

awk can do simple math

 awk programming
• You can also program in awk using if/else
structures. There are plenty of examples
of this around on the web etc.
awk programming
 paste command
• Paste is a Unix command which takes
input from two or more files (up to 12)
and pastes them next to each other in
columns.

Columns 1 Columns 2
abc 123
abc 123
abc 123
abc
 paste command
paste columns1 columns2 >
columns3
creates a file called columns3 with the
following entries:

a b c 1 2 3
a b c 1 2 3
a b c 1 2 3
a b c
 cat command
• Cat joins files together one after another.

cat columns1 columns2 > rows

gives
a b c
a b c
a b c
a b c
1 2 3
1 2 3
1 2 3
 grep command
• Grep is a command which gathers lines
from a file which match a certain pattern.
Usuage:

grep pattern file

• Example. Collect all data for well 3986

from the file well.sum

grep 3986 well.sum