Department of Industrial Engineering

Division of Information Technology Managment

Chapter 11
Software Security
Information Security
2025 Fall Semester
Younho Lee
Learning Objective (1/2)
❖ Describe how many computer security vulnerabilities are a result of poor
programming practices

❖ Describe an abstract view of a program, and detail where potential points

of vulnerability exist in this view

❖ Describe how a defensive programming approach will always validate any

assumptions made, and is designed to fail gracefully and safely whenever
errors occur

❖ Detail the many problems that occur as a result of incorrectly handling

program input, failing to check its size or interpretation

2
Learning Objective (2/2)
❖Describe problems that occur in implementing some algorithms

❖ Describe problems that occur as a result of interaction between

programs and OS components

❖ Describe problems that occur when generating program output

3
Software Security Issues (1/7)

Introducing Software Security and Defensive Programming

❖ Critical Web application
security flaws include Software error
injection flaws categories:
▪ The flaws occur as a
consequence of
insufficient checking and • Insecure interaction
validation of data and between
error codes components
▪ Awareness of theses issues is
a critical initial step in
• Risky resource
writing more secure program management
code • Porous defenses

4
CWE/SANS Top 25 Most Dangerous Software Errors (2022)

5
Software Security Issues (1/7)

Reducing Software Vulnearabilities

❖ The NIST report NISTIR 8151 presents a range of approaches to

reduce the number of software vulnerabilities

❖ It recommends:
▪ Stopping vulnerabilities before they occur by using improved methods for
specifying, designing, and building software

▪ Finding vulnerabilities before they can be exploited by using better testing

techniques and more efficient use of multiple testing methods

▪ Reducing the impact of vulnerabilities by building more resilient architectures

6
Software Security Issues (3/7)

Software Security, Quality, and Reliability

❖ Software quality and ❖ Software security:
reliability: ▪ Attacker chooses probability
▪ Concerned with the accidental distribution, specifically targeting
failure of program as a result of bugs that result in a failure that
some theoretically random, can be exploited by the attacker
unanticipated input, system ▪ Triggered by inputs that differ
interaction, or use of incorrect dramatically from what is usually
code expected
▪ Improve using structured design ▪ Unlikely to be identified by
and testing to identify and common testing approaches
eliminate as many bugs as possible
from a program
▪ Concern is not how many bugs,
but how often they are triggered

7
Software Security Issues (4/7)

Defensive Programming (1/3)

❖ Designing and implementing software so that it continues to function
even when under attack

❖ Requires attention to all aspects of..

▪ Program execution
▪ Environment
▪ Type of data the program processes

❖ Software is able to detect erroneous conditions resulting from some

attack

❖ Also referred to as secure programming

❖ Key rule is to never assume anything, check all assumptions and handle
any possible error states

8
Software Security Issues (5/7)

Defensive Programming (2/3)

❖ Abstract view of program

9
Software Security Issues (6/7)

Defensive Programming (3/3)

Programmers often make Conflicts with

Requires a changed business pressures to
assumptions about the type mindset to traditional
of inputs a program will keep development
receive and the programming practices times as short as
possible to maximize
environment it executes in market advantage
Programmers have to
understand how failures can
occur and the steps needed
to reduce the chance of
Assumptions need to be them occurring in their
validated by the program and
all potential failures handled programs
gracefully and safely

10
Software Security Issues (7/7)

Security by Design
❖ Security and reliability are common design goals in most engineering
principles
▪ Society is intolerant of bridge collapsing, building falling down, or airplanes
crashing
❖ Software development not as mature
▪ Society tolerates far higher levels of failure in software
❖ Recent years have seen increasing efforts to improve secure software
development processes
▪ SAFECode (Software Assurance Forum for Excellence in Code)

Develop publications outlining industry best practices for software

assurance and providing practical advice for implementing proven
methods for secure software development

11
Handling Program Input (1/15)

Input is any source

of data from
outside and whose
Incorrect handling value is not
is a very common
explicitly known by
failing
the programmer
when the code was
written

Explicitly validate
Must identify all assumptions on size
data sources and type of values
before use

12
Handling Program Input (2/15)

Input Size & Buffer Overflow

❖ Programmers often make assumptions about the maximum expected
size of input
▪ Allocated buffer size is not confirmed
▪ Resulting in buffer overflow

❖ Testing may not identify vulnerability

▪ Test inputs are unlikely to include large enough inputs to trigger the
overflow

❖ Safe coding treats all input as dangerous

▪ Should this occur, the program must handle this error gracefully

13
Handling Program Input (3/15)

Interpretation of Program Input

❖ Program input may be binary or text
▪ Binary interpretation depends on encoding and is usually application-specific
E.g.> integers, floating point numbers, complex binary structures used by
network protocols

❖ There is an increasing variety of character sets being used

▪ Care is needed to identify just which set is being used and what characters
are being read

❖ Failure to validate may result in an exploitable vulnerability

▪ E.g> 2014 Heartbleed OpenSSL bug:

An attacker could access the contents of adjacent memory because of a

coding error, failing to check the amount of the data requested for return

14
Handling Program Input (4/15)

Injection Attacks
❖ Flaws relating to invalid handling of input data,
specifically when program input data can accidentally or
deliberately influence the flow of execution of the program

Most often occur in

scripting languages
• Encourage reuse of other programs and
system utilities where possible to save
coding effort
• Often used as Web CGI scripts

15
Handling Program Input (5/15)

Injection Attacks – A Web CGI Injection Attack (1/2)

❖ Unsafe Perl finger CGI script

❖ Finger form

16
Handling Program Input (6/15)

Injection Attacks – A Web CGI Injection Attack (2/2)

❖ Expected and surverted finger CGI responses
‘lpb’ is provided as input

‘xxx; echo attack success; ls -l finger*’ is given as an input

❖ Safety extension to Perl finger CGI script

17
Handling Program Input (7/15)

Injection Attacks – SQL Injection Attack

❖Vulnerable PHP code
Bob’; drop table suppliers can be given as an input

❖ Safety extension to Perl finger CGI script

mysql_real_escape_string(): add ‘backslash’ character after each of

dangerous characters

18
Handling Program Input (8/15)

Injection Attacks – PHP Code Injection Example

❖ Vulnerable PHP code

❖ HTTP exploit request

19
Handling Program Input (9/15)

Cross Site Scripting (XSS) Attacks (1/3)

Commonly seen
in scripted Web XSS reflection
applications Exploit vulnerability
• Vulnerability involves assumption that • Attacker includes the
Attacks where the inclusion of script all content from malicious script
code in the HTML
input provided by content one site is content in data
equally trusted supplied to a site
one user is • Script code may need
subsequently to access data and hence is
output to associated with other permitted to
pages
another user • Browsers impose interact with
security checks and other content
restrict data access from the site
to pages originating
from the same site

20
Handling Program Input (10/15)

Cross Site Scripting (XSS) Attacks (2/3) – XSS example

❖ Plain XSS example

▪ Suppose the following is stored in a guestbook
▪ The script is executed when the text is viewed

Encoded XSS example

21
Handling Program Input (11/15)

Cross Site Scripting (XSS) Attacks (3/3)

❖ XSS reflection attack

Trusted web sites

Malicious URL link is sent including malicious scripts or
iframe

When the link is clicked, attached variables are

also sent

The victim site sends back the variables with normal

HTML content
Victim

22
Handling Program Input (12/15)

Validating Input Syntax (1/3)

It is necessary to Input data By only

ensure that data should be Alternative is to
accepting known
conform with compared compare the
safe data the
any assumptions against what is input data with
program is more
made about the wanted known
likely to remain
data before dangerous values
(whitelisting) secure
subsequent use

23
Handling Program Input (13/15)

Validating Input Syntax (2/3) – Alternate Encodings

Growing requirement to support

May have multiple means of users around the globe and to
encoding text interact with them using their
own languages

Unicode used for Canonicalization

internationalization • Transforming input data into a single,

• Uses 16-bit value for characters standard, minimal representation

• Once this is done the input data can be
• UTF-8 encodes as 1-4 byte sequences
compared with a single representation of
• Many Unicode decoders accept any valid
acceptable input values
equivalent sequence

24
Handling Program Input (14/15)

Validating Input Syntax (3/3) – Validating Numeric Input

❖ Additional concern when input data represents numeric

values

❖ Internally stored in fixed sized value

▪ 8, 16, 32, 64 bit integers
▪ Floating point numbers depend on the processor used
▪ Values may be signed or unsigned

❖ Must correctly interpret text form and process consistently

▪ Have issues comparing signed to unsigned
▪ Could be used to thwart buffer overflow attack

25
Handling Program Input (15/15)

Input Fuzzing
❖ Developed by Prof. Barton Miller at the University of
Wisconsin Madison in 1989
❖ Software testing technique that uses randomly generated
data as inputs to a program
▪ Range of inputs is very large
▪ Intent is to determine if the program or function correctly handles abnormal
inputs
▪ Simple, free of assumptions, cheap
▪ Assists with reliability as well as security
❖ Can also use templates to generate classes of known-
problem inputs
▪ Disadvantage is that bugs triggered by other forms of input would be missed
▪ Combination of approaches is needed for reasonably comprehensive
coverage of the inputs

26
Writing Safe Program Code (1/6)
❖ Processing of the input data according to some algorithm
→ the 2nd component of our model of computer program

❖ High-level languages are typically compiled and linked into

machine code which is then directly executed by the target
processor

Security issues:
• Correct algorithm implementation
• Correct machine instructions for algorithm
• Valid manipulation of data

27
Writing Safe Program Code (2/6)

Correct Algorithm Implementation

Another variant is when the

Issue of good program programmers deliberately include
development technique additional code in a program to
help test and debug it

Often code remains in production release of

a program and could inappropriately release
Algorithm may not correctly handle information
all problem variants

May permit a user to bypass security checks

and perform actions they would not
otherwise be allowed to perform

Consequence of deficiency is a bug in

the resulting program that could be
exploited This vulnerability was exploited by the
Morris Internet Worm

28
Ensuring Machine Language Corresponds to
Algorithm
❖ Issue is ignored by most programmers
▪ Assumption is that the compiler or interpreter generates or executes code
that validly implements the language statements

❖ Requires comparing machine code with original source

▪ Slow and difficult

❖ Development of computer systems with very high assurance level is the

one area where this level of checking is required
▪ Specifically common criteria assurance level of EAL7

29
Writing Safe Program Code (4/6)

Correct Data Interpretation

❖ Data stored as ❖ Different languages
bits/bytes in computer provide different
▪ Grouped as words or long capabilities for restricting
words and validating
▪ Accessed and manipulated in
memory or copied into
interpretation of data in
processor registers before variables
being used ▪ Strongly-typed languages are
▪ Interpretation depends on more limited, safer
machine instruction executed • c.f.> C language
▪ Other languages allow more
liberal interpretation of data and
permit program code to explicitly
change their interpretation

30
10.1 Stack Overflows - Stack Buffer Overflow (11/15)

Shellcode development (Cont’d)

NOP
JMPSHORT POPA %esi
(Go ahead by 26 XOR From mov %al,
(0x58+0x06)
byte addresses) eax to 0x7(%esi)
eax

0xb0: move the INT 0x80

next byte value ASCII code call -31 (relative
to %al representation of address)
“/bin/sh”

31
Writing Safe Program Code (5/6)

Correct Use of Memory

❖ Issue of dynamic memory allocation
▪ Used to manipulate unknown amounts of data
▪ Allocated when needed, released when done

❖ Memory leak
▪ Steady reduction in memory available on the heap to the point where it is
completely exhausted

❖ Many old languages have no explicit support for dynamic

memory allocation
▪ Use standard library routines to allocate and release memory
▪ Difficulty in determining when the dynamic memory is no longer needed

❖ Modern languages handle automatically

▪ Resulting programs are far more reliable
32
Writing Safe Program Code (6/6)

Race Conditions
❖ Without synchronization of accesses it is possible that values may be
corrupted or changes lost due to overlapping access, use, and
replacement of shared values

❖ Arise when writing concurrent code whose solution requires the correct
selection and use of appropriate synchronization primitives

❖ Deadlock
▪ Processes or threads wait on a resource held by the other
▪ One or more programs has to be terminated

*Example of race
condition
33
Interacting with The operating system and Other Programs (1/15)

Operating System Interaction

❖ Programs execute on systems under the control of
an operating system
▪ Mediates and shares access to resources
▪ Constructs execution environment
▪ Includes environment variables and arguments

❖ Systems are of multiple users

▪ Resources are owned by a user and have permissions granting access
with various rights to different categories of users
▪ Programs need access to various resources, however excessive levels of
access are dangerous
▪ Concerns when multiple programs access shared resources such as a
common file

34
Interacting with The operating system and Other Programs (2/15)

Environment Variables
❖ Collection of string values inherited by each process from
its parent
▪ Can affect the way a running process behaves
▪ Included in memory when it is constructed
▪ E.g.> LD_LIBRARY_PATH, PATH

❖ Can be modified by the program process at any time

▪ Modifications will be passed to its children

❖ Another sources of untrusted program input

❖ Most common use is by a local user attempting to gain

increased privileges
▪ Attacker’s program can be called by a setuid script by modifying PATH
variable
▪ Goal is to subvert a program that grants superuser or administrator privileges

35
Interacting with The operating system and Other Programs (3/15)

Environment Variables - example

36
Interacting with The operating system and Other Programs (4/15)

Vulnerable Compiled Programs

Programs can be vulnerable to PATH

variable manipulation
• Must reset to “safe” values

Dynamically linked may be vulnerable to

manipulation of LD_LIBRARY_PATH
• Used to locate suitable dynamic library
• Must either statically link privileged programs
or prevent use of this variable

37
Interacting with The operating system and Other Programs (5/15)

Use of Appropriate, Least Privileges

Privilege escalation
•Exploit of flaws may give attacker greater privileges

Least privilege
•Run programs with least privilege needed to complete their
function

Determine appropriate user and group

privileges required
•Decide whether to grant extra user or just group privileges

Ensure that privileged program can modify only

those files and directories necessary
c.f> Configuration of conventional Web servers and their document
directories

38
Interacting with The operating system and Other Programs (6/15)

Root/Administrator Privileges

Programs with root/ • They provide highest levels of

administrator privileges system access and control
are a major target of • Are needed to manage access to
protected system resources
attackers

Often privilege is only • Can then run as normal user

needed at start

Good design partitions • Provides a greater degree of

isolation between the components
complex programs in • Reduces the consequences of a
smaller modules with security breach in one component
needed privileges • Easier to test and verify

39
Interacting with The operating system and Other Programs (7/15)

System Calls and Standard Library Functions

Programs use system calls and standard library functions for common
operations

Programmers make assumptions about their operations

In some cases, the assumptions may not work as intended

May be a result of system optimizing access to shared resources

Results in requests for services being buffered, resequenced, or otherwise modified to

optimize system use
Optimizations can conflict with program goals

40
Interacting with The operating system and Other Programs (8/15)

Secure File Shredding Problem (1/2)

❖ ‘delete’ operation supported by OS does not work
▪ Only performs deleting the linkage between the file’s information and
the file’s content

❖ Standard recommendation
▪ Repeatedly overwrite the data contents with several distinct bit
patterns to minimize the likelihood of the original data being
recovered

❖ Initial secure file shredding program algorithm

Problems?
→ Buffering, File I/O scheduling

41
Interacting with The operating system and Other Programs (9/15)

Secure File Shredding Problem (2/2)

❖ Better secure file shredding program algorithm

42
Interacting with The operating system and Other Programs (10/15)

Preventing Race Conditions

❖ Programs may need to access a common system resources
▪ Examples
• Mail client and mail delivery programs sharing access to mailbox
• Various users of a Web CGI script updating the same file used to save submitted
form values

❖ Need suitable synchronization mechanisms

▪ Most common technique is to acquire a lock on the shared file
File
❖ Lockfile
▪ Process must create and own the lockfile in order to gain access to the shared
resource File for Timestamp/
resource 1 Creator’s ID
▪ Concerns
• If a program chooses to ignore the existence of the lockfile and access the shared
resource the system will not prevent this
• All programs using this form of synchronization must cooperate
• Implementation
– Must guarantee ‘atomic operation’

43
Interacting with The operating system and Other Programs (12/15)

Perl File Locking Example

❖ Exclusive access of the opened file is possible after ‘flock’
operation with $EXCL_LOCK parameter

44
Interacting with The operating system and Other Programs (13/15)

Safe Temporary Files (1/2)

❖ Many programs use temporary files

❖ Characteristics regarding using temporary files

▪ Files are in common, shared area
▪ Names should be unique, not accessible by others
• Process ID can be included in the name → Predictable by attackers

❖ Attackers might guess and attempt to modify a specific

temp file used by a target process Passwd
file
▪ Example in [WHEE03] Symbolic
link
Tripwire Temp
program file Tripwire Temp
program file
Access Attacker Owned
with root Access by
privilege with root attacker
privilege

45
 Interacting with The operating system and Other Programs (14/15)

Safe Temporary Files (2/2)

❖ Secure temporary file creation and use requires the use of
random names

❖ C Temporary file creation example

▪ Once the program finishes using the file, it should be closed and kept unlinked

46
Interacting with The operating system and Other Programs (15/15)

Interacting with Other Programs

Programs may use functionality and services of other

programs

Security vulnerabilities can result unless care is taken with

this interaction

Issue of data confidentiality/integrity

Detection and handling of exceptions and errors generated

by interaction is also important from a security perspective

47
Handling Program Output
❖ Final component is program output
▪ May be stored for future use, sent over net, displayed
▪ May be binary or text

❖ Important from a program security perspective that the output conform

to the expected form and interpretation
▪ Can affect the security if the output is used for other entities’ input
▪ E.g> Output looking like a pure text but has many control characters, email
including XSS messages

❖ Programs must identify what is permissible output content and filter any
possibly untrusted data to ensure only valid output is displayed
▪ E.g.> removing all HTML markup, identifying the character set

48
Summary
❖ Software security issues ❖Handling program input
▪ Input size and buffer overflow
▪ Introducing software security and
▪ Interpretation of program input
defensive programming
▪ Validating input syntax
❖ Writing safe program code ▪ Input fuzzing
▪ Correct algorithm implementation ❖ Interacting with the
▪ Ensuring that machine language operating system and other
corresponds to algorithm programs
▪ Environment variables
▪ Correct interpretation of data
▪ Using appropriate, least
values
privileges
▪ Correct use of memory ▪ System calls and standard
▪ Preventing race conditions with library functions
shared memory ▪ Safe temporary file use
❖ Handling program output ▪ Preventing race conditions with
shared system resources
▪ Interacting with other programs
49