Part IV: Software

Part 4  Software 1
Why Software?
 Why is software as important to security as crypto, access
control, protocols?
 Virtually all information security features are implemented
in software
 If your software is subject to attack, your security can be
broken
o Regardless of strength of crypto, access control, or protocols
 Software is a poor foundation for security

Part 4  Software 2
Bad Software is Ubiquitous
 NASA Mars Lander (cost $165 million)
o Crashed into Mars due to…
o …error in converting English and metric units of measure
o Believe it or not
 Denver airport
o Baggage handling system  very buggy software
o Delayed airport opening by 11 months
o Cost of delay exceeded $1 million/day
o What happened to person responsible for this fiasco?
 MV-22 Osprey
o Advanced military aircraft
o Faulty software can be fatal

Part 4  Software 3
Software Issues

Alice and Bob (Normal Trudy

User)  Actively looks for
 Find bugs and flaws bugs and flaws
by accident  Likes bad software…
 Hate bad software…
 …and tries to make
 …but they learn to it misbehave
live with it  Attacks systems via
bad software

Part 4  Software 4
 Complexity
 “Complexity is the enemy of security”, Paul
Kocher, Cryptography Research, Inc.

System Lines of Code (LOC)

Netscape 17 million
Space Shuttle 10 million
Linux kernel 2.6.0 5 million
Windows XP 40 million
Mac OS X 10.4 86 million
Boeing 777 7 million
 A new car contains more LOC than was required
to land the Apollo astronauts on the moon
Part 4  Software 5
Lines of Code and Bugs
 Conservative estimate: 5 bugs/10,000 LOC
 Do the math
o Typical computer: 3k exe’s of 100k LOC each
o Conservative estimate: 50 bugs/exe
o Implies about 150k bugs per computer
o So, 30,000-node network has 4.5 billion bugs
o Maybe only 10% of bugs security-critical and only 10% of those
remotely exploitable
o Then “only” 45 million critical security flaws!

Part 4  Software 6
Software Security Topics
 Program flaws (unintentional)
o Buffer overflow
o Incomplete mediation
o Race conditions
 Malicious software (intentional)
o Viruses
o Worms
o Other breeds of malware

Part 4  Software 7
Program Flaws
 An error is a programming mistake
o To err is human
 An error may lead to incorrect state: fault
o A fault is internal to the program
 A fault may lead to a failure, where a system departs from
its expected behavior
o A failure is externally observable

error fault failure

Part 4  Software 8
 Example
char array[10];
for(i = 0; i < 10; ++i)
array[i] = `A`;
array[10] = `B`;
 This program has an error
 This error might cause a fault
o Incorrect internal state
 If a fault occurs, it might lead to a failure
o Program behaves incorrectly (external)
 We use the term flaw for all of the above
Part 4  Software 9
Secure Software
 In software engineering, try to ensure that a program
does what is intended
 Secure software engineering requires that software does
what is intended…
 …and nothing more
 Absolutely secure software? Dream on…
o Absolute security anywhere is impossible
 How can we manage software risks?

Part 4  Software 10
Program Flaws
 Program flaws are unintentional
o But can still create security risks
 We’ll consider 3 types of flaws
o Buffer overflow (smashing the stack)
o Incomplete mediation
o Race conditions
 These are the most common flaws

Part 4  Software 11
 Buffer Overflow

Part 4  Software 12
Attack Scenario
 Users enter data into a Web form
 Web form is sent to server
 Server writes data to array called buffer, without checking
length of input data
 Data “overflows” buffer
o Such overflow might enable an attack
o If so, attack could be carried out by anyone with Internet access

Part 4  Software 13
 Buffer Overflow
int main(){
int buffer[10];
buffer[20] = 37;}

 Q: What happens when code is executed?

 A: Depending on what resides in memory
at location “buffer[20]”
o Might overwrite user data or code
o Might overwrite system data or code
o Or program could work just fine

Part 4  Software 14
 Simple Buffer Overflow
 Consider boolean flag for authentication
 Buffer overflow could overwrite flag
allowing anyone to authenticate
Boolean flag
buffer
F OU R S C … T
F

 In some cases, Trudy need not be so lucky

as in this example
Part 4  Software 15
 Memory Organization
¬ low
 Text  code text address
 Data  static variables data
 Heap  dynamic data
heap
 Stack  “scratch paper” 
¬ stack
o Dynamic local variables pointer (SP)
o Parameters to functions 
stack ¬ high
o Return address
address

Part 4  Software 16
 Simplified Stack Example
(Let func is called)
low 

void func(int a, int b){ :

char buffer[10];
:
}
void main(){
func(1,2); ¬ SP
buffer
}
¬ return
SP
ret address
a ¬ SP

high  b ¬ SP

Part 4  Software 17
 Smashing the Stack
low 

 What happens if :
??? :
buffer overflows?
 Program “returns” ¬ SP
to wrong location buffer
overflow
ret ¬ ret…
SP NOT!
A crash is likely
overflow
a ¬ SP

high  b ¬ SP

Part 4  Software 18
 Smashing the Stack
low 
 Trudy has a
:
better idea… :
 Code injection
 Trudy can run ¬ SP
code of her evil code

choosing… ret
ret ¬ SP
a ¬ SP
o …on your machine
high  b ¬ SP

Part 4  Software 19
 Smashing the Stack
:
:
 Trudy may not know… NOP
1) Address of evil code :
2) Location of ret on stack NOP

 Solutions evil code

1) Precede evil code with ret

¬ ret
NOP “landing pad” ret
:
2) Insert ret many times
ret
:
Part 4  Software 20 :
 Stack Smashing Summary
 A buffer overflow must exist in the code
 Not all buffer overflows are exploitable
o Things must align properly
 If exploitable, attacker can inject code
 Trial and error is likely required
o Fear not, lots of help is available online
o Smashing the Stack for Fun and Profit, Aleph One
 Stack smashing is “attack of the decade”…
o …for many recent decades
o Also heap & integer overflows, format strings, etc.

Part 4  Software 21
 Stack Smashing Example
 Suppose program asks for a serial number
that Trudy does not know
 Also, Trudy does not have source code
 Trudy only has the executable (exe)

 Program quits on incorrect serial number

Part 4  Software 22
 Buffer Overflow Present?
 By trial and error, Trudy discovers
apparent buffer overflow

 Note that 0x41 is ASCII for “A”

 Looks like ret overwritten by 2 bytes!
Part 4  Software 23
 Disassemble Code
 Next, disassemble bo.exe to find

 The goal is to exploit buffer overflow

to jump to address 0x401034
Part 4  Software 24
 Buffer Overflow Attack
 Find that, in ASCII, 0x401034 is “@^P4”

 Byte order is reversed? What the … ?

 X86 processors are “little-endian”
Part 4  Software 25
 Overflow Attack, Take 2
 Reverse the byte order to “4^P@” and…

 Success! We’ve bypassed serial number

check by exploiting a buffer overflow
 What just happened?
o Overwrote return address on the stack

Part 4  Software 26
Buffer Overflow
 Trudy did not require access to the source code
 Only tool used was a disassembler to determine
address to jump to
 Find desired address by trial and error?
o Necessary if attacker does not have exe
o For example, a remote attack

Part 4  Software 27
 Source Code
 Source code for buffer overflow example
 Flaw easily
exploited by
attacker…
 …without
access to
source code!

Part 4  Software 28
Stack Smashing Defenses
 Employ non-executable stack
o “No execute” NX bit (if available)
o Seems like the logical thing to do, but some real code executes on
the stack (Java, for example)
 Use a canary
 Address space layout randomization (ASLR)
 Use safe languages (Java, C#)
 Use safer C functions
o For unsafe functions, safer versions exist
o For example, strncpy instead of strcpy

Part 4  Software 29
 Stack Smashing Defenses
low 
:
 Canary :
o Run-time stack check
o Push canary onto stack
o Canary value: buffer
 Constant 0x000aff0d overflow
canary ¬

 Or, may depends on ret overflow

ret
a
high  b
Part 4  Software 30
 Microsoft’s Canary
 Microsoft added buffer security check
feature to C++ with /GS compiler flag
o Based on canary (or “security cookie”)
Q: What to do when canary dies?
A: Check for user-supplied “handler”
 Handler shown to be subject to attack
o Claimed that attacker can specify handler code
o If so, formerly “safe” buffer overflows become
exploitable when /GS is used!

Part 4  Software 31
 ASLR
 Address Space Layout Randomization
o Randomize place where code loaded in memory
 Makes most buffer overflow attacks
probabilistic
 Windows Vista used 256 random layouts
o So about 1/256 chance buffer overflow works
 Similar thing in Mac OS X and other OSs
 Attacks against Microsoft’s ASLR do exist
o Possible to “de-randomize”

Part 4  Software 32
 Buffer Overflow
 A major security threat yesterday, today,
and tomorrow
 The good news?
o It is possible to reduce overflow attacks (safe
languages, NX bit, ASLR, education, etc.)
 The bad news?
o Buffer overflows will exist for a long time
o Why? Legacy code, bad development practices,
clever attacks, etc.

Part 4  Software 33
 Incomplete Mediation

Part 4  Software 34
Input Validation
 Consider: strcpy(buffer, argv[1])
 A buffer overflow occurs if
len(buffer) < len(argv[1])
 Software must validate the input by checking the length
of argv[1]
 Failure to do so is an example of a more general problem:
incomplete mediation

Part 4  Software 35
 Input Validation
 Consider web form data
 Suppose input is validated on client
 For example, the following is valid
http://www.things.com/orders/
final&custID=112&num=55A&qty=20&price=10&shipping=5&t
otal=205
 Suppose input is not checked on server
o Why bother since input checked on client?
o Then attacker could send http message
http://www.things.com/orders/
final&custID=112&num=55A&qty=20&price=10&shipping=5&t
otal=25

Part 4  Software 36
 Incomplete Mediation
 Linux kernel
o Research revealed many buffer overflows
o Lots of these due to incomplete mediation
 Linux kernel is “good” software since
o Open-source
o Kernel  written by coding gurus
 Tools exist to help find such problems
o But incomplete mediation errors can be subtle
o And tools useful for attackers too!

Part 4  Software 37
 Race Conditions

Part 4  Software 38
Race Condition
 Security processes should be atomic
o Occur “all at once”
 Race conditions can arise when security-
critical process occurs in stages
 Attacker makes change between stages
o Often, between stage that gives authorization,
but before stage that transfers ownership
 Example: Unix mkdir

Part 4  Software 39
 mkdir Race Condition
 mkdircreates new directory
 How mkdir is supposed to work

mkdir
1. Allocate
space
2. Transfer
ownership

Part 4  Software 40
 mkdir Attack
 The mkdir race condition
mkdir
1. Allocate
space
3. Transfer
ownership

2. Create link to
password file

 Not really a “race”

o But attacker’s timing is critical
Part 4  Software 41
 Race Conditions
 Race conditions are common
 Race conditions may be more prevalent
than buffer overflows
 But race conditions harder to exploit
o Buffer overflow is “low hanging fruit” today
 To prevent race conditions, make security-
critical processes atomic
o Occur all at once, not in stages
o Not always easy to accomplish in practice

Part 4  Software 42
 Malware

Part 4  Software 43
 Malicious Software
 Malware is not new…
o Fred Cohen’s seminal virus work in 1980’s
 Types of malware (no standard definition)
o Virus  passive propagation
o Worm  active propagation
o Trojan horse  unexpected functionality
o Trapdoor/backdoor  unauthorized access
o Rabbit  exhaust system resources
o Spyware  steals info, such as passwords
o Botnets  malware for hire
o Ransomware  Use malware to encrypt data

Part 4  Software 44
 Where do Viruses Live?
 They live just about anywhere, such as…
 Boot sector
o Take control before anything else
 Memory resident
o Stays in memory
 Applications, macros, data, etc.
 Library routines
 Compilers, debuggers, virus checker, etc.
o These would be particularly nasty!

Part 4  Software 45
Malware Timeline
 Brain virus (1986)
 Morris worm (1988)
 Code Red (2001)
 SQL Slammer (2004)
 Botnets
 Stuxnet (2010)
 Ransomware
 Future of malware?

Part 4  Software 46
 Brain
q First appeared in 1986
q More annoying than harmful
q A prototype for later viruses
q Not much reaction by users
q What it did
1. Placed itself in boot sector (and other places)
2. Screened disk calls to avoid detection
3. Each disk read, checked boot sector to see if
boot sector infected; if not, goto 1
q Brain did nothing really malicious
Part 4  Software 47
 Morris Worm
 Appeared in 1988
 What it tried to do
o Determine where it could spread, then…
o …spread its infection and…
o …remain undiscovered
 Morris claimed his worm had a bug!
o It tried to re-infect infected systems
o Led to resource exhaustion
o Effect was like a so-called rabbit
Part 4  Software 48
How Morris Worm Spread

 Obtained access to machines by…

o User account password guessing
o Exploit buffer overflow in fingerd
o Exploit trapdoor in sendmail
 Flaws in fingerd and sendmail were well-known, but
not widely patched

Part 4  Software 49
 Bootstrap Loader
 Once Morris worm got access…
 “Bootstrap loader” sent to victim
o 99 lines of C code
 Victim compiled and executed code
 Bootstrap loader fetched the worm
 Victim authenticated sender
o Don’t want user to get a bad worm…

Part 4  Software 50
How to Remain Undetected?

 If transmission interrupted, all code

deleted
 Code encrypted when downloaded
 Code deleted after decrypt/compile
 When running, worm regularly changed
name and process identifier (PID)

Part 4  Software 51
 Morris Worm: Bottom Line
 Shocked the Internet community of 1988
o Internet of 1988 much different than today
 Internet designed to survive nuclear war
o Yet, brought down by one graduate student!
o At the time, Morris’ father worked at NSA…
 Could have been much worse
 Result? CERT, more security awareness
 But should have been a wakeup call

Part 4  Software 52
 Code Red Worm
 Appeared in July 2001
 Infected more than 250,000 systems
in about 15 hours
 Eventually infected 750,000 out of
about 6,000,000 vulnerable systems
 Exploited buffer overflow in
Microsoft IIS server software
o Then monitor traffic on port 80, looking
for other susceptible servers
Part 4  Software 53
 Code Red: What it Did
 Day 1 to 19 of month: spread its infection
 Day 20 to 27: distributed denial of service
attack (DDoS) on www.whitehouse.gov
 Later version (several variants)
o Included trapdoor for remote access
o Rebooted to flush worm, leaving only trapdoor
 Some said it was “beta test for info warfare”
o But, no evidence to support this

Part 4  Software 54
 SQL Slammer
 Infected 75,000 systems
in 10 minutes!
 At its peak, infections
doubled every 8.5 seconds
 Spread “too fast”…
 …so it “burned out”
available bandwidth

Part 4  Software 55
 Why was Slammer Successful?
 Worm size: one 376-byte UDP packet
 Firewalls often let one packet thru
o Then monitor ongoing “connections”
 Expectation was that much more data
required for an attack
o So no need to worry about 1 small packet
 Slammer defied “experts”

Part 4  Software 56
 Stuxnet
 Malware for information warfare…
 Discovered in 2010
o Origins go back to 2008, or earlier
 Apparently, targeted Iranian nuclear
processing facility
o Reprogrammed specific type of PLC
o Changed speed of centrifuges, causing
damage to 1000 (or more) of them

Part 4  Software 57
 Stuxnet
 Many advanced features including…
o Infect system via removable drives 
able to get behind “airgap” firewalls
o Used 4 unpatched MS vulnerabilities
o Updates via P2P over LAN
o Contact C&C server for code/updates
o Includes a Windows rootkit for stealth
o Significant exfiltration/recon capability
o Used a compromised private key
Part 4  Software 58
Malware Related to Stuxnet

 Duqu (2011)
o Likely that developers had access to Stuxnet source
code
o Apparently, used mostly for info stealing
 Flame (2012)
o May be “most complex” malware ever
o Very sophisticated spyware mechanisms

Part 4  Software 59
 Trojan Horse Example
 Trojan: unexpected functionality
 Prototype trojan for the Mac
 File icon for freeMusic.mp3:
 For a real mp3, double click on icon
o iTunes opens
o Music in mp3 file plays
 But for freeMusic.mp3, unexpected results…

Part 4  Software 60
 Mac Trojan
 Double click on freeMusic.mp3
o iTunes opens (expected)
o “Wild Laugh” (not expected)
o Message box (not expected)

Part 4  Software 61
 Trojan Example
 How does freeMusic.mp3 trojan work?
 This “mp3” is an application, not data

 This trojan is harmless, but…

 …could have done anything user could do
o Delete files, download files, launch apps, etc.
Part 4  Software 62
Botnet
 Botnet: a “network” of infected machines
 Infected machines are “bots”
o Victim is unaware of infection (stealthy)
 Botmaster controls botnet
o Often, using IRC
o P2P botnet architectures exist
 Botnets used for…
o Spam, DoS attacks, keylogging, ID theft, etc.

Part 4  Software 63
 Botnet Examples
 XtremBot
o Similar bots: Agobot, Forbot, Phatbot
o Highly modular, easily modified
o Source code readily available (GPL license)
 UrXbot
o Similar bots: SDBot, UrBot, Rbot
o Less sophisticated than XtremBot type
 GT-Bots and mIRC-based bots
o mIRC is common IRC client for Windows

Part 4  Software 64
 More Botnet Examples
 Mariposa
o Used to steal credit card info
o Creator arrested in July 2010
 Conficker
o Estimated 10M infected hosts (2009)
 Kraken
o Largest as of 2008 (400,000 infections)
 Srizbi
o For spam, one of largest as of 2008

Part 4  Software 65
Ransomware

 Recently,ransomware very popular

 Malware encrypts important info
o Key is provided only if ransom paid
o Sometimes key works, sometimes not…
 To pay or not to pay?
o Recent case, reportedly paid $5M
o In 2020 total ransom estimated $350M
o Payments are in cryptocurrency
Part 4  Software 66
Malware Detection
 Malware detection techniques
o Signature detection
o Change detection
o Anomaly detection
o Machine learning
 We briefly discuss each of these
o And consider advantages…
o …and disadvantages

Part 4  Software 67
 Signature Detection
 A signature may be a string of bits in exe
o Might also use wildcards, hash values, etc.
 For example, W32/Beast virus has signature
83EB 0274 EB0E 740A 81EB 0301 0000
o That is, this string of bits appears in virus
 We can search for this signature in all files
 If string found, have we found W32/Beast?
o Not necessarily  string could be in normal code
o At random, chance is only 1/2112
o But software is not random…
Part 4  Software 68
 Signature Detection
 Advantages
o Effective on “ordinary” malware
o Minimal burden for users/administrators
 Disadvantages
o Signature file can be large (10s of thousands)…
o …making scanning slow
o Signature files must be kept up to date
o Cannot detect unknown viruses
o Cannot detect some advanced types of malware
 The most popular detection method
Part 4  Software 69
 Change Detection
 Viruses must live somewhere
 If you detect a file has changed
unexpectedly, it might have been infected
 How to detect changes?
o Hash files and (securely) store hash values
o Periodically re-compute hashes and compare
o If hash changes, file might be infected

Part 4  Software 70
 Change Detection
 Advantages
o Few (if any) false negatives
o Can detect previously unknown malware
 Disadvantages
o Many files change  and often
o May be many false alarms (false positives)
o Heavy burden on users/administrators
o If suspicious change detected, then what?
Might have to fall back on signature detection

Part 4  Software 71
 Anomaly Detection
 Monitor system for anything “unusual” or
“virus-like” or “potentially malicious” or …
 Examples of anomalous things
o Files change in some unexpected way
o System misbehaves in some way
o Unexpected network activity
o Unexpected file access, etc., etc., etc., etc.
 But, we must first define “normal”
o And normal can (and must) change over time

Part 4  Software 72
 Anomaly Detection
 Advantages
o Chance of detecting unknown malware
 Disadvantages
o No proven track record
o Trudy might make abnormal look normal (go slow)
o Typically, anomaly detection is combined with another
method (e.g., signature detection)
 Also popular in intrusion detection (IDS)
 Difficult unsolved (unsolvable?) problem
o This sounds like a job for AI (equivalently, machine
learning, deep learning, big data)

Part 4  Software 73
 Machine Learning
 Machine learning and AI is not widely
applied to malware detection problem
o Some AV systems claim to only use AI
 Models trained on features extracted
from known malware samples
 Then trained models used to classify
 Models can be viewed as higher-level
“signatures”
Part 4  Software 74
 Future of Malware
 Recent trends
o Encrypted, polymorphic, metamorphic malware
o Fast replication/Warhol worms
o Flash worms, slow worms
o Botnets, ransomware, and ???
 The future is bright for malware
o Good news for the bad guys…
o …bad news for the good guys
 Future of malware detection?
Part 4  Software 75
Encrypted Viruses
 Virus writers know signature detection used
 So, how to evade signature detection?
 Encrypting the virus is a good approach
o Ciphertext looks like random bits
o Different key, then different “random” bits
o So, different copies have no common signature
 Encryption often used in viruses today

Part 4  Software 76
Encrypted Viruses
 How to detect encrypted viruses?
 Scan for the decryptor code
o More-or-less standard signature detection
o But may be more false alarms
 Why not encrypt the decryptor code?
o Then encrypt the decryptor of the decryptor (and so on…)
 Encryption of limited value to virus writers

Part 4  Software 77
 Polymorphic Malware
 Polymorphic worm
o Body of worm is encrypted
o Decryptor code is “mutated” (or “morphed”)
o Trying to hide decryptor signature
o Like an encrypted worm on steroids…

Q: How to detect?
A: Emulation might work
o Let the code decrypt itself
o Slow, and anti-emulation is possible

Part 4  Software 78
 Metamorphic Malware
 A metamorphic worm mutates before
infecting a new system
o Sometimes called “body polymorphic”
 Such a worm can, in principle, evade
signature-based detection
 Mutated worm must function the same
o And be “different enough” to avoid detection
 Detection is a difficult research problem

Part 4  Software 79
 Metamorphic Worm
 One approach to metamorphic replication…
o The worm is disassembled
o Worm then stripped to a base form
o Random variations inserted into code (permute
the code, insert dead code, etc., etc.)
o Assemble the resulting code

 Result is a worm with same functionality as

original, but different signature

Part 4  Software 80
 Future Malware Detection?
 There is so much malware, maybe it is better to
“detect” good code
o If code not on approved list, assume it’s bad?
o That is, use whitelist instead of blacklist

 Machine learning (AI) is clearly the wave of the

future in malware detection
o Offers many practical advantages
o Trudy may be able to fight back with AI too
o The arms race will likely continue!

Part 4  Software 81
 Miscellaneous
Software-Based
Attacks

Part 4  Software 82
Miscellaneous Attacks
 Numerous attacks involve software
 We’ll discuss a few issues that do not fit into
previous categories
o Salami attack
o Linearization attack
o Time bomb
o Can you ever trust software?

Part 4  Software 83
 Salami Attack
 What is Salami attack?
o Programmer “slices off” small amounts of money
o Slices are hard for victim to detect
 Example
o Bank calculates interest on accounts
o Programmer “slices off” any fraction of a cent
and puts it in his own account
o No customer notices missing partial cent
o Bank may not notice any problem
o Over time, programmer makes lots of money!

Part 4  Software 84
 Salami Attack
 Such attacks are possible for insiders
 Do salami attacks actually occur?
o Or is it just Office Space folklore?
 Programmer added a few cents to every
employee payroll tax withholding
o But money credited to programmer’s tax
o Programmer got a big tax refund!
 Rent-a-car franchise in Florida inflated gas
tank capacity to overcharge customers
Part 4  Software 85
 Salami Attacks
 Employee reprogrammed Taco Bell cash
register: $2.99 item registered as $0.01
o Employee pocketed $2.98 on each such item
o A large “slice” of salami!
 In LA, four men installed computer chip
that overstated amount of gas pumped
o Customers complained when they had to pay for
more gas than tank could hold
o Hard to detect since chip programmed to give
correct amount when 5 or 10 gallons purchased
o Inspector usually asked for 5 or 10 gallons

Part 4  Software 86
Linearization Attack
 Program checks for
serial number
S123N456
 For efficiency,
check made one
character at a time
 Can attacker take
advantage of this?

Part 4  Software 87
 Linearization Attack
 Correct number takes longer than incorrect
 Trudy tries all 1st characters
o Find that S takes longest
 Then she guesses all 2nd characters: S
o Finds S1 takes longest
 And so on…
 Trudy can recover one character at a time!
o Same principle as used in lock picking

Part 4  Software 88
Linearization Attack
 What is the advantage to attacking serial number one
character at a time?
 Suppose serial number is 8 characters and each has 128
possible values
o Then 1288 = 256 possible serial numbers
o Attacker would guess the serial number in about 255 tries  a lot
of work!
o Using the linearization attack, the work is about 8  (128/2) = 29
which is easy

Part 4  Software 89
Linearization Attack
 A real-world linearization attack
 TENEX, an ancient OS (timeshare)
o Passwords checked one character at a time
o Careful timing was not necessary, instead…
o …could arrange for a “page fault” when next
unknown character guessed correctly
o Page fault register was user accessible
 This attack was very easy in practice

Part 4  Software 90
 Time Bomb
 In 1986 Donald Gene Burleson told employer
to stop withholding taxes from his paycheck
 His company refused
 He planned to sue his company
o He used company time to prepare legal docs
o Company found out and fired him
 Burleson had been working on malware…
o After being fired, his software “time bomb”
deleted important company data

Part 4  Software 91
 Time Bomb
 Company was reluctant to pursue the case
o So Burleson sued company for back pay!
o Then the company finally sued Burleson
 In 1988 Burleson fined $11,800
o Case took years to prosecute…
o Cost company many thousands of dollars…
o Resulted in a slap on the wrist for attacker
 An early example of a computer crime cases
 Many cases since follow a similar pattern
o That is, companies are reluctant to prosecute

Part 4  Software 92
 Trusting Software
 Can you ever trust software?
o See Reflections on Trusting Trust
 Consider the following thought experiment
 Suppose C compiler has a virus
o When compiling login program, virus creates
backdoor (account with known password)
o When recompiling the C compiler, virus
incorporates itself into new C compiler
 Difficult to get rid of this virus!
Part 4  Software 93
 Trusting Software
 Suppose you notice something is wrong
 So you start over from scratch
 First, you recompile the C compiler
 Then you recompile the OS
o Including login program…
o You have not gotten rid of the problem!
 Real-world example
o Underhanded C Contest
o Write code that looks like it does something
innocent, but instead is “subtly evil”

Part 4  Software 94
 Chapter 12:
Insecurity in Software
Every time I write about the impossibility of effectively protecting digital files
on a general-purpose computer, I get responses from people decrying the
death of copyright. “How will authors and artists get paid for their work?”
they ask me. Truth be told, I don’t know. I feel rather like the physicist
who just explained relativity to a group of would-be interstellar travelers,
only to be asked: “How do you expect us to get to the stars, then?”
I’m sorry, but I don't know that, either.
 Bruce Schneier

So much time and so little to do! Strike that. Reverse it. Thank you.
 Willy Wonka

Part 4  Software 95
 Software Reverse
Engineering (SRE)

Part 4  Software 96
 SRE
 Software Reverse Engineering
o Also known as Reverse Code Engineering (RCE)
o Or simply “reversing”
 Can be used for good...
o Understand malware
o Understand legacy code
 …or not-so-good
o Remove usage restrictions from software
o Find and exploit flaws in software
o Cheat at games, etc.

Part 4  Software 97
 SRE
 We assume…
o Reverse engineer is an attacker
o Attacker only has exe (no source code)
o No bytecode (i.e., not Java, .Net, etc.)
 Attacker might want to
o Understand the software
o Modify (“patch”) the software
 SRE usually focused on Windows
o So we focus on Windows

Part 4  Software 98
 SRE Tools
 Disassembler
o Converts exe to assembly (as best it can)
o Cannot always disassemble 100% correctly
o In general, not possible to re-assemble
disassembly into working executable
 Debugger
o Must step thru code to completely understand it
o Labor intensive  lack of useful tools
 Hex Editor
o To patch (modify) exe file
 Process Monitor, VMware, etc.
Part 4  Software 99
 Specific Tools
 IDA Pro  good disassembler/debugger
o Costs a few hundred dollars (free version exists)
o Converts binary to assembly (as best it can)
 OllyDbg  high-quality shareware debugger
o Includes a good disassembler
 Hex editor  to view/modify bits of exe
o UltraEdit is good  freeware
o HIEW  useful for patching exe
 Process Monitor  freeware

Part 4  Software 100

 Why is Debugger Needed?
 Disassembly gives static results
o Good overview of program logic
o User must “mentally execute” program
o Difficult to jump to specific place in the code
 Debugging is dynamic
o Can set break points
o Can treat complex code as “black box”
o And code not always disassembled correctly
 Disassembly and debugging both required
for any serious SRE task
Part 4  Software 101
 SRE Necessary Skills
 Working knowledge of target assembly code
 Experience with the tools
o IDA Pro  sophisticated and complex
o OllyDbg  good choice for this class
 Knowledge of Windows Portable Executable
(PE) file format
 Boundless patience and optimism
o SRE is a tedious and labor-intensive process
o Often, like finding a needle in a haystack
Part 4  Software 102
 Java Example
 Consider this
Java program
o User inputs n
o Program
generates n
Fibonacci
numbers

Part 4  Software 103

 Reversing Java Bytecode
 Decompile
bytecode from
Java program on
previous slide
o Used Fernflower
Java decompiler
 It is possible to
obfuscate Java
o Not too effective
(homework)

Part 4  Software 104

SRE Example
 We consider a simple example
 This example only requires disassembly (IDA Pro used
here) and hex editor
o Trudy disassembles to understand code
o Trudy also wants to patch (modify) the code
 For most real-world code, would also need a debugger (e.g.,
OllyDbg)

Part 4  Software 105

 SRE Example
 Program requires serial number
 But Trudy doesn’t know the serial number…

 Can Trudy get serial number from exe?

Part 4  Software 106

SRE Example
 IDA Pro disassembly

 Looks like serial number is S123N456

Part 4  Software 107

SRE Example

 Try the serial number S123N456

 It works!
 Can Trudy do “better”?
Part 4  Software 108
 SRE Example
 Again, IDA Pro disassembly

 And hex view…

Part 4  Software 109

 SRE Example

 “test eax, eax” is AND of eax with itself

o So, zero flag set only if eax is 0
o If test yields 0, then jz is true
 Trudy wants jz to always be true
 Can Trudy patch exe so jz always holds?

Part 4  Software 110

 SRE Example
 Can Trudy patch exe so that jz always true?

xor  jz always true!!!

Assembly Hex
test eax,eax 85 C0 …
xor eax,eax 33 C0 …

Part 4  Software 111

 SRE Example
 Can edit serial.exe with hex editor

serial.exe

serialPatch.exe

 Save as serialPatch.exe
Part 4  Software 112
SRE Example

 Any “serial number” now works!

 Very convenient for Trudy

Part 4  Software 113

 SRE Example
 Back to IDA Pro disassembly…

serial.exe

serialPatch.exe

Part 4  Software 114

 SRE Attack Mitigation
 Impossible to prevent SRE on open systems
 Can we make such attacks more difficult?
 Anti-disassembly techniques
o To confuse static view of code
 Anti-debugging techniques
o To confuse dynamic view of code
 Tamper-resistance
o Code checks itself to detect tampering
 Code obfuscation
o Make code more difficult to understand

Part 4  Software 115

Anti-disassembly
 Anti-disassembly methods include
o Encrypted or “packed” object code
o False disassembly
o Self-modifying code
o Many other techniques
 Encryption prevents disassembly
o But need plaintext decryptor to decrypt code!
o Same problem as with polymorphic viruses

Part 4  Software 116

 Anti-disassembly Example
 Suppose actual code instructions are

inst 1 jmp junk inst 3 inst 4 …

 What a “dumb” disassembler sees

inst 1 inst 2 inst 3 inst 4 inst 5 inst 6 …

 This is example of “false disassembly”

 Persistent attacker will figure it out

Part 4  Software 117

 Anti-debugging
 IsDebuggerPresent() function
 Can also monitor for
o Use of debug registers
o Inserted breakpoints
 Debuggers don’t handle threads well
o Interacting threads may confuse debugger…
o …and therefore, confuse attacker
 Many other debugger-unfriendly tricks
o See next slide for one example

Part 4  Software 118

Anti-debugger Example

inst 1 inst 2 inst 3 inst 4 inst 5 inst 6 …

 Suppose when program gets inst 1, it pre-

fetches inst 2, inst 3, and inst 4
o This is done to increase efficiency
 Suppose when debugger executes inst 1, it
does not pre-fetch instructions
 Can we use this difference to confuse the
debugger?
Part 4  Software 119
 Anti-debugger Example
inst 1 inst 2 inst 3 inst
junk4 inst 5 inst 6 …

 Suppose inst 1 overwrites inst 4 in memory

 Then program (without debugger) will be OK
since it fetched inst 4 at same time as inst 1
 Debugger will be confused when it reaches
junk where inst 4 is supposed to be
 Problem if this segment of code executed
more than once!
o Also, self-modifying code is platform-dependent
 Again, clever attacker can figure this out
Part 4  Software 120
Tamper-resistance
 Goal is to make patching more difficult
 Code can hash parts of itself
 If tampering occurs, hash check fails
 Research has shown, can get good coverage of code with
small performance penalty
 But don’t want all checks to look similar
o Or else easy for attacker to remove checks
 This approach sometimes called “guards”

Part 4  Software 121

 Code Obfuscation
 Goal is to make code hard to understand
o Opposite of good software engineering
o Spaghetti code is a good example
 Much research into more robust obfuscation
o Example: opaque predicate
int x,y
:
if((xy)(xy) > (xx2xy+yy)){…}
o The if() conditional is always false
 Attacker wastes time analyzing dead code
Part 4  Software 122
 Code Obfuscation
 Code obfuscation sometimes promoted as a
powerful security technique
 Diffie and Hellman’s original idea for public
key crypto was based on code obfuscation
o But did not prove to be useful approach
 It has been shown that obfuscation probably
cannot provide strong, crypto-like security
o On the (im)possibility of obfuscating programs
 Obfuscation might still have practical uses
o Even if it can never be as strong as crypto

Part 4  Software 123

 Obfuscation
 Obfuscation forces attacker to analyze
larger amounts of code
 Method could be combined with
o Anti-disassembly techniques
o Anti-debugging techniques
o Code tamper-checking
 All of these increase work/pain for attacker
 But a persistent attacker can ultimately win

Part 4  Software 124

 Secure Software
Development

Part 4  Software 125

Penetrate and Patch

 Usual approach to software development

o Develop product as quickly as possible
o Release it without adequate testing
o Patch the code as flaws are discovered
 In security, this is “penetrate and patch”
o A bad approach to software development
o An even worse approach to secure software!

Part 4  Software 126

Why Penetrate and Patch?
 First to market advantage
o First to market likely to become market leader
o Market leader has huge advantage in software
o Users find it safer to “follow the leader”
o Boss won’t complain if your system has a flaw,
as long as everybody else has same flaw…
o User can ask more people for support, etc.
 Sometimes called “network economics”

Part 4  Software 127

Why Penetrate and Patch?
 Secure software development is hard
o Costly and time consuming development
o Costly and time consuming testing
o Cheaper to let customers do the work!
 No serious economic disincentive
o Even if software flaw causes major losses, the
software vendor is not liable
o Is any other product sold this way?
o Would it matter if vendors were legally liable?

Part 4  Software 128

Penetrate and Patch Fallacy
 Fallacy: If you keep patching software,
eventually it will be secure
 Why is this a fallacy?
 Empirical evidence to the contrary
 Patches often add new flaws
 Software is a moving target: new versions,
features, changing environment, new uses,…

Part 4  Software 129

Open vs Closed Source

 Open source software

o The source code is available to user
o For example, Linux
 Closed source
o The source code is not available to user
o For example, Windows
 What are the security implications?

Part 4  Software 130

 Open Source Security
 Claimed advantages of open source is
o More eyeballs: more people looking at the code
should imply fewer flaws
o A variant on Kerchoffs Principle
 Is this valid?
o How many “eyeballs” looking for security flaws?
o How many “eyeballs” focused on boring parts?
o How many “eyeballs” belong to security experts?
o Attackers can also look for flaws!
o Evil coder might be able to insert a flaw

Part 4  Software 131

Open Source Security
 Open source example: wu-ftp
o About 8,000 lines of code
o A security-critical application
o Was deployed and widely used
o After 10 years, serious security flaws discovered!
 More generally, open source software has
done little to reduce security flaws
 Why?
o Open source follows penetrate and patch model!

Part 4  Software 132

Closed Source Security
 Claimed advantage of closed source
o Security flaws not as visible to attacker
o This is a form of “security by obscurity”
 Is this valid?
o Many exploits do not require source code
o Possible to analyze closed source code…
o …though it is a lot of work!
o Is “security by obscurity” real security?

Part 4  Software 133

Open vs Closed Source
 Advocates of open source often cite the
Microsoft fallacy which states
1. Microsoft makes bad software
2. Microsoft software is closed source
3. Therefore all closed source software is bad
 Why is this a fallacy?
o Not logically correct
o More relevant is the fact that Microsoft
follows the penetrate and patch model

Part 4  Software 134

 Open vs Closed Source
 No obvious security advantage to
either open or closed source
 More significant than open vs closed
source is software development
practices
 Both open and closed source follow the
“penetrate and patch” model

Part 4  Software 135

 Open vs Closed Source
 If there is no security difference, why is
Microsoft software attacked so often?
o Microsoft is a big target!
o Attacker wants most “bang for the buck”
 Few exploits against Mac OS X
o Not because OS X is inherently more secure
o An OS X attack would do less damage
o Would bring less “glory” to attacker
 Next, we consider the theoretical
differences
o See this paper

Part 4  Software 136

Security and Testing
 Can be shown that probability of a security
failure after t units of testing is about
E = K/t where K is a constant
 This approximation holds over large range of t
 Then the “mean time between failures” is

MTBF = t/K
 The good news: security improves with testing
 The bad news: security only improves linearly
with testing!
Part 4  Software 137
 Security and Testing
 The “mean time between failures” is
approximately
MTBF = t/K
 To have 1,000,000 hours between security
failures, must test 1,000,000 hours!

Part 4  Software 138

Security and Testing
 The fundamental problem
o Good guys must find (almost) all flaws
o Bad guy only needs 1 (exploitable) flaw
 Software reliability far more difficult in security
than elsewhere
 How much more difficult?
o See the next slide…

Part 4  Software 139

 Security Testing: Do the Math
 Recall that MTBF = t/K
 Suppose 106 security flaws in some software
o Say, Windows XP
 Suppose each bug has MTBF of 109 hours
 Expect to find 1 bug for every 103 hours testing
 Good guys spend 107 hours testing: find 104 bugs
o Good guys have found 1% of all the bugs
 Trudy spends 103 hours of testing: finds 1 bug
 Chance good guys found Trudy’s bug is only 1% !!!
Part 4  Software 140
 Secure Software Development
 Goal: move away from “penetrate and patch”
 Penetrate and patch will always exist
o But if more care taken in development, then
fewer and less severe flaws to patch
 Secure software development not easy
 Much more time and effort required thru
entire development process
 Today, little economic incentive for this!

Part 4  Software 141

 Security Testing: The
Bottom Line
 Security testing is far more demanding
than non-security testing
 Non-security testing  does system do
what it is supposed to?
 Security testing  does system do what it
is supposed to and nothing more?
 Usually impossible to do exhaustive testing
 How much testing is enough?

Part 4  Software 142

 Security Testing: The
Bottom Line
 How much testing is enough?
 Recall MTBF = t/K
 Seems to imply testing is nearly hopeless!
 But there is some hope…
o If we eliminate an entire class of flaws then
statistical model breaks down
o For example, if a single test (or a few tests)
find all buffer overflows

Part 4  Software 143

 Software Security
 First to market advantage
o Also known as “network economics”
o Security suffers as a result
o Little economic incentive for secure software!
 Penetrate and patch
o Fix code as security flaws are found
o Fix can result in worse problems
o Mostly done after code delivered
 Proper development can reduce flaws
o But costly and time-consuming
Part 4  Software 144
 Software and Security
 Even with best development practices,
security flaws will still exist
 Absolute security is (almost) never possible
 So, it is not surprising that absolute
software security is impossible
 The goal is to minimize and manage risks of
software flaws
 Do not expect dramatic improvements in
consumer software security anytime soon!

Part 4  Software 145

Software Summary
 Software flaws
o Buffer overflow
o Race conditions
o Incomplete mediation
 Malware
o Viruses, worms, etc.
 Other software-based attacks

Part 4  Software 146

Software Summary
 Software Reverse Engineering (SRE)
 Secure software development
o Penetrate and patch
o Open vs closed source
o Testing

Part 4  Software 147

 Course Summary
 Crypto
o Symmetric key, public key, hash functions,
cryptanalysis
 Access Control
o Authentication, authorization
 Security Protocols
o Simple auth., SSL, IPSec, Kerberos, GSM, etc.
 Software
o Flaws, malware, SRE, Software development

Part 4  Software 148