Software Security

Low level Security

(Memory Attacks)
Edited By: Areej Althubaity

Course Material inspired from several Universities

(Maryland, Mit, etc )

1
Acknoledgment
} Slides are adapted from Dr. Akrami
} Also, other resources avilable online and I have cited them within
slides

2
Memory layout
Memory Layout Refresher
} How is program data laid out in memory?

} What does the stack look like?

} What effect does calling (and returning from) a function have on

memory?

} We are focusing on the Linux process model running on an Intel

x86, 32, or 64 bit processor
} Similar to other operating systems

4
All programs are stored in memory

} During program execution, the OS allocates

a specific memory space for the program to
store its data and instructions.
} This memory space can be visualized as a
continuous block of memory, where each
individual byte has a unique address.
} The size of the address space depends on
your operating system and CPU https://www.cs.virginia.edu/~evans/cs216/guides/x86.html

architecture.
} In a 32-bit system, memory addresses are
32 bits long, which means the address space
has 232 bytes of memory.

5
All programs are stored in memory
4G 0xffffffff

In reality, these are

The process’s view virtual addresses;
of memory is that the OS/CPU map
it owns all of it them to physical
addresses
The first element in
this array is located at
address 0x00000000, 0 0x00000000
while the last element
is at address
0xFFFFFFFF.

6
All programs are stored in memory
} During program execution, the memory space is
divided into four main sections:
C Memory
1. Text/Code Section: This section stores the
program's executable instructions, which are the
machine code generated by the compiler and linker.
2. Static Section: This section holds constant values,
static variables, and global variables. These variables
have a fixed memory location throughout the
program's execution.
3. Heap: This section is used for dynamic memory
allocation. When a program requests memory using
functions like malloc in C, it is allocated from the heap.
The heap grows upwards in memory, meaning newer
allocations have higher addresses.
4. Stack: This section is used for storing local variables https://textbook.cs161.org/memory-safety/x86.html
and function call information. The stack grows
downwards in memory, with older function calls having
higher addresses.

7
The instructions themselves are in
memory

4G 0xffffffff

Text
0 0x00000000

8
Location of data areas
4G 0xffffffff
Set when
process starts cmdline & env
Stack int f() {
int x;
…
Runtime

Heap malloc(sizeof(long));
Uninit’d data static int x;
Known at Init’d data static const int y=10;
compile time
Text
0 0x00000000

9
Memory allocation
Stack and heap grow in opposite directions
As more memory is needed in the heap,
it grows towards the higher addresses.
Where as more memory is needed for
the stack, is grows downward
0x00000000 toward the lower address. 0xffffffff
Heap Stack

push 1
Stack push 2
pointer push 3

10
Memory allocation
Stack and heap grow in opposite directions

0x00000000 0xffffffff
Heap 1 Stack

push 1
Stack push 2
pointer push 3

11
Memory allocation
Stack and heap grow in opposite directions

0x00000000 0xffffffff
Heap 2 1 Stack

push 1
Stack push 2
pointer push 3

12
Memory allocation

Compiler emits instructions adjust

the size of the stack at run-time

0x00000000 0xffffffff
Heap 3 2 1 Stack

push 1
apportioned by the OS;
managed in-process Stack push 2
pointer push 3
by malloc
return

Focusing on the stack for now

13
 Registers

} Registers hold data that can be

readily accessed by the CPU.
} A register is a special state
machine that stores multiple bits
of data.
} Modern x86 processors, have
eight 32-bit general-purpose
registers.
} The registers are like variables
built in the processor.
} Using registers instead of
memory to store values makes
the process faster and cleaner.
14
14
 Registers

} These registers, while historically named for

specific functions (like the accumulator or
counter), now serve a more general
purpose.
} Each register can hold any 32-bit integer
value.
} Registers do not have addresses. Instead, we
refer to registers using names.

https://www.cs.virginia.edu/~evans/cs216/guides/x86.html

15
15
 Registers

} There are three special x86 registers

(important for our topic for today!)
• EIP (Instruction Pointer): This register
stores the memory address of the next
machine instruction to be executed. Also,
known as the PC (Program Counter).
• EBP (Base Pointer): This register points
to the top of the current stack frame. Also,
known as the FP (Frame Pointer).
• ESP (Stack Pointer): This register points
to the bottom of the current stack frame.
Also, called the SP (Stack Pointer).

What does the “E” stands for? https://www.cs.virginia.edu/~evans/cs216/guides/x86.html

16
16
Registers and Memory
} Note that the top of the current stack frame is the highest address
associated with the current stack frame, and the bottom of the stack frame
is the lowest address associated with the current stack frame.
} The values stored in these three registers, EIP, EBP, and ESP, typically
represent memory addresses.
} When we say a register "points" to a memory location, we mean that the
address stored within the register refers to a specific memory address.
} For example, if EIP points to 0xDEADBEEF, it indicates that the value
0xDEADBEEF is stored in the EIP register, and this value can be interpreted
as an address to access a particular memory location.

17
Registers and Memory

Which section of C memory (code, static, heap, stack)

do eip, ebp and esp registers usually point to?

18
Pushing and Popping Stack
} To save a value in the stack:
1) Allocating space on the stack by decreasing the ESP register, and
2) Storing the value in the newly allocated space.
} The x86 push instruction simplifies this process by performing
both steps in a single operation.

19
Pushing and Popping Stack
} To remove a value in the stack, we use the x86 pop instruction.
} This instruction performs two actions:
1) It increments the ESP register to remove the top value from the stack, and
2) It copies the removed value into a specified register.
} It's important to note that once a value is popped from the stack, it is effectively erased from
memory.
} However, by incrementing the ESP register, the popped value is now below the current stack
pointer and is considered to be in undefined memory.

20
Stack and function calls
} How the program use the stack while it is running?

} What happens when we call a function?

} What data needs to be stored?
} Where does it go?
} What happens when we return from a function?
} What data needs to be restored?
} Where does it come from?

21
Function Calls
} When a function is invoked, the stack allocates additional memory to store
local variables and other function-specific data.
} This allocated space, located at lower memory addresses, is known as a
stack frame.
} Once the function completes, this space is released, making it available for
future function calls.
} A function call involves the caller and the callee.
} The caller initiates the function call, and the program's execution
temporarily shifts to the callee.
} Once the callee finishes its execution, control is returned back to the caller,
allowing it to resume its execution.

22
Function Calls

} When making a function call in x86, we need to adjust the values of three
key registers:
1. EIP (Instruction Pointer): Initially pointing to the caller's instructions,
EIP must be updated to point to the first instruction of the callee.
2. EBP (Base Pointer): EBP, which points to the top of the caller's stack
frame, needs to be updated to point to the top of the newly created stack
frame for the callee.
3. ESP (Stack Pointer): Similarly, ESP, which points to the bottom of the
caller's stack frame, must be adjusted to point to the bottom of the callee's
stack frame.

23
Function Calls

} When the functions return, we need to preserve the original values of the
registers before the function call.
} This involves saving these values onto the stack. Once the function finishes,
these saved values are restored to their original state, allowing the program
to resume execution from where it left off in the caller function.
} The complete process of calling a function and returning involves 11 steps.
} Example: main is the caller function and foo is the callee function.
int main(void) {
foo(1, 2);
}

void foo(int a, int b) {

int bar[4];
}

24
Function Calls
• The stack before the function foo is called.
• ebp and esp point to the top and bottom of the caller stack frame.

25
Function Calls

1. Push arguments onto the stack.

• To pass arguments in x86, we push them

onto the stack in reverse order.
• This means the first argument goes on the
stack first, followed by the second
argument, and so on.
• As each argument is pushed, the ESP
register is decremented to allocate space
on the stack.

26
Function Calls

2. Push the old eip (rip) on the stack.

• Before modifying the EIP register to point

to the callee's instructions, we must save
its current value on the stack.
• This saved value is often referred to as the
"old EIP" or "RIP" (Return Instruction
Pointer).

27
Function Calls

3. Update eip.

• We change eip to point to the

start of the code of the callee
function.

28
Function Calls
4. Push the old ebp (sfp) on the stack.

• We also need to save the current value of

the EBP register before modifying it.
• This saved value is known as the "old EBP"
or "SFP" (Saved Frame Pointer).
• Since we've pushed a new value onto the
stack, the ESP register has been
decremented.

29
Function Calls
5. Move ebp down.

• With the old EBP value safely stored, we

can now update the EBP register to point
to the top of the new stack frame.
• This new top is currently indicated by the
ESP register, as we are about to allocate
additional space below it for the callee's
stack frame.

30
Function Calls
6. Move esp down.

• We'll allocate space for the new stack

frame by decrementing the ESP register.
• The amount of space required depends
on the function's complexity, specifically
the number and size of its local variables.
• A function with fewer local variables will
need less stack space, resulting in a
smaller decrement of ESP.
• Conversely, a function with large local
variables, such as arrays, will require a
larger decrement of ESP to accommodate
the additional space.
• Local variables will be stores in the same
order as in the program

31
Function Calls
7. Execute the function.

• The newly allocated stack frame can now

be used to store local variables and other
relevant data for the function.
• Since EBP consistently points to the top
of the current stack frame, it can be used
as a reference point to access other
variables on the stack.
• For instance, the function arguments will
be located at the address stored in EBP,
offset by 8 bytes (8(%ebp) )

32
Function Calls
Inside a function

High Memory

sum(2,5) 5
2 To access 2= ebp + 8 bytes
int sum(int x,int y){ 4 rid
int total;
total = x + y; 4 ebp
EBP:Stack base pointer
return total;
}

ESP:Stack frame pointer

low Memory

2323 System Programming 33

33
Function Calls

High Memory

sum(2,5) 5
4 2 To access 5= ebp + 12 bytes
int sum(int x,int y){ 4 Ret Add
int total;
total = x + y; 4 EBP EBP:Stack base pointer
return total;
}

ESP:Stack frame pointer

low Memory

2323 System Programming 34

34
Function Calls

High Memory

sum(2,5) 5 Arguments
pushed in
2 reverse order
int sum(int x,int y){ Ret Add of code
int total;
EBP EBP:Stack base pointer
total = x + y;
return total; 4 0xC To access total= ebp – 4 bytes
} =-4(%ebp)
Local variables
pushed in the
same order as ESP:Stack frame pointer
they appear low Memory
in the code

2323 System Programming 35

35
Function Calls
8. Move esp up.

• When the function is ready to return, we

increment the ESP register to point to the
top of the old stack frame (where EBP is
pointing).
• This action effectively removes the
current stack frame, as the memory
below ESP is now considered undefined.

36
Function Calls
9. Restore the old ebp (sfp).

• The next value on the stack is the saved

frame pointer (SFP), which is the original
value of EBP before the function call.
• We pop this value from the stack and
store it back into the EBP register,
effectively restoring EBP to its pre-
function state.

37
Function Calls
10. Restore the old eip (rip).

• The next value on the stack is the return

instruction pointer (RIP), which is the
original value of EIP before the function
call.
• We pop this value from the stack and
store it back into the EIP register,
effectively returning program execution to
the instruction following the function call.

38
Function Calls
11. Remove arguments from the stack.

• Since the function call has completed, the

arguments passed to the function are no
longer needed.
• We can remove them from the stack by
incrementing the ESP register. Remember,
any memory location below the ESP is
considered undefined.

39
Function Calls

• You might notice that we saved the old values of EIP and EBP during the
function call, but not the old value of ESP.
• This is because the ESP register naturally adjusts itself as values are pushed
onto and popped off the stack.
• As we push arguments and local variables onto the stack, ESP is
decremented.
• Conversely, as we pop values off the stack, ESP is incremented.
• This automatic adjustment of ESP eliminates the need to save its old value
before the function call.

40
 Basic stack layout
void func(char *arg1, int arg2, int arg3)
{
char loc1[4]
int loc2;
...
}

0xffffffff
… loc2 loc1 ??? ??? arg1 arg2 arg3 caller’s data
Local variables Arguments
pushed in the pushed in
same order as reverse order
they appear of code
in the code

The local variable allocation is ultimately up to the compiler: Variables could be allocated in any
order, or not allocated at all and stored only in registers,
depending on the optimization level used.

41
Accessing variables
void func(char *arg1, int arg2, int arg3)
{
... Q: Where is (this) loc2?
loc2++;
...
}

0xffffffff
… loc2 loc1 ??? ??? arg1 arg2 arg3 caller’s data

0xbffff323
Can’t know absolute
address at compile time
But can know the relative address
• loc2 is always 8B before ???s

42
Accessing variables
void func(char *arg1, int arg2, int arg3)
{
... Q: Where is (this) loc2?
loc2++;
... A: -8(%ebp)
}

0xffffffff
… loc2 loc1 ??? ??? arg1 arg2 arg3 caller’s data

%ebp Stack frame

Frame pointer for func

But can know the relative address

• loc2 is always 8B before ???s

43
Returning from functions
int main()
{
...
func(“Hey”, 10, -3);
... Q: How do we restore %ebp?
}

0xffffffff
… loc2 loc1 ??? ??? arg1 arg2 arg3 caller’s data

%ebp Stack frame

for func %ebp

44
Returning from functions
int main()
{
...
Q: How do we restore %ebp?
func(“Hey”, 10, -3);
...
}

%esp
0xffffffff
Old
??? arg1 arg2 arg3 caller’s data
%ebp

Stack frame
%ebp Old %ebp
for func

Push %ebp before locals

Set %ebp to current (%esp)
Set %ebp to(%ebp) at return

45
Returning from functions
int main()
{
...
func(“Hey”, 10, -3);
... Q: How do we resume here?
}

0xffffffff
Old
… loc2 loc1 ??? arg1 arg2 arg3 caller’s data
%ebp

Stack frame
%ebp Old %ebp
for func

46
The instructions themselves are in
memory

4G 0xffffffff

%eip
Text
0 0x00000000

47
The instructions themselves are in
memory

4G 0xffffffff

%eip

Text
0 0x00000000

48
Returning from functions
int main()
{
...
func(“Hey”, 10, -3);
... Q: How do we resume here?
}

0xffffffff
Old Old
… loc2 loc1 ??? arg1 arg2 arg3 caller’s data
%ebp %eip

Stack frame
%ebp Old %ebp
for func

Set %eip to 4(%ebp) Push next %eip

at return before call

49
Stack and functions: Summary
} Calling function:
} 1.Push arguments onto the stack (in reverse)
} 2.Push the return address, i.e., the address of the instruction you
want run after control returns to you (%eip)
} 3.Jump to the function’s address
} Called function:
} 4.Push the old frame pointer onto the stack (%ebp)
} 5.Set frame pointer (%ebp) to where the end of the stack is right
now (%esp)
} 6.Push local variables onto the stack
} Returning function:
} 7.Reset the previous stack frame: %esp = current %ebp, %ebp = (old
%ebp)
} 8.Jump back to return address: %eip = 4(%esp)
50