Computer Organization and
Architecture (AT70.01)
COD Ch. 7
Large and Fast: Exploiting
Memory Hierarchy
Memories: Review
◼ DRAM (Dynamic Random Access Memory):
◼ value is stored as a charge on capacitor that must be periodically
refreshed, which is why it is called dynamic
◼ very small – 1 transistor per bit – but factor of 5 to 10 slower than
SRAM
◼ used for main memory
◼ SRAM (Static Random Access Memory):
◼ value is stored on a pair of inverting gates that will exist indefinitely
as long as there is power, which is why it is called static
◼ very fast but takes up more space than DRAM – 4 to 6 transistors
per bit
◼ used for cache
basic structure of memory hierarchy
Speed CPU Size Cost ($/bit)
Fastest Memory Smallest Highest
Memory
Slowest Memory Biggest Lowest
Memory Hierarchy
◼ Users want large and fast memories…
◼ expensive and they don’t like to pay…
◼ Make it seem like they have what they want…
◼ memory hierarchy
◼ hierarchy is inclusive, every level is subset of lower level
◼ performance depends on hit rates
Processor
Block of data
(unit of data copy)
Data are transferred
CPU
Increasing distance
Level 1
from the CPU in
access time
Levels in the Level 2
memory hierarchy
Level n
Size of the memory at each level
Locality
◼ Locality is a principle that makes having a memory hierarchy a
good idea
◼ If an item is referenced then because of
◼ temporal locality: it will tend to be again referenced soon
◼ spatial locality: nearby items will tend to be referenced soon
Hit and Miss
◼ Focus on any two adjacent levels – called, upper (closer to CPU)
and lower (farther from CPU) – in the memory hierarchy,
because each block copy is always between two adjacent levels
◼ Terminology:
◼ block: minimum unit of data to move between levels
◼ hit: data requested is in upper level
◼ miss: data requested is not in upper level
◼ hit rate: fraction of memory accesses that are hits (i.e., found at
upper level)
◼ miss rate: fraction of memory accesses that are not hits
◼ miss rate = 1 – hit rate
◼ hit time: time to determine if the access is indeed a hit + time to
access and deliver the data from the upper level to the CPU
◼ miss penalty: time to determine if the access is a miss + time to
replace block at upper level with corresponding block at lower level
+ time to deliver the block to the CPU
Caches
◼ By simple example
◼ assume block size = one word of data
X4 X4
X1 X1
Reference to Xn
causes miss so
Xn – 2 Xn – 2
it is fetched from
memory
Xn – 1 Xn – 1
X2 X2
Xn
X3 X3
a. Before the reference to Xn b. After the reference to Xn
◼ Issues:
◼ how do we know if a data item is in the cache?
◼ if it is, how do we find it?
◼ if not, what do we do?
◼ Solution depends on cache addressing scheme…
Direct Mapped Cache
◼ Addressing scheme in direct mapped cache:
◼ cache block address = memory block address mod cache size (unique)
◼ if cache size = 2m, cache address = lower m bits of n-bit memory address
◼ remaining upper n-m bits kept kept as tag bits at each cache block
also need a valid bit torecognize valid entry
Cache
000
001
010
011
111
100
101
110
Like for 8 block= 23
So lower 3 bits used for cache adress
00001 00101 01001 01101 10001 10101 11001 11101
Memory
Direct Mapped Cache
Address showing
Address (showing bit positions
bit positions)
31 30 13 12 11 210
Byte
offset
20 10
Hit Data
Tag
Index Address is 32 bit
Cache has 2^10=1024 words
So 10 bit are used to index the
Index Valid Tag Data
0
1
2
[Link] 32-10-2=20 bits
compared with the tag
If tag and 20 bit matched=hit ,if not
1021
miss
1022
1023
20 32
Cache Read Hit/Miss
◼ Cache read hit: no action needed
◼ Instruction cache read miss:
1. Send original PC value to memory
2. Instruct main memory to perform read and wait for memory to
complete access – stall on read
3. After read completes write cache entry
4. Restart instruction execution at first step to refetch instruction
◼ Data cache read miss:
◼ Similar to instruction cache miss
◼ To reduce data miss penalty allow processor to execute
instructions while waiting for the read to complete until the word
is required – stall on use (why won’t this work for instruction
misses?)
Cache Write Hit/Miss
◼ Write-through scheme
◼ on write hit: replace data in cache and memory with every write hit
to avoid inconsistency
◼ on write miss: write the word into cache and memory – obviously
no need to read missed word from memory!
◼ Write-through is slow because of always required memory write
◼ performance is improved with a write buffer where words are stored
while waiting to be written to memory – processor can continue
execution until write buffer is full
◼ when a word in the write buffer completes writing into main that buffer
slot is freed and becomes available for future writes
◼ DEC 3100 write buffer has 4 words
◼ Write-back scheme
◼ write the data block only into the cache and write-back the block to
main only when it is replaced in cache
◼ more efficient than write-through, more complex to implement
Direct Mapped Cache: Taking
Advantage of Spatial Locality
◼ Cache replacement in large (multiword) blocks:
◼ word read miss: read entire block from main memory
◼ word write miss: cannot simply write word and tag! Why?!
◼ writing in a write-through cache:
◼ if write hit, i.e., tag of requested address and and cache entry are
equal, continue as for 1-word blocks by replacing word and writing
block to both cache and memory
◼ if write miss, i.e., tags are unequal, fetch block from memory, replace
word that caused miss, and write block to both cache and memory
◼ therefore, unlike case of 1-word blocks, a write miss with a multiword
block causes a memory read
Decreasing Miss Rates with
Associative Block Placment
◼ Direct mapped: one unique cache location for each memory block
◼ cache block address = memory block address mod cache size
◼ Fully associative: each memory block can locate anywhere in cache
◼ all cache entries are searched (in parallel) to locate block
◼ Set associative: each memory block can place in a unique set of
cache locations – if the set is of size n it is n-way set-associative
◼ cache set address = memory block address mod number of sets in
cache
◼ all cache entries in the corresponding set are searched (in parallel) to
locate block
◼ Increasing degree of associativity
◼ reduces miss rate
◼ increases hit time because of the parallel search and then fetch
Decreasing Miss Rates with
Associative Block Placment
Direct Mapped
Direct mapped
2-way Set Associative
Set associative
Fully Associative
Fully associative
Block # 0 1 2 3 4 5 6 7 Set # 0 1 2 3
Data Data Data
12 mod 8 = 4 12 mod 4 = 0
1 1 1
Tag Tag Tag
2 2 2
Search Search Search
Location of a memory block with address 12 in a cache with 8 blocks
with different degrees of associativity
Decreasing Miss Penalty with
Multilevel Caches
◼ Add a second-level cache
◼ primary cache is on the same chip as the processor
◼ use SRAMs to add a second-level cache, sometimes off-chip,
between main memory and the first-level cache
◼ if miss occurs in primary cache second-level cache is accessed
◼ if data is found in second-level cache miss penalty is access time of
second-level cache which is much less than main memory access
time
◼ if miss occurs again at second-level then main memory access is
required and large miss penalty is incurred
◼ Design considerations using two levels of caches:
◼ try and optimize the hit time on the 1st level cache to reduce clock
cycle
◼ try and optimize the miss rate on the 2nd level cache to reduce
memory access penalties
◼ In other words, 2nd level allows 1st level to go for speed without
“worrying” about failure…
Virtual Memory
◼ Motivation: main memory acts as cache for secondary storage,
e.g., magnetic disk
◼ Virtual address space, i.e., space addressable by a program is
determined by ISA
◼ e.g., 64-bit MIPS address space size is 264 – recall jr instruction
◼ typically: main memory size disk size virtual address space size
◼ Program can “pretend” it has main memory of the size of the disk
– which is smaller than thevirtual memory (= whole virtual
address space), but bigger than the actual physical memory
(=DRAM main memory)
◼ Page table (as we shall see) transparently converts a virtual memory
address to a physical memory address, if the data is already in main;
if not, it issues call to OS to fetch the data from disk into main
◼ Virtual memory is organized in fixed-size (power of 2, typically at
least 4 KB) blocks, called pages. Physical memory is also
considered a collection of pages of the same size.
◼ the unit of data transfer between disk and physical memory is a page
Virtual Memory
Page
Virtual Address
Virtual addresses Physical Address
Physical addresses
Address translation
Main Memory
Virtual
Memory
Disk addresses
Secondary Storage
Mapping of pages from a virtual address to a
physical address or disk address
Page Faults
◼ Page fault: page is not in memory, must retrieve it from disk
◼ enormous miss penalty = millions of cycles
◼ therefore, page size should be large (e.g., 32 or 64 KB)
◼ to make one trip to disk worth a lot
◼ reducing page faults is critical
◼ LRU replacement policy – implemented approximately by setting a use
bit each time a page is accessed, and then periodically clearing all
these bits so that pages accessed in a fixed time period are known
◼ fully associative page placement – consequence of page table
◼ handle faults in software instead of hardware
◼ as software overhead is still small compared to disk access time
◼ using write-through is too expensive, so always use write-back
Resolving Page Faults using
the Page Table to Access Disk
◼ There is a data structure, either part of or auxiliary to the
page table, which records where each virtual page is stored on
disk (cylinder, sector, block, etc.)
Virtual page
number
Page table
Physical memory
Physical page or
Valid disk address
1
1
1
1
0
1
1
0
1 Disk storage
1
0
1
Page table maps virtual page to
either physical page or disk page
Modern Systems
C h a r a c te r is tic In te l P e n tiu m P r o P o w e rP C 6 0 4
V ir tu a l a d d r e s s 3 2 b its 5 2 b its
P h y s ic a l a d d r e s s 3 2 b its 3 2 b its
P a g e s iz e 4 KB, 4 M B 4 K B , s e le c ta b le , a n d 2 5 6 M B
T L B o r g a n iz a tio n A T L B fo r in s tr u c tio n s a n d a T L B fo r d a ta A T L B fo r in s tr u c tio n s a n d a T L B fo r d a ta
B o th fo u r - w a y s e t a s s o c ia tiv e B o th tw o - w a y s e t a s s o c ia tiv e
P s e u d o - L R U r e p la c e m e n t L R U r e p la c e m e n t
In s tr u c tio n T L B : 3 2 e n tr ie s In s tr u c tio n T L B : 1 2 8 e n tr ie s
D a ta T L B : 6 4 e n tr ie s D a ta T L B : 1 2 8 e n tr ie s
T L B m is s e s h a n d le d in h a r d w a r e T L B m is s e s h a n d le d in h a r d w a r e
Ch a
ra c
ter i
sti
c I
n t
elP ent
ium Pr
o PowerPC 6
0 4
C
acheor g
aniz
a t
ion S
plitinst
ructionand d
a tac
a c
hesSplitint
ruc
tio
n and data c
ac
h e
s
C
achesize 8
K B each f
o rin
st
ru c
t
io n
s/
dat
a 1 6 K B eac
h f
orinstru
c t
ion
s/
dat
a
C
acheas s
ocia
tivit
y F
ou r-way seta s
socia
tive F
ou r-waysetassocia
tive
R
eplac
em en
t A
pp roximate dLRU repla
cement L
RU repla
cem e
n t
B
locks
ize 3
2 b yt
e s 3
2 b yt
es
Wr
it
e p
olic
y Wr
ite-bac k Wr
ite-backorwrit
e-th
ro ug
h
Some Issues
◼ Processor speeds continue to increase very fast
◼ much faster than either DRAM or disk access times
◼ Design challenge: dealing with this growing disparity
◼ Trends:
◼ synchronous SRAMs (provide a burst of data)
◼ redesign DRAM chips to provide higher bandwidth or processing
◼ restructure code to increase locality
◼ use pre-fetching (make cache visible to ISA)
