Computer Organization and Architecture

Characteristics of Memory Systems

Chapter 4
Cache Memory

Note: Appendix 4A will not be covered in class, but

the material is interesting reading and may be used in
some homework problems.

Location Capacity
• CPU • Word size
— Registers and control unit memory — The natural unit of organisation
• Internal — Typically number of bits used to represent an
— Main memory and cache integer in the processor

• External • Number of words

— Storage devices (paper tape, cards, tapes, discs, — Most memory sizes are now expressed in bytes
flash cards, etc.) — Most modern processors have byte-addressable
memory but some have word addressable memory
— Memory capacity for A address lines is 2A
addressable units

Unit of Transfer Access Methods (1)

• Internal • Sequential
— Usually governed by data bus width — Start at the beginning and read through in order
• External — Access time depends on location of data and
— Usually a block which is much larger than a word previous location
(typical disk 512 - 4096 bytes) — e.g. tape
• Addressable unit • Direct
— Smallest location which can be uniquely addressed — Individual blocks have unique address
— Some systems have only word addressable memory — Access is by jumping to vicinity plus sequential
while many have byte addressable memory search
— A block or even cluster of blocks on most disks — Access time depends on location and previous
location
— e.g. disk

1
Access Methods (2) Performance
• Random • From user’s perspective the most important
— Individual addresses identify locations exactly characteristics of memory are capacity and
— Access time is independent of location or previous performance
access • Three performance parameters:
— e.g. RAM — Access time
• Associative — Cycle Time
— Data is located by a comparison with contents of a — Transfer Rate
portion of the store
• Access time (latency)
— Access time is independent of location or previous
— For RAM access time is the time between presenting an
access
address to memory and getting the data on the bus
— All memory is checked simultaneously; access time
— For other memories the largest component is
is constant
positioning the read/write mechanism
— e.g. cache

Performance Transfer rate for other memories

• Cycle Time • Tn = Ta + (n/r) where
— Primarily applied to RAM; access time + additional
time before a second access can start
• Tn =Average time to read or write N bits
— Function of memory components and system bus, not
the processor • Ta =Average access time
• n = number of bits
• Transfer Rate – the rate at which data can be • r = transfer rate in bits / second
transferred into or out of a memory unit
— For RAM TR = 1 / (cycle time)

Physical Types of Memory Physical Characteristics

• Semiconductor • Volatility
— RAM (volatile or non-volatile) — Does the memory retain data in the absence of
• Magnetic Surface Memory electrical power?
— Disk & Tape • Decay
• Optical — Ranges from tiny fractions of a second (volatile
DRAM) to many years (CDs, DVDs)
— CD & DVD
• Erasable
• Others
— Can the memory be rewritten? If so, how fast? How
— Magneto-optical many erase cycles can occur?
— Bubble
• Power consumption
— Hologram

2
Organization Memory Hierarchy
• Physical arrangement of bits into words • For any memory:
• Not always obvious, e.g., interleaved memory — How fast?
(examples later) — How much?
— How expensive?
• Faster memory => greater cost per bit
• Greater capacity => smaller cost / bit
• Greater capacity => slower access
• Going down the hierarchy:
— Decreasing cost / bit
— Increasing capacity
— Increasing access time
— Decreasing frequency of access by processor

Memory Hierarchy - Diagram Memory Hierarchy

• Registers
— In CPU
• Internal or Main memory
— May include one or more levels of cache
— “RAM”
• External memory
— Backing store

Hierarchy List Locality of Reference

• Registers • Two or more levels of memory can be used to
• L1 Cache produce average access time approaching the
• L2 Cache highest level
• Main memory • The reason that this works well is called
“locality of reference”
• Disk cache
• In practice memory references (both instructions
• Magnetic Disk and data) tend to cluster
• Optical — Instructions: iterative loops and repetitive subroutine
• Tape calls
• (and we could mention punch cards, etc at the — Data: tables, arrays, etc. Memory references cluster
in short run
very bottom)

3
Cache Cache Memory Principles
• A small amount of fast memory that sits • If data sought is not present in cache, a block
between normal main memory and CPU of memory of fixed size is read into the cache
• May be located on CPU chip or module • Locality of reference makes it likely that other
• Intended to allow access speed approaching words in the same block will be accessed soon
register speed
• When processor attempts to read a word from
memory, cache is checked first

Cache and Main Memory A Simple two-level cache

• Level 1: 1000 words, 0.01s
• Level 2: 100,000 words 0.1s
• If word in L1 processor has direct access else
word copied from L2 into L1
• Av Access Time as function of hit ratio H:
H * 0.01s + (1-H)* 0.11s
• With H near 1 access time approaches 0.01s

Two-level cache performance Two-level disk access

• Principles of two-level memories can be
applied to disk as well as RAM
• A portion of main memory can be used as a disk
cache
— Allows disk writes to be clustered; largest
component of disk access time is seek time
— Dirty (modified) datamay be requested by the
program before it is even written back to disk

4
Cache/Main Memory Structure Cache view of memory
• N address lines => 2n words of memory
• Cache stores fixed length blocks of K words
• Cache views memory as an array of M blocks
where M = 2n/K
• A block of memory in cache is referred to as a
line. K is the line size
• Cache size of C blocks where C < M
(considerably)
• Each line includes a tag that identifies the
block being stored
• Tag is usually upper portion of memory address

Cache operation – overview Cache Read Operation - Flowchart

• CPU requests contents of memory location
• Check cache for this data
• If present, get from cache (fast)
• If not present, read required block from main
memory to cache
• Then deliver from cache to CPU
• Cache includes tags to identify which block of
main memory is in each cache slot

Typical Cache Organization Cache organization

• The preceding diagram illustrates a shared
connection between the processor, the cache
and the system bus (look-aside cache)

• Another way to organize this system is to

interpose the cache between the processor and
the system bus for all lines (look-through
cache)

5
Elements of Cache Design Cache Size does matter
• Addresses (logical or physical) • Cost
• Size — More cache is expensive
• Mapping Function (direct, assoociative, set associative) — Would like cost/bit to approach cost of main
• Replacement Algorithm (LRU, LFU, FIFO, random) memory
• Write Policy (write through, write back, write once) • Speed
• Line Size — But we want speed to approach cache speed for all
• Number of Caches (how many levels, unified or split) memory access
— More cache is faster (up to a point)
Note that cache design for High Performance Computing (HPC) is very
different from cache design for other computers — Checking cache for data takes time
Some HPC applications perform poorly with typical cache designs — Larger caches are slower to operate

Comparison of Cache Sizes Virtual Memory

• Almost all modern processors support virtual
memory (Ch 8)
• Virtual memory allows a program to treat its
memory space as single contiguous block that
may be considerably larger than main memory
• A memory management unit takes care of the
mapping between virtual and physical
addresses

Logical Cache Logical and Physical Cache

• A logical (virtual) cache stores virtual
addresses rather than physical addresses
• Processor addresses cache directly without
going through MMU
• Obvious advantage is that addresses do not
have to be translated by the MMU
• A not-so-obvious disadvantage is that all
processes have the same virtual address space –
a block of memory starting at 0
— The same virtual address in two processes usually
refers to different physical addresses
— So either flush cache with every context switch or
add extra bits

6
 Look-aside and Look-through Look-through cache
• Look-aside cache is parallel with main memory • Cache checked first when processor requests
• Cache and main memory both see the bus cycle data from memory
— Cache hit: processor loaded from cache, bus cycle — Hit: data loaded from cache
terminates — Miss: cache loaded from memory, then processor
— Cache miss: processor AND cache loaded from loaded from cache
memory in parallel • Pro:
• Pro: less expensive, better response to cache — Processor can run on cache while another bus
miss master uses the bus
• Con: Processor cannot access cache while • Con:
another bus master accesses memory — More expensive than look-aside, cache misses slower

Mapping Function Direct Mapping

• There are fewer cache lines than memory • Each block of main memory maps to only one cache
line
blocks so we need — i.e. if a block is in cache, it must be in one specific place
— An algorithm for mapping memory into cache lines • Mapping function is i = j modulo m
— A means to determine which memory block is in (i = j % m) where
which cache line i = cache line number
• Example elements: j = main memory block number
m = number of cache lines
— Cache of 64kByte
— Cache block of 4 bytes • Address is in two parts
– i.e. cache is 16k (214) lines of 4 bytes
• Least Significant w bits identify unique word
— 16MBytes main memory • Most Significant s bits specify one memory block
— 24 bit address (224=16M) • The MSBs are split into a cache line field r and a tag of
s-r bits (most significant)
(note: Pentium cache line = 32 bytes until Pentium 4 (128 bytes))

Direct Mapping Address Structure Direct Mapping

Tag s-r Line or Slot r Word w • Parking lot analogy: think of the cache as a
14 2
parking lot, with spaces numbered 0000-9999
8
• With a 9 digit student id, we could assign
• 24 bit address, 2 bit word identifier (4 byte block)
parking spaces based on the middle 4 digits:
• 22 bit block identifier (s)
xxx PPPP yy
— 8 bit tag (=22-14) and 14 bit slot or line • Easy to find your parking space
— Example: AB1402 tag=AB line=0500 word=2
Note: 1402 = 0001 0100 0000 0010
• Problem if another student is already there!
Remove l.s. 2 bits = 0001 0100 0000 00 = 00 0101 0000 0000 = 0500

• There are 2s blocks in memory

• No two blocks with the same line number can have the same Tag field
• Note that with memory addresses, the middle
— AC1400, 041403, C71401 … bits are used as a line number
• Check contents of cache by finding line and checking Tag — Locality of reference suggests that memory
— Line is 0500 for all of these references close in time will have the same high-
— If mem request is AB1402 tag at 0500 must = AB order bits

7
 Example
Direct Mapping Cache Organization

Direct Mapping Summary Direct Mapping Cache Line Table

• Address length = (s + w) bits where w = Cache line Main Memory blocks held
log2(block size) • 0 0, m, 2m, 3m…2s-m
• 1 1,m+1, 2m+1…2s-m+1
• Number of addressable units = 2s+w words or
• …
bytes
• m-1 m-1, 2m-1,3m-1…2s-1
• Block size = line size = 2w words or bytes
• Number of blocks in main memory • 0 000000,010000,…,FF0000
= 2s+ w/2w = 2s • 1 000004,010004,…,FF0004
• Size of line field is r bits • …
— Number of lines in cache = m = 2r • m-1 00FFFC,01FFFC,…,FFFFFC
— Size of tag = (s – r) bits
• Size of cache 2r+w bytes or words

Direct Mapping Pros & Cons Associative Mapping

• Pro • A main memory block can load into any line of
— Simple cache
— Inexpensive • Memory address is interpreted as 2 fields: tag
• Con and word
— Fixed location for given block • Tag uniquely identifies block of memory
— If a program accesses 2 blocks that map to the same • Every line’s tag is examined simultaneously for a
line repeatedly, cache misses are very high match
(thrashing)
— Cache searching gets expensive because a comparator
• Victim cache must be wired to each tag
— A solution to direct mapped cache thrashing — A comparator consists of XNOR gates (true when both
— Discarded lines are stored in a small “victim” cache inputs are true)
(4 to 16 lines) — Complexity of comparator circuits makes fully
— Victim cache is fully associative and resides associative cache expensive
between L1 and next level of memory

8
Associative Mapping Direct Mapping compared to Associative
• Because no bit field in the address specifies a
line number the cache size is not determined
by the address size
• Associative-mapped memory is also called
“content-addressable memory.”
• Items are found not by their address but by
their content
— Used extensively in routers and other network
devices
— Corresponds to associative arrays in Perl and other
languages
• Primary disadvantage is the cost of circuitry

Associative Mapping Address Structure Fully Associative Cache Organization

Word
Tag 22 bit 2 bit
• 22 bit tag stored with each 32 bit block of data
• Compare tag field with tag entry in cache to check for
hit
• Least significant 2 bits of address identify which 16 bit
word is required from 32 bit data block
• e.g.
— Address Tag Data Cache line
— FFFFFD FFFFFC 24682468 3FFF

Example
Associative Mapping
• Parking lot analogy: there are more permits than
spaces
• Any student can park in any space
• Makes full use of parking lot
— With direct mapping many spaces may be unfilled

• Note that associative mapping allows flexibility

in choice of replacement blocks when cache is
full
• Discussed below

9
Associative Mapping Summary Set Associative Mapping
• Address length = (s + w) bits where w = • A compromise that provides strengths of both
log2(block size) direct and associative approaches
• Number of addressable units = 2s+w words or • Cache is divided into a number of sets of lines
bytes • Each set contains a fixed number of lines
• Block size = line size = 2w words or bytes • A given block maps to any line in a given set
• Number of blocks in main memory determined by that block’s address
= 2s+ w/2w = 2s — e.g. Block B can be in any line of set i
• Number of lines in cache = undetermined • e.g. 2 lines per set
• Size of tag = s bits — 2-way associative mapping
— A given block can be in one of 2 lines in only one set

Set Associative Mapping Set Associative Mapping: Parking Analogy

• m=v*k • If we have 10,000 parking spaces we can divide
— Where m = number of lines in cache, v = number of them into 1000 sets of 10 spaces each
sets and k = lines/set • Still use middle digits of id to find your parking
— Lines in cache = sets * lines per set place set: xxx PPP yyy
• i = j modulo v • You have a choice of any place in your set
— Where I = set number and j = main memory block
number
• Our parking lots actually work like this, but the
— Set number = block number % number of sets
sets are fairly large: Fac/Staff; Commuter;
Resident; Visitor
• This is referred to as a “k-way” set associative
mapping
• Block Bi can be mapped only into lines of set j.

Set Associative Mapping Example K-Way Set Associative Cache Organization

• Assume 13 bit set number
• Block number in main memory is modulo 213
(0010 0000 0000 0000 = 2000h
• 000000, 002000, 004000, … map to same set

10
Set Associative Mapping Address Structure Example

Word
Tag 9 bit Set 13 bit 2 bit
• Cache control logic sees address as three fields: tag,
set and word
• Use set field to determine cache set to look in
• Compare tag field to see if we have a hit
• e.g
— Address Tag Data Set number
— 1FF 7FFC 1FF 12345678 1FFF
— 001 7FFC 001 11223344 1FFF
• Tags are much smaller than fully associative memories
and comparators for simultaneous lookup are much less
expensive

Set Associative Mapping Summary Additional Notes

• For a k-way set associative cache with v sets (each set • Where v (# sets) = m (# lines in cache) and k =
contains k lines): 1 (one line/set) then set associative mapping
— Address length = (t+d+w) bits where w = log2(block size) and d
= log2(v)
reduces to direct mapping
— Number of addressable units = 2t+d+w words or bytes • For v=1 (one set) and k=m (# sets = # lines) it
— Size of tag = t bits reduces to pure associative mapping
— Block size = line size = 2w words or bytes
— Number of blocks in main memory = 2 t+d
• 2 lines/set; v=m/2, k=2 is quite common.
— Number of lines in set = k • Significant improvement in hit ratio over direct
— Number of sets = v = 2d mapping
— Number of lines in cache = kv = k * 2d
• Four-way mapping v=m/4, k=4 provides further
Word modest improvement
Tag (t bits) Set (d bits) (w bits)

Set Associative Mapping Implementation Direct mapped implementation

• A set associative cache can be implemented as
k direct mapped caches OR as v associative
caches
• With k direct mapped caches each direct
mapped cache is referred to as a way
• The direct mapped implementation is used for
small degrees of associativity (small k) and the
associative mapped implementation for higher
degrees.

11
Associative Mapped Implementation Varying associativity over cache size

Replacement Algorithms (1)

Cache replacement algorithms Direct mapping
• When a line is read from memory it replaces • No choice
some other line already in cache • Each block only maps to one line
• Other than direct mapping, there are choices • Replace that line
for replacement algorithm

• Any given choice can result in a great speedup

for one program and slow-down for some other
program

• There is no “best choice” that works for all

programs

Replacement Algorithms (2)

Associative & Set Associative MRU replacement
• Algorithm is hardware implemented for speed • Most Recently Used (MRU) would seem to be an
• Least Recently used (LRU) assumes locality of improbable algorithm
reference so most recently used is likely to be • It is useful for some specialized caches where
used again the type of code executing is known (example:
• LRU is easy for 2-way set associative cache database index scan)
— Each line has a USE bit
— When a line is referenced, set USE bit to 1 and set
the USE bit for the other line to 0
— Replace the block whose USE bit is 0
• Implemented in fully-associative caches by
keeping list of lines
— Most recently referenced lines goes to head of list

12
Other Algorithms Write Policy
• First in first out (FIFO) • When a block of memory about to be
— replace block that has been in cache longest overwritten in cache:
— Implemented as circular queue — No problem if not modified in cache
• Least frequently used — Has to written back to main memory if modified
(dirty)
— replace block which has had fewest hits
• Random • Must not overwrite a cache block unless main
memory is up to date
— Almost as good other choices
• LRU is often favored because of ease of
hardware implementation

Problems with dirty memory Write through

• More than one device may have access to main • Simplest technique
memory • All writes go to main memory as well as cache
— I/O may address main memory directly • Multiple CPUs can monitor main memory traffic
— If word altered in cache, then main memory is to keep local (to CPU) cache up to date (cache
invalid
coherency)
— If word altered in memory, then cache is invalid
• Lots of memory traffic, slows down writes
• Multiple CPUs may have individual caches
— Word altered in one cache may invalidate other
caches

Write back Cache Coherency

• Updates initially made in cache only • In a bus organization with shared memory and
multiple caches coherency has to be maintained
• Update bit for cache slot is set when update between caches as well as cache and memory
occurs
• Possible approaches:
• If block is to be replaced, write to main 1. Bus watching with write through. Cache controller monitors
memory only if update bit is set bus lines and detects writes to memory in cache. Requires
write-through policy for ALL cache controllers
— Other caches can get out of sync 2. Hardware transparency. Extra hardware ensures that a write
• I/O must access main memory through cache to one cache updates memory and all other caches
3. Noncacheable memory. Memory shared between processors is
• N.B. Typically 15% of memory references are designated as non-cacheable. All accesses to shared memory
writes; but can be as high as 50% in some HPC are cache misses. Mem identified with chip-select logic or
high address bits
apps
• More info in Ch. 18

13
Line Size Line Size
• When a cache line is filled it normally includes • Relationship between block size and hit ratio is
more than the requested data – some adjacent complex and program-dependent
words are retrieved • No optimal formula exists
• As block size increases, cache hit ratio will also • General purpose computing uses blocks of 8 to
increase because of locality of reference – to a 64 bytes
limit • In HPC 64 and 128 byte lines are most common
• If block size is too large, possibility of
reference to parts of block decreases; there
are fewer blocks in cache so more chance of
block being overwritten

Number of caches: multilevel caches Multilevel Caches

• With increased logic density caches can be on • It is usually desirable to have external as well
same chip as processor as internal cache
• Reduces external bus activity and speeds up • With only 1 level bus access to memory is slow
execution times • Most contemporary computers have at least 2
• No bus cycles; shorter data path is faster than levels
0-wait bus cycles — Internal: Level 1 (L1)
• Bus is free to do other transfers — External: Level 2 (L2)
• External L2 cache typically built with fast
SRAM; uses separate and faster data bus
• Now incorporated on processor chip

L2 and L3 Cache Unified and Split Caches

• Performance improvements depend on hit rates • Split caches have separate caches for
• Complicates replacement algorithms and write instructions and data
policy — These tend to be stored in different areas of
memory
• With L2 cache on-board L3 cache can improve
performance just as L2 can improve over L1 • Pros of unified cache:
alone — Higher rate for given cache size because cache is
automatically balanced between instructions and
data
— Only one cache needs to implemented

14
Split Cache Pentium Cache Evolution
• Current trend favors split caches • 80386 – no on chip cache
— Useful for superscalar machines with parallel • 80486 – 8k using 16 byte lines and four way set
associative organization
execution of instructions and prefetching of
predicted instructions • Pentium (all versions) – two on chip L1 caches
— Data & instructions
— Split cache eliminates contention for cache between
• Pentium III – L3 cache added off chip
instruction fetch/decode unit and the execution
• Pentium 4
unit (when accessing data)
— L1 caches
— Helps to keep pipeline full because the EU will block – 8k bytes
the fetch/decode unit otherwise – 64 byte lines
– four way set associative
— L2 cache
– Feeding both L1 caches
– 256k
– 128 byte lines
– 8 way set associative
— L3 cache on chip

Pentium Cache Evolution Pentium 4 Block Diagram

Pentium 4 Core Processor Pentium 4 Design Reasoning

• Fetch/Decode Unit • Decodes instructions into RISC like micro-ops before L1 cache
— Fetches instructions from L2 cache • Micro-ops fixed length
— Decode into micro-ops — Superscalar pipelining and scheduling
— Store micro-ops in L1 cache • Pentium instructions long & complex
• Performance improved by separating decoding from scheduling &
• Out of order execution logic pipelining
— Schedules micro-ops — (More later – ch14)
— Based on data dependence and resources • Data cache is write back
— May speculatively execute — Can be configured to write through
• Execution units • L1 cache controlled by 2 bits in register
— Execute micro-ops — CD = cache disable
— Data from L1 cache — NW = not write through
— Results in registers — 2 instructions to invalidate (flush) cache and write back then
invalidate
• Memory subsystem • L2 and L3 8-way set-associative
— L2 cache and systems bus — Line size 128 bytes

15
Pentium 4 Cache Operating Modes ARM Cache Organization
• ARM3 started with 4KB of cache
• ARM design emphasis on few transistors and
small, low-power chips has kept cache fairly
small

ARM Cache Features Write Buffer

• Distinctive feature of ARM cache is a FIFO write
buffer between cache and main memory
• When data is written to a bufferable area of
memory, data are placed in write buffer at
CPU clock speed and CPU continues execution
• Write buffer performs memory write in parallel
with processor
• If write buffer is full then CPU is stalled until
write buffer drains
• Data from same addresses as write buffer
cannot be read until write is complete

16