VM Migration

CONTENT SOURCE

• “Live Migration of Virtual Machines”, Christopher Clark, Keir Fraser, Steven Hand,
Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, Andrew Warfield
• “Post-Copy Based Live Virtual Machine Migration Using Adaptive Pre-Paging and
Dynamic Self-Ballooning”, Michael R. Hines and Kartik Gopalan
Source Contents Credit to:

• Prof. Rajkumar Buyya (University of Melbourne, Australia.)

• Prof. Mythili Vutukuru (IIT Bombay, India.)

 VM Migration

Physical Machine A Physical Machine B

VM

Source Target
Review: Virtual Machines
A virtual machine provides interface identical to underlying bare hardware
▫ i.e., all devices, interrupts, memory, page tables etc.

Virtualization Software: VMWare, Hyper-V, Oracle virtual box, Xen etc.

Ø It allows cleanseparation between hardware and software.
Ø Process level migration problems can be avoided by migrating a virtual machine.

Ø Virtualization provides facility to migrate virtual machine from one host (source)to
another physical host (destination).
Ø Virtual Machine Migration is a useful tool foradministrator of data center andclusters.
Review: Virtual Machines in Cloud
q Benefits of Virtual Machines
Virtualization helps in making efficient use of hardware resources
Facilitates a greater degree of abstraction
Easily move from one piece of hardware to another
Replicate them at will
Create more scalable and flexible infrastructure
High Availability and Fault tolerance (How?)

q Cloud computing has taken that degree of efficiency and agility realized from
virtualization
Pooled resources
Geographic diversity
Universal connectivity
7

Motivation
• Consider a data center consisting of “n” physical machines (PM) hosting “m” VMs
implementing one customer application each
• Resources(CPU, Network, Memory, I/O) are allocated to each VM to handle the
workload and operate at certain performance level (SLA)
• Each VM sees workload fluctuation (WHY?) from time to time =>
resource requirement changes Network Bandwidth = 10 Gbps
PM Memory = 16 GB
# of user visit
Capacity CPU = 8 cores
increases
VM1 VM2
VM1 VM2
cricinfo mail server Resource N = 5Gbps
N = 5Gbps
Allocation M = 8GB
Virtualization Layer M = 8GB
C = 4 cores C = 4 cores
Hardware
Motivation

• An increase in workload can be handled by allocating more resources to it, if

idle resources are available

q Main Issues:
What if PM does not have (enough or no) idle resources to satisfy VM's
requirement?
• Performance of the application degrades
• SLA violation occurs

q Key Ideas
• Replication VMs
• Migrating VMs
Virtual Machine Migration

• Why do we need migration?

• When do we need to migrate?
• How migration is done?
• Issues in long distance migration (across data
centers)
•Why do we need migration?

• To Maintain/Upgrade
• To Optimize the Performance
• To Protect SLA
• To Improve QoE
When we need to migrate? [NSDI’ 07]

• Hotspots can cause SLA violations

▫ Burden on some Virtual or Physical Machines are called hotspots

• Hotspot Detection
▫ Black-box Monitoring
CPU
Network
Memory
▫ Gray-box Monitoring
Gather OS level statistics and application logs

• A hotspot is flagged only if thresholds or SLAs are exceeded for a sustained time
When we need to migrate? [FGCS’ 12]

• SLA violation detection

▫ Mapping low-level resource metrics to high-level SLAs
CPU speed maps to Response Time
Occupied memory size maps to number of concurrent clients
▫ Predictive Strategy for detection of possible SLAviolations
▫ Detection interval
Short measurement intervals may degrade performance
Long measurement intervals may cause ignorance of heavy SLA violations.
Migration Techniques

▪ The different virtual machine migration techniques areas follows:

– FaultTolerant MigrationTechniques
– Load Balancing MigrationTechniques
– Energy Efficient MigrationTechniques
FaultTolerant Migration Techniques

Ø Fault tolerance allows theVMs to continue its jobeven any part of system fails.

Ø This technique migrates theVM from one physical serverto another physical server
based upon the prediction of the failure occurred.
Ø Fault tolerant migration technique is to improve the availability of physical server
and avoids performance degradation of applications.
Load Balancing Migration Techniques

Ø The Load balancing migration technique aims to distribute load across the physical
servers to improve the scalability of physical servers in cloud environment.
Ø The Load balancing improves the
Ø resource consumption
Ø implementation offailover
Ø enhancing scalability
Energy Efficient Migration Techniques

Ø The power consumption of Data center is mainly based on the utilization of the
servers and theircooling systems.
Ø The servers typically need up to 70 percentage of their maximum power
consumption even at theirlow utilization level.
Ø Therefore, there is a need for migration techniques that conserves the energy of servers
by optimum resourceutilization.
V M Migration Methods

▪ Virtual Machine Migration methods aredivided into two types:

– Cold (non-live)migration
– Hot (live)migration

§ The status of the VM loses and user can notice the service interruption in cold
migration. First, VM is suspended, then its state is transferred, and lastly, VM is
resumed at destination host.

§ In live migration process, the state of a VM to migrate is transferred.

§ The state consists of its CPU registers, memory contents and local file system. Local
file systems need not be transferred.
§ VM keeps running while migrating and does not lose its status. User doesn't feel any
interruption in service in hot (live) migration.
Live (Hot)VM Migration
Live Migration

§VM live migration can be a extremely powerful tool for cluster

administrators.
Hardware / Software maintenance / upgrades
Load balancing / resource management
Distributed power management
 Live Migration

• Move VM instances across distinct physical hosts with little or

no downtime for running services.
o Services are unaware of the migration.
o Maintain network connections of the guest OS.
o VM is treaded as a black box.

“Live Migration of Virtual Machines”, Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, Andrew Warfield
“Post-Copy Based Live Virtual Machine Migration Using Adaptive Pre-Paging and Dynamic Self-Ballooning”, Michael R. Hines and Kartik Gopalan
Live Migration

Migrate an entire VM from one physical host to another

• All user processes and kernel state
• Without having to shut down the machine
• VM Migration is easier than migrating processes
Live Migration

Why OS-level migration, instead of process-level?

•Avoid ‘residual dependencies’
•Original host can be power-off / sleep once migration completed.
•Can transfer in-memory state in a consistent and efficient fashion
•E.g. No reconnection for media streaming application
•Allow a separation of concerns b/w the users and operator of a cluster
•Users can fully control of the software and services within their VM.
•Operators don’t care about what’s occurring within the VM.
23

What ismigrated?

• CPU context of VM, contents of main memory

• Narrow interface, easier than process migration
• Disk: assume NAS (network attached storage) that is accessible
from both hosts, or local disk is mirrored
• We do not consider migrating disk data
• Network: assume both hosts on same LAN
• Migrate IP address, advertise new MAC address to IP mapping via ARP reply
• Migrate MAC address, let switches learn new MAC location
• Network packets redirected to new location (with transient losses)
• I/O devices are provisioned at target
• Virtual I/O devices easier to migrate.
Design-local resources

Virtual Machine Virtual Machine

Source Destination
25
Steps to migrate aVM
• Broad steps in any migration technique: Suppose we are migrating a
VM from host A to host B
1. Setup target host B, reserve resources for the VM
2. Push phase: push some memory of VM from A to B
3. Stop-and-copy: stop the VM at A, copy CPU context, and some memory
4. Pull phase: Start VM at host B, pull any further memory required from A
5. Clean up state from host A, migration complete
• Total migration time: time for steps 2,3,4
• Service downtime: time for step 3
• Other metrics: impact on application performance, network
bandwidth consumed, total pages transferred
26
Flavors of migrationtechniques

• Pure stop-and-copy: VM stopped, all state transferred to target,

VM restarts
• Too much downtime to be classified as “live” migration
• Pre-copy: most state is transferred in the push phase, followed by
a brief stop-and-copy phase
• Post-copy: VM stopped, bare minimum state required to run the
VM is transferred to the target host. Remaining state is pulled on
demand while the VM is running at the new location.
• Hybrid: a mix of pre-copy and post-copy. Some pushing of state
followed by stop-and-copy, followed by pulling of state on demand.
Design-challenges

qMinimize service downtime

qMinimize migration duration
qAvoid disrupting running service

Source Host
Destination Host
.BI
.VS
.X
.VH
V
DML
N
Storage
Design-memory migration

Phase service downtime migration duration

push - -
stop-and-copy longest shortest
pull (demand) shortest longest

§ Careful to avoid service degradation

29
Pre-copy based livemigration

• Iterative pre-copy + stop-and-copy

for remaining memory
• First push round copies all pages
• Every round copies pages dirtied in
previous round
• A page maybe copied multiple times
• Writable Working Set (WWS):
pages commonly written to
• WWS will be copied multiple times
• Finally transferred in stop-and-copy
• How many rounds?
• Stop when rate of dirtying > rate of
transfer
30
Impact of iterative pre-copy

Page dirtying rate (WWS) Downtime after successive rounds

• If stop-and-copy, 512MB VM, 128 Mbps network, downtime = 32 sec

• With one pre-copy round, downtime goes to 2-3 sec
• ~1 second for 2 or more rounds
 Tracking dirtypages Guest page table
• Xen-based implementation
• Page tables in Xen maintained by guest
• Move to shadow page tables for migration Shadow page table in Xen

• Migration managed by control software in domain0

• Shadow page table constructed on demand for every round
• Dirty bitmap maintained for every round
• Any page access by guest -> page fault to Xen, shadow page table updated
• PTE marked as read-only by default in shadow
• If valid write access, shadow PTE marked writeable, page marked dirty in bitmap
• At end of round, dirty pages are marked for transfer in control software
• Shadow page table and dirty bitmap reinitialized after every round
• Last set of dirty pages copied in stop-and-copy
• Guest page table in target host changed based on new physical addresses
33
Some optimizations

• Avoid transferring page multiple times

• Before transmitting page, peek into the current round's dirty bitmap
• Skip transmission if page is already dirtied in ongoing round
• Move non-interactive processes generating dirty pages to wait queue
• Execution paused until migration completes
• Free up page cache and other unnecessary pages
• Reduce memory footprint
• Much like ballooning
34
Pre-copyperformance

• Downtime: ~100 millisec, total migration time of few tens of seconds

• Worse for memory-intensive applications, better for interactive apps
35
Pre-copy vs Post-copy livemigration

• Avoid multiple transfers of same

page as happens in pre-copy
• Prepare target, stop VM, copy
CPU context and minimum
memory to target
• Start VM at target, pull memory
from source via demand paging
• Memory access at target causes page
fault, page fetched from source
36

Implementation details(Xen)

• How are pages pulled at target? “Pseudo-paging”

• Page to a pseudo, in-memory, swap device (part of domain0). No memory copy, just transfer pages
across domains. Guest page table updated suitably.
• Only non-pageable memory transferred during stop-and-copy
• When guest resumes at target, fetch memory from pseudo-paging device via page fault mechanism
• Special swap device driver fetches from source over the network
• Alternative: use shadow page tables
• If page fault to non-existent page at target, trap to hypervisor, fetch from source and update
Implementation-dynamic rate limiting
More network bandwidth, less service downtime !
Less network bandwidth, less impact on running service !
Dynamically adapt the bandwidth limit during each round
-Set a minimum and a maximum bandwidth limit, begin with the minimum
limit
-𝑏𝑎𝑛𝑑𝑤𝑖𝑑𝑡ℎ!"#$ = dirty 𝑟𝑎𝑡𝑒%&''()* + 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑖𝑛𝑐𝑟𝑒𝑚𝑒𝑛𝑡
-𝑑𝑖𝑟𝑡𝑦 𝑟𝑎𝑡𝑒+,--"!$ = ./-$0 123"47.,-2$/5!

When terminate push, and switch to stop-and-

copy ? downtime performance
-𝑑𝑖𝑟𝑡𝑦 𝑟𝑎𝑡𝑒!"##$%& > 𝑏𝑎𝑛𝑑𝑤𝑖𝑑𝑡ℎ'()
-𝑑𝑖𝑟𝑡𝑦 𝑝𝑎𝑔𝑒𝑠 < 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑
39
Optimizations

• Active pushing: source proactively pushes important pages, in

addition to pulling pages via page faults
• Pre-paging: a “bubble” of pages around faulted page and proactively
pushed, in anticipation of future accesses
• Dynamic self-ballooning: VM periodically frees up unnecessary
memory and gives it back to hypervisor
• Reduces memory footprint, speeds up page transfer
• Performed carefully without hurting application performance
• Can be used to optimize pre-copy migration as well
• Hybrid: one pre-copy round, followed by post copy
40

What aboutfailures?

• What if target machine fails during migration?

• Pre-copy can simply abort the migration, restart with another target
• With pre-copy, latest state is on source only, so can recover
• With post copy, source has stale memory, target has updated memory
• If target crashes during post copy, cannot recover application data (unless
some replication is performed)
41

Post copy vs pre copyperformance

• Longer downtime as compared to pre-copy, but lower total migration
time, fewer page transfers, lesser disruption to application
42

Post copy vs pre copyperformance

43
Summary

• VM live migration techniques

• Iterative pre-copy vs post-copy via demand paging
• Implementation details on Xen
• Performance comparison
• Dynamically adapting network-bandwidth
-Balance service downtime and service performance degradation
• Which is better?
• Pre-copy suited for interactive application
• Post copy is better for memory-intensive applications with large WWS
• Hybrid techniques are also used
Challenges /Concerns
India: +91-7022374614

US: 1-800-216-8930 (TOLL FREE)

[email protected]

24/7 Chat with Our Course Advisor