Virtual I/O Architecture and Performance

Li Ming Jun
[email protected]

Outline

Why Virtualization
Virtual I/O on system p5

Virtual I/O overview

Power virtualization architecture
Virtual I/O devices
Virtual I/O deployment example

Virtual I/O performance comparison test with physical I/O

IVE on Power 6 server
References and useful links

Outline

Why virtualization
Virtual I/O on system p5

Virtual I/O overview

Power virtualization architecture
Virtual I/O devices
Virtual I/O deployment example

Virtual I/O performance comparison test with physical I/O

IVE on Power 6 server
References and useful links

Industry Trends: Increasing Capacity

Network Bandwidth

Significant growth in capacity

Network bandwidth has grown 100x in less

than 10 years
Fibre Channel bandwidth has grown 4x in 5
years.
CPU performance per socket has grown
almost 4x in 4 years.

10

Bandwidth (Gb)

4
2
0
1998

Significant power and cooling requirements

The increasing microprocessor performance

per socket will both drive and enable the
growing use of virtualization

1999

2001

2003

2004

2006

Year

Relative Performance per Socket

Performance

Over allocated server and network capacity

FC (Trend)

Result


Ethernet (Trend)

3
2

System p (Trend)

1
0

2003

2004

2005

2006

2007

Year

4
4

2007 IBM Corporation

Outline

Why Virtualization
Virtual I/O on system p5

Virtual I/O overview

Power virtualization architecture
Virtual I/O devices
Virtual I/O deployment example

Virtual I/O performance comparison test with physical I/O

IVE on Power 6 server
References and useful links

IBM GCG ATS

Virtual I/O Overview

Virtual
I/O Server*
Virtual
SCSI
Function

AIX 5.3 or Linux

Virtual
Ethernet
Function

Ethernet

FC

AIX 5.3 or Linux

Ethernet

Virtual
I/O Server*
Virtual
Ethernet
Function

Virtual
SCSI
Function

Hypervisor
Ethernet
B

Virtual I/O Architecture

Benefits

Mix of virtualized and/or physical devices

Reduces adapters, I/O drawers, and ports

Multiple VIO Servers* supported

Improves speed to deployment

Virtual SCSI

Ethernet

Virtual Ethernet

Virtual SCSI, Fibre Channel, and DVD

VLAN and link aggregation support

Logical and physical volume virtual disks

LPAR to LPAR virtual LANs

Multi-path and redundancy options

High availability options

* Available on System p via the Advanced POWER virtualization features. IVM supports a single Virtual I/O Server.

6
2007 IBM Corporation

IBM GCG ATS

POWER5 Virtualization Architecture

Hypervisor
Type 1 Hypervisor
Thin Hypervisor layer No Host O/S

Hypervisor: Type 1
Apps

...

OS

Apps
OS

Paravirtualization
Architected Hypervisor calls
More efficient than trap and emulate

Hardware Assists
POWER Hypervisor mode
Hypervisor timer
Processor utilization registers
Real mode offset registers
I/O drawers and towers (isolation/performance)
SMT Dormant thread

Hypervisor

SMP Server

Hypervisor: Type 2
Apps
OS

...

Apps
OS

Hypervisor

Host OS
SMP Server

7
2007 IBM Corporation

IBM GCG ATS

POWER5 Virtualization Architecture

I/O
Multiple I/O models
Virtualized and/or nonvirtualized devices

Virtual I/O Performance

Size VIO server(s) to match
performance requirements

VIO
Server

LPAR 2

LPAR 3

Buffer

Virtual
SCSI
Server

Virtual
Disk

FC
Adapter

Read Message

Intra-partition DMA to avoid

data copies
MTU sizes up to 65280 bytes
for LPAR to LPAR traffic

LPAR 1

POWER5 Hypervisor (Firmware)

Disk

Disk

TCP/IP checksum bypass

8
2007 IBM Corporation

IBM GCG ATS

POWER Hypervisor support for virtual I/O

The POWER Hypervisor provides two classifications of virtual adapters:
Hypervisor simulated class and partition managed class.

Hypervisor simulated class.

the POWER Hypervisor simulates
an I/O adapter. This class is used in
virtual Ethernet support This
technique provides reliable and fast
communication between partitions
using network protocols..

9
2007 IBM Corporation

IBM GCG ATS

POWER Hypervisor support for virtual I/O(cont.)

Partition managed class. A server partition

provides the services of one of its physical
I/O adapters to client partition. A server
partition provides support to handle I/O
requests from the client partition and
perform those requests on one or more of
its devices, targeting the client partitions
direct memory access (DMA) buffer areas
using LRDMA facilitie. It then passes I/O
responses back to the client partition. This
classification of virtual adapter is used by
virtual SCSI.

10

10
2007 IBM Corporation

IBM GCG ATS

Virtual I/O infrastructure

The virtual I/O infrastructure provides several primitives that are used to build
connections between partitions for various purposes. These primitives include:
A Command/Response Queue (CRQ) facility that provides a pipe between
partitions. A partition can enqueue a command to the target partitions CRQ
for processing by that partition. The partition can set up the CRQ to receive
an interrupt when an entry is placed in the queue.
An extended Translation Control Entry (TCE) table called the Remote TCE
(RTCE) table, which enables a partition to provide windows to its memory for
other partitions to use, while maintaining addressing and access control to its
memory.
Remote DMA services that enable a server partition to transfer data to
another partitions memory via the RTCE windows. This enables a device
driver in a server partition to efficiently transfer data to and from another
partition. This is key to sharing a virtual I/O adapter in the server partition.

11

11
2007 IBM Corporation

IBM GCG ATS

Virtual I/O devices -VSA

The POWER Hypervisor supports three types of virtual I/O devices:
Virtual LAN (VLAN)
Virtual SCSI (VSCSI)
Virtual Serial Adapter (VSA).
The VSA can only be used for providing a virtual console to the
partitions. This console is visible to the end user in the HMC display.
The virtual serial port cannot be used for any other purpose. For
example, it cannot be used for HACMP heartbeat monitoring.
There are no specific performance considerations to address
regarding the VSA.

12

12
2007 IBM Corporation

Advanced Technical Support System p

Virtual SCSI

Virtual I/O helps reduce hardware costs by sharing disk drives

POWER5 Server
Micro-partitions

External Storage

A1 A2

A3

A4
A5

Shared
Fiber Chan
Adapter
Shared
SCSI
Adapter

VIOS
v
S
C
S
I

v
L
A
N

AIX 5L Linux AIX 5L Linux

V5.3
V5.3
A1
B1

B2

A2
B3

A3

B1 B2 B4 B5
B3
VIOS owns physical disk resources
LVM based storage on VIO Server
Physical Storage can be SCSI or FC
Local or remote
Partition sees disks as vSCSI (Virtual SCSI)
devices
Virtual SCSI devices added to partition
via HMC
LUNs on VIOS accessed as vSCSI disk
VIOS must be active for client to boot

Virtual SCSI

POWER Hypervisor

Multiple LPARs can use same or different physical

disk
Configure as logical volume on VIOS
Appear a hdisk on the partition
Can assign entire hdisk to a single client

Available via optional Advance POWER Virtualization or POWER Hypervisor and VIOS features.
13

13

2003 IBM Corporation

IBM GCG ATS

Virtual SCSI Basic Architecture

Client Partition

Virtual I/O Server

vSCSI Target Device
PV
VSCSI

LV
VSCSI

Optical
VSCSI

LVM

DVD

Multi-Path
or
Disk Drivers

Hdisk

vSCSI
Client
Adapter

vSCSI
Server
Adapter

Optical
Driver

Adapter /
Drivers

POWER5 Hypervisor

FC or SCSI
Device

14

14
2007 IBM Corporation

IBM GCG ATS

Logic Remote Direct Memory Access

15

15
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI device relationship on VIOS

16

16
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI device relationship on VIOC

17

17
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI HA Options 1/2

AIX Client Mirroring, with SCSI MPIO in VIO Server, PV VSCSI Disks

Complexity
Requires LVM mirroring to be setup on the client

AIX A
(LVM Mirror)

If a VIOS is rebooted, the mirrored disks will need to

be resynchronized via a varyonvg on the VIOC

AIX B
(LVM Mirror)

Resilience
Protection against failure of single VIOS / SCSI disk
/ SCSI controller

Notes
Requires multiple adapters on each VIOS

18

VIOS 1

VIOS 2

vSCSI

vSCSI

SCSI
MPIO

SCSI
MPIO

18
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI HA Options 1/2- Details

AIX Client Mirroring, with SCSI MPIO in VIOS, PV VSCSI Disks
AIX Client LPAR A
A

LVM Mirroring
vscsi0

vscsi1

AIX Client LPAR B

B

LVM Mirroring
vscsi0

vscsi1

Hypervisor
vhost0

vhost1

vhost0

vhost1

vtscsi0

vtscsi1

vtscsi0

vtscsi1

SCSI MPIO
VIOS 1

19

scsi0

scsi1

SCSI MPIO
VIOS 2

scsi0

scsi1

19
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI HA Options 2/2

AIX MPIO Default PCM Driver in Client, Multi-Path I/O in VIOS

Complexity
Requires MPIO to be setup on the client

Default MPIO PCM in the Client

Supports Failover Only

VIOS 1

Requires Multi-Path I/O setup on the VIOS

Resilience
Protection against failure of a single VIOS, FC
adapter, or path.
Protection against FC adapter failures within VIOS

vSCSI

AIX A
(MPIO Default
PCM)
AIX B
(MPIO Default
PCM)

MPATH*

VIOS 2

vSCSI
MPATH*

Throughput / Scalability
Potential for increased bandwidth due to Multi-Path
I/O
Primary LUNs can be split across multiple VIOS to
help balance the I/O load.

FC
FCSAN
SAN

Notes
Must be PV VSCSI disks.
A
B
PV LUNs

* Note: See the slide labeled VIOS Multi-Path Options for a high level overview of MPATH options.

20

20
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI HA Options 2/2 - Details

AIX MPIO Default PCM Driver in Client, Multi-Path I/O in VIOS
AIX Client LPAR 1

AIX Client LPAR 2

MPIO Default PCM

MPIO Default PCM

vscsi0

vscsi1

vscsi0

vscsi1

Hypervisor
vhost0

vhost1

vhost0

vhost1

vtscsi0

vtscsi1

vtscsi0

vtscsi1

Multi-Path Driver

VIOS 1

fcs0

Multi-Path Driver

fcs1

VIOS 2

fcs0

fcs1

A
B
PV LUNs

21

Active
Passive

21
2007 IBM Corporation

IBM POWER5 Virtual Ethernet

Two basic components


VLAN-aware Ethernet switch in the

Hypervisor
Comes standard with a
POWER5 server.

Virtual I/O Server

Ent0
(Phy)

Shared Ethernet Adapter

Part of the VIO Server
Acts as a bridge allowing
access to and from an external
networks.
Available via the Advanced
POWER virtualization feature.

Client 1

Client 2

Shared
Ethernet
Adapter

en0
(if)

en0
(if)

ent1
(Vir)

ent0
(Vir)

ent0
(Vir)

VLAN-Aware Ethernet Switch

Hypervisor

Ethernet
Switch

22

Bridging, Routing, and Switching

Ethernet
Header

Bridge

Router

Bridge

Router

TCP/IP
Header

Data

Ethernet
Header

TCP/IP
Header

Data

Switch (Multi-port Bridge)

VLAN Switch (Smart Multi-port Bridge)

Switch

Switch

Ethernet
Header

TCP/IP
Header

Data

Ethernet V
Header L

TCP/IP
Header

Data

23

POWER5 VIO Server Switch Concepts

Virtual I/O Server Switch

Client

Client LPAR

en0
(if)

en0
(if)

ent0

ent0
(Virt)

Switch

Port

VIOS

ent1
(Virt)

Logic

ent2
(SEA)

Port

ent0
(phy)

To other device or switch

To other switch

POWER5 Server

Traditional Switches

24

Ethernet Virtual LAN (VLAN)

IEEE 802.1Q VLAN


Operates at layer 2 (data link) of the OSI model

Creates logical networks within a physical network

A trunk is a network link carrying multiple VLANS which

are identified by tags

VLAN Benefits

Ethernet
Switch

Reduces the size of a broadcast domain

Reduces hardware (adapter, port) requirements

Segments network traffic (can improve security)

Can move/add devices to different networks without re-cabling

Preamble

Start
Frame
Delimiter

Dest.
MAC
Address

Source
MAC
Address

Tag
Prot. ID
( 0x8100 )

User Priority
( 3 bits)

Tag
Control
Info

CFI
(1 bit = 0)

Length

Data

Pad

Frame
Check
Sequence

VLAN ID
(12 bits)

25

IEEE 802.3ad Link Aggregation

Link Aggregation


Allows multiple Ethernet adapters to be aggregated together and

behave like a single pseudo network adapter.

All the adapters have one common hardware (MAC) and IP

address.

All adapters must be configured for the same speed and must be
full duplex.

All network adapters that form the link aggregation (not including a
backup adapter) must be connected to the same network switch.

Sometimes called NIC teaming, port teaming, or NIC bonding.

A Link Aggregation of more than one active virtual Ethernet adapter

is not supported.

Host

NIC

NIC

NIC

Ethernet Switch

Link Aggregation Benefit



Greater reliability

Greater total network bandwidth

26

EtherChannel
Standard Algorithm


Standard algorithm uses the last byte of the

destination IP to determine the outbound NIC.

All traffic to the same host goes out the same NIC.

Host
Source
IP Addr

Dest.
IP Addr

Hash Mode (src_dst_port)




Dest
Port

Hash
Function

Hashes the source and designation TCP or UDP

port values to determine the outbound NIC.
Probably best initial choice for mode.

Source
Port

NIC

NIC

NIC

NIC

Round Robin (EtherChannel only)



Traffic spread evenly across all adapters.

Ideal when there are no intervening switches due

to increased risk of out-of-order packets.

Port Port Port Port

Incoming Traffic from Switch



Governed by the algorithm of the switch.

Ethernet Switch

27

Shared Ethernet Adapter

In most cases, it is
unnecessary to create
more than one Virtual
Ethernet adapter for a
SEA.
(Think simple!)

VIOS 1

Multiple VLANs can be

added to a single SEA.

ent2
(Phy)

Client 1
en8
(if)
ent6
(LA)

ent8
(SEA)

ent1
(Phy)

ent4
(Vir)
VID
200

Client 2

en7
(if)

ent5
(Vir)

PVID
2

ent0
(Phy)

ent7
(SEA)

en0
(if)

en1
(if)

en0
(if)

en1
(if)

ent3
(Vir)

ent0
(Vir)

ent1
(Vir)

ent0
(Vir)

ent1
(Vir)

VID 300
( PVID 3 )

PVID 2
VID 200,300

PVID 100

PVID
300

PVID
2

PVID 100

mkvdev -sea ent0

-vadapter ent3

-default ent3

-defaultid 100

mkvdev -sea ent6

-vadapter ent4,ent5

-default ent4

-defaultid

Physical Ethernet
adapter or link
aggregation device

PVID
200

Virtual Ethernet
adapters in the VIOS
that will be used with
this SEA

Virtual Ethernet
that will contain
the default VLAN

Default
VLAN

28

High Availability VIOS Options

Network Interface Backup
 Must

be set up in each client.

Shared Ethernet Adapter Failover

 Set

up in the VIOSs only.

 Needs

 Optional

 Can

 Cannot

to ping outside host from each

client to initiate NIB failover.
load share clients across SEAs.

 VLAN-tagged
 Supported

traffic is not supported.

only on AIX.

ping is done in VIOS on behalf of

all clients.
load-share clients between the
primary and backup SEA.

 VLAN-tagged
 Supported

POWER5 Server
AIX Client

traffic is supported.

on AIX and Linux on POWER.

POWER5 Server
Client
Virt
Enet

Virt Virt
Enet Enet

NIB
VIOS 1

VIOS 2

VIOS 1

VIOS 2

SEA

SEA

SEA

SEA

Enet
PCI

Enet
PCI

Enet
PCI

Enet
PCI

Primary

Backup

29

Deployment Example of Virtual I/O

IBM GCG ATS

VIO server

Database lpar

Application lpar

VIO server

en0
LA
en2

pvid101

vlan
101

SEA

en1

vent0
LA
ent2

SCSI0

LA
ent2

vent1
vent0

vlan

hdisk0

vhost0

IO server
rootvg

LVs for
client
rootvgs

VSCSI0

hdisk0

VSCSI1
rootvg
mirror

LV2

LV1

hdisk1
MPIO
disk1

vhost3

hdisk3
vpath1

VSCSI0

VSCSI2

SDD

hdisk1

vhost0

hdisk1

vhost1

vpath1

en1

LV2
hdisk2
vpath1

rootvg
mirror

hdisk0

LV1

VSCSI1

vhost1

VSCSI3

hdisk0

SCSI0
IO server
rootvg

LVs for
client
rootvgs

vhost3

vpath1

SDD

MPIO
disk 2

hdisk4

VSCSI4

VSCSI5

LUN1
hdisk9

vhost2

Note: vhost and vscsi are

shown as more than
required. VIO server/client
disks can be presented on
shared vhost/vscsi adapters

LUN0
hdisk8

LUN0
hdisk7
HBA or
SCSI

vhost2

hdisk5

HBA0

HBA1

LUN1
hdisk9

HBA1

LUN0
hdisk8

HBA0

HBA or
SCSI

LUN0
hdisk7

SAN Switch

30

LA
en2

SEA

pvid102

hdisk1

Real
Virtual

en0

vent1

SAN Switch
LUN0

LUN1

LUN2

SEA = Shared Ethernet

Adapter
LA = Link Aggregation

30

2007
Special thanks to Tom Prokop
forIBM
this Corporation
slide

Outline

Why Virtualization
Virtual I/O on system p5

31

Virtual I/O overview

Power virtualization architecture
Virtual I/O devices
Virtual I/O deployment example

Virtual I/O performance comparison test with physical I/O

IVE on Power 6 server
References and useful links

31

IBM GCG ATS

Virtual I/O performance Test

There are worries about the use of the VIO server for real work
when using disk or network resources
Resolution: A test system was configured to test disk and
network throughput using both direct attach and virtual
interfaces
Test tools
dd,iozone,NetPipe,swingbench,ftp ,nmon and nmon analyzer
Testing servers:

System p5 550 1.65GHz*4C8G

DS4500 14*15k*136G (6+1Raid5*2)

VIOS:1C1G Dedicate 2*GbE/2*2GFC

VIOC: 1C2G Dedicate 1*vEth 6*vSCSI

Direct connection both for storage and network

All parameters remains default if not specified.

32

32
2007 IBM Corporation

IBM GCG ATS

Testing Environment Topology

vtscsi0-n

ent3
SEA

hdisk2-n

ent1

vhost0-n

fcs0
fcs1

Suspended
VIO Client:vLPAR1
Partition

hdisk0 hdiskn

ent0

VIO
Suspended
Client:vLPAR2(3)
Partition

en3
(if)

ent2
VIOServer

hdisk1
hdiskx

hdisk0

en0
(if)

hdisk1
hdiskx

hdisk0

en0
(if)

vscsi1
vscsix

vscsi0

ent0

vscsi1
vscsix

vscsi0

ent0

VLAN-Aware Ethernet Switch

P550-02 Hypervisor

DS4500
DS4500

AIX Test Client:p550-01

SwingBench Test Client

33

33
2007 IBM Corporation

IBM GCG ATS

Virtual SCSI Testing conclusion

Function

Throughput

Expense

Ultra 320 15rpm local disk

Write 63MB/s

100%

Read 71MB/s

79%

Write 62MB/s

100%+3%

Read 70MB/s

79%+3%

Write 200MB/s

100%

Read 200MB/s

70%

Write 160MB/s

100%+10%

Read 185MB/s

73%+10%

Ultra 320 15rpm local disk backed virtual

disk

One LUN disk backed by 2G HBA

One LUN backed virtual disk

Comments

Note: max throughput data depends on backend storage system, the data in this table is only used for
compare the differences of virtual SCSI and physical SCSI.
1.
2.
3.
34

Virtual SCSI provides high efficient performance at low resource consumption at VIO Server
side
Multiple virtual adapters do not help to improve performance, setup one virtual adapter for each
client LPAR is enough.
There seems no bandwidth restriction of virtual SCSI, but depends on the backed physical
adapters.

34

2007 IBM Corporation

IBM GCG ATS

Virtual Ethernet Testing conclusion

Function
Virtual LAN Performance Data

Throughput

Expense

GbE Physical LAN

900Mbps

32%

Virtual LAN

1262Mbps

98%

Gb Physical LAN with jumbo

frame

940Mbps

20%

Virtual LAN with jumbo frame

4062Mbps

85%

Shared Ethernet adapter

839Mbps

75%+75%

Comments

Note: max throughput data depends on system resource, the data in this table is only used for
compare the differences of virtual LAN and physical LAN
1.
2.
3.
4.
35

Virtual Ethernet adapter provides high throughput than physical adapters if enough CPU available.
Virtual Ethernet need more CPU cycle than physical network adapter
SEA doubles the cost of CPU at VIOS and client LPAR.
Jumbo frame help to improve performance at higher bandwidth and lower resource consumption.
35
2007 IBM Corporation

Outline

Why Virtualization
Virtual I/O on system p5

36

Virtual I/O overview

Power virtualization architecture
Virtual I/O devices
Virtual I/O deployment example

Virtual I/O performance comparison test with physical I/O

IVE on Power 6 server
References and useful links

36

IBM GCG ATS

Integrated Virtual Ethernet Adapter

Dual 1Gb

Quad 1Gb

Address Sharing:
Dual 1GB: 16 MAC Addresses / port group
Quad 1GB: 16 MAC Addresses / port group
Dual 10GB: 16 MAC Addresses / port group

Dual
10Gb

Total: 16 (1 port group)

Total: 32 (2 port groups)
Total: 32 (2 port groups)

Non VIOS Partition: Address Sharing (MAC Addresses)

Time Slicing Physical Ethernet adapter resources
VIOS Partitions: IVE logical/physical port is dedicated
37
37

2007 IBM Corporation

IBM GCG ATS

Logic Components of IVE

38
38

2007 IBM Corporation

IBM GCG ATS

IVE advantages
Linux

i5/OS

AIX

Advantages:
No POWER Hypervisor hits

Ethernet
Driver

Ethernet
Driver

PHYP

Ethernet
Driver

HEA

Does not require a VIO server to be

running
No configuration required on any VIO
servers
Removes SW packet forwarding
overhead from VIO server
Provides equivalent performance as a
dedicated Ethernet adapter

39
39

2007 IBM Corporation

Outline

Why Virtualization
Virtual I/O architecture on system p5

40

Virtual I/O overview

Power virtualization architecture
Virtual I/O devices
Virtual I/O deployment example

Virtual I/O performance comparison test with physical I/O

IVE on Power 6 server
References and useful links

40

References and useful links

IBM Redbooks
http://www.redbooks.ibm.com

IBM Virtualization wiki

http://www-941.ibm.com/collaboration/wiki/display/virtualization/VIO

VIO Server Home

http://www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/home.html

IBM Systems Information Center

http://publib.boulder.ibm.com/infocenter/systems/index.jsp

Virtual I/O Server FAQ

http://www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/faq.htm

41

41