0% found this document useful (0 votes)
114 views55 pages

Kubernetes Installation

The document discusses the need for operationalizing Kubernetes and introduces the Kubernetes toolchain. It then outlines Pivotal Container Service (PKS) for Day 1 deployment and Day 2 operations of Kubernetes.

Uploaded by

Vel_st
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
114 views55 pages

Kubernetes Installation

The document discusses the need for operationalizing Kubernetes and introduces the Kubernetes toolchain. It then outlines Pivotal Container Service (PKS) for Day 1 deployment and Day 2 operations of Kubernetes.

Uploaded by

Vel_st
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 55

CNA2080BU

Deep Dive: How to Deploy


t i o n
and Operationalize i s tr ibu
or d
Kubernetes t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
Cornelia Davis, Pivotal V M
Nathan Ness
Technical Product Manager, CNABU
@nvpnathan

#VMworld #CNA2080BU
Disclaimer
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product. t i o n
tr ibu
r dis purchase orders, or
• Features are subject to change, and must not be included in contracts,
o
sales agreements of any kind.
a t i on
c u b li
• Technical feasibility and market demand will affect o p
r final delivery.
o t f
• n
Pricing and packaging for any new technologiest : N or features discussed or presented have not
ont e
been determined.
17 C
2 0
o r ld
VMw

#CNA2080BU CONFIDENTIAL 2
Agenda

1 What is the need?


t i o n
i s tr ibu
or d
2 Introducing the toolchain
t ion
bli c a
r p u
3
o t fo
Pivotal Container Service (PKS)
n t : N
4 onte1
PKSCDay
7
r ld 201
Mw5o PKS Day 2
V

#CNA2080BU CONFIDENTIAL 3
t i o n
The Need for Operationalizing i s tr ibu
o r d
Kubernetes t io n
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Companies Have Many Ways to Package and Run Their
Workloads in the Cloud

t i o n
i s tr ibu
or d
t ion
CONTAINERS
bli c a
p u
BATCHES EVENT-DRIVEN
t f or FUNCTIONS
:N o
t e n t
C on
1 7
r ld 20
Mw o
V
MICROSERVICES MONOLITHIC
DATA SERVICES APPLICATIONS

#CNA2080BU CONFIDENTIAL 5
Workloads that Might Be Suitable for Kubernetes
Those:
• Requiring Persistence
t i o n
– MongoDB, CouchDB, Couchbase, Elastic Search, …
i s tr ibu
• Managed as a cluster or d
t ion
– nodes need to communicate with one another
bli c a
r p u
o t fo
– often with the help of service meshes such as Istio or Linkerd
– Spark, Elastic Search nt: N
o n te
• 1 7 C
Needing new architectural primitives
d 2 0
• w o rl ports, etc.
Misc things like multiple
VM

#CNA2080BU CONFIDENTIAL 6
Serving up Kubernetes Dial-tone

Kubernetes

t i o n
i s tr ibu
kubectl
o r
Routing d
Master n
Master
b licatio
rp u
t f o
:N o
t
Responsible for the

t e n K8s cluster(s)

on
Responsible for the
themselves
C
workloads running
in K8s
1 7
ld
etcd
r 20 Worker
Worker
Mw o etcd
etcd
Worker
V manage

#CNA2080BU CONFIDENTIAL 7
Operational Challenges with Any Platform
Day 1 - Build Day 2 - Operate
Multi-cloud Provide a reliable and smooth Patches Patching platform components with
experience for any cloud. tion feel normal.
thousands of apps runningushould
b
i s t ri
Open APIs Allow platform operations from rd
n o scale platform components to
i o
Scaling Seamlessly
a t changing demand.
different toolsets and the creation of CD pipelines.
ub l i c
accommodate
o r p
Notf
Consistency Provide a consistent setup
e n t : Upgrades How do you roll out new versions of the
C ont
experience, across different cloud environment platform with the lights on?
configurations.
d 2017
or l
w
VMdoes it take to setup a real Operating Effort Operating a platform should
Setup time How long
world working environment? Think hours, not require very few resources and minimum manual
weeks. intervention. Otherwise, is it really providing
operational benefits?

#CNA2080BU CONFIDENTIAL 8
Kubernetes - Especially Hard to Operationalize
High Availability. No out-of-the-box fault-tolerance
for the cluster components themselves (masters,
workers and etcd nodes).
i tion
b u
t r
Scaling. Kubernetes r dis
n oclusters handle scaling the
a i o
t the Nodes, but doesn’t provide a
pod/service
u b l i cwithin
p
o t for
mechanism to scale Workers, Masters & etcd VMs.
nt: N
o n te Health checks and healing. The Kubernetes cluster
17 C
2 0 only does routine health checks for the health of
w orld workloads running on Nodes.
V M
Upgrades. Rolling upgrades on a large fleet of
clusters is hard. Who manages the system it runs
on?

#CNA2080BU CONFIDENTIAL 9
t i o n
i s tr ibu
or d
Introducing BOSH t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Powered by BOSH
Pivotal container service ops
Packaging w/ embedded OS
BOSH t
Server provisioning on any IaaS i o n
turib
rd i s
o across availability zones
ion
Software deployment
c a t
BOSH is an open source tool ub l i
o r p monitoring (server AND processes)
o t fHealth
for release engineering, n t : N
o n te Self-healing w/ Resurrector
deployment, lifecycle 17 C
2 0
o r ld Storage management
management,VM wand monitoring
of distributed systems. Rolling upgrades via canaries

Easy scaling of clusters

#CNA2080BU CONFIDENTIAL 11
Powered by BOSH
Pivotal container service ops
Kubernetes Packaging w/ embedded OS
Master
t
Server provisioning on any IaaS i o n
Master
turib
rd i s
o across availability zones
ion
Software deployment
c a t
ub l i
o r p monitoring (server AND processes)
etcd Worker
Worker o t fHealth
etcd
etcd Worker
nt: N
o n te Self-healing w/ Resurrector
17 C
2 0
w o rld Storage management
V M
BOSH
Rolling upgrades via canaries

Easy scaling of clusters

#CNA2080BU CONFIDENTIAL 12
Primary BOSH Entities
The definition of each of the nodes in the
cluster, including:
Kubernetes •TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
cluster, including: started on a node (jobs)
• The processes
Master •• The t i o n
ibu
Master bits installed on a node (packages)
Parameterized
s tr
• The processes started on a node (jobs)
i
or d BOSH release
t ion
bli c a
r p uA declaration of the desired state of the
etcd Worker
Worker o t fo cluster:
etcd
etcd Worker
nt: N • Assembly of the components from BOSH
C onte releases (relationships, dependencies)
1 7
r ld 20 • Parameter values

Mw o BOSH deployment
V BOSH
Relationship to the underlying infrastructure
BOSH cloud config

#CNA2080BU CONFIDENTIAL 13
The Workflow
The definition of each of the nodes in the
cluster, including:
Kubernetes •TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
cluster, including: started on a node (jobs)
• The processes
Master •• The t i o n
ibu
Master bits installed on a node (packages)
Parameterized
s tr
• The processes started on a node (jobs)
i
or d BOSH release
STEP a2:tio n and Manage Kubernetes
Install
u b l ic
Worker
f or p A declaration of the desired state of the
etcd Worker o t cluster:
etcd
etcd Worker
e n t:N • Assembly of the components from BOSH
on t
1 7 C releases (relationships, dependencies)

r ld 20 • Parameter values

Mw o BOSH deployment
V BOSH
Relationship to the underlying infrastructure
STEP 1: Install and configure BOSH
BOSH cloud config

#CNA2080BU CONFIDENTIAL 14
t i o n
i s tr ibu
or d
Pivotal Container Service (PKS) a t i on
blic p u
fo r
N o t
n tent:
C o
0 17
rld 2
Mw o
V
Project Kubo

“Day 1” Build t i o n
Uniform way to instantiate, deploy, and t r
● Deploy Kubernetes i b ucluster via BOSH
is
r d
manage highly available Kubernetes on o
cia tOperate
l
“Day i 2”
clusters. On any cloud. r pu●b Self-healing VMs and monitoring via
o t fo
n t : N BOSH
ont e ● Elastic scaling for clusters
Launched by Pivotal & Google
1 7 CFeb
d 2 0 ● Rolling upgrades to latest Kubernetes
w o rl
2017, Donated to Cloud Foundry release
VM2017
Foundation June ● High-availability and multi-AZ support

#CNA2080BU CONFIDENTIAL 16
Kubo Provides Specification of K8S Components
Kubernetes
This forms the
Open Core
of Master
Pivotal Container Master
t i o n
Kubo Release Service
i s tr ibu
(PKS)
or d
t ion
Release
bli c a
templates
etcd r p u Worker
o t
etcdfo Worker
Worker
nt: N etcd

o n te
1 7 C
2 0
w o rld
V M bosh deploy
Manifest BOSH

17
#CNA2080BU CONFIDENTIAL
Kubernetes Dial Tone:
• Health management
• Aggregated Metricsuand t i onLogging
Provides the control plane i s t r ib
• Autoscalingr d
for provisioning and o n o
i
u licat
• Persistence
b
interface
managing Kubo releases or p
o t f
e n t:N Control Plane:
on t • Provisioning Engine
1 7 C
Joint development 2 0
effort
o r ld • Self-service Clusters
VMw
between Pivotal, VMWare • Software Update Automation
and Google • Load balancing
• Networking
• Multi-tenancy
#CNA2080BU CONFIDENTIAL 18
PKS Leverages the Power of BOSH
Kubo Release

Release
templates
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
Manifest
nt: N
o n te
17 C
2 0
w orld
PKS

V M
BOSH

19
t i o n
i s tr ibu
or d
Kubernetes Cluster – Day 1 t ion
bli c a
r p u
Deploy
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Starting with a BOSH Deployment...
The definition of each of the nodes in the
cluster, including:
Kubernetes
•TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
•cluster, including: started on a node (jobs)
Master
The processes
•• The t i o n
ibu
Master bits installed on a node (packages)
Parameterized
i s tr
• The processes started on a node (jobs)
or d BOSH release
t ion
bli c a
r p uA declaration of the desired state of the
etcd Worker
Worker o t fo
etcd
etcd Worker
nt: N cluster:
• Assembly of the components from BOSH
C onte releases (relationships, dependencies)
1 7
r ld 20 • Parameter values

Mw o BOSH deployment
V BOSH

#CNA2080BU CONFIDENTIAL 21
Deploying a Kubernetes Cluster with Cloud Foundry BOSH

DB
t i o n
Deploy my
BOSH Director
i s tr ibuWorker VMs
K8s
or d
t ion
Blobs
bli c a
r p u
otfo etcd

n t : NMessage Bus
Deployment
ont e Master
Target VM
• Packages 1
Health Monitor7 C
d 2 0 Worker
Target VM
• Blobs
w o rl
• Source VM Target VM

• Jobs
• Manifest BOSH vSphere

#CNA2080BU CONFIDENTIAL 22
t i o n
i s tr ibu
or d
Kubernetes Cluster – Day 2 t ion
bli c a
r p u
Operationalize
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Day 2: Operationalize

t i o n
1 Managing Health i s tr ibu
or d
t ion
bli c a
2 Scaling
r p u
o t fo
nt: N
3 Upgrade
o n te
17 C
2 0
w orld
V M
K8s Cluster Health: Processes are Monitored

AGENT

t i o n
Health Monitor
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7

r ld 20
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 25
K8s Cluster Health: Processes are Monitored

AGENT

t i o n
Health Monitor
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7

r ld 20
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 26
K8s Cluster Health: Processes are Monitored

AGENT

t i o n
Health Monitor
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7

r ld 20
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 27
K8s Cluster Health: VMs are Monitored

Desired State Actual State AGENT

t i o n
Health Monitor BOSH Director
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
ressurector
r ld 20

Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 28
K8s Cluster Health: VMs are Monitored

Desired State Actual State AGENT

t i o n
Health Monitor BOSH Director
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
ressurector
r ld 20

Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 29
K8s Cluster Health: VMs are Monitored
CPI
Desired State AGENT
Actual State
t i o n
Health Monitor BOSH Director
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
ressurector
r ld 20

Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 30
Day 2: Operationalize

t i o n
1 Managing Health i s tr ibu
or d
t ion
bli c a
2 Scaling
r p u
o t fo
nt: N
3 Upgrade
o n te
17 C
2 0
w orld
V M

#CNA2080BU CONFIDENTIAL 31
Primary BOSH Entities The definition of each of the nodes in the
cluster, including:
•TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
Kubernetes cluster, including: started on a node (jobs)
• The processes
Master •• The bits installed on a node (packages)
Master Parameterized
• The processes started on a node (jobs)t i o n
i s tr ibu
BOSH release
or d
t ion
Worker bli c a
A declaration of the desired state of the
etcd Worker
r p u
cluster:
etcd
etcd Worker
o t fo • Assembly of the components from BOSH
nt: N
o n te releases (relationships, dependencies)
• Parameter values
17 C
2 0
w orld BOSH deployment
M
V BOSH
Relationship to the underlying infrastructure
BOSH cloud config

#CNA2080BU CONFIDENTIAL 32
instance_groups:
- name: etcd - name: master
instances: 2 - name: worker
instances: 3 instances: 3
networks: networks:
- name: *network-name networks:
- name: &network-name ((deployments_network)) - name: *network-name
azs: [z1] azs: [z1]
jobs: azs: [z1]
jobs: jobs:
- name: etcd - name: cloud-provider
release: kubo - name: docker
t i o n
ibu
release: kubo-etcd release: docker
properties: properties: {}
i s tr
etcd: - name: kubernetes-api properties:
or d
release: kubo t io
...n
require_ssl: false
bli c a
- name: kubeconfig
peer_require_ssl: false properties:
r p u release: kubo
stemcell: trusty
o fo
admin-username: admin
t properties:
vm_type: common
nt: N
admin-password: ((kubo-admin-password))
persistent_disk_type: 5120 o n te
... ...

17 C
- name: kubeconfig - name: kubelet
2 0 release: kubo
w o rld release: kubo
properties: properties:
Manifest
V M ... ...
... - name: kubernetes-proxy
stemcell: trusty release: kubo
vm_type: master properties:
...
stemcell: trusty
vm_type: worker
persistent_disk_type: 10240 33
instance_groups:
- name: etcd - name: master
instances: 2 - name: worker
instances: 3 instances: 3
networks: networks:
- name: *network-name networks:
- name: &network-name ((deployments_network)) - name: *network-name
azs: [z1] azs: [z1]
jobs: azs: [z1]
jobs: jobs:
- name: etcd - name: cloud-provider
release: kubo - name: docker
t i o n
ibu
release: kubo-etcd release: docker
properties: properties: {}
i s tr
etcd: - name: kubernetes-api properties:
or d
release: kubo t io
...n
require_ssl: false
bli c a
- name: kubeconfig
peer_require_ssl: false properties:
r p u release: kubo
stemcell: trusty
o fo
admin-username: admin
t properties:
vm_type: common
nt: N
admin-password: ((kubo-admin-password))
persistent_disk_type: 5120 o n te
... ...

17 C
- name: kubeconfig - name: kubelet
2 0 release: kubo
w o rld release: kubo
properties: properties:
V M ... ...
Scaling is a matter of changing...
the number of - name: kubernetes-proxy
stemcell: trusty release: kubo
instances and telling BOSH to
vm_type: master properties:
“make it so” ...
stemcell: trusty
vm_type: worker
persistent_disk_type: 10240 34
Day 2: Operationalize

t i o n
1 Managing Health i s tr ibu
or d
t ion
bli c a
2 Scaling
r p u
o t fo
Upgrade te nt: N
3
C on
2 017
orl d
VMw
K8s Cluster Upgrade: Canary Deployments

update: t i o n
t r i b u
canaries: 1 r d is
o n o
max_in_flight: 1
c a t i
u b l i
serial: true or p
o t f
n t : N
canary_watch_time: 10000-300000
t e
Con 10000-300000
update_watch_time:
1 7
2 0
w o rld
Manifest
V M

#CNA2080BU CONFIDENTIAL 36
K8s Cluster Upgrade: Canary Deployments
EXAMPLE:

# OF CANARIES: 2

MAX IN FLIGHT: 2
t i o n
i s t ribu
or d
t ion
bli c a
r p u
o t fo
nt: N
te
CANARIES

C o n
0 1 7
rld 2
Mw o
V
V1.0 V1.1

#CNA2080BU CONFIDENTIAL 37
K8s Cluster Upgrade: Canary Deployments
EXAMPLE:

# OF CANARIES: 2

MAX IN FLIGHT: 2
t i o n
i s t ribu
or d
t io n
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

V1.1 V1.2
Once failed, Canary VMs are kept
for troubleshooting purposes.

#CNA2080BU CONFIDENTIAL 38
t i o n
i s tr ibu
or d
Operationalizing at Scale t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Supporting Kubernetes Needs at Scale

Kubo Release

Release
templates
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
Manifest
o n te
17 C
2 0
w o rld
PKS Service Broker
VM
BOSH

40
Supporting Kubernetes Needs at Scale
https://thenewstack.io/comcast-1500-developers-working-cloud-foundry

Kubo Release

create cluster Release


(with upgrade policy) templates
t i o n Ones

i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N manage

Thousands Manifest
o n te
17 C
2 0
w o rld
PKS Service Broker
VM
BOSH

41
t i o n
i s tr ibu
or d
Let Us Show You… t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#CNA2080BU CONFIDENTIAL 43
NSX-T Integration
NSX Container Plugin
Kubernetes
Adapter

t i o n
ibu
CloudFoundry
PaaS Control Plane Adapter NSX
i s tr
NCM
Infra
Manager
or d
API-Server
Mesos API Client
t io n
Adapter

bli c a
etcd
Libnetwork
r p u
Scheduler Adapter
o t fo
nt: N
te
NSX topology for K8s / CF

C o n
NSX Container Plugin (NCP) for integrating with Kubernetes
0 1 7 Proj: foo Proj: bar
• NSX Features for K8s PODs
rl d 2
• IP address per container / w o
VM
POD
• Container Network – Routed (BGP) & NATed mode
• Microsegmentation – via K8s Network Policy or native NSX APIs (mapping
K8s labels to NSX tags)
• Network & Security automation – created as part of app deployment
• Multi-tenant network topologies
vRealize Ops, vRealize Log Insight For Comprehensive Visibility
Launch in
VMware vRealize Context VMware vRealize
Operations Log Insight
Capacity, Performance and Log analytics, aggregation,

n
Configuration Management Events and search
t i o
i s tr ibu
Structured Data
or
Unstructured Data d
t ionMessages
Metrics Alerts Events Logs
bli c a
r p u
o t fo
nt: N
o n te
1 7 C
2 0 Virtual Applications

w orld
V M
46

vRealize Ops – Managing Kubernetes Clusters

K8S Summary –
Nodes, Pods, etc. i o n
ibu t
i s tr
or d
t ion
K8S Topology -
K8S Pods - pHealthbli c a
u
Health
t f or
: N o
t e n t
C on
0 1 7
l d 2
or
VMw
47

vRealize Ops – Kubernetes Integration Details


K8S Pod Relationship
to Components
K8S Alerts n
t i o
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
7 C
r 201
K8S Alerts
ld
Mw o
V
Introducing Wavefront By VMware
SaaS-Based Metrics Monitoring and Analytics Platform
Iterate & Troubleshoot Trend & Alert Visualize Self-Service Metrics
Issues on Anomalies Metrics at Scale Analytics for All

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
UI and API Backend
o n te
17 C
0
Advanced Analytics Engine
2
w o rld
V M
Metrics Collection and Storage

Engineering &
Business

4
Wavefront – Container Monitoring Suite

t i o n
i s tr ibu
Amazon Docker
or d
ECS Swarm
t ion
bli c a
Docker Host Docker Host
r p u
o t fo
nt: N
o n te
17 C
2 0
rld
Container

Mw o
App Containers Metric Collector

V Docker Host

Docker
Cluster
Real-time insight into Docker containers and orchestration
systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS
50

Registry – Enterprise-grade Private Registry

user management & access control


role-based access control
t i o n
i b
AD/LDAP integration
t r u
r d is
n o
security
o
c at i
b l i vulnerability scanning
r pu
o t fo content trust - image signing
n t : N
ont e policy based image replication
1 7
Need Harbor Cscreenshot
0 d 2 audit and logs
or l
V Mw restful API
lightweight & easy deployment
open-source under Apache 2 license
51

Registry – Content Trust,


When Enabled Un-signed Images Can’t Be Pulled

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
52

Registry – Image Vulnerability Scanning Details

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
VMware PKS
t i o n
Kubernetes on BOSH (Kubo)
i s t ribu
Analytics Automation
or d
a t i on GCP

lic
Service

pub
etcd master worker etcd master worker
Container Broker
Registry
fo r
N o t
e n t : BOSH
nt
Logging Monitoring

C o
0 1 7
l d 2 NSX
or
Operations Security
VMw
vSphere vSAN

Physical Infrastructure
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

You might also like