VMworld 2017 vSAN Network Design

Editor's Notes

• #5: Go to storagehub. Select vSAN. Select Plan and Design.
• #6: This section will describe the fundamentals behind vSAN’s architecture.
• #7: There are a number of distinct parts to vSAN networking. First there is the communication that takes place between all of the ESXi hosts in the vSAN cluster, indicating that they are still actively participating in vSAN. This has traditionally been done via multicast traffic, and a heartbeat is sent from the master to all hosts once every second to ensure they are still active. However, since the release of vSAN 6.6, this communication is now done via unicast traffic. This is a significant change compared to previous versions of vSAN, and should make vSAN configuration much easier from a networking perspective Lastly, there is virtual machine disk I/O. This makes up the majority of the traffic on the vSAN network. Because VMs on the vSAN datastore are made up of a set of objects, these objects may be made up of one or more components. For example, a number of RAID-0 stripes or a number of RAID-1 mirrors. Invariably, a VMs compute and a VMs storage will be located on different ESXi hosts in the cluster. It may also transpire that if a VM has been configured to tolerate one or more failures, the compute may be on node 1, the first RAID-0 mirror may be on host 2 and the second RAID-0 mirror could be on host 3. In this case, disk reads and writes for this virtual machine will have to traverse the vSAN network. This is unicast traffic, and forms the bulk of the vSAN network traffic. RDT traffic has always been unicast. VSANVP (Virtual SAN VASA Provider – Storage Awareness APIs) Used for Storage Policy Based Management Each vSAN node registers a VASA provider to vCenter Server via TCP
• #8: KMS server port varies from vendor to vendor Enabling encryption requires a rolling upgrade approach to write the DEKs (Disk Encryption Keys) to disk.
• #10: Speaker notes: In some cases, 10GB may not be required for stretched cluster. It will depend on the number of components, and rebuild bandwidth. Details are in our documentation. In many cases, when the witness and the vSAN traffic are separated in 2-node, witness traffic is placed on the same network as the management traffic.
• #11: Speaker notes. Single switches become a pain in large configs where you have to ensure MTU, VLAN, subnet mask, gatewa Distributed switches, while being a little more complex to configure, do offer greater advantages to vSAN customers. And it is free.
• #13: Keep is Simple! Use a single VMkernel port for vSAN and have the networking stack provide resiliency. On this please deploy what your networking team is most comfortable with! Designing vSAN networks – Dedicated or Shared Interfaces? https://blogs.vmware.com/virtualblocks/2017/01/20/designing-vsan-networks-dedicated-shared-interfaces/ IP HASH balance on a per host basis, so a connection between any two hosts will ONLY use one NIC. As a cluster grows the opportunity for a host to use more than one NIC and balance flows grows. It should be noted that seeing 80% on uplink 1, and 20% usage on uplink 2 is not unheard of - this is the nature of how great LACP is vMotion uses a less than documented hash that includes Source and Destination port and opens multiple connections so LACP (even if its benefit for vSAN is not significant) can help vMotion with large VM evacuations.
• #14: This section will describe the fundamentals behind vSAN’s architecture.
• #15: An IP Multicast address is called a Multicast Group (MG). Internet Group Management Protocol (IGMP) is a communication protocol used to dynamically add receivers to IP Multicast group membership. There are multiple versions V1, V2 , V3 Protocol Independent Multicast (PIM) is a family of Layer 3 multicast routing protocols that provide different communication techniques for IP Multicast traffic to reach receivers that are in different Layer 3 segments from the Multicast Groups sources. IP multicast sends source packets to multiple receivers as a group transmission, and provides an efficient delivery of data to a number of destinations with minimum network bandwidth consumption IGMP is a communication protocol used to dynamically add receivers to IP Multicast group membership. The IGMP operations are restricted within individual Layer 2 domains. IGMP allows receivers to send requests to the Multicast Groups they would like to join. Becoming a member of a Multicast Group allows routers to know to forward traffic that is destined for the Multicast Groups on the Layer 3 segment where the receiver is connected (switch port). This allows the switch to keep a table of the individual receivers that need a copy of the Multicast Group traffic.   IP Multicast is a fundamental requirement of vSAN prior to v6.6. Earlier vSAN versions depended on IP multicast communication for the process of joining and leaving cluster groups as well as other intra-cluster communication services. IP multicast must be enabled and configured in the IP Network segments that will carry the vSAN traffic service Some customers who do not wish to to use PIM for routing multicast traffic may consider encapsulating the multicast traffic in a VxLAN, or some other fabric overlay.
• #16: IGMP snooping is a mechanism to constrain multicast traffic to only the ports that have receivers attached. The mechanism adds efficiency because it enables a Layer 2 switch to selectively send out multicast packets on only the ports that need them When a network/VLAN does not have a router that can take on the multicast router role and provide the multicast router discovery on the switches, you can turn on the IGMP querier feature. The feature allows the Layer 2 switch to proxy for a multicast router and send out periodic IGMP queries in that network. This action causes the switch to consider itself an multicast router port. If it’s a small layer 2 environment and you only have 1 VSAN cluster per VLAN and NOTHING else goes on that VLAN besides Vmkernel ports you can use “flooding” and disable snooping for that VLAN. (No significant overhead as your basically having the switch broadcast that traffic). In a larger environment you will
• #17: This is true even in vSAN 6.6, in case there is a ’revert’ to multicast from unicast. Port 23451 is used by the master for sending a heartbeat to each host in the cluster every second. Port 12345 is used for the CMMDS updates.
• #18: This section will describe the fundamentals behind vSAN’s architecture.
• #21: Speaker notes: vCenter Server and ESXi hosts must be 6.50d EP2 or later vSAN cluster IP address list is maintained by vCenter and is pushed to each node. The following changes will trigger an update from vCenter: A vSAN cluster is formed A new vSAN node is added or removed from vSAN enabled cluster An IP address change or vSAN UUID change on an existing node
• #22: This section will describe the fundamentals behind vSAN’s architecture.
• #23: Table for education purposes – only really interested in first 3 behaviours.
• #25: When vCenter server recovers, vCenter (vSAN health) will attempt to reconcile its current list of unicast addresses with the vSAN cluster, and will push down stale unicast addresses to vSAN nodes. This may trigger a vSAN cluster partition and vCenter may no longer be accessible (since it runs on that vSAN cluster). DHCP with reservations (i.e. assigned IP addresses that are bound to the mac addresses of vSAN VMkernel ports) is supported, as is DHCP without reservations but with the managing vCenter hosted residing outside of the vSAN cluster IPv6 is supported with unicast communications in vSAN 6.6. With IPv6, a link-local address is an IPv6 unicast address that can be automatically configured on any interface using the link-local prefix. vSAN, by default, does not add a node’s link local address to other cluster nodes (as a neighbor). As a consequence, IPv6 Link local addresses are not supported for unicast communications on vSAN 6.6. A link-local address is an IPv6 unicast address that can be automatically configured on any interface using the link-local prefix FE80::/10 (1111 1110 10) . IPv6 link-local addresses are a special scope of address which can be used only within the context of a single layer two domain
• #28: This section will describe the fundamentals behind vSAN’s architecture.
• #29: Note: vSAN does not use NIC teaming for load balancing
• #30: In a simple I/O test performed in our labs - using 120 VMs with a 70:30 read/write ratio with a 64K block size on a four node all flash vSAN cluster, we can clearly see vSAN make no attempt to balance the traffic
• #32: Again using a simple I/O test performed in our labs, using 120 VMs with a 70:30 read/write ratio with a 64K block size on a four node all flash vSAN cluster
• #33: Note: this is contrary to what we had in our documentation. Not recommended, if subnet not routed
• #34: This section will describe the fundamentals behind vSAN’s architecture.
• #37: Multiple TOR (top of rack) switches Explain IGMP needed on all switches
• #38: Multiple TOR (top of rack) switches Explain IGMP needed on all switches
• #39: This section will describe the fundamentals behind vSAN’s architecture.
• #40: Multiple TOR (top of rack) switches Explain IGMP needed on all switches Between Data Site 1 and Data Site 2, VMware supports implementing a stretched L2 (switched) configuration or a L3 (routed) configuration. Both topologies are supported. Between Data Sites and Witness Site, VMware supports an L3 (routed) configuration
• #41: Multiple TOR (top of rack) switches Explain IGMP needed on all switches
• #42: This section will describe the fundamentals behind vSAN’s architecture.
• #43: Key Message/Talk track: Remote offices and branch offices (ROBO) is a common geographic model for many organizations. vSAN makes it easy to run a 2 node vSAN cluster to provide all of the storage needs in these branch offices, while using the primary site as the location to house the witness appliances. This makes for a fast, affordable, and flexible management and delivery of services in environments that require this type of topology. Additionally, vSAN ROBO Edition licensing has no host limit restriction. There is only a restriction of the number of VMs that may be run in a site. A maximum of 25 VM’s can be run in a single site, or across sites. Any multiple of 25 requires an additional 25 VM-pack license. There is no upgrade path from vSAN ROBO licenses to regular CPU or Desktop vSAN licenses. ---------------------------------- Overview: 2 Node ROBO for vSAN designed for branch office scenarios (retail, etc.) Both hosts in remote office store data. Each remote office seen as a 2 Node cluster Witness VM(s) lives in primary site – one witness VM for each remote office Can easily scale from 2 Node to more by adding additional hosts and removing the vSAN Witness. All sites managed by one vCenter instance Minimum requirements when using 2 Node from the site to the location of the vSAN Witness Appliance. 500ms RTT latency 1.5Mbps bandwidth
• #45: This section will describe the fundamentals behind vSAN’s architecture.
• #48: 4k blocksize, threads=32, 10x VMs, Stretch-Cluster native, no latency introduced: ~ 46938.13 IOPS 1ms RTT latency: ~ 39377.07 IOPS, ~ 83% from native 5ms RTT latency: ~ 32679.18 IOPS, ~ 69% from native 10ms RTT latency: ~ 20333.63 IOPS, ~ 43% from native 20ms RTT latency: ~ 11828.53 IOPS, ~ 25% from native
• #50: Native ~ 46k IOPS First test 1% -- 44k IOPS Second test 2% -- 30k IOPS 3rd test 5% -- 8k IOPS Native again ~ 46k IOPS 0.1% loss: ~ 45298.27 IOPS, ~ 97% from native 0.5% loss: ~ 44689.50 IOPS, ~ 96% from native 1% loss : ~ 43337.53 IOPS, ~ 93% from native 5% loss : ~ 9373.43 IOPS, ~ 20% from native 10% loss : ~ 3117.98 IOPS, ~ 6% from native
• #53: Mention metrics in talk track