Best Practices for Virtualizing Hadoop
10 likes1,893 views
This document outlines best practices for virtualizing Hadoop, emphasizing the advantages of using virtual machines such as flexibility, rapid provisioning, and improved operational efficiency. It covers essential aspects of virtualization including hardware configuration, networking, storage solutions, and the use of Hadoop virtualized extensions for enhanced performance. Attendees are encouraged to follow recommended guidelines and configurations to optimize their Hadoop deployments in a virtual environment.
Best Practices for Virtualizing Hadoop
- 1. Best Practices for Virtualizing Hadoop March 21st, 2013 2:20 – 3:00pm George Trujillo © Hortonworks Inc. 2012
- 2. George Trujillo Master Principal Big Data Specialist - Hortonworks Tier One Big Data/BCA Specialist – VMware Center of Excellence VMware Certified Professional VMware Certified Instructor MySQL Certified DBA Sun Microsystem's Ambassador for Java Platforms Author of Linux Administration and Advanced Linux Administration Video Training Oracle Double ACE Served on Oracle Fusion Council & Oracle Beta Leadership Council, Independent Oracle Users Group (IOUG) Board of Directors, Recognized as one of the “Oracles of Oracle” by IOUG Page 2 © Hortonworks Inc. 2012
- 3. Agenda Title: Best Practices for Virtualizing Hadoop Summary: This presentation will discuss best practices for designing and building a solid, robust and flexible Hadoop platform on an enterprise virtual infrastructure. Attendees will learn the flexibility and operational advantages of Virtual Machines such as fast provisioning, cloning, high levels of standardization, hybrid storage, vMotioning, increased stabilization of the entire software stack, High Availability and Fault Tolerance. This is a can't miss presentation for anyone wanting to understand design, configuration and deployment of Hadoop in virtual infrastructures. Agenda: A. Hypervisor’s today B. Building an enterprise virtual platform C. Virtualizing Master and Slave servers D. Key best practices Page 3 © Hortonworks Inc. 2012
- 4. Grand Junction Colorado Page 4 © Hortonworks Inc. 2012
- 5. Hypervisors Today: Faster/Less Overhead • VMware vSphere, Microsoft Hyper-V Server, Citrix XenServer and RedHat RHEV Hypervisor Performance Benchmarks % Overhead VMware 1M IOPS with 1 microsecond of latency (5.1) 2 – 10% KVM 1M transactions/minute (IBM hardware RHEL) 10% Hypervisor Performance vSphere 5.1 VMware vCPUs 64 RAM per VM, RAM per Host 1TB / 2TB Network 36 GB/s IOPS 1,000,000 vSphere 5.1 – 1m IOPS with 1 1 μs latency Page 5 © Hortonworks Inc. 2012
- 6. Why Virtualize Hadoop • Virtual Servers offer advantages over Physical Servers • Enabling Hadoop as a service in a public or private cloud • Cloud providers are making it easy to deploy Hadoop for POCs, dev and test environments • Running a Consistent, Highly Reliable Hardware Environment • Standardizing on a Single Common Hardware Platform (software stack) • Virtualization is a natural step towards the cloud • Cloud and virtualization vendors are offering elastic MapReduce solutions Page 6 © Hortonworks Inc. 2012
- 7. Virtualization Features Faster provisioning Live Cloning Live migrations Templates Live storage migrations Distributed Resource Scheduling High Availability Hot CPU and Memory add Live Cloning VM Replication Network isolation using VXLANs Multi-VM trust zones VM Backups Distributed Power Management Elasticity Multi-tenancy Storage I/O Control Network I/O Control 16Gb FC Support iSCSI Jumbo Frame Support Page 7 © Hortonworks Inc. 2012
- 8. Building an Enterprise Virtual Platform Hortonworks Data Platform Sqoop Talend WebHDFS FlumeNG “Others” Data Extraction (DB Transfer) (Data Transfer) (REST API) (Data Transfer) (Vendors and 3rd party tools) And Load Ambari Oozie Ganglia Nagios Management (Management) (Workflow) (Monitoring) (Alerts) Monitoring WebHCatalog Pig Mahout HCatalog Hive HBase Zookeeper Hadoop (Rest-like APIs) (Scripting) (Machine Learning) (Metadata Mgmt) (Query) (Column DB) (Coordination) Essentials Distributed Processing Distributed Storage Core Hadoop (MapReduce) (HDFS) (kernel) Linux Windows Hypervisor Hardware Page 8 © Hortonworks Inc. 2012
- 9. Virtualizing Master Servers • Virtualize the master servers (NameNode, JobTracker, HBase Master, Secondary NameNode) – Ganglia, Nagios, Ambari, Active Directory, Metadata databases • Virtualizing Hadoop master servers offers: – Virtualization features to master servers – VMware High Availability • Develop a hybrid storage solution Page 9 © Hortonworks Inc. 2012
- 10. Protecting Master Servers • Shared storage is required to fully leverage virtualization features. • A virtual enterprise platform will provide: – Less down time (Live migrations, cloning, …) – A more reliable software stack – A higher Quality of Service – Reduced CapEx and OpEx – Increased operational flexibility Page 10 © Hortonworks Inc. 2012
- 11. Configure Hardware Properly • Do not overcommit SLA or production environments • Understand how NUMA affects your VMs – try to keep the VM size within the NUMA node – Look at disabling node interleaving (leave NUMA enabled) – Maintain memory locality • Leverage hyperthreading – make sure there is hardware and BIOS support – Hyper Threading – can improve performance up to 20% • Do not set memory limits on production servers. • Let hypervisor control power mgmt by BIOS setting “OS Controled Mode” • Enable C1E in BIOS Page 11 © Hortonworks Inc. 2012
- 12. Configure Hardware Properly (2) • Run latest version of BIOS and VMware Tools • Verify BIOS settings enable all populated processor sockets and enable all cores in each socket. • Enable “Turbo Boost” in BIOS if processors support it. • Disabling hardware devices (in BIOS) can free interrupt resources. – COM andn LPT ports, USB controllers, floppy drives, network interfaces, optical drives, storage controllers, etc • Enable virtualization features in BIOS (VT-x, AMD-V, EPT, RVI) • Initially leave memory scrubbing rate at manufacturer’s default setting. Page 12 © Hortonworks Inc. 2012
- 13. More Best Practices • Use latest hypervisor and virtual tools • Configure an OS kernel as a single-core or multi-core kernel based on the number of vCPUs being used. • Size virtual machines to avoid entering host “soft” memory state and the likely breaking of host large pages into small pages. Leave at least 6% of memory for the hypervisor and VM memory overhead is conservative. – If free memory drops below minFree (“soft” memory state), memory will be reclaimed through ballooning and other memory management techniques. All these techniques require breaking host large pages into small pages. • Have a very good reason for using CPU affinity otherwise avoid it like the plague Page 13 © Hortonworks Inc. 2012
- 14. Linux Best Practices • Kernel parameters: – nofile=16384 – nproc=32000 – Mount with noatime and nodiratime attributes disabled – File descriptors set to 65535 – File system read-ahead buffer should be increased to 1024 or 2,048. – Epoll file descriptor limit should be increased to 4096 • Turn off swapping • Use ext4 or xfs (mount noatime) – Ext can be about 5% better on reads than xfs – XFS can be 12-25% better on writes (and auto defrags in the background) • Linux 2.6.30+ can give 60% better energy consumption. Page 14 © Hortonworks Inc. 2012
- 15. Networking Best Practices • Separate VM traffic from live migration and management traffic – Separate NICs with separate vSwitches • Leverage NIC teaming (at least 2 NICS per vSwitch) • Leverage latest drivers for hypervisor vendor • Avoid multi-queue networking: Hadoop drives a high packet rate, but not high enough to justify the overhead of multi-queue. • Network: – Channel bonding two GbE ports can give better I/O performance – 8 Queues per port Page 15 © Hortonworks Inc. 2012
- 16. Networking Best Practices (2) • Evaluate these features with network adapters to leverage hardware features: – Checksum offload – TCP segmentation offload(TSO) – Jumbo frames (JF) – Large receive offload(LRO) – Ability to handle high-memory DMA (that is, 64-bit DMA addresses) – Ability to handle multiple Scatter Gather elements per Tx frame • Optimize 10 Gigabit Ethernet network adapters – Features like NetQueue can significantly improve performance of 10 Gigabit Ethernet network adapters in virtualized environments. Page 16 © Hortonworks Inc. 2012
- 17. Storage Best Practices • Make good storage decisions – i.e. VMFS or Raw Device Mappings (RDM) – VMDK – leverages all features of virtualization – RDM – leverages features of storage vendors (replication, snapshots, …) – Run in Advanced Host Controller interface mode (AHCI). – Native Command Queuing enabled (NCQ) • Use multiple vSCSI adapters and evenly distribute target devices • Use eagerzeroedthick for VMDK files or uncheck Windows “Quick Format” option • Makes sure there is block alignment for storage Page 17 © Hortonworks Inc. 2012
- 18. Hadoop Virtualized Extensions (HVE) • HVE is a new feature that extends the Hadoop topology awareness mechanism to support rack and node groups with hosts containing VMs. – Data locality-related policies maintained within a virtual layer • HVE merged into branch-1 – Available in Apache Hadoop 1.2 • Extensions include: – Block placement and removal policies – Balancer policies – Task scheduling – Network topology awareness Page 18 © Hortonworks Inc. 2012
- 19. HVE: Virtualization Topology Awareness • HVE is a new feature that extends the Hadoop topology awareness mechanism to support rack and node groups with hosts containing VMs. – Data locality-related policies maintained within a virtual layer. Data Center Rack1 Rack2 NodeG 1 NodeG 2 NodeG 3 NodeG 4 Host1 Host2 Host3 Host4 Host5 Host6 Host7 Host8 V V V V V V V V V V V V V V V V M M M M M M M M M M M M M M M M V V V V V V V V V V V V V V V V M M M M M M M M M M M M M M M M Page 19 © Hortonworks Inc. 2012
- 20. HVE: Replica Policies Multiple replicas are not placed on the same node with standard or extension replica placement/removal policies. Rules are maintained for the balancer. Standard Replica Policies Extension Replica Policies 1st replica is on local (closest) node of Multiple replicas are not be placed on the writer the same node or on nodes under the same node group 2nd replica is on separate rack of 1st 1st replica is on the local node or local replica; node group of the writer 3rd replica is on the same rack as the 2nd replica is on a remote rack of the 2nd replica; 1st replica Remaining replicas are placed randomly across rack to meet minimum restriction. Page 20 © Hortonworks Inc. 2012
- 21. Follow Virtualization Best Practices Validate virtualization and Hadoop configurations with Hardware vendor hardware compatibility lists. Hadoop Follow recommended Hadoop reference architectures. Follow virtualization vendors best practices, Virtualization deployment guides and workload characterizations. Storage Review storage vendor recommendations. Validate internal best practices for configuring and Internal managing VMs. Page 21 © Hortonworks Inc. 2012
- 22. Best Practices for Virtualizing Hadoop Thank You For Attending George Trujillo Blog: http://cloud-dba-journey.blogspot.com Twitter: GeorgeTrujillo