Kubernetes: Managed or Not Managed?
0 likes251 views
This document discusses whether to use a managed or self-managed Kubernetes cluster. It describes Talend's initial deployment of Kubernetes using Kubespray on VMs, which required significant effort for upgrades and maintenance. It then discusses moving to Amazon EKS which simplified deployment and reduced monthly costs from $438 to $144, allowing Talend to focus more on applications. While EKS has limitations like a black box control plane, it offers easier setup and an evolving platform supported by AWS. The conclusion is that managed Kubernetes is good for application focus with small ops teams, while self-managed is better if fully controlling the infrastructure is important.
Kubernetes: Managed or Not Managed?
- 1. Kubernetes: Managed or Not Managed? CNCF Meetup - February 2020
- 2. Mathieu Herbert SRE Streamroot @MathieuHerbert Sébastien Charret SRE Talend @SCharret
- 3. Kubernetes: Managed or Not Managed? CNCF Meetup - February 2020
- 4. “ A long time ago …” January 2018
- 7. “Can we move our applications to Kubernetes ?” January 2018
- 8. How can we deploy K8S cluster ?
- 12. ● still in early stage (2018) ● strong blocker due to major upgrade that needs to recreate the cluster
- 13. ● Go based tool ● Big community
- 14. Kubespray ● Ansible role based ● Support multi-cloud provider
- 17. Kubespray - KISS (or not)
- 18. What needs to be done ?
- 20. ETCD ETCD * 3 Master * 3 API Server Scheduler Controller Manager Kubespray
- 21. ETCD ETCD * 3 Master * 3 API Server Scheduler Controller Manager Worker * N Kubelet Kube-proxy Container Engine Pods Kubespray
- 23. VM OS (Amazon centos AMI) VM customization (SSH key) Kubernetes component (kubespray)
- 24. Talend responsibility - ETCD ● Backup of the cluster state and procedure to restore it ● Node Rotation script for OS upgrade and ETCD Upgrade ○ Strategy: Rolling Update Node by Node ○ Critical!! Bunch of script to check the cluster status
- 25. Talend responsibility - Master nodes ● Load Balancing between 3 Master nodes ● Root Certificate Authority creation / Update / Backup ● Node Rotation, taking care that Kubespray is storing all the certificates on the 1st master (only) ○ Strategy: double the number of master nodes ○ Moving all the certificates from the 1st node to the new 1st node ○ Remove the old nodes ○ Be able to handle a restart from a checkpoint
- 26. Talend responsibility - Worker Nodes ● Procedure and script to remove one faulty Node ● Procedure to add new nodes (Manual scalability) ● Procedure to rotate all the nodes ○ Double the number of nodes ○ Moving all the pods from old nodes to new ones ○ Remove Old Nodes ○ Clean Calico database
- 27. Talend responsability - Cluster itself ● Update all components ○ Kubernetes ○ Calico ○ Kube-proxy ○ Kube-dns -> core DNS …. ○ …. ● Manage user creation based on certificate ● RBAC with groups (Dev / Admin)
- 28. Finally the apps
- 29. ...
- 30. “Hey Guys! Good news we need to upgrade Kubernetes from 1.10 to 1.11 then to 1.12” January 2019
- 33. Chill and let’s get back to Talend business
- 34. “Let’s move to managed K8S and concentrate our effort on the applications” January 2019
- 35. effort on the application” January 2019
- 36. State of the art of K8S deployment tools in 2019
- 37. EKS deployment With terraform pretty simple… Master/ETCD part
- 38. EKS deployment With terraform pretty simple… Workers part
- 39. What do we manage? Network Layer
- 40. What do we manage? Network Layer
- 41. What do we manage? Network Layer Worker
- 42. What do we manage ? Network Layer Autoscaling Worker Worker Worker WorkerWorker
- 43. Shared Responsibility Model Network Layer Autoscaling Worker Worker Worker Worker Worker
- 44. Who manage what - Control plane upgrade / resiliency - Worker upgrade - IAM / RBAC - SSL Certificate update (nodes)
- 45. EKS deployment Quick cost comparison (monthly cost): Kubespray: ● master = $219 (3*m4.Large) ● ETCD = $219 (3*m4.Large) Total = $438 EKS: ● control plane = $144 Total = $144!!! At talend we have roughly ~ 30 clusters, I let you imagine how happy were the manager...
- 47. EKS gain ● Less time on the cluster operation ● More time on monitoring, scalability, security … ● Growing community with lot of tool! ● AWS Support!
- 48. EKS pain ● Control plane black box ● Master logs are in cloudwatch erk! ● CM authentication is shitty (make mistake and your cluster will gone wild) ● Far behind the brand new feature brought by Kubernetes on premise
- 49. EKS: our Opinion ● Very easy to setup ● EKS platform is still evolving (GitHub AWS container roadmap is the place to follow) ● Not fully managed (Try to change your subnet in place if you dare...)
- 50. Managed VS Not Managed Managed Not Managed Comments ● Multi tenant clusters cannot be implemented by managed clusters Company business is infrastructure ● From OS configuration to core-dns versionWants to manage everything ● Less time on the infrastructure == More time on the application layer (monitoring / scalability / Resiliency) Application focused ● The time to maintain its own cluster is hugeSmall Ops team
- 52. Kubernetes: Managed or Not Managed? CNCF Meetup - February 2020