MMUG18 - MySQL Failover and Orchestrator
5 likes1,451 views
The presentation discusses handling MySQL failover and various clustering solutions while addressing downtime concerns for systems needing high availability. It introduces Orchestrator as a tool for managing master failovers, providing a GUI and CLI for topology visualization and recovery options, and emphasizes the need for a robust setup to run 24/7. Additionally, it covers the future development goals for the Orchestrator tool and recruitment opportunities at Booking.com.
More Related Content
![[2018] MySQL 이중화 진화기](https://cdn.slidesharecdn.com/ss_thumbnails/cloudinfra03-190131073325-thumbnail.jpg?width=560&fit=bounds)
Similar to MMUG18 - MySQL Failover and Orchestrator (20)
MMUG18 - MySQL Failover and Orchestrator
- 2. Gracias a Tuenti 117/05/2017 Madrid MySQL Users Group - MMUG18 • Por permitir el uso de sus oficinas para esta presentación
- 3. Content • Handling failover with MySQL • Downtime & Requirements • MySQL Clustering solutions • Non-clusterings solutions and considerations • Orchestrator • Questions 217/05/2017 Madrid MySQL Users Group - MMUG18
- 4. Is Downtime Acceptable? • Do you have a system that needs to run 24 x 7 ? • Not everyone does • If you have a website then generally downtime is not acceptable 317/05/2017 Madrid MySQL Users Group - MMUG18
- 5. Requirements Goal: Run 24 x 7 x 365 with no downtime • Is this really necessary? • If you ask management they’ll always say yes… • What is the cost? • Shorter downtime requirements mean more effort spent to achieve that • How do you reliably detect failure? Hard problem to solve If you accept downtime how much can you really tolerate? • 1s, 5s, 30s, 1min ? 417/05/2017 Madrid MySQL Users Group - MMUG18
- 6. What options are available? • MySQL Cluster • carrier grade • very high uptime • Not InnoDB – specialised workloads • Galera • Often with asynchronous replication between datacentres • InnoDB Cluster • Very new • All require clients to take action on failure of a node • If you use a proxy that can fail too… 517/05/2017 Madrid MySQL Users Group - MMUG18
- 7. What options are available? “Cluster solutions” • Do not work well cross-DC due to latency • If you accept writes into multiple masters there’s a chance of conflict • Slows things down • InnoDB Cluster now does not recommend this behaviour – requires care • Only small setups work in a single data-centre so adaptation here is also needed • Cluster setups do not scale easily to 10 or more servers 617/05/2017 Madrid MySQL Users Group - MMUG18
- 8. What options are available? • Standard MySQL, MariaDB, Amazon RDS, Google Cloud SQL, … • Read scale-out • Asynchronous replication • Semi-sync helps improve performance and ensure data is “somewhere else” when acknowledging a transaction • If you are out of the cloud then: different setups • SBR or RBR? • No GTID, Oracle or MySQL GTID? • Optional semi-sync? • If you are out of the cloud then: do it yourself • MHA • MariaDB Replication Manager • Orchestrator 717/05/2017 Madrid MySQL Users Group - MMUG18
- 10. Orchestrator • Handles master failover, but more… • GUI to manage and visualise topology – very handy • CLI to do the same things – good for scripting • API calls to run at a distance (more generic interface) • Needs a DB backend to store state. • Normally MySQL but can be SQLite 917/05/2017 Madrid MySQL Users Group - MMUG18
- 11. Orchestrator What failures does it handle? • Master failures – needs to talk to external systems • Intermediate master failures – can handle on its own • Does not care about slaves or applications • Works with GTID: Oracle or MariaDB • Works without using GTID: Can add Pseudo-GTID (events injected on the master are used to find a match) so no need to migrate to GTID if not wanted • Handles multi-level topologies 1017/05/2017 Madrid MySQL Users Group - MMUG18
- 15. Orchestrator CLI Over 100 commands you can use • E.g. • relocate • discover • begin-downtime, end-downtime • topology 1417/05/2017 Madrid MySQL Users Group - MMUG18
- 19. Orchestrator Setup • Source at github.com/github/orchestrator • Binaries written in go • Daemon runs web service and discovery, client on each MySQL server • State stored in MySQL / SQLite • Single json configuration file: /etc/orchestrator.conf.json • How to reach backend database (stores state) • How to recognise delay • Most defaults are good to get you going • Which systems you want to trigger recovery on • Hooks to handle recovery (talking to external systems) • If you need help please ask 1817/05/2017 Madrid MySQL Users Group - MMUG18
- 20. Orchestrator Characteristics • Discover one server in your cluster and orchestrator will find the others • Detects new servers in the cluster automatically • Notifies you of problems seen • Recovery is optional (per cluster) • Optional selection of candidate masters or servers to blacklist • Global ON / OFF switch – handy if several failures happen at once • For paranoid DBAs, so far orchestrator has always done the right thing 1917/05/2017 Madrid MySQL Users Group - MMUG18
- 21. Orchestrator HA ? Orchestrator can be run in HA mode • Multiple daemons will co-operate so if one fails another one takes over (they share the database backend) • Use a load balancer to provide an HA GUI service • Use nginx (or similar) for authentication and TLS if needed • Upgrades are easier • Replicate the orchestrator MySQL backend to not lose data 2017/05/2017 Madrid MySQL Users Group - MMUG18
- 22. Does it Scale? Yes • Booking.com has a large installation with a single cluster monitoring thousands of MySQL servers • Recommended by YouTube for managing Vitess servers • Quite a number of other users but they are not very visible 2117/05/2017 Madrid MySQL Users Group - MMUG18
- 23. Future work • Simplify configuration and setup so more people can use it • Improve scalability • Make it work on Amazon RDS • Spread the word… 17/05/2017 Madrid MySQL Users Group - MMUG18 22
- 24. Further help needed? • github.com/github/orchestrator • for Issues (Problems / Questions) and Pull Requests (patches) • Google Group: Orchestrator MySQL • https://groups.google.com/forum/#!forum/orchestrator-mysql • Feel free to contact me and I will try to help provide pointers 2317/05/2017 Madrid MySQL Users Group - MMUG18
- 25. Oh, and Booking.com is hiring! • Almost any role: • MySQL Engineer / DBA • System Administrator • System Engineer • Site Reliability Engineer • Developer • Designer • Technical Team Lead • Product Owner • Data Scientist • And many more… • https://workingatbooking.com/ 17/05/2017 Madrid MySQL Users Group - MMUG18 24