AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mobile Gaming

Editor's Notes

• #5: Tip #1 is to automate the role of the NOC. Traditionally, some companies have a NOC team that is responsible to: - collect all the alerts from the different monitoring systems - do the triage - assess the severity of the alert - follow run-books - eventually escalate the issue to the Ops team or Dev teams This model have 2 problems: - by the time the right person is involved to troubleshoot, the incident is already going on for 10-15 minutes - of course you need to maintain a NOC team   - show in the same diagram human request, as above - show % of alerts in each bucket: app / system / email
• #6: - We learned that a NOC function can be mostly automated using services like Pagerduty But for that to be automated, you need to do some of the triage in the source of the alerts: Critical alerts should go to Pagerduty non-critical can be sent by email and handled later during business hours Tip #2 is to add the developers on-call in parallel to Ops. This workflow is common in many companies, where Ops will get all the pages, try to resolve, and eventually may escalate the issue to Dev if it’s a problem in the application code. The problem here is similar with when we are using a NOC… Ops is still doing triage of what is a system-level alert that he can resolve himself, or doing the escalation manually, if it’s an application issue that requires developer involvement. Effectively, Ops would be doing part of the NOC role, of Triage and Escalation. Ops guy would have to have that awkward call with Dev at 4am: “sorry man to call at this hour so late in the night… your game is down, and I need you to get online and help me fixed it…”  Following the generic DevOps recommendation, some companies started to put Dev on the Ops on-call rotation, so the Dev can also “feel the pain” and help to fix the root-cause of the alerts, but that doesn’t work well when you have multiple projects (in our case Games), and the devs know only 1 of the games.
• #7: Instead of putting Dev in the Ops on-call rotation, we found that we had to put the Dev team to receive page at the same time as Ops. And the Dev teams would receive only the alerts from the application they developed. GREE currently have about 20 games, considering versions for Android and iOS. On the development side, each game as a server-side development team, and usually 2 client-side teams: 1 for Android, and 1 for iOS. The application-level alerts, for instance, if it’s an issue of Android client crashing, would page the Android client-side developers of that game.
• #8: And that leads us to tip #3 – you need to standardize and separate clearly who is the right team to handle which alert. At GREE, we ask each game team to provide us with 3 different health-checks. The first one is a simple check for the elastic load balancer. This is used to automatically remove the server from rotation if it breaks. No one is paged if that happens, as a new server is created automatically by auto-scale. The second health-check is to detect system-level issues, like disk / network, checks the backend like Database and Memcache. Checks if replication is in-sync. If it fails, it alerts *ONLY* the Ops team, as Ops is able to resolve this kind of issue without dev assistance. The last type of health-check is for application-level issues, usually bugs in application. Those bugs generate warns and errors, that if above certain threshold would alert *BOTH* Dev and Ops team, as Ops will likely need Dev assistance to troubleshoot. The application-level alerts can be further broke-down by Server-side, or by Client-side, either iOS or Android platform. In that way, that alert will go only to the exact team that is responsible for that part of the code that is mal-functioning.
• #9: Replace by photos Add *many red* dashboard to illustrate how a major outage looks like 5)  - Service Desk / Monitoring - show before / after     - before: traditional escalation model - ops do triage, escalate to engineering manually, need to look-up list of contacts per game     - after: devops, page engineering directly for app-level alerts  / page correct engineer for each game    - urgent: page via email + Pagerduty integration    - non-urgent: open ticket via email + Jira integration   - show diagram with workflow of email/pingdom/zabbix alerts for email (non-urgent)/pagerduty (urgent)    - show in the same diagram human request, as above - show % of alerts in each bucket: app / system / email - Incident Management + Problem Management    - System-level monitoring - alerts go to Ops    - Application-level monitoring - alerts go to Ops and Devels    - Availability Management    - Pingdom (external) + Zabbix (internal) + Pagerduty    - Skypebot (mention hackaton origin)    - Status dashboard screenshot (mention hackaton) – add screenshots status dashboard green/one-red/many-red    - Monday meetings to discuss previous week Incidents        - chart: number of email alerts decrease over time / warn/errors decrease    - RCA meetings to major incidents (GLUE? for transition between topics - put Release Mgmt first, and glue with Incident Management when things go awry / or other way - Incident caused by a Release, then transition to Release)
• #10: Tip #4 – is to put a Big Status Dashboard near your Ops team. A Big Dashboard allow us to very quickly view the status of all of our games.