From vagrant to production - Mark Eijsermans
- 1. From Vagrant to production Mark Eijsermans HootSuite - Software Engineer @markeijsermans http://code.hootsuite.com
- 2. HootSuite • • • • • • • Social media dashboard 8M users 40 people committing code 4 ops AWS (2/3) & private cloud (1/3) 100M requests/day (hootsuite.com) 70M requests/day (ow.ly)
- 3. • Monolithic web app (PHP) • Transitioning towards service oriented architecture (Scala)
- 4. In the beginning dev server (LAMP) smb dev 1 svn smb dev 2 production (LAMP)
- 5. In the beginning Release anytime • Small team • Is intimately aware of prod • Low overhead to release DevOps?
- 6. ..a while later web web web web web web web web svn dev server web web web web gearman smb gearman 0mq worker dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev cron dev dev cron dev dev cron 0mq worker ops ops cron
- 7. ..a while later 1 release / 2-4 weeks • • • • • Medium sized team Branching, pre-release code freezes Only some devs knowledgeable of prod Ops mostly handles deploy Complicated process Devs vs. Ops?
- 8. …and a few years more (now) Vagrant dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev monolithic web app dev dev dev dev dev dev dev dev ops ops ops automate build test QA Service A Service B Service C Service D
- 9. …and a few years more (now) 6 - 10 release / day • • • • • • Larger team No branching Anyone deploys Automated process Commit to production in 15min Much higher % of devs understand prod
- 11. Perfection is the enemy of good • You’re never going to get a perfect system • How far along are we? • How many things are actually: • Automated • Reproducible • In a state we want them to be • When can we start feeling good about it?
- 12. The move to artifacts • svn • svn mirror • PHP • phar • tar + zsync • Scala • debian packages
- 13. Dev Build Deploy Jenkins project code unit test create artifact Artifact Repo build #124 build #123 smoke test staging dash4deploy production production service A/B/C
- 21. Broke the build? “I’m on it”
- 23. Vagrant • Working on a shared web server over samba was simply painful • extremely volatile environment • Vagrant simply kicks ass • Dev / prod parity • Encourages experimentation • go ahead and try out that new X
- 24. Dev Build Deploy Jenkins project code (vagrant) unit test create artifact Artifact Repo build #124 build #123 smoke test staging dash4deploy production production service A/B/C
- 25. Dev Build Deploy Jenkins project code (vagrant) Ansible unit test create artifact smoke test build vagrant box Artifact Repo vagrant.box build #124 build #123 dash4deploy production production service A/B/C staging
- 28. Ansible (and why we love it) …yes, use any CM, but why we chose Ansible: • New team members up to speed in a few days: • Gentle learning curve – YAML • Push over ssh - easy to understand • The more declarative it is, the more it documents • It's all about disseminating information • Agentless model suits immutable ephemeral instances
- 30. Auditing our infrastructure There are gems in that dirt • logs • you mean logs aren’t saved to /dev/null ? • app was rotating logs, vs. using logrotate • memcached, mongodb servers not all the same version • what the hell is server X for ???
- 31. Cache for gold • With configuration management gold images become a caching layer to the build process • Private cloud - idle instances are cheap • AMI - need auto spinning to save $$$ & time
- 32. Simplify for sanity • 200+ server types is hard to manage • Amortize the stack • Custom bespoke vs. generic appliances
- 33. Devs on call • Learning happens in production • Production is still running at 3am
- 34. “People who are really serious about software should make their own hardware” Alan Kay
- 35. Devs on call • Is your monitoring and tooling truly effective? • Make sure they can ssh into the damn box! (if they have to) • Throwing some poor dev onto a broken system they know nothing about doesn't work • Choose your own adventure cookbooks • Create registry of specialists
- 36. Is this working?
- 37. Working out loud • Yammer (& HipChat for on call) • Tried IRC, was too silo’ed • 5 whys post mortems • Accountability • Build fails - “I'm on it”
- 38. Learning happens in production