Immutable infrastructure isn’t the answer
Download as PPTX, PDF0 likes288 views
Immutable infrastructure wasn't suitable for the consultancy's needs as it led to long deployment times and a lack of visibility into instance configurations. Instead, they developed a hybrid approach using Packer, Puppet, S3, and AWS services that provides faster deployments, self-healing infrastructure, and a known, verifiable state for instances. This allows them to focus on application development rather than infrastructure management.
Immutable infrastructure isn’t the answer
- 1. Immutable Infrastructure Isn’t the Answer Sam Bashton
- 3. Who am I? • Sam Bashton • Ran a cloud (AWS + GCP) consultancy firm until 2016 when it was acquired by Claranet Group • Working with config management (Puppet) since 2007 • Working with AWS since 2009 • Working with GCP since 2014
- 5. What is this talk about? • How we tried to use immutable infrastructure • How and why it wasn’t right for us • What we do instead
- 6. Business Model • Charge customer for building super reliable infrastructure • Charge customers a monthly support fee
- 7. Hard won experience • Migrated over 1000 apps to public cloud • Variety of approaches to managing infra and deploying code - Including Immutable Infrastructure • 2011 onwards
- 11. Terraform
- 12. AWS Concepts • Each customer in one or more region • Each region has two or more data centres (‘availability zones’) - Most have three • SLA says that no more than one data centre will be down at once in a region “Region Unavailable” and “Region Unavailability” mean that more than one Availability Zone in which you are running an instance or task (one or more containers), as applicable, within the same Region, is “Unavailable” to you.
- 13. Data Lives in Services • We use AWS services to store all state - RDS (MySQL, Postgres, Oracle, MS SQL) - Elasticache (Redis) - DynamoDB - AWS Elasticsearch • The instances in question are ‘stateless’
- 15. Immutable immutable /ɪˈmjuːtəb(ə)l/ unchanging over time or unable to be changed
- 16. What is immutable infrastructure? • Automatically build a golden image • New infrastructure using the new image replaces the old infrastructure
- 17. Why would I want to do that? • Unit of deployment becomes a machine image • Test the artifact and have confidence it’ll be the same in production
- 20. Fudgetown • All the images are the same, except.. - We need to specify a different database location in each environment • And we need to specify it in an XML config file - We have different sizes of machine in each environment, and need to using different JVM settings
- 21. Why not just build lots of images? • Image building is automatic - why don’t we just build an image for each environment?
- 22. Why not just build lots of images? • Unit of deploy is a machine image • Images are created via an imperative set of commands - Shell Script - Ansible • What is in each image? What is different?
- 23. Immutable-ish • Scripts at startup handle differences • Consul cluster? - consul-template
- 24. Fudgetown
- 25. Fudgetown • Many dozens of microservices • All with configuration files - XML, yaml, ini, other
- 26. Fudgetown • Multiple processes make up a single ‘service’ • All have to be started in a specific order
- 27. Fudgetown • Deploying changes takes much longer - ~10-15 minutes for a Packer build and deployment to test infra • Tests on minor changes take a lot longer
- 28. Fudgetown • We don’t know what the state of our instances is, or should be • We don’t know whether config files were written successfully • It takes ages to test things
- 29. Back to the drawing board • Doing the thing the ‘cool kids’ say they are doing is not the path to technical success • Our customers care whether their app is working, not how
- 30. What do we actually need? • Infrastructure and configuration in a known and verifiable state • Self-healing • Fault tolerant - should continue to work even if a whole data centre (‘AZ’) fails
- 31. • Autoscaling which works every time • New instances which provision quickly (autoscaling) • Automated deployments - Possibly Canary, Blue/Green • Nice to have: quick to test changes What do we actually need?
- 32. Instance configuration in a known state • We need a way to describe configuration on the machine • A declarative language • Should tell us if something went wrong
- 34. Except.. • Puppet master doesn’t lend itself to an autoscaling world - Performance bottleneck bringing up new instances - Single point of failure - Especially in the zone failure scenario
- 35. The rules • Terminating an instance should always automatically give you a replacement which works - Even if external repos are down • CentOS mirrors • EPEL • Elasticsearch yum repo • Gem • Pip • We should expect data centre (‘AZ’) failure
- 36. How do we do it? • Packer - base common AMI • Puppet • S3 • yum/apt • Jenkins
- 37. Jenkins
- 38. Packer • Build a base image • Generally common to all roles • Sometimes will have per-role AMIs • pip/gem dependencies generally installed here - Easier than building a package, even with FPM • Install big RPMs here to save time at provisioning
- 39. Masterless Puppet • Put the Puppet manifests and modules on an instance • Run puppet apply
- 40. Distributing Puppet • Puppet needs to be on every instance • Build an RPM/DEB containing Puppet manifests/modules • Add to a RPM/DEB repo in S3 • Script at startup (cloud-init) installs Puppet • Puppet runs from systemd❤
- 41. External Repos • Mirror CentOS, etc repos in S3 • Repos are copied as part of deployment process - Dev repos continually updated - When code is promoted to next step (eg staging), repos also copied - OS upgrades are a part of the normal deployment process
- 42. Repos in S3 • Puppet, application code in yum repos in S3 • Repo created from a Terraform module • Just drop your RPM in, it handles metadata generation https://registry.terraform.io/ modules/claranet/s3-yum- repo/aws
- 43. Config updates • AWS provides SSM • SSM triggers updating Puppet RPM, running Puppet • ~120 seconds from commit to Puppet run finishing
- 44. Success • We have been using this approach for 6+ years • Tried other approaches • Always came back for apps unsuitable for containerisation
- 45. Your problems are not my problems • Have lovely 12 factor apps? • Why are you wasting time building infrastructure?!
- 47. Career advice • You don’t get paid to build infrastructure • ‘Serverless’ isn’t NoOps • Understanding distributed systems and their many failure modes the path to future success
- 48. Conclusions • Concentrate on the desired outcome, not what somebody at a conference said worked for them • Find the things that will give you the most success most easily, then iterate
- 49. • Architect for ease of management • Don’t be constrained by ‘best practice’ • Don’t be embarrassed by ‘ugly hacks’ when they solve real problems Conclusions