Uploaded byGeert van der Cruijsen
Chaos engineering - The art of breaking stuff in production on purpose
This document discusses chaos engineering, which is the practice of experimenting on a distributed system in production to build confidence in its ability to withstand failures. It describes introducing controlled failures or experiments to test a system's resilience. The key aspects covered are defining hypotheses about potential failures before experiments, designing and executing small experiments initially, learning from the results to identify issues, fixing any problems found, and embedding chaos engineering into the development process and culture. Patterns for building resilient systems like parallelism, async communication, and circuit breakers are also overviewed.
More Related Content
Similar to Chaos engineering - The art of breaking stuff in production on purpose
More from Geert van der Cruijsen
Chaos engineering - The art of breaking stuff in production on purpose
- 1. CHAOS ENGINEERING THE FINEART OF BREAKING STUFF IN PRODUCTION ON PURPOSE GEERT VANDER CRUIJSEN @GEERTVDC
- 2. GEERT VAN DERCRUIJSEN @GEERTVDC CLOUD NATIVE ARCHITECT #DOEPICSHIT FULL CYCLE DEVELOPER DEVOPS COACH
- 3. CHAOS ENGINEERING ? WHYDO WE NEED
- 7. “IN A COMPLEXLANDSCAPE YOUR APPLICATION IS NEVER FULLY UP”
- 8.
- 9.
- 10. MEASURE USER IMPACT RELIABILITY AVAILABILITYLATENCY THROUGHPUT CORRECTNESS FRESHNESS COVERAGE QUALITY DURABILITY
- 11.
- 12.
- 13. GRACEFUL DEGRADATION FAIL OPEN BUTWE DO TESTS?
- 14. BUT WE DOTESTS? UNIT A INPUT OUTPUT UNIT TESTS
- 15. BUT WE DOTESTS? COMPONENT / SERVICE A INPUT OUTPUT COMPONENT /SERVICE B INTEGRATION TESTS
- 16.
- 17. CHAOS ENGINEERING IS NOT RANDOMLYBREAKING STUFF IN PRODUCTION
- 18. CHAOS ENGINEERING “Chaos Engineeringis the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” https//principlesofchaos.org
- 19. CHAOS ENGINEERING “Chaos Engineeringis the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” https//principlesofchaos.org
- 20.
- 21. CHAOS ENGINEERING EXPERIMENTS HOSTFAILURE RESOURCE CAPACITY ATTACKS APPLICATION FAILURE NETWORK ATTACKS BRENT ATTACK
- 22. CHAOS ENGINEERING ONLY INPRODUCTION?
- 23. YOUR FIRST EXPERIMENT HOWTO START
- 24.
- 25. INCIDENT RESPONSE LEARNING OUTAGENORMAL DETECT& ANALYSIS FIX LEARNIMPROVE
- 26. CHAOS GAME DAY CHAOS EXPERIMENT NORMAL DETECT& ANALYSIS FIX LEARNIMPROVE
- 27.
- 28.
- 29. STEADY STATE MEASURE BUSINESSMETRICS 100ms extra load time drop Amazon’s sale by 1%
- 30.
- 31. STEADY STATE SERVICE UNDER TEST ROUTINGSERVICE B CONTROL SERVICE EXPERIMENT SERVICE
- 32. STEADY STATE SERVICE UNDER TEST ROUTINGSERVICE B CONTROL SERVICE EXPERIMENT SERVICE 98% 1% 1%
- 33. ALWAYS BE ABLETO ABORT
- 34.
- 35. DEFINE HYPOTHESIS BRAINSTORM WHATCAN GO WRONG BRING EVERYONE DEVELOPERS SRE /OPERATIONS NETWORKS BUSINESS INFRASTRUCTURE TESTERS WHAT CAN GO WRONG? WHAT IFDATABASE IS DOWN? WHAT IFSERVICE RESPONDS SLOWER? WHAT IFMY CACHE RESPONDS SLOW? WHAT IFA POD DIES? WHAT IF LOADBALANCER STOPS? WHAT IF….?
- 36. STOP IF YOUKNOW THE EXPERIMENT WILL BREAK
- 37. DESIGN & EXECUTEEXPERIMENT STEADY STATE DEFINE HYPOTHESIS DESIGN& EXECUTE LEARN FIX EMBED
- 38. DESIGN & EXECUTEEXPERIMENT START SMALL NOTIFY PEOPLE INVOLVED SLOWLY INCREASE BLAST RADIUS TOOLS: GREMLIN.COM CHAOSTOOLKIT.ORG GITHUB.COM/NETFLIX/SIMIANARMY GITHUB.COM/ASOBTI/KUBE-MONKEY
- 39.
- 40. LEARN HOW FAST DIDWE RECOVER? HOW FAST DID WE DETECT? DO NOT BLAME!
- 41.
- 42.
- 43.
- 44.
- 45.
- 46. MULTI PARALELLISM PARALLELISM AVAILABILITYDOWNTIME PER YEAR 1 99% 3 DAYS 16 HOURS 2 99,99% 53 MINUTES 3 99,9999% 32 SECONDS HOW PARALEL IS YOUR CLOUD COMPONENT ? REGIONSAVAILABILITY ZONES
- 47. ASYNC COMMUNICATION SYNC REQUIRESA CONNECTION PER REQUEST FOCUS ON MESSAGE BASED COMMUNICATION DECOUPLING PUB SUB LISTENER
- 48. QUEUE BASED LOADDISTRIBUTION
- 49. QUEUE BASED LOADDISTRIBUTION
- 50. QUEUE BASED LOADDISTRIBUTION SERVICE BUS
- 51. IDEMPOTENT APIS HTTP METHODIDEMPOTENCE SAFETY GET YES YES HEAD YES YES PUT YES NO DELETE YES NO POST NO NO PATCH NO NO
- 52. BULKHEAD PATTERN ISOLATE WORKLOADSLIKE THE HULL OF A SHIP
- 53.
- 54. CIRCUIT BREAKER ADD JITTERTO RETRIES
- 55. SPLIT RESPONSIBILITIES READ /WRITE SHARDING CQRS
- 56. WRAP UP BIG CULTURECHANGE FULL CYCLE DEVELOPERSPRODUCTION ACCESS START EXPERIMENTING START SMALL CHECK OUT TOOLSOBSERVABILITY
- 57. “CHAOS ENGINEERING DOESN’TCAUSE PROBLEMS, IT JUST REVEALS THEM” NORA JONES – CHAOS ENGINEERING LEAD SLACK
- 58. GEERT VAN DERCRUIJSEN @GEERTVDC THANK YOU!ALL PICTURES USED ARE FROM UNSPLASHED.COM RESOURCES BOOKS: Chaosengineering-O’Reilly Chaosengineeringobservability -O’Reilly TOOLS: chaostoolkit.org gremlin.com github.com/netflix/simianarmy github.com/asobti/kube-monkey RESOURCES: principlesofchaos.org github.com/dastergon/awesome-chaos-engineering docs.microsoft.com/en-us/azure/architecture/patterns/category/resiliency
Editor's Notes
- #4 Why? Easy you can put “chaos engineer” as function title on your resume Who did fail over test to other data center?
- #6 We’ve started using the cloud and building distributed applications Deathstar of amazon
- #8 Who did fail over test to other data center?
- #9 How do we monitor this kind of stuff?
- #10 Netflix SPS
- #11 Netflix SPS It has been reported that every 100ms of latency costs Amazon 1% of profit
- #12 INFRA: cloud is providing this for us right? NETWORK: The network is always reliable? We know it is not. How about switching over? How do we test that? It’s often one of the easiest APPLICATION: how do applications hold up when errors occur? What if the database is not accesible? PEOPLE: People intervention. Is that making it wose? Fire drills. Do we fire drill for IT?
- #13 Partial failure mode You have to think TOGETHER with business of the impact of failure. Fail open example
- #14 Partial failure mode
- #15 Partial failure mode
- #16 Partial failure mode
- #17 Chaos engineering is like vaccination. We add small amounts of harm to make the full system more immune to the effects
- #19 Why? Easy you can put “chaos engineer” as function title on your resume
- #20 Why? Easy you can put “chaos engineer” as function title on your resume
- #21 Partial failure mode
- #25 incident-response learnin
- #26 incident-response learning MTTR
- #27 incident-response learnin
- #47 MULTI REGION, MULTI AVAILABILITY ZONE
- #52 Idempotency: can i do the same thing with same effect? Safety: does it change the end state?
- #53 Example in kubernetes fixed CPU / memory reservations Not 1 application can kill others by using max CPU/MEMORY
- #54 Fusebox When things go wrong stop retrying
- #55 Polly
- #56 Command and Query Responsibility Segregation
- #60 Chaos engineering is like vaccination. We add small amounts of harm to make the full system more immune to the effects
- #61 Why? Easy you can put “chaos engineer” as function title on your resume