A chaos experiment a day, keeping the outage away

one chaos experiment a day.
keep the outages away.
yan cui,@theburningmonk

MUST KILL SERVERS!
RAWR!!
RAWR!!

@theburningmonk theburningmonk.com
“the discipline of experimenting on a system in order to build conﬁdence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org

microservices death stars circa 2015

“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun

“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun
it’s not about
preventing failures!

everything fails, all the time

“You don't choose the moment, the moment chooses you!
You only choose how prepared you are when it does.”
Fire Chief Mike Burtch

anything that can go wrong, will go wrong.
MURPHY’s LAW

identify weaknesses before they manifest in system-wide, aberrant behaviors
GOAL

learn about the system’s behavior by observing it during a controlled experiments
HOW

learn about the system’s behavior by observing it during a controlled experiments
HOW
game days
failure injection

Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

Yan Cui
@theburningmonk
http://bit.ly/yubl-serverless

Yan Cui
@theburningmonk
Developer Advocate @

Yan Cui
@theburningmonk
Independent Consultant
advisetraining delivery

“using serverless reduces the blast radius”
www.buzzsprout.com/877747/4615985

serverless improves resilience
as platform takes care of infrastructure failures

by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

Shared Responsibility Model

chaos monkey kills an
EC2 instance
latency monkey induces
artiﬁcial delay in APIs
chaos gorilla kills an AWS
Availability Zone
chaos kong kills an entire
AWS region

there are no servers to kill!
SERVERLESS

improperly tuned timeouts

missing error handling

missing fallbacks

STEP 1.
deﬁne steady state
i.e. “what does normal look like”

STEP 2.
hypothesis that steady state continues in control and experimental group
e.g. “the system stays up if a server dies”

STEP 3.
inject realistic failures
e.g. “slow response from 3rd-party service”

STEP 4.
try to disprove hypothesis
i.e. “look for difference between control and experimental group”

latency inject latency to function invocation

“what if service X has elevated latency?”

API Gateway Lambda API Gateway Lambda

hypothesis: API would timeout and our try-catch
would handle it and return default response

result: function times out after 6s
(hypothesis is disproved)

502
200

3s timeout
6s timeout

max 29s integration
max 15 mins timeout

and then there’s
cold starts…

TIL: most HTTP client libraries have default timeout of 60s.
API Gateway has an integration timeout of 29s.
Most Lambda functions default to timeout of 3-6s.
Don’t forget about the cold starts!

https://bit.ly/2Wvfort

outcome: a more resilient system

latency
exception
inject latency to function invocation
throws exception

latency
exception
statuscode
throws exception
return HTTP status code

latency
exception
statuscode
diskspace
throws exception
ﬁlls up /tmp directory

latency
exception
statuscode
diskspace
denylist
throws exception
ﬁlls up /tmp directory
looses network connectivity

“what if DynamoDB has an elevated error rate?”

API Gateway Lambda DynamoDB

hypothesis: the AWS SDK retries would handle it

TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms

TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms
delay = Math.random() * (Math.pow(2, retryCount) * base)
this is Marc Brooker’s
fav formula!

action: set max retry count + fallback

https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

@theburningmonk
theburningmonk.com
github.com/theburningmonk

A chaos experiment a day, keeping the outage away

More Related Content

Similar to A chaos experiment a day, keeping the outage away

More from Yan Cui

Recently uploaded

A chaos experiment a day, keeping the outage away