Dynamic Infrastructure and The Cloud

Editor's Notes

• #2: Dynamic Infrastructure and The CloudAdventures in Keeping Your Application Running…at Scale AWS Summit - Sydney, Australia Lee Atchison ∙ Senior Director Strategic Architecture at New Relic, Inc.
• #4: I’d like to tell you a story. Does this story sound familiar to you?
• #5: It’s Sunday. The day of the big game. You’ve invited 20 of your closes friends over to watch the game on your new 300” ultra max TV. Everyone has come, your house is full of snacks and beer. Everyone is laughing. The game is about to start. And…
• #6: …the lights go out……the TV goes dark……the game, for you and your friends, is over.
• #7: Obviously disappointed, what happened? You decide to pick up the phone and call the local power company.
• #8: The representative, unsympathetically, says: “We’re sorry, but we only guarantee 95% availability of our power grid.” They could not understand why you were complaining, after all you had power “most of the time”. Why is availability important? * Because your customers expect your service to work…all the time. * Anything less than 100% availability can be catastrophic to your business.
• #9: A hope and a prayer… Laugh at it, but more people do this than you might expect.
• #10: Keeping your application running is possible. I will discuss three points to making it happen. {c} First, availability…is more than you think it is...
• #11: I want to tell you about an overheard OPs conversation. I want you to tell me if this sounds like something you’ve heard yourself in your OPs organizations…
• #12: We were wondering how changing a setting on our MySQL database might impact our performance…
• #13: … but we were worried that the change may cause our production database to fail…
• #14: … Since we didn’t want to bring down production, we decided to make the change to our backup (replica) database instead…
• #15: … After all, it wasn’t being used for anything at the moment.
• #16: Until…of course...the backup was needed...
• #17: Does this story sound familiar? This exact story is a true story, and unfortunately is not uncommon.
• #18: Availability issues such as I described here may seem obvious…but many are much more subtle. For example...
• #19: Imagine we are a e-commerce website. We’ve got a mobile app that can purchase items in ourshop. {C} Bob uses his phone, buys something, and it takes 300ms. That’s great! {C} Sally logs in, buys something, but the database is slow. It takes much longer. She is not a happy customer. Availability is not just whether a page responds, but how long it takes to respond. The customer doesn’t care why a problem occurred, they don’t care why your app is slow. If it doesn’t meet their expectations at a time they expect, nothing else matters…
• #20: But keeping your application available can be tough. It may be fuzzy. Performance may be good for some users, and bad for others. But, can you even detect this, or do you just show that, on average, your site is doing fine? The real answer to how your application is doing is not a hope and a wish. It’s in the details. It’s in the data.
• #21: Modern application monitoring can’t be done by simply looking from the outside in. It can’t be done with averaged or sampled data. You must collect data from all areas of your application, and from all transactions. You must collect tons and tons of data. ---
• #22: In fact, you typically need to collect more monitoring data than data that is within your application. And it grows continuously, every day, every second. Everything that anyone does on your application, generates performance data. If anybody is using your application, you must collect data about exactly how they are using it and how the infrastructure behind it works together. All of it is important.
• #23: All parts of your application, from your servers thru your apps, to the business outcomes they represent {C} All generate data that you must analyze together.
• #24: So, you expect your site to be up. And when it is down, what do you do? Do you look to attach blame? No, you want to find the problem.
• #25: You want to know what happened.
• #26: To know what happened, we need data. We need data from every level of our application. Here is a typical, simple, web application. It consists of an application and some services. It consists of servers running an operationg system, and they consist of virtual hardware that all that runs on. They may also run in our customers browsers, or in their mobile applications.
• #27: Often people think that all they need is low level virtual hardware monitoring. They monitor their instances using tools like CloudWatch. But CloudWatch provides a very limited view of the world. You get virtual hardware level information, but that’s about it. You don’t even get information about the operating system, memory, processes, or system configuration. And you absolutely get no information about your application.
• #28: To know how your application is really performing. You need an application performance monitoring tool. You also need to know how the rest of your infrastructure is running (the operating system for instance). You also need to know how your remote application, such as those running on mobile devices or your customer’s browsers are running.
• #29: To monitor the application, you need full stack performance monitoring.
• #30: Because if you don’t monitor the data you need at the time you need it. You’ll: 1) Waste time fire fighting, 2) Meaningless finger pointing across teams, 3) Lose money, 4) Make customers unhappy, 5) Unhappy customers tell other people…
• #31: You also need the right data. You need to know how your application is performing, to answer questions as simple as, “Am I actually open for business?”. But you also want to know how easy it is for your customers to make use of your application. What is their experience? And you need to know how your business is doing. You need to monitor the right components…and you need to monitor the right data. Success involves all three types of analytics. Is the software working? Is it meeting the customer’s needs? Is it meeting your business needs? All of these three things are interconnected.
• #32: Because, avoiding this is critical to every business.
• #33: Point 2, there are technologies that can help you keep your application running…technologies such as the dynamic cloud. How do I mean? Let’s take a look.
• #34: How can the cloud help? Well, it turns out that there are two fundamental ways people make use of the cloud. The first is to use the cloud as a “Better Data Center”. The second is to use the “Dynamic Nature” of the cloud to build better apps faster. I’m going to talk about each of these methods.
• #35: Let’s first look at using the cloud as a “Better Data Center”.
• #36: What do I mean by using the cloud as a “Better Data Center”? I mean:* Resources are allocated to uses, just like in a regular data center <click> * The provisioning process for new resources, though, is significantly faster <click> * The lifetime of the resources you create is relatively long…usually measured in days, weeks, months, or years. <click> * However, even with a faster provisioning process, traditional “capacity planning” is still important and still applies.
• #37: Why would we want to use the cloud simply as a “better data center”? What are the benefits to us building applications? Since we can add new capacity faster, we can build and scale our applications easier in the cloud. In addition to adding servers easier and quicker, we can add entire new data centers easier, which can improve our application availability and redundancy. Additionally, this ability to add additional data centers can improve our compliance, especially when it comes to things like EU Safe Harbor laws.
• #38: So, now, let’s switch to talking about using the cloud in a dynamic environment.
• #39: What do I mean by using the cloud as a “dynamic tool for dynamic applications”? I mean:   Use only the resources you need <click> * Allocate and deallocate resources on the fly <click> * Resource allocation becomes an integral part of your application architecture.
• #40: In a dynamic application, resources are allocated, consumed, and deallocated on the fly. And the application is aware of and is controlling this management of resources. The application is essentially performing traditional OPs resource management tasks.
• #41: New Relic did an analysis recently about how our customers are making use of Docker. The question we wanted to answer was, how long do docker containers live? This diagram shows the answer to that question. The horizontal axis is the number of hours a docker container has lived for, and the vertical axis is the number of containers in that time bucket. As you can see, there is a long tail, with some docker containers running for well over a year. However, there is a huge number of docker containers that run for less than one hour. In fact, if we zoom in on just that one hour time period…
• #42: we can see that most docker containers we run actually only run for less than one minute! Over 11% of all docker containers we run will run for less than 60 seconds.   This is some customer’s application or service, some business logic, that starts up, runs, and shuts down all within 60 seconds. This is very rapid. These are containers that are launched only for a specific business purpose and are terminated when that purpose is completed. This is what we mean by dynamic infrastructure.
• #43: And there are lots of different cloud technologies that can be used in this dynamic manner…from queues to routing to auto scaled EC2 instances. Many resources in the cloud can be used in this dynamic fashion.
• #44: The dynamic cloud allows you to build better applications, faster. The way you’ve done things in the past won’t work in the future.
• #45: Change happens faster in the cloud. This is because of dynamic servers, dynamic infrastructure, and, more recently, {c} the cloud is even more dynamic due to technologies such as AWS Lambda.
• #46: What is Lambda? Lambda is one of many technologies that implement what’s called “Functions as a Service” or “Compute as a Service”. You might also know it as “Serverless”, but that is not as accurate of a description of it. Lambda allows creating microcomputing environments. This allows creating highly dynamic and incredibly scalable functions that can be executed without the need to provision any infrastructure what-so-ever. They provide automatic scaling using a massively shared infrastructure.
• #47: In a nutshell, AWS Lambda simply takes an event from some AWS resource. This is called the “trigger”. This event can be something like an object being updated in an S3 bucket…or a database update in DynamoDB, or a call to an API Gateway. Some sort of event within the AWS ecosystem.
• #48: Lambda takes that event and creates an instance of a Lambda function, on the fly, that can process that event.
• #49: The processing is usually a very simple action...something like updating another object in S3, or responding to the API Gateway request...whatever action the lambda script was designed to execute in response to that trigger.
• #50: Any number of triggers can occur as fast as possible, and multiple instances of the lambda function will automatically be created to handle all of the concurrent events, instantly scaling the function to as many instances as is necessary to handle all events as quickly as possible. This automatic scaling is designed to be transparent to everyone, including the customer who created the script. This is the definition of near infinite scaling.
• #51: Building dynamic infrastructures in the cloud allows you to {c} scale your applications better. {c} It also allows you to make changes to your application faster and easier. {c} Both of these ultimately result in higher availability… But only if you know what your application is actually doing…
• #52: (But only if you know what your application is actually doing…) This brings up an interesting concern. In a dynamic cloud, you have dynamic resources. Resources that are coming and going fast. Instances are starting and stopping. Containers are coming and going. And functions are executing and terminating. If resources are coming and going so fast, how can you monitor them? How do you monitor a dynamic application in a dynamic cloud?
• #53: Here is an example of a dynamic application. It looks much like the static application. It might have more services and microservices that compose the application, this is typical of a more modern application.
• #54: We still have AWS CloudWatch monitoring the low level cloud infrastructure.
• #55: And we still have traditional application performance monitoring that monitors the static nature of the application components.
• #56: Overall, this provides **almost** top to bottom monitoring of the entire application.
• #57: But what about this piece? How do you monitor the provisioning process itself? Given that resources are coming and going regularly, how do you monitor that?
• #58: How do you monitor components that are there one moment, but less than 60 seconds later, they are gone? <click> Remember the docker information…
• #59: It turns out that monitoring a dynamic application in a dynamic cloud is very different than monitoring traditional data center components.   You must of course still monitor each of the cloud components themselves…each of the services and resources and components that make up your application. {c} But you also must monitor the lifecycle of the cloud components. This is because it matters not only **that** a resource was used, it matters **when** that resource was used. Because just looking at the resources running right now is inadequate when trying to diagnose a problem from even a few minutes ago. The resources that were in use when the problem occurred are **not** the same resources in use now.
• #60: So, in the old world, your operations team was comfortable. They knew the resources they controlled, they created them, they managed them. All was simple and manageable.
• #61: But in this new world, resources are created and destroyed dynamically. The world of the operations team can no longer be as simple as tracking resources on a spreadsheet. The resources they are responsible for are dynamic and transient. Their world has gotten a lot more complicated.
• #62: This change is inevitable. The change is needed because our customers are expecting more and more from our applications. The change is needed because our customers are expecting better and more reliable performance from our applications. The change is inevitable because to meet the needs of our customers, our organizations must grow quickly and build applications that are more reliable than ever before. The cloud helps achieve this, and this more and more the reason why moving to the cloud is so important for us.
• #63: The third point, is getting to the cloud. Migrating to the cloud is easy, right?
• #64: How do we move to the cloud? Often, we start our migration to the cloud with lofty expectations. But we find out that moving to the cloud isn’t necessarily as easy as we would like it to me. Problems occur. The cloud doesn’t meet our expectations that have been promised to us.{c}There is pressure to declare ”victory” before we are ready.{c}Promised performance gains are not occurring. Costs run out of control.{c}And schedules just don’t matter anymore.{c}How can we meet our promises to our stakeholders if we can’t get the cloud to do what we want it to do? Most companies moving to the cloud struggle with this. Some struggle more than others. Some fail to overcome the struggle.
• #65: But moving to the cloud does not have to be scary or dangerous. It can be done safely, but you must be willing to learn as you go. Learn and adapt the cloud to meet your company’s needs, and learn and adopt your expectations to the reality of what the cloud can offer.
• #66: Let’s take a look at how most enterprises figure out how to migrate to the cloud. There are six *typical* steps that most companies take to move to the cloud. They don’t all use all the steps. Some stop part way up the path. Some skip steps. But this is typical… Let’s look at each of these in turn.
• #67: Let’s start with “Experiment”.
• #68: This is the first, tentative step into the cloud. It involves using safe technologies. Technologies that we can use in simple and subtle ways in parts of our applications that may be less critical. There are no cloud policies created. We just build one off implementations to see how the cloud can fit into our needs. Most companies have at least started on this step.
• #69: After you’ve done some basic “feet wetting” in the cloud, security typically becomes a concern.
• #70: Critical evolution point in the company’s culture …all displines in the company are involved (Legal, Finance, Security) …companies that can’t get past this point, can’t be successful in the cloud
• #71: Once policies are in place and the cloud can be trusted…you start using other features the cloud has to offer.
• #72: Three choices: ...1) Put some workloads in the cloud, some in your own data center ...2) Resiliency - additional data center(s) ...3) Move applications to the cloud, out of existing data centers Independently: SaaS uage increases (internal apps first)
• #73: Now the cloud is important to you, so you start to see what else the cloud can do for us.
• #74: ”Managed Services”
• #75: Now, we start looking at cloud native services…services only available in the cloud.
• #76: Point of commitment…now dependent on the cloud
• #77: So now we are committed to the cloud…now comes the last step. Mandated use.
• #78: Mandate use of the cloud >>>Typically wanting to get out of the data center business Netflix, etc
• #79: The steps aren’t easy…
• #80: But ultimately, these are the steps involved. Different companies go thru these steps at different speeds. Different companies find the right “stopping point” that matches their needs
• #81: While these are the steps our *company* may go thru. As we build new and migrate existing applications, our applications go thru a similar learning process… How can a given application take advantage of the cloud? This adoption may happen faster or slower for different types of applications.
• #82: Let’s take a look at these as two different axis on a chart. Coporate adoption process on the left, Application adoption process on the bottom Another way to look at this: based on application types and requirements...
• #83: So we can see we are more likely to use the “newer” technologies, such as Lambda, in new applications. But we are much less willing to use these technologies in our more business critical applications.
• #84: There exists a sweet spot… >Corporate adoption is strong, but not “mandated” >Application adoption is strong, but not “committed” *This is the destination for a lot of companies and applications
• #85: Very near some of the common, core AWS services
• #86: So, that’s all great data. I know I need to move to the cloud to keep my company moving forward. But what about the nuts and bolts. What should *I* do to be successful in moving to the cloud?
• #87: How can I make sure a cloud migration is successful? Understand where your culture is Risk tolerance, Cloud commitment, Expertise Understand your needs Redundancy? Cost? New Opportunity? Consciously plan your acceptance What level are you? What level do you need to be? Drive your culture to where you feel you need to be Monitor your adoption
• #88: Before migration Baseline application Servers Databases Caches Applications Microservices Determine your steady state Important before you migrate!
• #89: During migration Incorporate Cloud’s internal monitoring …provides cloud specific infrastructure monitoring …AWS CloudWatch Continue application monitoring *Here, looking for performance deviations from steady state Track down & explain all deviations before moving on Understand all deviations from norm Solve problematic deviations/problems
• #90: Deviation in performance before and after migration give us a clue to migration related issues
• #91: Continue monitoring post migration Should understand: The infrastructure is now out of your control…you need to keep an eye on it Cloud infrastructure changes can impact your application…you need to keep an eye on it There are some cloud specific concerns: EC2 instance failures Greater part of your availability plans Often impacts other AWS systems as well Instance degradation (more common than you’d think) Ongoing application & infrastructure monitoring is essential
• #92: APM, Insights, Browser, Synthetics
• #93: So, that’s the third point in keeping your application running at scale…successful cloud migration. {c}Together, these three points can keep your application highly available and running at scale. {c}And underlying all three is monitoring your application and your infrastructure.
• #94: It used to be, long ago, that all it took to make sure an application was running was to look at the server. Did the amount of CPU or memory utilization change recently? If it did, there might be a problem. Everything was static, everything was smooth. Everything was constant. A change indicated a problem.
• #95: But in this new world, resources are created and destroyed dynamically. The world of the operations team can no longer be as simple as tracking resources on a spreadsheet. The resources they are responsible for are dynamic and transient. Their world has gotten a lot more complicated.
• #96: In order to monitor your dynamic applications in the dynamic cloud, you must monitor all aspects of your application, top to bottom, using a full stack monitoring solution, a solution such as New Relic.
• #97: Dynamic applications require dynamic scaling and use of dynamic technologies. (how many streams during each day?)
• #98: Our customers won’t stand by waiting for us to solve availability problems.
• #99: And panic is not the solution. Nor is blame.
• #100: The dynamic cloud has caused significant change to our world. Our world has sped up, and the rate of change in application development has increased. The cloud alone has speed things up, and the dynamic cloud has sped things up even more. The way you’ve done things in the past just won’t work in the future. This is good…but it is also scary.
• #101: In order to monitor your dynamic applications in the dynamic cloud, you must monitor all aspects of your application, top to bottom, using a full stack monitoring solution, a solution such as New Relic.
• #103: Let’s be careful out there…