SELF 2011: Deploying Django Application Stacks with Chef

Editor's Notes

• #2: \n
• #3: \n
• #4: \n
• #5: \n
• #6: \n
• #7: \n
• #8: Managing infrastructure with Chef.\n
• #9: How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
• #10: How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
• #11: How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
• #12: How’s and why’s of managing infrastructure with Chef.\nA quick overview of Chef\nWe’ll talk about the steps involved for automatically building our n-tier Django stack\nWe’ll discuss how your Chef code is reusable across environments\n\n
• #13: Why not just use some shell scripts?\n
• #14: \n
• #15: This is the dream. When everyone’s 2nd girlfriend ‘nagios’ wakes you up in the middle of the night you are ready for the worst.\n
• #16: Keep track of all the steps required to take bare metal systems to doing their job in the infrastructure.\n\nIt is all about the policy. This can be lo-fi (ie a wiki page and the ‘meat’ cloud) or fully automated with something like Chef.\n\nUsually this is the context of a single node.\n
• #17: Taking all the systems that have been configured to do their job, and make them work together to actually run the infrastructure.\n\nWRT Chef, we talk about Fully Automated Infrastructure.\n\nThis brings all the single nodes together.\n
• #18: Introducing Chef.\n\nMaybe you’ve already met!\n
• #19: Chef provides a framework for fully automating infrastructure, and has some important design principles.\n
• #20: Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
• #21: Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
• #22: Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
• #23: Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
• #24: Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. “Tim Toady”\n
• #25: Idempotent: Consistency, known system state, easy to build\nChef takes idempotent actions to put the system in the desired state.\n
• #26: Chef makes it easy to reason about your infrastructure.\n\nWe build an API, with primitives and libraries, and let you model your data. We don’t try to discover because we might make an incorrect assumption about the system.\n\nYou need an API to manage your infrastructure. The standard tools on Unix and Windows already use APIs to the system. Use an API to manage your infrastructure.\n
• #27: Chef comes from the perspective that the tool should do pretty much what you expect, hence sane defaults. To get there we had express some opinions. Those opinions can be changed easily, whether that is modifying the configuration of the chef-client, or how Apache is set up.\n
• #28: Chef is a flexible framework. You can make it do pretty much anything you want. Just like Perl doesn’t tell the programmer how to program, Chef doesn’t tell the system administrator how to manage the system. You can extend it, mold it and modify it to do what you need.\n\nBeing written in Ruby, using Ruby for the main language was very purposeful. The tool shouldn’t get in your way.\n
• #29: \n
• #30: -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
• #31: -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
• #32: -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
• #33: -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
• #34: \n
• #35: -thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
• #36: -thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
• #37: -thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
• #38: -thin server...fat clients => all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
• #39: \n
• #40: Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
• #41: Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
• #42: Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
• #43: Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
• #44: Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
• #45: Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
• #46: \n
• #47: \n
• #48: Communication with the server is secure. Clients hold the private key, the server has the public key. The server is not a centralized store of authentication information, pushing the responsibility to the nodes.\n
• #49: \n
• #50: \n
• #51: \n
• #52: Manage system configuration as resources.\nPut resources together in recipes.\nDistribute recipes with cookbooks.\nAssign recipes to systems through roles.\nStore arbitrary data in data bags.\nTrack it all like source code.\n
• #53: Resources are an abstraction we feed data into. When you write recipes in Chef, you create resources of things you want to configure.\n
• #54: Resources declare a description of the state a part of the node should be in.\n
• #55: Resources declare a description of the state a part of the node should be in.\n
• #56: Resources declare a description of the state a part of the node should be in.\n
• #57: Resources declare a description of the state a part of the node should be in.\n
• #58: Resources declare a description of the state a part of the node should be in.\n
• #59: Resources declare a description of the state a part of the node should be in.\n
• #60: Resources declare a description of the state a part of the node should be in.\n
• #61: Resources declare a description of the state a part of the node should be in.\n
• #62: Resources declare a description of the state a part of the node should be in.\n
• #63: Resources declare a description of the state a part of the node should be in.\n
• #64: Resources declare a description of the state a part of the node should be in.\n
• #65: Resources declare a description of the state a part of the node should be in.\n
• #66: Resources declare a description of the state a part of the node should be in.\n
• #67: Resources declare a description of the state a part of the node should be in.\n
• #68: Resources declare a description of the state a part of the node should be in.\n
• #69: Resources declare a description of the state a part of the node should be in.\n
• #70: Resources declare a description of the state a part of the node should be in.\n
• #71: Resources declare a description of the state a part of the node should be in.\n
• #72: Resources declare a description of the state a part of the node should be in.\n
• #73: Resources declare a description of the state a part of the node should be in.\n
• #74: Resources declare a description of the state a part of the node should be in.\n
• #75: Resources declare a description of the state a part of the node should be in.\n
• #76: Resources declare a description of the state a part of the node should be in.\n
• #77: Resources declare a description of the state a part of the node should be in.\n
• #78: Resources declare a description of the state a part of the node should be in.\n
• #79: Resources declare a description of the state a part of the node should be in.\n
• #80: The abstraction over the commands or API calls that will configure the resource to be in the state you have defined.\n
• #81: Providers know how to actually configure the resources to be in the declared state\n\n
• #82: \n
• #83: \n
• #84: Just like recipes themselves are processed in order, the recipes included are processed in order, so when you include a recipe, all its resources are added to the resource collection, then Chef continues to the next.\n\n\n
• #85: \n
• #86: \n
• #87: \n
• #88: \n
• #89: \n
• #90: \n
• #91: \n
• #92: -the cpan, PyPI or rubygems.org of the cookbbook world!\n
• #93: \n
• #94: \n
• #95: \n
• #96: \n
• #97: \n
• #98: \n
• #99: \n
• #100: \n
• #101: \n
• #102: \n
• #103: \n
• #104: \n
• #105: Data bags are containers of grouped information.\n\nUsers, application information, network info, cabinet/rack locations. Describe components of your infrastructure with data, and use that data to configure systems.\n\nProcess data bags in recipes to configure systems dynamically.\n
• #106: \n
• #107: \n
• #108: In our example we are deploy Django CMS but this would apply to any Django application.\n
• #109: Provisioning is the first step. We need some computers on the internet. They’re going to be load balancers, webservers, database servers. Launch those with a cloud API. Every cloud does this.\n
• #110: You need to do something and you manage the configuration. Install packages, create users, etc. You can use pre-built images for this, but then just for a simple infrastructure (w/o monitoring/trending, middleware, etc) you have four discrete images to maintain for updates both for the OS and app updates.\n
• #111: This is the last mile. The systems are configured, but also need to be configured to talk to each other.\n
• #112: \n
• #113: \n
• #114: \n
• #115: \n
• #116: Every database backed application needs a master database. For this simple example we haven’t done any complex setup of master/slave replication, but the recipes are built such that this would be relatively easy to add.\n
• #117: The database master recipe will read the application information from the data bag and use it to create the database so the application can store its data.\n
• #118: all done fully automated\n
• #119: all done fully automated\n
• #120: all done fully automated\n
• #121: We start with one but scale horizontally as needed. We can even scale up and down based on demands.\n
• #122: The main thing in this role is the application recipe. \n\nThe recipe will read in data from the data bag (in a predefined format) to determine what kind of application to deploy, the repository where it lives, details on where to put it, what roles to search for to find the database, and many more customizable properties.\n\nWe launched two of these to have something to load balance :).\n
• #123: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #124: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #125: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #126: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #127: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #128: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #129: -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
• #130: You no longer need to track which system has an IP that should be applied as the database master. We can just use its fqdn from a search.\n
• #131: \n
• #132: We’re using haproxy, and we’ll search for a specific application to load balance. The recipe is written to search for the django_cms role to find systems that should be pool members.\n
• #133: \n
• #134: -This search occurs everytime Chef runs on the load balancer.\n-If you add more app servers to the mix, the load balancer will *automatically* become aware!\n
• #135: A single set of Chef cookbooks, roles and databags allows you to deploy your application stack in numerous ways.\n
• #136: \n
• #137: \n
• #138: \n
• #139: \n
• #140: \n
• #141: \n
• #142: \n
• #143: \n
• #144: https://github.com/opscode/cookbooks/tree/master/application\nhttp://www.opscode.com/blog/category/quickstarts/\n
• #145: Before we go to questions, here’s some resources to be aware of.\n
• #146: \n
• #147: \n