Lessons from Branch's launch

Editor's Notes

• #2: Hi Im andrew blah blah blah\n
• #3: What is branch anyway?\nThink of a dinner table conversation on the web\n\nIn between a blog post, 1 person, and a forum open to anyone\n\nWe’re trying to create a place for thoughtful communication\nSo, \nhow did we build it??\n\n
• #4: We’re running a pretty standard Rails app off of heroku,\nPostgres for our DB, Redis on EC2 for caching and misc. key value\nCoffee/Haml for the frontend\nHow do we monitor it?\nadd ec2\n
• #5: a few tools Using statsD on ec2 paired with datadog for most of our app stats - \n\netsy developed statsD as a metrics aggregator-\n we send over all our metrics from logs and internal events to statsd, and it gets aggregated and sent to datadog\n\nddog makes it very easy to define and track custom metrics , \nand alerts\n\nTo keep tabs on DB< New Relic through heroku\n\nTo keep an eye on async tasks, Resque web app (more on that later)\n
• #6: We learned alot during our launch, tonight I’m going to talk about the three biggest take aways we got from it.\n\nThese aren’t branch-specific lessons, should be applicable to any consumer-facing rails app \n
• #7: Probably stating the obvious, but we can’t predict the future as much as we might like to believe.\n\nThe week before launch we thought about what to optimize.\n\nWe set up a waitlist system so we could throttle the number of people using our site\n\nwhat about people who are using our site?\n\n“Creating branches!” Since we’re granting 100s of ppl access, that will cause a bunch of load- lets optimize that\n\nopened the floodgates\n
• #8: New Relic ++\nThe ability to drill down into your database load made optimizing so much easier\n\nNR catalogues every request and database transaction, \nWe can examine the stack traces for slow queries,\nView response time geographically\n\nBeyond just which transactions are slow- which components of those transactions are slow?\nare you stuck in ruby, or doing too many sql calls.\n\nSo as we looked at our db load during the first few days,\n
• #9: This is what we set up to throttle load!, but it was regularly making up >20% of our database operations\n\nWHY??\n\nWhat does the validates_uniqueness do?\nPostgres scans for matches, so if there are not indexes, you are doing close to a full table scan!\n\nIn the end, the public facing actions that we weren’t throttling were the biggest load\n\nFirst day we didn’t even invite anyone since we were just struggling to keep up with waitlist signups\n\nsolution?\n
• #10: We needed to optimize those validations, one specific culprit was the email uniqueness\nwe added a functional index on the email column\nwe LOWERCASE FUNCTION t\n\nsidenote: active record doesn’t support creating functional indexes, hence the explicit sql\n\nthis also means we needed to store our scheme as a structure.sql dump instead of schema.rb\nin order to reconstruct it\n\nSo that was the first thing we learned\n-- Optimizing based on assumptions is going to be wrong sometimes, so be ready for that and have the tools to deal with it.\n
• #11: Everyone loves async right?\n\nAlot of our app happens asynchronously at branch, \neverything from sending emails to pushing realtime updates,\n to firing off tweets\n\nsidenote -- give you a rundown of how we handle that\n
• #12: We used DJob until a few weeks before launch\n\nDelayedJob is great and super easy to work with,\nin the end we wanted more granular control over our background tasks since a lot of stuff happens there\n\n_different queues, priorities, complete visibility into them , failed jobs etc.\n\n\n
• #13: We used DJob until a few weeks before launch\n\nDelayedJob is great and super easy to work with,\nin the end we wanted more granular control over our background tasks since a lot of stuff happens there\n\n_different queues, priorities, complete visibility into them , failed jobs etc.\n\n\n
• #14: We used DJob until a few weeks before launch\n\nDelayedJob is great and super easy to work with,\nin the end we wanted more granular control over our background tasks since a lot of stuff happens there\n\n_different queues, priorities, complete visibility into them , failed jobs etc.\n\n\n
• #15: We used DJob until a few weeks before launch\n\nDelayedJob is great and super easy to work with,\nin the end we wanted more granular control over our background tasks since a lot of stuff happens there\n\n_different queues, priorities, complete visibility into them , failed jobs etc.\n\n\n
• #16: We used DJob until a few weeks before launch\n\nDelayedJob is great and super easy to work with,\nin the end we wanted more granular control over our background tasks since a lot of stuff happens there\n\n_different queues, priorities, complete visibility into them , failed jobs etc.\n\n\n
• #17: Being able to see into the queues individually (and individually kill them) saved our asses more than once.\n\nESPECIALLY EXAMINING failed queue in detail, \nwhich jobs are failing, \nhow frequently, \nare there stuck workers\n\n\nRuns on Redis, created by @defunkt, has a great community and a host of plugins \n
• #18: When thinking about how to optimize Branch before launch, we mostly concentrated on how to optimize user response time. \n\nSo we ended up making a bunch of actions asynchronous-\n\nwe even have quite a few background jobs which QUEUE background jobs.\n\nIf they’re processed in the background, who cares if they’re fast?\n\n\n
• #19: When thinking about how to optimize Branch before launch, we mostly concentrated on how to optimize user response time. \n\nSo we ended up making a bunch of actions asynchronous-\n\nwe even have quite a few background jobs which QUEUE background jobs.\n\nIf they’re processed in the background, who cares if they’re fast?\n\n\n
• #20: A big lesson for us is that you can’t think like that- especially when so much of your app is async\n\nespecially at peak activity times like when we would send batches of invites\n\nHOW\n\n\n\n
• #21: There are a bunch of great blog posts and gems to do this\n\nbut be careful!\n\n\n
• #22: They are not fire and forget\n\nWhen you have a lot of workers doing silly things in your DB, the result can be catastrophic\n
• #23: They are not fire and forget\n\nWhen you have a lot of workers doing silly things in your DB, the result can be catastrophic\n
• #24: They are not fire and forget\n\nWhen you have a lot of workers doing silly things in your DB, the result can be catastrophic\n
• #25: Rails is great- gives us alot of convenience, makes it easy to develop quickly\n\nactive record abstracts your database for you, but \n\nthat’s a double edged sword\n
• #26: I can run this migration, even on production data, and it won’t be THAT painful.\nI even think I’m being efficient with find_each\n\nIt should be. Running a migration like this on prod can brick for ??? how long\n
• #27: there’s probably a SQL ninja here who could accomplish this in 1 call,\n\nbut we can all agree this is much better than the first\n\nThe ORM makes it easy to ignore sql completely, but that will bite you. FOR INSTANCE\n
• #28: At first: this sounds super convenient\n\n
• #29: At first: this sounds super convenient\n\n
• #30: Especially with big objects! \n\ngod forbid you do any type of validations on those serialized hashes, since they dont support indices\n\nWe were storing twitter token objects, \nUpdating/ changing the tables, or moving the data became EXTREMELY expensive. \n\n\nyou have to de-serialize in memory! \n
• #31: \nIf your object is so important, maybe it needs its own table?\n\nIf you don’t have to store it, dont!\n\nHstore fsho!\n
• #32: \nIf your object is so important, maybe it needs its own table?\n\nIf you don’t have to store it, dont!\n\nHstore fsho!\n
• #33: \nIf your object is so important, maybe it needs its own table?\n\nIf you don’t have to store it, dont!\n\nHstore fsho!\n
• #34: Hstore is the way postgres stores hashes\n\nit’s not supported in rails natively yet, but there are gems that will allow you to use\n\nmuch faster!\n\nthose are just two examples of ways to shoot yourself in the foot with rails\n\nstay alert!\n
• #35: \n
• #36: \n
• #37: \n
• #38: \n
• #39: \n