Download as PDF, PPTX
![re.compile(‘f[o0][1l][o0]w’)](https://image.slidesharecdn.com/qconsf2013-mikekrieger-howasmallteamscalesinstagram-160428182758/75/Mike-Krieger-A-Brief-Rapid-History-of-Scaling-Instagram-with-a-tiny-team-217-2048.jpg)
Uploaded byJean-Luc David
Mike Krieger - A Brief, Rapid History of Scaling Instagram (with a tiny team)
The document outlines a history of Instagram's scaling process from its inception, detailing the challenges and strategies employed by a small team of engineers. Key principles discussed include simplifying solutions, effective infrastructure management, and careful monitoring, along with the importance of recruiting and adapting as the company grew. It emphasizes the need for balancing technological trade-offs and making informed decisions in the face of rapid growth.
More Related Content
![[A31]AWS上でOracleを利用するためのはじめの一歩！by Masatoshi Yoshida](https://cdn.slidesharecdn.com/ss_thumbnails/a31oracleyoshida-131122001746-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![[AWSマイスターシリーズ] Instance Store & Elastic Block Store](https://cdn.slidesharecdn.com/ss_thumbnails/20140129aws-meister-reloaded-ephemeral-ebs-public-140130055222-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Mike Krieger - A Brief, Rapid History of Scaling Instagram (with a tiny team)
Mike Krieger - A Brief, Rapid History of Scaling Instagram (with a tiny team)
- 1. A Brief, RapidHistory of Scaling Instagram (with a tiny team) Mike Krieger QConSF 2013 !
- 2.
- 3.
- 4. 30 million with2 eng (2010-end 2012)
- 5. 150 million with6 eng (2012-now)
- 6.
- 7. What I wouldhave done differently
- 8. What tradeoffs youmake when scaling with that size team
- 9. (if you canhelp it, have a bigger team)
- 10.
- 11.
- 12.
- 13.
- 14. Do the simplestthing first
- 15. Every infra movingpart is another “thread” your team has to manage
- 16.
- 17. This talk Early days Year1: Scaling Up Year 2: Scaling Out Year 3-present: Stability, Video, FB
- 18.
- 19. 2010 2 guys ona pier
- 23.
- 24.
- 26. Mike iOS, KevinServer
- 27. Early Stack Django +Apache mod_wsgi Postgres Redis Gearman Memcached Nginx
- 28. If today Django +uWSGI Postgres Redis Celery Memcached HAproxy
- 29.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37. “What’s a loadaverage?”
- 38. “Can we getanother server?”
- 39. Doritos & Red Bull& Animal Crackers & Amazon EC2
- 40.
- 41.
- 42. Scale "just enough"to get back to working on app
- 43. Every weekend wasan accomplishment
- 44. “Infra is whathappens when you’re busy making other plans” —Ops Lennon
- 45.
- 46. First bottleneck: diskIO on old Amazon EBS
- 47. At the time:~400 IOPS max
- 48.
- 49.
- 50.
- 51.
- 52.
- 53. Bought us sometime
- 54. Almost no applicationlogic changes (other than some primary keys)
- 55. Today: SSD andprovisioned IOPS get you way further
- 56.
- 57.
- 58.
- 59.
- 60. No easy migrationstory; mostly double-writing
- 61. Replicating + deleting oftenleaves fragmentation
- 62.
- 63.
- 64.
- 65. Mind that hashfunction
- 66. Why not Redisfor kv caching?
- 67.
- 68.
- 69. fabric + parallelgit pull (sorry GitHub)
- 70. All AMI basedsnapshots for new instances
- 71.
- 72.
- 73. Should have doneChef earlier
- 74.
- 75.
- 76.
- 77. Infra going from10% time to 70%
- 78.
- 79. Testing & monitoringkept concurrent fires to a minimum
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87. Hard to trackdown kernel panics
- 88. Skip rabbit hole;use instance- status to detect and restart
- 89.
- 90. Out of IOagain (Pre SSDs)
- 91.
- 92.
- 93.
- 94.
- 95. Theory: partitioning andrebalancing are hard to get right, let DB take care of it
- 96. MongoDB (1.2 atthe time)
- 97.
- 98.
- 99.
- 100.
- 101. Could have madeit work…
- 102. …but it wouldhave been someone’s full time job
- 103. (and we stillonly had 3 people)
- 104.
- 105.
- 106. QCon to therescue
- 107. Similar approach toFB (infra foreshadowing?)
- 108. Logical partitioning, done atapplication level
- 109. Simplest thing; skipped abstractions& proxies
- 110.
- 111.
- 112. note to self:pick a power of 2 next time
- 113.
- 114.
- 115.
- 116.
- 117.
- 118. Still how wescale PG today
- 119. 9.2 upgrade: bucardoto move schema by schema
- 120.
- 121. Requirements No extra movingparts 64 bits max Time ordered Containing partition key
- 122. 41 bits: timein millis (41 years of IDs) 13 bits: logical shard ID 10 bits: auto-incrementing sequence, modulo 1024.
- 123. This means wecan generate 1024 IDs, per shard, per table, per millisecond
- 124.
- 125. A new dbis a full time commitment
- 126. Be thrifty withyour existing tech
- 127. = minimize movingparts
- 128.
- 129.
- 130. No team tomaintain
- 131.
- 132.
- 133.
- 134.
- 135. But zero additionalinfra, got the job done, easy to debug
- 136.
- 137. Munin: too coarse,too hard to add new stats
- 138.
- 139.
- 140.
- 141. Step change indeveloper attitude towards stats
- 142. <5 min fromwanting to measure, to having a graph
- 143. 580 statsd counters 164statsd timers
- 144.
- 145.
- 146. (doubling all ofour infra, most of which was now horizontally scalable)
- 147. Doubled active usersin < 6 months
- 148. Finally, slowly, buildingup team
- 149.
- 150. Scale tools tomatch team
- 151.
- 152. Finally 100% onChef
- 153. Simple thing first:knife and chef-solo
- 154. Every new hirelearns Chef
- 155.
- 156.
- 157.
- 158. But push stillneeds a driver
- 159.
- 160. Humans are terrible distributedlocking systems
- 161.
- 162.
- 163. Rollout / majorconfig changes / live deployment tracking
- 164. Extracting approach Hit issue Developmanual approach Build tools to improve manual / hands on approach Replace manual with automated system
- 165.
- 166.
- 167.
- 168.
- 169.
- 170.
- 171. s = time.time() #do work statsd.incr("waittime.VIEWNAME.C OMPONENT", time.time() - s)
- 172.
- 174.
- 175.
- 176.
- 177. Every churned orinactive user
- 179.
- 180.
- 181.
- 182. C* cluster is20% of the size of Redis one
- 183. Main feed (timeline)still in Redis
- 184.
- 185.
- 186.
- 187.
- 188.
- 189.
- 190.
- 191.
- 192. Uses Incremental feature rollouts Dynamicpage sizing (shedding load) Feature killswitches
- 194. As more teamsaround FB contribute
- 195. Decouple deploy from featurerollout
- 196.
- 197. Launch a top10 video site on day 1 with team of 6 engineers, in less than 2 months
- 198. Reuse what weknow
- 199.
- 200.
- 201. Separate from mainApp servers
- 202.
- 203.
- 204. ZooKeeper ephemeral nodes fordetection
- 205. (finally worth it/ doable to deploy ZK)
- 206.
- 207. Priority list forclients
- 208. Transcoding tier is completelystateless
- 209.
- 210. holding area for debuggingbad videos
- 211. 5 million videosin first day 40h of video / hour
- 212. (other than perfimprovements we’ve basically not touched it since launch)
- 213.
- 214. Where can weskip a few years?
- 215. (at our ownpace)
- 216.
- 217.
- 218. Simplest thing didnot last
- 219. Generic features + machinelearning
- 220. Hadoop + Hive+ Presto
- 221. "I wonder howthey..."
- 222.
- 223.
- 224.
- 225.
- 226.
- 227.
- 228. Do the simplestthing first
- 229. Every infra movingpart is another “thread” your team has to manage
- 230.
- 231.
- 232. Recruit way earlierthan you'd think
- 233.
- 234. Rocketship scaling hasbeen (somewhat) democratized
- 235. Huge thanks toIG Eng Team
- 236.