Download to read offline
More Related Content
Similar to CREAM - That Conference Austin - January 2024.pptx
![[Hanoi-August 13] Tech Talk on Caching Solutions](https://cdn.slidesharecdn.com/ss_thumbnails/nitecocachingsolution-130826204343-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
![[B5]memcached scalability-bag lru-deview-100](https://cdn.slidesharecdn.com/ss_thumbnails/b5memcached-scalability-baglru-deview-100-120919013502-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
More from Matthew Groves
![[DSC DACH 25] Ali Mokh - Symbolic Reasoning in telecom oriented LLMs.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/dscdach25alimokh-symbolicreasoningintelecomorientedllms-251027100200-99093bc8-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC DACH 25] Yudan Lin_The Human - AI Paradox and Effective Fix.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/dscdach2025yudanlinthehuman-aiparadoxandeffectivefix-251024080438-f3a7a9e1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC DACH 25] Brinnae Bent - Hacking the Blackbox.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/brinnaebenthackingtheblackbox1-251024080947-002ebd1f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC DACH 25] From Data to Decisions: Building Responsible AI for the Future ...](https://cdn.slidesharecdn.com/ss_thumbnails/dscdach2025shpresentation-251024080322-abe65846-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC DACH 25] Jacqueline Berger - Beyond the Hype: The Real Story of Building...](https://cdn.slidesharecdn.com/ss_thumbnails/dsc-dach-2025jacquelineberger-251024080522-7625155a-thumbnail.jpg?width=640&height=640&fit=bounds)
CREAM - That Conference Austin - January 2024.pptx
- 3. C.R.E.A.M. get thememory (dollar dollar bill ya'll) January 2024 Matthew D. Groves | DevRel Engineer aka “Passionate Bravo” Cache Rules Everything Around Me
- 4. 4 Latency Numbers EveryProgrammer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
- 5. 5 The Need forSpeed https://www.slideshare.net/devonauerswald/walmart-pagespeedslide
- 6. 6 Cache Rules EverythingAround Me Dollar dollar bill ya'll Android apps iPhone apps CPU cache CDNs Web browser DNS
- 7. 01/ 02/ 03/ 04/ 05/ 06/ What is caching? UseCases Reading from Cache Writing into Cache Cache Invalidation Gotchas Agenda
- 8. 8 ho am I? •Matthew D. Groves • Microsoft MVP, author at Pluralsight, Manning, Apress • DevRel Engineer at Couchbase • https://twitter.com/mgroves • https://www.linkedin.com/in/mgroves/ • C# Advent: https://csadvent.christmas/
- 9. 9 1 hat isCaching?
- 10. 10 Caching Collection of duplicateddata to serve future requests faster. • First formally described by Maurice Wilkes in 1965 • "Slave Memories and Dynamic Storage Allocation" https://bit.ly/WilkesPaper What is Caching? Key Data 1 Proteck Ya Neck 2 C.R.E.A.M. 3 Brooklyn Zoo 4 Criminology … … Key Data 2 C.R.E.A.M. 4 Criminology Records Cache
- 11. 11 Powerpoint as aCache What is Caching? Key Value What year was Wu-Tang Clan Formed? 1992
- 12. 12 2 Caching UseCases
- 13. 13 Caching Use Cases Whatis Caching used for?
- 14. 14 Caching Use Case1: Faster Database Access • Databases can rely heavily on disk • Reading the same data over again could mean accessing the disk over again • Store frequently accessed data in a cache, to reduce pressure on database/disk
- 15. 15 Caching Use Case1: Faster Database Access DATABASE APP LAN CACHE CACHE CACHE
- 16. 16 Caching Use Case2: Faster Web Browsing • Any given URL could result in hundreds of files that are loaded over HTTP • A single JS file may be loaded by every page on a site. • Cache static assets (JS, images, sound, icons, etc) on a user's laptop to avoid loading them over the network every time
- 17. 17 Caching Use Case2: Faster Web Browsing CACHE
- 18. 18 Caching Use Case3: Efficient API Use • API usage may involve an internet call (with latency) • API calls may cost per use, or may be limited • Storing results in cache may help reduce latency AND reduce costs
- 19. 19 Caching Use Case3: Efficient API Use API APP LAN/WAN CACHE CACHE CACHE
- 20. 20 Caching Use Case4: Microservice Availability • Retrieving data from other processes • Latency is secondary concern
- 21. 21 Caching Use Case4: Microservice Availability CACHE
- 22. 22 Cache Rules EverythingAround Me Dollar dollar bill ya'll Disconnected Availability Slow Repetition Constrained Costs
- 23. 23 3 Reading FromCache
- 24. 24 Using a Cache101 Your Code Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo Cache Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo D Criminology E Triumph F Uzi G Method Man H Wu-Tang Clan Ain't . . . . . . . . . Database 1. Lookup "A" 2. Lookup "B" 3. Lookup "C" HIT! HIT! MISS! 4. Lookup "C" again? WALK OF SHAME
- 25. 25 4 Writing IntoCache
- 26. 26 What about writes? •Data is duplicated, stored in cache and record • The system is responsible for keeping the cache updated • Commons methods: • write-through • write-around • write-back Stuff needs to get in the cache somehow
- 27. 27 write-through 1. System getsa new/updated piece of data 2. Write to cache and record "at the same time" a) Write to record b) If that didn't succeed, the write failed. c) Write to cache d) If that didn't succeed, the write failed. Rollback (a). e) If the system reaches this point, the write succeeded! Why or why not use this?
- 28. 28 write-around 1. System getsa new/updated piece of data 2. Write the data to the record ONLY 3. Invalidate the record in the cache Why or why not use this?
- 29. 29 write-back 1. System getsa new/updated piece of data 2. Write the data to the cache. 3. Update the record asynchronously (don't wait on it to finish) 4. If the cache write succeeded, then consider the write succeeded. Why or why not use this?
- 30. 30 Cache Writing Methods MethodWrite performance Write integrity Best for…? write-through ➖ ➕ High % reads write-around ➖ ➕ High % read later write-back ➕ ❔ Mixed reads / writes
- 31. 31 Couchbase: A write-backimplementation A database with a built-in managed cache https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html Other Couchbase Server nodes
- 32. 32 Couchbase and Writingto the Cache await collection.InsertAsync("A", new { title = "Protect Ya Neck" }); await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options => { options.Durability(DurabilityLevel.None); // OR: options.Durability(DurabilityLevel.Majority); // OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive); // OR: options.Durability(DurabilityLevel.PersistToMajority); });
- 33. 33 Durability (with 3replicas) None Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ⚠ ⚠ Node 4 ⚠ ⚠
- 34. 34 Durability (with 3replicas) Majority Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
- 35. 35 Durability (with 3replicas) MajorityAndPersistToActive Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
- 36. 36 Durability (with 3replicas) PersistToMajority Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ✔ Node 4 ✔ ✔
- 37. 37 Latency Numbers EveryProgrammer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
- 38.
- 39. 39 Powerpoint as aCache Key Value What year was Wu-Tang Clan Formed? 1992 What was Wu-Tang's first album called? Enter the Wu-Tang What year was Maurice Wilkes born? 1913 What year did Maurice Wilkes die? 2010 What is the most recent Wu- Tang studio album? Once Upon a Time in Shaolin
- 40. 40 Cache Invalidation isHard There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton How to decide what to invalidate (what data to "evict"): cache policy https://www.nndb.com/people/400/000031307/
- 41. 41 Cache Policy 1:LRU (Least Recently Used) Key How new? A 0 B 1 3 C 2 D 4 Cache max size: 3 Order of reads: A, B, C, B, D X ---------------------------------------------------------------------------------
- 42. 42 Cache Policy 2:NRU (Not Recently Used) Lower scores evicted (at random) Key Referenced? Modified? Score A 0 0 0 B 0 0 0 C 0 1 1 D 1 0 2 E 1 1 3
- 43. 43 Cache Policy 3:None Don't evict anything
- 44. 44 More Cache Policies •RR – random replacement • FIFO and LIFO – treat a cache like a queue • MRU – MOST recently used • LFU – least frequently used • Belady's algorithm – not possible to implement • Machine learning • https://en.wikipedia.org/wiki/Cache_replacement_policies
- 45.
- 46. 46 Gotcha 1: CacheSize • Problem: too small or too big cache • Symptom: High turnover and churn (many evictions) or long lifetimes (few evictions) • Solutions: • Monitoring • Size the cache • "Value" vs "Full" eviction
- 47. 47 Gotcha 2: "Close"Caching • Problem: running the cache on the same machine as the application • Symptom: redundancy, hot spots, inconsistency, availability • Solutions: • Use a distributed cache • Running on its own machine(s) / cluster • Couchbase, of course!
- 48. 48 Gotcha 3: BadEviction Policy • Problem: using the wrong eviction policy • Symptom: lots of overhead, cache checking • Solutions: • Don't Write an Eviction Policy (or better yet don't write a Cache) • Benchmarking: "hit ratio" and "latency" • "Hit ratio" – how often a hit vs miss (e.g. "45% hit ratio") • "Latency" – how long it takes from request to response (e.g. 10μs latency)
- 49.
- 50. 50 References • https://www.geeksforgeeks.org/not-recently-used-nru-page-replacement-algorithm/ • https://shahriar.svbtle.com/Understanding-writethrough-writearound-and-writeback- caching-with-python •https://en.wikipedia.org/wiki/Maurice_Wilkes • https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf • https://stackoverflow.com/a/4087315/40015 • https://www.techtarget.com/searchstorage/definition/cache • https://en.wikipedia.org/wiki/Cache_replacement_policies • https://docs.couchbase.com/dotnet-sdk/current/howtos/kv-operations.html#durability • https://docs.couchbase.com/dotnet-sdk/current/concept-docs/durability-replication-failure- considerations.html#durable-writes