LMCache Lab

𝗞𝗦𝗲𝗿𝘃𝗲 𝘃0.15: 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀-𝗡𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗧𝗵𝗮𝘁 𝗔𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗦𝗰𝗮𝗹𝗲𝘀 The gap between getting an LLM working on your laptop and running it reliably in production is still enormous. KServe is the first serving platform I've seen that actually addresses these problems at an architectural level. Multi-node inference lets you run models like Llama 405B by distributing them across GPU clusters with simple YAML configuration. No custom orchestration code, no manual sharding logic. The KEDA integration brings autoscaling that understands LLM workloads. Instead of scaling on CPU percentage or request count, you scale on running inference requests. This matters when one request might generate 10 tokens and another generates 500. Distributed KV cache through LMCache Lab means your inference pods can share cached computations. When a user asks a follow-up question, any pod in your cluster can reuse the context from previous turns, even if a different instance handled the original request. The Envoy AI Gateway adds token-based rate limiting, intelligent routing, and multi-tenant isolation—features that generic API gateways don't provide because they weren't designed for generative AI traffic patterns. The platform is open source, CNCF-backed, and used by Bloomberg, IBM, and NVIDIA. I wrote a detailed technical breakdown covering the architecture, implementation examples, and YAML configurations: https://lnkd.in/eaY9Dtkx

We’re thrilled to see LMCache featured alongside other leading open-source projects at IBM TechXchange! 🎉

Check out our new blog and colab with Google GKE about LMCache on Google Kubernetes Engine: Boosting LLM Inference Performance with KV Cache on Tiered Storage https://lnkd.in/g4PZXB7s

KV Caching is an optimization to accelerate inference workloads' time to first token and total tokens per second throughput by preventing LLM models from recalculating these tokens for the entire text sequence. Our team has been working with LMCache Lab to develop KV Cache offloading with multi-tier storage on #GCP to increase the tokens / second throughput of vLLM by 264% and decrease the E2E latency by 73%. Read more in: https://lnkd.in/gQv9rcZH Nathan Beach Akshay Ram Nishtha Jain Connor McCoy Clayton Coleman Danna Wang Brian Kaufman

Fantastic to see real-world stories like this! Thanks, Manoranjan, for spotlighting how LMCache helps teams unlock longer context windows and maximize their GPU investment.

We’re excited to see Redis highlighting the power of LMCache for faster, more efficient LLM inference!

Why is my LLM still slow — even with a powerful GPU? In this tutorial, I walk through how LMCache Lab + NVIDIA Dynamo boosts vLLM by 13.1×, and how you can replicate the setup on your own. YouTube: https://lnkd.in/eBuPhu2k

In large-scale language model inference scenarios, efficient memory management and KV cache optimization are crucial. LMCache, as a KV cache management system specifically designed for vLLM, requires more flexible extension mechanisms to meet the needs of monitoring, troubleshooting, and state insight when facing complex production environments. However, instead of directly customizing the lmcache core, we introduced the LMCache Plugin Framework - a lightweight yet powerful plugin system that allows developers to run custom scripts within LMCache processes. Based on this plugin framework, we implemented lmcache_frontend(https://lnkd.in/gkBe8Hmi), a monitoring and proxy service that runs as a subprocess only on scheduler nodes. It provides a Web interface for cluster status visualization and implements request forwarding functionality through HTTP proxy services. This design not only facilitates deployment and management but also provides developers with an excellent plugin implementation example, demonstrating how to use the Plugin Framework to enhance system observability and control capabilities. Read more: https://lnkd.in/gNSGmgmS Doc: https://lnkd.in/gdh5cd_B

A deep dive on LMCache x NVIDIA Dynamo: https://lnkd.in/gaYZzPjR to learn how LMCache integrates with NVIDIA Dynamo to slash KV-cache bottlenecks and push LLM inference efficiency to the next level. We’re also honored to be featured in NVIDIA’s official blog on Dynamo: https://lnkd.in/gaF35XaT #LMCache #NVIDIA #Dynamo #LLM #Inference #KVcache

Missed our tutorial at SIGCOMM 2025? The full recording is now available! Watch the session here: https://lnkd.in/e6EbM48Y #SIGCOMM #SIGCOMM2025 #LMCache