vLLM

We are excited to announce KServe 0.15 release, marking a significant leap forward in serving both predictive and generative AI models. Key GenAI features include: Envoy AI Gateway integration, multi-node inference via vLLM, LLM autoscaler with KEDA, and distributed KV cache with LMCache Lab.

Thank you RunLLM! 13k per month questions answered and growing!

the PyTorch Foundation is becoming a hub for open-source AI excellence. vLLM and Deepspeed are joining PyTorch as the first two projects after this change! Super excited to be able to bring together multiple like-minded, high-quality projects into the foundation. Props to Joseph Spisak, Matt W., Luca Antiga and others for driving this change! Full announcement here: https://lnkd.in/eQ-htiaN

vLLM is one of the fastest growing open-source communities, and they've had a meteoric rise that mirrors the growing popularity of open-source models. Given how widely the project is used, the team gets hundreds of questions a day — all of which are extremely technical. Answering each question requires an understanding of cloud infrastructure, model architectures, and deployment options. Most generic RAG applications simply can't keep up with this level of complexity, which is why the vLLM team turned to RunLLM. Since they've deployed RunLLM, we've had a first hand seat to see how technical their user base and how much they benefit from a high quality AI Support Engineer. RunLLM's now answering 13k questions a month (🤯) for vLLM and enabling the team to scale without needing dedicated support. More in the case study below 👇

The Open Language of AI.. My journey in AI started over 10 years ago when I met Soumith, Yangqing, Dmytro, Andrew and Pieter - folks that I am proud to this day to call friends and colleagues. It’s hard to believe it’s been that long and honestly hard to believe I’ve spent 7 of those years helping to build PyTorch what it is today - an amazing and inspirational project that is foundational to ChatGPT, Llama, DeepSeek, Stable Diffusion and so many other breakthroughs. In this next phase we have a much bigger and broader vision and as part of that we are bringing in vLLM and DeepSpeed under our umbrella foundation laying the groundwork for PyTorch to continue its growth trajectory with additional high caliber projects on the way. For more on how we see the future check out the vision blog that Luca and coauthored.  https://lnkd.in/gZRmv2e9 I’m excited to work with this community to realize our expanded vision of being the open language of AI. Thank you to everyone who made this next step possible including a few key folks - Ion Stoica, Aparna Ramani, Soumith Chintala, Matt W., Simon Mo and Woosuk Kwon and many others.. And thank you to all of the members: AMD, Arm, Amazon Web Services (AWS), Google, Huawei, Hugging Face, IBM, Intel Corporation, Lightning AI, NVIDIA, and Snowflake. Cheers to what's next! Details: Vision post: https://lnkd.in/gZRmv2e9 Press Release - https://lnkd.in/gBzYkyaJ vLLM Announcement - https://lnkd.in/gCEwbq3p DeepSpeed Announcement - https://lnkd.in/gXAWTNnt

𝗖𝗮𝗻 𝗔𝗜 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗯𝗲 𝗲𝘃𝗲𝗻 𝗳𝗮𝘀𝘁𝗲𝗿? At Snowflake AI Research, we’ve been working to push the limits of real world speculative decoding —and we’re excited to share a major step forward. With our latest improvements, we’re seeing some of the 𝗳𝗮𝘀𝘁𝗲𝘀𝘁 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝘀𝗽𝗲𝗲𝗱𝘀 𝘆𝗲𝘁 across a range of real-world workloads. 🚀  • 𝟰𝘅 𝗳𝗮𝘀𝘁𝗲𝗿 LLM inference for coding agents like OpenHands All Hands AI  💬  • 𝟮.𝟰𝘅 𝗳𝗮𝘀𝘁𝗲𝗿 LLM inference for conservational/interactive use cases These improvements result from multiple technologies all working together:  • 𝗦𝘂𝗳𝗳𝗶𝘅 𝗗𝗲𝗰𝗼𝗱𝗶𝗻𝗴 for speculating long repetitive patterns.  • 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴-𝘁𝗼-𝗱𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 framework for creating powerful but lightweight custom draft models for non-repetitive generation.  • 𝗦𝘆𝘀𝘁𝗲𝗺-𝗹𝗲𝘃𝗲𝗹 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀 to streamline the entire speculation code path. We are open-sourcing all of our improvements as part of Arctic Inference, an easy-to-install plugin for vLLM Check out our blog post to learn more: https://lnkd.in/gPAERCcN

This is a MUCH BIGGER deal than one could think at first sight: ➡️Transformers as consistent architecture definition, made seamlessly available in vLLM’s super efficient runtime I’m looking forward to even more collaboration in the coming months 🔥🔥🔥

We are excited to announce Day 0 Support for Meta's Llama 4 Scout and Maverick. You can find useful serving commands in this blog post https://lnkd.in/gm_JUCqp

Announcing native LLM APIs in Ray Data and Ray Serve Libraries. These are experimental APIs we are announcing today that abstract two things: 1. Serve LLM: simplifies the deployment of LLM engines (e.g. vLLM) through ray serve APIs. Enables things like auto-scaling, monitoring, LoRA management, resource allocation etc. 2. Data LLM: Helps you scale up offline inference horizontally for throughput sensitive applications (e.g. data curation, evaluation, etc). Ray data's lazy execution engine helps you pipeline complex heterogenous stages that involve LLMs. Say you want to create a pipeline that reformats input with Llama-8B and then queries Llama-70B in another stage. How do you maximize throughput for this pipeline? Or a vision language model that needs to pull images from s3 (a network bounded operation), tokenization (a cpu bounded op) and then inference with Pixtral (a gpu bounded op). This is the type of problem that Data LLM API will simplify. https://lnkd.in/gSnAs4kf