Piyush A.

API Gateway ≠ AI Gateway Both sit in front of services, but they solve different classes of problems. API Gateway -Protocol-level: HTTP, REST, GraphQL, gRPC -Stateless request/response routing -Focus on auth, throttling, caching, logging -Examples: Kong, NGINX, AWS API Gateway AI Gateway -Model-level: LLMs, embeddings, multi-model orchestration -Stateful: tracks context, conversation, memory -Handles token accounting, latency metrics, model routing -AI-specific observability (usage, cost per token, provider SLAs) -Examples: OpenRouter, LangChain Gateway, Hugging Face LLM Router Think of it this way: API Gateway optimizes API traffic AI Gateway optimizes model traffic As AI stacks evolve, I expect AI Gateways to become as foundational as API Gateways — especially in enterprises juggling multiple LLMs across vendors.

777

58 Comments

If you’re working with Kubernetes, here are 6 scaling strategies you should know — and when to use each one. Before we start — why should you care about scaling strategies? Because when Kubernetes apps face unpredictable demand, you need scaling mechanisms in place to keep them running smoothly and cost-effectively. Here are 6 strategies worth knowing: 1. Human Scaling ↳ Manually adjust pod counts using kubectl scale. ↳ Direct but not automated. When to use ~ For debugging, testing, or small workloads where automation isn’t worth it. 2. Horizontal Pod Autoscaling (HPA) ↳ Changes pod count based on CPU/memory usage. ↳ Adds/removes pods as workload fluctuates. When to use ~ For stateless apps with variable load (e.g., web apps, APIs). 3. Vertical Pod Autoscaling (VPA) ↳ Adjusts CPU/memory requests for existing pods. ↳ Ensures each pod gets the right resources. When to use ~ For steady workloads where pod count is fixed, but resource needs vary. 4. Cluster Autoscaling ↳ Adds/removes nodes based on pending pods. ↳ Ensures pods always have capacity to run. When to use ~ For dynamic environments where pod scheduling fails due to lack of nodes. 5. Custom Metrics Based Scaling ↳ Scale pods using application-specific metrics (e.g., queue length, request latency). ↳ Goes beyond CPU/memory. When to use ~ For workloads with unique performance signals not tied to infrastructure metrics. 6. Predictive Scaling ↳ Uses ML/forecasting to scale in advance of demand. ↳ Tries to prevent traffic spikes before they happen. When to use ~ For workloads with predictable traffic patterns (e.g., sales events, daily peaks). Now know this — scaling isn’t one-size-fits-all. The best teams often combine multiple strategies (for example, HPA + Cluster Autoscaling) for resilience and cost efficiency. What did I miss? • • • If you found this useful.. 🔔 Follow me (Vishakha) for more Cloud & DevOps insights ♻️ Share so others can learn as well

674

79 Comments

In Kubernetes, HPA uses a fixed 10% tolerance for both scale-up and scale-down. Meaning, HPA waits for at least 10% change in CPU/memory usage before adding or removing pods For example, If you set 70% as the usage target, Kubernetes would only scale up if it went above 77%, and scale down if it went below 63%. Here is the problem with with fixed 10%? Sometimes you want the system to react quickly when traffic spikes. Other times, you want it to be more cautious to avoid constant scaling up and down. Now in Kubernetes v1.33 (alpha feature - HPAConfigurableTolerance), You can set separate tolerance levels for scaling up and scaling down. I have published a short guide with hands on example in my newsletter. 𝗥𝗲𝗮𝗱 𝗶𝘁 𝗵𝗲𝗿𝗲: https://lnkd.in/gyAJTMXS 10,000+ professionals read my DevOps newsletter I share deep dives and insights on Kubernetes, DevOps, Cloud, MLOps and GitOps ♻️ If you find this useful, Repost with your network! #kubernetes

289

7 Comments

#Kubernetes Journey – Day 8: Volumes & Persistent Volumes In Kubernetes, containers are ephemeral – once a Pod is deleted or restarted, its data is lost. That’s where Volumes & Persistent Volumes (PV) come into play. 🔹 Volume – Provides storage to a Pod, ensuring data is available during container restarts inside the same Pod. 🔹 Persistent Volume (PV) – A cluster-wide storage resource that exists independently of Pods. 🔹 Persistent Volume Claim (PVC) – A request made by a Pod to use storage from a PV. ✅ Key Takeaways: Volumes solve the ephemeral nature of containers. PVs abstract the underlying storage (NFS, AWS EBS, GCP PD, Azure Disk, etc.). PVCs let developers request storage without worrying about backend implementation. StorageClasses automate dynamic provisioning of volumes. 💡 Real-World Example: Imagine running a database Pod (MySQL/Postgres) – you cannot afford to lose data every time the Pod restarts. By using PVCs with PVs, you ensure your data persists reliably. #Interview Question for You: Q.What is the difference between Volume, PV, and PVC in Kubernetes? #Kubernetes #DevOps #CloudComputing #Containers #LearningJourney Pod → PVC → PV → Storage

60

6 Comments

You can code and run your MCP server remotely on Kubernetes. Here's how: 1. Write code which conforms to the MCP standards. What does this mean? it means, it should work with JSON RPC 2.0 and can process standard MCP methods such as tools/list should send the name and description of the tools, tools/call should be able to invoke the tools etc. These standards are defined by Anthropic 2. You can either code the methods for MCP yourself, which can be tedious. Or use MCP implementation with library like FastMCP 3. Run Dockerfile to create a container and save it in ECR 4. Deploy to the Kubernetes cluster such as in Amazon EKS, and expose it using a ALB via service or ingress. ALB and container support streamable HTTP out of the box, and works nicely 5. Invoke your MCP Server using the ALB Url. For Streamable HTTP, you need to initialize a session, and then use the sessionID for subsequent calls to do tool discovery, and tool calls Check out detailed step by step demo, along with related codes, and pros-cons of local vs remote MCP server: https://lnkd.in/eXKENy6R --- Get byte sized system design, behavioral, Gen AI, and other interview and career switch tips in weekly newsletter : https://lnkd.in/eG7XdHmN #genai #ai #systemdesign

293

31 Comments

Ingress vs Service Mesh: The battle of Kubernetes traffic control. One rules the edge. The other rules everything inside. In Kubernetes, Ingress and Service Mesh both are used for managing traffic, but in fundamentally different scopes. With the help of a simple illustration Let’s understand how they are different. 𝗜𝗻𝗴𝗿𝗲𝘀𝘀 handles North-South traffic, which means it manages external client requests entering the cluster. It offers features like host/path-based routing, TLS termination, and basic auth. You can think of it as the gatekeeper of your Kubernetes environment. 𝗦𝗲𝗿𝘃𝗶𝗰𝗲 𝗠𝗲𝘀𝗵, on the other hand, is used to manage East-West traffic, which means it controls the internal communication between services. It introduces sidecar proxies (e.g., Envoy) to handle secure (mTLS), observable, and reliable communication.

96

32 Comments

Normal Pod vs Secured Pod – Why Security Context Matters in Kubernetes In today’s cloud-native world, containers and Kubernetes have become the backbone of modern applications. But with great agility comes an equally important responsibility: security. As solution architects, we must not only design for scalability and performance but also embed security as a first-class citizen in our architecture. One of the most overlooked areas in Kubernetes is the difference between a Normal Pod and a Secured Pod. At first glance, both seem identical—they run workloads, expose services, and scale as needed. But under the hood, the security posture can make all the difference between a resilient system and an open door for attackers. 🔹 Normal Pod A normal pod often runs with default settings. This usually means the container may run as root, the filesystem is writable, Linux capabilities are not restricted, and host-level access is possible. While this works for development speed, it leaves the environment vulnerable. If compromised, such a pod could allow privilege escalation, lateral movement across the cluster, or even host takeovers. 🔹 Secured Pod A secured pod, on the other hand, is designed with defense-in-depth principles. It enforces a SecurityContext and Pod Security Standards such as: Running as a non-root user Using a read-only filesystem Dropping all unnecessary Linux capabilities Applying seccomp/AppArmor profiles to restrict system calls Enforcing resource limits to prevent denial-of-service attacks By shifting left and securing pods at the design stage, we significantly reduce the attack surface. This also aligns with compliance frameworks like CIS Benchmarks, PCI DSS, and zero-trust models. 🔑 Why does this matter at an architectural level? Because Kubernetes security is not just a DevOps problem—it’s an enterprise resilience challenge. When you standardize secured pods, you embed trust into your CI/CD pipelines, accelerate audits, and prevent costly breaches. More importantly, you enable teams to innovate faster without the constant fear of vulnerabilities. ✅ Takeaway: A normal pod may be enough to run an app, but a secured pod ensures that the app runs safely, responsibly, and compliantly. As architects and technology leaders, we must advocate for secure defaults and foster a culture where security is seen not as a blocker, but as an enabler of sustainable innovation.

28

1 Comment

If you're preparing for a cloud engineer role (in 1 month or less) - you must read these case studies 1. LinkedIn's Infrastructure as Code at Scale → Standardize multi-cloud infrastructure with Terraform → Manage thousands of resources via IaC pipelines → Enable self-service model for global engineering teams Case study link: https://lnkd.in/dXpaAu9t 2. Spotify's Gradual Cloud Migration → Methodically transition massive infrastructure to GCP → Empower teams to provision resources autonomously → Focus engineers on building product, not infrastructure Case study link: https://lnkd.in/dPqtQmNm 3. Netflix's Cloud Migration Journey → Deploy thousands of servers in minutes → Support 100M+ members on cloud-native infrastructure → Rapidly innovate and roll out new features globally Case study link: https://lnkd.in/dE9aKC6d Cloud Engineer version: https://lnkd.in/dexcS5DQ 4. Dropbox’s Reverse Migration, From Cloud to Own Data Centers Key learnings: → Migrate from cloud to in-house hardware → Build custom infrastructure to optimize for costs → Achieve massive scale through tailored technical solutions Case study link: https://lnkd.in/dU5_ecGJ 5. Canva's Cost Optimization → Leverage Savings Plans and RI to reduce costs → Distribute service costs via scalable microservices → Maintain reliability while optimizing AWS spend Case study link: https://lnkd.in/dT9ryqj3 Here's what you should focus on: - What design patterns & migration strategy were chosen - How scalability and reliability were achieved - Key challenges and how they were solved - Cloud services and tool stack used for a business problem Don’t get distracted by cloud services — start with the problem. Most successful case studies solved core issues with clear architecture and thoughtful workflows. If you found this useful.. • • • ♻️ Share so others can learn as well

60

12 Comments

Ever wondered how to protect your Kubernetes workloads during maintenance or node scaling? In this video, I break down Pod Disruption Budgets - what they are, why they matter, and how to configure them effectively. 💡 What’s Inside? ✅ Clear explanation of PDB concepts ✅ Real-world scenarios where PDB saves the day ✅ Step-by-step hands-on demonstration in Kubernetes ✅ Tips for avoiding downtime during voluntary disruptions Whether you’re a Kubernetes beginner or an experienced DevOps engineer, this session will help you confidently set up PDBs and keep your applications running smoothly. Watch now and level up your Kubernetes reliability game! #Kubernetes #DevOps #SRE #Cloud Native Computing Foundation (CNCF) #PodDisruptionBudget https://lnkd.in/dm4wCnUh

48

1 Comment