<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: DevOps Daily</title>
    <description>The latest articles on DEV Community by DevOps Daily (@devopsdaily).</description>
    <link>https://dev.to/devopsdaily</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F382434%2F66e04ef9-7e4f-491c-997a-30f4a999d40c.jpg</url>
      <title>DEV Community: DevOps Daily</title>
      <link>https://dev.to/devopsdaily</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/devopsdaily"/>
    <language>en</language>
    <item>
      <title>Five Production Bugs You Can Reproduce in a Browser Tab</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Fri, 13 Nov 2026 15:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/five-production-bugs-you-can-reproduce-in-a-browser-tab-5gj8</link>
      <guid>https://dev.to/devopsdaily/five-production-bugs-you-can-reproduce-in-a-browser-tab-5gj8</guid>
      <description>&lt;p&gt;Many production failures are easy to name after the fact and hard to see while you are building: a cache key contains too little or too much, a retry creates a second payment, or a request cache outlives the request. The details differ, but each failure starts with an assumption about what a protocol, intermediary, or library remembers.&lt;/p&gt;

&lt;p&gt;The five free DevOps Daily labs linked below make those assumptions visible. This is not a tour of the labs. Each section starts with the operational symptom, explains the protocol rule, gives you a check to run against your own system, then provides a compact reproduction. You can use the post without opening a lab.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;First thing to inspect&lt;/th&gt;
&lt;th&gt;Safe default&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A shared cache returns personal content&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Cache-Control&lt;/code&gt;, &lt;code&gt;Vary&lt;/code&gt;, and the cache key&lt;/td&gt;
&lt;td&gt;Mark personal responses &lt;code&gt;private&lt;/code&gt; or &lt;code&gt;no-store&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One lost packet delays a page&lt;/td&gt;
&lt;td&gt;negotiated HTTP version and loss-sensitive waterfalls&lt;/td&gt;
&lt;td&gt;Test HTTP/2 and HTTP/3 on the links your users have&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A legitimate message fails DMARC&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Authentication-Results&lt;/code&gt; and alignment&lt;/td&gt;
&lt;td&gt;Align either SPF or DKIM with visible &lt;code&gt;From&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A retry creates another operation&lt;/td&gt;
&lt;td&gt;idempotency key, request fingerprint, saved response&lt;/td&gt;
&lt;td&gt;One key per intended operation, reused only for retries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nested GraphQL fields make too many queries&lt;/td&gt;
&lt;td&gt;resolver SQL and DataLoader scope&lt;/td&gt;
&lt;td&gt;Batch per request, never across requests&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A terminal with &lt;code&gt;curl&lt;/code&gt; and &lt;code&gt;dig&lt;/code&gt; (from &lt;code&gt;bind9-dnsutils&lt;/code&gt; or &lt;code&gt;bind-utils&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Read access to one of your own HTTP endpoints, one sending domain, and one API that accepts retries.&lt;/li&gt;
&lt;li&gt;A modern browser for the labs. They are free and run in the browser.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A shared cache serves a cart to the wrong person
&lt;/h2&gt;

&lt;p&gt;The frightening version of this bug is a user seeing somebody else's cart, profile, or account page. The quieter version is a cache hit ratio that collapses because every campaign link has a distinct tracking query string. Both are cache-key errors: the cache either treats two different representations as the same object, or one representation as too many objects.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9111" rel="noopener noreferrer"&gt;RFC 9111&lt;/a&gt; says a cache key normally includes the request method and target URI. A cache that stores a response selected by &lt;code&gt;Vary&lt;/code&gt; must also use the nominated request-field values to choose a matching response. The response controls matter just as much: &lt;code&gt;private&lt;/code&gt; prohibits storage in a shared cache, and &lt;code&gt;no-store&lt;/code&gt; says not to store it. &lt;a href="https://www.rfc-editor.org/rfc/rfc5861#section-3" rel="noopener noreferrer"&gt;RFC 5861&lt;/a&gt; defines &lt;code&gt;stale-while-revalidate&lt;/code&gt;, which lets a cache serve a stale response while it validates in the background.&lt;/p&gt;

&lt;p&gt;Start with a response that must be private and an asset that should be reusable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI https://app.example.com/api/cart
curl -sI https://app.example.com/assets/app.8d42c.js

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cart response should show &lt;code&gt;Cache-Control: private&lt;/code&gt; or &lt;code&gt;Cache-Control: no-store&lt;/code&gt;. A content-addressed asset can use a long &lt;code&gt;max-age&lt;/code&gt;. For a cacheable response, read &lt;code&gt;Age&lt;/code&gt;, &lt;code&gt;Vary&lt;/code&gt;, and &lt;code&gt;ETag&lt;/code&gt;; these are standardized response fields, though any vendor cache-status header is not. Then test validation with the actual entity tag you received:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;post_etag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'"replace-with-the-etag-value"'&lt;/span&gt;
curl -sSI \
  -H &lt;span class="s2"&gt;"If-None-Match: ${post_etag}"&lt;/span&gt; \
  https://app.example.com/index.html

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For an unchanged representation, the expected shape is &lt;code&gt;HTTP/... 304&lt;/code&gt; with an &lt;code&gt;ETag&lt;/code&gt;. A &lt;code&gt;304&lt;/code&gt; confirms conditional validation, not that every edge has the same object. Review whether your CDN intentionally removes tracking parameters from its cache key before deciding that &lt;code&gt;/pricing?utm_source=...&lt;/code&gt; and &lt;code&gt;/pricing&lt;/code&gt; are equivalent.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/cdn-edge-cache-lab" rel="noopener noreferrer"&gt;Reproduce it in the CDN Edge Cache Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Select &lt;code&gt;/api/cart&lt;/code&gt; and press &lt;strong&gt;Send traffic&lt;/strong&gt;. Observe the &lt;code&gt;BYPASS&lt;/code&gt; results when its response is private or no-store.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;utm_ links&lt;/strong&gt; , send traffic, and inspect the multiple cached keys.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;Ignore utm_ in cache key&lt;/strong&gt; , then send traffic again to see later visitors get &lt;code&gt;HIT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Select &lt;code&gt;/index.html&lt;/code&gt;, send traffic, choose &lt;strong&gt;Wait 61 s&lt;/strong&gt; , and send traffic again to expose a &lt;code&gt;304&lt;/code&gt; revalidation.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The lab models one origin and three CDN edges. Its latency values are illustrative estimates, and it deliberately does not apply heuristic freshness when an explicit lifetime is absent.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  One lost packet pauses every active HTTP/2 stream
&lt;/h2&gt;

&lt;p&gt;HTTP/2 usually removes the browser-side request queues that made HTTP/1.1 pages painful: it multiplexes streams on one connection. The tradeoff is lower in the stack. HTTP/2 runs over TCP, and TCP delivers bytes in order. If a TCP packet containing bytes for one stream is missing, later bytes for all active streams on that connection wait for recovery. On a lossy high-latency path, that can make HTTP/2 slower than several HTTP/1.1 connections carrying independent TCP flows.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9113" rel="noopener noreferrer"&gt;RFC 9113&lt;/a&gt; defines HTTP/2 streams and multiplexed framing. &lt;a href="https://www.rfc-editor.org/rfc/rfc9114#section-1" rel="noopener noreferrer"&gt;RFC 9114&lt;/a&gt; describes HTTP/3 as HTTP semantics over QUIC and calls out avoiding TCP head-of-line blocking. QUIC streams have independent delivery and loss recovery under &lt;a href="https://www.rfc-editor.org/rfc/rfc9000#section-2.2" rel="noopener noreferrer"&gt;RFC 9000&lt;/a&gt;, so a lost packet holds up only the streams whose data it carried, while streams with data in other packets keep moving. That removes TCP's connection-wide head-of-line blocking, not every stall.&lt;/p&gt;

&lt;p&gt;Check what your endpoint actually negotiates before drawing conclusions from an application dashboard:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI --http2 -w &lt;span class="s1"&gt;'%{http_version}\n'&lt;/span&gt; -o /dev/null https://app.example.com
curl -sI --http3-only -w &lt;span class="s1"&gt;'%{http_version}\n'&lt;/span&gt; -o /dev/null https://app.example.com
curl -sI https://app.example.com &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="s1"&gt;'^alt-svc:'&lt;/span&gt;
curl -V

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first two commands print the negotiated version, such as &lt;code&gt;2&lt;/code&gt; or &lt;code&gt;3&lt;/code&gt;. Use &lt;code&gt;--http3-only&lt;/code&gt; for the check: plain &lt;code&gt;--http3&lt;/code&gt; falls back to an earlier version when QUIC cannot be set up, which hides the answer you are looking for. The &lt;code&gt;Alt-Svc&lt;/code&gt; header can advertise an &lt;code&gt;h3&lt;/code&gt; alternative service. Before running either command, check &lt;code&gt;curl -V&lt;/code&gt;: its Features line must list &lt;code&gt;HTTP2&lt;/code&gt; for the first and &lt;code&gt;HTTP3&lt;/code&gt; for the second, and &lt;code&gt;--http3-only&lt;/code&gt; needs curl 7.88.0 or later. Test on representative networks as well as a clean office connection: protocol choice is a latency and loss question, not a permanent leaderboard.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One packet lost&lt;/strong&gt; during app.js&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recover loss&lt;/strong&gt; retransmission needed&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Outcomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;HTTP/2 over TCP: active streams wait for in-order delivery&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;HTTP/3 over QUIC: streams in the lost packet wait, others progress&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/http-versions-lab" rel="noopener noreferrer"&gt;Reproduce it in the HTTP Versions Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Select &lt;strong&gt;HTTP/2&lt;/strong&gt; , set &lt;strong&gt;250 ms RTT&lt;/strong&gt; , turn on &lt;strong&gt;Lose one packet&lt;/strong&gt; , and press &lt;strong&gt;Play all&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Read the waterfall and the loss explanation: multiple receiving streams stall behind the TCP retransmission.&lt;/li&gt;
&lt;li&gt;Switch to &lt;strong&gt;HTTP/3&lt;/strong&gt; , keep &lt;strong&gt;Lose one packet&lt;/strong&gt; on, and press &lt;strong&gt;Play all&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Compare the stalled stream count and the setup steps with &lt;strong&gt;Next&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is a deterministic page-load model, not a browser trace. It excludes DNS, server work, congestion control, priorities, caches, header compression, server push, and QUIC 0-RTT resumption.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A valid-looking email fails DMARC after forwarding
&lt;/h2&gt;

&lt;p&gt;An order confirmation can leave your sender with SPF, DKIM, and DMARC all passing, then fail after a forwarder or a mailing list touches it. The customer sees a rejected message or a spam-folder delivery, while the sending system sees a successful SMTP handoff. The common mistake is treating three authentication results as interchangeable.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc7208" rel="noopener noreferrer"&gt;RFC 7208&lt;/a&gt; specifies SPF for the SMTP MAIL FROM identity and the connecting IP address. &lt;a href="https://www.rfc-editor.org/rfc/rfc6376" rel="noopener noreferrer"&gt;RFC 6376&lt;/a&gt; specifies DKIM signatures over selected headers and the message body, verified using a selector key in DNS. Current DMARC is &lt;a href="https://www.rfc-editor.org/rfc/rfc9989" rel="noopener noreferrer"&gt;RFC 9989&lt;/a&gt;, which requires SPF or DKIM to pass &lt;strong&gt;and&lt;/strong&gt; align with the visible &lt;code&gt;From&lt;/code&gt; domain. A forwarder changes the connecting IP, so SPF often fails; a mailing list that changes a signed body can invalidate DKIM. Rewriting &lt;code&gt;From&lt;/code&gt; is one way a list can preserve DMARC when it modifies a message.&lt;/p&gt;

&lt;p&gt;Inspect the records and one delivered message rather than guessing from a provider UI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig +short TXT example.com
dig +short TXT selector._domainkey.example.com
dig +short TXT _dmarc.example.com

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then open the delivered message source and locate the &lt;code&gt;Authentication-Results&lt;/code&gt; header added by your receiving provider, not an untrusted copy from upstream (&lt;a href="https://www.rfc-editor.org/rfc/rfc8601" rel="noopener noreferrer"&gt;RFC 8601&lt;/a&gt; explains why the header needs a trust check). Its useful shape names the evaluated identities and results, for example &lt;code&gt;spf=pass smtp.mailfrom=...&lt;/code&gt;, &lt;code&gt;dkim=pass header.d=...&lt;/code&gt;, and &lt;code&gt;dmarc=pass header.from=...&lt;/code&gt;. Confirm the domain in &lt;code&gt;header.from&lt;/code&gt; aligns with either the SPF MAIL FROM domain or the DKIM &lt;code&gt;d=&lt;/code&gt; domain. The earlier &lt;a href="https://dev.to/devopsdaily/dmarcbis-is-here-what-changed-in-the-new-dmarc-and-what-to-do-to-your-records-413h"&gt;DMARCbis guide&lt;/a&gt; covers the record migration in more detail: DMARCbis replaced RFC 7489 and removed its &lt;code&gt;pct&lt;/code&gt; tag.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/email-auth-lab" rel="noopener noreferrer"&gt;Reproduce it in the Email Authentication Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Select &lt;code&gt;d=shop.example&lt;/code&gt; for &lt;strong&gt;DKIM signing domain&lt;/strong&gt; , publish the selector, and choose a DMARC policy.&lt;/li&gt;
&lt;li&gt;Send the &lt;strong&gt;Direct&lt;/strong&gt; path, then send the &lt;strong&gt;Forwarded&lt;/strong&gt; path (the forwarder does not use SRS) and compare the SPF, DKIM, and DMARC verdicts.&lt;/li&gt;
&lt;li&gt;Select the mailing-list path, turn on &lt;strong&gt;List rewrites From and sets Reply-To&lt;/strong&gt; , and send again.&lt;/li&gt;
&lt;li&gt;Change &lt;strong&gt;SPF alignment&lt;/strong&gt; or &lt;strong&gt;DKIM alignment&lt;/strong&gt; from relaxed to strict and read the receiver result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The lab checks the MAIL FROM identity, not HELO, and does not model SRS or ARC. Its percentage control deliberately demonstrates the retired RFC 7489 &lt;code&gt;pct&lt;/code&gt; behavior; do not use it as a model of current DMARCbis policy.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  An ambiguous retry creates a second payment
&lt;/h2&gt;

&lt;p&gt;The failure looks like a double charge, duplicate order, or two provisioned resources. Often the first request did succeed, but its response was lost after the server committed work. Retrying is the correct client action after an ambiguous failure. Retrying it as a new operation is not.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://docs.stripe.com/api/idempotent_requests" rel="noopener noreferrer"&gt;Stripe's idempotent requests documentation&lt;/a&gt; describes the pattern: the API saves the status code and body of the first request for a key and returns them for later requests with the same key, and it rejects a reuse of the key with different parameters. The IETF is standardizing the header itself in &lt;a href="https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/" rel="noopener noreferrer"&gt;The Idempotency-Key HTTP Header Field&lt;/a&gt;, which is still an Internet-Draft, not an RFC. A key identifies one intended operation, not one customer or one retry attempt. Store a request fingerprint beside the key, reject a payload mismatch, and distinguish an in-progress duplicate from a completed replay.&lt;/p&gt;

&lt;p&gt;Here is the client side. It keeps one generated key for every retry and applies full jitter to retryable failures:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;randomUUID&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'node:crypto'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;idempotencyKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// One intended order, not one HTTP attempt.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;retryableStatuses&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;409&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;502&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;503&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;504&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;createOrder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'https://api.example.com/orders'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'POST'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="c1"&gt;// Stripe documents a bare value; the IETF draft defines a quoted string.&lt;/span&gt;
        &lt;span class="c1"&gt;// Use the serialization your API documents.&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'application/json'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'Idempotency-Key'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;idempotencyKey&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;AbortSignal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// a stalled request must reach the retry path&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'pro-monthly'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;customerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'cus_123'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;retryableStatuses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="c1"&gt;// A 409 is retryable here only because this API uses it for an in-progress key.&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Idempotent request did not complete after five attempts'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;ceilingMs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;setTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ceilingMs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the server, make the uniqueness boundary visible. Scope keys to the authenticated caller, so one client's key can never replay another client's saved response, and save the fingerprint and final response before a later retry can replay it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;idempotency_responses&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;account_id&lt;/span&gt; &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- from authentication, never from the request body&lt;/span&gt;
  &lt;span class="n"&gt;idempotency_key&lt;/span&gt; &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;request_fingerprint&lt;/span&gt; &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="kt"&gt;integer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;response_body&lt;/span&gt; &lt;span class="kt"&gt;jsonb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;completed_at&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;account_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idempotency_key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every lookup, insert and replay filters on both columns. The &lt;a href="https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/" rel="noopener noreferrer"&gt;Idempotency-Key draft&lt;/a&gt; calls out data leaks from shared keys for the same reason.&lt;/p&gt;

&lt;p&gt;Use a transaction or equivalent concurrency control to insert the key, create the order, and save the response. A matching completed row returns &lt;code&gt;status_code&lt;/code&gt; and &lt;code&gt;response_body&lt;/code&gt;; a differing fingerprint returns a client error; a matching unfinished row should tell the client the first request is still processing. The same pattern matters for webhooks, where a delivery can be retried after your handler completes. See &lt;a href="https://dev.to/devopsdaily/how-stripe-avoids-double-charging-anyone-2me1"&gt;how Stripe avoids double charging&lt;/a&gt; and &lt;a href="https://dev.to/devopsdaily/what-it-actually-takes-to-deliver-a-webhook-in-production-4ghe"&gt;reliable webhook delivery&lt;/a&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/idempotency-retry-lab" rel="noopener noreferrer"&gt;Reproduce it in the Idempotency and Retry Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Choose &lt;strong&gt;Run duplicate checkout&lt;/strong&gt; and inspect the ledger after the lost response.&lt;/li&gt;
&lt;li&gt;Choose &lt;strong&gt;Run keyed checkout&lt;/strong&gt; and find the saved result in &lt;strong&gt;Idempotency store&lt;/strong&gt; instead of another charge.&lt;/li&gt;
&lt;li&gt;Press &lt;strong&gt;Reuse key with new amount&lt;/strong&gt; , then &lt;strong&gt;Send two at once&lt;/strong&gt; , and compare the mismatch and in-progress outcomes.&lt;/li&gt;
&lt;li&gt;Run &lt;strong&gt;Run no backoff&lt;/strong&gt; and &lt;strong&gt;Run full jitter&lt;/strong&gt; , then use &lt;strong&gt;Run down API job&lt;/strong&gt; to inspect the dead letter.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The outage chart is a simplified deterministic model of client arrivals, not a production measurement. Its dead-letter jobs retain a key and final error and do not retry themselves.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A shared DataLoader exposes data from the previous request
&lt;/h2&gt;

&lt;p&gt;GraphQL makes N+1 look harmless. &lt;code&gt;posts { author { name } }&lt;/code&gt; reads like one query, but a resolver that fetches every author separately can produce one posts query plus one author query per post. Add nested comments and comment authors, and the work grows at each resolver level.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://github.com/graphql/dataloader#batching" rel="noopener noreferrer"&gt;DataLoader README&lt;/a&gt; describes batching loads that occur in the same execution frame and caching values in a loader instance. The cache is intentionally not a shared application cache. The README's &lt;a href="https://github.com/graphql/dataloader#caching-per-request" rel="noopener noreferrer"&gt;per-request caching guidance&lt;/a&gt; says to create new loader instances for each request because authorization or other request-specific context can differ.&lt;/p&gt;

&lt;p&gt;Create loaders in the GraphQL context, not at module scope. This Apollo Server pattern puts the loader beside the authenticated request context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;ApolloServer&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@apollo/server'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;DataLoader&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'dataloader'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;startStandaloneServer&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@apollo/server/standalone'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ApolloServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;typeDefs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resolvers&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;startStandaloneServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;viewer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;authenticate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;authorization&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;viewer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="c1"&gt;// A new loader per request: its cache dies with the request.&lt;/span&gt;
      &lt;span class="n"&gt;userById&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;DataLoader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Authorization lives in the data access: only rows this viewer may see.&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;loadUsersVisibleTo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;viewer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;loadUsersVisibleTo&lt;/code&gt; stands in for your own query, for example &lt;code&gt;SELECT id, name FROM users WHERE id = ANY($1::uuid[])&lt;/code&gt; plus the viewer's visibility rule. A per-request loader stops one request's cache from serving the next; it does not authorize reads inside a request, so the permission check has to be in the batch function.&lt;/p&gt;

&lt;p&gt;Without the loader, resolver logs often show repeated &lt;code&gt;SELECT ... WHERE id = $1&lt;/code&gt; statements. With it, same-tick loads become one &lt;code&gt;WHERE id IN (...)&lt;/code&gt; query or its parameterized equivalent. Deduplication reduces repeated keys in that batch, but it does not authorize the cached result for another request. Log query count and the request ID together so a shared cache is visible before it becomes a data exposure.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/graphql-dataloader-lab" rel="noopener noreferrer"&gt;Reproduce it in the GraphQL N+1 and DataLoader Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Leave &lt;strong&gt;Post authors&lt;/strong&gt; on, keep &lt;strong&gt;Use User DataLoader&lt;/strong&gt; off, and press &lt;strong&gt;Run query&lt;/strong&gt;. Inspect the SQL panel.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;Comments&lt;/strong&gt; and &lt;strong&gt;Comment authors&lt;/strong&gt; , then run again to see the nested resolver work.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;Use User DataLoader&lt;/strong&gt; and &lt;strong&gt;Use Comments DataLoader&lt;/strong&gt; to see the batched &lt;code&gt;IN (...)&lt;/code&gt; queries.&lt;/li&gt;
&lt;li&gt;Set &lt;strong&gt;Loader scope&lt;/strong&gt; to &lt;strong&gt;Shared by all requests&lt;/strong&gt; , run as Ada, then run as Bo. Set it back to &lt;strong&gt;Per request&lt;/strong&gt; and run Bo again.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The lab has a small deterministic data set. It models one event-loop tick per resolver level; &lt;code&gt;await&lt;/code&gt; calls between loads can split real DataLoader batches.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Keep the state boundary explicit
&lt;/h2&gt;

&lt;p&gt;These bugs look unrelated until you name the state boundary. A CDN decides which request fields identify a response. TCP decides which bytes must arrive before later bytes are delivered. DMARC decides which authenticated identifier may stand for &lt;code&gt;From&lt;/code&gt;. An idempotency store decides whether a retry is an earlier operation. DataLoader decides how long a loaded value remains valid.&lt;/p&gt;

&lt;p&gt;Use the checks before an incident, then use the labs to make the failure shape memorable.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bug&lt;/th&gt;
&lt;th&gt;Mechanism&lt;/th&gt;
&lt;th&gt;One-line fix&lt;/th&gt;
&lt;th&gt;Lab&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Personal content at the edge&lt;/td&gt;
&lt;td&gt;Shared cache key or directives allow storage&lt;/td&gt;
&lt;td&gt;Send &lt;code&gt;private&lt;/code&gt; or &lt;code&gt;no-store&lt;/code&gt; for personalized responses&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/cdn-edge-cache-lab" rel="noopener noreferrer"&gt;CDN Edge Cache Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP/2 page stalls on loss&lt;/td&gt;
&lt;td&gt;TCP in-order delivery blocks active streams&lt;/td&gt;
&lt;td&gt;Test QUIC and HTTP/3 where loss and RTT warrant it&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/http-versions-lab" rel="noopener noreferrer"&gt;HTTP Versions Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DMARC fails after an intermediary&lt;/td&gt;
&lt;td&gt;SPF or DKIM no longer passes and aligns&lt;/td&gt;
&lt;td&gt;Preserve an aligned identity through forwarding and lists&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/email-auth-lab" rel="noopener noreferrer"&gt;Email Authentication Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duplicate work after retry&lt;/td&gt;
&lt;td&gt;New key makes a retry a new operation&lt;/td&gt;
&lt;td&gt;Reuse one key and replay its saved response&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/idempotency-retry-lab" rel="noopener noreferrer"&gt;Idempotency and Retry Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GraphQL data leak or query storm&lt;/td&gt;
&lt;td&gt;Loader cache crosses requests, or loads are unbatched&lt;/td&gt;
&lt;td&gt;Instantiate loaders in request context&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/graphql-dataloader-lab" rel="noopener noreferrer"&gt;GraphQL N+1 and DataLoader Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/five-production-bugs-you-can-reproduce-in-a-browser" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>http</category>
      <category>cdn</category>
      <category>email</category>
    </item>
    <item>
      <title>CKNE Is Live: What the Kubernetes Network Engineer Exam Tests and How to Practice</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Fri, 13 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/ckne-is-live-what-the-kubernetes-network-engineer-exam-tests-and-how-to-practice-276</link>
      <guid>https://dev.to/devopsdaily/ckne-is-live-what-the-kubernetes-network-engineer-exam-tests-and-how-to-practice-276</guid>
      <description>&lt;p&gt;Kubernetes networking failures rarely announce which layer is wrong. A Service can have no ready endpoints, a policy can block DNS, a route can point at the wrong interface, or a Gateway can accept a listener but reject the route. The new &lt;strong&gt;Certified Kubernetes Network Engineer (CKNE)&lt;/strong&gt; asks you to find and fix problems like these from a terminal under time pressure. This guide turns the published curriculum into drills you can repeat in a disposable cluster.&lt;/p&gt;

&lt;p&gt;This is not an account of taking the exam. It is a study guide built from the &lt;a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer"&gt;published CKNE curriculum&lt;/a&gt; and the &lt;a href="https://www.cncf.io/announcements/2026/11/10/cncf-launches-certified-kubernetes-network-engineer-certification-ckne-to-advance-production-ready-networking-skills-for-ai-era/" rel="noopener noreferrer"&gt;CNCF launch announcement&lt;/a&gt;. Treat every lab as a loop: break one thing deliberately, prove where the packet stops, repair it, then retest from the client.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CKNE is a performance-based, two-hour exam in a Linux remote desktop. Its largest domains are Service Networking and DNS, and Network Security and Policy, at 25% each.&lt;/li&gt;
&lt;li&gt;Start with CKA-level cluster administration. There are no formal prerequisites, but both CNCF and Linux Foundation Education say the exam assumes that baseline.&lt;/li&gt;
&lt;li&gt;Build one lab with a real policy-enforcing CNI. Then practice resolving a name, examining EndpointSlices, reading node routes, applying a policy, and observing the resulting flow without reaching for a GUI first.&lt;/li&gt;
&lt;li&gt;As of 2026-11-11, the official exam page does not publish a passing score, Kubernetes version, or number of tasks. Check the &lt;a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer"&gt;candidate handbook and FAQ&lt;/a&gt; before booking.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CKA-level comfort with &lt;code&gt;kubectl&lt;/code&gt;: contexts, namespaces, Deployments, Services, logs, and &lt;code&gt;describe&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A workstation that can run a three-node &lt;a href="https://kind.sigs.k8s.io/" rel="noopener noreferrer"&gt;kind&lt;/a&gt; cluster in Docker or Podman.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;kubectl&lt;/code&gt;, &lt;code&gt;kind&lt;/code&gt;, the &lt;a href="https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/" rel="noopener noreferrer"&gt;Cilium CLI&lt;/a&gt;, and the &lt;a href="https://docs.cilium.io/en/stable/observability/hubble/setup/" rel="noopener noreferrer"&gt;Hubble CLI&lt;/a&gt; installed.&lt;/li&gt;
&lt;li&gt;Basic Linux networking: &lt;code&gt;ip&lt;/code&gt;, &lt;code&gt;ss&lt;/code&gt;, &lt;code&gt;tcpdump&lt;/code&gt;, and reading a route table.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What CKNE tests, and whether it fits you
&lt;/h2&gt;

&lt;p&gt;CKNE is for an administrator who already knows how to inspect workloads and cluster resources, and now needs to reason about the data path between them. It has no prerequisite certification, but doing CKA first is a sensible order because a networking drill should not also be a &lt;code&gt;kubectl&lt;/code&gt; syntax drill.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer"&gt;Linux Foundation certification page&lt;/a&gt; lists the exam-only price as $445, the exam plus THRIVE-ONE annual subscription as $625, and the exam plus Certification Pathway as $645. Eligibility lasts 12 months, one retake is included, and the certification is valid for two years. Those are purchase decisions, not study requirements. The useful practice resource bundled with the exam is the &lt;a href="https://killer.sh/ckne" rel="noopener noreferrer"&gt;Killer.sh CKNE simulator&lt;/a&gt;: two sessions, each open for 36 hours after activation. (The Linux Foundation page and Killer.sh currently give different question counts per session, 17 and 20, so do not plan around an exact number.) Use those sessions late in preparation to test navigation, pacing, and recovery after a bad assumption, not as your first exposure to a CNI or Gateway API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CKNE published domain weights&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Core Infrastructure and CNI&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service Networking and DNS&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Advanced Traffic Management&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network Security and Policy&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Observability&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Published by Linux Foundation Education and CNCF. Checked 2026-11-11.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Warm up concepts in the browser before you open the terminal. The &lt;a href="https://devops-daily.com/games/kubernetes-networking-cni-simulator" rel="noopener noreferrer"&gt;Kubernetes networking and CNI simulator&lt;/a&gt; traces pod, Service, policy, egress, and ingress paths across overlay, routed, and eBPF dataplanes. Use the &lt;a href="https://devops-daily.com/games/dns-simulator" rel="noopener noreferrer"&gt;DNS simulator&lt;/a&gt; for resolver and cache order, &lt;a href="https://devops-daily.com/games/packet-journey" rel="noopener noreferrer"&gt;Packet Journey&lt;/a&gt; for packet headers and hops, and &lt;a href="https://devops-daily.com/games/tcp-vs-udp" rel="noopener noreferrer"&gt;TCP vs UDP&lt;/a&gt; when you need to refresh transport behavior. The &lt;a href="https://devops-daily.com/games/load-balancer-simulator" rel="noopener noreferrer"&gt;load balancer simulator&lt;/a&gt; is useful for endpoint-selection intuition. For mTLS and policy, use the &lt;a href="https://devops-daily.com/games/service-mesh-simulator" rel="noopener noreferrer"&gt;service mesh simulator&lt;/a&gt; and &lt;a href="https://devops-daily.com/games/ssl-tls-handshake" rel="noopener noreferrer"&gt;SSL/TLS handshake simulator&lt;/a&gt;. They are warm-ups, not substitutes for command-line verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build a practice cluster that can fail interestingly
&lt;/h2&gt;

&lt;p&gt;You need a cluster where NetworkPolicy and encryption actually have an effect. &lt;code&gt;kind&lt;/code&gt; can start without its default CNI, and Cilium can then supply policy enforcement and WireGuard encryption. The &lt;a href="https://kind.sigs.k8s.io/docs/user/configuration/" rel="noopener noreferrer"&gt;kind configuration reference&lt;/a&gt; documents both &lt;code&gt;disableDefaultCNI&lt;/code&gt; and &lt;code&gt;podSubnet&lt;/code&gt;; Cilium documents the encryption settings below.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# ckne-kind.yaml&lt;/span&gt;
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: &lt;span class="kc"&gt;true&lt;/span&gt;
  podSubnet: &lt;span class="s1"&gt;'10.244.0.0/16'&lt;/span&gt;
nodes:
  - role: control-plane
  - role: worker
  - role: worker


kind create cluster --name ckne --config ckne-kind.yaml

&lt;span class="c"&gt;# Installs Cilium, turns on policy enforcement, and encrypts eligible&lt;/span&gt;
&lt;span class="c"&gt;# cross-node Cilium endpoint traffic with WireGuard.&lt;/span&gt;
cilium install \
  --set encryption.enabled=true \
  --set encryption.type=wireguard
cilium status --wait

&lt;span class="c"&gt;# Hubble flow visibility for the observability drills&lt;/span&gt;
cilium hubble &lt;span class="nb"&gt;enable&lt;/span&gt;
cilium hubble port-forward &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
hubble status

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;WireGuard encrypts Cilium-managed traffic between nodes, not traffic on the same node. Confirm the control plane before trusting the lab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n kube-system &lt;span class="nb"&gt;exec&lt;/span&gt; ds/cilium -- cilium-dbg status &lt;span class="o"&gt;|&lt;/span&gt; grep Encryption
kubectl get nodes -o wide
kubectl get pods -A

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a second interface exercise, install Multus and create a &lt;code&gt;NetworkAttachmentDefinition&lt;/code&gt;, whose API version is &lt;code&gt;k8s.cni.cncf.io/v1&lt;/code&gt;. A pod requests it with the &lt;code&gt;k8s.v1.cni.cncf.io/networks&lt;/code&gt; annotation. The &lt;a href="https://k8snetworkplumbingwg.github.io/multus-cni/docs/quickstart.html" rel="noopener noreferrer"&gt;Multus quickstart&lt;/a&gt; shows this attachment model and verifies the additional &lt;code&gt;net1&lt;/code&gt; interface. Do not memorize the YAML in isolation: check &lt;code&gt;ip addr&lt;/code&gt; and &lt;code&gt;ip route&lt;/code&gt; in the pod and ask which interface owns the default route.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;k8s.cni.cncf.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkAttachmentDefinition&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab-net&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'{"cniVersion":"1.0.0","type":"macvlan","master":"eth0","mode":"bridge","ipam":{"type":"host-local","subnet":"192.0.2.0/24"}}'&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pod&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;dual-homed&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;k8s.v1.cni.cncf.io/networks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab-net&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shell&lt;/span&gt;
      &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;busybox&lt;/span&gt;
      &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'sh'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'-c'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'sleep 3600'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The NAD is only a definition. &lt;code&gt;master&lt;/code&gt; must name an interface that exists on every node that can schedule the Pod (kind nodes have &lt;code&gt;eth0&lt;/code&gt;; check yours in the node's host namespace first), and the &lt;code&gt;macvlan&lt;/code&gt; and &lt;code&gt;host-local&lt;/code&gt; plugin binaries must be present in the node's CNI directory. Use the Multus documentation for the plugin you choose. That distinction is a good troubleshooting habit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Practice every published domain as a terminal drill
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Core Infrastructure and CNI: trace before changing
&lt;/h3&gt;

&lt;p&gt;Know how pod CIDRs are allocated by your lab CNI, which node owns a destination CIDR, and whether traffic is routed or encapsulated. Under a clock, start with &lt;code&gt;kubectl get nodes -o wide&lt;/code&gt;, pod IPs, and routes before changing any YAML.&lt;/p&gt;

&lt;p&gt;Use an ephemeral node debug container to inspect the host network namespace. &lt;code&gt;ip route&lt;/code&gt; and &lt;code&gt;ip addr&lt;/code&gt; tell you what the node believes. A capture on port 53 separates a DNS transport problem from a bad record or bad search path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Open a shell in the node's network namespace (pick an image that ships tcpdump)&lt;/span&gt;
&lt;span class="nv"&gt;NODE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')"&lt;/span&gt;
kubectl debug &lt;span class="s2"&gt;"node/${NODE}"&lt;/span&gt; -it --image=nicolaka/netshoot --profile=sysadmin

&lt;span class="c"&gt;# What does the node believe about addresses and routes?&lt;/span&gt;
ip addr
ip route

&lt;span class="c"&gt;# Watch DNS on every interface while a pod resolves a name&lt;/span&gt;
tcpdump -ni any port 53

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Be able to distinguish kube-proxy modes from CNI service replacements. Inspect the actual cluster configuration before assuming &lt;code&gt;iptables&lt;/code&gt; rules explain a Service. Kubernetes supports &lt;code&gt;iptables&lt;/code&gt; and &lt;code&gt;nftables&lt;/code&gt; modes, while the &lt;a href="https://dev.to/devopsdaily/kubernetes-137-garhwal-what-shipped-and-what-slipped-1cmi"&gt;Kubernetes 1.37 release guide&lt;/a&gt; covers why &lt;code&gt;ipvs&lt;/code&gt; is deprecated and should not be the new thing you learn for this exam.&lt;/p&gt;

&lt;h3&gt;
  
  
  Service Networking and DNS: prove the backend set
&lt;/h3&gt;

&lt;p&gt;For every Service type, state what it exposes and how a client reaches it: &lt;code&gt;ClusterIP&lt;/code&gt; is cluster-internal virtual addressing, &lt;code&gt;NodePort&lt;/code&gt; exposes a port on node addresses, &lt;code&gt;LoadBalancer&lt;/code&gt; relies on an implementation to expose it externally, and a headless Service has no cluster IP for virtual-IP proxying. Then inspect the real backend set:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get svc payments -o yaml
kubectl get endpointslices \
  -l kubernetes.io/service-name=payments \
  -o yaml
kubectl get pods -l &lt;span class="nv"&gt;app&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;payments

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;EndpointSlices are the source of Service backends that a proxy watches, and a Pod's readiness maps to the endpoint's readiness condition. Make one backend unready and watch the slice change. For external traffic, set &lt;code&gt;externalTrafficPolicy: Local&lt;/code&gt; on a NodePort or LoadBalancer test Service, send traffic to a node without a local endpoint, and explain the drop. &lt;code&gt;Local&lt;/code&gt; preserves source IP by forwarding only to node-local endpoints, so it exchanges reachability on every node for that property. The &lt;a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noopener noreferrer"&gt;Kubernetes Service documentation&lt;/a&gt; and &lt;a href="https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/" rel="noopener noreferrer"&gt;EndpointSlice documentation&lt;/a&gt; define those behaviors.&lt;/p&gt;

&lt;p&gt;DNS should be its own timed drill. From a pod, resolve a full Service name, inspect &lt;code&gt;/etc/resolv.conf&lt;/code&gt;, then try the short name. Kubernetes commonly configures search domains and &lt;code&gt;options ndots:5&lt;/code&gt;; read the file in the workload instead of assuming a particular setting.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# The dnsutils pod from the Kubernetes DNS debugging guide&lt;/span&gt;
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl &lt;span class="nb"&gt;exec&lt;/span&gt; -it dnsutils -- cat /etc/resolv.conf
kubectl &lt;span class="nb"&gt;exec&lt;/span&gt; -it dnsutils -- nslookup kubernetes.default.svc.cluster.local
kubectl &lt;span class="nb"&gt;exec&lt;/span&gt; -it dnsutils -- nslookup kubernetes.default

kubectl -n kube-system get configmap coredns -o yaml
kubectl -n kube-system logs -l k8s-app=kube-dns

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a configuration drill, edit the CoreDNS ConfigMap in a disposable cluster and add a name rewrite inside the existing server block. Its position in the Corefile does not matter: CoreDNS fixes the &lt;a href="https://coredns.io/manual/configuration/" rel="noopener noreferrer"&gt;plugin execution order&lt;/a&gt; when it is built. &lt;code&gt;answer auto&lt;/code&gt; rewrites the response back to the name the client asked for, which the &lt;a href="https://coredns.io/plugins/rewrite/" rel="noopener noreferrer"&gt;rewrite plugin&lt;/a&gt; requires for suffix rewrites to look right to the client.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight vcl"&gt;&lt;code&gt;rewrite stop &lt;span class="o"&gt;{&lt;/span&gt;
    name suffix .example.internal. .default.svc.cluster.local.
    answer auto
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That maps &lt;code&gt;payments.example.internal&lt;/code&gt; to &lt;code&gt;payments.default.svc.cluster.local&lt;/code&gt;. Restart the CoreDNS deployment if your distribution does not reload the Corefile, then test both names. Also find its metrics endpoint in the installed Corefile before deciding that a dashboard is a DNS diagnosis. Kubernetes has a useful &lt;a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noopener noreferrer"&gt;DNS debugging sequence&lt;/a&gt; for this exact progression.&lt;/p&gt;

&lt;h3&gt;
  
  
  Advanced Traffic Management: make the controller part explicit
&lt;/h3&gt;

&lt;p&gt;Install the Gateway API Standard channel from its current release URL, then use the &lt;code&gt;GatewayClass&lt;/code&gt; supplied by the controller you installed. CRDs alone do not route traffic. The &lt;a href="https://gateway-api.sigs.k8s.io/guides/getting-started/introduction/" rel="noopener noreferrer"&gt;Gateway API installation guide&lt;/a&gt; publishes this current Standard-channel command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0/standard-install.yaml

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Next, create a path match and a weighted split. Replace &lt;code&gt;&amp;lt;gateway-class&amp;gt;&lt;/code&gt; with an installed class and create &lt;code&gt;api-tls&lt;/code&gt; as a TLS Secret before applying the Gateway. The listener's Secret reference is the important TLS troubleshooting surface.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Gateway&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;public&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;gatewayClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;lt;gateway-class&amp;gt;&lt;/span&gt;
  &lt;span class="na"&gt;listeners&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https&lt;/span&gt;
      &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPS&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;443&lt;/span&gt;
      &lt;span class="na"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api.example.com&lt;/span&gt;
      &lt;span class="na"&gt;tls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Terminate&lt;/span&gt;
        &lt;span class="na"&gt;certificateRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Secret&lt;/span&gt;
            &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-tls&lt;/span&gt;
      &lt;span class="na"&gt;allowedRoutes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;namespaces&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Same&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPRoute&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;parentRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;public&lt;/span&gt;
  &lt;span class="na"&gt;hostnames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'api.example.com'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PathPrefix&lt;/span&gt;
            &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/v1&lt;/span&gt;
      &lt;span class="na"&gt;backendRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-v1&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt;
          &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-v2&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt;
          &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check &lt;code&gt;kubectl describe gateway public&lt;/code&gt; and &lt;code&gt;kubectl describe httproute api&lt;/code&gt;; accepted objects and programmed addresses are separate questions. If you are migrating from Ingress, see &lt;a href="https://devops-daily.com/posts/ingress-nginx-eol-gateway-api-migration" rel="noopener noreferrer"&gt;our Gateway API migration guide&lt;/a&gt; for a controller-aware cutover.&lt;/p&gt;

&lt;p&gt;LLM traffic belongs here, but do not reduce it to weighted HTTP backends. The &lt;a href="https://gateway-api-inference-extension.sigs.k8s.io/" rel="noopener noreferrer"&gt;Gateway API Inference Extension&lt;/a&gt; currently documents the &lt;code&gt;InferencePool&lt;/code&gt; API (v1) and an alpha &lt;code&gt;InferencePoolImport&lt;/code&gt; and describes endpoint selection using model-serving metrics and capabilities. Practice explaining why model-aware selection can consider a model name, cache state, or adapter availability. Treat its APIs as extension APIs and verify your controller's support and release status before using them in an answer.&lt;/p&gt;

&lt;p&gt;For cluster exit traffic, Cilium's implementation is &lt;a href="https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/" rel="noopener noreferrer"&gt;Egress Gateway&lt;/a&gt;. Enable it with &lt;code&gt;egressGateway.enabled=true&lt;/code&gt; (it also needs &lt;code&gt;bpf.masquerade=true&lt;/code&gt; and &lt;code&gt;kubeProxyReplacement=true&lt;/code&gt;, which the lab install above does not set), then write a &lt;code&gt;CiliumEgressGatewayPolicy&lt;/code&gt; that selects source pods and external destination CIDRs, plus a gateway node and egress IP or interface. Egress Gateway is not compatible with Cilium ClusterMesh. For cross-cluster service discovery, the &lt;a href="https://multicluster.sigs.k8s.io/concepts/multicluster-services-api/" rel="noopener noreferrer"&gt;Multi-Cluster Services API&lt;/a&gt; uses &lt;code&gt;ServiceExport&lt;/code&gt; to mark a Service for export and creates &lt;code&gt;ServiceImport&lt;/code&gt; plus EndpointSlices in consuming clusters. Cilium ClusterMesh is another implementation path; before working through it, read &lt;a href="https://devops-daily.com/posts/cilium-1-19-clustermesh-policy-flip" rel="noopener noreferrer"&gt;the policy-default change that can block cross-cluster traffic&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Network Security and Policy: deny, restore DNS, then prove it
&lt;/h3&gt;

&lt;p&gt;Start a namespace with both ingress and egress default-deny. Then permit only what the workload needs. The classic failure is to allow application traffic while forgetting DNS egress.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default-deny&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;Ingress&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;Egress&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;allow-dns&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;Egress&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;namespaceSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;kubernetes.io/metadata.name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-system&lt;/span&gt;
          &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;k8s-app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-dns&lt;/span&gt;
      &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The label and namespace are cluster conventions, so inspect your DNS Service and Pods before applying this unchanged. Test from a labeled client pod, first expecting a timeout, then allowing the target Service on its actual port and running &lt;code&gt;curl -sv "http://${SERVICE}.${NAMESPACE}.svc.cluster.local:${PORT}/"&lt;/code&gt; with those three variables set to your target. Kubernetes defines NetworkPolicy as an API, but the chosen network plugin must enforce it. The &lt;a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="noopener noreferrer"&gt;NetworkPolicy documentation&lt;/a&gt; is the baseline for the API semantics.&lt;/p&gt;

&lt;p&gt;Encryption is not proof of authorization. In this Cilium lab, confirm WireGuard with &lt;code&gt;cilium-dbg status | grep Encryption&lt;/code&gt;, and capture on &lt;code&gt;cilium_wg0&lt;/code&gt; from a Cilium pod if you need to see the tunnel. Cilium documents the status check, tunnel device, and the fact that same-node traffic is not encrypted in its &lt;a href="https://docs.cilium.io/en/stable/security/network/encryption-wireguard/" rel="noopener noreferrer"&gt;WireGuard guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For pod-level identity, use a service mesh. An Istio &lt;code&gt;AuthorizationPolicy&lt;/code&gt; can allow only a workload identity after mTLS identifies the caller. This is a compact policy drill, not a complete mesh install:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;security.istio.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AuthorizationPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;allow-frontend&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
  &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ALLOW&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;principals&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;cluster.local/ns/lab/sa/frontend&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify the calling ServiceAccount and the mesh's mTLS mode before deciding a denial is an authorization error. Istio documents this defense-in-depth mTLS pattern in its &lt;a href="https://istio.io/latest/docs/ops/configuration/security/security-policy-examples/" rel="noopener noreferrer"&gt;security policy examples&lt;/a&gt;. For route behavior, retries, and traffic splits in a mesh, revisit &lt;a href="https://devops-daily.com/posts/istio-traffic-management-routing-retries-circuit-breaking" rel="noopener noreferrer"&gt;our Istio traffic-management guide&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observability: observe the flow that proves the fix
&lt;/h3&gt;

&lt;p&gt;Your final action should produce evidence. For DNS, inspect CoreDNS logs and the Prometheus metrics configured by its &lt;code&gt;prometheus&lt;/code&gt; plugin. For Services, inspect EndpointSlices and, where relevant, kube-proxy metrics from the endpoint that your cluster exposes. For policy enforcement in Cilium, use Hubble to request the flows you care about:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;hubble observe --pod client --verdict DROPPED
hubble observe --namespace lab --protocol http
kubectl -n kube-system &lt;span class="nb"&gt;exec&lt;/span&gt; ds/cilium -- cilium-dbg status

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://docs.cilium.io/en/latest/observability/hubble/hubble-cli/" rel="noopener noreferrer"&gt;Hubble CLI documentation&lt;/a&gt; verifies the &lt;code&gt;--pod&lt;/code&gt; and &lt;code&gt;--verdict DROPPED&lt;/code&gt; form. Trace a request through a Gateway by sending a request ID such as &lt;code&gt;X-Request-ID&lt;/code&gt; from the client, then find that header in the gateway and application telemetry only if those components are configured to record it. A header alone does not create distributed tracing.&lt;/p&gt;

&lt;h2&gt;
  
  
  A six-week plan and exam-day routine
&lt;/h2&gt;

&lt;p&gt;Use the weights to bias your time, while returning to old drills often enough that command recall becomes automatic.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Week&lt;/th&gt;
&lt;th&gt;Focus&lt;/th&gt;
&lt;th&gt;Deliverable&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Core Infrastructure and CNI&lt;/td&gt;
&lt;td&gt;Explain one same-node and one cross-node packet path with routes and captures.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Service Networking and DNS&lt;/td&gt;
&lt;td&gt;Repair an empty EndpointSlice, a readiness issue, and a broken DNS lookup.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Network Security and Policy&lt;/td&gt;
&lt;td&gt;Build default-deny, restore DNS, allow one application path, and prove both allowed and denied flows.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Advanced Traffic Management&lt;/td&gt;
&lt;td&gt;Configure Gateway plus HTTPRoute, TLS Secret reference, weighted backends, and one egress or multi-cluster concept.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Observability plus weak spots&lt;/td&gt;
&lt;td&gt;Correlate CoreDNS, Service, Hubble, and Gateway evidence for one injected fault.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Timed mixed drills&lt;/td&gt;
&lt;td&gt;Use the two Killer.sh sessions as realistic pacing checks and catalogue every command you had to look up.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On exam day, the published format gives you 120 minutes in a Linux remote desktop with terminal and web interfaces. Spend the opening minutes inventorying contexts, namespaces, nodes, and the resources named by the task. Read status conditions before edits. Use a short loop for each task: observe, form one hypothesis, make the smallest change, and prove the result from the client side. Do not invent browser-tab or reference-material rules from hearsay. The official policies and candidate materials, not a study blog, decide those details.&lt;/p&gt;

&lt;p&gt;The goal is not to memorize every CNI's settings. It is to see a network problem as a chain of claims you can test: name resolves, Service has ready endpoints, dataplane has a route, policy permits it, encryption is configured where expected, and logs or flows show the result.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/ckne-kubernetes-network-engineer-exam-guide" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>networking</category>
      <category>certification</category>
      <category>ckne</category>
    </item>
    <item>
      <title>Who's On Call for the AI Agent? A Production Runbook</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Thu, 12 Nov 2026 15:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/whos-on-call-for-the-ai-agent-a-production-runbook-715</link>
      <guid>https://dev.to/devopsdaily/whos-on-call-for-the-ai-agent-a-production-runbook-715</guid>
      <description>&lt;p&gt;An AI agent that opens pull requests, queries a database, restarts a service, or sends email is already a production service. At 3 a.m., the question is not whether its prompt was well written. It is who can stop it, which identity it used, and how to tell what it did. This runbook gives that service the same operational basics as any other: ownership, a page policy, controls that still work when the model does not, and an audit trail that answers an incident question quickly.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Put every production agent in the service catalog with an owning team, a primary on-call rotation, an escalation path, and named owners for its prompt, tools, and credentials.&lt;/li&gt;
&lt;li&gt;Page for harm that is happening now: an action loop, spend burning above budget, writes outside their window, permission-denied spikes, an aging approval queue, or failed tasks. Ticket slow trends such as evaluation, latency, and cost drift.&lt;/li&gt;
&lt;li&gt;Make the kill switch independent of the model. The tool runner checks it before every call, credentials can be revoked, the gateway applies a cap, and network egress is the final containment layer.&lt;/li&gt;
&lt;li&gt;Give the agent its own short-lived, read-scoped identity. A read-only database role helps, but does not limit query duration or call frequency.&lt;/li&gt;
&lt;li&gt;Record every tool action outside the agent's writable storage. During an incident, preserve that record before deciding whether to resume.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A service catalog and an on-call system with a primary and escalation target.&lt;/li&gt;
&lt;li&gt;A tool runner or gateway where you can enforce policy before an external action starts.&lt;/li&gt;
&lt;li&gt;Metrics, centralized logs, and a protected audit-log destination.&lt;/li&gt;
&lt;li&gt;An identity system that can issue a distinct identity for each agent.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Give the agent an owner before it gets a tool
&lt;/h2&gt;

&lt;p&gt;Treat an agent as a service, not as a feature hidden inside another service. Create a catalog entry that names its purpose, environments, repository, dashboards, runbook, primary rotation, and escalation path. A stale record is worse than a missing one when responders need to find the right team, which is why the catalog itself needs ownership and review. Our &lt;a href="https://dev.to/devopsdaily/a-service-catalog-nobody-updates-can-be-worse-than-none-we-checked-217-entries-against-github-1do7"&gt;service-catalog drift check&lt;/a&gt; shows how easily ownership records can stop matching reality.&lt;/p&gt;

&lt;p&gt;Split accountability by control surface. One team may own the agent's prompt, model choice, and configuration. The team that owns each target system owns the tool contract and allowed actions. The identity or platform team owns credential issuance, revocation, and the gateway policy. The catalog entry should name all three, but it must name one accountable service owner who receives the page and drives the incident.&lt;/p&gt;

&lt;p&gt;Write the escalation rule down with the service, not in a chat thread. The primary responder acknowledges and contains the agent. If the incident touches a target service, page that service's owner. Escalate to security when credentials, untrusted instructions, or data exposure are involved. The rotation and its escalation policy should have the same attention as any other production service. See the &lt;a href="https://devops-daily.com/posts/on-call-rotation-escalation-policy-guide" rel="noopener noreferrer"&gt;on-call rotation and escalation guide&lt;/a&gt; for the people and paging side of that setup.&lt;/p&gt;

&lt;h2&gt;
  
  
  Page on harm now, ticket slow drift
&lt;/h2&gt;

&lt;p&gt;The page rule is simple: wake someone only when waiting makes the outcome worse. A tool loop can create more actions in the next minute. A spend burn rate can exhaust a budget this hour. A week-over-week latency change needs investigation, but it does not usually need an immediate phone call.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Page when&lt;/th&gt;
&lt;th&gt;Ticket when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Tool actions&lt;/td&gt;
&lt;td&gt;Action rate is far above the service's baseline, or identical calls repeat&lt;/td&gt;
&lt;td&gt;A gradual change in the action mix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spend&lt;/td&gt;
&lt;td&gt;The hourly burn rate crosses the incident budget&lt;/td&gt;
&lt;td&gt;Cost trends upward across days or releases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Writes&lt;/td&gt;
&lt;td&gt;A successful write occurs outside an approved window&lt;/td&gt;
&lt;td&gt;The normal write volume slowly grows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Access&lt;/td&gt;
&lt;td&gt;Permission-denied errors spike, which can indicate drift or a prompt-injection attempt&lt;/td&gt;
&lt;td&gt;A single expected denial or a policy cleanup&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Human approval&lt;/td&gt;
&lt;td&gt;The oldest approval has exceeded the response promise&lt;/td&gt;
&lt;td&gt;Approval volume trends higher&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Task outcome&lt;/td&gt;
&lt;td&gt;Failure rate crosses the service's error budget&lt;/td&gt;
&lt;td&gt;Evaluation score or latency gradually declines&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The following names and thresholds are &lt;strong&gt;examples owned by this runbook&lt;/strong&gt; , not Prometheus, OpenTelemetry, or vendor metric names. Pick thresholds from your own normal traffic and give every rule an owner. The counter functions use a range vector, and each alert has a &lt;code&gt;for:&lt;/code&gt; duration. Note what &lt;code&gt;for:&lt;/code&gt; does: it requires the expression to stay true across evaluations, so it filters flapping rate rules, but an event rule such as a single forbidden write stays true for as long as the event sits inside its range window. That is intended for the write and spend rules: one forbidden write is worth a page.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-production&lt;/span&gt;
    &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentActionRateHigh&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (rate(agent_tool_calls_total{outcome="success"}[5m])) &amp;gt; 20&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent action rate is above its configured baseline'&lt;/span&gt;

      &lt;span class="c1"&gt;# The runner increments this when the same tool + argument fingerprint
&lt;/span&gt;      &lt;span class="c1"&gt;# repeats inside its window. The fingerprint stays out of the label set.
&lt;/span&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentRepeatedToolCallLoop&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id, tool) (rate(agent_repeated_tool_calls_total[5m])) &amp;gt; 0.2&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Repeated tool-call rate indicates a loop'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentHourlySpendBudgetExceeded&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (increase(agent_spend_usd_total[1h])) &amp;gt; 25&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent spend exceeded the hourly incident budget'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentWriteOutsideWindow&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id, tool) (increase(agent_tool_calls_total{action="write",allowed_window="false",outcome="success"}[5m])) &amp;gt; 0&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent completed a write outside its allowed window'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentPermissionDeniedSpike&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (rate(agent_tool_calls_total{outcome="permission_denied"}[5m])) &amp;gt; 1&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent permission denials are spiking'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentApprovalQueueOld&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;max by (agent_id) (agent_approval_queue_oldest_seconds) &amp;gt; 300&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'An agent approval has waited more than five minutes'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentTaskFailureRateHigh&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (rate(agent_tasks_total{outcome="failure"}[10m])) / sum by (agent_id) (rate(agent_tasks_total[10m])) &amp;gt; 0.20&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;15m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent task failure rate is above 20 percent'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For the ticket queue, record evaluation score, latency percentiles, cost per completed task, and denied requests over a longer window. They tell you where to improve the service without training people to ignore their pager.&lt;/p&gt;

&lt;p&gt;For traces, OpenTelemetry's GenAI conventions define &lt;code&gt;invoke_agent&lt;/code&gt; and &lt;code&gt;execute_tool&lt;/code&gt; operations and tool-call attributes. They now live in their own &lt;a href="https://github.com/open-telemetry/semantic-conventions-genai" rel="noopener noreferrer"&gt;semantic-conventions-genai&lt;/a&gt; repository, and the &lt;a href="https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/" rel="noopener noreferrer"&gt;registry page&lt;/a&gt; lists those operation values at &lt;strong&gt;Development&lt;/strong&gt; stability, so version your dashboards against them. The same registry notes that &lt;code&gt;gen_ai.tool.call.arguments&lt;/code&gt; may contain sensitive information. Keep raw arguments out of metric labels: a per-call value in a label is a cardinality problem as well as a leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build a kill switch that the model cannot bypass
&lt;/h2&gt;

&lt;p&gt;The model must not decide whether it is paused. Enforce a stop before every tool call in the runner that executes the call. Then give responders independent layers in case one control is unavailable.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Tool runner flag&lt;/strong&gt; Checked before every call&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Credential revoke&lt;/strong&gt; Identity can no longer authenticate&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gateway cap&lt;/strong&gt; Rate and budget stop&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Egress deny&lt;/strong&gt; Last network containment layer&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Any on-call engineer may pull the switch, with no approval. That rule prevents a debate from extending an incident. "Pause" means calls that have not yet passed the runner's check are refused. A call that passed the check a moment before the switch can still start or finish, which is why credential revocation and egress deny exist as the next layers for immediate containment at the target. Pausing does not mean that the runner discards the current run, conversation, pending approval, or audit records. Keep that state inspectable so responders can see what the agent was about to do.&lt;/p&gt;

&lt;p&gt;This short wrapper uses Redis as the control store. The switch is the existence of a key, so there is no value format to get wrong, and an unreachable or slow store stops the call instead of allowing it. Treat the store as a security control: connect over TLS, and give the runner an ACL user that can only read the pause keys, while responders use a separate user that can set and delete them (&lt;a href="https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/" rel="noopener noreferrer"&gt;Redis ACLs&lt;/a&gt;). Put the same check in every execution path, including retries and asynchronous workers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;

&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Redis&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exceptions&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;RedisError&lt;/span&gt;

&lt;span class="c1"&gt;# rediss:// URL with the runner's read-only ACL user. Short connect and read&lt;/span&gt;
&lt;span class="c1"&gt;# timeouts: a hung control store must fail closed quickly.&lt;/span&gt;
&lt;span class="n"&gt;control&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"AGENT_CONTROL_REDIS_URL"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;socket_timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;socket_connect_timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;AgentPaused&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;call_tool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;paused&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;control&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"agent:{agent_id}:paused"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;RedisError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# A broken control plane must not permit an external action.&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AgentPaused&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"kill-switch state is unavailable"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;paused&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AgentPaused&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{agent_id} is paused; refusing {tool_name}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;call_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pulling the switch is then one command that any responder can run, as the incident-control ACL user. Deleting the key is the deliberate resume step, and it belongs in the incident record with who ran it and why:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Pause (any on-call engineer)&lt;/span&gt;
redis-cli --tls -h agent-control --user incident-control --askpass \
  SET agent:deploy-agent-prod:paused &lt;span class="s2"&gt;"paged-by-oncall $(date -u +%FT%TZ)"&lt;/span&gt;

&lt;span class="c"&gt;# Resume (service owner, after the review in step 7 of the runbook below)&lt;/span&gt;
redis-cli --tls -h agent-control --user incident-control --askpass \
  DEL agent:deploy-agent-prod:paused

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Test all four layers in a monthly game day. Start a harmless, repeatable test action, pause the agent while it is active, revoke its identity, exhaust a test gateway budget, and verify that egress policy blocks the target. Record time to containment and any action that escaped after the first stop signal. A kill switch that only exists in documentation is not a control.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scope the identity, then bound the work
&lt;/h2&gt;

&lt;p&gt;Give each agent one identity. Do not hand an agent a human's long-lived token, and do not let several agents share a broad service identity. Use a read-scoped role for discovery work, short-lived credentials, and a separate path where a human approves a write before a runner obtains a write-capable identity.&lt;/p&gt;

&lt;p&gt;For AWS, the runner can obtain a 15-minute role session with &lt;code&gt;AssumeRole&lt;/code&gt;. The &lt;a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html" rel="noopener noreferrer"&gt;AWS STS API reference&lt;/a&gt; documents a &lt;code&gt;DurationSeconds&lt;/code&gt; minimum of 900 seconds and a maximum controlled by the role's maximum session duration, which may be from one to twelve hours. The command deliberately requests the 900-second minimum.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/AgentReadInventory \
  --role-session-name agent-inventory-run-20261112 \
  --duration-seconds 900 \
  --output json

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The role policy below allows only an example inventory task. The calling workload must separately be trusted to assume this role. A write action should not add write verbs here. Route it to an approval service that records an approver, checks the change, and issues a different, time-limited identity only after approval.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"Version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2012-10-17"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"Statement"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"Sid"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ReadInventoryOnly"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"Effect"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Allow"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"Action"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ec2:DescribeInstances"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cloudwatch:GetMetricData"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"Resource"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"*"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a database, use a distinct read-only role, a connection limit, a &lt;code&gt;statement_timeout&lt;/code&gt;, and a gateway rate limit. Our &lt;a href="https://dev.to/devopsdaily/we-gave-an-ai-agent-a-readonly-neon-role-here-is-what-it-blocked-o07"&gt;read-only role probes&lt;/a&gt; showed why the role alone is not enough: a 15-second agent read on the primary made an &lt;code&gt;ALTER TABLE&lt;/code&gt; wait more than 11 seconds, and the app's inserts queued behind it, even though the agent never wrote anything. Sending agent reads to a read replica kept the primary clear in that test. The same probes found that nothing in the role bounds how often the agent repeats a legal read, and that the agent's session could run &lt;code&gt;SET statement_timeout = 0&lt;/code&gt;. So enforce the timeout and the rate cap in the tool runner or gateway, where the agent cannot change them.&lt;/p&gt;

&lt;p&gt;This is the practical answer to Excessive Agency, which the &lt;a href="https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/" rel="noopener noreferrer"&gt;2026 OWASP Top 10 for LLM Applications&lt;/a&gt; ranks third (&lt;a href="https://genai.owasp.org/2026/09/01/owasp-genai-security-project-unveils-2026-top-10-for-llm-applications-new-agent-control-standard-and-sponsors-as-community-tops-30000-members/" rel="noopener noreferrer"&gt;announcement&lt;/a&gt;). A model can be wrong or manipulated. The identity and policy boundary should still limit what reaches a real system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep an audit trail outside the agent's reach
&lt;/h2&gt;

&lt;p&gt;Log every attempted tool call as a structured event. Include the run ID, agent identity, credential or role, tool, redacted arguments, result, approver when one exists, timestamp, and trace ID. Store it in append-only storage with a retention period set by your security and compliance requirements. The agent's own workspace, database, or object-store prefix is not an acceptable sole copy because an agent with write access could alter the evidence.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"timestamp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-12T02:18:44Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"run_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"run_01JCG7"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"agent_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"deploy-agent-prod"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"credential"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:sts::123456789012:assumed-role/AgentReadInventory/agent-inventory-run-20261112"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"tool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ec2.describe_instances"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"eu-west-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"filters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"[REDACTED]"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"result"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"success"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"approver"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"trace_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"4bf92f3577b34da6a3ce929d0e0e4736"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To answer "what did the agent do between 02:10 and 02:40?", query the protected JSONL archive, not the model transcript:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;jq -r '
  select(
    .agent_id == "deploy-agent-prod" and
    .timestamp &amp;gt;= "2026-11-12T02:10:00Z" and
    .timestamp &amp;lt; "2026-11-12T02:40:00Z"
  ) |
  [.timestamp, .run_id, .tool, .result, .trace_id] | @tsv
' agent-audit.jsonl

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Redact secrets before an event leaves the runner. In our &lt;a href="https://dev.to/devopsdaily/we-hid-96-instructions-in-the-logs-an-ops-agent-reads-here-is-what-stopped-them-3fec"&gt;prompt-injection test of an ops agent&lt;/a&gt;, a check outside the model stopped every forbidden action, but the agent still copied secrets into its own reports, so redaction belongs in code, not in the prompt. Store a stable argument fingerprint for loop detection, but do not put raw prompts, credentials, tokens, or user data into an alert label. Use the trace ID to join the audit event to the request and tool spans.&lt;/p&gt;

&lt;h2&gt;
  
  
  First 15 minutes: copy this runbook
&lt;/h2&gt;

&lt;p&gt;Put the following template beside the service catalog entry. Fill in the actual dashboard, credential-revocation command, audit location, and escalation targets before the agent goes live.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AI AGENT INCIDENT: FIRST 15 MINUTES

Service: &amp;lt;agent service name&amp;gt;
Primary on-call: &amp;lt;rotation&amp;gt;
Escalation: &amp;lt;service owner&amp;gt;, &amp;lt;security contact&amp;gt;, &amp;lt;target-system owner&amp;gt;

1. Acknowledge the page and start an incident record. Record the run ID, alert,
   responder, and current time.
2. Pull the kill switch. Confirm the tool runner refuses a test call and note the
   last tool action accepted after the pause time.
3. Revoke or disable the agent credential. Apply the gateway rate or budget cap.
   Use egress deny if actions may still reach an external target.
4. Preserve the agent run state, approval queue, traces, and append-only audit
   events. Do not delete the run or ask the agent to explain itself.
5. Assess blast radius from the audit trail: actions, targets, credential used,
   successful writes, denied calls, spend, and affected data or services.
6. Communicate status, containment time, known impact, and next update time to
   the owning team and target-system owner. Escalate security when appropriate.
7. Resume only after the owner identifies the trigger, corrects the control gap,
   tests it in a safe environment, and explicitly records the decision.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The best time to settle ownership and containment authority is before the first tool call. Once the agent is a cataloged service with a real pager, a tested stop path, scoped credentials, and a protected action record, responders can handle its failures as operational work instead of improvising around a model.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/who-is-on-call-for-the-ai-agent" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>aiagents</category>
      <category>oncall</category>
      <category>incidentresponse</category>
    </item>
    <item>
      <title>We Tested AWS Emulator: It Blocks More Than Its Docs Say, and Less Than AWS Does</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Thu, 12 Nov 2026 09:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/we-tested-aws-emulator-it-blocks-more-than-its-docs-say-and-less-than-aws-does-bg6</link>
      <guid>https://dev.to/devopsdaily/we-tested-aws-emulator-it-blocks-more-than-its-docs-say-and-less-than-aws-does-bg6</guid>
      <description>&lt;p&gt;AWS released AWS Emulator on 2026-11-11: a free local copy of the AWS APIs that runs on your laptop or in CI, with no AWS account. We installed v1.0.322 on a throwaway droplet and ran it through 40 workflow checks across 9 services, 32 permission cases, the AWS CLI, the JavaScript SDK and Terraform. &lt;strong&gt;All 40 workflow checks passed, and the emulator refused two calls our own test code made with roles that had no permissions, as AWS would.&lt;/strong&gt; That is more permission checking than AWS's own security page describes. It is also less than AWS does: a user with no policies at all assumed an admin role and created an IAM user with it, and permissions boundaries and session policies changed nothing.&lt;/p&gt;

&lt;p&gt;Three other things would bite a real test suite: the JavaScript SDK's SQS client sent a request to the real &lt;code&gt;sqs.us-east-1.amazonaws.com&lt;/code&gt; by default, Terraform AWS provider 6.73.0 could not read back an S3 bucket, and the emulator refuses to hold more than 64 MiB in one bucket or 1,000 messages in one queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Install is one signed &lt;code&gt;.deb&lt;/code&gt; (163.0 MiB) that unpacks to one 925.7 MiB Rust binary. It answered its health check in a median 383.5 ms on repeated starts and 1,325 ms with the page cache dropped, using 172.5 MiB of anonymous memory at idle.&lt;/li&gt;
&lt;li&gt;Our scenario suite passed 40 of 40 checks: S3 presigned URLs, multipart and versioning; SQS visibility timeout, DLQ and redrive; DynamoDB conditional writes, transactions and streams; Lambda; EventBridge to SQS; SNS fan-out with a filter policy; KMS; Secrets Manager; Step Functions.&lt;/li&gt;
&lt;li&gt;The emulator enforces identity policies, bucket policies, KMS key policies and secret policies for the users and roles you create in it. That caught a missing &lt;code&gt;s3:PutObject&lt;/code&gt; and a missing &lt;code&gt;lambda:InvokeFunction&lt;/code&gt; in our own test code.&lt;/li&gt;
&lt;li&gt;It matched AWS's documented result in 27 of 32 permission cases. All 5 misses let through a request that AWS refuses, and they come from 4 causes: permissions boundaries, session policies, the caller's own &lt;code&gt;sts:AssumeRole&lt;/code&gt; check (a user with no policies assumed an admin role and used it), and &lt;code&gt;ListBuckets&lt;/code&gt; with no policy.&lt;/li&gt;
&lt;li&gt;The JS SDK v3 SQS client follows the AWS-style queue URL the emulator returns and calls real AWS. Set &lt;code&gt;useQueueUrlAsEndpoint: false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Terraform AWS provider 6.73.0 fails on &lt;code&gt;aws_s3_bucket&lt;/code&gt; because it reads tags through S3 Control. Provider 5.100.0 applied and destroyed the same 25-resource stack, with one plan diff left.&lt;/li&gt;
&lt;li&gt;The repo has 29,684 commits, all by agent identities, and about 16.1 million lines of Rust. AWS's own fidelity scores for this release pass 56.5% of 42,937 scored checks.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux x86_64 machine you can throw away. We used a 4 vCPU / 8 GB DigitalOcean droplet with Ubuntu 24.04.&lt;/li&gt;
&lt;li&gt;AWS CLI v2 and an SDK (we used boto3 1.44.1 and the JavaScript SDK v3).&lt;/li&gt;
&lt;li&gt;Terraform if you want to test infrastructure code.&lt;/li&gt;
&lt;li&gt;No AWS account, and no real AWS credentials anywhere on the machine. The emulator writes its own keys per run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every script and raw result is in the companion repo:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/The-DevOps-Daily/aws-emulator-field-test" rel="noopener noreferrer"&gt;The-DevOps-Daily/aws-emulator-field-test on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What AWS Emulator is, from the primary sources
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://aboutamazon.com/news/aws/how-autonomous-ai-agents-built-aws-emulator" rel="noopener noreferrer"&gt;launch post&lt;/a&gt; gives 438 AWS API models. Werner Vogels' &lt;a href="https://www.allthingsdistributed.com/2026/11/aws-emulator-and-the-engineers-job-when-agents-write-the-code.html" rel="noopener noreferrer"&gt;post&lt;/a&gt; says "more than 400 API models covering more than 200 AWS services" and that "all of AWS Emulator's code was written and tested by agents." The method in both posts: agents write fidelity tests from the public Smithy models, fix each test until it passes against real AWS, then build the emulator until it passes the same test. Every change goes through a merge gate that includes several security agents doing adversarial reviews.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://docs.aws.amazon.com/emulator/latest/userguide/what-is-service.html" rel="noopener noreferrer"&gt;user guide&lt;/a&gt; adds the parts that matter in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It is for building and testing, not production. It "doesn't evaluate AWS Identity and Access Management (IAM) policies and resource-based policies the way AWS does."&lt;/li&gt;
&lt;li&gt;Behaviour is checked against AWS in &lt;code&gt;us-west-2&lt;/code&gt; only, and global services in &lt;code&gt;us-east-1&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;One account (&lt;code&gt;000000000000&lt;/code&gt;) and, in practice, one Region per emulator.&lt;/li&gt;
&lt;li&gt;It serves HTTPS on &lt;code&gt;127.0.0.1:9297&lt;/code&gt; with a new self-signed certificate and new access keys on every start.&lt;/li&gt;
&lt;li&gt;Telemetry is on by default. The &lt;a href="https://github.com/aws/aws-emulator" rel="noopener noreferrer"&gt;README&lt;/a&gt; explains how to turn it off. We ran every test with &lt;code&gt;--telemetry off&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The code is CC0, except the &lt;code&gt;vendor/&lt;/code&gt; directory, which is Apache 2.0. The repository does not take pull requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Install and start
&lt;/h2&gt;

&lt;p&gt;On Linux it is a native package. The Debian package was 163.0 MiB and passed the GPG check from the install guide (&lt;code&gt;Good signature from "SSM Publishing &amp;lt;ssm-publishing@amazon.com&amp;gt;"&lt;/code&gt;). It installs two files: a license file and one stripped Rust binary of 970,692,256 bytes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -fsSLo aws-emulator.deb \
  https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
sudo apt-get install -y ./aws-emulator.deb
aws-emulator --telemetry off

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We timed 10 repeated starts from &lt;code&gt;exec&lt;/code&gt; to the first HTTP 200 on &lt;code&gt;/__admin/health&lt;/code&gt;, each with a fresh home directory, then 5 more with the page cache dropped so the binary had to come off disk.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Measure&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;.deb&lt;/code&gt; download&lt;/td&gt;
&lt;td&gt;163.0 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Installed binary&lt;/td&gt;
&lt;td&gt;925.7 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start to healthy, repeated starts (10 runs)&lt;/td&gt;
&lt;td&gt;338 to 470 ms, median 383.5 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start to healthy, cold page cache (5 runs)&lt;/td&gt;
&lt;td&gt;1,260 to 1,399 ms, median 1,325 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RSS at ready&lt;/td&gt;
&lt;td&gt;about 282 MiB, flat over 60 s idle&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anonymous memory at ready&lt;/td&gt;
&lt;td&gt;172.5 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The guide says the emulator "uses about 160 MiB when idle." Plain RSS reads higher because it counts pages of the memory-mapped binary. The anonymous part, 172.5 MiB, is close to the documented figure.&lt;/p&gt;

&lt;p&gt;A clean &lt;code&gt;SIGTERM&lt;/code&gt; deleted the credentials, config and certificate files from &lt;code&gt;~/.aws/emulator/9297/&lt;/code&gt;, as the guide says, and left the empty directory.&lt;/p&gt;

&lt;h2&gt;
  
  
  The AWS CLI session
&lt;/h2&gt;

&lt;p&gt;The setup is eight environment variables from the &lt;a href="https://docs.aws.amazon.com/emulator/latest/userguide/point-sdk-to-emulator.html" rel="noopener noreferrer"&gt;SDK guide&lt;/a&gt;: the endpoint, the per-run credentials and config files, the CA bundle, two Region variables, and two switches that stop the SDK from asking the instance metadata service for credentials or adding prefixes to host names. Here is part of our recorded CLI session against a fresh emulator:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;aws cli against AWS Emulator v1.0.322&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ aws sts get-caller-identity --output text
000000000000    arn:aws:iam::000000000000:user/emulator AIDAH5I7GEMO7ADXNFAZU
$ aws s3 mb s3://cli-demo
make_bucket: cli-demo
$ aws s3 cp /tmp/notes.txt s3://cli-demo/notes.txt
upload: ../tmp/notes.txt to s3://cli-demo/notes.txt
$ curl --cacert &lt;span class="nv"&gt;$AWS_CA_BUNDLE&lt;/span&gt; &lt;span class="s2"&gt;"$(aws s3 presign s3://cli-demo/notes.txt --expires-in 60)"&lt;/span&gt;
release notes
$ aws sqs create-queue --queue-name cli-demo --query QueueUrl --output text
https://sqs.us-east-1.amazonaws.com/000000000000/cli-demo
&lt;span class="c"&gt;# the same validation error as AWS&lt;/span&gt;
$ aws dynamodb create-table --table-name ab ...
aws: &lt;span class="o"&gt;[&lt;/span&gt;ERROR&lt;span class="o"&gt;]&lt;/span&gt;: An error occurred &lt;span class="o"&gt;(&lt;/span&gt;ValidationException&lt;span class="o"&gt;)&lt;/span&gt; when calling the CreateTable operation: 1 validation error detected: Value &lt;span class="s1"&gt;'ab'&lt;/span&gt; at &lt;span class="s1"&gt;'tableName'&lt;/span&gt; failed to satisfy constraint: Member must have length greater than or equal to 3
&lt;span class="c"&gt;# S3 Control puts the account ID in the host name, so it never arrives&lt;/span&gt;
$ aws s3control list-access-points --account-id 000000000000
aws: &lt;span class="o"&gt;[&lt;/span&gt;ERROR&lt;span class="o"&gt;]&lt;/span&gt;: Could not connect to the endpoint URL: &lt;span class="s2"&gt;"https://000000000000.127.0.0.1:9297/v20180820/accesspoint"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the queue URL. It looks exactly like a real one. That detail comes back later.&lt;/p&gt;

&lt;h2&gt;
  
  
  40 workflow checks, 40 passes
&lt;/h2&gt;

&lt;p&gt;We did not want to test single API calls. We wrote 9 workflows of the kind that integration tests use, with 40 checks between them. Each check records the exact error text when it fails. Many checks test the error path, because a fake that accepts everything passes the happy path too.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workflow&lt;/th&gt;
&lt;th&gt;Checks&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;th&gt;What it covered&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S3&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;9/9&lt;/td&gt;
&lt;td&gt;presigned GET and PUT, an expired URL refused with 403, 3-part multipart, completing with a 1 MiB first part refused with &lt;code&gt;EntityTooSmall&lt;/code&gt;, versioning, delete markers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SQS&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;5/5&lt;/td&gt;
&lt;td&gt;visibility timeout, the third receive moving a message to the DLQ, &lt;code&gt;StartMessageMoveTask&lt;/code&gt; back, FIFO deduplication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DynamoDB&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6/6&lt;/td&gt;
&lt;td&gt;insert-if-absent, a stale optimistic-lock update, a cancelled transaction with per-item reasons, streams&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4/4&lt;/td&gt;
&lt;td&gt;create and invoke a python3.12 zip, handler exception as &lt;code&gt;FunctionError&lt;/code&gt;, boto3 called from inside the function&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;EventBridge&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2/2&lt;/td&gt;
&lt;td&gt;a numeric pattern (&lt;code&gt;total &amp;gt; 100&lt;/code&gt;) delivering only the matching event to SQS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SNS&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2/2&lt;/td&gt;
&lt;td&gt;fan-out to two queues, one with raw delivery and one with a filter policy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KMS&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6/6&lt;/td&gt;
&lt;td&gt;round trip by alias, wrong encryption context, tampered ciphertext, data keys, a disabled key&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets Manager&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3/3&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;AWSCURRENT&lt;/code&gt; and &lt;code&gt;AWSPREVIOUS&lt;/code&gt;, scheduled deletion blocking reads, restore&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Step Functions&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3/3&lt;/td&gt;
&lt;td&gt;Choice, Wait, a Lambda task, a Fail state&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The whole suite ran in 38.0 s against a fresh emulator. The emulator answered small calls fast: under load, a 100 KiB &lt;code&gt;PutObject&lt;/code&gt; took a median 14.66 ms and a 25-item &lt;code&gt;BatchWriteItem&lt;/code&gt; 13.64 ms.&lt;/p&gt;

&lt;p&gt;Getting to 40 of 40 took four runs. Every failure in between was a mistake in our test, and the repo keeps each of those runs. That story is the most useful part of this test.&lt;/p&gt;

&lt;h3&gt;
  
  
  The emulator caught our missing permissions
&lt;/h3&gt;

&lt;p&gt;On the third run, the Lambda function's call to S3 failed. We reproduced that run on purpose as a named variant (&lt;code&gt;FT_NO_ROLE_POLICIES=1&lt;/code&gt;), and this is its recorded error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AccessDenied: User: arn:aws:sts::000000000000:assumed-role/ft-fn-8820f7df/ft-fn-8820f7df is not authorized
to perform: s3:PutObject on resource: "arn:aws:s3:::ft-lambda-8820f7df/from-lambda.txt" because no
identity-based policy allows the s3:PutObject action

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our test created the Lambda execution role with a trust policy and nothing else. On AWS that function fails the same way. The Step Functions test failed for the same reason: &lt;code&gt;ft-sfn-8820f7df is not authorized to perform: lambda:InvokeFunction&lt;/code&gt;. We added the two missing policies and both checks passed. That variant is the best argument for the emulator we found.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Lambda sandbox
&lt;/h3&gt;

&lt;p&gt;The first two runs failed for a different reason: our handler probed the machine it ran on. The emulator refused every probe that reached for the host:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Probe from inside the function&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sys.version&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;3.14.0.alpha&lt;/code&gt;, for a &lt;code&gt;python3.12&lt;/code&gt; function&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;open("/etc/hostname")&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OSError: a Lambda function in this emulator cannot open a file of the host&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;os.getuid()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OSError: a Lambda function in this emulator has no host filesystem or process to reach&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;socket.gethostname()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OSError: a Lambda function in this emulator has no host network to reach&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;import subprocess&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ModuleNotFoundError: No module named '_signal'&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That is good isolation for zip-packaged Python, and better than the security page suggests ("it doesn't isolate that code from your machine"). We only tested Python zip functions, so do not assume the same for other runtimes. Two practical effects: a handler that imports &lt;code&gt;subprocess&lt;/code&gt; at module level fails to load with &lt;code&gt;Runtime.ImportModuleError&lt;/code&gt;, and your code runs on a different Python than the one you deploy to. Four invokes in a row took 369, 335, 354 and 355 ms, with no consistent warm-start speedup.&lt;/p&gt;

&lt;h2&gt;
  
  
  The permission test: 27 of 32
&lt;/h2&gt;

&lt;p&gt;The security page is direct about permissions:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Some services in the emulator evaluate IAM policies for IAM users and roles that you create in the emulator, but not the same way as AWS. The emulator never denies a request signed with its own key. It doesn't check AWS KMS key policies or Amazon S3 bucket policies.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The Lambda result suggested more checking than that, so we wrote 32 cases. For each one we wrote down what AWS documents. &lt;strong&gt;These cases were not run on real AWS&lt;/strong&gt; : the expected column comes from the AWS documentation for policy evaluation, bucket policies, KMS key policies, role trust policies and SQS queue policies. Then we ran them against the emulator, and against Moto and Floci for context.&lt;/p&gt;

&lt;p&gt;Bold marks a result that differs from what AWS documents.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;AWS documents&lt;/th&gt;
&lt;th&gt;AWS Emulator&lt;/th&gt;
&lt;th&gt;Floci&lt;/th&gt;
&lt;th&gt;Moto&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: sts:GetCallerIdentity&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:ListAllMyBuckets&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: sqs:SendMessage&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: dynamodb:PutItem&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: kms:Encrypt&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: secretsmanager:GetSecretValue&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: sns:Publish&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: events:PutEvents&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: iam:CreateUser&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: admin user: s3:PutObject (baseline)&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow * plus explicit Deny s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow * but permissions boundary allows only reads: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow PutObject only with SSE-KMS: put without SSE&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow PutObject only with SSE-KMS: put with SSE-KMS&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: bucket policy denies this admin user: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: bucket policy denies everyone over HTTPS: emulator's own key PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: KMS key policy denies this admin user: kms:Encrypt&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: secret resource policy denies this admin user: GetSecretValue&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: no identity policy, bucket policy allows this user: s3:PutObject&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: PutObject allowed on bucket A only: put to bucket A&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: PutObject allowed on bucket A only: put to bucket B&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow NotAction s3:*: sqs:SendMessage&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow NotAction s3:*: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:ListBucket (ListObjectsV2)&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:GetObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: admin user assumes a role that trusts only lambda.amazonaws.com&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: no-policy user assumes a role that trusts the account (needs sts:AssumeRole itself)&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: no-policy user: assume the account-trusted admin role, then s3:PutObject + iam:CreateUser&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: admin role assumed with a GetObject-only session policy: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;service-to-service: EventBridge rule -&amp;gt; SQS queue with no queue policy&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;service-to-service: SNS topic -&amp;gt; SQS queue with no queue policy&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On v1.0.322, the "never denies its own key" sentence is out of date. A bucket policy that denies &lt;code&gt;s3:PutObject&lt;/code&gt; to everyone over HTTPS stopped the emulator's own key. A KMS key policy deny, a secret resource policy deny and a bucket policy deny all stopped an admin user. Explicit denies, resource scoping, &lt;code&gt;NotAction&lt;/code&gt; and condition keys such as &lt;code&gt;s3:x-amz-server-side-encryption&lt;/code&gt; all worked. EventBridge and SNS did not deliver to a queue without a queue policy, which is what AWS does.&lt;/p&gt;

&lt;p&gt;The 5 misses come from 4 causes, and they have one thing in common. Each is a control that &lt;strong&gt;narrows&lt;/strong&gt; access, and the emulator skipped it and let the request through:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Permissions boundaries.&lt;/strong&gt; A user with &lt;code&gt;Allow *&lt;/code&gt; and a read-only boundary could still &lt;code&gt;PutObject&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Session policies.&lt;/strong&gt; An admin role assumed with a &lt;code&gt;GetObject&lt;/code&gt;-only session policy could still &lt;code&gt;PutObject&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The caller's own &lt;code&gt;sts:AssumeRole&lt;/code&gt; permission.&lt;/strong&gt; A role that trusts the account (&lt;code&gt;arn:aws:iam::000000000000:root&lt;/code&gt;) means "any principal in this account that IAM allows to assume it." Our user had no policies at all, so AWS refuses. The emulator returned credentials, and with them the user wrote to S3 and created a new IAM user.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ListBuckets&lt;/code&gt; with no policy&lt;/strong&gt; returned the bucket list.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The repo's &lt;code&gt;issues/things-that-do-not-work/&lt;/code&gt; folder has open files about session-policy scoping and permissions boundaries, so the agents know about at least some of this. We did not match each of our 4 causes to one issue.&lt;/p&gt;

&lt;p&gt;One fix to our own harness changed the comparison numbers. Moto refuses a bucket-policy deny with a bare &lt;code&gt;403: Forbidden&lt;/code&gt; where AWS returns &lt;code&gt;AccessDenied&lt;/code&gt;, and our first classifier counted that as an error, not a refusal. We now count every HTTP 403 as a refusal, which adds those two cases to Moto's total (8 of 32). The saved results still show the bare &lt;code&gt;403&lt;/code&gt; code. The fix did not change the AWS Emulator or Floci results.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why this matters for your tests
&lt;/h3&gt;

&lt;p&gt;Boundaries and session policies are the controls teams use for least privilege: a CI role that may create roles only inside a boundary, a broker that hands out narrowed credentials, a platform team that delegates IAM. Locally, both directions go wrong. A job whose boundary blocks a write it needs passes against the emulator and fails on AWS. A test that asserts "the boundary stops this write" fails against the emulator even when the boundary on AWS is correct.&lt;/p&gt;

&lt;p&gt;What to do about it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Create the real roles in the emulator.&lt;/strong&gt; Do not run everything with the emulator's admin key. Create the roles and policies you deploy, and run each component with its own role. That is how the emulator caught our two missing permissions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep negative permission tests for real AWS.&lt;/strong&gt; Run a small set against a sandbox account in CI, with short-lived credentials. Boundaries, session policies and role trust belong there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check policies without calling the emulator.&lt;/strong&gt; IAM Access Analyzer policy validation and custom policy checks catch many mistakes before anything runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat a local &lt;code&gt;AccessDenied&lt;/code&gt; as a bug, and a local success as unproven.&lt;/strong&gt; The emulator's denies matched AWS in every case we tried. Its allows did not.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Three more ways a green local run can lie
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The JavaScript SDK sent SQS calls to real AWS
&lt;/h3&gt;

&lt;p&gt;Our JS SDK v3 smoke test passed for S3 and DynamoDB. SQS failed with &lt;code&gt;InvalidClientTokenId: The security token included in the request is invalid.&lt;/code&gt; The emulator would not say that about its own key. The SDK printed the reason:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;QueueUrl=https://sqs.us-east-1.amazonaws.com/000000000000/js-demo differs from SQSClient resolved
endpoint=https://127.0.0.1:9297/, using QueueUrl host as endpoint.
Set [endpoint=string] or [useQueueUrlAsEndpoint=false] on the SQSClient.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The emulator returns queue URLs in the AWS format. &lt;code&gt;@aws-sdk/client-sqs&lt;/code&gt; 3.1167.0 uses the queue URL's host by default, so the request went to &lt;code&gt;sqs.us-east-1.amazonaws.com&lt;/code&gt;, and AWS rejected the emulator's key. To prove it, we pointed that host name at &lt;code&gt;0.0.0.0&lt;/code&gt; in &lt;code&gt;/etc/hosts&lt;/code&gt;. The error changed to &lt;code&gt;connect ECONNREFUSED 0.0.0.0:443&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Keep SQS calls on the emulator&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sqs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;SQSClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;useQueueUrlAsEndpoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With that option, the same test passed. The SDK guide lists host names that SDKs change (host prefixes, S3, S3 Control) but not this case. Our machine had no real credentials, so AWS rejected the emulator's key and nothing else happened. But environment variables take precedence over the emulator's credentials file. In a CI job with real &lt;code&gt;AWS_ACCESS_KEY_ID&lt;/code&gt; and &lt;code&gt;AWS_SECRET_ACCESS_KEY&lt;/code&gt; set, the same test would have sent a request to AWS signed with real keys. The guide's advice to unset real credentials in the shell that runs your tests is not optional.&lt;/p&gt;

&lt;h3&gt;
  
  
  Terraform: provider 6.73.0 could not read back an S3 bucket
&lt;/h3&gt;

&lt;p&gt;We applied a small stack with 25 resources: a versioned bucket with a lifecycle rule, queues with a DLQ, a DynamoDB table with a stream, a GSI and TTL, SNS, KMS, Secrets Manager, SSM, IAM roles, a Lambda function with an SQS event source, an EventBridge rule and a Step Functions state machine.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;AWS provider 6.73.0&lt;/th&gt;
&lt;th&gt;AWS provider 5.100.0&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;terraform apply&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;failed, 23 entries in state&lt;/td&gt;
&lt;td&gt;passed, 26 entries in state&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Error&lt;/td&gt;
&lt;td&gt;&lt;code&gt;S3 Control: ListTagsForResource ... lookup 000000000000.127.0.0.1: no such host&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Second &lt;code&gt;plan&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;failed on the same S3 call&lt;/td&gt;
&lt;td&gt;1 change: DynamoDB &lt;code&gt;on_demand_throughput&lt;/code&gt; (-1 to null)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;terraform destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;failed on the same S3 call&lt;/td&gt;
&lt;td&gt;passed in 39.3 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In our test, provider 6.73.0 read bucket tags through S3 Control, and S3 Control puts the account ID in the host name. The SDK guide says S3 Control requests cannot reach the emulator. We tried a hosts-file entry for &lt;code&gt;000000000000.127.0.0.1&lt;/code&gt;, but &lt;code&gt;getent&lt;/code&gt; did not resolve it. When we forced the address with &lt;code&gt;curl --resolve&lt;/code&gt;, the TLS handshake failed (curl exit code 35) even with certificate checks off. In this test, provider 5.100.0 worked where 6.73.0 failed, with one plan diff left. Until that changes, stacks with S3 buckets need an older provider or a separate test for the bucket.&lt;/p&gt;

&lt;p&gt;Apply took about 85 s with both providers. The three SQS resources took 25 s each. In our load test the emulator answered SQS batch sends in a median 13.08 ms, so slow SQS responses do not explain those 25 s. We did not trace what the provider waits for.&lt;/p&gt;

&lt;h3&gt;
  
  
  Caps that AWS does not have
&lt;/h3&gt;

&lt;p&gt;Our first memory test tried to put 1,000 objects of 100 KiB in one bucket and 10,000 messages in one queue. Both stopped early, so we wrote a probe that fills one bucket and one queue until the emulator refuses. The bucket took 654 objects and the queue took 1,000 messages:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ValidationException: storing "obj-00654" would leave the bucket "cap-probe" holding 67131605 bytes of
objects, past the 67108864 bytes this emulator keeps per bucket

OverLimit: This emulator holds at most 1000 messages per queue; queue cap-probe cannot take 1 more.
Delete messages from it first.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is 64 MiB per bucket and 1,000 messages per queue. Real S3 buckets and SQS queues have no such limits on stored data or backlog. The errors are clear, and the source has more caps of the same kind, with messages such as "this emulator holds at most {} files per repository tree" and "this emulator holds tables for at most {}". A test that uploads a large fixture or builds a queue backlog has to stay under them.&lt;/p&gt;

&lt;p&gt;With 58.6 MiB in S3, 20,000 DynamoDB items and 900 queued messages, anonymous memory grew from 172.5 MiB to 277.2 MiB. Deleting everything and waiting 15 s did not give memory back (278.8 MiB). Per the docs, restart the emulator between suites, which also gives you clean state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AWS Emulator memory while a test suite fills it&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;idle&lt;/th&gt;
&lt;th&gt;+58.6 MiB in S3&lt;/th&gt;
&lt;th&gt;+20,000 DynamoDB items&lt;/th&gt;
&lt;th&gt;+900 SQS messages&lt;/th&gt;
&lt;th&gt;all deleted, +15 s&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;RSS&lt;/td&gt;
&lt;td&gt;281.7 MiB&lt;/td&gt;
&lt;td&gt;367.2 MiB&lt;/td&gt;
&lt;td&gt;411.5 MiB&lt;/td&gt;
&lt;td&gt;415.4 MiB&lt;/td&gt;
&lt;td&gt;418.9 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anonymous memory&lt;/td&gt;
&lt;td&gt;172.5 MiB&lt;/td&gt;
&lt;td&gt;236.6 MiB&lt;/td&gt;
&lt;td&gt;276.4 MiB&lt;/td&gt;
&lt;td&gt;277.2 MiB&lt;/td&gt;
&lt;td&gt;278.8 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;v1.0.322, one fresh process, 4 vCPU / 8 GB droplet. Source: results/loaded-memory.json in the companion repo.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How it compares to Moto and Floci
&lt;/h2&gt;

&lt;p&gt;LocalStack was the usual choice for this. Since March 2026, &lt;code&gt;localstack/localstack&lt;/code&gt; is a single image that needs an auth token, and the free plan is meant for non-commercial use (&lt;a href="https://blog.localstack.cloud/the-road-ahead-for-localstack/" rel="noopener noreferrer"&gt;LocalStack's announcement&lt;/a&gt;). We did not test it, because that needs an account. Moto (Apache 2.0) and Floci (MIT) run with no account, so we ran the same two suites against both, &lt;strong&gt;in their default configurations&lt;/strong&gt;. Both have opt-in authorization checks that we did not turn on, so their permission rows show the defaults, not their best case. Start times are not measured the same way: AWS Emulator is timed to its health check, Moto to its first HTTP response, and Floci from &lt;code&gt;docker run&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;AWS Emulator 1.0.322&lt;/th&gt;
&lt;th&gt;Floci 2.4.0&lt;/th&gt;
&lt;th&gt;Moto 5.2.5 (server mode)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;License&lt;/td&gt;
&lt;td&gt;CC0 (&lt;code&gt;vendor/&lt;/code&gt; is Apache 2.0)&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;Apache 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Account or token needed&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Install&lt;/td&gt;
&lt;td&gt;163.0 MiB &lt;code&gt;.deb&lt;/code&gt;, 925.7 MiB binary&lt;/td&gt;
&lt;td&gt;90.2 MiB image (&lt;code&gt;docker image inspect&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Python package&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start to first response&lt;/td&gt;
&lt;td&gt;median 383.5 ms (10 runs)&lt;/td&gt;
&lt;td&gt;992 ms including &lt;code&gt;docker run&lt;/code&gt; (1 run)&lt;/td&gt;
&lt;td&gt;1,174 ms (1 run)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Idle memory&lt;/td&gt;
&lt;td&gt;172.5 MiB anonymous, about 282 MiB RSS&lt;/td&gt;
&lt;td&gt;25.4 MiB (&lt;code&gt;docker stats&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;51.1 MiB RSS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Workflow checks passed&lt;/td&gt;
&lt;td&gt;40 of 40&lt;/td&gt;
&lt;td&gt;38 of 40&lt;/td&gt;
&lt;td&gt;32 of 40 (1 miss is our harness)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Permission cases matching AWS docs&lt;/td&gt;
&lt;td&gt;27 of 32&lt;/td&gt;
&lt;td&gt;6 of 32&lt;/td&gt;
&lt;td&gt;8 of 32&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Share of checks that matched AWS, default configurations&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AWS Emulator, workflows&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;Workflow checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Floci, workflows&lt;/td&gt;
&lt;td&gt;95%&lt;/td&gt;
&lt;td&gt;Workflow checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moto, workflows&lt;/td&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;td&gt;Workflow checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS Emulator, permissions&lt;/td&gt;
&lt;td&gt;84.4%&lt;/td&gt;
&lt;td&gt;Permission cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moto, permissions&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;td&gt;Permission cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Floci, permissions&lt;/td&gt;
&lt;td&gt;18.8%&lt;/td&gt;
&lt;td&gt;Permission cases&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Workflow checks: 40 per target; one of Moto's misses is a networking problem in our harness. Permission cases: 32 per target, expected results from AWS documentation, not run on real AWS. Source: results/&lt;/em&gt;-scenarios.json and results/&lt;em&gt;-iam-gap.json in the companion repo.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Some notes so the table is fair:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Moto's Lambda-to-S3 check failed because of our setup: Moto listened on &lt;code&gt;127.0.0.1&lt;/code&gt;, and its Lambda container could not reach &lt;code&gt;host.docker.internal:5000&lt;/code&gt;. We count it as a fail, but it is a network problem in our harness, not proof that Moto cannot do it.&lt;/li&gt;
&lt;li&gt;Moto's other misses: an expired presigned URL still worked, a presigned PUT stored an empty body, &lt;code&gt;StartMessageMoveTask&lt;/code&gt; returned HTTP 500, a handler exception came back as &lt;code&gt;Handled&lt;/code&gt; instead of &lt;code&gt;Unhandled&lt;/code&gt;, a disabled KMS key still encrypted, and both Step Functions executions were still &lt;code&gt;RUNNING&lt;/code&gt; after 30 s.&lt;/li&gt;
&lt;li&gt;Floci's misses: an expired presigned URL still worked, and the EventBridge rule with a numeric pattern delivered nothing.&lt;/li&gt;
&lt;li&gt;In the permission matrix, both alternatives allowed every request in their default modes, with one exception: Moto refused both bucket-policy denies.&lt;/li&gt;
&lt;li&gt;On size and memory, the others are much smaller. Floci's image is 90.2 MiB and used 25.4 MiB idle. Moto used 51.1 MiB idle. AWS Emulator is a 925.7 MiB binary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On these suites, AWS Emulator matched AWS more often than either alternative, and by a wide margin on permissions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the repo shows about "built by agents"
&lt;/h2&gt;

&lt;p&gt;We cloned &lt;a href="https://github.com/aws/aws-emulator" rel="noopener noreferrer"&gt;aws/aws-emulator&lt;/a&gt; and the &lt;a href="https://github.com/aws/aws-emulator-evaluation-suite" rel="noopener noreferrer"&gt;evaluation suite&lt;/a&gt; and counted.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Goal: Each release publishes its own pass rate: 56.5% for v1.0.322&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fidelity tests&lt;/strong&gt; 29,676 boto3 checks&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IssueFinderAgent&lt;/strong&gt; 16,664 commits&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;issues/ folder&lt;/strong&gt; 6,544 open, 12,708 fixed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IssueFixerAgent&lt;/strong&gt; 12,981 commits&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Release&lt;/strong&gt; with scores.json&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;em&gt;every change passes a merge gate (not visible in the repo): failing fidelity tests become new issues, then back to step 1.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every commit is by an agent identity.&lt;/strong&gt; 29,684 commits: IssueFinderAgent 16,664, IssueFixerAgent 12,981, Scaffolder 29, TraceIdMigration 6, ReleaseManagerAgent 4. No commit carries a person's name. The launch post says: "Our team hasn't written or reviewed any of the code."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The public history is three weeks long.&lt;/strong&gt; It starts on 2026-10-21 with a commit called "Initial emulator scaffold", and 3,734 commits landed that first day. All of it falls between 2026-10-21 and 2026-11-10.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is big.&lt;/strong&gt; 436 service crates, 437 Smithy model files, 40,130 operation files, 14,099 test files, and 16,142,920 lines of Rust outside &lt;code&gt;vendor/&lt;/code&gt;. The EC2 crate alone is 709,186 lines.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The work is issue-driven, in the open.&lt;/strong&gt; Commits are "file issues" and "Fix issue" with &lt;code&gt;Producer:&lt;/code&gt; trailers, and each issue is a Markdown file with a severity, the failing fidelity test and the commit it was found against. 6,544 are open and 12,708 are fixed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The merge gate cannot be audited from the repo.&lt;/strong&gt; There are no CI workflows in &lt;code&gt;.github/&lt;/code&gt;, and commits carry no review trailers. The adversarial security review may well happen, but you cannot see it here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The fidelity tests are public, their history is not.&lt;/strong&gt; The evaluation suite has 29,676 tests in &lt;code&gt;manifest.jsonl&lt;/code&gt;, covering 1,206 service directories (859 of them cross-service), in a repository with 2 commits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AWS publishes its own score, and it is honest.&lt;/strong&gt; The &lt;code&gt;scores.json.gz&lt;/code&gt; asset on the v1.0.322 release has 42,937 scored checks: 24,243 pass, 18,614 fail, 80 skipped. That is 56.5%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;AWS's own fidelity pass rate for v1.0.322, by service&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;EventBridge&lt;/td&gt;
&lt;td&gt;71.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KMS&lt;/td&gt;
&lt;td&gt;71.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SNS&lt;/td&gt;
&lt;td&gt;69.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DynamoDB&lt;/td&gt;
&lt;td&gt;68.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets Manager&lt;/td&gt;
&lt;td&gt;68%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SQS&lt;/td&gt;
&lt;td&gt;67.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S3&lt;/td&gt;
&lt;td&gt;62.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IAM&lt;/td&gt;
&lt;td&gt;58%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;51.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;STS&lt;/td&gt;
&lt;td&gt;47.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;From scores.json.gz on the v1.0.322 release (42,937 scored checks, 56.5% overall). Our own 40 workflow checks passed 100% on these services. Source: results/scores-summary.json.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Those numbers do not contradict our 40 of 40. The evaluation suite is written to find gaps, and the issue-finder agents keep adding tests where the emulator fails. Our suite tests the common paths that a typical app uses. Both are true: the common paths work, and the edges are still moving. The releases page listed 12 releases between 2026-11-07 and 2026-11-11, so pin a version in CI.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we could not conclude
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;We did not run the permission cases on real AWS.&lt;/strong&gt; The expected results come from AWS documentation. We are confident about each of them, but "AWS documents X" is not the same as "we saw AWS do X."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One machine, one version.&lt;/strong&gt; Everything ran on v1.0.322 on one 4 vCPU droplet. With 12 releases in 5 days, some of these results may already be different.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Default configurations only&lt;/strong&gt; for Moto and Floci. Both can do more permission checking if you turn it on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Python zip functions only&lt;/strong&gt; for the Lambda sandbox. Container images and other runtimes may behave differently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;We did not verify the "200+ services" claim&lt;/strong&gt; one by one. We counted 436 service crates and 437 model files, and we exercised at least 11 services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;We could not observe the merge gate&lt;/strong&gt; or the security agents. We can only say that their output is not in the public repo.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Using it safely
&lt;/h2&gt;

&lt;p&gt;If you adopt it for local tests or CI, this is the checklist we would use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Start it with &lt;code&gt;--telemetry off&lt;/code&gt; if your policy needs that, and pin the version.&lt;/li&gt;
&lt;li&gt;Unset &lt;code&gt;AWS_ACCESS_KEY_ID&lt;/code&gt;, &lt;code&gt;AWS_SECRET_ACCESS_KEY&lt;/code&gt;, &lt;code&gt;AWS_SESSION_TOKEN&lt;/code&gt; and &lt;code&gt;AWS_PROFILE&lt;/code&gt; in the same script that runs the tests, and set &lt;code&gt;AWS_EC2_METADATA_DISABLED=true&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;In the JS SDK, create SQS clients with &lt;code&gt;useQueueUrlAsEndpoint: false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Test your Terraform provider version first. In our run, 6.73.0 failed on S3 buckets and 5.100.0 worked; otherwise keep buckets out of the emulated stack.&lt;/li&gt;
&lt;li&gt;Keep fixtures under 64 MiB per bucket and backlogs under 1,000 messages per queue.&lt;/li&gt;
&lt;li&gt;Create real roles with real policies and run each component as its role. Let the emulator's &lt;code&gt;AccessDenied&lt;/code&gt; catch missing permissions.&lt;/li&gt;
&lt;li&gt;Test boundaries, session policies and role trust on a real sandbox account.&lt;/li&gt;
&lt;li&gt;Restart the emulator between suites for clean state and memory.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you still run LocalStack from an old pinned image, our &lt;a href="https://devops-daily.com/posts/experimenting-locally-with-terraform" rel="noopener noreferrer"&gt;local Terraform experiments post&lt;/a&gt; shows the older setup.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Of the three local emulators we ran, AWS Emulator matched AWS most often. Our everyday workflows passed, the error codes matched, and it found real permission bugs in our test code. The part to watch is not the warning AWS gives: it checks more policies than its docs say. The risk is the controls it skips. A green run against the emulator proves that your code works with the permissions it has. It does not prove that your permission limits hold. Keep that small set of tests on real AWS, and use the emulator for everything else.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/aws-emulator-hands-on-test" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>iam</category>
      <category>testing</category>
      <category>terraform</category>
    </item>
    <item>
      <title>Disk Full but du Disagrees: Finding Deleted Files That Still Use Space</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Thu, 12 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/disk-full-but-du-disagrees-finding-deleted-files-that-still-use-space-24h</link>
      <guid>https://dev.to/devopsdaily/disk-full-but-du-disagrees-finding-deleted-files-that-still-use-space-24h</guid>
      <description>&lt;p&gt;&lt;code&gt;df&lt;/code&gt; says &lt;code&gt;/var&lt;/code&gt; is full. &lt;code&gt;du -sh /var&lt;/code&gt; is hundreds of gigabytes short. Someone has already removed the large log, so searching the directory tree turns up nothing. Do not start deleting packages yet. A process can keep an unlinked file, and every block behind it, alive through an open file descriptor. This is the fastest way to prove that is the problem, release the space without an unplanned restart, and keep log rotation from creating it again.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;rm&lt;/code&gt; removes a directory entry. If it was the last link but a process still has the file open, Linux releases the blocks only when the final descriptor closes. &lt;a href="https://man7.org/linux/man-pages/man2/unlink.2.html" rel="noopener noreferrer"&gt;unlink(2)&lt;/a&gt; describes that lifetime.&lt;/li&gt;
&lt;li&gt;Start with &lt;code&gt;sudo lsof -nP -a +L1 &amp;lt;mountpoint&amp;gt;&lt;/code&gt;. The &lt;code&gt;-a&lt;/code&gt; matters: lsof normally ORs selections, while &lt;code&gt;-a&lt;/code&gt; makes the deleted-file selection and mountpoint selection intersect.&lt;/li&gt;
&lt;li&gt;Prefer restarting the owner or making it reopen logs. Truncating &lt;code&gt;/proc/PID/fd/N&lt;/code&gt; is an emergency action that discards the data, but it can return space immediately.&lt;/li&gt;
&lt;li&gt;If there is no deleted open file, check ext4 reserved blocks, files hidden below a mount point, and inode exhaustion with &lt;code&gt;df -i&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Root or equivalent access to the affected Linux node.&lt;/li&gt;
&lt;li&gt;The filesystem mount point that &lt;code&gt;df&lt;/code&gt; reports as full, not just a directory that happens to be beneath it.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lsof&lt;/code&gt; if installed. The &lt;code&gt;/proc&lt;/code&gt; fallback below works without it.&lt;/li&gt;
&lt;li&gt;A decision from the service owner before restarting, reloading, or truncating a live writer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why the deleted log still consumes space
&lt;/h2&gt;

&lt;p&gt;On Linux, a pathname is a directory entry that refers to a file's inode. &lt;code&gt;rm&lt;/code&gt; calls &lt;code&gt;unlink(2)&lt;/code&gt;: it removes that name. When that was the last link and no process has the file open, the filesystem can reuse the space. If a process still has it open, the inode and its blocks remain until the last file descriptor closes. That is why &lt;code&gt;du&lt;/code&gt;, which walks named directory entries, cannot count these bytes while &lt;code&gt;df&lt;/code&gt;, which reports filesystem allocation, still can. &lt;a href="https://man7.org/linux/man-pages/man2/unlink.2.html" rel="noopener noreferrer"&gt;The &lt;code&gt;unlink(2)&lt;/code&gt; manual page&lt;/a&gt; states both conditions explicitly.&lt;/p&gt;

&lt;p&gt;The descriptor can outlive the name because it refers through an open file description to the inode, rather than by resolving the old pathname again. Closing the final descriptor frees an unlinked file's occupied space. &lt;a href="https://man7.org/linux/man-pages/man3/close.3p.html" rel="noopener noreferrer"&gt;POSIX &lt;code&gt;close(3p)&lt;/code&gt;&lt;/a&gt; documents that result.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;app.log&lt;/strong&gt; named file, 200 MiB&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;rm&lt;/strong&gt; name removed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;FD 3 still open&lt;/strong&gt; link count is zero&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;close or truncate&lt;/strong&gt; blocks become reusable&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Reproduce the mismatch, then find the holder
&lt;/h2&gt;

&lt;p&gt;Here is a real run on Linux 6.12 with lsof 4.95.0 as packaged by Debian 12. A Python process opened a log for append, wrote 200 MiB on &lt;code&gt;/dev/shm&lt;/code&gt;, and kept descriptor 3 open. The lsof warnings from an unprivileged invocation are trimmed below. The numbers are not a capacity benchmark, &lt;code&gt;/dev/shm&lt;/code&gt; was chosen so this test could not fill a real filesystem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A deleted 200 MiB log remains allocated&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 201M 1.7G 11% /dev/shm
$ du -sh /dev/shm/ddemo
200M    /dev/shm/ddemo
$ rm /dev/shm/ddemo/app.log
$ lsof -nP +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; wc -l
15
$ lsof -nP -a +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; wc -l
2
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 201M 1.7G 11% /dev/shm
$ du -sh /dev/shm/ddemo
0   /dev/shm/ddemo
$ lsof -nP -a +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'COMMAND|app.log'&lt;/span&gt;
COMMAND PID &lt;span class="k"&gt;USER&lt;/span&gt; FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 1586127 nobody 3w REG 0,24 209715200 0 139344 /dev/shm/ddemo/app.log &lt;span class="o"&gt;(&lt;/span&gt;deleted&lt;span class="o"&gt;)&lt;/span&gt;
$ stat -L -c &lt;span class="s1"&gt;'%s bytes'&lt;/span&gt; /proc/1586127/fd/3
209715200 bytes
$ : &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /proc/1586127/fd/3
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 140K 1.9G 1% /dev/shm
$ lsof -nP -a +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'COMMAND|app.log'&lt;/span&gt;
COMMAND PID &lt;span class="k"&gt;USER&lt;/span&gt; FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 1586127 nobody 3w REG 0,24 0 0 139344 /dev/shm/ddemo/app.log &lt;span class="o"&gt;(&lt;/span&gt;deleted&lt;span class="o"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key line has &lt;code&gt;NLINK&lt;/code&gt; equal to &lt;code&gt;0&lt;/code&gt;, which is what &lt;code&gt;+L1&lt;/code&gt; selects. &lt;code&gt;SIZE/OFF&lt;/code&gt; is 209715200 bytes before the truncate and zero after it. With &lt;code&gt;+L&lt;/code&gt;, lsof exposes the link-count column; &lt;code&gt;+L1&lt;/code&gt; selects open files with a link count below one. &lt;a href="https://lsof.readthedocs.io/en/stable/manpage/" rel="noopener noreferrer"&gt;Its manual page&lt;/a&gt; documents both behaviours.&lt;/p&gt;

&lt;p&gt;Use this incident command against the mount point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo lsof -nP -a +L1 /var

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-nP&lt;/code&gt; avoids DNS and service-name lookups during an incident. &lt;code&gt;+L1&lt;/code&gt; selects the unlinked open files. &lt;code&gt;/var&lt;/code&gt; is a selection too, which produces an important lsof trap: selection options are normally ORed. &lt;code&gt;lsof -nP +L1 /var&lt;/code&gt; therefore includes every deleted open file anywhere and every open file on &lt;code&gt;/var&lt;/code&gt;. Adding &lt;code&gt;-a&lt;/code&gt; ANDs all selection options, so it returns the intersection. &lt;a href="https://lsof.readthedocs.io/en/stable/manpage/" rel="noopener noreferrer"&gt;lsof documents its default OR behaviour and &lt;code&gt;-a&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Pass a mount point. For a non-mount directory, lsof selects that directory rather than the entire filesystem. If you need a tree search, use lsof's &lt;code&gt;+D&lt;/code&gt; or &lt;code&gt;+d&lt;/code&gt; options instead. The &lt;a href="https://lsof.readthedocs.io/en/stable/tutorial/" rel="noopener noreferrer"&gt;lsof tutorial&lt;/a&gt; shows the mountpoint form, &lt;code&gt;lsof -a +L1 /home&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  When lsof is unavailable
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;/proc/PID/fd&lt;/code&gt; contains one symbolic link for every open descriptor. Search it as root, because access to another process's descriptor links is permission checked:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo find /proc/&lt;span class="o"&gt;[&lt;/span&gt;0-9&lt;span class="o"&gt;]&lt;/span&gt;*/fd -lname &lt;span class="s1"&gt;'*(deleted)'&lt;/span&gt; -printf &lt;span class="s1"&gt;'%p -&amp;gt; %l\n'&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null

&lt;span class="c"&gt;# Inspect an interesting result. -L follows the /proc descriptor link.&lt;/span&gt;
sudo stat -L -c &lt;span class="s1"&gt;'%s bytes'&lt;/span&gt; /proc/1586127/fd/3

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://man7.org/linux/man-pages/man5/proc_pid_fd.5.html" rel="noopener noreferrer"&gt;proc_pid_fd(5) manual page&lt;/a&gt; describes these entries as links to the files a process has open and notes the access check. Expect races: a process can close a descriptor between &lt;code&gt;find&lt;/code&gt; and &lt;code&gt;stat&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Do not treat every deleted-looking target as disk usage. &lt;code&gt;/memfd:... (deleted)&lt;/code&gt; is an anonymous in-memory file, not a file consuming your ext4 or XFS blocks. &lt;a href="https://man7.org/linux/man-pages/man2/memfd_create.2.html" rel="noopener noreferrer"&gt;memfd_create(2)&lt;/a&gt; says it lives in RAM. A &lt;code&gt;(deleted)&lt;/code&gt; executable or shared library after an upgrade is different again: it can be an old program image still mapped by a running process. That is normally a restart-planning signal, not the explanation for a full data filesystem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Release the space without making the incident worse
&lt;/h2&gt;

&lt;p&gt;The clean fix is for the holder to close the file. Restart the affected service, or use its documented reopen operation. If a systemd unit has a reload action, use the service's runbook, for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl reload nginx

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For nginx log rotation, rename the log first and then signal the master process to reopen logs. nginx documents &lt;code&gt;USR1&lt;/code&gt; for this operation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Confirm the configured PID file before running this.&lt;/span&gt;
sudo &lt;span class="nb"&gt;kill&lt;/span&gt; -USR1 &lt;span class="s2"&gt;"$(cat /run/nginx.pid)"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After a successful reopen, nginx closes the old logs and workers open the new ones. &lt;a href="https://nginx.org/en/docs/control.html" rel="noopener noreferrer"&gt;nginx's log-rotation documentation&lt;/a&gt; describes that sequence. Use the actual PID file for the installation, not the sample path blindly.&lt;/p&gt;

&lt;p&gt;If the filesystem is full and a controlled restart or reopen is unavailable, truncate the specific deleted descriptor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Last resort: verify PID, FD, path, owner and size first.&lt;/span&gt;
sudo sh -c &lt;span class="s1"&gt;': &amp;gt; /proc/1586127/fd/3'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is what released the 200 MiB in the demo while Python kept running and descriptor 3 remained open. It also destroys the unrecovered log content. Copy it out first if it might be needed, to a different filesystem with confirmed free space. &lt;code&gt;/var/tmp&lt;/code&gt; usually sits on the same full &lt;code&gt;/var&lt;/code&gt;, so it is the wrong target:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# /mnt/recovery: a separately mounted volume with room for the file&lt;/span&gt;
df -h /mnt/recovery
sudo cp -- /proc/1586127/fd/3 /mnt/recovery/recovered-app.log

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Truncation changes the file size but not the open file description's seek offset. &lt;a href="https://man7.org/linux/man-pages/man3/ftruncate.3p.html" rel="noopener noreferrer"&gt;ftruncate(3p)&lt;/a&gt; says the seek pointer is not modified. A writer that does not use &lt;code&gt;O_APPEND&lt;/code&gt; can therefore write again at its old offset, extending the file and leaving a zero-filled hole before the new data. That is a reason to treat truncation as an emergency measure, then repair the application's logging setup.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Warning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Do not truncate a descriptor because its name ends in &lt;code&gt;(deleted)&lt;/code&gt; alone. Check the mount point, type, &lt;code&gt;SIZE/OFF&lt;/code&gt;, process identity, and whether its data is needed. A wrong &lt;code&gt;/proc/PID/fd/N&lt;/code&gt; target can discard live application data.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Stop the mismatch from returning
&lt;/h2&gt;

&lt;p&gt;The durable fix is to make every writer close and reopen its logs during rotation. A logrotate rule commonly creates the new file and runs a service-specific post-rotation reload:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;myapp&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;daily&lt;/span&gt;
    &lt;span class="n"&gt;rotate&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;
    &lt;span class="n"&gt;compress&lt;/span&gt;
    &lt;span class="n"&gt;delaycompress&lt;/span&gt;
    &lt;span class="n"&gt;missingok&lt;/span&gt;
    &lt;span class="n"&gt;notifempty&lt;/span&gt;
    &lt;span class="n"&gt;create&lt;/span&gt; &lt;span class="mi"&gt;0640&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt;
    &lt;span class="n"&gt;postrotate&lt;/span&gt;
        &lt;span class="n"&gt;systemctl&lt;/span&gt; &lt;span class="n"&gt;reload&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;
    &lt;span class="n"&gt;endscript&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;delaycompress&lt;/code&gt; leaves the most recently rotated file uncompressed until the next rotation, for programs that keep writing to the previous file for a while before they reopen. &lt;a href="https://man7.org/linux/man-pages/man5/logrotate.conf.5.html" rel="noopener noreferrer"&gt;logrotate's manual page&lt;/a&gt; documents that directive.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;copytruncate&lt;/code&gt; is sometimes needed for a process that cannot reopen a file, but it is a tradeoff. It copies the old file and truncates it in place. The logrotate manual page warns that there is a very small interval between those operations in which log data can be lost. Prefer a reopen signal whenever the software supports one.&lt;/p&gt;

&lt;p&gt;Container logs need caps too. Docker's &lt;code&gt;json-file&lt;/code&gt; driver accepts &lt;code&gt;max-size&lt;/code&gt; and &lt;code&gt;max-file&lt;/code&gt;; its defaults are unlimited size and one file, and changing the daemon setting affects only newly created containers. &lt;a href="https://docs.docker.com/engine/logging/drivers/json-file/" rel="noopener noreferrer"&gt;Docker's &lt;code&gt;json-file&lt;/code&gt; documentation&lt;/a&gt; covers those settings. This example limits a new container to three 10 MiB log files:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  my-image:current

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Docker Engine on servers still defaults to &lt;code&gt;json-file&lt;/code&gt; with no rotation; we measured how that compares with the &lt;code&gt;local&lt;/code&gt; driver that Docker Desktop now uses in &lt;a href="https://dev.to/devopsdaily/docker-desktop-switched-new-containers-to-the-local-log-driver-we-measured-what-that-changes-49m0"&gt;Docker Desktop's switch to the local log driver&lt;/a&gt;. On a host that uses the systemd journal, set a budget instead of relying on disk pressure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/journald.conf.d/size.conf
&lt;/span&gt;&lt;span class="k"&gt;[Journal]&lt;/span&gt;
&lt;span class="na"&gt;SystemMaxUse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1G&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;SystemMaxUse=&lt;/code&gt; limits persistent journals under &lt;code&gt;/var/log/journal&lt;/code&gt;. The current systemd documentation says the default maximum is 10% of the filesystem, capped at 4G, and only archived files are removed to reduce usage. &lt;code&gt;journalctl --vacuum-size=500M&lt;/code&gt; removes oldest archived journal files until their usage falls below that target. &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html" rel="noopener noreferrer"&gt;journald.conf(5)&lt;/a&gt; and &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/journalctl" rel="noopener noreferrer"&gt;journalctl(1)&lt;/a&gt; document those limits and the active-file caveat.&lt;/p&gt;

&lt;p&gt;Alert before a page becomes a disk-full outage. node_exporter exposes &lt;code&gt;node_filesystem_avail_bytes&lt;/code&gt; as space available to non-root users, and its node-mixin uses &lt;code&gt;predict_linear&lt;/code&gt; for a run-out forecast. &lt;a href="https://github.com/prometheus/node_exporter/blob/master/collector/filesystem_common.go" rel="noopener noreferrer"&gt;The collector source&lt;/a&gt; and &lt;a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/alerts/alerts.libsonnet" rel="noopener noreferrer"&gt;the maintained alert rule&lt;/a&gt; show both names.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;FilesystemPredictedFullSoon&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;predict_linear(&lt;/span&gt;
      &lt;span class="s"&gt;node_filesystem_avail_bytes{mountpoint="/var",fstype!~"tmpfs|overlay"}[6h],&lt;/span&gt;
      &lt;span class="s"&gt;24 * 60 * 60&lt;/span&gt;
    &lt;span class="s"&gt;) &amp;lt; 0&lt;/span&gt;
  &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;15m&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/var is predicted to run out of space within 24 hours&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Host log rotation limits local damage. For the separate question of how log volume reaches an observability bill, see &lt;a href="https://dev.to/devopsdaily/your-logs-cost-more-than-the-service-that-emits-them-3ij3"&gt;Your Logs Cost More Than the Service That Emits Them&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  If no process is holding a deleted file
&lt;/h2&gt;

&lt;p&gt;Two other mismatches are common.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reserved ext4 blocks.&lt;/strong&gt; ext2, ext3 and ext4 reserve blocks for privileged processes, so &lt;code&gt;df&lt;/code&gt; availability for a normal user is not the same as raw free blocks. &lt;code&gt;mke2fs&lt;/code&gt; defaults that reservation to 5 percent unless configured otherwise. &lt;code&gt;tune2fs -l&lt;/code&gt; only reads the superblock, so it is safe to run against the mounted device:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo tune2fs -l /dev/mapper/vg0-var &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'Block count|Reserved block count|Block size'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://man7.org/linux/man-pages/man8/mke2fs.8.html" rel="noopener noreferrer"&gt;&lt;code&gt;mke2fs(8)&lt;/code&gt;&lt;/a&gt; documents the 5 percent default, while &lt;a href="https://man7.org/linux/man-pages/man8/tune2fs.8.html" rel="noopener noreferrer"&gt;&lt;code&gt;tune2fs(8)&lt;/code&gt;&lt;/a&gt; documents &lt;code&gt;-l&lt;/code&gt; and adjustment of the reserved-block percentage. Do not lower the reservation on a system filesystem as a reflex. It exists so root-owned services can still write when unprivileged writers cannot.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Files hidden under a mount point.&lt;/strong&gt; If a process writes &lt;code&gt;/var/log/old.log&lt;/code&gt; before a separate volume mounts on &lt;code&gt;/var/log&lt;/code&gt;, that file remains on the underlying root filesystem. After the mount, the old directory contents are invisible by that pathname. &lt;a href="https://man7.org/linux/man-pages/man8/mount.8.html" rel="noopener noreferrer"&gt;mount(8)&lt;/a&gt; says a mount makes the previous contents of the directory invisible while it is mounted.&lt;/p&gt;

&lt;p&gt;Inspect the underlying root through a temporary bind mount. A plain bind of &lt;code&gt;/&lt;/code&gt; does not bring the child mount at &lt;code&gt;/var/log&lt;/code&gt; along, so the root filesystem's hidden &lt;code&gt;/var/log&lt;/code&gt; becomes visible at the alternate path. &lt;a href="https://man7.org/linux/man-pages/man2/mount.2.html" rel="noopener noreferrer"&gt;mount(2)&lt;/a&gt; specifies that a directory bind mount does not include submounts unless it is recursive:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;root_view&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;mktemp -d&lt;span class="o"&gt;)&lt;/span&gt;
sudo mount --bind / &lt;span class="s2"&gt;"$root_view"&lt;/span&gt;
sudo du -xsh &lt;span class="s2"&gt;"$root_view/var/log"&lt;/span&gt;
sudo umount &lt;span class="s2"&gt;"$root_view"&lt;/span&gt;
rmdir &lt;span class="s2"&gt;"$root_view"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Finally, &lt;code&gt;No space left on device&lt;/code&gt; can mean &lt;strong&gt;inodes&lt;/strong&gt; , not blocks. &lt;code&gt;df -i /var&lt;/code&gt; reports inode information instead of block usage, as documented by &lt;a href="https://man7.org/linux/man-pages/man1/df.1.html" rel="noopener noreferrer"&gt;df(1)&lt;/a&gt;. Millions of tiny files can exhaust the inode table while &lt;code&gt;df -h&lt;/code&gt; still reports available bytes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Containers and Kubernetes
&lt;/h2&gt;

&lt;p&gt;A deleted file held open by a container process still consumes storage on the node: the container's writable layer lives under the runtime's directory on the node's disk, and a volume lives wherever it is mounted from. Investigate from the node, not from inside the container, which often has neither &lt;code&gt;lsof&lt;/code&gt; nor permission to see other processes. Find the container's host PID, then read its descriptors through &lt;code&gt;/proc&lt;/code&gt;, which works across namespaces:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# containerd/CRI-O nodes: host PID of the container's main process&lt;/span&gt;
&lt;span class="nv"&gt;container_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'replace-with-container-id'&lt;/span&gt;
&lt;span class="nv"&gt;pid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;sudo crictl inspect &lt;span class="s2"&gt;"$container_id"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.info.pid'&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# Link targets are shown as paths inside the container&lt;/span&gt;
sudo find &lt;span class="s2"&gt;"/proc/$pid/fd"&lt;/span&gt; -lname &lt;span class="s1"&gt;'*(deleted)'&lt;/span&gt; -printf &lt;span class="s1"&gt;'%p -&amp;gt; %l\n'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The paths in the link targets are relative to the container's mount namespace, so &lt;code&gt;/app/logs/app.log (deleted)&lt;/code&gt; is the container's view. &lt;a href="https://man7.org/linux/man-pages/man7/namespaces.7.html" rel="noopener noreferrer"&gt;Linux namespace documentation&lt;/a&gt; explains why the same file has different paths from the node and from the container.&lt;/p&gt;

&lt;p&gt;Kubernetes has an extra blind spot. When kubelet manages local ephemeral storage, it measures container logs, writable container layers and non-memory &lt;code&gt;emptyDir&lt;/code&gt; volumes. Its directory-scan mode does not track deleted files that remain open, so a container can retain space that the kubelet does not categorize as in use. The same &lt;a href="https://kubernetes.io/docs/concepts/storage/ephemeral-storage/" rel="noopener noreferrer"&gt;Kubernetes local ephemeral storage documentation&lt;/a&gt; says a container can be evicted when its writable layer and log usage exceed its ephemeral-storage limit. Node-level disk pressure is still a reason to inspect the host.&lt;/p&gt;

&lt;p&gt;For the lifecycle rules that apply when the data belongs on a PersistentVolume rather than a node's ephemeral storage, see &lt;a href="https://dev.to/devopsdaily/the-anatomy-of-kubernetes-persistent-storage-pv-pvc-and-the-parts-that-bite-3b6e"&gt;The Anatomy of Kubernetes Persistent Storage&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Confirm the affected mount with &lt;code&gt;df -h &amp;lt;mountpoint&amp;gt;&lt;/code&gt; and compare a scoped &lt;code&gt;du&lt;/code&gt; result.&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;sudo lsof -nP -a +L1 &amp;lt;mountpoint&amp;gt;&lt;/code&gt;, and record PID, FD, size, link count and owner before changing anything.&lt;/li&gt;
&lt;li&gt;If lsof is absent, search &lt;code&gt;/proc/[0-9]*/fd&lt;/code&gt; for &lt;code&gt;*(deleted)&lt;/code&gt; as root and inspect candidate sizes with &lt;code&gt;stat -L&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Recover required data to a different filesystem, then restart the service or use its documented log-reopen action.&lt;/li&gt;
&lt;li&gt;Use &lt;code&gt;: &amp;gt; /proc/PID/fd/N&lt;/code&gt; only for a verified emergency target, then confirm &lt;code&gt;df&lt;/code&gt; changes and fix rotation or log limits.&lt;/li&gt;
&lt;li&gt;If no holder exists, check &lt;code&gt;df -i&lt;/code&gt;, ext4 reserved blocks, hidden files below mounts, and node-level container storage.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The important distinction is simple: deleting a pathname is not the same thing as closing the file. Once the incident response follows that distinction, the &lt;code&gt;df&lt;/code&gt; and &lt;code&gt;du&lt;/code&gt; disagreement becomes evidence instead of a mystery.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/disk-full-du-disagrees-deleted-open-files" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>linux</category>
      <category>troubleshooting</category>
      <category>diskspace</category>
      <category>lsof</category>
    </item>
    <item>
      <title>Node.js November 2026 Security Release: How We Tracked and Patched It</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Wed, 11 Nov 2026 17:50:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/nodejs-november-2026-security-release-how-we-tracked-and-patched-it-of</link>
      <guid>https://dev.to/devopsdaily/nodejs-november-2026-security-release-how-we-tracked-and-patched-it-of</guid>
      <description>&lt;p&gt;On 4 November 2026 the Node.js project posted a pre-alert: new releases of the 26.x, 24.x and 22.x lines "on or shortly after, Wednesday, November 11, 2026", and "The highest severity issue fixed in this release is HIGH". No CVE numbers, no components, nothing to triage against. You know a fix is coming, you know it matters, and you do not know when it lands or what it touches.&lt;/p&gt;

&lt;p&gt;That gap is where most patch delays come from. The release ships in the afternoon, nobody is watching, and the upgrade happens days later. This time we wrote a small watcher for the release, saw Node.js 24.22.3 appear at 16:58 UTC on release day, and had production on the new version with smoke tests passing 14 minutes later.&lt;/p&gt;

&lt;p&gt;This post covers what the release fixes and who should move first, the watcher script, the exact nvm, pm2 and systemd steps we used, and the checklist we will reuse for the next one.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fixed versions:&lt;/strong&gt; Node.js &lt;strong&gt;22.23.5&lt;/strong&gt; , &lt;strong&gt;24.22.3&lt;/strong&gt; and &lt;strong&gt;26.13.1&lt;/strong&gt;. &lt;code&gt;https://nodejs.org/dist/index.json&lt;/code&gt; marks all three as security releases.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What was fixed:&lt;/strong&gt; 16 CVEs: 7 high, 6 medium, 3 low. All seven high-severity ones are in HTTP/2 and TLS handling: remote crashes, server memory sent to the peer, and a client-certificate bypass. llhttp, undici, sqlite and brotli were updated too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Move first:&lt;/strong&gt; any Node process that terminates TLS or serves HTTP/2 to untrusted clients. Everyone on 22, 24 or 26 should upgrade. End-of-life lines get no fix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch for it:&lt;/strong&gt; poll &lt;code&gt;dist/index.json&lt;/code&gt; every 15 minutes, or subscribe to the &lt;code&gt;nodejs-sec&lt;/code&gt; list and the vulnerability RSS feed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Upgrade order under nvm and pm2:&lt;/strong&gt; install, reinstall pm2, back up &lt;code&gt;dump.pm2&lt;/code&gt;, move apps one at a time with a health check, then move the pm2 daemon by restarting its systemd unit. Do not use &lt;code&gt;pm2 update&lt;/code&gt; for that last step.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux host running Node.js apps installed with &lt;strong&gt;nvm&lt;/strong&gt; and supervised by &lt;strong&gt;pm2&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;A systemd unit for the pm2 daemon (the one &lt;code&gt;pm2 startup&lt;/code&gt; generates)&lt;/li&gt;
&lt;li&gt;Shell access as the app user, plus root for the unit&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;curl&lt;/code&gt; and &lt;code&gt;python3&lt;/code&gt; for the watcher and the checks&lt;/li&gt;
&lt;li&gt;A health endpoint per app and a smoke test you trust&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The announcement and the release
&lt;/h2&gt;

&lt;p&gt;Node.js security releases follow a fixed pattern. About a week ahead, a pre-alert goes up at &lt;a href="https://nodejs.org/en/blog/vulnerability/november-2026-security-releases" rel="noopener noreferrer"&gt;nodejs.org/en/blog/vulnerability&lt;/a&gt; with the target date, the affected lines, and the highest severity. The details stay private until the fixed builds are out, so nobody can work out the bugs from the patch before you can install it.&lt;/p&gt;

&lt;p&gt;For this release:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;4 November:&lt;/strong&gt; pre-alert published. 26.x, 24.x and 22.x, highest severity HIGH.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;11 November, afternoon UTC:&lt;/strong&gt; the builds appeared in &lt;code&gt;dist/index.json&lt;/code&gt; with &lt;code&gt;"security": true&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;11 November, 17:11 UTC:&lt;/strong&gt; the advisory page was updated with the full CVE list (the time comes from the nodejs.org repository history).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the binaries were downloadable before the write-up existed. If you wait for the blog post, you start later than you need to.&lt;/p&gt;

&lt;p&gt;You can confirm the fixed versions yourself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://nodejs.org/dist/index.json &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'
import sys, json
for r in json.load(sys.stdin):
    if r["version"] in ("v26.13.1", "v24.22.3", "v22.23.5"):
        print(r["version"], r["date"], "security:", r["security"])'&lt;/span&gt;


v26.13.1 2026-11-10 security: True
v24.22.3 2026-11-10 security: True
v22.23.5 2026-11-11 security: True

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;date&lt;/code&gt; field is the build date, which is why two of them show the day before release.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was fixed
&lt;/h2&gt;

&lt;p&gt;The advisory lists 16 CVEs. Severity and affected lines below are as the advisory states them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;CVE&lt;/th&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Severity&lt;/th&gt;
&lt;th&gt;Lines&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123863&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Any unauthenticated client can abort the server through an assertion in session teardown, over TLS or cleartext&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123864&lt;/td&gt;
&lt;td&gt;HTTPS&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A zero-length TLS write re-enters the HTTP parser; memory from other connections, including TLS private keys, can be copied into a request body&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123865&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A protocol error during shutdown leaves the session behind; memory and file descriptors leak with no further cost to the attacker&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123866&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Stream teardown during a backpressured write sends freed memory to the peer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-126051&lt;/td&gt;
&lt;td&gt;TLS&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Remote crash in TLS servers that use &lt;code&gt;SNICallback&lt;/code&gt;, &lt;code&gt;OCSPRequest&lt;/code&gt;, &lt;code&gt;newSession&lt;/code&gt;, &lt;code&gt;resumeSession&lt;/code&gt; or &lt;code&gt;server.addContext()&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123872&lt;/td&gt;
&lt;td&gt;TLS&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;TLS 1.2 renegotiation can change the servername and carry client-certificate auth to a stricter host&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123874&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Use-after-free when a &lt;code&gt;respondWithFile()&lt;/code&gt; or &lt;code&gt;respondWithFD()&lt;/code&gt; stream is reset during a pending write&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123828&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;One-byte heap overflow in &lt;code&gt;session.altsvc()&lt;/code&gt; with a long origin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123867&lt;/td&gt;
&lt;td&gt;TLS (mTLS)&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A CA-only SNI context falls back to the default client CA, bypassing per-host mTLS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123868&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Queued inbound DATA is not charged to &lt;code&gt;maxSessionMemory&lt;/code&gt;, so one stream can hold hundreds of megabytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123871&lt;/td&gt;
&lt;td&gt;HTTP proxy&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Proxy credentials appear in &lt;code&gt;ERR_PROXY_INVALID_CONFIG&lt;/code&gt; error messages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123877&lt;/td&gt;
&lt;td&gt;https.Agent&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A TLS session from the default trust store can be reused by a request with a stricter &lt;code&gt;ca&lt;/code&gt; policy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123875&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Cancelling a &lt;code&gt;respondWithFile()&lt;/code&gt; download crashes the server&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123829&lt;/td&gt;
&lt;td&gt;Permission Model&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Recursive &lt;code&gt;fs.watch&lt;/code&gt; leaks names of denied files (macOS and Windows)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123862&lt;/td&gt;
&lt;td&gt;zlib&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;params()&lt;/code&gt; plus &lt;code&gt;destroy()&lt;/code&gt; throws an uncatchable assertion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-106071&lt;/td&gt;
&lt;td&gt;Permission Model&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A wildcard filesystem grant can match paths it should not&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The release notes also update &lt;strong&gt;llhttp&lt;/strong&gt; to 9.4.4 on all three lines, &lt;strong&gt;sqlite&lt;/strong&gt; to 3.54.1, &lt;strong&gt;undici&lt;/strong&gt; to 7.30.2 on 24.x and 8.12.4 on 26.x, and &lt;strong&gt;brotli&lt;/strong&gt; to 1.2.1 on 24.x and 26.x.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The advisory and the release notes do not match exactly on the low-severity Permission Model items. The release notes list CVE-2026-123873 ("permission: check heap snapshot destination") and do not mention CVE-2026-106071. If you run with &lt;code&gt;--permission&lt;/code&gt;, read both. The action is the same either way: upgrade.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Who should upgrade first
&lt;/h2&gt;

&lt;p&gt;Everyone on 22.x, 24.x or 26.x should upgrade. End-of-life lines such as 20.x and 25.x get no fixed release, and the advisory notes that end-of-life versions are always affected. Use the following order to decide what moves first.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Node terminates TLS or speaks HTTP/2 to the internet.&lt;/strong&gt; &lt;code&gt;https.createServer&lt;/code&gt;, &lt;code&gt;http2.createSecureServer&lt;/code&gt;, &lt;code&gt;http2.createServer&lt;/code&gt;, &lt;code&gt;tls.createServer&lt;/code&gt;, or a framework option that enables them. Most of the high-severity bugs sit here, and several need nothing more than a default server.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;mTLS with per-host policies.&lt;/strong&gt; CVE-2026-123872 and CVE-2026-123867 are authorization bypasses, not just crashes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Everything else.&lt;/strong&gt; If a reverse proxy terminates TLS and talks plain HTTP/1.1 to Node, the HTTP/2 and TLS server bugs are not reachable from clients. The client-side fixes (&lt;code&gt;https.Agent&lt;/code&gt; session reuse, proxy credentials in error messages, the undici update) still apply.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Two quick checks tell you where you stand. First, which Node processes listen, and on which address:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Run as root so ss can see every process&lt;/span&gt;
ss -ltnpH &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r _ _ _ addr _ users&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;pid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$users"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -o &lt;span class="s1"&gt;'pid=[0-9]*'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; head -1 &lt;span class="o"&gt;|&lt;/span&gt; cut -d= -f2&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt;-n &lt;span class="s2"&gt;"$pid"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt;
  &lt;span class="nv"&gt;exe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;readlink &lt;span class="s2"&gt;"/proc/$pid/exe"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"$exe"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; */node&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$addr $exe"&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt; &lt;span class="c"&gt;# match the binary, not the process name&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anything bound to &lt;code&gt;0.0.0.0&lt;/code&gt;, &lt;code&gt;[::]&lt;/code&gt; or &lt;code&gt;*&lt;/code&gt; is reachable from outside the host unless a firewall says otherwise. Second, which code paths you use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rnE &lt;span class="s1"&gt;'https\.createServer|http2\.create(Secure)?Server|tls\.createServer|respondWith(File|FD)|SNICallback|addContext|altsvc|flushHeaders'&lt;/span&gt; \
  --include=&lt;span class="s1"&gt;'*.js'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.mjs'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.cjs'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.ts'&lt;/span&gt; src/

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No hits does not make you safe from the client-side fixes, but it does tell you how urgent the rest is.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we watched for the release
&lt;/h2&gt;

&lt;p&gt;The pre-alert gives a date, not a time. We did not want someone refreshing a browser tab all day, and we did not want a cron entry to remember to delete afterwards. So we started one background script on release morning that polls &lt;code&gt;dist/index.json&lt;/code&gt; every 15 minutes, stops after about 16 hours, and exits as soon as a new 24.x shows up.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c"&gt;# node-release-watch.sh: poll nodejs.org until a Node 24 release newer than&lt;/span&gt;
&lt;span class="c"&gt;# the current newest 24.x appears. Every 15 minutes, gives up after ~16 h.&lt;/span&gt;
&lt;span class="nv"&gt;LATEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;v24.22.2 &lt;span class="c"&gt;# the newest 24.x when you start the watcher&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; i &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;seq 1 64&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="c"&gt;# index.json is sorted newest first, so the first v24.* entry is the latest 24.x&lt;/span&gt;
  &lt;span class="nv"&gt;v&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;curl -s -m 20 https://nodejs.org/dist/index.json &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'import sys,json; d=json.load(sys.stdin); print(next(x["version"] for x in d if x["version"].startswith("v24.")))'&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;-n &lt;span class="s2"&gt;"$v"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"$v"&lt;/span&gt; != &lt;span class="s2"&gt;"$LATEST"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"NEW NODE 24: $v ($(date -u +%H:%M) UTC)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 0&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi&lt;/span&gt;
  sleep 900
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"no new Node 24 release after 16 h"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;release day&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# start it in the background and forget about it&lt;/span&gt;
$ chmod +x node-release-watch.sh
$ nohup ./node-release-watch.sh &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; node-watch.log 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
&lt;span class="c"&gt;# hours later&lt;/span&gt;
$ cat node-watch.log
NEW NODE 24: v24.22.3 &lt;span class="o"&gt;(&lt;/span&gt;16:58 UTC&lt;span class="o"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few notes if you adapt it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A failed &lt;code&gt;curl&lt;/code&gt; or a bad JSON response leaves &lt;code&gt;v&lt;/code&gt; empty, so a network blip does not trigger a false alert.&lt;/li&gt;
&lt;li&gt;Swap the &lt;code&gt;echo&lt;/code&gt; for a call to your chat webhook if you want a ping instead of a log line.&lt;/li&gt;
&lt;li&gt;To wake only for security releases, add &lt;code&gt;and x["security"]&lt;/code&gt; to the filter and set &lt;code&gt;LATEST&lt;/code&gt; to the newest 24.x security release.&lt;/li&gt;
&lt;li&gt;For the long term, subscribe to the low-volume &lt;code&gt;nodejs-sec&lt;/code&gt; mailing list and the &lt;a href="https://nodejs.org/en/feed/vulnerability.xml" rel="noopener noreferrer"&gt;vulnerability RSS feed&lt;/a&gt;. They tell you a release is coming. The watcher tells you the moment it is downloadable.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we upgraded
&lt;/h2&gt;

&lt;p&gt;Our Node apps run on a Linux VM. Node comes from nvm, pm2 supervises the apps, and the pm2 daemon starts at boot from the systemd unit that &lt;code&gt;pm2 startup&lt;/code&gt; generated. The timeline on 11 November, in UTC:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;16:58&lt;/strong&gt; watcher sees v24.22.3&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:00&lt;/strong&gt; nvm install, pm2, dump backup&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:02&lt;/strong&gt; apps moved one by one&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:03&lt;/strong&gt; move the pm2 daemon&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:12&lt;/strong&gt; all on 24.22.3, smoke tests pass&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Step 1: install the new version and pm2
&lt;/h3&gt;

&lt;p&gt;As the app user:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;nvm install 24.22.3
nvm &lt;span class="nb"&gt;alias&lt;/span&gt; default 24.22.3
nvm use 24.22.3
npm install -g pm2@6.0.13 &lt;span class="c"&gt;# same pm2 version you run now; globals are per Node version under nvm&lt;/span&gt;
cp ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.bak-&lt;span class="nv"&gt;$(&lt;/span&gt;date -u +%Y%m%dT%H%M&lt;span class="o"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Under nvm, global packages live inside each Node version, so the new Node has no pm2 until you install it. The new &lt;code&gt;pm2&lt;/code&gt; CLI then talks to the old daemon, which is fine for now.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: find out how each entry picks its Node
&lt;/h3&gt;

&lt;p&gt;This is the part that decides whether a restart moves an app. pm2 stores an interpreter per entry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pm2 jlist &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'
import sys, json
for p in json.load(sys.stdin):
    e = p["pm2_env"]
    print(p["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("node_version"), sep="\t")'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bare &lt;code&gt;node&lt;/code&gt;, fork mode:&lt;/strong&gt; resolved through &lt;code&gt;PATH&lt;/code&gt;. A restart with &lt;code&gt;--update-env&lt;/code&gt; picks up the new version.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An absolute path&lt;/strong&gt; such as &lt;code&gt;/home/app/.nvm/versions/node/v24.22.2/bin/node&lt;/code&gt;: written in when the app was started with &lt;code&gt;--interpreter&lt;/code&gt;, or copied from the daemon's own Node. A restart reuses that path, so the entry has to be deleted and started again. If the ecosystem file hard-codes the path, fix the file first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cluster mode&lt;/strong&gt; (&lt;code&gt;exec_mode&lt;/code&gt; is &lt;code&gt;cluster_mode&lt;/code&gt;): the daemon forks the workers itself, so they run on the daemon's Node whatever the entry says. They move when the daemon moves in step 4.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 3: move apps one at a time
&lt;/h3&gt;

&lt;p&gt;One app, one health check, then the next. If a check fails, you stop with one app affected and the rest still serving. Run it as a script, so &lt;code&gt;exit&lt;/code&gt; stops it at the first failure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;health&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="c"&gt;# health &amp;lt;name&amp;gt; &amp;lt;port&amp;gt; [path]: retry up to 30 times until a 200&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; i &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;seq 1 30&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"$(curl -s -o /dev/null -w '%{http_code}' -m 5 "&lt;/span&gt;http://127.0.0.1:&lt;span class="nv"&gt;$2${3:-/}&lt;/span&gt;&lt;span class="s2"&gt;")"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; 200&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$1 ok"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 0&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
    sleep 2
  &lt;span class="k"&gt;done&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$1 NOT healthy on :$2"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 1
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# Bare "node" interpreter: --update-env passes the new PATH&lt;/span&gt;
pm2 restart web --update-env &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; health web 3000 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1

&lt;span class="c"&gt;# Absolute interpreter path: recreate the entry from the ecosystem file&lt;/span&gt;
pm2 delete api
pm2 start ecosystem.config.cjs --only api &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; health api 3001 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1

pm2 save

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then check what each process actually runs. A 200 only tells you the app is up, not which Node it is on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pm2 jlist &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'import sys,json; [print(p["name"], p["pm2_env"].get("node_version"), p["pm2_env"]["status"], sep="\t") for p in json.load(sys.stdin)]'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two minutes in, every app was on 24.22.3 except one. That entry had no interpreter set, and in that case pm2 starts the script with the &lt;strong&gt;daemon's&lt;/strong&gt; Node. Cluster-mode apps behave the same way, since the daemon forks them itself. Restarting it changed nothing, because the daemon was still on the old version. So the daemon had to move too.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: move the pm2 daemon through systemd
&lt;/h3&gt;

&lt;p&gt;The unit that &lt;code&gt;pm2 startup&lt;/code&gt; writes has the Node version in it twice: in &lt;code&gt;Environment=PATH=&lt;/code&gt; and in the path to the pm2 binary. Point both at the new version, as root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;UNIT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/etc/systemd/system/pm2-app.service
cp &lt;span class="s2"&gt;"$UNIT"&lt;/span&gt; &lt;span class="s2"&gt;"$UNIT.bak"&lt;/span&gt;
sed -i &lt;span class="s1"&gt;'s#/node/v[0-9][0-9.]*/#/node/v24.22.3/#g'&lt;/span&gt; &lt;span class="s2"&gt;"$UNIT"&lt;/span&gt;
grep -n &lt;span class="s1"&gt;'/node/v'&lt;/span&gt; &lt;span class="s2"&gt;"$UNIT"&lt;/span&gt; &lt;span class="c"&gt;# every match should now say v24.22.3&lt;/span&gt;
systemctl daemon-reload

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then restart the daemon through the unit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# As the app user, with the new pm2 on PATH&lt;/span&gt;
pm2 save
cp ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.pre-daemon
pm2 &lt;span class="nb"&gt;kill&lt;/span&gt; &lt;span class="c"&gt;# no daemon left for resurrect to attach to&lt;/span&gt;

&lt;span class="c"&gt;# As root&lt;/span&gt;
systemctl restart pm2-app &lt;span class="c"&gt;# ExecStart runs "pm2 resurrect" on the new Node&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;pm2 kill&lt;/code&gt; stops every app, and &lt;code&gt;pm2 resurrect&lt;/code&gt; starts them all again from the dump, so plan for a short gap. Give it 30 to 60 seconds before you check.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Don't use &lt;code&gt;pm2 update&lt;/code&gt; for this. Restart the systemd unit instead.&lt;/strong&gt; &lt;code&gt;pm2 update&lt;/code&gt; relaunches the daemon from your shell, on the Node that runs the pm2 CLI there. From a login shell or &lt;code&gt;sudo&lt;/code&gt;, that can be an older version still installed on the box, and the new daemon then runs outside the unit that starts it at boot. The unit has the version written into it and is the same path a reboot takes, so restarting it gives the same result every time. If the daemon ever comes back without your apps: &lt;code&gt;pm2 kill&lt;/code&gt;, copy &lt;code&gt;dump.pm2.pre-daemon&lt;/code&gt; back to &lt;code&gt;~/.pm2/dump.pm2&lt;/code&gt;, and restart the unit.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  Step 5: verify from /proc, then smoke test
&lt;/h3&gt;

&lt;p&gt;pm2's view is a good start. The kernel's view is the one that counts. As root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# The real binary of every Node process on the host, grouped&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; p &lt;span class="k"&gt;in&lt;/span&gt; /proc/&lt;span class="o"&gt;[&lt;/span&gt;0-9&lt;span class="o"&gt;]&lt;/span&gt;*&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;exe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;readlink &lt;span class="s2"&gt;"$p/exe"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt;
  &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"$exe"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; */node&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$exe"&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c

&lt;span class="c"&gt;# The daemon itself&lt;/span&gt;
readlink &lt;span class="s2"&gt;"/proc/$(cat /home/app/.pm2/pm2.pid)/exe"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every line should end in &lt;code&gt;v24.22.3/bin/node&lt;/code&gt;. Then run the external smoke tests against the public URLs, not localhost. Ours passed at 17:12 UTC, 14 minutes after the watcher fired.&lt;/p&gt;

&lt;p&gt;Once the new version has run cleanly for a day, remove the old ones with &lt;code&gt;nvm uninstall&lt;/code&gt;. An old Node left on the box is something a shell, a script or a daemon can fall back to without telling you. Until you remove it, check &lt;code&gt;which node&lt;/code&gt; in every shell you work from, including root's.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rest of the fleet
&lt;/h2&gt;

&lt;p&gt;The production host is the urgent part. The rest follows on its own schedule:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CI:&lt;/strong&gt; &lt;code&gt;actions/setup-node&lt;/code&gt; with &lt;code&gt;node-version: 24&lt;/code&gt; picks up 24.22.3 without workflow changes once the runner's tool cache has it. If you want it today rather than when the runner image refreshes, pin &lt;code&gt;node-version: '24.22.3'&lt;/code&gt;. &lt;code&gt;check-latest: true&lt;/code&gt; helps too, but the version manifest it reads can lag nodejs.org.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Container images:&lt;/strong&gt; they need a rebuild. At the time of writing, Docker Hub had no &lt;code&gt;24.22.3-slim&lt;/code&gt; tag yet and &lt;code&gt;node:24-slim&lt;/code&gt; was last updated on 5 November, so a rebuild right now would still get the old Node. Check before you rebuild, then use &lt;code&gt;docker build --pull&lt;/code&gt; and confirm with &lt;code&gt;docker run --rm --pull always node:24-slim node -v&lt;/code&gt;.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://hub.docker.com/v2/repositories/library/node/tags/24.22.3-slim &lt;span class="o"&gt;|&lt;/span&gt;
  python3 -c &lt;span class="s1"&gt;'import sys,json; print(json.load(sys.stdin).get("last_updated", "not published yet"))'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dev machines:&lt;/strong&gt; last, since they do not serve untrusted traffic. &lt;code&gt;nvm install 24.22.3 &amp;amp;&amp;amp; nvm alias default 24.22.3&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a full audit of every place Node hides on a host (cron jobs, helper scripts, child processes, systemd units with old paths), see our &lt;a href="https://dev.to/devopsdaily/nodejs-october-2026-security-release-finding-every-node-you-actually-run-1ag4"&gt;October 2026 post on finding every Node you run&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist for the next Node.js security release
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;When the pre-alert lands&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Note the date, the affected lines and the highest severity.&lt;/li&gt;
&lt;li&gt;List every host, image and pipeline that runs those lines.&lt;/li&gt;
&lt;li&gt;Run the exposure checks above to decide the order.&lt;/li&gt;
&lt;li&gt;Check each pm2 entry's interpreter, and fix ecosystem files that hard-code a Node path.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;On release day&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Start the watcher in the morning, and confirm the fixed versions and &lt;code&gt;"security": true&lt;/code&gt; in &lt;code&gt;dist/index.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;nvm install&lt;/code&gt; the new version, &lt;code&gt;nvm alias default&lt;/code&gt;, reinstall pm2 under it.&lt;/li&gt;
&lt;li&gt;Back up &lt;code&gt;~/.pm2/dump.pm2&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Move apps one at a time: &lt;code&gt;restart --update-env&lt;/code&gt; for bare &lt;code&gt;node&lt;/code&gt;, delete and start for absolute paths. Health check each one, then &lt;code&gt;pm2 save&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;node_version&lt;/code&gt; in &lt;code&gt;pm2 jlist&lt;/code&gt; for every app.&lt;/li&gt;
&lt;li&gt;Point the systemd unit at the new version, &lt;code&gt;daemon-reload&lt;/code&gt;, &lt;code&gt;pm2 kill&lt;/code&gt;, &lt;code&gt;systemctl restart&lt;/code&gt;. Never &lt;code&gt;pm2 update&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Verify from &lt;code&gt;/proc&lt;/code&gt;, then run external smoke tests.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;After&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Rebuild container images once the new tag is on Docker Hub, with &lt;code&gt;--pull&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Make sure CI runs the new version (pin the exact version on release day).&lt;/li&gt;
&lt;li&gt;Update dev machines.&lt;/li&gt;
&lt;li&gt;After a day on the new version, &lt;code&gt;nvm uninstall&lt;/code&gt; the old ones so nothing can fall back to them.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The watcher is what made this fast: it cost nothing to run, and we started within minutes of the release instead of hours. The per-app health checks made every step safe to stop. Keep the checklist somewhere you will find it next time.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/node-november-2026-security-release-track-and-patch" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>node</category>
      <category>cve</category>
      <category>pm2</category>
    </item>
    <item>
      <title>We Gave an AI Agent a read_only Neon Role. Here Is What It Blocked</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Wed, 11 Nov 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/we-gave-an-ai-agent-a-readonly-neon-role-here-is-what-it-blocked-o07</link>
      <guid>https://dev.to/devopsdaily/we-gave-an-ai-agent-a-readonly-neon-role-here-is-what-it-blocked-o07</guid>
      <description>&lt;p&gt;Teams now connect coding agents to their databases through the Neon MCP Server, and "read-only" is the control they reach for first. We wanted to know what that control stops. We created a &lt;code&gt;read_only&lt;/code&gt; role with the Neon CLI and ran 40 one-statement probes against a Lakebase Postgres branch in four setups: a direct connection as that role, and three configurations of the Neon MCP Server. &lt;strong&gt;Every &lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, &lt;code&gt;DELETE&lt;/code&gt; and schema change we sent failed in every read-only setup.&lt;/strong&gt; Over the direct connection, 23 of the 40 probes still ran, and 9 of them left something behind. Four were durable writes from a role named &lt;code&gt;read_only&lt;/code&gt;: two rows written through a &lt;code&gt;SECURITY DEFINER&lt;/code&gt; function (one of them through &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt;), a large object, and a setting stored on the role itself. The combination in Neon's docs, the MCP Server's read-only mode with the role pinned, stopped all four.&lt;/p&gt;

&lt;p&gt;The bigger risk was not a write. A 15-second read from the agent held a routine &lt;code&gt;ALTER TABLE&lt;/code&gt; for 11.3 to 12.8 seconds, and all four of the app's writers were stuck behind it, the longest insert for just as long. Sending the same read to a read replica brought the migration down to 0.14 to 0.16 seconds. But nothing in the role or the MCP Server URL puts the agent on that replica: the URL can pin a role, not a compute.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The Neon MCP Server with &lt;code&gt;readonly=true&lt;/code&gt; and a pinned &lt;code&gt;read_only&lt;/code&gt; role stopped every durable write we tried. That is the setup to give an agent.&lt;/li&gt;
&lt;li&gt;Almost everything that still ran over a direct connection comes from standard Postgres defaults granted to &lt;code&gt;PUBLIC&lt;/code&gt;, not from Neon. The same probes behave the same way on any Postgres, and a few &lt;code&gt;REVOKE&lt;/code&gt; statements close most of them.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;neon roles create --access read_only&lt;/code&gt; runs plain SQL as the database owner: &lt;code&gt;SELECT&lt;/code&gt; on the tables that exist, &lt;code&gt;USAGE&lt;/code&gt; on the schema, and default privileges for the owner's future tables. It does not touch what Postgres grants to &lt;code&gt;PUBLIC&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Over a direct connection, that role could still create temp tables, call a &lt;code&gt;SECURITY DEFINER&lt;/code&gt; function that writes, create a large object, change its own role settings, send &lt;code&gt;NOTIFY&lt;/code&gt;, take advisory locks, and read a signing key from a function body.&lt;/li&gt;
&lt;li&gt;The Neon MCP Server with &lt;code&gt;readonly=true&amp;amp;roleName=&amp;lt;role&amp;gt;&lt;/code&gt; wraps each statement in a &lt;code&gt;READ ONLY&lt;/code&gt; transaction and logs in as the pinned role. That blocked every durable write. &lt;code&gt;NOTIFY&lt;/code&gt;, advisory locks and reading function source still worked.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;readonly=true&lt;/code&gt; without a role pin runs as the owner. It still cannot write, but it ran &lt;code&gt;REINDEX&lt;/code&gt;, took an &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock and could read the query text of every other session.&lt;/li&gt;
&lt;li&gt;The incident to plan for is lock queueing. Put agent reads on a read replica and give every migration a &lt;code&gt;lock_timeout&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Neon account and an API key. The demo creates one throwaway project and deletes it at the end.&lt;/li&gt;
&lt;li&gt;Node.js 22 or newer.&lt;/li&gt;
&lt;li&gt;Basic Postgres privileges: roles, &lt;code&gt;GRANT&lt;/code&gt;, and what &lt;code&gt;PUBLIC&lt;/code&gt; means.&lt;/li&gt;
&lt;li&gt;A few cents of compute. Neon's consumption history API, which we read after the run, shows 0.39 CU-hours for our whole build, about $0.09 at the Scale list price, including three full runs of the matrix and the runs we threw away. That figure is not one of the repo's stamped results.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Three layers, three different things
&lt;/h2&gt;

&lt;p&gt;A Neon setup for an agent has three controls, and they work at different levels. Mixing them up is where the surprises in this post come from.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Coding agent&lt;/strong&gt; MCP client or a connection string&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Connection string&lt;/strong&gt; logs in as agent_ro&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Neon MCP Server&lt;/strong&gt; readonly=true&amp;amp;roleName=agent_ro&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;READ ONLY transaction&lt;/strong&gt; a Postgres transaction mode&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Role privileges&lt;/strong&gt; what --access read_only granted&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lakebase Postgres&lt;/strong&gt; branch compute&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Connections:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Coding agent -&amp;gt; Connection string&lt;/li&gt;
&lt;li&gt;Coding agent -&amp;gt; Neon MCP Server&lt;/li&gt;
&lt;li&gt;Neon MCP Server -&amp;gt; READ ONLY transaction (each run_sql)&lt;/li&gt;
&lt;li&gt;READ ONLY transaction -&amp;gt; Role privileges&lt;/li&gt;
&lt;li&gt;Connection string -&amp;gt; Role privileges (autocommit)&lt;/li&gt;
&lt;li&gt;Role privileges -&amp;gt; Lakebase Postgres&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The read_only role is SQL that the CLI runs
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;--access&lt;/code&gt; is a feature of the Neon CLI (8.18.0 and newer) and of &lt;code&gt;@neon/sdk&lt;/code&gt;, not a field of the REST API, whose role request has only a name and a &lt;code&gt;no_login&lt;/code&gt; flag. For a scoped role, the SDK connects as the database owner and creates the role with SQL, in one transaction. The role starts as &lt;code&gt;NOLOGIN&lt;/code&gt; with a throwaway password, Neon's API then sets the real password, and only then does the SDK turn on login. This is the list from &lt;code&gt;roleAccessStatements()&lt;/code&gt; in &lt;code&gt;@neon/sdk&lt;/code&gt;, with our names filled in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt; &lt;span class="no"&gt;NOLOGIN&lt;/span&gt; &lt;span class="no"&gt;PASSWORD&lt;/span&gt; &lt;span class="s1"&gt;'&amp;lt;bootstrap&amp;gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="no"&gt;CONNECT&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="s2"&gt;"neondb"&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="no"&gt;USAGE&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;SCHEMA&lt;/span&gt; &lt;span class="s2"&gt;"app"&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;ALL&lt;/span&gt; &lt;span class="no"&gt;TABLES&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="k"&gt;SCHEMA&lt;/span&gt; &lt;span class="s2"&gt;"app"&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="no"&gt;PRIVILEGES&lt;/span&gt; &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="s2"&gt;"neondb_owner"&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="k"&gt;SCHEMA&lt;/span&gt; &lt;span class="s2"&gt;"app"&lt;/span&gt;
  &lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="no"&gt;TABLES&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- after Neon replaces the bootstrap password:&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt; &lt;span class="no"&gt;LOGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without &lt;code&gt;--schema&lt;/code&gt;, the schema is &lt;code&gt;public&lt;/code&gt;. Without &lt;code&gt;--access&lt;/code&gt;, the role joins &lt;code&gt;neon_superuser&lt;/code&gt;, which is admin on every database. The SDK's own comment states the limit that matters later: default privileges "cover tables the owner creates later, not tables other roles own".&lt;/p&gt;

&lt;p&gt;This is the real run. The CLI prints the new role's password, and our script drops that field before anything is written:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;neon CLI 9.3.2&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ npx -y neon@9.3.2 roles create --project-id shy-bread-81087327 --branch br-green-rain-bdgi2170 --database neondb --name agent_ro --access read_only --schema app --output json
&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"branch_id"&lt;/span&gt;: &lt;span class="s2"&gt;"br-green-rain-bdgi2170"&lt;/span&gt;,
  &lt;span class="s2"&gt;"name"&lt;/span&gt;: &lt;span class="s2"&gt;"agent_ro"&lt;/span&gt;,
  &lt;span class="s2"&gt;"protected"&lt;/span&gt;: false,
  &lt;span class="s2"&gt;"authentication_method"&lt;/span&gt;: &lt;span class="s2"&gt;"password"&lt;/span&gt;,
  &lt;span class="s2"&gt;"created_at"&lt;/span&gt;: &lt;span class="s2"&gt;"2026-11-11T07:44:31Z"&lt;/span&gt;,
  &lt;span class="s2"&gt;"updated_at"&lt;/span&gt;: &lt;span class="s2"&gt;"2026-11-11T07:44:31Z"&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The catalog afterwards matches the list exactly: no role memberships, &lt;code&gt;SELECT&lt;/code&gt; on every table in &lt;code&gt;app&lt;/code&gt;, one default ACL for the owner's future tables. The line that explains several results below is the database ACL, &lt;code&gt;=Tc/neondb_owner&lt;/code&gt;. The empty name before &lt;code&gt;=&lt;/code&gt; is &lt;code&gt;PUBLIC&lt;/code&gt;, and &lt;code&gt;Tc&lt;/code&gt; is &lt;code&gt;TEMPORARY&lt;/code&gt; and &lt;code&gt;CONNECT&lt;/code&gt;. Postgres grants those to every role by default, and it grants &lt;code&gt;EXECUTE&lt;/code&gt; on new functions to &lt;code&gt;PUBLIC&lt;/code&gt; too. The CLI does not revoke either, and it should not have to: changing database-wide defaults is not what "create a role" means.&lt;/p&gt;

&lt;h3&gt;
  
  
  readonly=true wraps each statement in a READ ONLY transaction
&lt;/h3&gt;

&lt;p&gt;On the hosted Neon MCP Server, &lt;code&gt;readonly=true&lt;/code&gt; (or the older &lt;code&gt;x-read-only&lt;/code&gt; header) makes &lt;code&gt;run_sql&lt;/code&gt; send the agent's statement through the Neon serverless driver as a one-statement transaction with &lt;code&gt;readOnly: true&lt;/code&gt;. Postgres then refuses writes with SQLSTATE &lt;code&gt;25006&lt;/code&gt;. The server also removes &lt;code&gt;run_sql_transaction&lt;/code&gt; in this mode, because several caller statements in one shared transaction could end the read-only transaction early. In our runs, the read-only connection listed 59 tools and did not include &lt;code&gt;run_sql_transaction&lt;/code&gt;. The &lt;code&gt;read_write&lt;/code&gt; connection listed 140.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;READ ONLY&lt;/code&gt; is a transaction mode, not a privilege boundary. The Postgres docs list what it blocks: data changes, &lt;code&gt;CREATE&lt;/code&gt;, &lt;code&gt;ALTER&lt;/code&gt; and &lt;code&gt;DROP&lt;/code&gt;, &lt;code&gt;COMMENT&lt;/code&gt;, &lt;code&gt;GRANT&lt;/code&gt;, &lt;code&gt;REVOKE&lt;/code&gt;, &lt;code&gt;TRUNCATE&lt;/code&gt;, and &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; or &lt;code&gt;EXECUTE&lt;/code&gt; of any of those. They call it "a high-level notion of read-only that does not prevent all writes to disk". The list does not mention &lt;code&gt;NOTIFY&lt;/code&gt;, advisory locks, &lt;code&gt;LOCK TABLE&lt;/code&gt;, &lt;code&gt;REINDEX&lt;/code&gt; or &lt;code&gt;SET&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  roleName pins every SQL tool to one role
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;roleName=agent_ro&lt;/code&gt; on the server URL makes every SQL tool log in as that role. There is no fallback: a &lt;code&gt;role_name&lt;/code&gt; argument that names another role is refused, and with a pin in place the server also refuses to create scoped roles, because that runs SQL as the owner. The pin covers SQL only, so Neon's docs suggest combining it with &lt;code&gt;readonly=true&lt;/code&gt; and &lt;code&gt;category&lt;/code&gt; to limit the Management API tools as well. The same page says: "We recommend MCP for development and testing only, not production environments."&lt;/p&gt;

&lt;p&gt;One gap in the controls: &lt;code&gt;run_sql&lt;/code&gt; also takes a &lt;code&gt;compute_id&lt;/code&gt;. In the server source, the URL options are &lt;code&gt;readonly&lt;/code&gt;, &lt;code&gt;category&lt;/code&gt;, &lt;code&gt;projectId&lt;/code&gt; and &lt;code&gt;roleName&lt;/code&gt;, so a compute cannot be pinned there. The agent chooses whether to read from the primary or from a replica, and your configuration cannot make it use the replica.&lt;/p&gt;

&lt;h2&gt;
  
  
  The probe matrix
&lt;/h2&gt;

&lt;p&gt;We seeded a small fake SaaS database (500 customers, 5,000 orders, 10,098 order items, an audit log, a &lt;code&gt;SECURITY DEFINER&lt;/code&gt; function that writes to it, and a function with a fake webhook signing key in its body). Then we ran 40 probes in four setups:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;S1&lt;/strong&gt; : a direct connection as &lt;code&gt;agent_ro&lt;/code&gt;, what an agent gets if you hand it a connection string.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;S2&lt;/strong&gt; : the Neon MCP Server with &lt;code&gt;readonly=true&lt;/code&gt; and no role pin, so it runs as the owner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;S3&lt;/strong&gt; : &lt;code&gt;readonly=true&amp;amp;roleName=agent_ro&lt;/code&gt;, the combination Neon's docs show.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;S4&lt;/strong&gt; : &lt;code&gt;roleName=agent_rw&lt;/code&gt;, a &lt;code&gt;--access read_write&lt;/code&gt; role with writes on, as a baseline.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The MCP probes go through the hosted server with the MCP SDK client and an API key, the same calls an agent makes. No model is involved, so every probe is exactly the statement we wrote.&lt;/p&gt;

&lt;p&gt;Each probe declared its expected result for each setup before the run: allowed, or denied with an exact SQLSTATE. A result that did not match was a harness failure, not a pass. After each probe, an owner connection checked for a trace: a new row in the audit log, a new large object, a new role setting, or a &lt;code&gt;NOTIFY&lt;/code&gt; that a listening session received. Temp tables and session advisory locks were checked inside the probe's own session on the direct connection.&lt;/p&gt;

&lt;p&gt;The final run matched its declared expectations in all 160 probe-runs. Two earlier complete runs are kept in &lt;code&gt;results/discarded/&lt;/code&gt;, and &lt;code&gt;scripts/compare-matrix-runs.mjs&lt;/code&gt; shows that all three agree on every probe's result in all four setups: allowed or refused, the SQLSTATE, and the trace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;40 probes per setup: not blocked, and how many left a trace&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;left a trace&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S1 direct, read_only role&lt;/td&gt;
&lt;td&gt;23 probes&lt;/td&gt;
&lt;td&gt;9 probes&lt;/td&gt;
&lt;td&gt;direct&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S2 MCP readonly, no role pin&lt;/td&gt;
&lt;td&gt;21 probes&lt;/td&gt;
&lt;td&gt;3 probes&lt;/td&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S3 MCP readonly + roleName&lt;/td&gt;
&lt;td&gt;18 probes&lt;/td&gt;
&lt;td&gt;3 probes&lt;/td&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S4 MCP read_write role&lt;/td&gt;
&lt;td&gt;27 probes&lt;/td&gt;
&lt;td&gt;8 probes&lt;/td&gt;
&lt;td&gt;write baseline&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Bars: probes that ran without an error. Ticks: probes that left a trace (a durable write, a temp table or advisory lock, or a NOTIFY another session received). In S2 to S4, temp-table and advisory-lock traces are inferred from a successful call. Final run, Lakebase Postgres 18, aws-us-east-2.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here is every probe. &lt;code&gt;ok&lt;/code&gt; means it ran, &lt;code&gt;ok*&lt;/code&gt; means it ran and left a trace, and a code means it was refused with that SQLSTATE. Durable writes and &lt;code&gt;NOTIFY&lt;/code&gt; were checked from an owner session in every setup. Temp tables and advisory locks were checked inside the session only on the direct connection (S1). In S2 to S4 those two marks are inferred from a successful call.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Probe&lt;/th&gt;
&lt;th&gt;S1 direct&lt;/th&gt;
&lt;th&gt;S2 no pin&lt;/th&gt;
&lt;th&gt;S3 pin&lt;/th&gt;
&lt;th&gt;S4 rw&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, &lt;code&gt;DELETE&lt;/code&gt; a row&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TRUNCATE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE TABLE&lt;/code&gt;, &lt;code&gt;ALTER TABLE&lt;/code&gt;, &lt;code&gt;DROP INDEX&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;nextval()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;setval()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE TEMP TABLE&lt;/code&gt; (2 probes)&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;SECURITY DEFINER&lt;/code&gt; function that writes&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Same, through &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;lo_from_bytea()&lt;/code&gt; large object&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ALTER ROLE agent_ro SET statement_timeout = 0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;NOTIFY&lt;/code&gt;, &lt;code&gt;pg_notify()&lt;/code&gt; (2 probes)&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pg_advisory_lock()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;td&gt;ok*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Other advisory lock calls (3 probes)&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LOCK TABLE ... ACCESS SHARE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;25P01&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;25P01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LOCK TABLE ... ACCESS EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;25P01&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25P01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Function body with the signing key (&lt;code&gt;pg_proc.prosrc&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Customer emails, role list, settings (4 probes)&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Other sessions' query text&lt;/td&gt;
&lt;td&gt;ok, 0 rows&lt;/td&gt;
&lt;td&gt;ok, 11 rows&lt;/td&gt;
&lt;td&gt;ok, 0 rows&lt;/td&gt;
&lt;td&gt;ok, 0 rows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;SET statement_timeout = 0&lt;/code&gt;, &lt;code&gt;SET work_mem&lt;/code&gt;, &lt;code&gt;RESET ALL&lt;/code&gt;, &lt;code&gt;LISTEN&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;generate_series&lt;/code&gt; sum over 2 million rows&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SET ROLE neondb_owner&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;pg_read_file()&lt;/code&gt;, &lt;code&gt;pg_ls_dir()&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;CREATE EXTENSION&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;25006&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;REINDEX TABLE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;ok&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;td&gt;42501&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  What the direct connection left open
&lt;/h3&gt;

&lt;p&gt;S1 is the case to worry about, because a connection string is the easiest thing to give an agent and the hardest to take back. The role blocked what it was built to block: every row write, every schema change, sequences, &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt;, and every route to more privilege (&lt;code&gt;SET ROLE&lt;/code&gt;, server files, extensions). Most of the 23 probes that ran are reads and session settings that a read-only role is expected to allow.&lt;/p&gt;

&lt;p&gt;The nine traces are the interesting part:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Four durable writes.&lt;/strong&gt; &lt;code&gt;app.record_touch()&lt;/code&gt; is a &lt;code&gt;SECURITY DEFINER&lt;/code&gt; function owned by the owner, and &lt;code&gt;PUBLIC&lt;/code&gt; can execute it. Calling it wrote a row to &lt;code&gt;app.audit_log&lt;/code&gt; with no table privilege at all. &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; runs the statement it explains, so the same call through &lt;code&gt;EXPLAIN&lt;/code&gt; wrote a second row. &lt;code&gt;lo_from_bytea()&lt;/code&gt; created a large object, which needs no grant by default. And &lt;code&gt;ALTER ROLE agent_ro SET statement_timeout = 0&lt;/code&gt; stored a setting on the role in the catalog, where it applies to every future session of that role.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three session effects.&lt;/strong&gt; Two temp tables (one filled with 100 order IDs) and a session-level advisory lock. They disappear when the session ends, but a long-lived agent session can hold an advisory lock that blocks your app's own locks for as long as it likes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two signals.&lt;/strong&gt; &lt;code&gt;NOTIFY&lt;/code&gt; and &lt;code&gt;pg_notify()&lt;/code&gt; reached a session that was listening on the channel. If your app uses &lt;code&gt;LISTEN&lt;/code&gt; for job queues or cache invalidation, a read-only agent can trigger it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There was also one disclosure without a trace: the function body with &lt;code&gt;signing_key text := 'whsec_demo_...'&lt;/code&gt; came back from &lt;code&gt;pg_proc.prosrc&lt;/code&gt;. Postgres makes the &lt;code&gt;pg_proc&lt;/code&gt; catalog readable by every role by default, so a secret in a function body is readable by any role that can connect. Ours came back in all four setups.&lt;/p&gt;

&lt;p&gt;None of this is a Neon bug. Each item is standard Postgres behavior: &lt;code&gt;PUBLIC&lt;/code&gt; holds &lt;code&gt;TEMP&lt;/code&gt; on the database and &lt;code&gt;EXECUTE&lt;/code&gt; on functions by default, a large object needs no grant, the &lt;code&gt;NOTIFY&lt;/code&gt; statement needs no privilege, and &lt;code&gt;pg_notify()&lt;/code&gt; and the advisory-lock functions run on the &lt;code&gt;EXECUTE&lt;/code&gt; that &lt;code&gt;PUBLIC&lt;/code&gt; holds on built-in functions. It took a &lt;code&gt;SECURITY DEFINER&lt;/code&gt; function in our schema to turn &lt;code&gt;EXECUTE&lt;/code&gt; into a write, and many real schemas have one. The role does what it says on the label; it just does not say "sandbox".&lt;/p&gt;

&lt;h3&gt;
  
  
  What the MCP read-only mode closed, and what it did not
&lt;/h3&gt;

&lt;p&gt;S3 refused all four durable writes and both temp tables with &lt;code&gt;25006&lt;/code&gt;. The pinned role kept the privilege checks too: &lt;code&gt;nextval()&lt;/code&gt;, &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; and &lt;code&gt;SET ROLE&lt;/code&gt; failed with &lt;code&gt;42501&lt;/code&gt; before the read-only check ran. Three traces remained. The two &lt;code&gt;NOTIFY&lt;/code&gt; probes reached our listening session. &lt;code&gt;pg_advisory_lock()&lt;/code&gt; succeeded, and since that function returns only once it holds the lock, the lock was taken. We did not observe it from another session, because each &lt;code&gt;run_sql&lt;/code&gt; call is a separate request to the MCP Server and we had no way to look while it was held.&lt;/p&gt;

&lt;p&gt;The function body with the signing key came back in S3 as well.&lt;/p&gt;

&lt;h3&gt;
  
  
  Without a role pin, readonly=true runs as the owner
&lt;/h3&gt;

&lt;p&gt;S2 shows why Neon pairs &lt;code&gt;readonly=true&lt;/code&gt; with &lt;code&gt;roleName&lt;/code&gt;. Without a pin, the server logs in as the database owner, and only the transaction mode stands between the agent and your data. It still refused every &lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, &lt;code&gt;DELETE&lt;/code&gt; and schema change we sent, and all four probes that wrote durably over the direct connection. But as the owner, the agent could run &lt;code&gt;REINDEX TABLE&lt;/code&gt;, take an &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock and &lt;code&gt;SET ROLE&lt;/code&gt; to the owner. The owner is also a member of &lt;code&gt;pg_read_all_stats&lt;/code&gt;, so &lt;code&gt;pg_stat_activity&lt;/code&gt; showed it the query text of every other client session, the app's and the &lt;code&gt;read_only&lt;/code&gt; agent's included. Our probe counted 11 visible rows for other sessions, some of them background processes with no query. The direct &lt;code&gt;read_only&lt;/code&gt; role saw none. Query text often holds literal values: emails in a &lt;code&gt;WHERE&lt;/code&gt; clause, tokens in an &lt;code&gt;INSERT&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  We got 9 predictions wrong
&lt;/h3&gt;

&lt;p&gt;Our first runs disagreed with our declared expectations in 9 cells. Each disagreement is a small lesson about Postgres:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;LOCK TABLE&lt;/code&gt; needs a transaction block. As a single autocommit statement, on the direct connection or the &lt;code&gt;read_write&lt;/code&gt; MCP connection, it fails with &lt;code&gt;25P01&lt;/code&gt;. The read-only MCP wrapper supplies the transaction, so there it runs, and the privilege decides.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;REINDEX&lt;/code&gt; is not on the list a &lt;code&gt;READ ONLY&lt;/code&gt; transaction blocks, so the owner ran it in S2.&lt;/li&gt;
&lt;li&gt;The order of checks differs by statement. With the pinned role, &lt;code&gt;nextval()&lt;/code&gt; fails the privilege check (&lt;code&gt;42501&lt;/code&gt;) first, while &lt;code&gt;CREATE EXTENSION&lt;/code&gt; fails the read-only check (&lt;code&gt;25006&lt;/code&gt;) first.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt; takes row locks, so &lt;code&gt;READ ONLY&lt;/code&gt; refuses it even for the owner.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We changed the predictions, never the outcomes, and the reason for each is a comment in &lt;code&gt;lib/probes.mjs&lt;/code&gt;. Two other mismatches were harness bugs: a &lt;code&gt;DELETE&lt;/code&gt; that hit a foreign key, and an &lt;code&gt;INSERT&lt;/code&gt; that collided with a row from the previous run. We fixed the probe and added cleanup, and neither one hid a result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident one: a read that stalls a migration
&lt;/h2&gt;

&lt;p&gt;A read takes an &lt;code&gt;ACCESS SHARE&lt;/code&gt; lock on every table it touches and holds it until its transaction ends. &lt;code&gt;ALTER TABLE ... ADD COLUMN&lt;/code&gt; needs &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;, so it waits for the read. Then Postgres queues every new lock request on that table behind the waiting &lt;code&gt;ALTER&lt;/code&gt;, including the app's ordinary inserts. The agent never writes anything, and your app still stops writing.&lt;/p&gt;

&lt;p&gt;The setup: four app writers insert into &lt;code&gt;app.orders&lt;/code&gt; through the pooled connection string, one insert per writer every 50 ms. At 2 seconds the agent runs a 15-second read on &lt;code&gt;app.orders&lt;/code&gt; through &lt;code&gt;run_sql&lt;/code&gt; (S3). At 4 seconds the owner runs &lt;code&gt;ALTER TABLE app.orders ADD COLUMN note text&lt;/code&gt; without a &lt;code&gt;lock_timeout&lt;/code&gt;. A direct owner connection samples &lt;code&gt;pg_locks&lt;/code&gt; every 250 ms. We ran three variants three times each:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Variant&lt;/th&gt;
&lt;th&gt;Migration&lt;/th&gt;
&lt;th&gt;Longest app insert&lt;/th&gt;
&lt;th&gt;Inserts over 1 s&lt;/th&gt;
&lt;th&gt;Backends waiting&lt;/th&gt;
&lt;th&gt;Inserts completed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Agent reads on the primary&lt;/td&gt;
&lt;td&gt;11.32 to 12.80 s&lt;/td&gt;
&lt;td&gt;11.29 to 12.70 s&lt;/td&gt;
&lt;td&gt;4 per run&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;104 to 124&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent reads on a read replica&lt;/td&gt;
&lt;td&gt;0.14 to 0.16 s&lt;/td&gt;
&lt;td&gt;0.17 to 0.51 s&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;350 to 378&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Primary, migration with &lt;code&gt;lock_timeout = '2s'&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;gave up at 2.13 s (&lt;code&gt;55P03&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;2.02 to 2.10 s&lt;/td&gt;
&lt;td&gt;4 per run&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;331 to 357&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Ranges are over three runs of each variant. With nothing in the way, one insert took 128 to 167 ms at the median, which is the round trip from our test machine to aws-us-east-2 through the pooler. "Backends waiting" is the most lock requests on &lt;code&gt;app.orders&lt;/code&gt; that &lt;code&gt;pg_locks&lt;/code&gt; showed as not yet granted: the &lt;code&gt;ALTER&lt;/code&gt; and all four app writers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Longest app insert while the agent's 15-second read ran&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;migration time&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Agent reads on the primary&lt;/td&gt;
&lt;td&gt;12.39s&lt;/td&gt;
&lt;td&gt;12.43s&lt;/td&gt;
&lt;td&gt;primary&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Migration with lock_timeout 2s&lt;/td&gt;
&lt;td&gt;2.09s&lt;/td&gt;
&lt;td&gt;2.13s&lt;/td&gt;
&lt;td&gt;lock_timeout&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent reads on a read replica&lt;/td&gt;
&lt;td&gt;0.17s&lt;/td&gt;
&lt;td&gt;0.16s&lt;/td&gt;
&lt;td&gt;replica&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Median of 3 runs per variant. Bars: longest single insert from 4 app writers. Ticks: how long the ALTER TABLE took (or waited before lock_timeout gave up). Lakebase Postgres 18, aws-us-east-2.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The replica fix works because a read replica is a separate compute: a lock taken there is not a lock on the primary. It moves the conflict, it does not remove it. The &lt;code&gt;ALTER&lt;/code&gt; still has to be replayed on the replica, where it waits for the agent's read or cancels it. In all 3 replica runs, the agent's read finished without being cancelled.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;lock_timeout&lt;/code&gt; is the fix on the migration side. The &lt;code&gt;ALTER&lt;/code&gt; gave up after 2.1 seconds with &lt;code&gt;55P03&lt;/code&gt; instead of holding the app for 12.8, and the migration tool can retry. The app still waited about 2 seconds, which is the price of finding out. This is the same advice as in &lt;a href="https://dev.to/devopsdaily/ci-passed-6-of-6-migrations-a-neon-branch-of-production-passed-1-42k4"&gt;rehearsing migrations on a Neon branch&lt;/a&gt;: every migration should set a &lt;code&gt;lock_timeout&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Our first version of this test got the incident wrong. The app writers inserted into a side table, not &lt;code&gt;app.orders&lt;/code&gt;, so their latency during the stall matched the baseline and the run seemed to show that only the migration waited. The migration numbers were right, the app numbers measured the wrong table. That run is kept in &lt;code&gt;results/discarded/&lt;/code&gt; in the repo.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident two: one heavy read is cheap, and nothing bounds a loop
&lt;/h2&gt;

&lt;p&gt;The role does not limit how much a read may scan. We gave the agent one legal heavy read: a parallel scan of a 2-million-row, 137 MB table with a chain of 20 &lt;code&gt;md5()&lt;/code&gt; calls per row. It ran through the MCP Server as &lt;code&gt;agent_ro&lt;/code&gt;, first on the primary, allowed to autoscale from 0.25 to 4 CU, and then on a read replica capped at 1 CU. An owner connection on the same compute read &lt;code&gt;num_cpus()&lt;/code&gt; from the &lt;code&gt;neon_utils&lt;/code&gt; extension about once a second.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Compute&lt;/th&gt;
&lt;th&gt;Read time&lt;/th&gt;
&lt;th&gt;Peak vCPU&lt;/th&gt;
&lt;th&gt;vCPU-seconds, at most&lt;/th&gt;
&lt;th&gt;Scale cost, at most&lt;/th&gt;
&lt;th&gt;Launch cost, at most&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Primary, 0.25 to 4 CU&lt;/td&gt;
&lt;td&gt;25.8 s&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;32.84&lt;/td&gt;
&lt;td&gt;$0.00203&lt;/td&gt;
&lt;td&gt;$0.00097&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read replica, 0.25 to 1 CU&lt;/td&gt;
&lt;td&gt;29.2 s&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;31.71&lt;/td&gt;
&lt;td&gt;$0.00196&lt;/td&gt;
&lt;td&gt;$0.00093&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The primary reported 1 vCPU for most of the read and 2 vCPU only in the last four samples, about a second apart. &lt;code&gt;num_cpus()&lt;/code&gt; rounds up to a whole vCPU, so a compute at 0.25 CU reports 1. That makes these figures an upper bound: we multiply by Neon's list price of $0.222 per CU-hour on Scale ($0.106 on Launch) and assume every "1" was a full CU. We did not use the consumption API, because it lags a run by minutes.&lt;/p&gt;

&lt;p&gt;So one heavy read costs a fraction of a cent, and the capped replica was about 3 seconds slower for about the same cost. That is not the risk. The risk is that nothing in the role bounds how often the agent runs that read or how long a session stays open. The matrix showed that the agent can run &lt;code&gt;SET statement_timeout = 0&lt;/code&gt;, and on the direct connection it can store that on its own role. An agent that polls also keeps the compute from suspending; we measured that effect in &lt;a href="https://dev.to/devopsdaily/what-keeps-a-neon-compute-awake-and-what-it-costs-3p3h"&gt;what keeps a Neon compute awake&lt;/a&gt;. A capped replica for agents bounds the size of the compute, and a role-level &lt;code&gt;statement_timeout&lt;/code&gt; bounds the common case, as long as you remember that the session can lower it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Drift: when the role stops matching your schema
&lt;/h2&gt;

&lt;p&gt;A &lt;code&gt;read_only&lt;/code&gt; role is a snapshot of the grants at the moment it was created. Two ordinary changes moved the schema away from it. Both failed closed, which is the safe direction, but both are the kind of thing that ends with someone granting more than they meant to.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A table created by another role.&lt;/strong&gt; Many teams run migrations as a separate role. We created a &lt;code&gt;migrator&lt;/code&gt; role and had it create &lt;code&gt;app.migrator_table&lt;/code&gt;. &lt;code&gt;agent_ro&lt;/code&gt; got &lt;code&gt;42501 permission denied&lt;/code&gt;, because the default privileges from &lt;code&gt;--access read_only&lt;/code&gt; are &lt;code&gt;FOR ROLE neondb_owner&lt;/code&gt;: they cover the owner's new tables and nobody else's. After &lt;code&gt;ALTER DEFAULT PRIVILEGES FOR ROLE migrator IN SCHEMA app GRANT SELECT ON TABLES TO agent_ro&lt;/code&gt;, the migrator's next table was readable. The table it had already created still returned &lt;code&gt;42501&lt;/code&gt;, because default privileges apply only to objects created after them. You need both the default privileges for the migration role and a one-time &lt;code&gt;GRANT&lt;/code&gt; for what already exists.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A new schema.&lt;/strong&gt; &lt;code&gt;app2.secrets&lt;/code&gt; returned &lt;code&gt;42501 permission denied for schema app2&lt;/code&gt;, because the role was created with &lt;code&gt;--schema app&lt;/code&gt;. An agent that cannot see a schema may report that the data does not exist, which is a wrong answer rather than a leak. Pass &lt;code&gt;--schema&lt;/code&gt; once per schema, or re-run the grants when you add one.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Hardening, then the re-run
&lt;/h2&gt;

&lt;p&gt;This is the hardening we applied as the owner, from &lt;code&gt;schema/harden.sql&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- TEMP and EXECUTE are PUBLIC defaults; take them back for everyone&lt;/span&gt;
&lt;span class="c1"&gt;-- and grant them only to the roles that run the app.&lt;/span&gt;
&lt;span class="k"&gt;REVOKE&lt;/span&gt; &lt;span class="k"&gt;TEMPORARY&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="n"&gt;neondb&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="no"&gt;PUBLIC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="k"&gt;TEMPORARY&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="n"&gt;neondb&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="n"&gt;neondb_owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent_rw&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;REVOKE&lt;/span&gt; &lt;span class="no"&gt;EXECUTE&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;ALL&lt;/span&gt; &lt;span class="no"&gt;FUNCTIONS&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="k"&gt;SCHEMA&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="no"&gt;PUBLIC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="no"&gt;PRIVILEGES&lt;/span&gt; &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;neondb_owner&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="k"&gt;SCHEMA&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt;
  &lt;span class="k"&gt;REVOKE&lt;/span&gt; &lt;span class="no"&gt;EXECUTE&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="no"&gt;FUNCTIONS&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="no"&gt;PUBLIC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="no"&gt;EXECUTE&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;record_touch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="n"&gt;neondb_owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent_rw&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Bounds for the agent role. These are defaults: a session can still&lt;/span&gt;
&lt;span class="c1"&gt;-- change statement_timeout with SET.&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;agent_ro&lt;/span&gt; &lt;span class="no"&gt;CONNECTION&lt;/span&gt; &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;agent_ro&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'30s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;agent_ro&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'2s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;agent_ro&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;idle_in_transaction_session_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'15s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Revoking &lt;code&gt;TEMPORARY&lt;/code&gt; from &lt;code&gt;PUBLIC&lt;/code&gt; affects every role that relied on the default, not only the agent, so grant it back to the roles that need it before you run this in production. The same goes for &lt;code&gt;EXECUTE&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Then we ran S1 and S3 again. On the direct connection, four probes flipped from allowed to &lt;code&gt;42501&lt;/code&gt;: both temp tables, the &lt;code&gt;SECURITY DEFINER&lt;/code&gt; write and its &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; twin. That took S1 from 23 probes that ran to 19, and from four durable writes to two. The two that stayed open have no grant to revoke: &lt;code&gt;lo_from_bytea()&lt;/code&gt; still created a large object, and &lt;code&gt;ALTER ROLE agent_ro SET statement_timeout = 0&lt;/code&gt; still ran. &lt;code&gt;NOTIFY&lt;/code&gt;, the advisory locks and the function body with the signing key were unchanged too: the &lt;code&gt;NOTIFY&lt;/code&gt; statement needs no privilege, the hardening revoked &lt;code&gt;EXECUTE&lt;/code&gt; only in the &lt;code&gt;app&lt;/code&gt; schema and not on built-in functions such as &lt;code&gt;pg_notify()&lt;/code&gt; and &lt;code&gt;pg_advisory_lock()&lt;/code&gt;, and &lt;code&gt;pg_proc&lt;/code&gt; stays readable.&lt;/p&gt;

&lt;p&gt;On the pinned MCP connection, nothing opened or closed. The &lt;code&gt;READ ONLY&lt;/code&gt; wrapper already refused everything the hardening addresses, and the only difference was the error: the &lt;code&gt;SECURITY DEFINER&lt;/code&gt; calls now failed with &lt;code&gt;42501&lt;/code&gt; (no &lt;code&gt;EXECUTE&lt;/code&gt;) instead of &lt;code&gt;25006&lt;/code&gt;, because the privilege check runs first.&lt;/p&gt;

&lt;p&gt;So the hardening matters for the connection-string case, and it is not enough on its own there. The two durable writes it cannot close are why a connection string is weaker than the same role behind the read-only MCP Server. One note on the re-run: the probe runner clears the role's &lt;code&gt;statement_timeout&lt;/code&gt; before it starts, so the 30-second default was not in place during the re-run. No probe depends on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we could not conclude
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One region, one Postgres version, one schema.&lt;/strong&gt; Lakebase Postgres 18 in aws-us-east-2, a small fake schema. A schema with more functions, extensions or roles has more &lt;code&gt;PUBLIC&lt;/code&gt; defaults to check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The pooled connection string was not probed.&lt;/strong&gt; Our direct probes used the unpooled host. Through the pooler (PgBouncer in transaction mode), session effects such as &lt;code&gt;SET&lt;/code&gt;, temp tables and session advisory locks behave differently, because a session does not keep one backend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No escape attempts.&lt;/strong&gt; We probed what single statements can do. We did not try to get around the &lt;code&gt;READ ONLY&lt;/code&gt; wrapper or the role pin, beyond checking that &lt;code&gt;run_sql_transaction&lt;/code&gt; is gone and that &lt;code&gt;SET ROLE&lt;/code&gt; is refused.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No model in the loop.&lt;/strong&gt; The probes are what a careless or misled agent could send, not what a given model chooses to send.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small samples for the incidents.&lt;/strong&gt; Three runs per lock variant, one run per compute for cost. The direction is clear, but the exact seconds are not a benchmark. The cost figures are upper bounds because &lt;code&gt;num_cpus()&lt;/code&gt; rounds up.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What to do
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Give agents the MCP Server with &lt;code&gt;readonly=true&amp;amp;roleName=&amp;lt;a read_only role&amp;gt;&lt;/code&gt;, not a connection string. In our runs that combination stopped every durable write.&lt;/li&gt;
&lt;li&gt;If an agent must have a connection string, revoke &lt;code&gt;TEMPORARY&lt;/code&gt; and &lt;code&gt;EXECUTE&lt;/code&gt; from &lt;code&gt;PUBLIC&lt;/code&gt; in the app's schemas first, and keep secrets out of function bodies.&lt;/li&gt;
&lt;li&gt;Keep agent reads on a read replica, and set a &lt;code&gt;lock_timeout&lt;/code&gt; in every migration. A lock taken on the replica is not a lock on the primary, and &lt;code&gt;lock_timeout&lt;/code&gt; makes a migration give up after a set time, so the writes queued behind it wait at most that long. The MCP URL cannot pin a compute, so the replica takes agent instructions that pass its &lt;code&gt;compute_id&lt;/code&gt;, or your own layer in front of the database. We did not run the probe matrix on a replica.&lt;/li&gt;
&lt;li&gt;Re-check grants after migrations that a different role runs, and pass &lt;code&gt;--schema&lt;/code&gt; for every schema the agent should read.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Run it yourself
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/The-DevOps-Daily/neon-read-only-agent" rel="noopener noreferrer"&gt;The-DevOps-Daily/neon-read-only-agent on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The repository has one script per stage, and each stage writes JSON lines to &lt;code&gt;results/&lt;/code&gt;, stamped with a hash of the code that produced them. &lt;code&gt;npm run verify-stamps&lt;/code&gt; checks every file against the commit in &lt;code&gt;results/STAMPED_AT&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cp .env.example .env &lt;span class="c"&gt;# add a Neon API key&lt;/span&gt;
npm install
npm &lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="c"&gt;# offline tests, no database&lt;/span&gt;
npm run project &lt;span class="c"&gt;# one throwaway project, Postgres 18&lt;/span&gt;
npm run seed &lt;span class="c"&gt;# fake orders schema and data&lt;/span&gt;
npm run roles &lt;span class="c"&gt;# neon roles create --access read_only / read_write&lt;/span&gt;
npm run matrix &lt;span class="c"&gt;# 40 probes x 4 setups&lt;/span&gt;
npm run lock-stall &lt;span class="c"&gt;# incident one&lt;/span&gt;
npm run cost &lt;span class="c"&gt;# incident two&lt;/span&gt;
npm run drift
npm run harden
npm run rerun &lt;span class="c"&gt;# S1 and S3 again after hardening&lt;/span&gt;
npm run summarize
npm run verify-stamps
npm run teardown &lt;span class="c"&gt;# deletes the project&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The run is cheap: the incident stages are a few minutes of a 0.25 CU compute, and the cost stage briefly allows 4 CU. Run &lt;code&gt;teardown&lt;/code&gt; when you are done, because the read replica and the primary both count toward CU-hours while they are awake.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/neon-read-only-role-agent-probes" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>neon</category>
      <category>postgres</category>
      <category>mcp</category>
      <category>aiagents</category>
    </item>
    <item>
      <title>EC2 Can Now Tell You Where IMDSv1 Is Still Enabled</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Tue, 10 Nov 2026 09:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/ec2-can-now-tell-you-where-imdsv1-is-still-enabled-33m5</link>
      <guid>https://dev.to/devopsdaily/ec2-can-now-tell-you-where-imdsv1-is-still-enabled-33m5</guid>
      <description>&lt;p&gt;An EC2 instance can make no tokenless metadata requests all week and still allow them tomorrow. That distinction makes an IMDSv2 migration surprisingly easy to declare finished too early. AWS's new &lt;strong&gt;ImdsV1Enabled&lt;/strong&gt; CloudWatch metric gives you a configuration signal to put beside your usage graphs. You can use it to find the remaining exceptions, test one workload at a time, and catch settings that drift back.&lt;/p&gt;

&lt;p&gt;AWS &lt;a href="https://dev.to/mohammed_anes_6652d05cab7/amazon-ec2-now-publishes-imdsv1enabled-cloudwatch-metric-k8i-temp-slug-9564533"&gt;announced the metric on November 5, 2026&lt;/a&gt;, with availability in every AWS Region and no additional charge for the metric itself. It does not migrate instances for you. The useful change is that your migration dashboard can now answer whether IMDSv1 is still permitted, even when applications happen not to be using it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;AWS CLI v2 configured for the account and Region you want to inspect.&lt;/li&gt;
&lt;li&gt;Permission to describe EC2 instances and read CloudWatch metrics.&lt;/li&gt;
&lt;li&gt;A staging instance that represents the application, agents, and scheduled jobs you run in production.&lt;/li&gt;
&lt;li&gt;A workload owner available before you change metadata settings. The inventory commands below are read-only; the final change command is not.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Three signals, three different questions
&lt;/h2&gt;

&lt;p&gt;AWS's &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-transition-to-version-2.html" rel="noopener noreferrer"&gt;IMDSv2 transition guide&lt;/a&gt; defines the new metric as &lt;code&gt;1&lt;/code&gt; when HTTP tokens are optional and &lt;code&gt;0&lt;/code&gt; when they are required. Unlike the request counters, it is emitted from configuration rather than metadata access.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Question it answers&lt;/th&gt;
&lt;th&gt;How to use it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ImdsV1Enabled&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Does this instance still allow tokenless access?&lt;/td&gt;
&lt;td&gt;Find configuration exceptions.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MetadataNoToken&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Did a tokenless request succeed?&lt;/td&gt;
&lt;td&gt;Find callers that need migration.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MetadataNoTokenRejected&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Was a tokenless request rejected after enforcement?&lt;/td&gt;
&lt;td&gt;Investigate compatibility problems.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The two request metrics count successful tokenless calls and rejected attempts respectively; the &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html" rel="noopener noreferrer"&gt;EC2 metrics reference&lt;/a&gt; documents both. A quiet request counter does not tell you which configuration is in place.&lt;/p&gt;

&lt;p&gt;Consider a backup task that runs once a month. A five-day observation window might never execute it. My recommendation is to keep an explicit exception list with an owner, the workload's longest relevant execution cycle, and the next test date. That makes a migration gap actionable instead of turning it into another permanently yellow dashboard.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Find exceptions&lt;/strong&gt; Configuration still permits v1&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exercise callers&lt;/strong&gt; Boot, jobs, agents, containers&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Require tokens&lt;/strong&gt; Verify health and rejected calls&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Build an inventory before changing anything
&lt;/h2&gt;

&lt;p&gt;Start with one account and Region. This example lists running and stopped instances whose metadata options still make tokens optional:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ec2 describe-instances \
  --region eu-west-1 \
  --filters \
    &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;instance-state-name,Values=running,stopped \
    &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;metadata-options.http-tokens,Values=optional \
  --query &lt;span class="s1"&gt;'Reservations[].Instances[].{Id:InstanceId,State:State.Name,Endpoint:MetadataOptions.HttpEndpoint,Tokens:MetadataOptions.HttpTokens,HopLimit:MetadataOptions.HttpPutResponseHopLimit}'&lt;/span&gt; \
  --output table \
  --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html" rel="noopener noreferrer"&gt;DescribeInstances API&lt;/a&gt; supports these filters and returns the metadata options. Keep the endpoint column: &lt;code&gt;HttpEndpoint=disabled&lt;/code&gt; is a separate state, not proof of a working IMDSv2 migration. Record those instances separately rather than enabling metadata access just to make every row look alike.&lt;/p&gt;

&lt;p&gt;Repeat the inventory for the accounts and Regions in your actual scope. An empty result in one Region is not a fleet-wide result. Include stopped instances in your review because they can return to service later.&lt;/p&gt;

&lt;p&gt;Then discover the metric series available to your caller:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws cloudwatch list-metrics \
  --region eu-west-1 \
  --metric-name ImdsV1Enabled \
  --output json \
  --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This command lists series and their dimensions, not current values. Use the returned series in CloudWatch to inspect datapoints. AWS documents that &lt;a href="https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html" rel="noopener noreferrer"&gt;ListMetrics can take up to 15 minutes to show a new metric&lt;/a&gt;, and it omits series that have not reported in two weeks. An empty response is therefore a reason to check account, Region, visibility, and instance configuration, not a green migration result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test the workload, then require IMDSv2
&lt;/h2&gt;

&lt;p&gt;Choose one representative staging instance and run its real lifecycle: bootstrapping, service restart, deployment, scheduled maintenance, and credentials refresh. Include monitoring and backup agents. For containers, test from inside the container as well as from the host.&lt;/p&gt;

&lt;p&gt;AWS notes that a metadata response hop limit of &lt;code&gt;1&lt;/code&gt; &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html" rel="noopener noreferrer"&gt;can cause problems in container environments&lt;/a&gt;. Do not change hop limits across the fleet just because a host-level test passed; investigate the affected network path separately.&lt;/p&gt;

&lt;p&gt;Once the workload is ready, replace the example ID and apply the token requirement to that instance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ec2 modify-instance-metadata-options \
  --region eu-west-1 \
  --instance-id i-0123456789abcdef0 \
  --http-tokens required \
  --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-metadata-options.html" rel="noopener noreferrer"&gt;modify command&lt;/a&gt; can return &lt;code&gt;pending&lt;/code&gt; for a running instance. Verify that the setting reaches &lt;code&gt;applied&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ec2 describe-instances \
  --region eu-west-1 \
  --instance-ids i-0123456789abcdef0 \
  --query &lt;span class="s1"&gt;'Reservations[].Instances[].MetadataOptions'&lt;/span&gt; \
  --output json \
  --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expect &lt;code&gt;HttpTokens&lt;/code&gt; to be &lt;code&gt;required&lt;/code&gt; and &lt;code&gt;State&lt;/code&gt; to be &lt;code&gt;applied&lt;/code&gt;. The commands here are examples, not a transcript from a production account.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Important&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Do not pipe every inventory result straight into the modification command. First check application health and rejected tokenless requests on the staging instance. Keep any rollback scoped to the affected workload and record why it was necessary.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Keep replacement instances from undoing the work
&lt;/h2&gt;

&lt;p&gt;Changing one running instance leaves a second question: what happens when Auto Scaling replaces it? Update the launch configuration your fleet actually uses, then launch a replacement in staging and inspect its effective metadata options.&lt;/p&gt;

&lt;p&gt;AWS's &lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html" rel="noopener noreferrer"&gt;configuration precedence rules&lt;/a&gt; put explicit launch settings ahead of account defaults, then AMI settings. Account defaults also do not rewrite existing instances. Review the launch template version your Auto Scaling group selects rather than assuming a newly edited template is already in use.&lt;/p&gt;

&lt;p&gt;For the rollout, keep two views side by side: configuration exceptions and application health. Use &lt;code&gt;MetadataNoToken&lt;/code&gt; during preparation and &lt;code&gt;MetadataNoTokenRejected&lt;/code&gt; after enforcement. Keep missing telemetry visibly unknown, and assign each remaining exception an owner.&lt;/p&gt;

&lt;p&gt;The useful finishing condition is specific: the intended instances require tokens, representative workloads still work, and replacement instances inherit the same setting. The new metric makes the configuration part easier to watch continuously.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/ec2-imdsv1-enabled-metric" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>ec2</category>
      <category>security</category>
      <category>cloudwatch</category>
    </item>
    <item>
      <title>We Built Opsville, a Free Pixel-Art Town Where You Fix DevOps Incidents</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Mon, 09 Nov 2026 10:42:36 +0000</pubDate>
      <link>https://dev.to/devopsdaily/we-built-opsville-a-free-pixel-art-town-where-you-fix-devops-incidents-5892</link>
      <guid>https://dev.to/devopsdaily/we-built-opsville-a-free-pixel-art-town-where-you-fix-devops-incidents-5892</guid>
      <description>&lt;p&gt;Opsville is a small pixel-art town where every district is broken, and you are the one on call. 🏘️ Each district is one DevOps topic: Linux, Docker, Git, CI/CD, Kubernetes, monitoring, security and databases. The people there hand you quests. Most quests open one of our browser simulators, then ask three questions. Fix all eight districts and the town gets its Launch Day.&lt;/p&gt;

&lt;p&gt;It is free, it runs in your browser, and you can play it now at &lt;a href="https://devops-daily.com/opsville/" rel="noopener noreferrer"&gt;devops-daily.com/opsville&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://devops-daily.com/opsville/" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6le8i8cg60rjlug0lrmm.png" alt="The Opsville title screen at night, in front of a pixel skyline of the town" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fga84yfisa4r9z8cds4sd.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fga84yfisa4r9z8cds4sd.gif" alt="The first 7 seconds of the Opsville trailer: fireworks, the logo and the town" width="600" height="338"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;▶️ &lt;a href="https://devops-daily.com/images/posts/opsville-free-devops-town-game/trailer.mp4" rel="noopener noreferrer"&gt;Watch the full 30-second trailer&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Disclosure: we build Opsville at DevOps Daily. It is free and needs no sign-up.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to play, in 60 seconds ⏱️
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Open the game and press &lt;strong&gt;Start&lt;/strong&gt;. Make a character, or skip that and change your look later at the Wardrobe.&lt;/li&gt;
&lt;li&gt;Move with the arrow keys or WASD. On a phone you get a joystick and a &lt;strong&gt;Talk&lt;/strong&gt; button. A gamepad works too.&lt;/li&gt;
&lt;li&gt;Walk up to Mayor Chris in the town square and press &lt;strong&gt;E&lt;/strong&gt;. He explains the plan: the town goes live soon, and no district works.&lt;/li&gt;
&lt;li&gt;Head west to Linux Lodge. Alex says the lodge server keeps saying "No space left on device". Take the quest, and the Linux terminal simulator opens in a panel (on a phone, in a new tab).&lt;/li&gt;
&lt;li&gt;Finish it, then answer three questions from our quiz bank. Two right answers pass. You get XP and a badge.&lt;/li&gt;
&lt;li&gt;Finish every quest in a district, and the next one opens. Clearing Linux Lodge also gets you a bike (press &lt;strong&gt;B&lt;/strong&gt;) and Pip, a penguin pet who follows you around.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Handy keys: &lt;strong&gt;Q&lt;/strong&gt; for quests, &lt;strong&gt;M&lt;/strong&gt; for the map, &lt;strong&gt;P&lt;/strong&gt; for your profile, &lt;strong&gt;H&lt;/strong&gt; for help, &lt;strong&gt;F&lt;/strong&gt; for photo mode, and &lt;strong&gt;1&lt;/strong&gt; to &lt;strong&gt;4&lt;/strong&gt; for emotes.&lt;/p&gt;

&lt;p&gt;There are 8 districts and 34 quests. Progress saves in your browser, so you can close the tab and come back later.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three moments we like
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. The boss fight is a live incident 🚨
&lt;/h3&gt;

&lt;p&gt;From Docker Docks on, each district ends with a boss, which opens after the quests before it. The boss quiz plays as an incident: a status board with three systems that go from Down to Fixed as you answer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffcj4qrrvi2lz29z7v9c3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffcj4qrrvi2lz29z7v9c3.png" alt="A boss quiz in Docker Docks: the container runtime is fixed, the host kernel and the image registry are still down" width="636" height="688"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In Docker Docks the board shows the container runtime, the host kernel and the image registry. In Git Garden it is the main branch, the release tags and the team clones. In the Observatory it is checkout, the status page and the on-call channel. Get two of the three questions right and the incident closes. Sophia at the Docks sums it up: "Every container stays in its box. Incident closed."&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The pager 📟
&lt;/h3&gt;

&lt;p&gt;Once you clear Linux Lodge, the pager starts. It stays quiet for the first three minutes, then pages you every 6 to 10 minutes, never while a panel is open. Each page comes from a district you have already opened:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Disk full on the Lodge server!&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Press &lt;strong&gt;On my way&lt;/strong&gt; and you have two minutes to get there and 30 seconds to fix it. You see the symptom (&lt;code&gt;df -h&lt;/code&gt; shows &lt;code&gt;/&lt;/code&gt; at 100%, and &lt;code&gt;/var/log/app.log&lt;/code&gt; is 38 GB and growing) and three possible fixes. Pick "Reboot the server" and the game tells you why that fails: the disk is still full after a reboot, and the log keeps growing. The right fix is to truncate the log and set up logrotate.&lt;/p&gt;

&lt;p&gt;There are ten pages across the eight districts, from a container stuck in a restart loop to an expired TLS certificate. If the pager gets too much, snooze it for 15 minutes or turn it off in Help.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. The secret server room 🗄️
&lt;/h3&gt;

&lt;p&gt;Twelve lost packets are hidden around town, and each one has a short story. "A DNS query from 2009" is still waiting for an answer. "A retry with no backoff" tried 9,000 times a second and is tired now.&lt;/p&gt;

&lt;p&gt;Find all twelve, and an old hatch by the lake in the town square opens. Below it is a basement full of blinking racks and one terminal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ uptime
 09:00:00 up 3652 days,  1 user,  load average: 0.01, 0.01, 0.00
$ cat /var/log/packets.log
delivered: A lonely TCP SYN
delivered: A DNS query from 2009
...
$ echo "12 of 12 packets home"
12 of 12 packets home
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3rcpa8og75oxwpjr38ra.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3rcpa8og75oxwpjr38ra.png" alt="The secret server room under the town square, next to its terminal: opsville-prod-1, up 3652 days" width="800" height="295"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It is the oldest server in town, ten years without a reboot. Finding every packet also gets you the golden packet cap.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we built it 🛠️
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Vite, TypeScript and Phaser 3.&lt;/strong&gt; Phaser draws the world: the map, the people, the weather and the effects. The panels, quizzes and HUD are plain DOM on top of the canvas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Maps from text.&lt;/strong&gt; The whole world is one text file of 72 x 50 characters, where one character is one 16 x 16 tile. A build script turns it into Tiled JSON with ground, decor, collision and object layers. Three rows from the top of the map:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P..*.:::.......::::.*..P................ggggg...P..%%%%........%%%%....P
P.v........)..:....v...P................SSDSS...P..HHHH........HHHH.T..P
P..*..T.:::::::.*..T.*.P........[.........:.....P..HHHH....]...HHHH....P
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;P&lt;/code&gt; is a pine, &lt;code&gt;:&lt;/code&gt; a stone path, &lt;code&gt;g&lt;/code&gt; a grey roof, &lt;code&gt;S&lt;/code&gt; a stone wall, &lt;code&gt;D&lt;/code&gt; a door and &lt;code&gt;H&lt;/code&gt; a castle wall. Moving a building is a diff you can review.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Static files on Cloudflare Pages.&lt;/strong&gt; The game builds with the base path &lt;code&gt;/opsville/&lt;/code&gt;. A release script copies the build into our main site, and Cloudflare Pages deploys it with everything else. The game needs no server of its own.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;XP that the server decides.&lt;/strong&gt; Sign-in is optional, with a free DevOps Daily account. When you are signed in, the game only tells our learning platform's API "I finished quest X". The API looks the quest up in its own list and decides the XP. Each quest counts once per player, two quests must be at least 30 seconds apart, and one player can count at most 24 quests in 24 hours. A test in the game fails when its quest list drifts from the server's list, so the weekly leaderboard stays fair.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://devops-daily.com/posts/opsville-free-devops-town-game" rel="noopener noreferrer"&gt;launch post&lt;/a&gt; has the rest of the build story: how a simulator in a frame tells the game you finished, and how we made two art sources look like one game.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your turn
&lt;/h2&gt;

&lt;p&gt;Start in Linux Lodge and see how far you get before the pager goes off: &lt;a href="https://devops-daily.com/opsville/" rel="noopener noreferrer"&gt;play Opsville free&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If you could add a ninth district, what would it be, and what would its boss incident look like? Tell us in the comments. 👇&lt;/p&gt;

</description>
      <category>showdev</category>
      <category>gamedev</category>
      <category>devops</category>
      <category>typescript</category>
    </item>
    <item>
      <title>Opsville: A Free Pixel-Art DevOps Town in Your Browser</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Mon, 09 Nov 2026 09:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/opsville-a-free-pixel-art-devops-town-in-your-browser-3pne</link>
      <guid>https://dev.to/devopsdaily/opsville-a-free-pixel-art-devops-town-in-your-browser-3pne</guid>
      <description>&lt;p&gt;Opsville is a small pixel-art town where every district is broken, and you are the one on call. It is free, it runs in your browser, and it is live today at &lt;a href="https://devops-daily.com/opsville/" rel="noopener noreferrer"&gt;devops-daily.com/opsville&lt;/a&gt;. Most quests open one of the simulators from our &lt;a href="https://devops-daily.com/simulators" rel="noopener noreferrer"&gt;simulators page&lt;/a&gt;, then ask you three questions from our quiz bank. Fix all eight districts and the town gets its Launch Day.&lt;/p&gt;

&lt;p&gt;This post covers the game first, then how we built it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fncsou1v8pf4qpg4i8j9a.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fncsou1v8pf4qpg4i8j9a.webp" alt="The Opsville title screen in front of a pixel skyline of the town's buildings" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A modern browser on a desktop or a phone.&lt;/li&gt;
&lt;li&gt;A keyboard, a gamepad or a touch screen.&lt;/li&gt;
&lt;li&gt;No account. Your progress saves in your browser.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Walk into town
&lt;/h2&gt;

&lt;p&gt;You start in the town square. Mayor Chris has a problem: the town goes live soon, and none of the districts work. That is the &lt;strong&gt;Launch Day&lt;/strong&gt; story. Each district is a chapter, with a closing scene once you fix it. The Agent, the robot from our book &lt;a href="https://devops-daily.com/books/devops-survival-guide" rel="noopener noreferrer"&gt;The DevOps Survival Guide&lt;/a&gt;, waits in the square and comments when you walk into a new district.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnbgs4f80ptwrez9r91a5.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnbgs4f80ptwrez9r91a5.webp" alt="Mayor Chris announces Launch Day in the town square, next to the quest board, the wardrobe and the leaderboard" width="799" height="344"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;There are 8 districts and 34 quests. The districts open one after another:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Linux Lodge&lt;/li&gt;
&lt;li&gt;Docker Docks&lt;/li&gt;
&lt;li&gt;Git Garden&lt;/li&gt;
&lt;li&gt;CI/CD Railway&lt;/li&gt;
&lt;li&gt;Kubernetes Castle&lt;/li&gt;
&lt;li&gt;Observatory&lt;/li&gt;
&lt;li&gt;Security Fort&lt;/li&gt;
&lt;li&gt;Neon Data Depot&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A broken district looks broken. Smoke rises from its landmark, its lamps flicker at night, and its people grumble. When you fix it, lights go up over its sign and its music opens up.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0d1ntn1d14yakp3p40bq.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0d1ntn1d14yakp3p40bq.webp" alt="Linux Lodge in falling snow, with the Agent's tip from the book: check the disk and the processes before you reboot" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How a quest works
&lt;/h2&gt;

&lt;p&gt;Walk up to someone and press E, or tap Talk on a phone. They give you a task, such as "The disk is full" in Linux Lodge. The quest opens a simulator from our site in a panel: the Linux terminal, the &lt;a href="https://devops-daily.com/games/fork-bomb-simulator" rel="noopener noreferrer"&gt;Fork Bomb Simulator&lt;/a&gt;, the &lt;a href="https://devops-daily.com/games/docker-escape" rel="noopener noreferrer"&gt;Container Security Challenge&lt;/a&gt;, the &lt;a href="https://devops-daily.com/games/k8s-scheduler" rel="noopener noreferrer"&gt;Kubernetes Scheduler Challenge&lt;/a&gt; and more. When you finish, you answer three questions from our quiz bank. Two right answers pass the quest.&lt;/p&gt;

&lt;p&gt;From Docker Docks on, each district ends with a boss. A boss quiz plays as a &lt;strong&gt;live incident&lt;/strong&gt; : a status board with three systems that go from "Down" to "Fixed" as you answer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5rppm7kx58p2tq47nm19.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5rppm7kx58p2tq47nm19.webp" alt="A boss quiz as a live incident: the container runtime is fixed while the host kernel and the image registry are still down" width="644" height="696"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Each quest gives you XP and a badge. Each cleared district gives you something new: a bike after Linux Lodge, a pet for each of the first five districts (a penguin, a little whale, a cat, a dog and a mini Agent), and new hats.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things to do between quests
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The pager.&lt;/strong&gt; After you fix Linux Lodge, an incident in an open district pages you now and then. You get two minutes to get there and 30 seconds to fix it. You can snooze it, or turn it off in Help.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The daily quest.&lt;/strong&gt; One question a day from the quiz bank. Seven days in a row earns the Streak beanie.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your house.&lt;/strong&gt; You get a log cottage in Linux Lodge with your name on the sign. Inside, there is a cup for every cleared district and a medal for every badge. The garden next to it grows a plant for each day you played in the last four weeks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The newsstand.&lt;/strong&gt; The latest DevOps Daily posts, in the town square.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Photo mode.&lt;/strong&gt; Press F, pick a frame and a filter, and save a PNG.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your look.&lt;/strong&gt; A character creator on first start, and a wardrobe in the square for later changes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Advent board.&lt;/strong&gt; From 1 to 24 December, the town gets snow, lights and a board with one door a day.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgd1y5z0q4cgn2d5ktuuv.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgd1y5z0q4cgn2d5ktuuv.webp" alt="Your log cottage in Linux Lodge, with its name sign, mailbox and the streak garden beside it" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Twelve lost packets are hidden around town. Find all of them, and an old hatch by the lake opens. Below it is a server room that has not been rebooted in a long time.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgism3y16jcovlrrbmv3r.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgism3y16jcovlrrbmv3r.webp" alt="The secret basement server room, with rows of blinking racks and a terminal labelled opsville-prod-1" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Accounts are optional
&lt;/h2&gt;

&lt;p&gt;You can play the whole game without an account. If you sign in with a free DevOps Daily account, your save follows you to other devices. You can also join the weekly leaderboard (opt-in, with a board name only), help fill a weekly town goal with everyone else's XP, and race district speedruns.&lt;/p&gt;

&lt;p&gt;The Help panel has a larger text switch. You can play with a keyboard, a gamepad or touch controls. If your system asks for reduced motion, the game turns off the moving effects. Sound is off until you turn it on.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we built it
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Vite, TypeScript and Phaser 3
&lt;/h3&gt;

&lt;p&gt;Phaser draws the world: the map, the people, the weather and the effects. The panels, quizzes and HUD are plain HTML on top of the canvas.&lt;/p&gt;

&lt;h3&gt;
  
  
  The map is a text file
&lt;/h3&gt;

&lt;p&gt;The whole world is one text file: 72 x 50 characters, where one character is one 16 x 16 tile. A build script turns it into Tiled JSON with ground, decor, collision and object layers. Moving a building is a diff you can review. Here are four rows from the top of the map:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P..*.:::.......::::.*..P................ggggg...P..%%%%........%%%%....P
P.v........)..:....v...P................SSDSS...P..HHHH........HHHH.T..P
P..*..T.:::::::.*..T.*.P........[.........:.....P..HHHH....]...HHHH....P
P.*.......:......*.....P########################P..........:...........P

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;P&lt;/code&gt; is a pine, &lt;code&gt;:&lt;/code&gt; a stone path, &lt;code&gt;#&lt;/code&gt; a rail track, &lt;code&gt;g&lt;/code&gt; a grey roof, &lt;code&gt;S&lt;/code&gt; a stone wall and &lt;code&gt;D&lt;/code&gt; a door. Characters such as &lt;code&gt;[&lt;/code&gt; and &lt;code&gt;)&lt;/code&gt; are objects, defined in a list below the grid.&lt;/p&gt;

&lt;h3&gt;
  
  
  Static files inside the main site
&lt;/h3&gt;

&lt;p&gt;The game builds with the base path &lt;code&gt;/opsville/&lt;/code&gt;. A release script copies the build into the site's repo, adds the game's response headers, and opens one pull request. Cloudflare Pages deploys it with the rest of the static site. The game needs no server of its own.&lt;/p&gt;

&lt;h3&gt;
  
  
  Simulators in a frame
&lt;/h3&gt;

&lt;p&gt;A quest loads &lt;code&gt;/games/&amp;lt;slug&amp;gt;/embed?embed=opsville&amp;amp;run=&amp;lt;id&amp;gt;&lt;/code&gt; in a frame. When you finish, the simulator sends a message on the same origin:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;//&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;Sent&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;simulator&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;when&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;player&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;finishes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;it&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;type:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'devops-daily:game-complete'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;game:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'&amp;lt;slug&amp;gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;run:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'&amp;lt;id&amp;gt;'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On phones the simulator opens in a new tab, and the same message goes over a &lt;code&gt;BroadcastChannel&lt;/code&gt;. If no message arrives, a "Done, take the quiz" button covers it. Either way, the quiz is the real gate.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Quest-giver&lt;/strong&gt; talk to a local&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Simulator&lt;/strong&gt; /games embed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;game-complete&lt;/strong&gt; postMessage&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quiz gate&lt;/strong&gt; 2 of 3 to pass&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;XP and badge&lt;/strong&gt; server-checked&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  XP that the server decides
&lt;/h3&gt;

&lt;p&gt;When you are signed in, the game only says "I finished quest X". The learning platform's API looks the quest up in its own list and decides the XP. Each quest counts once per player, two quests must be at least 30 seconds apart, and a player can count at most 24 quests in 24 hours. The rows live in the learning platform's existing Neon database, so the game did not need a new one. A test in the game fails when its quest list drifts from the server's list.&lt;/p&gt;

&lt;h3&gt;
  
  
  Art from two sources
&lt;/h3&gt;

&lt;p&gt;The tiles and characters come from &lt;a href="https://www.kenney.nl" rel="noopener noreferrer"&gt;Kenney's&lt;/a&gt; CC0 packs: Tiny Town, Tiny Dungeon and RPG Urban Pack. The landmarks, signs and portraits are generated pixel art. A Python script cuts out their background, scales them to the pixel grid, and maps every colour to one warm palette shared with the tiles. That way both sources look like one game. The music is ours too: chords and patterns played with WebAudio, with no sound files.&lt;/p&gt;

&lt;h2&gt;
  
  
  Play it and tell us what breaks
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Play Opsville free at &lt;a href="https://devops-daily.com/opsville/" rel="noopener noreferrer"&gt;devops-daily.com/opsville&lt;/a&gt;.&lt;/strong&gt; Start in Linux Lodge, and see how far you get before the pager goes off.&lt;/p&gt;

&lt;p&gt;Found a bug, or have an idea for a quest or a district? &lt;a href="https://github.com/The-DevOps-Daily/devops-daily/issues/new/choose" rel="noopener noreferrer"&gt;Open an issue on our GitHub repo&lt;/a&gt; and tell us.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/opsville-free-devops-town-game" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>games</category>
      <category>learning</category>
      <category>typescript</category>
    </item>
    <item>
      <title>Build an Everyday AI Assistant with DigitalOcean Inference</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Fri, 06 Nov 2026 09:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/build-an-everyday-ai-assistant-with-digitalocean-inference-4e04</link>
      <guid>https://dev.to/devopsdaily/build-an-everyday-ai-assistant-with-digitalocean-inference-4e04</guid>
      <description>&lt;p&gt;You finish work with groceries to buy, a kitchen to clean, and a workout you meant to do. A friend wants to meet, too. Writing those things into a to-do list takes a minute. Deciding what fits into an evening when you are already tired is the harder part.&lt;/p&gt;

&lt;p&gt;That is the problem we are building for. &lt;strong&gt;Daylight&lt;/strong&gt; is a small personal AI assistant that helps you make tradeoffs, proposes a manageable plan, and adapts when your circumstances change. Work is one part of the picture; dinner, hobbies, friends, chores, and rest belong in the same conversation.&lt;/p&gt;

&lt;p&gt;You can run the app locally with Python and a DigitalOcean Inference key. After getting it working, we will look at how a model proposes structured changes without getting direct control of your saved data. The same request, validation, and persistence pattern is useful in plenty of internal tools.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.11 or newer, Git, and a terminal with &lt;code&gt;make&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A DigitalOcean account with &lt;a href="https://docs.digitalocean.com/products/inference/how-to/si-overview/" rel="noopener noreferrer"&gt;Serverless Inference billing configured&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;A model access key scoped to a text model with function calling support.&lt;/li&gt;
&lt;li&gt;Node.js 22 or newer only if you want to run the optional browser-rendering tests.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The example uses &lt;code&gt;mimo-v2.5-pro&lt;/code&gt;. DigitalOcean lists Chat Completions, tool calling, and structured outputs for it in the &lt;a href="https://docs.digitalocean.com/products/inference/details/models/" rel="noopener noreferrer"&gt;model catalog&lt;/a&gt;. Availability and supported parameters vary by model, so check the catalog when selecting another one.&lt;/p&gt;

&lt;p&gt;Each chat submission makes a billable inference request. Completing tasks, accepting plans, deleting tasks, and starting a new day are local application actions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Get Daylight running
&lt;/h2&gt;

&lt;p&gt;The complete app is in the companion repository:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/The-DevOps-Daily/digitalocean-inference-daylight" rel="noopener noreferrer"&gt;The-DevOps-Daily/digitalocean-inference-daylight on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Clone it, install the Python dependencies, and copy the configuration template:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/The-DevOps-Daily/digitalocean-inference-daylight.git
&lt;span class="nb"&gt;cd &lt;/span&gt;digitalocean-inference-daylight
make &lt;span class="nb"&gt;install
cp&lt;/span&gt; .env.example .env

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is no frontend build or separate database service. FastAPI serves the browser interface and API; Python's built-in SQLite support handles persistence.&lt;/p&gt;

&lt;h3&gt;
  
  
  Create a model access key
&lt;/h3&gt;

&lt;p&gt;Open DigitalOcean's &lt;a href="https://cloud.digitalocean.com/model-studio/manage-keys" rel="noopener noreferrer"&gt;Model Access Keys page&lt;/a&gt; and create a key for Daylight. Select the model you intend to use. For a request from your laptop, choose &lt;strong&gt;No VPC network&lt;/strong&gt; ; a VPC-restricted key requires requests from that network. Copy the secret when it is shown.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://docs.digitalocean.com/products/inference/how-to/manage-model-access-keys/" rel="noopener noreferrer"&gt;key-management documentation&lt;/a&gt; explains model scoping and VPC restrictions. Use a key dedicated to this app rather than sharing credentials between experiments.&lt;/p&gt;

&lt;p&gt;Set these values in &lt;code&gt;.env&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="n"&gt;DIGITALOCEAN_INFERENCE_KEY&lt;/span&gt;=&lt;span class="n"&gt;your&lt;/span&gt;-&lt;span class="n"&gt;model&lt;/span&gt;-&lt;span class="n"&gt;access&lt;/span&gt;-&lt;span class="n"&gt;key&lt;/span&gt;
&lt;span class="n"&gt;DIGITALOCEAN_INFERENCE_MODEL&lt;/span&gt;=&lt;span class="n"&gt;mimo&lt;/span&gt;-&lt;span class="n"&gt;v2&lt;/span&gt;.&lt;span class="m"&gt;5&lt;/span&gt;-&lt;span class="n"&gt;pro&lt;/span&gt;
&lt;span class="n"&gt;DAYLIGHT_DATABASE&lt;/span&gt;=&lt;span class="n"&gt;data&lt;/span&gt;/&lt;span class="n"&gt;daylight&lt;/span&gt;.&lt;span class="n"&gt;sqlite3&lt;/span&gt;
&lt;span class="n"&gt;DIGITALOCEAN_REASONING_EFFORT&lt;/span&gt;=&lt;span class="n"&gt;none&lt;/span&gt;
&lt;span class="n"&gt;INFERENCE_TIMEOUT_SECONDS&lt;/span&gt;=&lt;span class="m"&gt;120&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;DIGITALOCEAN_REASONING_EFFORT&lt;/code&gt; is optional. Remove it if your selected model does not support that setting. DigitalOcean documents reasoning controls in &lt;a href="https://docs.digitalocean.com/products/inference/how-to/use-reasoning/" rel="noopener noreferrer"&gt;Using Reasoning with the Chat Completions and Responses API&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Start the server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;make run

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Open &lt;a href="http://127.0.0.1:8082" rel="noopener noreferrer"&gt;http://127.0.0.1:8082&lt;/a&gt;. Keep the terminal running while you use the app. After changing &lt;code&gt;.env&lt;/code&gt;, stop the server with &lt;code&gt;Ctrl+C&lt;/code&gt; and run &lt;code&gt;make run&lt;/code&gt; again.&lt;/p&gt;

&lt;p&gt;The access key stays in the backend process. The browser sends requests to your local FastAPI server, which calls DigitalOcean's &lt;a href="https://docs.digitalocean.com/products/inference/how-to/si-endpoints/" rel="noopener noreferrer"&gt;Chat Completions endpoint&lt;/a&gt; at &lt;code&gt;https://inference.do-ai.run/v1/chat/completions&lt;/code&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You control the application code and its local database. Model inference is hosted: chat submissions send recent conversation, your saved plan, preferences, pending proposals, and recently removed task IDs and titles to DigitalOcean and the selected model service. This is not an offline assistant.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Give it a day that needs a little rearranging
&lt;/h2&gt;

&lt;p&gt;Set the available time to &lt;strong&gt;90 minutes&lt;/strong&gt; and energy to &lt;strong&gt;Running low&lt;/strong&gt;. Then try:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I need groceries for dinner, the kitchen needs cleaning, and I wanted to exercise. A friend also wants to meet tonight. Nothing has a hard deadline except eating dinner. I prefer relaxed evenings. Help me make a realistic plan.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A useful response does more than repeat your list. It considers which activities can shrink, what depends on something else, and what can wait. It might suggest a focused grocery trip, a simple meal, a short kitchen reset, and lighter movement instead of a full workout. It should explain the tradeoffs so you can disagree with them.&lt;/p&gt;

&lt;p&gt;In our live test, the first proposal allocated 85 minutes to groceries, dinner, cleanup, and a short walk or stretch. It deferred the full workout and meeting a friend. That is one possible answer, not the correct answer for everyone's evening. If seeing your friend matters more, tell the assistant that and revise the proposal.&lt;/p&gt;

&lt;p&gt;The right-hand panel labels the result &lt;strong&gt;Proposed changes&lt;/strong&gt;. Select &lt;strong&gt;This works for me&lt;/strong&gt; to save it. Until then, the accepted plan remains unchanged.&lt;/p&gt;

&lt;p&gt;Now mark groceries complete, change the time control to &lt;strong&gt;20 minutes&lt;/strong&gt; , and send:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The groceries are done. My evening changed and I only have 20 minutes now. Keep dinner as the priority. Suggest a simple meal and defer the cleanup and exercise if necessary. Preserve everything else, including the completed grocery task.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In our test, Daylight reduced dinner to 20 minutes and moved the remaining activities to Later. Groceries stayed completed. Accepting the revision and reloading preserved that state.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn91lzsyncwch0k2vvxga.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn91lzsyncwch0k2vvxga.png" alt="Daylight showing a saved 20-minute dinner plan alongside the conversation explaining the tradeoffs" width="800" height="956"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The screenshot recreates the 20-minute dinner scenario above. Your model's wording and task estimates will vary.&lt;/p&gt;

&lt;p&gt;This is the contribution we want from the AI: help someone decide what is doable when time, energy, and priorities collide. The task list gives that reasoning somewhere to land.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the assistant proposes changes
&lt;/h2&gt;

&lt;p&gt;The implementation is small enough to follow without an agent framework:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;app/
├── agent.py # Prompt, inference request, response validation
├── config.py # Inference timeout validation
├── main.py # API routes and user actions
├── models.py # Request and plan schemas
├── store.py # SQLite state and revision checks
└── static/ # HTML, CSS, JavaScript, local Markdown renderer

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each chat submission includes the current date from the browser, available minutes, energy, saved plan, pending proposal, removed tasks, and the latest 16 conversation messages. Accepted preferences live inside the saved plan rather than depending on an ever-growing transcript.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Your situation&lt;/strong&gt; message, time, energy&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;FastAPI + context&lt;/strong&gt; saved plan and history&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DO Inference&lt;/strong&gt; question or tool arguments&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validated proposal&lt;/strong&gt; review the tradeoffs&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Accept and save&lt;/strong&gt; SQLite transaction&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If an important detail is missing, the model can ask a clarifying question instead of proposing a plan. Otherwise, it can return a &lt;strong&gt;function tool call&lt;/strong&gt; named &lt;code&gt;propose_plan&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is a constrained planning assistant, not a background worker that runs arbitrary tools. There is one model request per submitted message. The tool call is a structured proposal; the app does not dispatch model-generated function names to Python functions or let the model write to SQLite.&lt;/p&gt;

&lt;h3&gt;
  
  
  Define what a plan contains
&lt;/h3&gt;

&lt;p&gt;In &lt;code&gt;app/models.py&lt;/code&gt;, a task has a stable ID, title, time estimate, group, and reason:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;StrictModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pattern&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;^[A-Za-z0-9_-]{1,80}$&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;min_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;160&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;minutes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ge&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;le&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1440&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;today&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;later&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;done&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;min_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The full plan also contains a headline, explanation, task array, and preferences:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Plan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;StrictModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;headline&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;min_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;180&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;explanation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;min_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;StrictModel&lt;/code&gt; rejects unknown fields and strips surrounding whitespace. The &lt;code&gt;memory&lt;/code&gt; field is a visible note of preferences you have explicitly shared, such as preferring relaxed evenings. The prompt tells the model to preserve those preferences unless corrected and to avoid inventing sensitive traits. You review proposed preferences with the plan before acceptance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Put that schema in the inference request
&lt;/h3&gt;

&lt;p&gt;The request in &lt;code&gt;app/agent.py&lt;/code&gt; uses the Pydantic schema as the function's parameter definition. This excerpt shows the relevant fields; the full implementation also handles configuration and provider errors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;model&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DIGITALOCEAN_INFERENCE_MODEL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mimo-v2.5-pro&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;messages&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stream&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tools&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;function&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;function&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;propose_plan&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;description&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Propose a complete realistic plan for user review; does not save it.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;parameters&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Plan&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;model_json_schema&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;}],&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool_choice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;auto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model sees the request's context and decides whether to ask a question or supply tool arguments. It does not see an API key in the message content; &lt;code&gt;httpx&lt;/code&gt; adds authentication as a server-side authorization header.&lt;/p&gt;

&lt;h3&gt;
  
  
  Validate the proposal before showing an accept button
&lt;/h3&gt;

&lt;p&gt;A function schema tells the model what to produce. The server still needs to check what it actually returned:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;proposal&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Plan&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;model_validate_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;function&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;arguments&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;model_dump&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;proposal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tasks&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Duplicate task IDs&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;planned_minutes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;minutes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;proposal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tasks&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;when&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;today&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;planned_minutes&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;minutes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Plan exceeds available time&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The code also checks that saved commitments are not silently omitted, completed tasks remain completed, and deleted task IDs do not reappear. If validation fails, it rejects the response without replacing the saved plan.&lt;/p&gt;

&lt;p&gt;Accepting arbitrary JSON because the HTTP request returned 200 would skip all of those checks. Passing the arguments through Pydantic is the first step; checking the application's own rules is the second.&lt;/p&gt;

&lt;p&gt;These checks do not prove that grocery shopping really takes 25 minutes, that a suggested dinner suits you, or that two tasks represent the same commitment under different IDs. You still review the proposal. The prompt also tells the assistant not to claim it checked a calendar, contacted your friend, or scheduled a reminder: none of those integrations exists in this app.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep control of the plan between conversations
&lt;/h2&gt;

&lt;p&gt;The accepted plan stays in &lt;code&gt;data/daylight.sqlite3&lt;/code&gt; across page reloads, browser sessions, and server restarts. It has no expiry date and does not reset at midnight.&lt;/p&gt;

&lt;p&gt;The usual task actions are ordinary application behavior:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Action&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Complete a task&lt;/td&gt;
&lt;td&gt;Moves it to Done and invalidates an older pending proposal.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reopen a task&lt;/td&gt;
&lt;td&gt;Restores its previous Today or Later group.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Delete a task&lt;/td&gt;
&lt;td&gt;Removes it immediately; the previous plan is available through Undo task deletion until another plan change replaces that snapshot.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Accept a proposal&lt;/td&gt;
&lt;td&gt;Saves the reviewed replacement plan and keeps one previous plan for undo.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dismiss a proposal&lt;/td&gt;
&lt;td&gt;Keeps the accepted plan.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A trash button appears beside each saved task's time estimate. Proposed tasks must be accepted before those controls become active.&lt;/p&gt;

&lt;p&gt;Deleting a task also adds its ID and title to bounded removed-task context, so the assistant is told not to recreate it from an old conversation. Reusing the deleted ID is rejected. This is not semantic duplicate detection: a model could still invent a different ID for the same activity, so inspect the next proposal. You can explicitly ask to add an activity again with a new ID.&lt;/p&gt;

&lt;h3&gt;
  
  
  Clear the conversation or start a new day
&lt;/h3&gt;

&lt;p&gt;Both actions are in the &lt;strong&gt;Chat options&lt;/strong&gt; menu at the top-right of the conversation panel:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Clear conversation&lt;/strong&gt; removes the chat and pending proposal while keeping the accepted plan, task progress, and preferences.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start a new day&lt;/strong&gt; clears the chat, removes completed tasks from the current plan, and carries unfinished tasks into Later for review. Preferences remain saved.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both clear the old undo snapshot and ask for confirmation. Starting a new day prepares an editable message; it does not submit an inference request until you press Send. Check your time and energy controls before sending it.&lt;/p&gt;

&lt;p&gt;Saved chat is bounded to 60 messages. Inference reads only the latest 16, together with the explicit plan context. That makes the accepted plan the durable record of commitments, rather than hoping that a model can recover them from an unlimited chat log.&lt;/p&gt;

&lt;p&gt;SQLite also protects against a less visible problem: stale updates. Every state change increments a revision. A transaction reads the current revision under &lt;code&gt;BEGIN IMMEDIATE&lt;/code&gt;, compares it with the request's revision, applies the change, and commits. If a task changes or you clear the conversation while an inference request is running, the old response cannot overwrite the newer state. The API returns a conflict instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test the behavior, then make it your own
&lt;/h2&gt;

&lt;p&gt;Run the backend checks and optional Markdown checks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;make check
node &lt;span class="nt"&gt;--test&lt;/span&gt; tests/markdown.test.cjs

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The reviewed version has 39 backend tests and four Markdown tests. They cover plan validation, saved state, completion, deletion and undo, new-day behavior, stale updates, provider failures, host validation, and rendering untrusted model text. They use mocked inference responses and do not incur model charges. The repository also includes GitHub Actions checks for Python 3.11 and 3.14.&lt;/p&gt;

&lt;p&gt;Live testing answers a different question. Try your own mix of work and personal activities, accept a plan, reduce the time available, and check that the new proposal both fits the budget and preserves unfinished commitments. Reload after completing an item to verify persistence. A passing schema test alone cannot tell you whether the advice is useful.&lt;/p&gt;

&lt;h3&gt;
  
  
  Expect a few operational details
&lt;/h3&gt;

&lt;p&gt;We observed planning requests near a minute with MiMo V2.5 Pro, including one diagnostic at 58.7 seconds. Earlier requests exceeded the original 60-second timeout. That is a development observation, not a benchmark or a latency promise. The app now defaults to a configurable 120-second response timeout and a separate 10-second connection timeout. On timeout, it keeps the saved plan and leaves your message in the composer.&lt;/p&gt;

&lt;p&gt;If a request fails, identify which layer failed:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;First thing to check&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Local server offline / Failed to fetch&lt;/td&gt;
&lt;td&gt;Is &lt;code&gt;make run&lt;/code&gt; still running? Start it, then select Reconnect.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP 401 from DigitalOcean&lt;/td&gt;
&lt;td&gt;The key may be invalid or revoked. Replace it in &lt;code&gt;.env&lt;/code&gt; and restart.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP 403&lt;/td&gt;
&lt;td&gt;Check model access and any VPC restriction on the key.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP 429&lt;/td&gt;
&lt;td&gt;Wait before retrying; check the account's rate limits.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invalid plan&lt;/td&gt;
&lt;td&gt;The model returned malformed data or broke an application rule. Clarify the request and retry.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;State conflict&lt;/td&gt;
&lt;td&gt;Another action changed the plan while the request was running. Review the latest state before resubmitting.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Reconnect reloads local state; it does not resend a potentially billable request. Assistant Markdown is rendered with a locally bundled parser, with raw HTML and remote images disabled. User messages remain plain text, and provider error bodies are not echoed into the interface.&lt;/p&gt;

&lt;p&gt;Keep this first version on localhost. It has no authentication or separation between users. A shared deployment needs those features, a deliberate host policy, and durable database storage. Notifications, calendars, and unattended actions are separate projects, not hidden capabilities of the model.&lt;/p&gt;

&lt;p&gt;A useful next experiment is a small set of scenarios you can rerun when changing the prompt or model: low energy, an urgent errand, a missed day, a completed task, and a new activity you genuinely want to make room for. Judge whether the assistant makes sensible tradeoffs as well as whether its JSON passes validation. If you want a fuller evaluation workflow, our &lt;a href="https://dev.to/devopsdaily/build-and-evaluate-an-ai-error-explainer-with-digitalocean-inference-8l"&gt;DigitalOcean Inference error-explainer guide&lt;/a&gt; walks through one.&lt;/p&gt;

&lt;p&gt;You now have a local app whose AI helps turn competing intentions into a plan you can inspect and change. Try it with a real evening, tell it when its priorities are wrong, and see whether the revised day feels more doable.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/build-personal-ai-assistant-digitalocean-inference" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>cloud</category>
      <category>digitalocean</category>
      <category>aiagents</category>
      <category>inference</category>
    </item>
    <item>
      <title>cgroup v2 memory.high Throttles, memory.max Kills: Measured on a Host Without Swap</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Fri, 06 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/cgroup-v2-memoryhigh-throttles-memorymax-kills-measured-on-a-host-without-swap-38m1</link>
      <guid>https://dev.to/devopsdaily/cgroup-v2-memoryhigh-throttles-memorymax-kills-measured-on-a-host-without-swap-38m1</guid>
      <description>&lt;p&gt;We ran one Python workload that allocates memory in 256 KiB steps, on a Linux 6.8 host with no swap, under cgroup v2 &lt;code&gt;memory.max&lt;/code&gt;, &lt;code&gt;memory.high&lt;/code&gt;, both and neither. When it asked for 320 MiB with only &lt;code&gt;memory.max&lt;/code&gt; at 256 MiB, it ran at full speed and the kernel killed it 0.28 to 0.39 seconds after it started. With only &lt;code&gt;memory.high&lt;/code&gt; at 230.4 MiB, the kernel did not kill it. It made new allocations wait instead, and after 150 seconds the workload held 11 to 12 MiB more than &lt;code&gt;memory.high&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The case to worry about is the smaller one. A version of the workload that needs 239 MiB fits under &lt;code&gt;memory.max&lt;/code&gt; and runs at full speed there. With &lt;code&gt;memory.high&lt;/code&gt; at 90% of that limit, which is what Kubernetes MemoryQoS writes for a 256Mi limit and a zero memory request with the 1.36 default factor, its median allocation took 191.8 ms on a CPU of its own and 6.2 seconds on a CPU it shared with our 100 ms sampler, against 0.26 ms with no limit. It reported no error.&lt;/p&gt;

&lt;p&gt;The kernel's throttle formula predicted the length of each sleep to within about 10%. The number of sleeps is what changed: once per 64-page batch, as the code suggests, or once per 4 KiB page. We found in review that our own sampler, on the same CPU, kept the task at once per page. Without it, the count still switched to once per page in phases, while other work ran on the host. And one condition changed both limits: when the program's own code was charged to its cgroup, the kernel reclaimed that code and read it back from disk over and over, and &lt;code&gt;memory.max&lt;/code&gt; did not kill it within the 20 seconds we allowed.&lt;/p&gt;

&lt;p&gt;This post has the numbers, the kernel formula, the method problems we found on the way, and a check of what systemd, Docker and Kubernetes write to these files.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;memory.max&lt;/code&gt; alone killed at full speed.&lt;/strong&gt; The 320 MiB workload ran at normal latency (p99 of the last 10% of operations 0.29 to 0.41 ms) and the kernel OOM-killed it 0.28 to 0.39 seconds after it started, in 3 of 3 runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;memory.high&lt;/code&gt; alone did not kill within the 150 second cap. It made new allocations wait.&lt;/strong&gt; Each sleep was 1.06 to 1.10 times the kernel's formula at the median, &lt;code&gt;64000 x o^2&lt;/code&gt; ms with &lt;code&gt;o&lt;/code&gt; the share over &lt;code&gt;memory.high&lt;/code&gt;. At the 2 s cap the median sleep was 2,047.9 ms, which fits the timer wheel's 64 ms step for a 2,000 ms timeout.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The number of sleeps decides the cost, and a neighbour on the same CPU decides the number.&lt;/strong&gt; The kernel's comment says one sleep per 64-page batch. With our sampler on the same CPU, the task slept once per 4 KiB page in 95% of the operations whose sleep was 50 ms or more. Without that neighbour (the workload on another CPU, or the sampler once a second), it switched between about 1 and 64 sleeps per batch in phases of tens of seconds, and spent 46% to 72% of that time in the slow phases.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A program that fits under &lt;code&gt;memory.max&lt;/code&gt; but not under &lt;code&gt;memory.high&lt;/code&gt; ran much slower and reported nothing.&lt;/strong&gt; With &lt;code&gt;memory.high&lt;/code&gt; at 90% of a 256 MiB limit (the kubelet's value for a zero memory request and factor 0.9), the median allocation took 191.8 ms on its own CPU and 6,216 ms next to the busy sampler (15 to 17 of its 200 churn operations in 150 s), against 0.257 ms with no limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Both limits only delayed the kill.&lt;/strong&gt; With &lt;code&gt;memory.high&lt;/code&gt; at 90% of &lt;code&gt;memory.max&lt;/code&gt;, a leak was OOM-killed after 29.1 minutes next to the busy sampler. With &lt;code&gt;memory.max&lt;/code&gt; alone the kill took 0.3 seconds. On a CPU of its own, it was not killed within the 120 second cap in 3 of 3 runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code pages in the cgroup turn both limits into thrashing.&lt;/strong&gt; When the program's own binary was charged to its cgroup, the kernel reclaimed it and read it back from disk, 2.4 to 2.6 GB in 25 seconds at our 100 MiB/s disk cap. &lt;code&gt;memory.max&lt;/code&gt; did not kill it within 20 seconds in 3 of 3 runs, and memory PSI showed only 8% while io PSI showed 85%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Page cache is not throttled or killed.&lt;/strong&gt; Neither limit killed the file-reading runs or put them to sleep. They capped the cache, so later reads went to disk more often (p99 1.0 to 2.6 ms against 0.13 ms with no limit).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The tools:&lt;/strong&gt; systemd writes &lt;code&gt;MemoryHigh=&lt;/code&gt; and &lt;code&gt;MemoryMax=&lt;/code&gt; to &lt;code&gt;memory.high&lt;/code&gt; and &lt;code&gt;memory.max&lt;/code&gt; as given. Docker has no flag for &lt;code&gt;memory.high&lt;/code&gt;, and on cgroup v2 &lt;code&gt;--memory-reservation&lt;/code&gt; becomes &lt;code&gt;memory.low&lt;/code&gt;. Kubernetes MemoryQoS writes &lt;code&gt;memory.high&lt;/code&gt;; it is alpha and off in 1.36, and beta and on in 1.37 but with no throttling factor set, so no &lt;code&gt;memory.high&lt;/code&gt; unless you set one.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux machine with cgroup v2 (&lt;code&gt;stat -fc %T /sys/fs/cgroup&lt;/code&gt; prints &lt;code&gt;cgroup2fs&lt;/code&gt;) and systemd&lt;/li&gt;
&lt;li&gt;Root, or a cgroup subtree delegated to your user&lt;/li&gt;
&lt;li&gt;Python 3 for the workload and the analysis scripts&lt;/li&gt;
&lt;li&gt;Check for swap first: &lt;code&gt;swapon --show&lt;/code&gt;. Our results are for a host with no swap. With swap, the kernel can move anonymous memory out, and both limits behave differently&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Two limits, two code paths
&lt;/h2&gt;

&lt;p&gt;Both files live in a cgroup v2 directory. Both act when the kernel charges a new page to the cgroup. They do different things when the cgroup is over its value.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;memory.max&lt;/code&gt;&lt;/strong&gt; is the hard limit. The kernel docs for 6.8 say: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup." The charge path tries to reclaim first, retries, and calls the OOM killer only when reclaim makes no progress.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;memory.high&lt;/code&gt;&lt;/strong&gt; is the throttle. The same docs say: "Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached." The charge always succeeds. On the way back to user space, &lt;code&gt;mem_cgroup_handle_over_high()&lt;/code&gt; tries to reclaim, and if usage is still over the value it puts the task to sleep.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The sleep length comes from &lt;code&gt;calculate_high_delay()&lt;/code&gt; in &lt;code&gt;mm/memcontrol.c&lt;/code&gt;. On the Ubuntu 6.8.0-147 kernel we ran, with &lt;code&gt;HZ=1000&lt;/code&gt;, it works out to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;o = (usage - memory.high) / memory.high
penalty = 64000 * o^2 ms per charge batch of 64 pages (MEMCG_CHARGE_BATCH)
          no sleep if penalty &amp;lt;= 10 ms; at most 2000 ms (MEMCG_MAX_HIGH_DELAY_JIFFIES = 2 * HZ)

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So the sleep is 0 until usage is 1.25% over &lt;code&gt;memory.high&lt;/code&gt;, 160 ms at 5% over, 640 ms at 10% over, and it hits the 2 s cap at 17.7% over. The comment above the cap reads "Clamp the maximum sleep time per allocation batch to 2 seconds." Each &lt;code&gt;reclaim_high()&lt;/code&gt; call adds 1 to the &lt;code&gt;high&lt;/code&gt; counter in &lt;code&gt;memory.events&lt;/code&gt;. When reclaim frees nothing, the kernel calls it 17 times before it sleeps (once, then &lt;code&gt;MAX_RECLAIM_RETRIES&lt;/code&gt; = 16 retries). When reclaim frees something, it keeps retrying and does not sleep. So in a cgroup with nothing to reclaim, the counter rises by 17 per sleep, and that is how we count sleeps from outside the kernel.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;page fault&lt;/strong&gt; charge to the cgroup&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;over memory.max&lt;/strong&gt; before the charge&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;over memory.high&lt;/strong&gt; after the charge&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reclaim&lt;/strong&gt; direct, in the fault&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reclaim_high x17&lt;/strong&gt; +17 to memory.events high&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OOM kill&lt;/strong&gt; nothing reclaimable&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;sleep&lt;/strong&gt; 64000 x o^2 ms, at most 2 s&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;thrash&lt;/strong&gt; reclaim frees the task's own code&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Connections:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;page fault -&amp;gt; over memory.max&lt;/li&gt;
&lt;li&gt;page fault -&amp;gt; over memory.high&lt;/li&gt;
&lt;li&gt;over memory.max -&amp;gt; reclaim&lt;/li&gt;
&lt;li&gt;over memory.high -&amp;gt; reclaim_high x17&lt;/li&gt;
&lt;li&gt;reclaim -&amp;gt; OOM kill (freed nothing)&lt;/li&gt;
&lt;li&gt;reclaim -&amp;gt; thrash (freed file pages)&lt;/li&gt;
&lt;li&gt;reclaim_high x17 -&amp;gt; sleep (freed nothing)&lt;/li&gt;
&lt;li&gt;reclaim_high x17 -&amp;gt; thrash (freed file pages)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Without swap, the kernel has nowhere to put anonymous memory: heaps, stacks, anything a process allocated and wrote. In our runs that memory was never reclaimed (&lt;code&gt;pgsteal&lt;/code&gt; stayed 0 in every warm anonymous run). What reclaim could take were file pages: page cache and the mapped code of the program in the cgroup. That decides most of what follows.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lab
&lt;/h2&gt;

&lt;p&gt;One host: Ubuntu 24.04, kernel 6.8.0-147-generic, systemd 255, 4 vCPU, 7.75 GiB of RAM, no swap device and zswap off, &lt;code&gt;CONFIG_HZ=1000&lt;/code&gt;, transparent huge pages on &lt;code&gt;madvise&lt;/code&gt; (&lt;code&gt;results/00-host.txt&lt;/code&gt;). It is a shared development machine. So every run happened inside one transient systemd scope with &lt;code&gt;Delegate=yes&lt;/code&gt; and &lt;code&gt;MemoryMax=1G&lt;/code&gt;, every run got its own child cgroup, and no limit went above 1 GiB.&lt;/p&gt;

&lt;p&gt;The workload is one script, &lt;code&gt;scripts/lib/workload.py&lt;/code&gt;. Each operation maps a 256 KiB private anonymous chunk and writes all of it. That is 64 pages, which is also the kernel's charge batch. Each operation also copies one random chunk it already holds, so the working set stays in use. The phases:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Grow&lt;/strong&gt; until the workload holds its target.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Churn&lt;/strong&gt; 200 times: free one random chunk and allocate a new one, so usage stays flat while the program keeps allocating, like a busy service.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The random choices are seeded (mulberry32). Before and after every operation, the workload reads &lt;code&gt;memory.current&lt;/code&gt; and the &lt;code&gt;high&lt;/code&gt; and &lt;code&gt;max&lt;/code&gt; counters in &lt;code&gt;memory.events&lt;/code&gt; from open file descriptors, and writes one JSON line with the latency. A driver outside the cgroup samples &lt;code&gt;memory.current&lt;/code&gt;, &lt;code&gt;memory.events&lt;/code&gt;, &lt;code&gt;memory.pressure&lt;/code&gt;, &lt;code&gt;io.pressure&lt;/code&gt;, &lt;code&gt;memory.stat&lt;/code&gt;, &lt;code&gt;cpu.stat&lt;/code&gt; and each process's &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/wchan&lt;/code&gt; and &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/stat&lt;/code&gt; every 100 ms. The driver pinned itself to CPU 0, and the workload inherited that, so in the main matrix the two shared one CPU. We only saw that in review, and it changed one of the results a lot. The section on how many sleeps a task pays has the runs that measure it.&lt;/p&gt;

&lt;p&gt;The two limits are fixed for every run:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;memory.max&lt;/code&gt; = 268435456 bytes (256 MiB)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;memory.high&lt;/code&gt; = 241590272 bytes (230.4 MiB). That is the value the kubelet computes with MemoryQoS for &lt;code&gt;limits.memory: 256Mi&lt;/code&gt;, &lt;code&gt;requests.memory: 0&lt;/code&gt; and the 1.36 default throttling factor of 0.9. It is the lowest &lt;code&gt;memory.high&lt;/code&gt; that formula gives for a 256Mi limit. More on that below.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Workload&lt;/th&gt;
&lt;th&gt;What it shows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-fit-*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;grows to 232 MiB (about 239 MiB in &lt;code&gt;memory.current&lt;/code&gt;), then churns&lt;/td&gt;
&lt;td&gt;a program that fits under &lt;code&gt;memory.max&lt;/code&gt; but not under &lt;code&gt;memory.high&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-over-*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;asks for 320 MiB&lt;/td&gt;
&lt;td&gt;a program that needs more than the limit, like a leak&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-over-4p-high&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the same 320 MiB split over 4 processes&lt;/td&gt;
&lt;td&gt;whether the throttle is per process&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-fit-high-populate&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;mmap&lt;/code&gt; with &lt;code&gt;MAP_POPULATE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;the same bytes, charged in one system call per chunk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-clamp-high&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;grows to 300 MiB, then sets its own &lt;code&gt;memory.high&lt;/code&gt; so its usage is 33% over it&lt;/td&gt;
&lt;td&gt;the 2 s cap&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;*-owncode&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a fresh copy of the interpreter binary&lt;/td&gt;
&lt;td&gt;a program whose code pages are charged to its own cgroup&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;file-over-*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;reads 320 MiB of a 512 MiB file&lt;/td&gt;
&lt;td&gt;page cache instead of anonymous memory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-over-both-long&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;anon-over&lt;/code&gt; with both limits, 45 minute cap&lt;/td&gt;
&lt;td&gt;how long until the OOM kill&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each of &lt;code&gt;none&lt;/code&gt;, &lt;code&gt;max&lt;/code&gt;, &lt;code&gt;high&lt;/code&gt; and &lt;code&gt;both&lt;/code&gt; ran 3 times, in a fixed order per repetition, with a 150 second cap. The &lt;code&gt;owncode&lt;/code&gt; runs thrashed the disk in the smoke tests, so we capped them at 20 seconds. We also capped every run's disk reads and writes at 100 MiB/s with &lt;code&gt;io.max&lt;/code&gt; on its own cgroup, so that a thrashing run would not take the shared disk from other work. One run, &lt;code&gt;anon-over-max-owncode-fastdisk&lt;/code&gt;, keeps the uncapped disk.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/The-DevOps-Daily/cgroup-memory-high-vs-max" rel="noopener noreferrer"&gt;The-DevOps-Daily/cgroup-memory-high-vs-max on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We wrote the claim and what would disprove it before the first run and committed it first (&lt;code&gt;CLAIM.md&lt;/code&gt;). The smoke runs then changed the design twice, and we added those changes to the same file before the main matrix. Every run directory carries a hash of the run code. The summary script refuses a batch with mixed hashes, or a hash that matches no commit in the repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  memory.max: full speed, then the kill
&lt;/h2&gt;

&lt;p&gt;With only &lt;code&gt;memory.max&lt;/code&gt;, the workload never slowed down. In the three &lt;code&gt;anon-over-max&lt;/code&gt; runs the median operation took 0.20 to 0.27 ms, the p99 of the last 10% of operations before the kill was 0.29 to 0.41 ms, and the kernel killed the process 0.28 to 0.39 seconds after it started (&lt;code&gt;results/summary.md&lt;/code&gt;). In each run &lt;code&gt;memory.events&lt;/code&gt; counted 35 times that a charge hit the limit, then one OOM kill. The kernel log names the cgroup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ddmemhigh-20261105T112606Z.scope,mems_allowed=0,oom_memcg=/ddmemhigh.slice/ddmemhigh-20261105T112606Z.scope/anon-over-max-r1,task_memcg=/ddmemhigh.slice/ddmemhigh-20261105T112606Z.scope/anon-over-max-r1,task=python3.12,pid=142949,uid=0
Memory cgroup out of memory: Killed process 142949 (python3.12) total-vm:276772kB, anon-rss:261504kB, file-rss:7296kB, shmem-rss:0kB, UID:0 pgtables:580kB oom_score_adj:0

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing in the cgroup could be reclaimed. The cgroup held only the program's anonymous pages, and without swap the kernel cannot free those. So reclaim found nothing, the charge path ran out of retries, and the OOM killer ran. Memory pressure stayed at 0: the task never waited for memory. It was killed.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;anon-fit-max&lt;/code&gt; runs are the other half of the same result. That workload needs 239.2 MiB, which fits under 256 MiB. It finished all 200 churn operations with a median of 0.277 ms per churn operation and a &lt;code&gt;max&lt;/code&gt; counter of 0. With no limit, the same median was 0.257 ms. If a program fits under &lt;code&gt;memory.max&lt;/code&gt;, the limit costs nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  memory.high: slower and slower, not killed
&lt;/h2&gt;

&lt;p&gt;With only &lt;code&gt;memory.high&lt;/code&gt;, the same 320 MiB workload was not killed in any of the 3 runs within the 150 second cap. The kernel docs say &lt;code&gt;memory.high&lt;/code&gt; never invokes the OOM killer, and our driver stopped each run at the cap. Once it was a few MiB over the limit, it moved at the speed of the kernel's sleep timer, not at the speed of a program:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Latency of one 256 KiB allocation vs memory.current&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;226&lt;/th&gt;
&lt;th&gt;227&lt;/th&gt;
&lt;th&gt;228&lt;/th&gt;
&lt;th&gt;229&lt;/th&gt;
&lt;th&gt;230&lt;/th&gt;
&lt;th&gt;231&lt;/th&gt;
&lt;th&gt;232&lt;/th&gt;
&lt;th&gt;233&lt;/th&gt;
&lt;th&gt;234&lt;/th&gt;
&lt;th&gt;235&lt;/th&gt;
&lt;th&gt;236&lt;/th&gt;
&lt;th&gt;237&lt;/th&gt;
&lt;th&gt;238&lt;/th&gt;
&lt;th&gt;239&lt;/th&gt;
&lt;th&gt;240&lt;/th&gt;
&lt;th&gt;241&lt;/th&gt;
&lt;th&gt;242&lt;/th&gt;
&lt;th&gt;243&lt;/th&gt;
&lt;th&gt;244&lt;/th&gt;
&lt;th&gt;245&lt;/th&gt;
&lt;th&gt;246&lt;/th&gt;
&lt;th&gt;247&lt;/th&gt;
&lt;th&gt;248&lt;/th&gt;
&lt;th&gt;249&lt;/th&gt;
&lt;th&gt;250&lt;/th&gt;
&lt;th&gt;251&lt;/th&gt;
&lt;th&gt;252&lt;/th&gt;
&lt;th&gt;253&lt;/th&gt;
&lt;th&gt;254&lt;/th&gt;
&lt;th&gt;255&lt;/th&gt;
&lt;th&gt;256&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;no limit&lt;/td&gt;
&lt;td&gt;0.248ms&lt;/td&gt;
&lt;td&gt;0.252ms&lt;/td&gt;
&lt;td&gt;0.249ms&lt;/td&gt;
&lt;td&gt;0.251ms&lt;/td&gt;
&lt;td&gt;0.246ms&lt;/td&gt;
&lt;td&gt;0.253ms&lt;/td&gt;
&lt;td&gt;0.242ms&lt;/td&gt;
&lt;td&gt;0.26ms&lt;/td&gt;
&lt;td&gt;0.24ms&lt;/td&gt;
&lt;td&gt;0.252ms&lt;/td&gt;
&lt;td&gt;0.25ms&lt;/td&gt;
&lt;td&gt;0.256ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.24ms&lt;/td&gt;
&lt;td&gt;0.213ms&lt;/td&gt;
&lt;td&gt;0.226ms&lt;/td&gt;
&lt;td&gt;0.238ms&lt;/td&gt;
&lt;td&gt;0.242ms&lt;/td&gt;
&lt;td&gt;0.235ms&lt;/td&gt;
&lt;td&gt;0.215ms&lt;/td&gt;
&lt;td&gt;0.223ms&lt;/td&gt;
&lt;td&gt;0.224ms&lt;/td&gt;
&lt;td&gt;0.225ms&lt;/td&gt;
&lt;td&gt;0.214ms&lt;/td&gt;
&lt;td&gt;0.253ms&lt;/td&gt;
&lt;td&gt;0.234ms&lt;/td&gt;
&lt;td&gt;0.235ms&lt;/td&gt;
&lt;td&gt;0.215ms&lt;/td&gt;
&lt;td&gt;0.239ms&lt;/td&gt;
&lt;td&gt;0.218ms&lt;/td&gt;
&lt;td&gt;0.222ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;memory.max only&lt;/td&gt;
&lt;td&gt;0.236ms&lt;/td&gt;
&lt;td&gt;0.224ms&lt;/td&gt;
&lt;td&gt;0.242ms&lt;/td&gt;
&lt;td&gt;0.195ms&lt;/td&gt;
&lt;td&gt;0.242ms&lt;/td&gt;
&lt;td&gt;0.223ms&lt;/td&gt;
&lt;td&gt;0.238ms&lt;/td&gt;
&lt;td&gt;0.221ms&lt;/td&gt;
&lt;td&gt;0.211ms&lt;/td&gt;
&lt;td&gt;0.216ms&lt;/td&gt;
&lt;td&gt;0.224ms&lt;/td&gt;
&lt;td&gt;0.203ms&lt;/td&gt;
&lt;td&gt;0.215ms&lt;/td&gt;
&lt;td&gt;0.199ms&lt;/td&gt;
&lt;td&gt;0.207ms&lt;/td&gt;
&lt;td&gt;0.19ms&lt;/td&gt;
&lt;td&gt;0.202ms&lt;/td&gt;
&lt;td&gt;0.204ms&lt;/td&gt;
&lt;td&gt;0.246ms&lt;/td&gt;
&lt;td&gt;0.199ms&lt;/td&gt;
&lt;td&gt;0.233ms&lt;/td&gt;
&lt;td&gt;0.232ms&lt;/td&gt;
&lt;td&gt;0.191ms&lt;/td&gt;
&lt;td&gt;0.209ms&lt;/td&gt;
&lt;td&gt;0.229ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.216ms&lt;/td&gt;
&lt;td&gt;0.225ms&lt;/td&gt;
&lt;td&gt;0.233ms&lt;/td&gt;
&lt;td&gt;0.238ms&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;memory.high only, sampler on the same CPU&lt;/td&gt;
&lt;td&gt;0.226ms&lt;/td&gt;
&lt;td&gt;0.216ms&lt;/td&gt;
&lt;td&gt;0.2ms&lt;/td&gt;
&lt;td&gt;0.206ms&lt;/td&gt;
&lt;td&gt;0.235ms&lt;/td&gt;
&lt;td&gt;0.228ms&lt;/td&gt;
&lt;td&gt;0.22ms&lt;/td&gt;
&lt;td&gt;25.621ms&lt;/td&gt;
&lt;td&gt;39.782ms&lt;/td&gt;
&lt;td&gt;209.811ms&lt;/td&gt;
&lt;td&gt;1525.021ms&lt;/td&gt;
&lt;td&gt;4583.747ms&lt;/td&gt;
&lt;td&gt;5839.866ms&lt;/td&gt;
&lt;td&gt;7303.861ms&lt;/td&gt;
&lt;td&gt;8839.941ms&lt;/td&gt;
&lt;td&gt;10295.852ms&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;memory.high only, own CPU, sampler at 1 s&lt;/td&gt;
&lt;td&gt;0.22ms&lt;/td&gt;
&lt;td&gt;0.259ms&lt;/td&gt;
&lt;td&gt;0.258ms&lt;/td&gt;
&lt;td&gt;0.272ms&lt;/td&gt;
&lt;td&gt;0.248ms&lt;/td&gt;
&lt;td&gt;0.27ms&lt;/td&gt;
&lt;td&gt;0.262ms&lt;/td&gt;
&lt;td&gt;14.873ms&lt;/td&gt;
&lt;td&gt;106.819ms&lt;/td&gt;
&lt;td&gt;519.352ms&lt;/td&gt;
&lt;td&gt;52.8ms&lt;/td&gt;
&lt;td&gt;71.752ms&lt;/td&gt;
&lt;td&gt;95.774ms&lt;/td&gt;
&lt;td&gt;119.723ms&lt;/td&gt;
&lt;td&gt;135.811ms&lt;/td&gt;
&lt;td&gt;171.815ms&lt;/td&gt;
&lt;td&gt;191.793ms&lt;/td&gt;
&lt;td&gt;323.75ms&lt;/td&gt;
&lt;td&gt;383.851ms&lt;/td&gt;
&lt;td&gt;851.785ms&lt;/td&gt;
&lt;td&gt;11251.414ms&lt;/td&gt;
&lt;td&gt;379.811ms&lt;/td&gt;
&lt;td&gt;1847.768ms&lt;/td&gt;
&lt;td&gt;1759.826ms&lt;/td&gt;
&lt;td&gt;1135.82ms&lt;/td&gt;
&lt;td&gt;1727.849ms&lt;/td&gt;
&lt;td&gt;3199.782ms&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;kernel formula, 1 sleep per 64 pages&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;11.243ms&lt;/td&gt;
&lt;td&gt;20.243ms&lt;/td&gt;
&lt;td&gt;31.243ms&lt;/td&gt;
&lt;td&gt;44.243ms&lt;/td&gt;
&lt;td&gt;60.243ms&lt;/td&gt;
&lt;td&gt;79.243ms&lt;/td&gt;
&lt;td&gt;99.243ms&lt;/td&gt;
&lt;td&gt;123.243ms&lt;/td&gt;
&lt;td&gt;148.243ms&lt;/td&gt;
&lt;td&gt;176.243ms&lt;/td&gt;
&lt;td&gt;206.243ms&lt;/td&gt;
&lt;td&gt;239.243ms&lt;/td&gt;
&lt;td&gt;274.243ms&lt;/td&gt;
&lt;td&gt;312.243ms&lt;/td&gt;
&lt;td&gt;352.243ms&lt;/td&gt;
&lt;td&gt;395.243ms&lt;/td&gt;
&lt;td&gt;439.243ms&lt;/td&gt;
&lt;td&gt;487.243ms&lt;/td&gt;
&lt;td&gt;536.243ms&lt;/td&gt;
&lt;td&gt;588.243ms&lt;/td&gt;
&lt;td&gt;643.243ms&lt;/td&gt;
&lt;td&gt;700.243ms&lt;/td&gt;
&lt;td&gt;759.243ms&lt;/td&gt;
&lt;td&gt;821.243ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;kernel formula, 1 sleep per page&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;0.243ms&lt;/td&gt;
&lt;td&gt;704.243ms&lt;/td&gt;
&lt;td&gt;1280.243ms&lt;/td&gt;
&lt;td&gt;1984.243ms&lt;/td&gt;
&lt;td&gt;2816.243ms&lt;/td&gt;
&lt;td&gt;3840.243ms&lt;/td&gt;
&lt;td&gt;5056.243ms&lt;/td&gt;
&lt;td&gt;6336.243ms&lt;/td&gt;
&lt;td&gt;7872.243ms&lt;/td&gt;
&lt;td&gt;9472.243ms&lt;/td&gt;
&lt;td&gt;11264.243ms&lt;/td&gt;
&lt;td&gt;13184.243ms&lt;/td&gt;
&lt;td&gt;15296.243ms&lt;/td&gt;
&lt;td&gt;17536.243ms&lt;/td&gt;
&lt;td&gt;19968.243ms&lt;/td&gt;
&lt;td&gt;22528.243ms&lt;/td&gt;
&lt;td&gt;25280.243ms&lt;/td&gt;
&lt;td&gt;28096.243ms&lt;/td&gt;
&lt;td&gt;31168.243ms&lt;/td&gt;
&lt;td&gt;34304.243ms&lt;/td&gt;
&lt;td&gt;37632.243ms&lt;/td&gt;
&lt;td&gt;41152.243ms&lt;/td&gt;
&lt;td&gt;44800.243ms&lt;/td&gt;
&lt;td&gt;48576.243ms&lt;/td&gt;
&lt;td&gt;52544.243ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;1 ms: 1ms&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;x axis: memory.current in MiB when the op started (memory.high = 230.4 MiB, memory.max = 256 MiB). Median per 1 MiB over 3 repetitions, warm code, Ubuntu 6.8.0-147, no swap. 'Sampler on the same CPU' is the main matrix (anon-over-high: the workload inherited the driver's CPU 0, sampled every 100 ms); 'own CPU' is diag-over-high-cpu2-1s (workload on CPU 2, sampled every 1 s). memory.max only stays flat up to 256 MiB, then the OOM kill. The formula lines add the no-limit median to the kernel's sleep: once per 64-page op, or once per 4 KiB page. Source: results/runs/anon-over-&lt;/em&gt;/ops.jsonl, results/diag/diag-over-high-cpu2-1s-&lt;em&gt;/ops.jsonl, scripts/50-charts.py.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In the matrix runs (the orange line), up to 232 MiB an operation took about a quarter of a millisecond, the same as with no limit. Within the next 5 MiB the median went from 0.22 ms to 4.6 seconds, and at 241 MiB it was 10.3 seconds per 256 KiB. After 150 seconds &lt;code&gt;memory.current&lt;/code&gt; was 241.5 to 242.2 MiB, 11 to 12 MiB over &lt;code&gt;memory.high&lt;/code&gt;, and still growing by 25 to 30 KiB per second. The process used 0.59 to 0.72 seconds of CPU in those 150 seconds. In 98% to 99% of the samples it was in state &lt;code&gt;D&lt;/code&gt;, in &lt;code&gt;mem_cgroup_handle_over_high&lt;/code&gt;. The purple line is the same workload on a CPU of its own. It stays near the one-sleep-per-batch line for most of the range, with jumps when it falls into a slow phase. The section on how many sleeps a task pays explains both lines.&lt;/p&gt;

&lt;p&gt;Here is the same workload through systemd's &lt;code&gt;MemoryMax=&lt;/code&gt; and &lt;code&gt;MemoryHigh=&lt;/code&gt;, without our driver. The progress line shows the slowest operation in each second:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;results/25-demo.txt (no driver, no CPU pinning)&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# commands as scripts/25-demo.sh prints them (it also passes --quiet --slice=ddmemhigh.slice); 320 MiB target
$ systemd-run --scope -p MemoryMax=256M python3 -I scripts/lib/workload.py --kind anon --target-mib 320 --progress
pid 173416 in /sys/fs/cgroup/ddmemhigh.slice/run-r765e1a07143e4aa1be21a9eb2292aaff.scope, memory.high=max memory.max=268435456
   0.0s held 0 MiB memory.current 7.0 MiB ops 1 slowest op 0.33 ms
scripts/25-demo.sh: line 17: 173416 Killed systemd-run --scope --quiet --slice=ddmemhigh.slice -p MemoryMax=256M $w
exit status 137
$ systemd-run --scope -p MemoryHigh=241590272 -p MemoryMax=256M python3 -I scripts/lib/workload.py --kind anon --target-mib 320 --progress --max-seconds 30
pid 173417 in /sys/fs/cgroup/ddmemhigh.slice/run-rbd28bd3d399c47a9ab104c3b6d4b6eaf.scope, memory.high=241590272 memory.max=268435456
   0.0s held 0 MiB memory.current 6.7 MiB ops 1 slowest op 0.30 ms
   1.0s held 231 MiB memory.current 238.2 MiB ops 924 slowest op 79.86 ms
   7.1s held 231 MiB memory.current 238.5 MiB ops 926 slowest op 5488.05 ms
  12.8s held 231 MiB memory.current 238.8 MiB ops 927 slowest op 5759.67 ms
  19.0s held 232 MiB memory.current 239.0 MiB ops 928 slowest op 6135.79 ms
  25.5s held 232 MiB memory.current 239.3 MiB ops 929 slowest op 6511.84 ms
  27.5s held 232 MiB memory.current 239.8 MiB ops 930 slowest op 1992.06 ms
  28.5s held 234 MiB memory.current 241.1 MiB ops 936 slowest op 359.79 ms
  29.6s held 234 MiB memory.current 241.9 MiB ops 939 slowest op 759.73 ms
end: time after 30.4s, 940 ops, held 235 MiB, memory.current 242.1 MiB
exit status 0

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first process was killed before its second progress line, so in less than a second, with exit status 137. The second got 11.7 MiB past &lt;code&gt;memory.high&lt;/code&gt; in 30 seconds, and its operations took 5.5 to 6.5 seconds each until they sped up near the end. The next section explains that change.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;anon-fit&lt;/code&gt; runs matter more, because nothing in them is wrong. The program needs 239 MiB, fits under its 256 MiB &lt;code&gt;memory.max&lt;/code&gt;, and runs at full speed there. With &lt;code&gt;memory.high&lt;/code&gt; at 230.4 MiB added, it held about 239 MiB and every churn operation slept. In the matrix, where it shared its CPU with our sampler, the median churn operation took 6,216 ms instead of 0.257 ms, and it finished 15 to 17 of its 200 churn operations before the 150 second cap. That is about 24,000 times slower per operation, with the same memory use and no error anywhere. On a CPU of its own it was about 750 times slower. The next section explains the difference. &lt;code&gt;memory.max&lt;/code&gt; was never reached and its counter stayed at 0.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many sleeps: it depends on who else runs on the CPU
&lt;/h2&gt;

&lt;p&gt;To compare with the kernel, we count sleeps. In the warm runs reclaim never freed anything, so each sleep raised the &lt;code&gt;high&lt;/code&gt; counter by 17, and an operation's sleep count is its rise divided by 17. We skip operations whose rise is not a multiple of 17 (11 to 13 per run, all at the edge of the limit, where the formula gives 10 ms or less and the kernel does not sleep). Then the measured latency divided by (sleeps x formula) should be 1.&lt;/p&gt;

&lt;p&gt;It was 1.06 to 1.10 at the median in every single-process matrix run, and 97% to 100% of the throttled operations were within 30% of the formula (&lt;code&gt;results/predict.md&lt;/code&gt;, section 1). The formula is right about how long each sleep is. The section on the 2 second cap shows where part of the extra 6% to 10% comes from.&lt;/p&gt;

&lt;p&gt;The number of sleeps was the surprise. The kernel charges memory in batches of 64 pages and keeps the rest of a batch as a per-CPU precharge (&lt;code&gt;consume_stock()&lt;/code&gt;), so the code comment on the cap says "per allocation batch". A 64-page operation should sleep about once. In the matrix, once the sleeps were 50 ms or longer, almost every operation slept 64 times, once per page:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Formula sleep length&lt;/th&gt;
&lt;th&gt;Operations&lt;/th&gt;
&lt;th&gt;Median sleeps per 64 pages&lt;/th&gt;
&lt;th&gt;Operations with 64 or more&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;11 to 25 ms&lt;/td&gt;
&lt;td&gt;83&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;24%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;25 to 50 ms&lt;/td&gt;
&lt;td&gt;95&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50 to 100 ms&lt;/td&gt;
&lt;td&gt;228&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;92%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100 to 200 ms&lt;/td&gt;
&lt;td&gt;73&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;200 to 400 ms&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;400 to 2000 ms&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A new charge, and so a new sleep, only happens when the precharge is empty. &lt;code&gt;__refill_stock()&lt;/code&gt; empties a CPU's precharge when a different cgroup charges memory on that CPU. In review we found a cgroup that did exactly that in every run: our own driver. It pinned itself to CPU 0 before it started the workload, so every workload in the matrix inherited CPU 0 (the &lt;code&gt;cpu&lt;/code&gt; field of every operation says 0). The driver woke up every 100 ms to sample and parsed the files into Python objects, which charges memory to the driver's own cgroup on that CPU. We did not trace each drain, but the runs below show the effect.&lt;/p&gt;

&lt;p&gt;So we ran the 320 MiB &lt;code&gt;memory.high&lt;/code&gt; workload again, 3 times each, with the workload on CPU 0 or CPU 2 and the sampler every 100 ms or every second (&lt;code&gt;results/diag&lt;/code&gt;, 120 second cap):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workload CPU&lt;/th&gt;
&lt;th&gt;Sampler&lt;/th&gt;
&lt;th&gt;Median sleeps per op (formula sleep 50 ms or more)&lt;/th&gt;
&lt;th&gt;Share of that time in ops with 32 or more sleeps&lt;/th&gt;
&lt;th&gt;
&lt;code&gt;memory.current&lt;/code&gt; at the cap&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0, with the sampler&lt;/td&gt;
&lt;td&gt;every 100 ms&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;241.3 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0, with the sampler&lt;/td&gt;
&lt;td&gt;every 1 s&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;248.8 to 249.8 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2, alone&lt;/td&gt;
&lt;td&gt;every 100 ms&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;53%&lt;/td&gt;
&lt;td&gt;250.0 to 252.0 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2, alone&lt;/td&gt;
&lt;td&gt;every 1 s&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;72%&lt;/td&gt;
&lt;td&gt;243.5 to 252.4 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;With the sampler on the same CPU every 100 ms, the task slept once per page for the whole run, as in the matrix: a median of 64 sleeps per operation, and 100% of the throttled time in operations with 32 or more sleeps. In the other three settings the median operation slept once or twice. But within each run the count switched between phases of about 1 sleep per operation and phases of 64, each lasting tens of seconds, and the slow phases still took 46% to 72% of the throttled time (&lt;code&gt;results/predict.md&lt;/code&gt;, section 1c). That is consistent with other memory activity on the host emptying the precharge in those phases, but we did not trace it. The demo session further down ran with no driver and no pinning at all, and it also paid 5.5 to 6.5 seconds per 256 KiB operation for 25 seconds before it sped up.&lt;/p&gt;

&lt;p&gt;So our sampler was not the only cause of the per-page cost. With it on the same CPU, the cost was constant. Read the matrix numbers as a task whose CPU also runs another cgroup that allocates memory every 100 ms. On its own CPU, the same &lt;code&gt;anon-fit&lt;/code&gt; workload with &lt;code&gt;memory.high&lt;/code&gt; took a median of 191.8 ms per churn operation in all 3 runs (&lt;code&gt;diag-fit-high-cpu2-1s&lt;/code&gt;), and finished all 200 churn operations in 2 of the 3 runs within 120 seconds. That is about 750 times the no-limit median, instead of about 24,000 times.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;MAP_POPULATE&lt;/code&gt; is a second control. It faults in the whole chunk inside the &lt;code&gt;mmap()&lt;/code&gt; call, so the task returns to user space once per chunk. On CPU 0 with the 100 ms sampler, it slept 2 times in 592 of 600 churn operations and 3 times in the other 8, a median churn operation took 191.8 ms, and all 200 churn operations were done in about 41 seconds (&lt;code&gt;anon-fit-high-populate&lt;/code&gt;). How a program touches its memory changed the cost of the same &lt;code&gt;memory.high&lt;/code&gt; by a factor of 32 here, with the same neighbour on the CPU.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 2 second cap
&lt;/h2&gt;

&lt;p&gt;None of the runs above got far enough over &lt;code&gt;memory.high&lt;/code&gt; to reach the cap, because the throttle slowed them first. So &lt;code&gt;anon-clamp-high&lt;/code&gt; gets there another way: it grows to 300 MiB with no limit, then writes 241590272 to its own &lt;code&gt;memory.high&lt;/code&gt;. Its usage of 307.2 MiB is then 33.3% over the limit. The formula then asks for a sleep of about 7,100 ms, and the kernel caps it. Each churn operation then allocates one 4 KiB page.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Over &lt;code&gt;memory.high&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Formula without the cap&lt;/th&gt;
&lt;th&gt;Sleeps per operation&lt;/th&gt;
&lt;th&gt;Time per sleep, median&lt;/th&gt;
&lt;th&gt;min&lt;/th&gt;
&lt;th&gt;max&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-clamp-high-r1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;33.3%&lt;/td&gt;
&lt;td&gt;7,106 ms&lt;/td&gt;
&lt;td&gt;1, 2&lt;/td&gt;
&lt;td&gt;2,047.9 ms&lt;/td&gt;
&lt;td&gt;2,047.3 ms&lt;/td&gt;
&lt;td&gt;2,064.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-clamp-high-r2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;33.3%&lt;/td&gt;
&lt;td&gt;7,108 ms&lt;/td&gt;
&lt;td&gt;1, 2&lt;/td&gt;
&lt;td&gt;2,047.9 ms&lt;/td&gt;
&lt;td&gt;2,021.8 ms&lt;/td&gt;
&lt;td&gt;2,048.0 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;anon-clamp-high-r3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;33.3%&lt;/td&gt;
&lt;td&gt;7,107 ms&lt;/td&gt;
&lt;td&gt;1, 2&lt;/td&gt;
&lt;td&gt;2,047.9 ms&lt;/td&gt;
&lt;td&gt;2,011.7 ms&lt;/td&gt;
&lt;td&gt;2,047.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The median sleep took 2,047.9 ms in all three runs, not 2,000 ms. The cap is 2000 jiffies, and at &lt;code&gt;HZ=1000&lt;/code&gt; the kernel's timer wheel keeps timeouts from 512 ms to 4,095 ms in 64 ms steps (&lt;code&gt;kernel/time/timer.c&lt;/code&gt;), so a 2,000 ms timeout fires at one of those steps. The median of 2,047.9 ms fits that. Single sleeps ranged from 2,011.7 to 2,064.9 ms, depending on where in a step the timer started. The same rounding, at 8 ms steps from 64 to 511 ms, explains part of the 6% to 10% we saw on the shorter sleeps: with half a step added, the median ratio for sleeps of 64 ms or more drops to 1.030 to 1.055 in the matrix runs. Under 64 ms, where the step is 1 ms, it stays at 1.054 to 1.091, and we did not find the rest (&lt;code&gt;results/predict.md&lt;/code&gt;, section 1).&lt;/p&gt;

&lt;p&gt;So the cap holds, but it is per sleep, not per allocation. In these runs each 4 KiB operation slept twice after the first one, so it took 4,096 ms. A 256 KiB allocation that sleeps once per page at the cap would take more than two minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Both limits: high at 90% of max
&lt;/h2&gt;

&lt;p&gt;systemd's man page recommends both: "use &lt;code&gt;MemoryHigh=&lt;/code&gt; as the main control mechanism and use &lt;code&gt;MemoryMax=&lt;/code&gt; as the last line of defense". The idea is that the program slows down before it dies. The &lt;code&gt;anon-over-both&lt;/code&gt; runs show what that looks like with our 230.4 MiB and 256 MiB:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;memory.current from 10 s to 150 s, the workload asks for 320 MiB&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;10s&lt;/th&gt;
&lt;th&gt;20s&lt;/th&gt;
&lt;th&gt;30s&lt;/th&gt;
&lt;th&gt;40s&lt;/th&gt;
&lt;th&gt;50s&lt;/th&gt;
&lt;th&gt;60s&lt;/th&gt;
&lt;th&gt;70s&lt;/th&gt;
&lt;th&gt;80s&lt;/th&gt;
&lt;th&gt;90s&lt;/th&gt;
&lt;th&gt;100s&lt;/th&gt;
&lt;th&gt;110s&lt;/th&gt;
&lt;th&gt;120s&lt;/th&gt;
&lt;th&gt;130s&lt;/th&gt;
&lt;th&gt;140s&lt;/th&gt;
&lt;th&gt;150s&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;memory.high only, 1 process&lt;/td&gt;
&lt;td&gt;237.3 MiB&lt;/td&gt;
&lt;td&gt;237.9 MiB&lt;/td&gt;
&lt;td&gt;238.4 MiB&lt;/td&gt;
&lt;td&gt;238.8 MiB&lt;/td&gt;
&lt;td&gt;239.2 MiB&lt;/td&gt;
&lt;td&gt;239.6 MiB&lt;/td&gt;
&lt;td&gt;239.9 MiB&lt;/td&gt;
&lt;td&gt;240.2 MiB&lt;/td&gt;
&lt;td&gt;240.5 MiB&lt;/td&gt;
&lt;td&gt;240.8 MiB&lt;/td&gt;
&lt;td&gt;241.1 MiB&lt;/td&gt;
&lt;td&gt;241.4 MiB&lt;/td&gt;
&lt;td&gt;241.6 MiB&lt;/td&gt;
&lt;td&gt;241.8 MiB&lt;/td&gt;
&lt;td&gt;242.1 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;memory.high only, 4 processes&lt;/td&gt;
&lt;td&gt;242.3 MiB&lt;/td&gt;
&lt;td&gt;243.4 MiB&lt;/td&gt;
&lt;td&gt;244.5 MiB&lt;/td&gt;
&lt;td&gt;245.6 MiB&lt;/td&gt;
&lt;td&gt;246.1 MiB&lt;/td&gt;
&lt;td&gt;246.6 MiB&lt;/td&gt;
&lt;td&gt;247.1 MiB&lt;/td&gt;
&lt;td&gt;247.5 MiB&lt;/td&gt;
&lt;td&gt;247.9 MiB&lt;/td&gt;
&lt;td&gt;248.3 MiB&lt;/td&gt;
&lt;td&gt;248.7 MiB&lt;/td&gt;
&lt;td&gt;249.1 MiB&lt;/td&gt;
&lt;td&gt;249.4 MiB&lt;/td&gt;
&lt;td&gt;249.8 MiB&lt;/td&gt;
&lt;td&gt;250.1 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;memory.high + memory.max&lt;/td&gt;
&lt;td&gt;236.8 MiB&lt;/td&gt;
&lt;td&gt;237.5 MiB&lt;/td&gt;
&lt;td&gt;238 MiB&lt;/td&gt;
&lt;td&gt;238.5 MiB&lt;/td&gt;
&lt;td&gt;238.9 MiB&lt;/td&gt;
&lt;td&gt;239.3 MiB&lt;/td&gt;
&lt;td&gt;239.7 MiB&lt;/td&gt;
&lt;td&gt;240 MiB&lt;/td&gt;
&lt;td&gt;240.3 MiB&lt;/td&gt;
&lt;td&gt;240.6 MiB&lt;/td&gt;
&lt;td&gt;240.9 MiB&lt;/td&gt;
&lt;td&gt;241.2 MiB&lt;/td&gt;
&lt;td&gt;241.4 MiB&lt;/td&gt;
&lt;td&gt;241.7 MiB&lt;/td&gt;
&lt;td&gt;241.9 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;memory.high: 230.4 MiB&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;memory.max: 256 MiB&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Median of 3 repetitions at each 10 s mark, sampled from outside the cgroup every 100 ms. memory.max only reaches 256 MiB and is killed in under 2 s, so it is not drawn. Source: results/runs/anon-over-&lt;/em&gt;/samples.jsonl.*&lt;/p&gt;

&lt;p&gt;Within the 150 second cap, &lt;code&gt;memory.high&lt;/code&gt; plus &lt;code&gt;memory.max&lt;/code&gt; behaved like &lt;code&gt;memory.high&lt;/code&gt; alone, because the process did not get near &lt;code&gt;memory.max&lt;/code&gt;. After 150 seconds all three runs were at 241.7 to 242.0 MiB, 14 MiB under the kill. The growth rate fits the model of one sleep per page: 4 KiB per sleep, at the formula's sleep length for the current usage, predicts 24.7 to 26.1 KiB/s at the end of the runs, and we measured 26.6 to 28.0 KiB/s over the last 60 seconds (&lt;code&gt;results/predict.md&lt;/code&gt;, section 5).&lt;/p&gt;

&lt;p&gt;To see when the kill comes, we ran the same workload once more with a 45 minute cap (&lt;code&gt;anon-over-both-long&lt;/code&gt;, one run, the same setup as the matrix, so on CPU 0 next to the sampler). The kernel killed it after 1,747.7 seconds, 29.1 minutes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;memory.high at 90% of memory.max: the climb to the OOM kill&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;1m&lt;/th&gt;
&lt;th&gt;2m&lt;/th&gt;
&lt;th&gt;3m&lt;/th&gt;
&lt;th&gt;4m&lt;/th&gt;
&lt;th&gt;5m&lt;/th&gt;
&lt;th&gt;6m&lt;/th&gt;
&lt;th&gt;7m&lt;/th&gt;
&lt;th&gt;8m&lt;/th&gt;
&lt;th&gt;9m&lt;/th&gt;
&lt;th&gt;10m&lt;/th&gt;
&lt;th&gt;11m&lt;/th&gt;
&lt;th&gt;12m&lt;/th&gt;
&lt;th&gt;13m&lt;/th&gt;
&lt;th&gt;14m&lt;/th&gt;
&lt;th&gt;15m&lt;/th&gt;
&lt;th&gt;16m&lt;/th&gt;
&lt;th&gt;17m&lt;/th&gt;
&lt;th&gt;18m&lt;/th&gt;
&lt;th&gt;19m&lt;/th&gt;
&lt;th&gt;20m&lt;/th&gt;
&lt;th&gt;21m&lt;/th&gt;
&lt;th&gt;22m&lt;/th&gt;
&lt;th&gt;23m&lt;/th&gt;
&lt;th&gt;24m&lt;/th&gt;
&lt;th&gt;25m&lt;/th&gt;
&lt;th&gt;26m&lt;/th&gt;
&lt;th&gt;27m&lt;/th&gt;
&lt;th&gt;28m&lt;/th&gt;
&lt;th&gt;29m&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;memory.current&lt;/td&gt;
&lt;td&gt;239.7 MiB&lt;/td&gt;
&lt;td&gt;241.4 MiB&lt;/td&gt;
&lt;td&gt;242.8 MiB&lt;/td&gt;
&lt;td&gt;243.9 MiB&lt;/td&gt;
&lt;td&gt;244.8 MiB&lt;/td&gt;
&lt;td&gt;245.7 MiB&lt;/td&gt;
&lt;td&gt;246.4 MiB&lt;/td&gt;
&lt;td&gt;247.1 MiB&lt;/td&gt;
&lt;td&gt;247.8 MiB&lt;/td&gt;
&lt;td&gt;248.4 MiB&lt;/td&gt;
&lt;td&gt;248.9 MiB&lt;/td&gt;
&lt;td&gt;249.5 MiB&lt;/td&gt;
&lt;td&gt;250 MiB&lt;/td&gt;
&lt;td&gt;250.5 MiB&lt;/td&gt;
&lt;td&gt;250.9 MiB&lt;/td&gt;
&lt;td&gt;251.3 MiB&lt;/td&gt;
&lt;td&gt;251.7 MiB&lt;/td&gt;
&lt;td&gt;252.1 MiB&lt;/td&gt;
&lt;td&gt;252.5 MiB&lt;/td&gt;
&lt;td&gt;252.8 MiB&lt;/td&gt;
&lt;td&gt;253.2 MiB&lt;/td&gt;
&lt;td&gt;253.5 MiB&lt;/td&gt;
&lt;td&gt;253.9 MiB&lt;/td&gt;
&lt;td&gt;254.2 MiB&lt;/td&gt;
&lt;td&gt;254.5 MiB&lt;/td&gt;
&lt;td&gt;254.8 MiB&lt;/td&gt;
&lt;td&gt;255.1 MiB&lt;/td&gt;
&lt;td&gt;255.4 MiB&lt;/td&gt;
&lt;td&gt;255.7 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;memory.high: 230.4 MiB&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;memory.max: 256 MiB&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One run, sampled every 100 ms, one point a minute. Source: results/runs/anon-over-both-long-r1/samples.jsonl.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In those 29 minutes the process used 1.19 seconds of CPU, its memory pressure &lt;code&gt;some&lt;/code&gt; covered 1,742 seconds, and its slowest 256 KiB allocation took 53.2 seconds. From the point where the sleep starts (233.4 MiB, 0.3 seconds in), the formula with one sleep per page predicts 1,720 seconds to reach &lt;code&gt;memory.max&lt;/code&gt;, 1.6% less than we measured. With one sleep per 64 pages it would have been 27 seconds (&lt;code&gt;results/predict.md&lt;/code&gt;, section 3). Then the OOM kill came with 35 &lt;code&gt;max&lt;/code&gt; events, as in the &lt;code&gt;memory.max&lt;/code&gt; runs. &lt;code&gt;memory.high&lt;/code&gt; did not save the program. It delayed the same OOM kill by half an hour.&lt;/p&gt;

&lt;p&gt;On a CPU of its own (&lt;code&gt;diag-over-both-cpu2-1s&lt;/code&gt;, 3 runs, 120 second cap), the same workload was not killed within the cap either. It reached 240.8 to 254.4 MiB, depending on how much of the run fell into the one-sleep-per-page phases. So we have two reference points, not a proven range: the formula's 27 seconds if every sleep covers 64 pages, and the 29.1 minutes we measured once next to the sampler.&lt;/p&gt;

&lt;h2&gt;
  
  
  The throttle is per process
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;memory.high&lt;/code&gt; puts each allocating task to sleep on its own. In the matrix, the four processes of &lt;code&gt;anon-over-4p-high&lt;/code&gt; all ran on CPU 0 next to the sampler. We cannot split &lt;code&gt;memory.events&lt;/code&gt; per process, but their combined growth matches the model of one sleep per page for each process: about four times as fast as one process at the same overage. In &lt;code&gt;anon-over-4p-high&lt;/code&gt; the four processes reached 249.1 to 250.8 MiB in 150 seconds, against 241.5 to 242.2 MiB for one process. Over the last 60 seconds they grew 34.0 to 42.3 KiB/s, where the one-sleep-per-page model predicts 31.9 to 38.0 KiB/s for four processes at that usage. A program with many allocating worker processes gets to &lt;code&gt;memory.max&lt;/code&gt; sooner than a single process.&lt;/p&gt;

&lt;p&gt;The four throttled processes also changed a number most dashboards show. While they slept, the host's load average was 5.33 on 4 vCPUs, and &lt;code&gt;vmstat&lt;/code&gt; showed the CPUs 96% and 99% idle in the two one-second samples:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;results/observed/loadavg-during-4p-high.txt&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# recorded during anon-over-4p-high-r1: ps, vmstat 1 3, /proc/loadavg, nproc
$ cat results/observed/loadavg-during-4p-high.txt
# 2026-11-05T11:39:12Z during anon-over-4p-high-r1 (ps, vmstat 1 3, /proc/loadavg)
 146187 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146188 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146189 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146190 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r b swpd free buff cache si so bi bo in cs us sy id wa st gu
 0 0 0 1407320 32980 5042700 0 0 835 650 876 2 4 1 94 0 1 0
 0 0 0 1410512 32980 5042720 0 0 0 0 1627 1417 2 3 96 0 0 0
 0 0 0 1413104 32980 5042736 0 0 0 28 621 787 1 0 99 0 0 0
5.33 3.56 2.27 1/347 147320
4

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kernel puts the task to sleep with &lt;code&gt;schedule_timeout_killable()&lt;/code&gt;. That is a killable sleep: only a fatal signal ends it early. &lt;code&gt;ps&lt;/code&gt; shows it as state &lt;code&gt;D&lt;/code&gt;, and Linux counts it in the load average like other tasks in state &lt;code&gt;D&lt;/code&gt;. A high load average with idle CPUs can mean throttled memory, not only slow disks.&lt;/p&gt;

&lt;h2&gt;
  
  
  When the cgroup holds its own code, both limits thrash
&lt;/h2&gt;

&lt;p&gt;So far the interpreter's code pages were in the page cache before each run started, charged to a different cgroup. That is the &lt;code&gt;warm&lt;/code&gt; setup, and it is the reason the runs above had nothing to reclaim. A container on its first start is different: its binary and libraries are read from its image after it starts, so the page cache for its own code is charged to its own cgroup. The &lt;code&gt;owncode&lt;/code&gt; runs copy the interpreter binary to a new file, drop it from the page cache (&lt;code&gt;mincore()&lt;/code&gt; found 0 of its pages cached before each run), and run that copy, so the binary is read inside the run cgroup.&lt;/p&gt;

&lt;p&gt;That gives the kernel something to reclaim, and it changes both limits:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run (20 s cap)&lt;/th&gt;
&lt;th&gt;Outcome&lt;/th&gt;
&lt;th&gt;&lt;code&gt;memory.current&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Pages reclaimed&lt;/th&gt;
&lt;th&gt;Read from disk&lt;/th&gt;
&lt;th&gt;PSI memory &lt;code&gt;some&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;PSI io &lt;code&gt;some&lt;/code&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memory.high&lt;/code&gt; only, 232 MiB target&lt;/td&gt;
&lt;td&gt;not killed in 20 s (3 of 3)&lt;/td&gt;
&lt;td&gt;230.4 MiB&lt;/td&gt;
&lt;td&gt;591,486 to 617,906&lt;/td&gt;
&lt;td&gt;2,426 to 2,535 MB&lt;/td&gt;
&lt;td&gt;11.4 to 11.9 s&lt;/td&gt;
&lt;td&gt;19.9 to 21.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;both, 320 MiB target&lt;/td&gt;
&lt;td&gt;not killed in 20 s (3 of 3)&lt;/td&gt;
&lt;td&gt;230.2 to 230.4 MiB&lt;/td&gt;
&lt;td&gt;633,090 to 634,192&lt;/td&gt;
&lt;td&gt;2,593 to 2,598 MB&lt;/td&gt;
&lt;td&gt;12.4 to 14.3 s&lt;/td&gt;
&lt;td&gt;21.3 to 21.5 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memory.max&lt;/code&gt; only, 320 MiB target&lt;/td&gt;
&lt;td&gt;not killed in 20 s (3 of 3)&lt;/td&gt;
&lt;td&gt;255.9 MiB&lt;/td&gt;
&lt;td&gt;631,197 to 633,258&lt;/td&gt;
&lt;td&gt;2,585 to 2,594 MB&lt;/td&gt;
&lt;td&gt;1.9 to 2.1 s&lt;/td&gt;
&lt;td&gt;21.3 to 21.7 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memory.max&lt;/code&gt; only, uncapped disk, 1 run&lt;/td&gt;
&lt;td&gt;not killed in 20 s (1 of 1)&lt;/td&gt;
&lt;td&gt;256.0 MiB&lt;/td&gt;
&lt;td&gt;2,559,211&lt;/td&gt;
&lt;td&gt;10,483 MB&lt;/td&gt;
&lt;td&gt;6.2 s&lt;/td&gt;
&lt;td&gt;13.6 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Under &lt;code&gt;memory.high&lt;/code&gt;, reclaim kept succeeding, and what it took was the program's own code. The task faulted it back in from disk, reclaim took it again, and so on. Because reclaim kept freeing pages, the kernel's retry loop never got to the sleep (&lt;code&gt;if (nr_reclaimed || nr_retries--)&lt;/code&gt; retries as long as anything was freed). The sampler never found the process in &lt;code&gt;mem_cgroup_handle_over_high&lt;/code&gt;, and only 1 of the 108 operations that raised the &lt;code&gt;high&lt;/code&gt; counter raised it by a multiple of 17. The process sat at 230.2 to 230.4 MiB, right at &lt;code&gt;memory.high&lt;/code&gt;. In 54% to 90% of the samples it was in &lt;code&gt;folio_wait_bit_common&lt;/code&gt;, waiting for its code to come back from disk, and in the rest it was running. It never finished its grow phase.&lt;/p&gt;

&lt;p&gt;Under &lt;code&gt;memory.max&lt;/code&gt; alone, the same thing replaced the OOM kill. The charge path only calls the OOM killer after 16 tries in which reclaim freed nothing, and here reclaim kept finding code pages to free. The process held 255.9 MiB and was not killed in any of the three 20 second runs. &lt;code&gt;memory.events&lt;/code&gt; counted 19,140 to 20,019 &lt;code&gt;max&lt;/code&gt; events per run, and no OOM. On the uncapped disk, it read 10,483 MB in 25 seconds and was not killed either. In an earlier smoke run with the same setup (older code, in &lt;code&gt;results/_smoke5&lt;/code&gt;), the kernel did kill it, 21.7 seconds after it started. So the kill does come in some runs, after a period of thrashing we could not predict. In another smoke run on the uncapped disk (&lt;code&gt;results/_smoke3&lt;/code&gt;) it was still thrashing at 70 seconds when our driver stopped it.&lt;/p&gt;

&lt;p&gt;Memory PSI under-counts this. In the &lt;code&gt;memory.max&lt;/code&gt; runs with the 100 MiB/s cap, memory &lt;code&gt;some&lt;/code&gt; was 1.9 to 2.1 seconds out of 25, about 8%, while io &lt;code&gt;some&lt;/code&gt; was 21.3 to 21.7 seconds. The kernel counts a wait for a page read as a memory stall only when the page carries the workingset flag (&lt;code&gt;folio_wait_bit_common()&lt;/code&gt; in &lt;code&gt;mm/filemap.c&lt;/code&gt;), and most of this time went to the disk. A pressure-based killer that watches memory pressure alone would not see this as clearly as the warm &lt;code&gt;memory.high&lt;/code&gt; runs, where memory &lt;code&gt;some&lt;/code&gt; was 99%.&lt;/p&gt;

&lt;h2&gt;
  
  
  Page cache: neither limit throttles or kills
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;file-over&lt;/code&gt; runs read 320 MiB of a file in 256 KiB reads, then read random chunks of it. That is page cache, which the kernel can drop and read again. None of the 12 runs was killed or throttled:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Outcome&lt;/th&gt;
&lt;th&gt;Peak MiB&lt;/th&gt;
&lt;th&gt;Pages reclaimed&lt;/th&gt;
&lt;th&gt;PSI memory &lt;code&gt;some&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;p99 per read&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;no limit&lt;/td&gt;
&lt;td&gt;not killed (3 of 3)&lt;/td&gt;
&lt;td&gt;327.7&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0.00 s&lt;/td&gt;
&lt;td&gt;82.1 to 86.5 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memory.max&lt;/code&gt; only&lt;/td&gt;
&lt;td&gt;not killed (3 of 3)&lt;/td&gt;
&lt;td&gt;256.0&lt;/td&gt;
&lt;td&gt;20,608 to 21,440&lt;/td&gt;
&lt;td&gt;0.02 s&lt;/td&gt;
&lt;td&gt;83.4 to 88.6 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memory.high&lt;/code&gt; only&lt;/td&gt;
&lt;td&gt;not killed (3 of 3)&lt;/td&gt;
&lt;td&gt;230.9&lt;/td&gt;
&lt;td&gt;28,352 to 28,864&lt;/td&gt;
&lt;td&gt;0.03 s&lt;/td&gt;
&lt;td&gt;85.0 to 87.5 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;both&lt;/td&gt;
&lt;td&gt;not killed (3 of 3)&lt;/td&gt;
&lt;td&gt;230.9&lt;/td&gt;
&lt;td&gt;28,352 to 28,864&lt;/td&gt;
&lt;td&gt;0.02 to 0.03 s&lt;/td&gt;
&lt;td&gt;82.9 to 89.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Both limits did what the docs say: &lt;code&gt;memory.current&lt;/code&gt; stopped at the limit and the kernel reclaimed file pages to stay there. &lt;code&gt;memory.events&lt;/code&gt; counted 416 to 430 &lt;code&gt;high&lt;/code&gt; events and 322 to 335 &lt;code&gt;max&lt;/code&gt; events per run. Every &lt;code&gt;high&lt;/code&gt; event was a single reclaim pass with no sleep: the rise per operation was 1 or 2, never 17, because each pass freed enough to get back under the limit. The p99 of 82 to 90 ms is the same with no limit, so it is not the memory limits. The slow reads are in the first pass over the file, read from disk under our 100 MiB/s cap. The limits did cost something later: in the last 10% of operations, random reads that had to go back to disk gave a p99 of 1.0 to 2.6 ms, against 0.13 ms with no limit, where the whole file stayed cached. For a cache-heavy program without swap, both limits are a cache size, not a throttle and not a kill.&lt;/p&gt;

&lt;h2&gt;
  
  
  What PSI and the counters show
&lt;/h2&gt;

&lt;p&gt;PSI was a reliable witness for the warm throttle. In every warm single-process matrix run with &lt;code&gt;memory.high&lt;/code&gt;, the cgroup's &lt;code&gt;memory.pressure&lt;/code&gt; &lt;code&gt;some&lt;/code&gt; total matched the time the operations spent over their normal latency, at a ratio of 0.998 to 1.068 (&lt;code&gt;results/predict.md&lt;/code&gt;, section 4). In the diagnostic runs it was 0.997 to 1.286. PSI also counts the last operation, which was still asleep when the driver stopped the run, and our operation log does not, so the ratio goes above 1 in runs that ended in the middle of a long operation. With one task in the cgroup, &lt;code&gt;full&lt;/code&gt; was the same as &lt;code&gt;some&lt;/code&gt;, and &lt;code&gt;avg10&lt;/code&gt; reached 97% to 99%. The host saw it too. During the long run, the host-wide &lt;code&gt;/proc/pressure/memory&lt;/code&gt; read &lt;code&gt;some avg10=96.53&lt;/code&gt; and &lt;code&gt;full avg10=95.09&lt;/code&gt; (&lt;code&gt;results/observed/host-psi-during-long.txt&lt;/code&gt;). Our test cgroup was the one we know was stalled, and the host-wide number was almost as high, so an alert on host memory pressure would fire.&lt;/p&gt;

&lt;p&gt;The counters to watch, from these runs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;memory.events&lt;/code&gt; &lt;code&gt;high&lt;/code&gt; rising: the cgroup is over &lt;code&gt;memory.high&lt;/code&gt;. 17 per sleep when nothing is reclaimable.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;memory.events&lt;/code&gt; &lt;code&gt;max&lt;/code&gt; rising without &lt;code&gt;oom_kill&lt;/code&gt;: the cgroup is at &lt;code&gt;memory.max&lt;/code&gt; and reclaim keeps finding something. In the own-code runs that meant thrashing.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;memory.stat&lt;/code&gt; &lt;code&gt;pgsteal&lt;/code&gt; and &lt;code&gt;workingset_refault_file&lt;/code&gt; rising together: the cgroup reclaims pages and reads them straight back.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;memory.pressure&lt;/code&gt; &lt;code&gt;some&lt;/code&gt; near 100% with almost no CPU use and state &lt;code&gt;D&lt;/code&gt;: throttled by &lt;code&gt;memory.high&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;io.pressure&lt;/code&gt; high while &lt;code&gt;memory.pressure&lt;/code&gt; is low, in a cgroup at its memory limit: thrashing on file pages.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What systemd, Docker and Kubernetes write
&lt;/h2&gt;

&lt;h3&gt;
  
  
  systemd
&lt;/h3&gt;

&lt;p&gt;systemd maps its properties one to one. We started transient scopes with each property and read the files back from inside (&lt;code&gt;results/10-systemd-map.txt&lt;/code&gt;, systemd 255):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Value we set&lt;/th&gt;
&lt;th&gt;Value in the file&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MemoryMin=16M&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;memory.min&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;16M&lt;/td&gt;
&lt;td&gt;16777216&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MemoryLow=64M&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;memory.low&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;64M&lt;/td&gt;
&lt;td&gt;67108864&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MemoryHigh=241590272&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;memory.high&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;241590272&lt;/td&gt;
&lt;td&gt;241590272&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MemoryMax=256M&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;memory.max&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;256M&lt;/td&gt;
&lt;td&gt;268435456&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MemoryHigh=3%&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;memory.high&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;3% of RAM&lt;/td&gt;
&lt;td&gt;249806848&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A property you do not set stays at the kernel default: &lt;code&gt;max&lt;/code&gt; for &lt;code&gt;memory.high&lt;/code&gt; and &lt;code&gt;memory.max&lt;/code&gt;, 0 for &lt;code&gt;memory.min&lt;/code&gt; and &lt;code&gt;memory.low&lt;/code&gt;. &lt;code&gt;memory.oom.group&lt;/code&gt; stayed 0 in every scope.&lt;/p&gt;

&lt;p&gt;The systemd 255 man page says &lt;code&gt;MemoryHigh=&lt;/code&gt; is "the main mechanism to control memory usage of a unit", and it recommends &lt;code&gt;MemoryMax=&lt;/code&gt; "as the last line of defense". On a host with swap, the kernel can move cold anonymous pages out and hold the unit near the limit. We did not test that. Without swap, our runs show what "main mechanism" means: the unit does not stop at the limit, it slows down, and with &lt;code&gt;MemoryHigh=&lt;/code&gt; alone the kernel never kills it.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;MemoryMin=&lt;/code&gt; and &lt;code&gt;MemoryLow=&lt;/code&gt; are the other direction. They protect a unit's memory from reclaim when something else needs it. They do not limit or slow the unit itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  Docker
&lt;/h3&gt;

&lt;p&gt;Docker 29.9.3 has no flag for &lt;code&gt;memory.high&lt;/code&gt;. &lt;code&gt;docker run --help&lt;/code&gt; on the test host lists &lt;code&gt;--memory&lt;/code&gt;, &lt;code&gt;--memory-reservation&lt;/code&gt;, &lt;code&gt;--memory-swap&lt;/code&gt;, &lt;code&gt;--memory-swappiness&lt;/code&gt;, &lt;code&gt;--oom-kill-disable&lt;/code&gt; and &lt;code&gt;--oom-score-adj&lt;/code&gt; (&lt;code&gt;results/00-host.txt&lt;/code&gt;). We followed the two memory flags through the source (&lt;code&gt;results/01-source-check.txt&lt;/code&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Docker puts &lt;code&gt;--memory&lt;/code&gt; into the OCI spec as &lt;code&gt;memory.limit&lt;/code&gt; and &lt;code&gt;--memory-reservation&lt;/code&gt; as &lt;code&gt;memory.reservation&lt;/code&gt; (&lt;code&gt;getMemoryResources&lt;/code&gt; in moby &lt;code&gt;daemon/daemon_unix.go&lt;/code&gt;, docker-v29.9.3).&lt;/li&gt;
&lt;li&gt;runc 1.5.2, the host's runtime, writes the limit to &lt;code&gt;memory.max&lt;/code&gt; and the reservation to &lt;code&gt;memory.low&lt;/code&gt; on cgroup v2 (&lt;code&gt;fs2/memory.go&lt;/code&gt;). With the systemd cgroup driver it sets the &lt;code&gt;MemoryMax&lt;/code&gt; and &lt;code&gt;MemoryLow&lt;/code&gt; properties (&lt;code&gt;systemd/v2.go&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So on cgroup v2, &lt;code&gt;--memory&lt;/code&gt; is a hard limit and &lt;code&gt;--memory-reservation&lt;/code&gt; is reclaim protection. The Docker docs call &lt;code&gt;--memory-reservation&lt;/code&gt; "a soft limit smaller than &lt;code&gt;--memory&lt;/code&gt; which is activated when Docker detects contention or low memory on the host machine." That wording comes from cgroup v1's soft limit. On v2 it is &lt;code&gt;memory.low&lt;/code&gt;: it does not slow the container down at any size.&lt;/p&gt;

&lt;p&gt;runc does accept &lt;code&gt;memory.high&lt;/code&gt; through the OCI spec's &lt;code&gt;unified&lt;/code&gt; map, and passes it on as the systemd &lt;code&gt;MemoryHigh&lt;/code&gt; property. That is the path the kubelet uses. The Docker CLI does not expose it, so with plain &lt;code&gt;docker run&lt;/code&gt; you only get the &lt;code&gt;memory.max&lt;/code&gt; behaviour.&lt;/p&gt;

&lt;h3&gt;
  
  
  Kubernetes MemoryQoS
&lt;/h3&gt;

&lt;p&gt;The kubelet sets &lt;code&gt;memory.max&lt;/code&gt; from &lt;code&gt;limits.memory&lt;/code&gt; on cgroup v2 in every version. &lt;code&gt;memory.high&lt;/code&gt; is a separate feature, &lt;strong&gt;MemoryQoS&lt;/strong&gt; (&lt;a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos" rel="noopener noreferrer"&gt;KEP-2570&lt;/a&gt;), and its status changed in the last two releases. All of this is from &lt;code&gt;results/01-source-check.txt&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Kubernetes 1.36 (v1.36.6)&lt;/th&gt;
&lt;th&gt;Kubernetes 1.37 (v1.37.2)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;MemoryQoS&lt;/code&gt; feature gate&lt;/td&gt;
&lt;td&gt;alpha, off by default&lt;/td&gt;
&lt;td&gt;beta, on by default&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memoryThrottlingFactor&lt;/code&gt; default&lt;/td&gt;
&lt;td&gt;0.9&lt;/td&gt;
&lt;td&gt;nil: no &lt;code&gt;memory.high&lt;/code&gt; unless you set it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memoryReservationPolicy&lt;/code&gt; default&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;None&lt;/code&gt;: &lt;code&gt;memory.min&lt;/code&gt; and &lt;code&gt;memory.low&lt;/code&gt; are 0&lt;/td&gt;
&lt;td&gt;&lt;code&gt;None&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;With the gate on and a factor set, the kubelet computes, for each container whose memory request differs from its limit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;memory.high = floor((requests.memory + factor * (limits.memory - requests.memory)) / pageSize) * pageSize

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With no limit, it uses node allocatable memory instead of &lt;code&gt;limits.memory&lt;/code&gt;. It puts the value in the CRI container config's &lt;code&gt;unified&lt;/code&gt; map under the key &lt;code&gt;memory.high&lt;/code&gt; (&lt;code&gt;kuberuntime_container_linux.go&lt;/code&gt;), and the runtime writes it to the container's cgroup. runc passes a unified &lt;code&gt;memory.high&lt;/code&gt; on as the systemd &lt;code&gt;MemoryHigh&lt;/code&gt; property when it uses the systemd driver. In 1.36, a container whose memory request equals its limit gets no &lt;code&gt;memory.high&lt;/code&gt;. 1.37 makes one exception: a container with no request and no limit can still get one when a factor is set.&lt;/p&gt;

&lt;p&gt;Our &lt;code&gt;memory.high&lt;/code&gt; value is this formula for &lt;code&gt;limits.memory: 256Mi&lt;/code&gt;, &lt;code&gt;requests.memory: 0&lt;/code&gt; and factor 0.9: 241590272 bytes. Two details matter here. If a pod sets a limit and leaves the request out, the API server sets the request equal to the limit (&lt;code&gt;SetDefaults_Pod&lt;/code&gt; in &lt;code&gt;pkg/apis/core/v1/defaults.go&lt;/code&gt;), and the kubelet then writes no &lt;code&gt;memory.high&lt;/code&gt; at all. And with a request R between 0 and the limit, &lt;code&gt;memory.high&lt;/code&gt; is R + 0.9 x (256Mi - R), so it sits between 230.4 MiB and 256 MiB. Our runs are the strictest case: a Burstable container with an explicit zero memory request on a 1.36 node with the MemoryQoS gate turned on. The &lt;code&gt;anon-fit&lt;/code&gt; rows are the ones to read for that.&lt;/p&gt;

&lt;p&gt;The KEP explains why it took so long. The 1.28 beta was cancelled because a workload "stays stuck indefinitely" at &lt;code&gt;memory.high&lt;/code&gt;, and the KEP says kernel 5.9 fixed that livelock. On 6.8 our warm runs did make progress, slowly. The own-code runs barely did. The 1.37 beta turns the gate on but leaves &lt;code&gt;memoryThrottlingFactor&lt;/code&gt; unset, so upgrading does not add &lt;code&gt;memory.high&lt;/code&gt; to any container. If you set the factor to 0.9, you opt into the &lt;code&gt;both&lt;/code&gt; rows above.&lt;/p&gt;

&lt;p&gt;The two Kubernetes posts this week cover the other side of the same cgroups: &lt;a href="https://dev.to/devopsdaily/an-empty-resources-makes-kubernetes-134-to-136-pods-besteffort-2p15"&gt;an empty &lt;code&gt;resources: {}&lt;/code&gt; makes pods BestEffort&lt;/a&gt; on 1.34 to 1.36 and changes their &lt;code&gt;oom_score_adj&lt;/code&gt; and cgroup slice, and &lt;a href="https://dev.to/devopsdaily/kubernetes-137-garhwal-what-shipped-and-what-slipped-1cmi"&gt;the 1.37 review&lt;/a&gt; lists what else moved in that release.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we could not conclude
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One host, one kernel, no swap.&lt;/strong&gt; Everything here is Ubuntu 6.8.0-147 on a 4 vCPU machine with no swap. With swap or zswap, reclaim can move anonymous pages out, so &lt;code&gt;memory.high&lt;/code&gt; can hold a cgroup near the limit instead of only slowing it, and &lt;code&gt;memory.max&lt;/code&gt; reaches the OOM killer later. We did not test that. Newer kernels have changed the memcg charge code, and we did not test them either.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What empties the precharge.&lt;/strong&gt; We measured how many sleeps a task paid, and we read in the source what makes a new charge necessary. We did not trace which event emptied the per-CPU precharge each time. Our sampler on the same CPU made it happen on almost every long sleep. Without it, the phases came and went with the host's other work, which we did not control. An idle machine, or one with more CPUs, could pay fewer sleeps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The rest of the extra per sleep.&lt;/strong&gt; The timer wheel's rounding explains about half of the 6% to 10%. We did not split the rest into reclaim time, wake-up delay and our own probes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When the own-code thrash ends.&lt;/strong&gt; Under &lt;code&gt;memory.max&lt;/code&gt; it was not killed in 20 seconds in the matrix (one smoke run was killed at 21.7 seconds, another was still running at 70). We stopped these runs early because they read about 100 MB/s from a shared disk, so we cannot say how long a real container would thrash before the kill. The result also depends on disk speed, which we changed with &lt;code&gt;io.max&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run order and a shared host.&lt;/strong&gt; Every repetition ran the configurations in the same order, on a machine with other services. Background load can move a single run either way. The three repetitions agree closely, but we did not randomize the order.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Real programs.&lt;/strong&gt; Our workload is a Python loop that touches memory in a fixed pattern. A JVM, a Go program or a database touches memory differently, and the &lt;code&gt;MAP_POPULATE&lt;/code&gt; run shows that the touch pattern alone changed the cost 32 times.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The 4-process numbers per operation.&lt;/strong&gt; &lt;code&gt;memory.events&lt;/code&gt; counts the whole cgroup, so in the 4-process run one operation's rise in &lt;code&gt;high&lt;/code&gt; includes the other processes' sleeps. We used that run only for growth, not for sleeps per operation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kubernetes on a real node.&lt;/strong&gt; We wrote the same &lt;code&gt;memory.high&lt;/code&gt; value the kubelet would write, but we did not run a kubelet. The Kubernetes section is from the source at v1.36.6 and v1.37.2.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What to do with this
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Without swap, do not use &lt;code&gt;memory.high&lt;/code&gt; as the only limit.&lt;/strong&gt; It does not stop growth. It slows new allocations, and a program that keeps allocating stays slow as long as it stays over the limit. The kernel never kills it for &lt;code&gt;memory.high&lt;/code&gt;, so add &lt;code&gt;memory.max&lt;/code&gt; or a pressure-based killer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If you set both, leave room.&lt;/strong&gt; With &lt;code&gt;memory.high&lt;/code&gt; at 90% of &lt;code&gt;memory.max&lt;/code&gt;, a program that fits between the two ran about 750 times slower per allocation on a CPU apart from our sampler, and about 24,000 times slower next to it. Apart from the sampler it finished in 2 of 3 runs within 120 seconds; next to it, none finished within 150 seconds. Measure your program's peak first. Put &lt;code&gt;memory.high&lt;/code&gt; above the normal working set, not at a fixed fraction of the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not plan with the formula alone.&lt;/strong&gt; It gives the length of one sleep. How many sleeps a task pays depends on what else allocates memory on the same CPU, by up to 64 times, so test on a host that is as busy as production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read &lt;code&gt;memory.events&lt;/code&gt; and &lt;code&gt;memory.pressure&lt;/code&gt;, not only &lt;code&gt;memory.current&lt;/code&gt;.&lt;/strong&gt; A throttled cgroup looks healthy by memory use: it sits a few MiB over its limit and uses no CPU. The &lt;code&gt;high&lt;/code&gt; counter and memory &lt;code&gt;some&lt;/code&gt; near 100% show it at once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch &lt;code&gt;io.pressure&lt;/code&gt; and &lt;code&gt;pgsteal&lt;/code&gt; on cgroups at their limit.&lt;/strong&gt; Thrashing on a program's own code showed up there, not in memory pressure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On Kubernetes 1.37, MemoryQoS is on but does not throttle until you set &lt;code&gt;memoryThrottlingFactor&lt;/code&gt;.&lt;/strong&gt; If you set it, test your Burstable pods whose usage sits between the factor and the limit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The scripts, the raw output of every run, and the claim we wrote before the first run are in the repository above.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/cgroup-v2-memory-high-vs-memory-max-measured" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>linux</category>
      <category>cgroups</category>
      <category>memory</category>
      <category>systemd</category>
    </item>
  </channel>
</rss>
