<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DevOps Daily</title>
    <link>https://devops-daily.com</link>
    <description>The latest DevOps news, tutorials, and guides</description>
    <language>en</language>
    <lastBuildDate>Sun, 15 Nov 2026 17:49:54 GMT</lastBuildDate>
    <atom:link href="https://devops-daily.com/feed.xml" rel="self" type="application/rss+xml"/>
    
    <item>
      <title><![CDATA[Five Production Bugs You Can Reproduce in a Browser Tab]]></title>
      <link>https://devops-daily.com/posts/five-production-bugs-you-can-reproduce-in-a-browser</link>
      <description><![CDATA[A cache can leak a cart, one packet can pause an HTTP/2 page, and a shared DataLoader can disclose another user's data. Learn the mechanism, the production check, and a small lab reproduction for five failures worth recognizing before an incident.]]></description>
      <pubDate>Fri, 13 Nov 2026 15:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/five-production-bugs-you-can-reproduce-in-a-browser</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[HTTP]]></category><category><![CDATA[CDN]]></category><category><![CDATA[Email]]></category><category><![CDATA[APIs]]></category><category><![CDATA[GraphQL]]></category><category><![CDATA[Networking]]></category>
      <content:encoded><![CDATA[<p>Many production failures are easy to name after the fact and hard to see while you are building: a cache key contains too little or too much, a retry creates a second payment, or a request cache outlives the request. The details differ, but each failure starts with an assumption about what a protocol, intermediary, or library remembers.</p>
<p>The five free DevOps Daily labs linked below make those assumptions visible. This is not a tour of the labs. Each section starts with the operational symptom, explains the protocol rule, gives you a check to run against your own system, then provides a compact reproduction. You can use the post without opening a lab.</p>
<h2>TL;DR</h2><table>
<thead>
<tr>
<th>Failure</th>
<th>First thing to inspect</th>
<th>Safe default</th>
</tr>
</thead>
<tbody><tr>
<td>A shared cache returns personal content</td>
<td><code>Cache-Control</code>, <code>Vary</code>, and the cache key</td>
<td>Mark personal responses <code>private</code> or <code>no-store</code></td>
</tr>
<tr>
<td>One lost packet delays a page</td>
<td>negotiated HTTP version and loss-sensitive waterfalls</td>
<td>Test HTTP/2 and HTTP/3 on the links your users have</td>
</tr>
<tr>
<td>A legitimate message fails DMARC</td>
<td><code>Authentication-Results</code> and alignment</td>
<td>Align either SPF or DKIM with visible <code>From</code></td>
</tr>
<tr>
<td>A retry creates another operation</td>
<td>idempotency key, request fingerprint, saved response</td>
<td>One key per intended operation, reused only for retries</td>
</tr>
<tr>
<td>Nested GraphQL fields make too many queries</td>
<td>resolver SQL and DataLoader scope</td>
<td>Batch per request, never across requests</td>
</tr>
</tbody></table>
<h2>Prerequisites</h2><ul>
<li>A terminal with <code>curl</code> and <code>dig</code> (from <code>bind9-dnsutils</code> or <code>bind-utils</code>).</li>
<li>Read access to one of your own HTTP endpoints, one sending domain, and one API that accepts retries.</li>
<li>A modern browser for the labs. They are free and run in the browser.</li>
</ul>
<h2>A shared cache serves a cart to the wrong person</h2><p>The frightening version of this bug is a user seeing somebody else's cart, profile, or account page. The quieter version is a cache hit ratio that collapses because every campaign link has a distinct tracking query string. Both are cache-key errors: the cache either treats two different representations as the same object, or one representation as too many objects.</p>
<p><a href="https://www.rfc-editor.org/rfc/rfc9111" rel="noopener noreferrer">RFC 9111</a> says a cache key normally includes the request method and target URI. A cache that stores a response selected by <code>Vary</code> must also use the nominated request-field values to choose a matching response. The response controls matter just as much: <code>private</code> prohibits storage in a shared cache, and <code>no-store</code> says not to store it. <a href="https://www.rfc-editor.org/rfc/rfc5861#section-3" rel="noopener noreferrer">RFC 5861</a> defines <code>stale-while-revalidate</code>, which lets a cache serve a stale response while it validates in the background.</p>
<p>Start with a response that must be private and an asset that should be reusable:</p>
<pre><code class="hljs language-bash" tabindex="0">curl -sI https://app.example.com/api/cart
curl -sI https://app.example.com/assets/app.8d42c.js
</code></pre><p>The cart response should show <code>Cache-Control: private</code> or <code>Cache-Control: no-store</code>. A content-addressed asset can use a long <code>max-age</code>. For a cacheable response, read <code>Age</code>, <code>Vary</code>, and <code>ETag</code>; these are standardized response fields, though any vendor cache-status header is not. Then test validation with the actual entity tag you received:</p>
<pre><code class="hljs language-bash" tabindex="0">post_etag=<span class="hljs-string">'"replace-with-the-etag-value"'</span>
curl -sSI \
  -H <span class="hljs-string">"If-None-Match: <span class="hljs-variable">${post_etag}</span>"</span> \
  https://app.example.com/index.html
</code></pre><p>For an unchanged representation, the expected shape is <code>HTTP/... 304</code> with an <code>ETag</code>. A <code>304</code> confirms conditional validation, not that every edge has the same object. Review whether your CDN intentionally removes tracking parameters from its cache key before deciding that <code>/pricing?utm_source=...</code> and <code>/pricing</code> are equivalent.</p>
<blockquote>
<p><strong>Tip</strong></p>
<p><strong><a href="https://devops-daily.com/games/cdn-edge-cache-lab">Reproduce it in the CDN Edge Cache Lab</a></strong></p>
<ol>
<li>Select <code>/api/cart</code> and press <strong>Send traffic</strong>. Observe the <code>BYPASS</code> results when its response is private or no-store.</li>
<li>Turn on <strong>utm_ links</strong>, send traffic, and inspect the multiple cached keys.</li>
<li>Turn on <strong>Ignore utm_ in cache key</strong>, then send traffic again to see later visitors get <code>HIT</code>.</li>
<li>Select <code>/index.html</code>, send traffic, choose <strong>Wait 61 s</strong>, and send traffic again to expose a <code>304</code> revalidation.</li>
</ol>
<p>The lab models one origin and three CDN edges. Its latency values are illustrative estimates, and it deliberately does not apply heuristic freshness when an explicit lifetime is absent.</p>
</blockquote>
<h2>One lost packet pauses every active HTTP/2 stream</h2><p>HTTP/2 usually removes the browser-side request queues that made HTTP/1.1 pages painful: it multiplexes streams on one connection. The tradeoff is lower in the stack. HTTP/2 runs over TCP, and TCP delivers bytes in order. If a TCP packet containing bytes for one stream is missing, later bytes for all active streams on that connection wait for recovery. On a lossy high-latency path, that can make HTTP/2 slower than several HTTP/1.1 connections carrying independent TCP flows.</p>
<p><a href="https://www.rfc-editor.org/rfc/rfc9113" rel="noopener noreferrer">RFC 9113</a> defines HTTP/2 streams and multiplexed framing. <a href="https://www.rfc-editor.org/rfc/rfc9114#section-1" rel="noopener noreferrer">RFC 9114</a> describes HTTP/3 as HTTP semantics over QUIC and calls out avoiding TCP head-of-line blocking. QUIC streams have independent delivery and loss recovery under <a href="https://www.rfc-editor.org/rfc/rfc9000#section-2.2" rel="noopener noreferrer">RFC 9000</a>, so a lost packet holds up only the streams whose data it carried, while streams with data in other packets keep moving. That removes TCP's connection-wide head-of-line blocking, not every stall.</p>
<p>Check what your endpoint actually negotiates before drawing conclusions from an application dashboard:</p>
<pre><code class="hljs language-bash" tabindex="0">curl -sI --http2 -w <span class="hljs-string">'%{http_version}\n'</span> -o /dev/null https://app.example.com
curl -sI --http3-only -w <span class="hljs-string">'%{http_version}\n'</span> -o /dev/null https://app.example.com
curl -sI https://app.example.com | grep -i <span class="hljs-string">'^alt-svc:'</span>
curl -V
</code></pre><p>The first two commands print the negotiated version, such as <code>2</code> or <code>3</code>. Use <code>--http3-only</code> for the check: plain <code>--http3</code> falls back to an earlier version when QUIC cannot be set up, which hides the answer you are looking for. The <code>Alt-Svc</code> header can advertise an <code>h3</code> alternative service. Before running either command, check <code>curl -V</code>: its Features line must list <code>HTTP2</code> for the first and <code>HTTP3</code> for the second, and <code>--http3-only</code> needs curl 7.88.0 or later. Test on representative networks as well as a clean office connection: protocol choice is a latency and loss question, not a permanent leaderboard.</p>
<ol>
<li><strong>One packet lost</strong> during app.js</li>
<li><strong>Recover loss</strong> retransmission needed</li>
</ol>
<p>Outcomes:</p>
<ul>
<li><strong>HTTP/2 over TCP: active streams wait for in-order delivery</strong></li>
<li><strong>HTTP/3 over QUIC: streams in the lost packet wait, others progress</strong></li>
</ul>
<blockquote>
<p><strong>Tip</strong></p>
<p><strong><a href="https://devops-daily.com/games/http-versions-lab">Reproduce it in the HTTP Versions Lab</a></strong></p>
<ol>
<li>Select <strong>HTTP/2</strong>, set <strong>250 ms RTT</strong>, turn on <strong>Lose one packet</strong>, and press <strong>Play all</strong>.</li>
<li>Read the waterfall and the loss explanation: multiple receiving streams stall behind the TCP retransmission.</li>
<li>Switch to <strong>HTTP/3</strong>, keep <strong>Lose one packet</strong> on, and press <strong>Play all</strong>.</li>
<li>Compare the stalled stream count and the setup steps with <strong>Next</strong>.</li>
</ol>
<p>This is a deterministic page-load model, not a browser trace. It excludes DNS, server work, congestion control, priorities, caches, header compression, server push, and QUIC 0-RTT resumption.</p>
</blockquote>
<h2>A valid-looking email fails DMARC after forwarding</h2><p>An order confirmation can leave your sender with SPF, DKIM, and DMARC all passing, then fail after a forwarder or a mailing list touches it. The customer sees a rejected message or a spam-folder delivery, while the sending system sees a successful SMTP handoff. The common mistake is treating three authentication results as interchangeable.</p>
<p><a href="https://www.rfc-editor.org/rfc/rfc7208" rel="noopener noreferrer">RFC 7208</a> specifies SPF for the SMTP MAIL FROM identity and the connecting IP address. <a href="https://www.rfc-editor.org/rfc/rfc6376" rel="noopener noreferrer">RFC 6376</a> specifies DKIM signatures over selected headers and the message body, verified using a selector key in DNS. Current DMARC is <a href="https://www.rfc-editor.org/rfc/rfc9989" rel="noopener noreferrer">RFC 9989</a>, which requires SPF or DKIM to pass <strong>and</strong> align with the visible <code>From</code> domain. A forwarder changes the connecting IP, so SPF often fails; a mailing list that changes a signed body can invalidate DKIM. Rewriting <code>From</code> is one way a list can preserve DMARC when it modifies a message.</p>
<p>Inspect the records and one delivered message rather than guessing from a provider UI:</p>
<pre><code class="hljs language-bash" tabindex="0">dig +short TXT example.com
dig +short TXT selector._domainkey.example.com
dig +short TXT _dmarc.example.com
</code></pre><p>Then open the delivered message source and locate the <code>Authentication-Results</code> header added by your receiving provider, not an untrusted copy from upstream (<a href="https://www.rfc-editor.org/rfc/rfc8601" rel="noopener noreferrer">RFC 8601</a> explains why the header needs a trust check). Its useful shape names the evaluated identities and results, for example <code>spf=pass smtp.mailfrom=...</code>, <code>dkim=pass header.d=...</code>, and <code>dmarc=pass header.from=...</code>. Confirm the domain in <code>header.from</code> aligns with either the SPF MAIL FROM domain or the DKIM <code>d=</code> domain. The earlier <a href="https://devops-daily.com/posts/dmarcbis-what-changed-new-dmarc">DMARCbis guide</a> covers the record migration in more detail: DMARCbis replaced RFC 7489 and removed its <code>pct</code> tag.</p>
<blockquote>
<p><strong>Tip</strong></p>
<p><strong><a href="https://devops-daily.com/games/email-auth-lab">Reproduce it in the Email Authentication Lab</a></strong></p>
<ol>
<li>Select <code>d=shop.example</code> for <strong>DKIM signing domain</strong>, publish the selector, and choose a DMARC policy.</li>
<li>Send the <strong>Direct</strong> path, then send the <strong>Forwarded</strong> path (the forwarder does not use SRS) and compare the SPF, DKIM, and DMARC verdicts.</li>
<li>Select the mailing-list path, turn on <strong>List rewrites From and sets Reply-To</strong>, and send again.</li>
<li>Change <strong>SPF alignment</strong> or <strong>DKIM alignment</strong> from relaxed to strict and read the receiver result.</li>
</ol>
<p>The lab checks the MAIL FROM identity, not HELO, and does not model SRS or ARC. Its percentage control deliberately demonstrates the retired RFC 7489 <code>pct</code> behavior; do not use it as a model of current DMARCbis policy.</p>
</blockquote>
<h2>An ambiguous retry creates a second payment</h2><p>The failure looks like a double charge, duplicate order, or two provisioned resources. Often the first request did succeed, but its response was lost after the server committed work. Retrying is the correct client action after an ambiguous failure. Retrying it as a new operation is not.</p>
<p><a href="https://docs.stripe.com/api/idempotent_requests" rel="noopener noreferrer">Stripe's idempotent requests documentation</a> describes the pattern: the API saves the status code and body of the first request for a key and returns them for later requests with the same key, and it rejects a reuse of the key with different parameters. The IETF is standardizing the header itself in <a href="https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/" rel="noopener noreferrer">The Idempotency-Key HTTP Header Field</a>, which is still an Internet-Draft, not an RFC. A key identifies one intended operation, not one customer or one retry attempt. Store a request fingerprint beside the key, reject a payload mismatch, and distinguish an in-progress duplicate from a completed replay.</p>
<p>Here is the client side. It keeps one generated key for every retry and applies full jitter to retryable failures:</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-keyword">import</span> { randomUUID } <span class="hljs-keyword">from</span> <span class="hljs-string">'node:crypto'</span>;

<span class="hljs-keyword">const</span> idempotencyKey = <span class="hljs-title function_">randomUUID</span>(); <span class="hljs-comment">// One intended order, not one HTTP attempt.</span>
<span class="hljs-keyword">const</span> retryableStatuses = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Set</span>([<span class="hljs-number">409</span>, <span class="hljs-number">429</span>, <span class="hljs-number">500</span>, <span class="hljs-number">502</span>, <span class="hljs-number">503</span>, <span class="hljs-number">504</span>]);

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">createOrder</span>(<span class="hljs-params"></span>) {
  <span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> attempt = <span class="hljs-number">0</span>; attempt &lt; <span class="hljs-number">5</span>; attempt += <span class="hljs-number">1</span>) {
    <span class="hljs-keyword">try</span> {
      <span class="hljs-keyword">const</span> response = <span class="hljs-keyword">await</span> <span class="hljs-title function_">fetch</span>(<span class="hljs-string">'https://api.example.com/orders'</span>, {
        <span class="hljs-attr">method</span>: <span class="hljs-string">'POST'</span>,
        <span class="hljs-comment">// Stripe documents a bare value; the IETF draft defines a quoted string.</span>
        <span class="hljs-comment">// Use the serialization your API documents.</span>
        <span class="hljs-attr">headers</span>: { <span class="hljs-string">'Content-Type'</span>: <span class="hljs-string">'application/json'</span>, <span class="hljs-string">'Idempotency-Key'</span>: idempotencyKey },
        <span class="hljs-attr">signal</span>: <span class="hljs-title class_">AbortSignal</span>.<span class="hljs-title function_">timeout</span>(<span class="hljs-number">10_000</span>), <span class="hljs-comment">// a stalled request must reach the retry path</span>
        <span class="hljs-attr">body</span>: <span class="hljs-title class_">JSON</span>.<span class="hljs-title function_">stringify</span>({ <span class="hljs-attr">sku</span>: <span class="hljs-string">'pro-monthly'</span>, <span class="hljs-attr">customerId</span>: <span class="hljs-string">'cus_123'</span> }),
      });
      <span class="hljs-keyword">if</span> (response.<span class="hljs-property">ok</span> || !retryableStatuses.<span class="hljs-title function_">has</span>(response.<span class="hljs-property">status</span>)) <span class="hljs-keyword">return</span> response;
      <span class="hljs-comment">// A 409 is retryable here only because this API uses it for an in-progress key.</span>
    } <span class="hljs-keyword">catch</span> (error) {
      <span class="hljs-keyword">if</span> (attempt === <span class="hljs-number">4</span>) <span class="hljs-keyword">throw</span> error;
    }

    <span class="hljs-keyword">if</span> (attempt === <span class="hljs-number">4</span>) <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Error</span>(<span class="hljs-string">'Idempotent request did not complete after five attempts'</span>);
    <span class="hljs-keyword">const</span> ceilingMs = <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">min</span>(<span class="hljs-number">5_000</span>, <span class="hljs-number">100</span> * <span class="hljs-number">2</span> ** attempt);
    <span class="hljs-keyword">await</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function">(<span class="hljs-params">resolve</span>) =&gt;</span> <span class="hljs-built_in">setTimeout</span>(resolve, <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">random</span>() * ceilingMs));
  }
}
</code></pre><p>On the server, make the uniqueness boundary visible. Scope keys to the authenticated caller, so one client's key can never replay another client's saved response, and save the fingerprint and final response before a later retry can replay it:</p>
<pre><code class="hljs language-sql" tabindex="0"><span class="hljs-keyword">CREATE TABLE</span> idempotency_responses (
  account_id uuid <span class="hljs-keyword">NOT NULL</span>,          <span class="hljs-comment">-- from authentication, never from the request body</span>
  idempotency_key text <span class="hljs-keyword">NOT NULL</span>,
  request_fingerprint text <span class="hljs-keyword">NOT NULL</span>,
  status_code <span class="hljs-type">integer</span>,
  response_body jsonb,
  completed_at timestamptz,
  <span class="hljs-keyword">PRIMARY KEY</span> (account_id, idempotency_key)
);
</code></pre><p>Every lookup, insert and replay filters on both columns. The <a href="https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/" rel="noopener noreferrer">Idempotency-Key draft</a> calls out data leaks from shared keys for the same reason.</p>
<p>Use a transaction or equivalent concurrency control to insert the key, create the order, and save the response. A matching completed row returns <code>status_code</code> and <code>response_body</code>; a differing fingerprint returns a client error; a matching unfinished row should tell the client the first request is still processing. The same pattern matters for webhooks, where a delivery can be retried after your handler completes. See <a href="https://devops-daily.com/posts/how-stripe-avoids-double-charging-idempotency-keys">how Stripe avoids double charging</a> and <a href="https://devops-daily.com/posts/reliable-webhook-delivery-retries-signatures-idempotency">reliable webhook delivery</a>.</p>
<blockquote>
<p><strong>Tip</strong></p>
<p><strong><a href="https://devops-daily.com/games/idempotency-retry-lab">Reproduce it in the Idempotency and Retry Lab</a></strong></p>
<ol>
<li>Choose <strong>Run duplicate checkout</strong> and inspect the ledger after the lost response.</li>
<li>Choose <strong>Run keyed checkout</strong> and find the saved result in <strong>Idempotency store</strong> instead of another charge.</li>
<li>Press <strong>Reuse key with new amount</strong>, then <strong>Send two at once</strong>, and compare the mismatch and in-progress outcomes.</li>
<li>Run <strong>Run no backoff</strong> and <strong>Run full jitter</strong>, then use <strong>Run down API job</strong> to inspect the dead letter.</li>
</ol>
<p>The outage chart is a simplified deterministic model of client arrivals, not a production measurement. Its dead-letter jobs retain a key and final error and do not retry themselves.</p>
</blockquote>
<h2>A shared DataLoader exposes data from the previous request</h2><p>GraphQL makes N+1 look harmless. <code>posts { author { name } }</code> reads like one query, but a resolver that fetches every author separately can produce one posts query plus one author query per post. Add nested comments and comment authors, and the work grows at each resolver level.</p>
<p>The <a href="https://github.com/graphql/dataloader#batching" rel="noopener noreferrer">DataLoader README</a> describes batching loads that occur in the same execution frame and caching values in a loader instance. The cache is intentionally not a shared application cache. The README's <a href="https://github.com/graphql/dataloader#caching-per-request" rel="noopener noreferrer">per-request caching guidance</a> says to create new loader instances for each request because authorization or other request-specific context can differ.</p>
<p>Create loaders in the GraphQL context, not at module scope. This Apollo Server pattern puts the loader beside the authenticated request context:</p>
<pre><code class="hljs language-typescript" tabindex="0"><span class="hljs-keyword">import</span> { <span class="hljs-title class_">ApolloServer</span> } <span class="hljs-keyword">from</span> <span class="hljs-string">'@apollo/server'</span>;
<span class="hljs-keyword">import</span> <span class="hljs-title class_">DataLoader</span> <span class="hljs-keyword">from</span> <span class="hljs-string">'dataloader'</span>;
<span class="hljs-keyword">import</span> { startStandaloneServer } <span class="hljs-keyword">from</span> <span class="hljs-string">'@apollo/server/standalone'</span>;

<span class="hljs-keyword">const</span> server = <span class="hljs-keyword">new</span> <span class="hljs-title class_">ApolloServer</span>({ typeDefs, resolvers });

<span class="hljs-keyword">await</span> <span class="hljs-title function_">startStandaloneServer</span>(server, {
  <span class="hljs-attr">context</span>: <span class="hljs-title function_">async</span> ({ req }) =&gt; {
    <span class="hljs-keyword">const</span> viewer = <span class="hljs-keyword">await</span> <span class="hljs-title function_">authenticate</span>(req.<span class="hljs-property">headers</span>.<span class="hljs-property">authorization</span>);
    <span class="hljs-keyword">return</span> {
      viewer,
      <span class="hljs-comment">// A new loader per request: its cache dies with the request.</span>
      <span class="hljs-attr">userById</span>: <span class="hljs-keyword">new</span> <span class="hljs-title class_">DataLoader</span>(<span class="hljs-title function_">async</span> (ids) =&gt; {
        <span class="hljs-comment">// Authorization lives in the data access: only rows this viewer may see.</span>
        <span class="hljs-keyword">const</span> users = <span class="hljs-keyword">await</span> <span class="hljs-title function_">loadUsersVisibleTo</span>(viewer, ids);
        <span class="hljs-keyword">return</span> ids.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">id</span>) =&gt;</span> users.<span class="hljs-title function_">find</span>(<span class="hljs-function">(<span class="hljs-params">user</span>) =&gt;</span> user.<span class="hljs-property">id</span> === id) ?? <span class="hljs-literal">null</span>);
      }),
    };
  },
});
</code></pre><p><code>loadUsersVisibleTo</code> stands in for your own query, for example <code>SELECT id, name FROM users WHERE id = ANY($1::uuid[])</code> plus the viewer's visibility rule. A per-request loader stops one request's cache from serving the next; it does not authorize reads inside a request, so the permission check has to be in the batch function.</p>
<p>Without the loader, resolver logs often show repeated <code>SELECT ... WHERE id = $1</code> statements. With it, same-tick loads become one <code>WHERE id IN (...)</code> query or its parameterized equivalent. Deduplication reduces repeated keys in that batch, but it does not authorize the cached result for another request. Log query count and the request ID together so a shared cache is visible before it becomes a data exposure.</p>
<blockquote>
<p><strong>Tip</strong></p>
<p><strong><a href="https://devops-daily.com/games/graphql-dataloader-lab">Reproduce it in the GraphQL N+1 and DataLoader Lab</a></strong></p>
<ol>
<li>Leave <strong>Post authors</strong> on, keep <strong>Use User DataLoader</strong> off, and press <strong>Run query</strong>. Inspect the SQL panel.</li>
<li>Turn on <strong>Comments</strong> and <strong>Comment authors</strong>, then run again to see the nested resolver work.</li>
<li>Turn on <strong>Use User DataLoader</strong> and <strong>Use Comments DataLoader</strong> to see the batched <code>IN (...)</code> queries.</li>
<li>Set <strong>Loader scope</strong> to <strong>Shared by all requests</strong>, run as Ada, then run as Bo. Set it back to <strong>Per request</strong> and run Bo again.</li>
</ol>
<p>The lab has a small deterministic data set. It models one event-loop tick per resolver level; <code>await</code> calls between loads can split real DataLoader batches.</p>
</blockquote>
<h2>Keep the state boundary explicit</h2><p>These bugs look unrelated until you name the state boundary. A CDN decides which request fields identify a response. TCP decides which bytes must arrive before later bytes are delivered. DMARC decides which authenticated identifier may stand for <code>From</code>. An idempotency store decides whether a retry is an earlier operation. DataLoader decides how long a loaded value remains valid.</p>
<p>Use the checks before an incident, then use the labs to make the failure shape memorable.</p>
<table>
<thead>
<tr>
<th>Bug</th>
<th>Mechanism</th>
<th>One-line fix</th>
<th>Lab</th>
</tr>
</thead>
<tbody><tr>
<td>Personal content at the edge</td>
<td>Shared cache key or directives allow storage</td>
<td>Send <code>private</code> or <code>no-store</code> for personalized responses</td>
<td><a href="https://devops-daily.com/games/cdn-edge-cache-lab">CDN Edge Cache Lab</a></td>
</tr>
<tr>
<td>HTTP/2 page stalls on loss</td>
<td>TCP in-order delivery blocks active streams</td>
<td>Test QUIC and HTTP/3 where loss and RTT warrant it</td>
<td><a href="https://devops-daily.com/games/http-versions-lab">HTTP Versions Lab</a></td>
</tr>
<tr>
<td>DMARC fails after an intermediary</td>
<td>SPF or DKIM no longer passes and aligns</td>
<td>Preserve an aligned identity through forwarding and lists</td>
<td><a href="https://devops-daily.com/games/email-auth-lab">Email Authentication Lab</a></td>
</tr>
<tr>
<td>Duplicate work after retry</td>
<td>New key makes a retry a new operation</td>
<td>Reuse one key and replay its saved response</td>
<td><a href="https://devops-daily.com/games/idempotency-retry-lab">Idempotency and Retry Lab</a></td>
</tr>
<tr>
<td>GraphQL data leak or query storm</td>
<td>Loader cache crosses requests, or loads are unbatched</td>
<td>Instantiate loaders in request context</td>
<td><a href="https://devops-daily.com/games/graphql-dataloader-lab">GraphQL N+1 and DataLoader Lab</a></td>
</tr>
</tbody></table>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[CKNE Is Live: What the Kubernetes Network Engineer Exam Tests and How to Practice]]></title>
      <link>https://devops-daily.com/posts/ckne-kubernetes-network-engineer-exam-guide</link>
      <description><![CDATA[CKNE is a new hands-on Kubernetes networking exam. Use its published domains to build a lab, rehearse packet-level debugging, and turn DNS, traffic, policy, encryption, and observability into timed terminal drills.]]></description>
      <pubDate>Fri, 13 Nov 2026 07:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/ckne-kubernetes-network-engineer-exam-guide</guid>
      <category><![CDATA[Kubernetes]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Kubernetes]]></category><category><![CDATA[Networking]]></category><category><![CDATA[Certification]]></category><category><![CDATA[CKNE]]></category><category><![CDATA[Gateway API]]></category><category><![CDATA[Network Policy]]></category><category><![CDATA[CNCF]]></category>
      <content:encoded><![CDATA[<p>Kubernetes networking failures rarely announce which layer is wrong. A Service can have no ready endpoints, a policy can block DNS, a route can point at the wrong interface, or a Gateway can accept a listener but reject the route. The new <strong>Certified Kubernetes Network Engineer (CKNE)</strong> asks you to find and fix problems like these from a terminal under time pressure. This guide turns the published curriculum into drills you can repeat in a disposable cluster.</p>
<p>This is not an account of taking the exam. It is a study guide built from the <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer">published CKNE curriculum</a> and the <a href="https://www.cncf.io/announcements/2026/11/10/cncf-launches-certified-kubernetes-network-engineer-certification-ckne-to-advance-production-ready-networking-skills-for-ai-era/" rel="noopener noreferrer">CNCF launch announcement</a>. Treat every lab as a loop: break one thing deliberately, prove where the packet stops, repair it, then retest from the client.</p>
<h2>TLDR</h2><ul>
<li>CKNE is a performance-based, two-hour exam in a Linux remote desktop. Its largest domains are Service Networking and DNS, and Network Security and Policy, at 25% each.</li>
<li>Start with CKA-level cluster administration. There are no formal prerequisites, but both CNCF and Linux Foundation Education say the exam assumes that baseline.</li>
<li>Build one lab with a real policy-enforcing CNI. Then practice resolving a name, examining EndpointSlices, reading node routes, applying a policy, and observing the resulting flow without reaching for a GUI first.</li>
<li>As of 2026-11-11, the official exam page does not publish a passing score, Kubernetes version, or number of tasks. Check the <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer">candidate handbook and FAQ</a> before booking.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>CKA-level comfort with <code>kubectl</code>: contexts, namespaces, Deployments, Services, logs, and <code>describe</code>.</li>
<li>A workstation that can run a three-node <a href="https://kind.sigs.k8s.io/" rel="noopener noreferrer">kind</a> cluster in Docker or Podman.</li>
<li><code>kubectl</code>, <code>kind</code>, the <a href="https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/" rel="noopener noreferrer">Cilium CLI</a>, and the <a href="https://docs.cilium.io/en/stable/observability/hubble/setup/" rel="noopener noreferrer">Hubble CLI</a> installed.</li>
<li>Basic Linux networking: <code>ip</code>, <code>ss</code>, <code>tcpdump</code>, and reading a route table.</li>
</ul>
<h2>What CKNE tests, and whether it fits you</h2><p>CKNE is for an administrator who already knows how to inspect workloads and cluster resources, and now needs to reason about the data path between them. It has no prerequisite certification, but doing CKA first is a sensible order because a networking drill should not also be a <code>kubectl</code> syntax drill.</p>
<p>The <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer">Linux Foundation certification page</a> lists the exam-only price as $445, the exam plus THRIVE-ONE annual subscription as $625, and the exam plus Certification Pathway as $645. Eligibility lasts 12 months, one retake is included, and the certification is valid for two years. Those are purchase decisions, not study requirements. The useful practice resource bundled with the exam is the <a href="https://killer.sh/ckne" rel="noopener noreferrer">Killer.sh CKNE simulator</a>: two sessions, each open for 36 hours after activation. (The Linux Foundation page and Killer.sh currently give different question counts per session, 17 and 20, so do not plan around an exact number.) Use those sessions late in preparation to test navigation, pacing, and recovery after a bad assumption, not as your first exposure to a CNI or Gateway API.</p>
<p><strong>CKNE published domain weights</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Core Infrastructure and CNI</td>
<td>15%</td>
</tr>
<tr>
<td>Service Networking and DNS</td>
<td>25%</td>
</tr>
<tr>
<td>Advanced Traffic Management</td>
<td>20%</td>
</tr>
<tr>
<td>Network Security and Policy</td>
<td>25%</td>
</tr>
<tr>
<td>Observability</td>
<td>15%</td>
</tr>
</tbody></table>
<p><em>Published by Linux Foundation Education and CNCF. Checked 2026-11-11.</em></p>
<p>Warm up concepts in the browser before you open the terminal. The <a href="https://devops-daily.com/games/kubernetes-networking-cni-simulator">Kubernetes networking and CNI simulator</a> traces pod, Service, policy, egress, and ingress paths across overlay, routed, and eBPF dataplanes. Use the <a href="https://devops-daily.com/games/dns-simulator">DNS simulator</a> for resolver and cache order, <a href="https://devops-daily.com/games/packet-journey">Packet Journey</a> for packet headers and hops, and <a href="https://devops-daily.com/games/tcp-vs-udp">TCP vs UDP</a> when you need to refresh transport behavior. The <a href="https://devops-daily.com/games/load-balancer-simulator">load balancer simulator</a> is useful for endpoint-selection intuition. For mTLS and policy, use the <a href="https://devops-daily.com/games/service-mesh-simulator">service mesh simulator</a> and <a href="https://devops-daily.com/games/ssl-tls-handshake">SSL/TLS handshake simulator</a>. They are warm-ups, not substitutes for command-line verification.</p>
<h2>Build a practice cluster that can fail interestingly</h2><p>You need a cluster where NetworkPolicy and encryption actually have an effect. <code>kind</code> can start without its default CNI, and Cilium can then supply policy enforcement and WireGuard encryption. The <a href="https://kind.sigs.k8s.io/docs/user/configuration/" rel="noopener noreferrer">kind configuration reference</a> documents both <code>disableDefaultCNI</code> and <code>podSubnet</code>; Cilium documents the encryption settings below. To compare Cilium with the other common policy-enforcing CNI, see <a href="https://devops-daily.com/comparisons/cilium-vs-calico">Cilium vs Calico</a>.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># ckne-kind.yaml</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">Cluster</span>
<span class="hljs-attr">apiVersion:</span> <span class="hljs-string">kind.x-k8s.io/v1alpha4</span>
<span class="hljs-attr">networking:</span>
  <span class="hljs-attr">disableDefaultCNI:</span> <span class="hljs-literal">true</span>
  <span class="hljs-attr">podSubnet:</span> <span class="hljs-string">'10.244.0.0/16'</span>
<span class="hljs-attr">nodes:</span>
  <span class="hljs-bullet">-</span> <span class="hljs-attr">role:</span> <span class="hljs-string">control-plane</span>
  <span class="hljs-bullet">-</span> <span class="hljs-attr">role:</span> <span class="hljs-string">worker</span>
  <span class="hljs-bullet">-</span> <span class="hljs-attr">role:</span> <span class="hljs-string">worker</span>
</code></pre><pre><code class="hljs language-bash" tabindex="0">kind create cluster --name ckne --config ckne-kind.yaml

<span class="hljs-comment"># Installs Cilium, turns on policy enforcement, and encrypts eligible</span>
<span class="hljs-comment"># cross-node Cilium endpoint traffic with WireGuard.</span>
cilium install \
  --<span class="hljs-built_in">set</span> encryption.enabled=<span class="hljs-literal">true</span> \
  --<span class="hljs-built_in">set</span> encryption.type=wireguard
cilium status --<span class="hljs-built_in">wait</span>

<span class="hljs-comment"># Hubble flow visibility for the observability drills</span>
cilium hubble <span class="hljs-built_in">enable</span>
cilium hubble port-forward &amp;
hubble status
</code></pre><p>WireGuard encrypts Cilium-managed traffic between nodes, not traffic on the same node. Confirm the control plane before trusting the lab:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl -n kube-system <span class="hljs-built_in">exec</span> ds/cilium -- cilium-dbg status | grep Encryption
kubectl get nodes -o wide
kubectl get pods -A
</code></pre><p>For a second interface exercise, install Multus and create a <code>NetworkAttachmentDefinition</code>, whose API version is <code>k8s.cni.cncf.io/v1</code>. A pod requests it with the <code>k8s.v1.cni.cncf.io/networks</code> annotation. The <a href="https://k8snetworkplumbingwg.github.io/multus-cni/docs/quickstart.html" rel="noopener noreferrer">Multus quickstart</a> shows this attachment model and verifies the additional <code>net1</code> interface. Do not memorize the YAML in isolation: check <code>ip addr</code> and <code>ip route</code> in the pod and ask which interface owns the default route.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">k8s.cni.cncf.io/v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">NetworkAttachmentDefinition</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">lab-net</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">config:</span> <span class="hljs-string">'{"cniVersion":"1.0.0","type":"macvlan","master":"eth0","mode":"bridge","ipam":{"type":"host-local","subnet":"192.0.2.0/24"}}'</span>
<span class="hljs-meta">---</span>
<span class="hljs-attr">apiVersion:</span> <span class="hljs-string">v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">Pod</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">dual-homed</span>
  <span class="hljs-attr">annotations:</span>
    <span class="hljs-attr">k8s.v1.cni.cncf.io/networks:</span> <span class="hljs-string">lab-net</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">containers:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">shell</span>
      <span class="hljs-attr">image:</span> <span class="hljs-string">busybox</span>
      <span class="hljs-attr">command:</span> [<span class="hljs-string">'sh'</span>, <span class="hljs-string">'-c'</span>, <span class="hljs-string">'sleep 3600'</span>]
</code></pre><p>The NAD is only a definition. <code>master</code> must name an interface that exists on every node that can schedule the Pod (kind nodes have <code>eth0</code>; check yours in the node's host namespace first), and the <code>macvlan</code> and <code>host-local</code> plugin binaries must be present in the node's CNI directory. Use the Multus documentation for the plugin you choose. That distinction is a good troubleshooting habit.</p>
<h2>Practice every published domain as a terminal drill</h2><h3>Core Infrastructure and CNI: trace before changing</h3><p>Know how pod CIDRs are allocated by your lab CNI, which node owns a destination CIDR, and whether traffic is routed or encapsulated. Under a clock, start with <code>kubectl get nodes -o wide</code>, pod IPs, and routes before changing any YAML.</p>
<p>Use an ephemeral node debug container to inspect the host network namespace. <code>ip route</code> and <code>ip addr</code> tell you what the node believes. A capture on port 53 separates a DNS transport problem from a bad record or bad search path.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Open a shell in the node's network namespace (pick an image that ships tcpdump)</span>
NODE=<span class="hljs-string">"<span class="hljs-subst">$(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')</span>"</span>
kubectl debug <span class="hljs-string">"node/<span class="hljs-variable">${NODE}</span>"</span> -it --image=nicolaka/netshoot --profile=sysadmin

<span class="hljs-comment"># What does the node believe about addresses and routes?</span>
ip addr
ip route

<span class="hljs-comment"># Watch DNS on every interface while a pod resolves a name</span>
tcpdump -ni any port 53
</code></pre><p>Be able to distinguish kube-proxy modes from CNI service replacements. Inspect the actual cluster configuration before assuming <code>iptables</code> rules explain a Service. Kubernetes supports <code>iptables</code> and <code>nftables</code> modes, while the <a href="https://devops-daily.com/posts/kubernetes-1-37-garhwal-what-shipped">Kubernetes 1.37 release guide</a> covers why <code>ipvs</code> is deprecated and should not be the new thing you learn for this exam.</p>
<h3>Service Networking and DNS: prove the backend set</h3><p>For every Service type, state what it exposes and how a client reaches it: <code>ClusterIP</code> is cluster-internal virtual addressing, <code>NodePort</code> exposes a port on node addresses, <code>LoadBalancer</code> relies on an implementation to expose it externally, and a headless Service has no cluster IP for virtual-IP proxying. Then inspect the real backend set:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl get svc payments -o yaml
kubectl get endpointslices \
  -l kubernetes.io/service-name=payments \
  -o yaml
kubectl get pods -l app=payments
</code></pre><p>EndpointSlices are the source of Service backends that a proxy watches, and a Pod's readiness maps to the endpoint's readiness condition. Make one backend unready and watch the slice change. For external traffic, set <code>externalTrafficPolicy: Local</code> on a NodePort or LoadBalancer test Service, send traffic to a node without a local endpoint, and explain the drop. <code>Local</code> preserves source IP by forwarding only to node-local endpoints, so it exchanges reachability on every node for that property. The <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noopener noreferrer">Kubernetes Service documentation</a> and <a href="https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/" rel="noopener noreferrer">EndpointSlice documentation</a> define those behaviors.</p>
<p>DNS should be its own timed drill. From a pod, resolve a full Service name, inspect <code>/etc/resolv.conf</code>, then try the short name. Kubernetes commonly configures search domains and <code>options ndots:5</code>; read the file in the workload instead of assuming a particular setting.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># The dnsutils pod from the Kubernetes DNS debugging guide</span>
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl <span class="hljs-built_in">exec</span> -it dnsutils -- <span class="hljs-built_in">cat</span> /etc/resolv.conf
kubectl <span class="hljs-built_in">exec</span> -it dnsutils -- nslookup kubernetes.default.svc.cluster.local
kubectl <span class="hljs-built_in">exec</span> -it dnsutils -- nslookup kubernetes.default

kubectl -n kube-system get configmap coredns -o yaml
kubectl -n kube-system logs -l k8s-app=kube-dns
</code></pre><p>For a configuration drill, edit the CoreDNS ConfigMap in a disposable cluster and add a name rewrite inside the existing server block. Its position in the Corefile does not matter: CoreDNS fixes the <a href="https://coredns.io/manual/configuration/" rel="noopener noreferrer">plugin execution order</a> when it is built. <code>answer auto</code> rewrites the response back to the name the client asked for, which the <a href="https://coredns.io/plugins/rewrite/" rel="noopener noreferrer">rewrite plugin</a> requires for suffix rewrites to look right to the client.</p>
<pre><code class="hljs language-text" tabindex="0">rewrite stop {
    name suffix .example.internal. .default.svc.cluster.local.
    answer auto
}
</code></pre><p>That maps <code>payments.example.internal</code> to <code>payments.default.svc.cluster.local</code>. Restart the CoreDNS deployment if your distribution does not reload the Corefile, then test both names. Also find its metrics endpoint in the installed Corefile before deciding that a dashboard is a DNS diagnosis. Kubernetes has a useful <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noopener noreferrer">DNS debugging sequence</a> for this exact progression.</p>
<h3>Advanced Traffic Management: make the controller part explicit</h3><p>Install the Gateway API Standard channel from its current release URL, then use the <code>GatewayClass</code> supplied by the controller you installed. CRDs alone do not route traffic. The <a href="https://gateway-api.sigs.k8s.io/guides/getting-started/introduction/" rel="noopener noreferrer">Gateway API installation guide</a> publishes this current Standard-channel command:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0/standard-install.yaml
</code></pre><p>Next, create a path match and a weighted split. Replace <code>&lt;gateway-class&gt;</code> with an installed class and create <code>api-tls</code> as a TLS Secret before applying the Gateway. The listener's Secret reference is the important TLS troubleshooting surface.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">gateway.networking.k8s.io/v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">Gateway</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">public</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">gatewayClassName:</span> <span class="hljs-string">&lt;gateway-class&gt;</span>
  <span class="hljs-attr">listeners:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">https</span>
      <span class="hljs-attr">protocol:</span> <span class="hljs-string">HTTPS</span>
      <span class="hljs-attr">port:</span> <span class="hljs-number">443</span>
      <span class="hljs-attr">hostname:</span> <span class="hljs-string">api.example.com</span>
      <span class="hljs-attr">tls:</span>
        <span class="hljs-attr">mode:</span> <span class="hljs-string">Terminate</span>
        <span class="hljs-attr">certificateRefs:</span>
          <span class="hljs-bullet">-</span> <span class="hljs-attr">kind:</span> <span class="hljs-string">Secret</span>
            <span class="hljs-attr">name:</span> <span class="hljs-string">api-tls</span>
      <span class="hljs-attr">allowedRoutes:</span>
        <span class="hljs-attr">namespaces:</span>
          <span class="hljs-attr">from:</span> <span class="hljs-string">Same</span>
<span class="hljs-meta">---</span>
<span class="hljs-attr">apiVersion:</span> <span class="hljs-string">gateway.networking.k8s.io/v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">HTTPRoute</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">api</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">parentRefs:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">public</span>
  <span class="hljs-attr">hostnames:</span> [<span class="hljs-string">'api.example.com'</span>]
  <span class="hljs-attr">rules:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">matches:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">path:</span>
            <span class="hljs-attr">type:</span> <span class="hljs-string">PathPrefix</span>
            <span class="hljs-attr">value:</span> <span class="hljs-string">/v1</span>
      <span class="hljs-attr">backendRefs:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">api-v1</span>
          <span class="hljs-attr">port:</span> <span class="hljs-number">8080</span>
          <span class="hljs-attr">weight:</span> <span class="hljs-number">90</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">api-v2</span>
          <span class="hljs-attr">port:</span> <span class="hljs-number">8080</span>
          <span class="hljs-attr">weight:</span> <span class="hljs-number">10</span>
</code></pre><p>Check <code>kubectl describe gateway public</code> and <code>kubectl describe httproute api</code>; accepted objects and programmed addresses are separate questions. If you are migrating from Ingress, see <a href="https://devops-daily.com/posts/ingress-nginx-eol-gateway-api-migration">our Gateway API migration guide</a> for a controller-aware cutover.</p>
<p>LLM traffic belongs here, but do not reduce it to weighted HTTP backends. The <a href="https://gateway-api-inference-extension.sigs.k8s.io/" rel="noopener noreferrer">Gateway API Inference Extension</a> currently documents the <code>InferencePool</code> API (v1) and an alpha <code>InferencePoolImport</code> and describes endpoint selection using model-serving metrics and capabilities. Practice explaining why model-aware selection can consider a model name, cache state, or adapter availability. Treat its APIs as extension APIs and verify your controller's support and release status before using them in an answer.</p>
<p>For cluster exit traffic, Cilium's implementation is <a href="https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/" rel="noopener noreferrer">Egress Gateway</a>. Enable it with <code>egressGateway.enabled=true</code> (it also needs <code>bpf.masquerade=true</code> and <code>kubeProxyReplacement=true</code>, which the lab install above does not set), then write a <code>CiliumEgressGatewayPolicy</code> that selects source pods and external destination CIDRs, plus a gateway node and egress IP or interface. Egress Gateway is not compatible with Cilium ClusterMesh. For cross-cluster service discovery, the <a href="https://multicluster.sigs.k8s.io/concepts/multicluster-services-api/" rel="noopener noreferrer">Multi-Cluster Services API</a> uses <code>ServiceExport</code> to mark a Service for export and creates <code>ServiceImport</code> plus EndpointSlices in consuming clusters. Cilium ClusterMesh is another implementation path; before working through it, read <a href="https://devops-daily.com/posts/cilium-1-19-clustermesh-policy-flip">the policy-default change that can block cross-cluster traffic</a>.</p>
<h3>Network Security and Policy: deny, restore DNS, then prove it</h3><p>Start a namespace with both ingress and egress default-deny. Then permit only what the workload needs. The classic failure is to allow application traffic while forgetting DNS egress.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">networking.k8s.io/v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">NetworkPolicy</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">default-deny</span>
  <span class="hljs-attr">namespace:</span> <span class="hljs-string">lab</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">podSelector:</span> {}
  <span class="hljs-attr">policyTypes:</span> [<span class="hljs-string">Ingress</span>, <span class="hljs-string">Egress</span>]
<span class="hljs-meta">---</span>
<span class="hljs-attr">apiVersion:</span> <span class="hljs-string">networking.k8s.io/v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">NetworkPolicy</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">allow-dns</span>
  <span class="hljs-attr">namespace:</span> <span class="hljs-string">lab</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">podSelector:</span> {}
  <span class="hljs-attr">policyTypes:</span> [<span class="hljs-string">Egress</span>]
  <span class="hljs-attr">egress:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">to:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">namespaceSelector:</span>
            <span class="hljs-attr">matchLabels:</span>
              <span class="hljs-attr">kubernetes.io/metadata.name:</span> <span class="hljs-string">kube-system</span>
          <span class="hljs-attr">podSelector:</span>
            <span class="hljs-attr">matchLabels:</span>
              <span class="hljs-attr">k8s-app:</span> <span class="hljs-string">kube-dns</span>
      <span class="hljs-attr">ports:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">protocol:</span> <span class="hljs-string">UDP</span>
          <span class="hljs-attr">port:</span> <span class="hljs-number">53</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">protocol:</span> <span class="hljs-string">TCP</span>
          <span class="hljs-attr">port:</span> <span class="hljs-number">53</span>
</code></pre><p>The label and namespace are cluster conventions, so inspect your DNS Service and Pods before applying this unchanged. Test from a labeled client pod, first expecting a timeout, then allowing the target Service on its actual port and running <code>curl -sv "http://${SERVICE}.${NAMESPACE}.svc.cluster.local:${PORT}/"</code> with those three variables set to your target. Kubernetes defines NetworkPolicy as an API, but the chosen network plugin must enforce it. The <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="noopener noreferrer">NetworkPolicy documentation</a> is the baseline for the API semantics.</p>
<p>Encryption is not proof of authorization. In this Cilium lab, confirm WireGuard with <code>cilium-dbg status | grep Encryption</code>, and capture on <code>cilium_wg0</code> from a Cilium pod if you need to see the tunnel. Cilium documents the status check, tunnel device, and the fact that same-node traffic is not encrypted in its <a href="https://docs.cilium.io/en/stable/security/network/encryption-wireguard/" rel="noopener noreferrer">WireGuard guide</a>.</p>
<p>For pod-level identity, use a service mesh. An Istio <code>AuthorizationPolicy</code> can allow only a workload identity after mTLS identifies the caller. This is a compact policy drill, not a complete mesh install:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">security.istio.io/v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">AuthorizationPolicy</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">allow-frontend</span>
  <span class="hljs-attr">namespace:</span> <span class="hljs-string">lab</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">selector:</span>
    <span class="hljs-attr">matchLabels:</span>
      <span class="hljs-attr">app:</span> <span class="hljs-string">payments</span>
  <span class="hljs-attr">action:</span> <span class="hljs-string">ALLOW</span>
  <span class="hljs-attr">rules:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">source:</span>
            <span class="hljs-attr">principals:</span>
              <span class="hljs-bullet">-</span> <span class="hljs-string">cluster.local/ns/lab/sa/frontend</span>
</code></pre><p>Verify the calling ServiceAccount and the mesh's mTLS mode before deciding a denial is an authorization error. Istio documents this defense-in-depth mTLS pattern in its <a href="https://istio.io/latest/docs/ops/configuration/security/security-policy-examples/" rel="noopener noreferrer">security policy examples</a>. For route behavior, retries, and traffic splits in a mesh, revisit <a href="https://devops-daily.com/posts/istio-traffic-management-routing-retries-circuit-breaking">our Istio traffic-management guide</a>.</p>
<h3>Observability: observe the flow that proves the fix</h3><p>Your final action should produce evidence. For DNS, inspect CoreDNS logs and the Prometheus metrics configured by its <code>prometheus</code> plugin. For Services, inspect EndpointSlices and, where relevant, kube-proxy metrics from the endpoint that your cluster exposes. For policy enforcement in Cilium, use Hubble to request the flows you care about:</p>
<pre><code class="hljs language-bash" tabindex="0">hubble observe --pod client --verdict DROPPED
hubble observe --namespace lab --protocol http
kubectl -n kube-system <span class="hljs-built_in">exec</span> ds/cilium -- cilium-dbg status
</code></pre><p>The <a href="https://docs.cilium.io/en/latest/observability/hubble/hubble-cli/" rel="noopener noreferrer">Hubble CLI documentation</a> verifies the <code>--pod</code> and <code>--verdict DROPPED</code> form. Trace a request through a Gateway by sending a request ID such as <code>X-Request-ID</code> from the client, then find that header in the gateway and application telemetry only if those components are configured to record it. A header alone does not create distributed tracing.</p>
<h2>A six-week plan and exam-day routine</h2><p>Use the weights to bias your time, while returning to old drills often enough that command recall becomes automatic.</p>
<table>
<thead>
<tr>
<th>Week</th>
<th>Focus</th>
<th>Deliverable</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Core Infrastructure and CNI</td>
<td>Explain one same-node and one cross-node packet path with routes and captures.</td>
</tr>
<tr>
<td>2</td>
<td>Service Networking and DNS</td>
<td>Repair an empty EndpointSlice, a readiness issue, and a broken DNS lookup.</td>
</tr>
<tr>
<td>3</td>
<td>Network Security and Policy</td>
<td>Build default-deny, restore DNS, allow one application path, and prove both allowed and denied flows.</td>
</tr>
<tr>
<td>4</td>
<td>Advanced Traffic Management</td>
<td>Configure Gateway plus HTTPRoute, TLS Secret reference, weighted backends, and one egress or multi-cluster concept.</td>
</tr>
<tr>
<td>5</td>
<td>Observability plus weak spots</td>
<td>Correlate CoreDNS, Service, Hubble, and Gateway evidence for one injected fault.</td>
</tr>
<tr>
<td>6</td>
<td>Timed mixed drills</td>
<td>Use the two Killer.sh sessions as realistic pacing checks and catalogue every command you had to look up.</td>
</tr>
</tbody></table>
<p>On exam day, the published format gives you 120 minutes in a Linux remote desktop with terminal and web interfaces. Spend the opening minutes inventorying contexts, namespaces, nodes, and the resources named by the task. Read status conditions before edits. Use a short loop for each task: observe, form one hypothesis, make the smallest change, and prove the result from the client side. Do not invent browser-tab or reference-material rules from hearsay. The official policies and candidate materials, not a study blog, decide those details.</p>
<p>The goal is not to memorize every CNI's settings. It is to see a network problem as a chain of claims you can test: name resolves, Service has ready endpoints, dataplane has a route, policy permits it, encryption is configured where expected, and logs or flows show the result.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Who's On Call for the AI Agent? A Production Runbook]]></title>
      <link>https://devops-daily.com/posts/who-is-on-call-for-the-ai-agent</link>
      <description><![CDATA[An agent that can call production tools is a service with an incident path, not a clever prompt. Assign an owner, page only for harm in progress, test a kill switch, scope its identity, and keep enough evidence to explain every action.]]></description>
      <pubDate>Thu, 12 Nov 2026 15:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/who-is-on-call-for-the-ai-agent</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[AI Agents]]></category><category><![CDATA[On-Call]]></category><category><![CDATA[Incident Response]]></category><category><![CDATA[Security]]></category><category><![CDATA[Observability]]></category>
      <content:encoded><![CDATA[<p>An AI agent that opens pull requests, queries a database, restarts a service, or sends email is already a production service. At 3 a.m., the question is not whether its prompt was well written. It is who can stop it, which identity it used, and how to tell what it did. This runbook gives that service the same operational basics as any other: ownership, a page policy, controls that still work when the model does not, and an audit trail that answers an incident question quickly.</p>
<h2>TLDR</h2><ul>
<li>Put every production agent in the service catalog with an owning team, a primary on-call rotation, an escalation path, and named owners for its prompt, tools, and credentials.</li>
<li>Page for harm that is happening now: an action loop, spend burning above budget, writes outside their window, permission-denied spikes, an aging approval queue, or failed tasks. Ticket slow trends such as evaluation, latency, and cost drift.</li>
<li>Make the kill switch independent of the model. The tool runner checks it before every call, credentials can be revoked, the gateway applies a cap, and network egress is the final containment layer.</li>
<li>Give the agent its own short-lived, read-scoped identity. A read-only database role helps, but does not limit query duration or call frequency.</li>
<li>Record every tool action outside the agent's writable storage. During an incident, preserve that record before deciding whether to resume.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A service catalog and an on-call system with a primary and escalation target.</li>
<li>A tool runner or gateway where you can enforce policy before an external action starts.</li>
<li>Metrics, centralized logs, and a protected audit-log destination.</li>
<li>An identity system that can issue a distinct identity for each agent.</li>
</ul>
<h2>Give the agent an owner before it gets a tool</h2><p>Treat an agent as a service, not as a feature hidden inside another service. Create a catalog entry that names its purpose, environments, repository, dashboards, runbook, primary rotation, and escalation path. A stale record is worse than a missing one when responders need to find the right team, which is why the catalog itself needs ownership and review. Our <a href="https://devops-daily.com/posts/service-catalog-drift">service-catalog drift check</a> shows how easily ownership records can stop matching reality.</p>
<p>Split accountability by control surface. One team may own the agent's prompt, model choice, and configuration. The team that owns each target system owns the tool contract and allowed actions. The identity or platform team owns credential issuance, revocation, and the gateway policy. The catalog entry should name all three, but it must name one accountable service owner who receives the page and drives the incident.</p>
<p>Write the escalation rule down with the service, not in a chat thread. The primary responder acknowledges and contains the agent. If the incident touches a target service, page that service's owner. Escalate to security when credentials, untrusted instructions, or data exposure are involved. The rotation and its escalation policy should have the same attention as any other production service. See the <a href="https://devops-daily.com/posts/on-call-rotation-escalation-policy-guide">on-call rotation and escalation guide</a> for the people and paging side of that setup.</p>
<h2>Page on harm now, ticket slow drift</h2><p>The page rule is simple: wake someone only when waiting makes the outcome worse. A tool loop can create more actions in the next minute. A spend burn rate can exhaust a budget this hour. A week-over-week latency change needs investigation, but it does not usually need an immediate phone call.</p>
<table>
<thead>
<tr>
<th>Signal</th>
<th>Page when</th>
<th>Ticket when</th>
</tr>
</thead>
<tbody><tr>
<td>Tool actions</td>
<td>Action rate is far above the service's baseline, or identical calls repeat</td>
<td>A gradual change in the action mix</td>
</tr>
<tr>
<td>Spend</td>
<td>The hourly burn rate crosses the incident budget</td>
<td>Cost trends upward across days or releases</td>
</tr>
<tr>
<td>Writes</td>
<td>A successful write occurs outside an approved window</td>
<td>The normal write volume slowly grows</td>
</tr>
<tr>
<td>Access</td>
<td>Permission-denied errors spike, which can indicate drift or a prompt-injection attempt</td>
<td>A single expected denial or a policy cleanup</td>
</tr>
<tr>
<td>Human approval</td>
<td>The oldest approval has exceeded the response promise</td>
<td>Approval volume trends higher</td>
</tr>
<tr>
<td>Task outcome</td>
<td>Failure rate crosses the service's error budget</td>
<td>Evaluation score or latency gradually declines</td>
</tr>
</tbody></table>
<p>The following names and thresholds are <strong>examples owned by this runbook</strong>, not Prometheus, OpenTelemetry, or vendor metric names. Pick thresholds from your own normal traffic and give every rule an owner. The counter functions use a range vector, and each alert has a <code>for:</code> duration. Note what <code>for:</code> does: it requires the expression to stay true across evaluations, so it filters flapping rate rules, but an event rule such as a single forbidden write stays true for as long as the event sits inside its range window. That is intended for the write and spend rules: one forbidden write is worth a page.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">groups:</span>
  <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">agent-production</span>
    <span class="hljs-attr">rules:</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentActionRateHigh</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id)</span> <span class="hljs-string">(rate(agent_tool_calls_total{outcome="success"}[5m]))</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">20</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">10m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'Agent action rate is above its configured baseline'</span>

      <span class="hljs-comment"># The runner increments this when the same tool + argument fingerprint</span>
      <span class="hljs-comment"># repeats inside its window. The fingerprint stays out of the label set.</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentRepeatedToolCallLoop</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id,</span> <span class="hljs-string">tool)</span> <span class="hljs-string">(rate(agent_repeated_tool_calls_total[5m]))</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">0.2</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">5m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'Repeated tool-call rate indicates a loop'</span>

      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentHourlySpendBudgetExceeded</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id)</span> <span class="hljs-string">(increase(agent_spend_usd_total[1h]))</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">25</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">5m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'Agent spend exceeded the hourly incident budget'</span>

      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentWriteOutsideWindow</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id,</span> <span class="hljs-string">tool)</span> <span class="hljs-string">(increase(agent_tool_calls_total{action="write",allowed_window="false",outcome="success"}[5m]))</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">0</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">1m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'Agent completed a write outside its allowed window'</span>

      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentPermissionDeniedSpike</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id)</span> <span class="hljs-string">(rate(agent_tool_calls_total{outcome="permission_denied"}[5m]))</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">1</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">10m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'Agent permission denials are spiking'</span>

      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentApprovalQueueOld</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">max</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id)</span> <span class="hljs-string">(agent_approval_queue_oldest_seconds)</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">300</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">5m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'An agent approval has waited more than five minutes'</span>

      <span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">AgentTaskFailureRateHigh</span>
        <span class="hljs-attr">expr:</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id)</span> <span class="hljs-string">(rate(agent_tasks_total{outcome="failure"}[10m]))</span> <span class="hljs-string">/</span> <span class="hljs-string">sum</span> <span class="hljs-string">by</span> <span class="hljs-string">(agent_id)</span> <span class="hljs-string">(rate(agent_tasks_total[10m]))</span> <span class="hljs-string">&gt;</span> <span class="hljs-number">0.20</span>
        <span class="hljs-attr">for:</span> <span class="hljs-string">15m</span>
        <span class="hljs-attr">labels:</span> { <span class="hljs-attr">severity:</span> <span class="hljs-string">page</span> }
        <span class="hljs-attr">annotations:</span>
          <span class="hljs-attr">summary:</span> <span class="hljs-string">'Agent task failure rate is above 20 percent'</span>
</code></pre><p>For the ticket queue, record evaluation score, latency percentiles, cost per completed task, and denied requests over a longer window. They tell you where to improve the service without training people to ignore their pager.</p>
<p>For traces, OpenTelemetry's GenAI conventions define <code>invoke_agent</code> and <code>execute_tool</code> operations and tool-call attributes. They now live in their own <a href="https://github.com/open-telemetry/semantic-conventions-genai" rel="noopener noreferrer">semantic-conventions-genai</a> repository, and the <a href="https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/" rel="noopener noreferrer">registry page</a> lists those operation values at <strong>Development</strong> stability, so version your dashboards against them. The same registry notes that <code>gen_ai.tool.call.arguments</code> may contain sensitive information. Keep raw arguments out of metric labels: a per-call value in a label is a cardinality problem as well as a leak.</p>
<h2>Build a kill switch that the model cannot bypass</h2><p>The model must not decide whether it is paused. Enforce a stop before every tool call in the runner that executes the call. Then give responders independent layers in case one control is unavailable.</p>
<ol>
<li><strong>Tool runner flag</strong> Checked before every call</li>
<li><strong>Credential revoke</strong> Identity can no longer authenticate</li>
<li><strong>Gateway cap</strong> Rate and budget stop</li>
<li><strong>Egress deny</strong> Last network containment layer</li>
</ol>
<p>Any on-call engineer may pull the switch, with no approval. That rule prevents a debate from extending an incident. "Pause" means calls that have not yet passed the runner's check are refused. A call that passed the check a moment before the switch can still start or finish, which is why credential revocation and egress deny exist as the next layers for immediate containment at the target. Pausing does not mean that the runner discards the current run, conversation, pending approval, or audit records. Keep that state inspectable so responders can see what the agent was about to do.</p>
<p>This short wrapper uses Redis as the control store. The switch is the existence of a key, so there is no value format to get wrong, and an unreachable or slow store stops the call instead of allowing it. Treat the store as a security control: connect over TLS, and give the runner an ACL user that can only read the pause keys, while responders use a separate user that can set and delete them (<a href="https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/" rel="noopener noreferrer">Redis ACLs</a>). Put the same check in every execution path, including retries and asynchronous workers.</p>
<pre><code class="hljs language-python" tabindex="0"><span class="hljs-keyword">import</span> os

<span class="hljs-keyword">from</span> redis <span class="hljs-keyword">import</span> Redis
<span class="hljs-keyword">from</span> redis.exceptions <span class="hljs-keyword">import</span> RedisError

<span class="hljs-comment"># rediss:// URL with the runner's read-only ACL user. Short connect and read</span>
<span class="hljs-comment"># timeouts: a hung control store must fail closed quickly.</span>
control = Redis.from_url(
    os.environ[<span class="hljs-string">"AGENT_CONTROL_REDIS_URL"</span>],
    socket_timeout=<span class="hljs-number">0.5</span>,
    socket_connect_timeout=<span class="hljs-number">0.5</span>,
)

<span class="hljs-keyword">class</span> <span class="hljs-title class_">AgentPaused</span>(<span class="hljs-title class_ inherited__">RuntimeError</span>):
    <span class="hljs-keyword">pass</span>

<span class="hljs-keyword">def</span> <span class="hljs-title function_">run_tool</span>(<span class="hljs-params">agent_id: <span class="hljs-built_in">str</span>, tool_name: <span class="hljs-built_in">str</span>, arguments: <span class="hljs-built_in">dict</span>, call_tool</span>):
    <span class="hljs-keyword">try</span>:
        paused = control.exists(<span class="hljs-string">f"agent:<span class="hljs-subst">{agent_id}</span>:paused"</span>)
    <span class="hljs-keyword">except</span> RedisError <span class="hljs-keyword">as</span> error:
        <span class="hljs-comment"># A broken control plane must not permit an external action.</span>
        <span class="hljs-keyword">raise</span> AgentPaused(<span class="hljs-string">"kill-switch state is unavailable"</span>) <span class="hljs-keyword">from</span> error

    <span class="hljs-keyword">if</span> paused:
        <span class="hljs-keyword">raise</span> AgentPaused(<span class="hljs-string">f"<span class="hljs-subst">{agent_id}</span> is paused; refusing <span class="hljs-subst">{tool_name}</span>"</span>)

    <span class="hljs-keyword">return</span> call_tool(tool_name, arguments)
</code></pre><p>Pulling the switch is then one command that any responder can run, as the incident-control ACL user. Deleting the key is the deliberate resume step, and it belongs in the incident record with who ran it and why:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Pause (any on-call engineer)</span>
redis-cli --tls -h agent-control --user incident-control --askpass \
  SET agent:deploy-agent-prod:paused <span class="hljs-string">"paged-by-oncall <span class="hljs-subst">$(date -u +%FT%TZ)</span>"</span>

<span class="hljs-comment"># Resume (service owner, after the review in step 7 of the runbook below)</span>
redis-cli --tls -h agent-control --user incident-control --askpass \
  DEL agent:deploy-agent-prod:paused
</code></pre><p>Test all four layers in a monthly game day. Start a harmless, repeatable test action, pause the agent while it is active, revoke its identity, exhaust a test gateway budget, and verify that egress policy blocks the target. Record time to containment and any action that escaped after the first stop signal. A kill switch that only exists in documentation is not a control.</p>
<h2>Scope the identity, then bound the work</h2><p>Give each agent one identity. Do not hand an agent a human's long-lived token, and do not let several agents share a broad service identity. Use a read-scoped role for discovery work, short-lived credentials, and a separate path where a human approves a write before a runner obtains a write-capable identity.</p>
<p>For AWS, the runner can obtain a 15-minute role session with <code>AssumeRole</code>. The <a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html" rel="noopener noreferrer">AWS STS API reference</a> documents a <code>DurationSeconds</code> minimum of 900 seconds and a maximum controlled by the role's maximum session duration, which may be from one to twelve hours. The command deliberately requests the 900-second minimum.</p>
<pre><code class="hljs language-bash" tabindex="0">aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/AgentReadInventory \
  --role-session-name agent-inventory-run-20261112 \
  --duration-seconds 900 \
  --output json
</code></pre><p>The role policy below allows only an example inventory task. The calling workload must separately be trusted to assume this role. A write action should not add write verbs here. Route it to an approval service that records an approver, checks the change, and issues a different, time-limited identity only after approval.</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"Version"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2012-10-17"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"Statement"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span>
    <span class="hljs-punctuation">{</span>
      <span class="hljs-attr">"Sid"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"ReadInventoryOnly"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"Effect"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Allow"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"Action"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"ec2:DescribeInstances"</span><span class="hljs-punctuation">,</span> <span class="hljs-string">"cloudwatch:GetMetricData"</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"Resource"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"*"</span>
    <span class="hljs-punctuation">}</span>
  <span class="hljs-punctuation">]</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>For a database, use a distinct read-only role, a connection limit, a <code>statement_timeout</code>, and a gateway rate limit. Our <a href="https://devops-daily.com/posts/neon-read-only-role-agent-probes">read-only role probes</a> showed why the role alone is not enough: a 15-second agent read on the primary made an <code>ALTER TABLE</code> wait more than 11 seconds, and the app's inserts queued behind it, even though the agent never wrote anything. Sending agent reads to a read replica kept the primary clear in that test. The same probes found that nothing in the role bounds how often the agent repeats a legal read, and that the agent's session could run <code>SET statement_timeout = 0</code>. So enforce the timeout and the rate cap in the tool runner or gateway, where the agent cannot change them.</p>
<p>This is the practical answer to Excessive Agency, which the <a href="https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/" rel="noopener noreferrer">2026 OWASP Top 10 for LLM Applications</a> ranks third (<a href="https://genai.owasp.org/2026/09/01/owasp-genai-security-project-unveils-2026-top-10-for-llm-applications-new-agent-control-standard-and-sponsors-as-community-tops-30000-members/" rel="noopener noreferrer">announcement</a>). A model can be wrong or manipulated. The identity and policy boundary should still limit what reaches a real system.</p>
<h2>Keep an audit trail outside the agent's reach</h2><p>Log every attempted tool call as a structured event. Include the run ID, agent identity, credential or role, tool, redacted arguments, result, approver when one exists, timestamp, and trace ID. Store it in append-only storage with a retention period set by your security and compliance requirements. The agent's own workspace, database, or object-store prefix is not an acceptable sole copy because an agent with write access could alter the evidence.</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2026-11-12T02:18:44Z"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"run_id"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"run_01JCG7"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"agent_id"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"deploy-agent-prod"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"credential"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"arn:aws:sts::123456789012:assumed-role/AgentReadInventory/agent-inventory-run-20261112"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"tool"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"ec2.describe_instances"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"arguments"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"region"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"eu-west-1"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"filters"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"[REDACTED]"</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"result"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"success"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"approver"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">null</span></span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"trace_id"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"4bf92f3577b34da6a3ce929d0e0e4736"</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>To answer "what did the agent do between 02:10 and 02:40?", query the protected JSONL archive, not the model transcript:</p>
<pre><code class="hljs language-bash" tabindex="0">jq -r <span class="hljs-string">'
  select(
    .agent_id == "deploy-agent-prod" and
    .timestamp &gt;= "2026-11-12T02:10:00Z" and
    .timestamp &lt; "2026-11-12T02:40:00Z"
  ) |
  [.timestamp, .run_id, .tool, .result, .trace_id] | @tsv
'</span> agent-audit.jsonl
</code></pre><p>Redact secrets before an event leaves the runner. In our <a href="https://devops-daily.com/posts/prompt-injection-ops-agent-measured">prompt-injection test of an ops agent</a>, a check outside the model stopped every forbidden action, but the agent still copied secrets into its own reports, so redaction belongs in code, not in the prompt. Store a stable argument fingerprint for loop detection, but do not put raw prompts, credentials, tokens, or user data into an alert label. Use the trace ID to join the audit event to the request and tool spans.</p>
<h2>First 15 minutes: copy this runbook</h2><p>Put the following template beside the service catalog entry. Fill in the actual dashboard, credential-revocation command, audit location, and escalation targets before the agent goes live.</p>
<pre><code class="hljs language-text" tabindex="0">AI AGENT INCIDENT: FIRST 15 MINUTES

Service: &lt;agent service name&gt;
Primary on-call: &lt;rotation&gt;
Escalation: &lt;service owner&gt;, &lt;security contact&gt;, &lt;target-system owner&gt;

1. Acknowledge the page and start an incident record. Record the run ID, alert,
   responder, and current time.
2. Pull the kill switch. Confirm the tool runner refuses a test call and note the
   last tool action accepted after the pause time.
3. Revoke or disable the agent credential. Apply the gateway rate or budget cap.
   Use egress deny if actions may still reach an external target.
4. Preserve the agent run state, approval queue, traces, and append-only audit
   events. Do not delete the run or ask the agent to explain itself.
5. Assess blast radius from the audit trail: actions, targets, credential used,
   successful writes, denied calls, spend, and affected data or services.
6. Communicate status, containment time, known impact, and next update time to
   the owning team and target-system owner. Escalate security when appropriate.
7. Resume only after the owner identifies the trigger, corrects the control gap,
   tests it in a safe environment, and explicitly records the decision.
</code></pre><h2>Summary</h2><p>The best time to settle ownership and containment authority is before the first tool call. Once the agent is a cataloged service with a real pager, a tested stop path, scoped credentials, and a protected action record, responders can handle its failures as operational work instead of improvising around a model.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[We Tested AWS Emulator: It Blocks More Than Its Docs Say, and Less Than AWS Does]]></title>
      <link>https://devops-daily.com/posts/aws-emulator-hands-on-test</link>
      <description><![CDATA[AWS Emulator passed all 40 of our workflow checks and caught two IAM mistakes in our own test code. It also let a user with no policies assume an admin role, ignored permissions boundaries and session policies, and with the default JavaScript SDK settings, one SQS request went to real AWS. Here is what we measured on a throwaway droplet.]]></description>
      <pubDate>Thu, 12 Nov 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/aws-emulator-hands-on-test</guid>
      <category><![CDATA[AWS]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[AWS]]></category><category><![CDATA[IAM]]></category><category><![CDATA[Testing]]></category><category><![CDATA[Terraform]]></category><category><![CDATA[Local Development]]></category><category><![CDATA[AI Agents]]></category><category><![CDATA[CI/CD]]></category>
      <content:encoded><![CDATA[<p>AWS released AWS Emulator on 2026-11-11: a free local copy of the AWS APIs that runs on your laptop or in CI, with no AWS account. We installed v1.0.322 on a throwaway droplet and ran it through 40 workflow checks across 9 services, 32 permission cases, the AWS CLI, the JavaScript SDK and Terraform. <strong>All 40 workflow checks passed, and the emulator refused two calls our own test code made with roles that had no permissions, as AWS would.</strong> That is more permission checking than AWS's own security page describes. It is also less than AWS does: a user with no policies at all assumed an admin role and created an IAM user with it, and permissions boundaries and session policies changed nothing.</p>
<p>Three other things would bite a real test suite: the JavaScript SDK's SQS client sent a request to the real <code>sqs.us-east-1.amazonaws.com</code> by default, Terraform AWS provider 6.73.0 could not read back an S3 bucket, and the emulator refuses to hold more than 64 MiB in one bucket or 1,000 messages in one queue.</p>
<h2>TLDR</h2><ul>
<li>Install is one signed <code>.deb</code> (163.0 MiB) that unpacks to one 925.7 MiB Rust binary. It answered its health check in a median 383.5 ms on repeated starts and 1,325 ms with the page cache dropped, using 172.5 MiB of anonymous memory at idle.</li>
<li>Our scenario suite passed 40 of 40 checks: S3 presigned URLs, multipart and versioning; SQS visibility timeout, DLQ and redrive; DynamoDB conditional writes, transactions and streams; Lambda; EventBridge to SQS; SNS fan-out with a filter policy; KMS; Secrets Manager; Step Functions.</li>
<li>The emulator enforces identity policies, bucket policies, KMS key policies and secret policies for the users and roles you create in it. That caught a missing <code>s3:PutObject</code> and a missing <code>lambda:InvokeFunction</code> in our own test code.</li>
<li>It matched AWS's documented result in 27 of 32 permission cases. All 5 misses let through a request that AWS refuses, and they come from 4 causes: permissions boundaries, session policies, the caller's own <code>sts:AssumeRole</code> check (a user with no policies assumed an admin role and used it), and <code>ListBuckets</code> with no policy.</li>
<li>The JS SDK v3 SQS client follows the AWS-style queue URL the emulator returns and calls real AWS. Set <code>useQueueUrlAsEndpoint: false</code>.</li>
<li>Terraform AWS provider 6.73.0 fails on <code>aws_s3_bucket</code> because it reads tags through S3 Control. Provider 5.100.0 applied and destroyed the same 25-resource stack, with one plan diff left.</li>
<li>The repo has 29,684 commits, all by agent identities, and about 16.1 million lines of Rust. AWS's own fidelity scores for this release pass 56.5% of 42,937 scored checks.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Linux x86_64 machine you can throw away. We used a 4 vCPU / 8 GB DigitalOcean droplet with Ubuntu 24.04.</li>
<li>AWS CLI v2 and an SDK (we used boto3 1.44.1 and the JavaScript SDK v3).</li>
<li>Terraform if you want to test infrastructure code.</li>
<li>No AWS account, and no real AWS credentials anywhere on the machine. The emulator writes its own keys per run.</li>
</ul>
<p>Every script and raw result is in the companion repo:</p>
<p><a href="https://github.com/The-DevOps-Daily/aws-emulator-field-test" rel="noopener noreferrer">The-DevOps-Daily/aws-emulator-field-test on GitHub</a></p>
<h2>What AWS Emulator is, from the primary sources</h2><p>The <a href="https://aboutamazon.com/news/aws/how-autonomous-ai-agents-built-aws-emulator" rel="noopener noreferrer">launch post</a> gives 438 AWS API models. Werner Vogels' <a href="https://www.allthingsdistributed.com/2026/11/aws-emulator-and-the-engineers-job-when-agents-write-the-code.html" rel="noopener noreferrer">post</a> says "more than 400 API models covering more than 200 AWS services" and that "all of AWS Emulator's code was written and tested by agents." The method in both posts: agents write fidelity tests from the public Smithy models, fix each test until it passes against real AWS, then build the emulator until it passes the same test. Every change goes through a merge gate that includes several security agents doing adversarial reviews.</p>
<p>The <a href="https://docs.aws.amazon.com/emulator/latest/userguide/what-is-service.html" rel="noopener noreferrer">user guide</a> adds the parts that matter in practice:</p>
<ul>
<li>It is for building and testing, not production. It "doesn't evaluate AWS Identity and Access Management (IAM) policies and resource-based policies the way AWS does."</li>
<li>Behaviour is checked against AWS in <code>us-west-2</code> only, and global services in <code>us-east-1</code>.</li>
<li>One account (<code>000000000000</code>) and, in practice, one Region per emulator.</li>
<li>It serves HTTPS on <code>127.0.0.1:9297</code> with a new self-signed certificate and new access keys on every start.</li>
<li>Telemetry is on by default. The <a href="https://github.com/aws/aws-emulator" rel="noopener noreferrer">README</a> explains how to turn it off. We ran every test with <code>--telemetry off</code>.</li>
</ul>
<p>The code is CC0, except the <code>vendor/</code> directory, which is Apache 2.0. The repository does not take pull requests.</p>
<h2>Install and start</h2><p>On Linux it is a native package. The Debian package was 163.0 MiB and passed the GPG check from the install guide (<code>Good signature from "SSM Publishing &lt;ssm-publishing@amazon.com&gt;"</code>). It installs two files: a license file and one stripped Rust binary of 970,692,256 bytes.</p>
<pre><code class="hljs language-bash" tabindex="0">curl -fsSLo aws-emulator.deb \
  https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
<span class="hljs-built_in">sudo</span> apt-get install -y ./aws-emulator.deb
aws-emulator --telemetry off
</code></pre><p>We timed 10 repeated starts from <code>exec</code> to the first HTTP 200 on <code>/__admin/health</code>, each with a fresh home directory, then 5 more with the page cache dropped so the binary had to come off disk.</p>
<table>
<thead>
<tr>
<th>Measure</th>
<th>Result</th>
</tr>
</thead>
<tbody><tr>
<td><code>.deb</code> download</td>
<td>163.0 MiB</td>
</tr>
<tr>
<td>Installed binary</td>
<td>925.7 MiB</td>
</tr>
<tr>
<td>Start to healthy, repeated starts (10 runs)</td>
<td>338 to 470 ms, median 383.5 ms</td>
</tr>
<tr>
<td>Start to healthy, cold page cache (5 runs)</td>
<td>1,260 to 1,399 ms, median 1,325 ms</td>
</tr>
<tr>
<td>RSS at ready</td>
<td>about 282 MiB, flat over 60 s idle</td>
</tr>
<tr>
<td>Anonymous memory at ready</td>
<td>172.5 MiB</td>
</tr>
</tbody></table>
<p>The guide says the emulator "uses about 160 MiB when idle." Plain RSS reads higher because it counts pages of the memory-mapped binary. The anonymous part, 172.5 MiB, is close to the documented figure.</p>
<p>A clean <code>SIGTERM</code> deleted the credentials, config and certificate files from <code>~/.aws/emulator/9297/</code>, as the guide says, and left the empty directory.</p>
<h2>The AWS CLI session</h2><p>The setup is eight environment variables from the <a href="https://docs.aws.amazon.com/emulator/latest/userguide/point-sdk-to-emulator.html" rel="noopener noreferrer">SDK guide</a>: the endpoint, the per-run credentials and config files, the CA bundle, two Region variables, and two switches that stop the SDK from asking the instance metadata service for credentials or adding prefixes to host names. Here is part of our recorded CLI session against a fresh emulator:</p>
<p><strong>aws cli against AWS Emulator v1.0.322</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ aws sts get-caller-identity --output text
000000000000	arn:aws:iam::000000000000:user/emulator	AIDAH5I7GEMO7ADXNFAZU
$ aws s3 mb s3://cli-demo
make_bucket: cli-demo
$ aws s3 <span class="hljs-built_in">cp</span> /tmp/notes.txt s3://cli-demo/notes.txt
upload: ../tmp/notes.txt to s3://cli-demo/notes.txt
$ curl --cacert <span class="hljs-variable">$AWS_CA_BUNDLE</span> <span class="hljs-string">"<span class="hljs-subst">$(aws s3 presign s3://cli-demo/notes.txt --expires-in 60)</span>"</span>
release notes
$ aws sqs create-queue --queue-name cli-demo --query QueueUrl --output text
https://sqs.us-east-1.amazonaws.com/000000000000/cli-demo
<span class="hljs-comment"># the same validation error as AWS</span>
$ aws dynamodb create-table --table-name ab ...
aws: [ERROR]: An error occurred (ValidationException) when calling the CreateTable operation: 1 validation error detected: Value <span class="hljs-string">'ab'</span> at <span class="hljs-string">'tableName'</span> failed to satisfy constraint: Member must have length greater than or equal to 3
<span class="hljs-comment"># S3 Control puts the account ID in the host name, so it never arrives</span>
$ aws s3control list-access-points --account-id 000000000000
aws: [ERROR]: Could not connect to the endpoint URL: <span class="hljs-string">"https://000000000000.127.0.0.1:9297/v20180820/accesspoint"</span>
</code></pre><p>Note the queue URL. It looks exactly like a real one. That detail comes back later.</p>
<h2>40 workflow checks, 40 passes</h2><p>We did not want to test single API calls. We wrote 9 workflows of the kind that integration tests use, with 40 checks between them. Each check records the exact error text when it fails. Many checks test the error path, because a fake that accepts everything passes the happy path too.</p>
<table>
<thead>
<tr>
<th>Workflow</th>
<th>Checks</th>
<th>Result</th>
<th>What it covered</th>
</tr>
</thead>
<tbody><tr>
<td>S3</td>
<td>9</td>
<td>9/9</td>
<td>presigned GET and PUT, an expired URL refused with 403, 3-part multipart, completing with a 1 MiB first part refused with <code>EntityTooSmall</code>, versioning, delete markers</td>
</tr>
<tr>
<td>SQS</td>
<td>5</td>
<td>5/5</td>
<td>visibility timeout, the third receive moving a message to the DLQ, <code>StartMessageMoveTask</code> back, FIFO deduplication</td>
</tr>
<tr>
<td>DynamoDB</td>
<td>6</td>
<td>6/6</td>
<td>insert-if-absent, a stale optimistic-lock update, a cancelled transaction with per-item reasons, streams</td>
</tr>
<tr>
<td>Lambda</td>
<td>4</td>
<td>4/4</td>
<td>create and invoke a python3.12 zip, handler exception as <code>FunctionError</code>, boto3 called from inside the function</td>
</tr>
<tr>
<td>EventBridge</td>
<td>2</td>
<td>2/2</td>
<td>a numeric pattern (<code>total &gt; 100</code>) delivering only the matching event to SQS</td>
</tr>
<tr>
<td>SNS</td>
<td>2</td>
<td>2/2</td>
<td>fan-out to two queues, one with raw delivery and one with a filter policy</td>
</tr>
<tr>
<td>KMS</td>
<td>6</td>
<td>6/6</td>
<td>round trip by alias, wrong encryption context, tampered ciphertext, data keys, a disabled key</td>
</tr>
<tr>
<td>Secrets Manager</td>
<td>3</td>
<td>3/3</td>
<td><code>AWSCURRENT</code> and <code>AWSPREVIOUS</code>, scheduled deletion blocking reads, restore</td>
</tr>
<tr>
<td>Step Functions</td>
<td>3</td>
<td>3/3</td>
<td>Choice, Wait, a Lambda task, a Fail state</td>
</tr>
</tbody></table>
<p>The whole suite ran in 38.0 s against a fresh emulator. The emulator answered small calls fast: under load, a 100 KiB <code>PutObject</code> took a median 14.66 ms and a 25-item <code>BatchWriteItem</code> 13.64 ms.</p>
<p>Getting to 40 of 40 took four runs. Every failure in between was a mistake in our test, and the repo keeps each of those runs. That story is the most useful part of this test.</p>
<h3>The emulator caught our missing permissions</h3><p>On the third run, the Lambda function's call to S3 failed. We reproduced that run on purpose as a named variant (<code>FT_NO_ROLE_POLICIES=1</code>), and this is its recorded error:</p>
<pre><code class="hljs language-text" tabindex="0">AccessDenied: User: arn:aws:sts::000000000000:assumed-role/ft-fn-8820f7df/ft-fn-8820f7df is not authorized
to perform: s3:PutObject on resource: "arn:aws:s3:::ft-lambda-8820f7df/from-lambda.txt" because no
identity-based policy allows the s3:PutObject action
</code></pre><p>Our test created the Lambda execution role with a trust policy and nothing else. On AWS that function fails the same way. The Step Functions test failed for the same reason: <code>ft-sfn-8820f7df is not authorized to perform: lambda:InvokeFunction</code>. We added the two missing policies and both checks passed. That variant is the best argument for the emulator we found.</p>
<h3>The Lambda sandbox</h3><p>The first two runs failed for a different reason: our handler probed the machine it ran on. The emulator refused every probe that reached for the host:</p>
<table>
<thead>
<tr>
<th>Probe from inside the function</th>
<th>Result</th>
</tr>
</thead>
<tbody><tr>
<td><code>sys.version</code></td>
<td><code>3.14.0.alpha</code>, for a <code>python3.12</code> function</td>
</tr>
<tr>
<td><code>open("/etc/hostname")</code></td>
<td><code>OSError: a Lambda function in this emulator cannot open a file of the host</code></td>
</tr>
<tr>
<td><code>os.getuid()</code></td>
<td><code>OSError: a Lambda function in this emulator has no host filesystem or process to reach</code></td>
</tr>
<tr>
<td><code>socket.gethostname()</code></td>
<td><code>OSError: a Lambda function in this emulator has no host network to reach</code></td>
</tr>
<tr>
<td><code>import subprocess</code></td>
<td><code>ModuleNotFoundError: No module named '_signal'</code></td>
</tr>
</tbody></table>
<p>That is good isolation for zip-packaged Python, and better than the security page suggests ("it doesn't isolate that code from your machine"). We only tested Python zip functions, so do not assume the same for other runtimes. Two practical effects: a handler that imports <code>subprocess</code> at module level fails to load with <code>Runtime.ImportModuleError</code>, and your code runs on a different Python than the one you deploy to. Four invokes in a row took 369, 335, 354 and 355 ms, with no consistent warm-start speedup.</p>
<h2>The permission test: 27 of 32</h2><p>The security page is direct about permissions:</p>
<blockquote>
<p>Some services in the emulator evaluate IAM policies for IAM users and roles that you create in the emulator, but not the same way as AWS. The emulator never denies a request signed with its own key. It doesn't check AWS KMS key policies or Amazon S3 bucket policies.</p>
</blockquote>
<p>The Lambda result suggested more checking than that, so we wrote 32 cases. For each one we wrote down what AWS documents. <strong>These cases were not run on real AWS</strong>: the expected column comes from the AWS documentation for policy evaluation, bucket policies, KMS key policies, role trust policies and SQS queue policies. Then we ran them against the emulator, and against Moto and Floci for context.</p>
<p>Bold marks a result that differs from what AWS documents.</p>
<table>
<thead>
<tr>
<th>Case</th>
<th>AWS documents</th>
<th>AWS Emulator</th>
<th>Floci</th>
<th>Moto</th>
</tr>
</thead>
<tbody><tr>
<td>user with no policies: sts:GetCallerIdentity</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
</tr>
<tr>
<td>user with no policies: s3:ListAllMyBuckets</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: s3:PutObject</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: sqs:SendMessage</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: dynamodb:PutItem</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: kms:Encrypt</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: secretsmanager:GetSecretValue</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: sns:Publish</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: events:PutEvents</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: iam:CreateUser</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>identity policies: admin user: s3:PutObject (baseline)</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
</tr>
<tr>
<td>identity policies: Allow * plus explicit Deny s3:PutObject</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>identity policies: Allow * but permissions boundary allows only reads: s3:PutObject</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>identity policies: Allow PutObject only with SSE-KMS: put without SSE</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>identity policies: Allow PutObject only with SSE-KMS: put with SSE-KMS</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
</tr>
<tr>
<td>resource policies: bucket policy denies this admin user: s3:PutObject</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td>deny</td>
</tr>
<tr>
<td>resource policies: bucket policy denies everyone over HTTPS: emulator's own key PutObject</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td>deny</td>
</tr>
<tr>
<td>resource policies: KMS key policy denies this admin user: kms:Encrypt</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>resource policies: secret resource policy denies this admin user: GetSecretValue</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>resource policies: no identity policy, bucket policy allows this user: s3:PutObject</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
</tr>
<tr>
<td>identity policies: PutObject allowed on bucket A only: put to bucket A</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
</tr>
<tr>
<td>identity policies: PutObject allowed on bucket A only: put to bucket B</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>identity policies: Allow NotAction s3:*: sqs:SendMessage</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
<td>allow</td>
</tr>
<tr>
<td>identity policies: Allow NotAction s3:*: s3:PutObject</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: s3:ListBucket (ListObjectsV2)</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>user with no policies: s3:GetObject</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>roles and sessions: admin user assumes a role that trusts only lambda.amazonaws.com</td>
<td>deny</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>roles and sessions: no-policy user assumes a role that trusts the account (needs sts:AssumeRole itself)</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>roles and sessions: no-policy user: assume the account-trusted admin role, then s3:PutObject + iam:CreateUser</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>roles and sessions: admin role assumed with a GetObject-only session policy: s3:PutObject</td>
<td>deny</td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
<td><strong>allow</strong></td>
</tr>
<tr>
<td>service-to-service: EventBridge rule -&gt; SQS queue with no queue policy</td>
<td>not delivered</td>
<td>not delivered</td>
<td><strong>delivered</strong></td>
<td><strong>delivered</strong></td>
</tr>
<tr>
<td>service-to-service: SNS topic -&gt; SQS queue with no queue policy</td>
<td>not delivered</td>
<td>not delivered</td>
<td><strong>delivered</strong></td>
<td><strong>delivered</strong></td>
</tr>
</tbody></table>
<p>On v1.0.322, the "never denies its own key" sentence is out of date. A bucket policy that denies <code>s3:PutObject</code> to everyone over HTTPS stopped the emulator's own key. A KMS key policy deny, a secret resource policy deny and a bucket policy deny all stopped an admin user. Explicit denies, resource scoping, <code>NotAction</code> and condition keys such as <code>s3:x-amz-server-side-encryption</code> all worked. EventBridge and SNS did not deliver to a queue without a queue policy, which is what AWS does.</p>
<p>The 5 misses come from 4 causes, and they have one thing in common. Each is a control that <strong>narrows</strong> access, and the emulator skipped it and let the request through:</p>
<ol>
<li><strong>Permissions boundaries.</strong> A user with <code>Allow *</code> and a read-only boundary could still <code>PutObject</code>.</li>
<li><strong>Session policies.</strong> An admin role assumed with a <code>GetObject</code>-only session policy could still <code>PutObject</code>.</li>
<li><strong>The caller's own <code>sts:AssumeRole</code> permission.</strong> A role that trusts the account (<code>arn:aws:iam::000000000000:root</code>) means "any principal in this account that IAM allows to assume it." Our user had no policies at all, so AWS refuses. The emulator returned credentials, and with them the user wrote to S3 and created a new IAM user.</li>
<li><strong><code>ListBuckets</code> with no policy</strong> returned the bucket list.</li>
</ol>
<p>The repo's <code>issues/things-that-do-not-work/</code> folder has open files about session-policy scoping and permissions boundaries, so the agents know about at least some of this. We did not match each of our 4 causes to one issue.</p>
<p>One fix to our own harness changed the comparison numbers. Moto refuses a bucket-policy deny with a bare <code>403: Forbidden</code> where AWS returns <code>AccessDenied</code>, and our first classifier counted that as an error, not a refusal. We now count every HTTP 403 as a refusal, which adds those two cases to Moto's total (8 of 32). The saved results still show the bare <code>403</code> code. The fix did not change the AWS Emulator or Floci results.</p>
<h3>Why this matters for your tests</h3><p>Boundaries and session policies are the controls teams use for least privilege: a CI role that may create roles only inside a boundary, a broker that hands out narrowed credentials, a platform team that delegates IAM. Locally, both directions go wrong. A job whose boundary blocks a write it needs passes against the emulator and fails on AWS. A test that asserts "the boundary stops this write" fails against the emulator even when the boundary on AWS is correct.</p>
<p>What to do about it:</p>
<ul>
<li><strong>Create the real roles in the emulator.</strong> Do not run everything with the emulator's admin key. Create the roles and policies you deploy, and run each component with its own role. That is how the emulator caught our two missing permissions.</li>
<li><strong>Keep negative permission tests for real AWS.</strong> Run a small set against a sandbox account in CI, with short-lived credentials. Boundaries, session policies and role trust belong there.</li>
<li><strong>Check policies without calling the emulator.</strong> IAM Access Analyzer policy validation and custom policy checks catch many mistakes before anything runs.</li>
<li><strong>Treat a local <code>AccessDenied</code> as a bug, and a local success as unproven.</strong> The emulator's denies matched AWS in every case we tried. Its allows did not.</li>
</ul>
<h2>Three more ways a green local run can lie</h2><h3>The JavaScript SDK sent SQS calls to real AWS</h3><p>Our JS SDK v3 smoke test passed for S3 and DynamoDB. SQS failed with <code>InvalidClientTokenId: The security token included in the request is invalid.</code> The emulator would not say that about its own key. The SDK printed the reason:</p>
<pre><code class="hljs language-text" tabindex="0">QueueUrl=https://sqs.us-east-1.amazonaws.com/000000000000/js-demo differs from SQSClient resolved
endpoint=https://127.0.0.1:9297/, using QueueUrl host as endpoint.
Set [endpoint=string] or [useQueueUrlAsEndpoint=false] on the SQSClient.
</code></pre><p>The emulator returns queue URLs in the AWS format. <code>@aws-sdk/client-sqs</code> 3.1167.0 uses the queue URL's host by default, so the request went to <code>sqs.us-east-1.amazonaws.com</code>, and AWS rejected the emulator's key. To prove it, we pointed that host name at <code>0.0.0.0</code> in <code>/etc/hosts</code>. The error changed to <code>connect ECONNREFUSED 0.0.0.0:443</code>.</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-comment">// Keep SQS calls on the emulator</span>
<span class="hljs-keyword">const</span> sqs = <span class="hljs-keyword">new</span> <span class="hljs-title class_">SQSClient</span>({ <span class="hljs-attr">useQueueUrlAsEndpoint</span>: <span class="hljs-literal">false</span> });
</code></pre><p>With that option, the same test passed. The SDK guide lists host names that SDKs change (host prefixes, S3, S3 Control) but not this case. Our machine had no real credentials, so AWS rejected the emulator's key and nothing else happened. But environment variables take precedence over the emulator's credentials file. In a CI job with real <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> set, the same test would have sent a request to AWS signed with real keys. The guide's advice to unset real credentials in the shell that runs your tests is not optional.</p>
<h3>Terraform: provider 6.73.0 could not read back an S3 bucket</h3><p>We applied a small stack with 25 resources: a versioned bucket with a lifecycle rule, queues with a DLQ, a DynamoDB table with a stream, a GSI and TTL, SNS, KMS, Secrets Manager, SSM, IAM roles, a Lambda function with an SQS event source, an EventBridge rule and a Step Functions state machine.</p>
<table>
<thead>
<tr>
<th></th>
<th>AWS provider 6.73.0</th>
<th>AWS provider 5.100.0</th>
</tr>
</thead>
<tbody><tr>
<td><code>terraform apply</code></td>
<td>failed, 23 entries in state</td>
<td>passed, 26 entries in state</td>
</tr>
<tr>
<td>Error</td>
<td><code>S3 Control: ListTagsForResource ... lookup 000000000000.127.0.0.1: no such host</code></td>
<td>none</td>
</tr>
<tr>
<td>Second <code>plan</code></td>
<td>failed on the same S3 call</td>
<td>1 change: DynamoDB <code>on_demand_throughput</code> (-1 to null)</td>
</tr>
<tr>
<td><code>terraform destroy</code></td>
<td>failed on the same S3 call</td>
<td>passed in 39.3 s</td>
</tr>
</tbody></table>
<p>In our test, provider 6.73.0 read bucket tags through S3 Control, and S3 Control puts the account ID in the host name. The SDK guide says S3 Control requests cannot reach the emulator. We tried a hosts-file entry for <code>000000000000.127.0.0.1</code>, but <code>getent</code> did not resolve it. When we forced the address with <code>curl --resolve</code>, the TLS handshake failed (curl exit code 35) even with certificate checks off. In this test, provider 5.100.0 worked where 6.73.0 failed, with one plan diff left. Until that changes, stacks with S3 buckets need an older provider or a separate test for the bucket.</p>
<p>Apply took about 85 s with both providers. The three SQS resources took 25 s each. In our load test the emulator answered SQS batch sends in a median 13.08 ms, so slow SQS responses do not explain those 25 s. We did not trace what the provider waits for.</p>
<h3>Caps that AWS does not have</h3><p>Our first memory test tried to put 1,000 objects of 100 KiB in one bucket and 10,000 messages in one queue. Both stopped early, so we wrote a probe that fills one bucket and one queue until the emulator refuses. The bucket took 654 objects and the queue took 1,000 messages:</p>
<pre><code class="hljs language-text" tabindex="0">ValidationException: storing "obj-00654" would leave the bucket "cap-probe" holding 67131605 bytes of
objects, past the 67108864 bytes this emulator keeps per bucket

OverLimit: This emulator holds at most 1000 messages per queue; queue cap-probe cannot take 1 more.
Delete messages from it first.
</code></pre><p>That is 64 MiB per bucket and 1,000 messages per queue. Real S3 buckets and SQS queues have no such limits on stored data or backlog. The errors are clear, and the source has more caps of the same kind, with messages such as "this emulator holds at most {} files per repository tree" and "this emulator holds tables for at most {}". A test that uploads a large fixture or builds a queue backlog has to stay under them.</p>
<p>With 58.6 MiB in S3, 20,000 DynamoDB items and 900 queued messages, anonymous memory grew from 172.5 MiB to 277.2 MiB. Deleting everything and waiting 15 s did not give memory back (278.8 MiB). Per the docs, restart the emulator between suites, which also gives you clean state.</p>
<p><strong>AWS Emulator memory while a test suite fills it</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>idle</th>
<th>+58.6 MiB in S3</th>
<th>+20,000 DynamoDB items</th>
<th>+900 SQS messages</th>
<th>all deleted, +15 s</th>
</tr>
</thead>
<tbody><tr>
<td>RSS</td>
<td>281.7 MiB</td>
<td>367.2 MiB</td>
<td>411.5 MiB</td>
<td>415.4 MiB</td>
<td>418.9 MiB</td>
</tr>
<tr>
<td>Anonymous memory</td>
<td>172.5 MiB</td>
<td>236.6 MiB</td>
<td>276.4 MiB</td>
<td>277.2 MiB</td>
<td>278.8 MiB</td>
</tr>
</tbody></table>
<p><em>v1.0.322, one fresh process, 4 vCPU / 8 GB droplet. Source: results/loaded-memory.json in the companion repo.</em></p>
<h2>How it compares to Moto and Floci</h2><p>LocalStack was the usual choice for this. Since March 2026, <code>localstack/localstack</code> is a single image that needs an auth token, and the free plan is meant for non-commercial use (<a href="https://blog.localstack.cloud/the-road-ahead-for-localstack/" rel="noopener noreferrer">LocalStack's announcement</a>). We did not test it, because that needs an account. Moto (Apache 2.0) and Floci (MIT) run with no account, so we ran the same two suites against both, <strong>in their default configurations</strong>. Both have opt-in authorization checks that we did not turn on, so their permission rows show the defaults, not their best case. Start times are not measured the same way: AWS Emulator is timed to its health check, Moto to its first HTTP response, and Floci from <code>docker run</code>.</p>
<table>
<thead>
<tr>
<th></th>
<th>AWS Emulator 1.0.322</th>
<th>Floci 2.4.0</th>
<th>Moto 5.2.5 (server mode)</th>
</tr>
</thead>
<tbody><tr>
<td>License</td>
<td>CC0 (<code>vendor/</code> is Apache 2.0)</td>
<td>MIT</td>
<td>Apache 2.0</td>
</tr>
<tr>
<td>Account or token needed</td>
<td>no</td>
<td>no</td>
<td>no</td>
</tr>
<tr>
<td>Install</td>
<td>163.0 MiB <code>.deb</code>, 925.7 MiB binary</td>
<td>90.2 MiB image (<code>docker image inspect</code>)</td>
<td>Python package</td>
</tr>
<tr>
<td>Start to first response</td>
<td>median 383.5 ms (10 runs)</td>
<td>992 ms including <code>docker run</code> (1 run)</td>
<td>1,174 ms (1 run)</td>
</tr>
<tr>
<td>Idle memory</td>
<td>172.5 MiB anonymous, about 282 MiB RSS</td>
<td>25.4 MiB (<code>docker stats</code>)</td>
<td>51.1 MiB RSS</td>
</tr>
<tr>
<td>Workflow checks passed</td>
<td>40 of 40</td>
<td>38 of 40</td>
<td>32 of 40 (1 miss is our harness)</td>
</tr>
<tr>
<td>Permission cases matching AWS docs</td>
<td>27 of 32</td>
<td>6 of 32</td>
<td>8 of 32</td>
</tr>
</tbody></table>
<p><strong>Share of checks that matched AWS, default configurations</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>AWS Emulator, workflows</td>
<td>100%</td>
<td>Workflow checks</td>
</tr>
<tr>
<td>Floci, workflows</td>
<td>95%</td>
<td>Workflow checks</td>
</tr>
<tr>
<td>Moto, workflows</td>
<td>80%</td>
<td>Workflow checks</td>
</tr>
<tr>
<td>AWS Emulator, permissions</td>
<td>84.4%</td>
<td>Permission cases</td>
</tr>
<tr>
<td>Moto, permissions</td>
<td>25%</td>
<td>Permission cases</td>
</tr>
<tr>
<td>Floci, permissions</td>
<td>18.8%</td>
<td>Permission cases</td>
</tr>
</tbody></table>
<p><em>Workflow checks: 40 per target; one of Moto's misses is a networking problem in our harness. Permission cases: 32 per target, expected results from AWS documentation, not run on real AWS. Source: results/</em>-scenarios.json and results/<em>-iam-gap.json in the companion repo.</em></p>
<p>Some notes so the table is fair:</p>
<ul>
<li>Moto's Lambda-to-S3 check failed because of our setup: Moto listened on <code>127.0.0.1</code>, and its Lambda container could not reach <code>host.docker.internal:5000</code>. We count it as a fail, but it is a network problem in our harness, not proof that Moto cannot do it.</li>
<li>Moto's other misses: an expired presigned URL still worked, a presigned PUT stored an empty body, <code>StartMessageMoveTask</code> returned HTTP 500, a handler exception came back as <code>Handled</code> instead of <code>Unhandled</code>, a disabled KMS key still encrypted, and both Step Functions executions were still <code>RUNNING</code> after 30 s.</li>
<li>Floci's misses: an expired presigned URL still worked, and the EventBridge rule with a numeric pattern delivered nothing.</li>
<li>In the permission matrix, both alternatives allowed every request in their default modes, with one exception: Moto refused both bucket-policy denies.</li>
<li>On size and memory, the others are much smaller. Floci's image is 90.2 MiB and used 25.4 MiB idle. Moto used 51.1 MiB idle. AWS Emulator is a 925.7 MiB binary.</li>
</ul>
<p>On these suites, AWS Emulator matched AWS more often than either alternative, and by a wide margin on permissions.</p>
<h2>What the repo shows about "built by agents"</h2><p>We cloned <a href="https://github.com/aws/aws-emulator" rel="noopener noreferrer">aws/aws-emulator</a> and the <a href="https://github.com/aws/aws-emulator-evaluation-suite" rel="noopener noreferrer">evaluation suite</a> and counted.</p>
<p><em>Goal: Each release publishes its own pass rate: 56.5% for v1.0.322</em></p>
<ol>
<li><strong>Fidelity tests</strong> 29,676 boto3 checks</li>
<li><strong>IssueFinderAgent</strong> 16,664 commits</li>
<li><strong>issues/ folder</strong> 6,544 open, 12,708 fixed</li>
<li><strong>IssueFixerAgent</strong> 12,981 commits</li>
<li><strong>Release</strong> with scores.json</li>
</ol>
<p><em>every change passes a merge gate (not visible in the repo): failing fidelity tests become new issues, then back to step 1.</em></p>
<ul>
<li><strong>Every commit is by an agent identity.</strong> 29,684 commits: IssueFinderAgent 16,664, IssueFixerAgent 12,981, Scaffolder 29, TraceIdMigration 6, ReleaseManagerAgent 4. No commit carries a person's name. The launch post says: "Our team hasn't written or reviewed any of the code."</li>
<li><strong>The public history is three weeks long.</strong> It starts on 2026-10-21 with a commit called "Initial emulator scaffold", and 3,734 commits landed that first day. All of it falls between 2026-10-21 and 2026-11-10.</li>
<li><strong>It is big.</strong> 436 service crates, 437 Smithy model files, 40,130 operation files, 14,099 test files, and 16,142,920 lines of Rust outside <code>vendor/</code>. The EC2 crate alone is 709,186 lines.</li>
<li><strong>The work is issue-driven, in the open.</strong> Commits are "file issues" and "Fix issue" with <code>Producer:</code> trailers, and each issue is a Markdown file with a severity, the failing fidelity test and the commit it was found against. 6,544 are open and 12,708 are fixed.</li>
<li><strong>The merge gate cannot be audited from the repo.</strong> There are no CI workflows in <code>.github/</code>, and commits carry no review trailers. The adversarial security review may well happen, but you cannot see it here.</li>
<li><strong>The fidelity tests are public, their history is not.</strong> The evaluation suite has 29,676 tests in <code>manifest.jsonl</code>, covering 1,206 service directories (859 of them cross-service), in a repository with 2 commits.</li>
<li><strong>AWS publishes its own score, and it is honest.</strong> The <code>scores.json.gz</code> asset on the v1.0.322 release has 42,937 scored checks: 24,243 pass, 18,614 fail, 80 skipped. That is 56.5%.</li>
</ul>
<p><strong>AWS's own fidelity pass rate for v1.0.322, by service</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>EventBridge</td>
<td>71.4%</td>
</tr>
<tr>
<td>KMS</td>
<td>71.3%</td>
</tr>
<tr>
<td>SNS</td>
<td>69.3%</td>
</tr>
<tr>
<td>DynamoDB</td>
<td>68.1%</td>
</tr>
<tr>
<td>Secrets Manager</td>
<td>68%</td>
</tr>
<tr>
<td>SQS</td>
<td>67.8%</td>
</tr>
<tr>
<td>S3</td>
<td>62.7%</td>
</tr>
<tr>
<td>IAM</td>
<td>58%</td>
</tr>
<tr>
<td>Lambda</td>
<td>51.9%</td>
</tr>
<tr>
<td>STS</td>
<td>47.8%</td>
</tr>
</tbody></table>
<p><em>From scores.json.gz on the v1.0.322 release (42,937 scored checks, 56.5% overall). Our own 40 workflow checks passed 100% on these services. Source: results/scores-summary.json.</em></p>
<p>Those numbers do not contradict our 40 of 40. The evaluation suite is written to find gaps, and the issue-finder agents keep adding tests where the emulator fails. Our suite tests the common paths that a typical app uses. Both are true: the common paths work, and the edges are still moving. The releases page listed 12 releases between 2026-11-07 and 2026-11-11, so pin a version in CI.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>We did not run the permission cases on real AWS.</strong> The expected results come from AWS documentation. We are confident about each of them, but "AWS documents X" is not the same as "we saw AWS do X."</li>
<li><strong>One machine, one version.</strong> Everything ran on v1.0.322 on one 4 vCPU droplet. With 12 releases in 5 days, some of these results may already be different.</li>
<li><strong>Default configurations only</strong> for Moto and Floci. Both can do more permission checking if you turn it on.</li>
<li><strong>Python zip functions only</strong> for the Lambda sandbox. Container images and other runtimes may behave differently.</li>
<li><strong>We did not verify the "200+ services" claim</strong> one by one. We counted 436 service crates and 437 model files, and we exercised at least 11 services.</li>
<li><strong>We could not observe the merge gate</strong> or the security agents. We can only say that their output is not in the public repo.</li>
</ul>
<h2>Using it safely</h2><p>If you adopt it for local tests or CI, this is the checklist we would use:</p>
<ol>
<li>Start it with <code>--telemetry off</code> if your policy needs that, and pin the version.</li>
<li>Unset <code>AWS_ACCESS_KEY_ID</code>, <code>AWS_SECRET_ACCESS_KEY</code>, <code>AWS_SESSION_TOKEN</code> and <code>AWS_PROFILE</code> in the same script that runs the tests, and set <code>AWS_EC2_METADATA_DISABLED=true</code>.</li>
<li>In the JS SDK, create SQS clients with <code>useQueueUrlAsEndpoint: false</code>.</li>
<li>Test your Terraform provider version first. In our run, 6.73.0 failed on S3 buckets and 5.100.0 worked; otherwise keep buckets out of the emulated stack.</li>
<li>Keep fixtures under 64 MiB per bucket and backlogs under 1,000 messages per queue.</li>
<li>Create real roles with real policies and run each component as its role. Let the emulator's <code>AccessDenied</code> catch missing permissions.</li>
<li>Test boundaries, session policies and role trust on a real sandbox account.</li>
<li>Restart the emulator between suites for clean state and memory.</li>
</ol>
<p>If you still run LocalStack from an old pinned image, our <a href="https://devops-daily.com/posts/experimenting-locally-with-terraform">local Terraform experiments post</a> shows the older setup.</p>
<h2>Summary</h2><p>Of the three local emulators we ran, AWS Emulator matched AWS most often. Our everyday workflows passed, the error codes matched, and it found real permission bugs in our test code. The part to watch is not the warning AWS gives: it checks more policies than its docs say. The risk is the controls it skips. A green run against the emulator proves that your code works with the permissions it has. It does not prove that your permission limits hold. Keep that small set of tests on real AWS, and use the emulator for everything else.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Disk Full but du Disagrees: Finding Deleted Files That Still Use Space]]></title>
      <link>https://devops-daily.com/posts/disk-full-du-disagrees-deleted-open-files</link>
      <description><![CDATA[When df reports a full filesystem but du cannot find the bytes, an unlinked log held open by a process is often the missing space. Find the holder, release space safely, and prevent the next midnight disk-full incident.]]></description>
      <pubDate>Thu, 12 Nov 2026 07:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/disk-full-du-disagrees-deleted-open-files</guid>
      <category><![CDATA[Linux]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Linux]]></category><category><![CDATA[Troubleshooting]]></category><category><![CDATA[Disk Space]]></category><category><![CDATA[lsof]]></category><category><![CDATA[Logging]]></category><category><![CDATA[Incident Response]]></category>
      <content:encoded><![CDATA[<p><code>df</code> says <code>/var</code> is full. <code>du -sh /var</code> is hundreds of gigabytes short. Someone has already removed the large log, so searching the directory tree turns up nothing. Do not start deleting packages yet. A process can keep an unlinked file, and every block behind it, alive through an open file descriptor. This is the fastest way to prove that is the problem, release the space without an unplanned restart, and keep log rotation from creating it again.</p>
<h2>TLDR</h2><ul>
<li><code>rm</code> removes a directory entry. If it was the last link but a process still has the file open, Linux releases the blocks only when the final descriptor closes. <a href="https://man7.org/linux/man-pages/man2/unlink.2.html" rel="noopener noreferrer">unlink(2)</a> describes that lifetime.</li>
<li>Start with <code>sudo lsof -nP -a +L1 &lt;mountpoint&gt;</code>. The <code>-a</code> matters: lsof normally ORs selections, while <code>-a</code> makes the deleted-file selection and mountpoint selection intersect.</li>
<li>Prefer restarting the owner or making it reopen logs. Truncating <code>/proc/PID/fd/N</code> is an emergency action that discards the data, but it can return space immediately.</li>
<li>If there is no deleted open file, check ext4 reserved blocks, files hidden below a mount point, and inode exhaustion with <code>df -i</code>.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Root or equivalent access to the affected Linux node.</li>
<li>The filesystem mount point that <code>df</code> reports as full, not just a directory that happens to be beneath it.</li>
<li><code>lsof</code> if installed. The <code>/proc</code> fallback below works without it.</li>
<li>A decision from the service owner before restarting, reloading, or truncating a live writer.</li>
</ul>
<h2>Why the deleted log still consumes space</h2><p>On Linux, a pathname is a directory entry that refers to a file's inode. <code>rm</code> calls <code>unlink(2)</code>: it removes that name. When that was the last link and no process has the file open, the filesystem can reuse the space. If a process still has it open, the inode and its blocks remain until the last file descriptor closes. That is why <code>du</code>, which walks named directory entries, cannot count these bytes while <code>df</code>, which reports filesystem allocation, still can. <a href="https://man7.org/linux/man-pages/man2/unlink.2.html" rel="noopener noreferrer">The <code>unlink(2)</code> manual page</a> states both conditions explicitly.</p>
<p>The descriptor can outlive the name because it refers through an open file description to the inode, rather than by resolving the old pathname again. Closing the final descriptor frees an unlinked file's occupied space. <a href="https://man7.org/linux/man-pages/man3/close.3p.html" rel="noopener noreferrer">POSIX <code>close(3p)</code></a> documents that result.</p>
<ol>
<li><strong>app.log</strong> named file, 200 MiB</li>
<li><strong>rm</strong> name removed</li>
<li><strong>FD 3 still open</strong> link count is zero</li>
<li><strong>close or truncate</strong> blocks become reusable</li>
</ol>
<h2>Reproduce the mismatch, then find the holder</h2><p>Here is a real run on Linux 6.12 with lsof 4.95.0 as packaged by Debian 12. A Python process opened a log for append, wrote 200 MiB on <code>/dev/shm</code>, and kept descriptor 3 open. The lsof warnings from an unprivileged invocation are trimmed below. The numbers are not a capacity benchmark, <code>/dev/shm</code> was chosen so this test could not fill a real filesystem.</p>
<p><strong>A deleted 200 MiB log remains allocated</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ <span class="hljs-built_in">df</span> -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.9G  201M  1.7G  11% /dev/shm
$ <span class="hljs-built_in">du</span> -sh /dev/shm/ddemo
200M	/dev/shm/ddemo
$ <span class="hljs-built_in">rm</span> /dev/shm/ddemo/app.log
$ lsof -nP +L1 /dev/shm | <span class="hljs-built_in">wc</span> -l
15
$ lsof -nP -a +L1 /dev/shm | <span class="hljs-built_in">wc</span> -l
2
$ <span class="hljs-built_in">df</span> -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.9G  201M  1.7G  11% /dev/shm
$ <span class="hljs-built_in">du</span> -sh /dev/shm/ddemo
0	/dev/shm/ddemo
$ lsof -nP -a +L1 /dev/shm | grep -E <span class="hljs-string">'COMMAND|app.log'</span>
COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK   NODE NAME
python3 1586127 nobody    3w   REG   0,24 209715200     0 139344 /dev/shm/ddemo/app.log (deleted)
$ <span class="hljs-built_in">stat</span> -L -c <span class="hljs-string">'%s bytes'</span> /proc/1586127/fd/3
209715200 bytes
$ : &gt; /proc/1586127/fd/3
$ <span class="hljs-built_in">df</span> -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.9G  140K  1.9G   1% /dev/shm
$ lsof -nP -a +L1 /dev/shm | grep -E <span class="hljs-string">'COMMAND|app.log'</span>
COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
python3 1586127 nobody    3w   REG   0,24        0     0 139344 /dev/shm/ddemo/app.log (deleted)
</code></pre><p>The key line has <code>NLINK</code> equal to <code>0</code>, which is what <code>+L1</code> selects. <code>SIZE/OFF</code> is 209715200 bytes before the truncate and zero after it. With <code>+L</code>, lsof exposes the link-count column; <code>+L1</code> selects open files with a link count below one. <a href="https://lsof.readthedocs.io/en/stable/manpage/" rel="noopener noreferrer">Its manual page</a> documents both behaviours.</p>
<p>Use this incident command against the mount point:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">sudo</span> lsof -nP -a +L1 /var
</code></pre><p><code>-nP</code> avoids DNS and service-name lookups during an incident. <code>+L1</code> selects the unlinked open files. <code>/var</code> is a selection too, which produces an important lsof trap: selection options are normally ORed. <code>lsof -nP +L1 /var</code> therefore includes every deleted open file anywhere and every open file on <code>/var</code>. Adding <code>-a</code> ANDs all selection options, so it returns the intersection. <a href="https://lsof.readthedocs.io/en/stable/manpage/" rel="noopener noreferrer">lsof documents its default OR behaviour and <code>-a</code></a>.</p>
<p>Pass a mount point. For a non-mount directory, lsof selects that directory rather than the entire filesystem. If you need a tree search, use lsof's <code>+D</code> or <code>+d</code> options instead. The <a href="https://lsof.readthedocs.io/en/stable/tutorial/" rel="noopener noreferrer">lsof tutorial</a> shows the mountpoint form, <code>lsof -a +L1 /home</code>.</p>
<h3>When lsof is unavailable</h3><p><code>/proc/PID/fd</code> contains one symbolic link for every open descriptor. Search it as root, because access to another process's descriptor links is permission checked:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">sudo</span> find /proc/[0-9]*/fd -lname <span class="hljs-string">'*(deleted)'</span> -<span class="hljs-built_in">printf</span> <span class="hljs-string">'%p -&gt; %l\n'</span> 2&gt;/dev/null

<span class="hljs-comment"># Inspect an interesting result. -L follows the /proc descriptor link.</span>
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">stat</span> -L -c <span class="hljs-string">'%s bytes'</span> /proc/1586127/fd/3
</code></pre><p>The <a href="https://man7.org/linux/man-pages/man5/proc_pid_fd.5.html" rel="noopener noreferrer">proc_pid_fd(5) manual page</a> describes these entries as links to the files a process has open and notes the access check. Expect races: a process can close a descriptor between <code>find</code> and <code>stat</code>.</p>
<p>Do not treat every deleted-looking target as disk usage. <code>/memfd:... (deleted)</code> is an anonymous in-memory file, not a file consuming your ext4 or XFS blocks. <a href="https://man7.org/linux/man-pages/man2/memfd_create.2.html" rel="noopener noreferrer">memfd_create(2)</a> says it lives in RAM. A <code>(deleted)</code> executable or shared library after an upgrade is different again: it can be an old program image still mapped by a running process. That is normally a restart-planning signal, not the explanation for a full data filesystem.</p>
<h2>Release the space without making the incident worse</h2><p>The clean fix is for the holder to close the file. Restart the affected service, or use its documented reopen operation. If a systemd unit has a reload action, use the service's runbook, for example:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">sudo</span> systemctl reload nginx
</code></pre><p>For nginx log rotation, rename the log first and then signal the master process to reopen logs. nginx documents <code>USR1</code> for this operation:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Confirm the configured PID file before running this.</span>
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">kill</span> -USR1 <span class="hljs-string">"<span class="hljs-subst">$(cat /run/nginx.pid)</span>"</span>
</code></pre><p>After a successful reopen, nginx closes the old logs and workers open the new ones. <a href="https://nginx.org/en/docs/control.html" rel="noopener noreferrer">nginx's log-rotation documentation</a> describes that sequence. Use the actual PID file for the installation, not the sample path blindly.</p>
<p>If the filesystem is full and a controlled restart or reopen is unavailable, truncate the specific deleted descriptor:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Last resort: verify PID, FD, path, owner and size first.</span>
<span class="hljs-built_in">sudo</span> sh -c <span class="hljs-string">': &gt; /proc/1586127/fd/3'</span>
</code></pre><p>That is what released the 200 MiB in the demo while Python kept running and descriptor 3 remained open. It also destroys the unrecovered log content. Copy it out first if it might be needed, to a different filesystem with confirmed free space. <code>/var/tmp</code> usually sits on the same full <code>/var</code>, so it is the wrong target:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># /mnt/recovery: a separately mounted volume with room for the file</span>
<span class="hljs-built_in">df</span> -h /mnt/recovery
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">cp</span> -- /proc/1586127/fd/3 /mnt/recovery/recovered-app.log
</code></pre><p>Truncation changes the file size but not the open file description's seek offset. <a href="https://man7.org/linux/man-pages/man3/ftruncate.3p.html" rel="noopener noreferrer">ftruncate(3p)</a> says the seek pointer is not modified. A writer that does not use <code>O_APPEND</code> can therefore write again at its old offset, extending the file and leaving a zero-filled hole before the new data. That is a reason to treat truncation as an emergency measure, then repair the application's logging setup.</p>
<blockquote>
<p><strong>Warning</strong></p>
<p>Do not truncate a descriptor because its name ends in <code>(deleted)</code> alone. Check the mount point, type, <code>SIZE/OFF</code>, process identity, and whether its data is needed. A wrong <code>/proc/PID/fd/N</code> target can discard live application data.</p>
</blockquote>
<h2>Stop the mismatch from returning</h2><p>The durable fix is to make every writer close and reopen its logs during rotation. A logrotate rule commonly creates the new file and runs a service-specific post-rotation reload:</p>
<pre><code class="hljs language-text" tabindex="0">/var/log/myapp/app.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    postrotate
        systemctl reload myapp.service
    endscript
}
</code></pre><p><code>delaycompress</code> leaves the most recently rotated file uncompressed until the next rotation, for programs that keep writing to the previous file for a while before they reopen. <a href="https://man7.org/linux/man-pages/man5/logrotate.conf.5.html" rel="noopener noreferrer">logrotate's manual page</a> documents that directive.</p>
<p><code>copytruncate</code> is sometimes needed for a process that cannot reopen a file, but it is a tradeoff. It copies the old file and truncates it in place. The logrotate manual page warns that there is a very small interval between those operations in which log data can be lost. Prefer a reopen signal whenever the software supports one.</p>
<p>Container logs need caps too. Docker's <code>json-file</code> driver accepts <code>max-size</code> and <code>max-file</code>; its defaults are unlimited size and one file, and changing the daemon setting affects only newly created containers. <a href="https://docs.docker.com/engine/logging/drivers/json-file/" rel="noopener noreferrer">Docker's <code>json-file</code> documentation</a> covers those settings. This example limits a new container to three 10 MiB log files:</p>
<pre><code class="hljs language-bash" tabindex="0">docker run --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  my-image:current
</code></pre><p>Docker Engine on servers still defaults to <code>json-file</code> with no rotation; we measured how that compares with the <code>local</code> driver that Docker Desktop now uses in <a href="https://devops-daily.com/posts/docker-desktop-local-log-driver">Docker Desktop's switch to the local log driver</a>. On a host that uses the systemd journal, set a budget instead of relying on disk pressure:</p>
<pre><code class="hljs language-ini" tabindex="0"><span class="hljs-comment"># /etc/systemd/journald.conf.d/size.conf</span>
<span class="hljs-section">[Journal]</span>
<span class="hljs-attr">SystemMaxUse</span>=<span class="hljs-number">1</span>G
</code></pre><p><code>SystemMaxUse=</code> limits persistent journals under <code>/var/log/journal</code>. The current systemd documentation says the default maximum is 10% of the filesystem, capped at 4G, and only archived files are removed to reduce usage. <code>journalctl --vacuum-size=500M</code> removes oldest archived journal files until their usage falls below that target. <a href="https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html" rel="noopener noreferrer">journald.conf(5)</a> and <a href="https://www.freedesktop.org/software/systemd/man/latest/journalctl" rel="noopener noreferrer">journalctl(1)</a> document those limits and the active-file caveat.</p>
<p>Alert before a page becomes a disk-full outage. node_exporter exposes <code>node_filesystem_avail_bytes</code> as space available to non-root users, and its node-mixin uses <code>predict_linear</code> for a run-out forecast. <a href="https://github.com/prometheus/node_exporter/blob/master/collector/filesystem_common.go" rel="noopener noreferrer">The collector source</a> and <a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/alerts/alerts.libsonnet" rel="noopener noreferrer">the maintained alert rule</a> show both names.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-bullet">-</span> <span class="hljs-attr">alert:</span> <span class="hljs-string">FilesystemPredictedFullSoon</span>
  <span class="hljs-attr">expr:</span> <span class="hljs-string">|
    predict_linear(
      node_filesystem_avail_bytes{mountpoint="/var",fstype!~"tmpfs|overlay"}[6h],
      24 * 60 * 60
    ) &lt; 0
</span>  <span class="hljs-attr">for:</span> <span class="hljs-string">15m</span>
  <span class="hljs-attr">labels:</span>
    <span class="hljs-attr">severity:</span> <span class="hljs-string">warning</span>
  <span class="hljs-attr">annotations:</span>
    <span class="hljs-attr">summary:</span> <span class="hljs-string">/var</span> <span class="hljs-string">is</span> <span class="hljs-string">predicted</span> <span class="hljs-string">to</span> <span class="hljs-string">run</span> <span class="hljs-string">out</span> <span class="hljs-string">of</span> <span class="hljs-string">space</span> <span class="hljs-string">within</span> <span class="hljs-number">24</span> <span class="hljs-string">hours</span>
</code></pre><p>Host log rotation limits local damage. For the separate question of how log volume reaches an observability bill, see <a href="https://devops-daily.com/posts/your-logs-cost-more-than-the-service">Your Logs Cost More Than the Service That Emits Them</a>.</p>
<h2>If no process is holding a deleted file</h2><p>Two other mismatches are common.</p>
<p><strong>Reserved ext4 blocks.</strong> ext2, ext3 and ext4 reserve blocks for privileged processes, so <code>df</code> availability for a normal user is not the same as raw free blocks. <code>mke2fs</code> defaults that reservation to 5 percent unless configured otherwise. <code>tune2fs -l</code> only reads the superblock, so it is safe to run against the mounted device:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">sudo</span> tune2fs -l /dev/mapper/vg0-var | grep -E <span class="hljs-string">'Block count|Reserved block count|Block size'</span>
</code></pre><p><a href="https://man7.org/linux/man-pages/man8/mke2fs.8.html" rel="noopener noreferrer"><code>mke2fs(8)</code></a> documents the 5 percent default, while <a href="https://man7.org/linux/man-pages/man8/tune2fs.8.html" rel="noopener noreferrer"><code>tune2fs(8)</code></a> documents <code>-l</code> and adjustment of the reserved-block percentage. Do not lower the reservation on a system filesystem as a reflex. It exists so root-owned services can still write when unprivileged writers cannot.</p>
<p><strong>Files hidden under a mount point.</strong> If a process writes <code>/var/log/old.log</code> before a separate volume mounts on <code>/var/log</code>, that file remains on the underlying root filesystem. After the mount, the old directory contents are invisible by that pathname. <a href="https://man7.org/linux/man-pages/man8/mount.8.html" rel="noopener noreferrer">mount(8)</a> says a mount makes the previous contents of the directory invisible while it is mounted.</p>
<p>Inspect the underlying root through a temporary bind mount. A plain bind of <code>/</code> does not bring the child mount at <code>/var/log</code> along, so the root filesystem's hidden <code>/var/log</code> becomes visible at the alternate path. <a href="https://man7.org/linux/man-pages/man2/mount.2.html" rel="noopener noreferrer">mount(2)</a> specifies that a directory bind mount does not include submounts unless it is recursive:</p>
<pre><code class="hljs language-bash" tabindex="0">root_view=$(<span class="hljs-built_in">mktemp</span> -d)
<span class="hljs-built_in">sudo</span> mount --<span class="hljs-built_in">bind</span> / <span class="hljs-string">"<span class="hljs-variable">$root_view</span>"</span>
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">du</span> -xsh <span class="hljs-string">"<span class="hljs-variable">$root_view</span>/var/log"</span>
<span class="hljs-built_in">sudo</span> umount <span class="hljs-string">"<span class="hljs-variable">$root_view</span>"</span>
<span class="hljs-built_in">rmdir</span> <span class="hljs-string">"<span class="hljs-variable">$root_view</span>"</span>
</code></pre><p>Finally, <code>No space left on device</code> can mean <strong>inodes</strong>, not blocks. <code>df -i /var</code> reports inode information instead of block usage, as documented by <a href="https://man7.org/linux/man-pages/man1/df.1.html" rel="noopener noreferrer">df(1)</a>. Millions of tiny files can exhaust the inode table while <code>df -h</code> still reports available bytes.</p>
<h2>Containers and Kubernetes</h2><p>A deleted file held open by a container process still consumes storage on the node: the container's writable layer lives under the runtime's directory on the node's disk, and a volume lives wherever it is mounted from. Investigate from the node, not from inside the container, which often has neither <code>lsof</code> nor permission to see other processes. Find the container's host PID, then read its descriptors through <code>/proc</code>, which works across namespaces:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># containerd/CRI-O nodes: host PID of the container's main process</span>
container_id=<span class="hljs-string">'replace-with-container-id'</span>
pid=$(<span class="hljs-built_in">sudo</span> crictl inspect <span class="hljs-string">"<span class="hljs-variable">$container_id</span>"</span> | jq -r <span class="hljs-string">'.info.pid'</span>)

<span class="hljs-comment"># Link targets are shown as paths inside the container</span>
<span class="hljs-built_in">sudo</span> find <span class="hljs-string">"/proc/<span class="hljs-variable">$pid</span>/fd"</span> -lname <span class="hljs-string">'*(deleted)'</span> -<span class="hljs-built_in">printf</span> <span class="hljs-string">'%p -&gt; %l\n'</span>
</code></pre><p>The paths in the link targets are relative to the container's mount namespace, so <code>/app/logs/app.log (deleted)</code> is the container's view. <a href="https://man7.org/linux/man-pages/man7/namespaces.7.html" rel="noopener noreferrer">Linux namespace documentation</a> explains why the same file has different paths from the node and from the container.</p>
<p>Kubernetes has an extra blind spot. When kubelet manages local ephemeral storage, it measures container logs, writable container layers and non-memory <code>emptyDir</code> volumes. Its directory-scan mode does not track deleted files that remain open, so a container can retain space that the kubelet does not categorize as in use. The same <a href="https://kubernetes.io/docs/concepts/storage/ephemeral-storage/" rel="noopener noreferrer">Kubernetes local ephemeral storage documentation</a> says a container can be evicted when its writable layer and log usage exceed its ephemeral-storage limit. Node-level disk pressure is still a reason to inspect the host.</p>
<p>For the lifecycle rules that apply when the data belongs on a PersistentVolume rather than a node's ephemeral storage, see <a href="https://devops-daily.com/posts/anatomy-of-kubernetes-persistent-storage">The Anatomy of Kubernetes Persistent Storage</a>.</p>
<h2>Incident checklist</h2><ol>
<li>Confirm the affected mount with <code>df -h &lt;mountpoint&gt;</code> and compare a scoped <code>du</code> result.</li>
<li>Run <code>sudo lsof -nP -a +L1 &lt;mountpoint&gt;</code>, and record PID, FD, size, link count and owner before changing anything.</li>
<li>If lsof is absent, search <code>/proc/[0-9]*/fd</code> for <code>*(deleted)</code> as root and inspect candidate sizes with <code>stat -L</code>.</li>
<li>Recover required data to a different filesystem, then restart the service or use its documented log-reopen action.</li>
<li>Use <code>: &gt; /proc/PID/fd/N</code> only for a verified emergency target, then confirm <code>df</code> changes and fix rotation or log limits.</li>
<li>If no holder exists, check <code>df -i</code>, ext4 reserved blocks, hidden files below mounts, and node-level container storage.</li>
</ol>
<p>The important distinction is simple: deleting a pathname is not the same thing as closing the file. Once the incident response follows that distinction, the <code>df</code> and <code>du</code> disagreement becomes evidence instead of a mystery.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Node.js November 2026 Security Release: How We Tracked and Patched It]]></title>
      <link>https://devops-daily.com/posts/node-november-2026-security-release-track-and-patch</link>
      <description><![CDATA[Node.js 22.23.5, 24.22.3 and 26.13.1 fix 16 CVEs, seven of them high severity in HTTP/2 and TLS. Here is what was fixed, who should upgrade first, the 15-minute watcher that told us the release was out, and the nvm, pm2 and systemd steps we used to move production in minutes.]]></description>
      <pubDate>Wed, 11 Nov 2026 17:50:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/node-november-2026-security-release-track-and-patch</guid>
      <category><![CDATA[Security]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Security]]></category><category><![CDATA[Node.js]]></category><category><![CDATA[CVE]]></category><category><![CDATA[pm2]]></category><category><![CDATA[Patch Management]]></category><category><![CDATA[Linux]]></category><category><![CDATA[systemd]]></category>
      <content:encoded><![CDATA[<p>On 4 November 2026 the Node.js project posted a pre-alert: new releases of the 26.x, 24.x and 22.x lines "on or shortly after, Wednesday, November 11, 2026", and "The highest severity issue fixed in this release is HIGH". No CVE numbers, no components, nothing to triage against. You know a fix is coming, you know it matters, and you do not know when it lands or what it touches.</p>
<p>That gap is where most patch delays come from. The release ships in the afternoon, nobody is watching, and the upgrade happens days later. This time we wrote a small watcher for the release, saw Node.js 24.22.3 appear at 16:58 UTC on release day, and had production on the new version with smoke tests passing 14 minutes later.</p>
<p>This post covers what the release fixes and who should move first, the watcher script, the exact nvm, pm2 and systemd steps we used, and the checklist we will reuse for the next one.</p>
<h2>TL;DR</h2><ul>
<li><strong>Fixed versions:</strong> Node.js <strong>22.23.5</strong>, <strong>24.22.3</strong> and <strong>26.13.1</strong>. <code>https://nodejs.org/dist/index.json</code> marks all three as security releases.</li>
<li><strong>What was fixed:</strong> 16 CVEs: 7 high, 6 medium, 3 low. All seven high-severity ones are in HTTP/2 and TLS handling: remote crashes, server memory sent to the peer, and a client-certificate bypass. llhttp, undici, sqlite and brotli were updated too.</li>
<li><strong>Move first:</strong> any Node process that terminates TLS or serves HTTP/2 to untrusted clients. Everyone on 22, 24 or 26 should upgrade. End-of-life lines get no fix.</li>
<li><strong>Watch for it:</strong> poll <code>dist/index.json</code> every 15 minutes, or subscribe to the <code>nodejs-sec</code> list and the vulnerability RSS feed.</li>
<li><strong>Upgrade order under nvm and pm2:</strong> install, reinstall pm2, back up <code>dump.pm2</code>, move apps one at a time with a health check, then move the pm2 daemon by restarting its systemd unit. Do not use <code>pm2 update</code> for that last step.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Linux host running Node.js apps installed with <strong>nvm</strong> and supervised by <strong>pm2</strong></li>
<li>A systemd unit for the pm2 daemon (the one <code>pm2 startup</code> generates)</li>
<li>Shell access as the app user, plus root for the unit</li>
<li><code>curl</code> and <code>python3</code> for the watcher and the checks</li>
<li>A health endpoint per app and a smoke test you trust</li>
</ul>
<h2>The announcement and the release</h2><p>Node.js security releases follow a fixed pattern. About a week ahead, a pre-alert goes up at <a href="https://nodejs.org/en/blog/vulnerability/november-2026-security-releases" rel="noopener noreferrer">nodejs.org/en/blog/vulnerability</a> with the target date, the affected lines, and the highest severity. The details stay private until the fixed builds are out, so nobody can work out the bugs from the patch before you can install it.</p>
<p>For this release:</p>
<ul>
<li><strong>4 November:</strong> pre-alert published. 26.x, 24.x and 22.x, highest severity HIGH.</li>
<li><strong>11 November, afternoon UTC:</strong> the builds appeared in <code>dist/index.json</code> with <code>"security": true</code>.</li>
<li><strong>11 November, 17:11 UTC:</strong> the advisory page was updated with the full CVE list (the time comes from the nodejs.org repository history).</li>
</ul>
<p>So the binaries were downloadable before the write-up existed. If you wait for the blog post, you start later than you need to.</p>
<p>You can confirm the fixed versions yourself:</p>
<pre><code class="hljs language-bash" tabindex="0">curl -s https://nodejs.org/dist/index.json | python3 -c <span class="hljs-string">'
import sys, json
for r in json.load(sys.stdin):
    if r["version"] in ("v26.13.1", "v24.22.3", "v22.23.5"):
        print(r["version"], r["date"], "security:", r["security"])'</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">v26.13.1 2026-11-10 security: True
v24.22.3 2026-11-10 security: True
v22.23.5 2026-11-11 security: True
</code></pre><p>The <code>date</code> field is the build date, which is why two of them show the day before release.</p>
<h2>What was fixed</h2><p>The advisory lists 16 CVEs. Severity and affected lines below are as the advisory states them.</p>
<table>
<thead>
<tr>
<th>CVE</th>
<th>Component</th>
<th>Severity</th>
<th>Lines</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td>CVE-2026-123863</td>
<td>HTTP/2</td>
<td>High</td>
<td>24, 26</td>
<td>Any unauthenticated client can abort the server through an assertion in session teardown, over TLS or cleartext</td>
</tr>
<tr>
<td>CVE-2026-123864</td>
<td>HTTPS</td>
<td>High</td>
<td>22, 24, 26</td>
<td>A zero-length TLS write re-enters the HTTP parser; memory from other connections, including TLS private keys, can be copied into a request body</td>
</tr>
<tr>
<td>CVE-2026-123865</td>
<td>HTTP/2</td>
<td>High</td>
<td>22, 24, 26</td>
<td>A protocol error during shutdown leaves the session behind; memory and file descriptors leak with no further cost to the attacker</td>
</tr>
<tr>
<td>CVE-2026-123866</td>
<td>HTTP/2</td>
<td>High</td>
<td>22, 24, 26</td>
<td>Stream teardown during a backpressured write sends freed memory to the peer</td>
</tr>
<tr>
<td>CVE-2026-126051</td>
<td>TLS</td>
<td>High</td>
<td>24, 26</td>
<td>Remote crash in TLS servers that use <code>SNICallback</code>, <code>OCSPRequest</code>, <code>newSession</code>, <code>resumeSession</code> or <code>server.addContext()</code></td>
</tr>
<tr>
<td>CVE-2026-123872</td>
<td>TLS</td>
<td>High</td>
<td>22, 24, 26</td>
<td>TLS 1.2 renegotiation can change the servername and carry client-certificate auth to a stricter host</td>
</tr>
<tr>
<td>CVE-2026-123874</td>
<td>HTTP/2</td>
<td>High</td>
<td>24, 26</td>
<td>Use-after-free when a <code>respondWithFile()</code> or <code>respondWithFD()</code> stream is reset during a pending write</td>
</tr>
<tr>
<td>CVE-2026-123828</td>
<td>HTTP/2</td>
<td>Medium</td>
<td>24, 26</td>
<td>One-byte heap overflow in <code>session.altsvc()</code> with a long origin</td>
</tr>
<tr>
<td>CVE-2026-123867</td>
<td>TLS (mTLS)</td>
<td>Medium</td>
<td>22, 24, 26</td>
<td>A CA-only SNI context falls back to the default client CA, bypassing per-host mTLS</td>
</tr>
<tr>
<td>CVE-2026-123868</td>
<td>HTTP/2</td>
<td>Medium</td>
<td>24, 26</td>
<td>Queued inbound DATA is not charged to <code>maxSessionMemory</code>, so one stream can hold hundreds of megabytes</td>
</tr>
<tr>
<td>CVE-2026-123871</td>
<td>HTTP proxy</td>
<td>Medium</td>
<td>22, 24, 26</td>
<td>Proxy credentials appear in <code>ERR_PROXY_INVALID_CONFIG</code> error messages</td>
</tr>
<tr>
<td>CVE-2026-123877</td>
<td>https.Agent</td>
<td>Medium</td>
<td>22, 24, 26</td>
<td>A TLS session from the default trust store can be reused by a request with a stricter <code>ca</code> policy</td>
</tr>
<tr>
<td>CVE-2026-123875</td>
<td>HTTP/2</td>
<td>Medium</td>
<td>22, 24, 26</td>
<td>Cancelling a <code>respondWithFile()</code> download crashes the server</td>
</tr>
<tr>
<td>CVE-2026-123829</td>
<td>Permission Model</td>
<td>Low</td>
<td>22, 24, 26</td>
<td>Recursive <code>fs.watch</code> leaks names of denied files (macOS and Windows)</td>
</tr>
<tr>
<td>CVE-2026-123862</td>
<td>zlib</td>
<td>Low</td>
<td>22, 24, 26</td>
<td><code>params()</code> plus <code>destroy()</code> throws an uncatchable assertion</td>
</tr>
<tr>
<td>CVE-2026-106071</td>
<td>Permission Model</td>
<td>Low</td>
<td>22, 24, 26</td>
<td>A wildcard filesystem grant can match paths it should not</td>
</tr>
</tbody></table>
<p>The release notes also update <strong>llhttp</strong> to 9.4.4 on all three lines, <strong>sqlite</strong> to 3.54.1, <strong>undici</strong> to 7.30.2 on 24.x and 8.12.4 on 26.x, and <strong>brotli</strong> to 1.2.1 on 24.x and 26.x.</p>
<blockquote>
<p><strong>Note</strong></p>
<p>The advisory and the release notes do not match exactly on the low-severity Permission Model items. The release notes list CVE-2026-123873 ("permission: check heap snapshot destination") and do not mention CVE-2026-106071. If you run with <code>--permission</code>, read both. The action is the same either way: upgrade.</p>
</blockquote>
<h2>Who should upgrade first</h2><p>Everyone on 22.x, 24.x or 26.x should upgrade. End-of-life lines such as 20.x and 25.x get no fixed release, and the advisory notes that end-of-life versions are always affected. Use the following order to decide what moves first.</p>
<ol>
<li><strong>Node terminates TLS or speaks HTTP/2 to the internet.</strong> <code>https.createServer</code>, <code>http2.createSecureServer</code>, <code>http2.createServer</code>, <code>tls.createServer</code>, or a framework option that enables them. Most of the high-severity bugs sit here, and several need nothing more than a default server.</li>
<li><strong>mTLS with per-host policies.</strong> CVE-2026-123872 and CVE-2026-123867 are authorization bypasses, not just crashes.</li>
<li><strong>Everything else.</strong> If a reverse proxy terminates TLS and talks plain HTTP/1.1 to Node, the HTTP/2 and TLS server bugs are not reachable from clients. The client-side fixes (<code>https.Agent</code> session reuse, proxy credentials in error messages, the undici update) still apply.</li>
</ol>
<p>Two quick checks tell you where you stand. First, which Node processes listen, and on which address:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Run as root so ss can see every process</span>
ss -ltnpH | <span class="hljs-keyword">while</span> <span class="hljs-built_in">read</span> -r _ _ _ addr _ <span class="hljs-built_in">users</span>; <span class="hljs-keyword">do</span>
  pid=$(<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$users</span>"</span> | grep -o <span class="hljs-string">'pid=[0-9]*'</span> | <span class="hljs-built_in">head</span> -1 | <span class="hljs-built_in">cut</span> -d= -f2)
  [ -n <span class="hljs-string">"<span class="hljs-variable">$pid</span>"</span> ] || <span class="hljs-built_in">continue</span>
  exe=$(<span class="hljs-built_in">readlink</span> <span class="hljs-string">"/proc/<span class="hljs-variable">$pid</span>/exe"</span> 2&gt;/dev/null)
  <span class="hljs-keyword">case</span> <span class="hljs-string">"<span class="hljs-variable">$exe</span>"</span> <span class="hljs-keyword">in</span> */node) <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$addr</span>  <span class="hljs-variable">$exe</span>"</span> ;; <span class="hljs-keyword">esac</span>   <span class="hljs-comment"># match the binary, not the process name</span>
<span class="hljs-keyword">done</span>
</code></pre><p>Anything bound to <code>0.0.0.0</code>, <code>[::]</code> or <code>*</code> is reachable from outside the host unless a firewall says otherwise. Second, which code paths you use:</p>
<pre><code class="hljs language-bash" tabindex="0">grep -rnE <span class="hljs-string">'https\.createServer|http2\.create(Secure)?Server|tls\.createServer|respondWith(File|FD)|SNICallback|addContext|altsvc|flushHeaders'</span> \
  --include=<span class="hljs-string">'*.js'</span> --include=<span class="hljs-string">'*.mjs'</span> --include=<span class="hljs-string">'*.cjs'</span> --include=<span class="hljs-string">'*.ts'</span> src/
</code></pre><p>No hits does not make you safe from the client-side fixes, but it does tell you how urgent the rest is.</p>
<h2>How we watched for the release</h2><p>The pre-alert gives a date, not a time. We did not want someone refreshing a browser tab all day, and we did not want a cron entry to remember to delete afterwards. So we started one background script on release morning that polls <code>dist/index.json</code> every 15 minutes, stops after about 16 hours, and exits as soon as a new 24.x shows up.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># node-release-watch.sh: poll nodejs.org until a Node 24 release newer than</span>
<span class="hljs-comment"># the current newest 24.x appears. Every 15 minutes, gives up after ~16 h.</span>
LATEST=v24.22.2   <span class="hljs-comment"># the newest 24.x when you start the watcher</span>
<span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> $(<span class="hljs-built_in">seq</span> 1 64); <span class="hljs-keyword">do</span>
  <span class="hljs-comment"># index.json is sorted newest first, so the first v24.* entry is the latest 24.x</span>
  v=$(curl -s -m 20 https://nodejs.org/dist/index.json | python3 -c <span class="hljs-string">'import sys,json; d=json.load(sys.stdin); print(next(x["version"] for x in d if x["version"].startswith("v24.")))'</span> 2&gt;/dev/null)
  <span class="hljs-keyword">if</span> [ -n <span class="hljs-string">"<span class="hljs-variable">$v</span>"</span> ] &amp;&amp; [ <span class="hljs-string">"<span class="hljs-variable">$v</span>"</span> != <span class="hljs-string">"<span class="hljs-variable">$LATEST</span>"</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-built_in">echo</span> <span class="hljs-string">"NEW NODE 24: <span class="hljs-variable">$v</span> (<span class="hljs-subst">$(date -u +%H:%M)</span> UTC)"</span>; <span class="hljs-built_in">exit</span> 0; <span class="hljs-keyword">fi</span>
  <span class="hljs-built_in">sleep</span> 900
<span class="hljs-keyword">done</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"no new Node 24 release after 16 h"</span>
</code></pre><p><strong>release day</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># start it in the background and forget about it</span>
$ <span class="hljs-built_in">chmod</span> +x node-release-watch.sh
$ <span class="hljs-built_in">nohup</span> ./node-release-watch.sh &gt; node-watch.log 2&gt;&amp;1 &amp;
<span class="hljs-comment"># hours later</span>
$ <span class="hljs-built_in">cat</span> node-watch.log
NEW NODE 24: v24.22.3 (16:58 UTC)
</code></pre><p>A few notes if you adapt it:</p>
<ul>
<li>A failed <code>curl</code> or a bad JSON response leaves <code>v</code> empty, so a network blip does not trigger a false alert.</li>
<li>Swap the <code>echo</code> for a call to your chat webhook if you want a ping instead of a log line.</li>
<li>To wake only for security releases, add <code>and x["security"]</code> to the filter and set <code>LATEST</code> to the newest 24.x security release.</li>
<li>For the long term, subscribe to the low-volume <code>nodejs-sec</code> mailing list and the <a href="https://nodejs.org/en/feed/vulnerability.xml" rel="noopener noreferrer">vulnerability RSS feed</a>. They tell you a release is coming. The watcher tells you the moment it is downloadable.</li>
</ul>
<h2>How we upgraded</h2><p>Our Node apps run on a Linux VM. Node comes from nvm, pm2 supervises the apps, and the pm2 daemon starts at boot from the systemd unit that <code>pm2 startup</code> generated. The timeline on 11 November, in UTC:</p>
<ol>
<li><strong>16:58</strong> watcher sees v24.22.3</li>
<li><strong>17:00</strong> nvm install, pm2, dump backup</li>
<li><strong>17:02</strong> apps moved one by one</li>
<li><strong>17:03</strong> move the pm2 daemon</li>
<li><strong>17:12</strong> all on 24.22.3, smoke tests pass</li>
</ol>
<h3>Step 1: install the new version and pm2</h3><p>As the app user:</p>
<pre><code class="hljs language-bash" tabindex="0">nvm install 24.22.3
nvm <span class="hljs-built_in">alias</span> default 24.22.3
nvm use 24.22.3
npm install -g pm2@6.0.13          <span class="hljs-comment"># same pm2 version you run now; globals are per Node version under nvm</span>
<span class="hljs-built_in">cp</span> ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.bak-$(<span class="hljs-built_in">date</span> -u +%Y%m%dT%H%M)
</code></pre><p>Under nvm, global packages live inside each Node version, so the new Node has no pm2 until you install it. The new <code>pm2</code> CLI then talks to the old daemon, which is fine for now.</p>
<h3>Step 2: find out how each entry picks its Node</h3><p>This is the part that decides whether a restart moves an app. pm2 stores an interpreter per entry:</p>
<pre><code class="hljs language-bash" tabindex="0">pm2 jlist | python3 -c <span class="hljs-string">'
import sys, json
for p in json.load(sys.stdin):
    e = p["pm2_env"]
    print(p["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("node_version"), sep="\t")'</span>
</code></pre><ul>
<li><strong>Bare <code>node</code>, fork mode:</strong> resolved through <code>PATH</code>. A restart with <code>--update-env</code> picks up the new version.</li>
<li><strong>An absolute path</strong> such as <code>/home/app/.nvm/versions/node/v24.22.2/bin/node</code>: written in when the app was started with <code>--interpreter</code>, or copied from the daemon's own Node. A restart reuses that path, so the entry has to be deleted and started again. If the ecosystem file hard-codes the path, fix the file first.</li>
<li><strong>Cluster mode</strong> (<code>exec_mode</code> is <code>cluster_mode</code>): the daemon forks the workers itself, so they run on the daemon's Node whatever the entry says. They move when the daemon moves in step 4.</li>
</ul>
<h3>Step 3: move apps one at a time</h3><p>One app, one health check, then the next. If a check fails, you stop with one app affected and the rest still serving. Run it as a script, so <code>exit</code> stops it at the first failure.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-function"><span class="hljs-title">health</span></span>() {  <span class="hljs-comment"># health &lt;name&gt; &lt;port&gt; [path]: retry up to 30 times until a 200</span>
  <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> $(<span class="hljs-built_in">seq</span> 1 30); <span class="hljs-keyword">do</span>
    [ <span class="hljs-string">"<span class="hljs-subst">$(curl -s -o /dev/null -w '%{http_code}' -m 5 <span class="hljs-string">"http://127.0.0.1:$2<span class="hljs-variable">${3:-/}</span>"</span>)</span>"</span> = 200 ] &amp;&amp; { <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$1</span> ok"</span>; <span class="hljs-built_in">return</span> 0; }
    <span class="hljs-built_in">sleep</span> 2
  <span class="hljs-keyword">done</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$1</span> NOT healthy on :<span class="hljs-variable">$2</span>"</span>; <span class="hljs-built_in">return</span> 1
}

<span class="hljs-comment"># Bare "node" interpreter: --update-env passes the new PATH</span>
pm2 restart web --update-env &amp;&amp; health web 3000 || <span class="hljs-built_in">exit</span> 1

<span class="hljs-comment"># Absolute interpreter path: recreate the entry from the ecosystem file</span>
pm2 delete api
pm2 start ecosystem.config.cjs --only api &amp;&amp; health api 3001 || <span class="hljs-built_in">exit</span> 1

pm2 save
</code></pre><p>Then check what each process actually runs. A 200 only tells you the app is up, not which Node it is on:</p>
<pre><code class="hljs language-bash" tabindex="0">pm2 jlist | python3 -c <span class="hljs-string">'import sys,json; [print(p["name"], p["pm2_env"].get("node_version"), p["pm2_env"]["status"], sep="\t") for p in json.load(sys.stdin)]'</span>
</code></pre><p>Two minutes in, every app was on 24.22.3 except one. That entry had no interpreter set, and in that case pm2 starts the script with the <strong>daemon's</strong> Node. Cluster-mode apps behave the same way, since the daemon forks them itself. Restarting it changed nothing, because the daemon was still on the old version. So the daemon had to move too.</p>
<h3>Step 4: move the pm2 daemon through systemd</h3><p>The unit that <code>pm2 startup</code> writes has the Node version in it twice: in <code>Environment=PATH=</code> and in the path to the pm2 binary. Point both at the new version, as root:</p>
<pre><code class="hljs language-bash" tabindex="0">UNIT=/etc/systemd/system/pm2-app.service
<span class="hljs-built_in">cp</span> <span class="hljs-string">"<span class="hljs-variable">$UNIT</span>"</span> <span class="hljs-string">"<span class="hljs-variable">$UNIT</span>.bak"</span>
sed -i <span class="hljs-string">'s#/node/v[0-9][0-9.]*/#/node/v24.22.3/#g'</span> <span class="hljs-string">"<span class="hljs-variable">$UNIT</span>"</span>
grep -n <span class="hljs-string">'/node/v'</span> <span class="hljs-string">"<span class="hljs-variable">$UNIT</span>"</span>     <span class="hljs-comment"># every match should now say v24.22.3</span>
systemctl daemon-reload
</code></pre><p>Then restart the daemon through the unit:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># As the app user, with the new pm2 on PATH</span>
pm2 save
<span class="hljs-built_in">cp</span> ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.pre-daemon
pm2 <span class="hljs-built_in">kill</span>                      <span class="hljs-comment"># no daemon left for resurrect to attach to</span>

<span class="hljs-comment"># As root</span>
systemctl restart pm2-app     <span class="hljs-comment"># ExecStart runs "pm2 resurrect" on the new Node</span>
</code></pre><p><code>pm2 kill</code> stops every app, and <code>pm2 resurrect</code> starts them all again from the dump, so plan for a short gap. Give it 30 to 60 seconds before you check.</p>
<blockquote>
<p><strong>Tip</strong></p>
<p><strong>Don't use <code>pm2 update</code> for this. Restart the systemd unit instead.</strong> <code>pm2 update</code> relaunches the daemon from your shell, on the Node that runs the pm2 CLI there. From a login shell or <code>sudo</code>, that can be an older version still installed on the box, and the new daemon then runs outside the unit that starts it at boot. The unit has the version written into it and is the same path a reboot takes, so restarting it gives the same result every time. If the daemon ever comes back without your apps: <code>pm2 kill</code>, copy <code>dump.pm2.pre-daemon</code> back to <code>~/.pm2/dump.pm2</code>, and restart the unit.</p>
</blockquote>
<h3>Step 5: verify from /proc, then smoke test</h3><p>pm2's view is a good start. The kernel's view is the one that counts. As root:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># The real binary of every Node process on the host, grouped</span>
<span class="hljs-keyword">for</span> p <span class="hljs-keyword">in</span> /proc/[0-9]*; <span class="hljs-keyword">do</span>
  exe=$(<span class="hljs-built_in">readlink</span> <span class="hljs-string">"<span class="hljs-variable">$p</span>/exe"</span> 2&gt;/dev/null) || <span class="hljs-built_in">continue</span>
  <span class="hljs-keyword">case</span> <span class="hljs-string">"<span class="hljs-variable">$exe</span>"</span> <span class="hljs-keyword">in</span> */node) <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$exe</span>"</span> ;; <span class="hljs-keyword">esac</span>
<span class="hljs-keyword">done</span> | <span class="hljs-built_in">sort</span> | <span class="hljs-built_in">uniq</span> -c

<span class="hljs-comment"># The daemon itself</span>
<span class="hljs-built_in">readlink</span> <span class="hljs-string">"/proc/<span class="hljs-subst">$(cat /home/app/.pm2/pm2.pid)</span>/exe"</span>
</code></pre><p>Every line should end in <code>v24.22.3/bin/node</code>. Then run the external smoke tests against the public URLs, not localhost. Ours passed at 17:12 UTC, 14 minutes after the watcher fired.</p>
<p>Once the new version has run cleanly for a day, remove the old ones with <code>nvm uninstall</code>. An old Node left on the box is something a shell, a script or a daemon can fall back to without telling you. Until you remove it, check <code>which node</code> in every shell you work from, including root's.</p>
<h2>The rest of the fleet</h2><p>The production host is the urgent part. The rest follows on its own schedule:</p>
<ul>
<li><strong>CI:</strong> <code>actions/setup-node</code> with <code>node-version: 24</code> picks up 24.22.3 without workflow changes once the runner's tool cache has it. If you want it today rather than when the runner image refreshes, pin <code>node-version: '24.22.3'</code>. <code>check-latest: true</code> helps too, but the version manifest it reads can lag nodejs.org.</li>
<li><strong>Container images:</strong> they need a rebuild. At the time of writing, Docker Hub had no <code>24.22.3-slim</code> tag yet and <code>node:24-slim</code> was last updated on 5 November, so a rebuild right now would still get the old Node. Check before you rebuild, then use <code>docker build --pull</code> and confirm with <code>docker run --rm --pull always node:24-slim node -v</code>.</li>
</ul>
<pre><code class="hljs language-bash" tabindex="0">curl -s https://hub.docker.com/v2/repositories/library/node/tags/24.22.3-slim |
  python3 -c <span class="hljs-string">'import sys,json; print(json.load(sys.stdin).get("last_updated", "not published yet"))'</span>
</code></pre><ul>
<li><strong>Dev machines:</strong> last, since they do not serve untrusted traffic. <code>nvm install 24.22.3 &amp;&amp; nvm alias default 24.22.3</code>.</li>
</ul>
<p>For a full audit of every place Node hides on a host (cron jobs, helper scripts, child processes, systemd units with old paths), see our <a href="https://devops-daily.com/posts/node-october-2026-security-release-find-every-node">October 2026 post on finding every Node you run</a>.</p>
<h2>Checklist for the next Node.js security release</h2><p><strong>When the pre-alert lands</strong></p>
<ol>
<li>Note the date, the affected lines and the highest severity.</li>
<li>List every host, image and pipeline that runs those lines.</li>
<li>Run the exposure checks above to decide the order.</li>
<li>Check each pm2 entry's interpreter, and fix ecosystem files that hard-code a Node path.</li>
</ol>
<p><strong>On release day</strong></p>
<ol>
<li>Start the watcher in the morning, and confirm the fixed versions and <code>"security": true</code> in <code>dist/index.json</code>.</li>
<li><code>nvm install</code> the new version, <code>nvm alias default</code>, reinstall pm2 under it.</li>
<li>Back up <code>~/.pm2/dump.pm2</code>.</li>
<li>Move apps one at a time: <code>restart --update-env</code> for bare <code>node</code>, delete and start for absolute paths. Health check each one, then <code>pm2 save</code>.</li>
<li>Check <code>node_version</code> in <code>pm2 jlist</code> for every app.</li>
<li>Point the systemd unit at the new version, <code>daemon-reload</code>, <code>pm2 kill</code>, <code>systemctl restart</code>. Never <code>pm2 update</code>.</li>
<li>Verify from <code>/proc</code>, then run external smoke tests.</li>
</ol>
<p><strong>After</strong></p>
<ol>
<li>Rebuild container images once the new tag is on Docker Hub, with <code>--pull</code>.</li>
<li>Make sure CI runs the new version (pin the exact version on release day).</li>
<li>Update dev machines.</li>
<li>After a day on the new version, <code>nvm uninstall</code> the old ones so nothing can fall back to them.</li>
</ol>
<p>The watcher is what made this fast: it cost nothing to run, and we started within minutes of the release instead of hours. The per-app health checks made every step safe to stop. Keep the checklist somewhere you will find it next time.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[We Gave an AI Agent a read_only Neon Role. Here Is What It Blocked]]></title>
      <link>https://devops-daily.com/posts/neon-read-only-role-agent-probes</link>
      <description><![CDATA[A Neon read_only role blocked every INSERT, UPDATE, DELETE and schema change we sent, and the Neon MCP Server with a pinned role stopped every durable write. What still ran comes from standard Postgres defaults, and the real risk was a read that stalled a migration.]]></description>
      <pubDate>Wed, 11 Nov 2026 12:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/neon-read-only-role-agent-probes</guid>
      <category><![CDATA[Security]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Neon]]></category><category><![CDATA[Postgres]]></category><category><![CDATA[MCP]]></category><category><![CDATA[AI Agents]]></category><category><![CDATA[Database Security]]></category><category><![CDATA[Least Privilege]]></category><category><![CDATA[Security]]></category>
      <content:encoded><![CDATA[<p>Teams now connect coding agents to their databases through the Neon MCP Server, and "read-only" is the control they reach for first. We wanted to know what that control stops. We created a <code>read_only</code> role with the Neon CLI and ran 40 one-statement probes against a Lakebase Postgres branch in four setups: a direct connection as that role, and three configurations of the Neon MCP Server. <strong>Every <code>INSERT</code>, <code>UPDATE</code>, <code>DELETE</code> and schema change we sent failed in every read-only setup.</strong> Over the direct connection, 23 of the 40 probes still ran, and 9 of them left something behind. Four were durable writes from a role named <code>read_only</code>: two rows written through a <code>SECURITY DEFINER</code> function (one of them through <code>EXPLAIN ANALYZE</code>), a large object, and a setting stored on the role itself. The combination in Neon's docs, the MCP Server's read-only mode with the role pinned, stopped all four.</p>
<p>The bigger risk was not a write. A 15-second read from the agent held a routine <code>ALTER TABLE</code> for 11.3 to 12.8 seconds, and all four of the app's writers were stuck behind it, the longest insert for just as long. Sending the same read to a read replica brought the migration down to 0.14 to 0.16 seconds. But nothing in the role or the MCP Server URL puts the agent on that replica: the URL can pin a role, not a compute.</p>
<h2>TLDR</h2><ul>
<li>The Neon MCP Server with <code>readonly=true</code> and a pinned <code>read_only</code> role stopped every durable write we tried. That is the setup to give an agent.</li>
<li>Almost everything that still ran over a direct connection comes from standard Postgres defaults granted to <code>PUBLIC</code>, not from Neon. The same probes behave the same way on any Postgres, and a few <code>REVOKE</code> statements close most of them.</li>
<li><code>neon roles create --access read_only</code> runs plain SQL as the database owner: <code>SELECT</code> on the tables that exist, <code>USAGE</code> on the schema, and default privileges for the owner's future tables. It does not touch what Postgres grants to <code>PUBLIC</code>.</li>
<li>Over a direct connection, that role could still create temp tables, call a <code>SECURITY DEFINER</code> function that writes, create a large object, change its own role settings, send <code>NOTIFY</code>, take advisory locks, and read a signing key from a function body.</li>
<li>The Neon MCP Server with <code>readonly=true&amp;roleName=&lt;role&gt;</code> wraps each statement in a <code>READ ONLY</code> transaction and logs in as the pinned role. That blocked every durable write. <code>NOTIFY</code>, advisory locks and reading function source still worked.</li>
<li><code>readonly=true</code> without a role pin runs as the owner. It still cannot write, but it ran <code>REINDEX</code>, took an <code>ACCESS EXCLUSIVE</code> lock and could read the query text of every other session.</li>
<li>The incident to plan for is lock queueing. Put agent reads on a read replica and give every migration a <code>lock_timeout</code>.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Neon account and an API key. The demo creates one throwaway project and deletes it at the end.</li>
<li>Node.js 22 or newer.</li>
<li>Basic Postgres privileges: roles, <code>GRANT</code>, and what <code>PUBLIC</code> means.</li>
<li>A few cents of compute. Neon's consumption history API, which we read after the run, shows 0.39 CU-hours for our whole build, about $0.09 at the Scale list price, including three full runs of the matrix and the runs we threw away. That figure is not one of the repo's stamped results.</li>
</ul>
<h2>Three layers, three different things</h2><p>A Neon setup for an agent has three controls, and they work at different levels. Mixing them up is where the surprises in this post come from.</p>
<ol>
<li><strong>Coding agent</strong> MCP client or a connection string</li>
<li><strong>Connection string</strong> logs in as agent_ro</li>
<li><strong>Neon MCP Server</strong> readonly=true&amp;roleName=agent_ro</li>
<li><strong>READ ONLY transaction</strong> a Postgres transaction mode</li>
<li><strong>Role privileges</strong> what --access read_only granted</li>
<li><strong>Lakebase Postgres</strong> branch compute</li>
</ol>
<p>Connections:</p>
<ul>
<li>Coding agent -&gt; Connection string</li>
<li>Coding agent -&gt; Neon MCP Server</li>
<li>Neon MCP Server -&gt; READ ONLY transaction (each run_sql)</li>
<li>READ ONLY transaction -&gt; Role privileges</li>
<li>Connection string -&gt; Role privileges (autocommit)</li>
<li>Role privileges -&gt; Lakebase Postgres</li>
</ul>
<h3>The read_only role is SQL that the CLI runs</h3><p><code>--access</code> is a feature of the Neon CLI (8.18.0 and newer) and of <code>@neon/sdk</code>, not a field of the REST API, whose role request has only a name and a <code>no_login</code> flag. For a scoped role, the SDK connects as the database owner and creates the role with SQL, in one transaction. The role starts as <code>NOLOGIN</code> with a throwaway password, Neon's API then sets the real password, and only then does the SDK turn on login. This is the list from <code>roleAccessStatements()</code> in <code>@neon/sdk</code>, with our names filled in:</p>
<pre><code class="hljs language-sql" tabindex="0"><span class="hljs-keyword">CREATE</span> ROLE "agent_ro" NOLOGIN PASSWORD <span class="hljs-string">'&lt;bootstrap&gt;'</span>;
<span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">CONNECT</span> <span class="hljs-keyword">ON</span> DATABASE "neondb" <span class="hljs-keyword">TO</span> "agent_ro";
<span class="hljs-keyword">GRANT</span> USAGE <span class="hljs-keyword">ON</span> SCHEMA "app" <span class="hljs-keyword">TO</span> "agent_ro";
<span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">ON</span> <span class="hljs-keyword">ALL</span> TABLES <span class="hljs-keyword">IN</span> SCHEMA "app" <span class="hljs-keyword">TO</span> "agent_ro";
<span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">DEFAULT</span> PRIVILEGES <span class="hljs-keyword">FOR</span> ROLE "neondb_owner" <span class="hljs-keyword">IN</span> SCHEMA "app"
  <span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">ON</span> TABLES <span class="hljs-keyword">TO</span> "agent_ro";
<span class="hljs-comment">-- after Neon replaces the bootstrap password:</span>
<span class="hljs-keyword">ALTER</span> ROLE "agent_ro" LOGIN;
</code></pre><p>Without <code>--schema</code>, the schema is <code>public</code>. Without <code>--access</code>, the role joins <code>neon_superuser</code>, which is admin on every database. The SDK's own comment states the limit that matters later: default privileges "cover tables the owner creates later, not tables other roles own".</p>
<p>This is the real run. The CLI prints the new role's password, and our script drops that field before anything is written:</p>
<p><strong>neon CLI 9.3.2</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ npx -y neon@9.3.2 roles create --project-id shy-bread-81087327 --branch br-green-rain-bdgi2170 --database neondb --name agent_ro --access read_only --schema app --output json
{
  <span class="hljs-string">"branch_id"</span>: <span class="hljs-string">"br-green-rain-bdgi2170"</span>,
  <span class="hljs-string">"name"</span>: <span class="hljs-string">"agent_ro"</span>,
  <span class="hljs-string">"protected"</span>: <span class="hljs-literal">false</span>,
  <span class="hljs-string">"authentication_method"</span>: <span class="hljs-string">"password"</span>,
  <span class="hljs-string">"created_at"</span>: <span class="hljs-string">"2026-11-11T07:44:31Z"</span>,
  <span class="hljs-string">"updated_at"</span>: <span class="hljs-string">"2026-11-11T07:44:31Z"</span>
}
</code></pre><p>The catalog afterwards matches the list exactly: no role memberships, <code>SELECT</code> on every table in <code>app</code>, one default ACL for the owner's future tables. The line that explains several results below is the database ACL, <code>=Tc/neondb_owner</code>. The empty name before <code>=</code> is <code>PUBLIC</code>, and <code>Tc</code> is <code>TEMPORARY</code> and <code>CONNECT</code>. Postgres grants those to every role by default, and it grants <code>EXECUTE</code> on new functions to <code>PUBLIC</code> too. The CLI does not revoke either, and it should not have to: changing database-wide defaults is not what "create a role" means.</p>
<h3>readonly=true wraps each statement in a READ ONLY transaction</h3><p>On the hosted Neon MCP Server, <code>readonly=true</code> (or the older <code>x-read-only</code> header) makes <code>run_sql</code> send the agent's statement through the Neon serverless driver as a one-statement transaction with <code>readOnly: true</code>. Postgres then refuses writes with SQLSTATE <code>25006</code>. The server also removes <code>run_sql_transaction</code> in this mode, because several caller statements in one shared transaction could end the read-only transaction early. In our runs, the read-only connection listed 59 tools and did not include <code>run_sql_transaction</code>. The <code>read_write</code> connection listed 140.</p>
<p><code>READ ONLY</code> is a transaction mode, not a privilege boundary. The Postgres docs list what it blocks: data changes, <code>CREATE</code>, <code>ALTER</code> and <code>DROP</code>, <code>COMMENT</code>, <code>GRANT</code>, <code>REVOKE</code>, <code>TRUNCATE</code>, and <code>EXPLAIN ANALYZE</code> or <code>EXECUTE</code> of any of those. They call it "a high-level notion of read-only that does not prevent all writes to disk". The list does not mention <code>NOTIFY</code>, advisory locks, <code>LOCK TABLE</code>, <code>REINDEX</code> or <code>SET</code>.</p>
<h3>roleName pins every SQL tool to one role</h3><p><code>roleName=agent_ro</code> on the server URL makes every SQL tool log in as that role. There is no fallback: a <code>role_name</code> argument that names another role is refused, and with a pin in place the server also refuses to create scoped roles, because that runs SQL as the owner. The pin covers SQL only, so Neon's docs suggest combining it with <code>readonly=true</code> and <code>category</code> to limit the Management API tools as well. The same page says: "We recommend MCP for development and testing only, not production environments."</p>
<p>One gap in the controls: <code>run_sql</code> also takes a <code>compute_id</code>. In the server source, the URL options are <code>readonly</code>, <code>category</code>, <code>projectId</code> and <code>roleName</code>, so a compute cannot be pinned there. The agent chooses whether to read from the primary or from a replica, and your configuration cannot make it use the replica.</p>
<h2>The probe matrix</h2><p>We seeded a small fake SaaS database (500 customers, 5,000 orders, 10,098 order items, an audit log, a <code>SECURITY DEFINER</code> function that writes to it, and a function with a fake webhook signing key in its body). Then we ran 40 probes in four setups:</p>
<ul>
<li><strong>S1</strong>: a direct connection as <code>agent_ro</code>, what an agent gets if you hand it a connection string.</li>
<li><strong>S2</strong>: the Neon MCP Server with <code>readonly=true</code> and no role pin, so it runs as the owner.</li>
<li><strong>S3</strong>: <code>readonly=true&amp;roleName=agent_ro</code>, the combination Neon's docs show.</li>
<li><strong>S4</strong>: <code>roleName=agent_rw</code>, a <code>--access read_write</code> role with writes on, as a baseline.</li>
</ul>
<p>The MCP probes go through the hosted server with the MCP SDK client and an API key, the same calls an agent makes. No model is involved, so every probe is exactly the statement we wrote.</p>
<p>Each probe declared its expected result for each setup before the run: allowed, or denied with an exact SQLSTATE. A result that did not match was a harness failure, not a pass. After each probe, an owner connection checked for a trace: a new row in the audit log, a new large object, a new role setting, or a <code>NOTIFY</code> that a listening session received. Temp tables and session advisory locks were checked inside the probe's own session on the direct connection.</p>
<p>The final run matched its declared expectations in all 160 probe-runs. Two earlier complete runs are kept in <code>results/discarded/</code>, and <code>scripts/compare-matrix-runs.mjs</code> shows that all three agree on every probe's result in all four setups: allowed or refused, the SQLSTATE, and the trace.</p>
<p><strong>40 probes per setup: not blocked, and how many left a trace</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>left a trace</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>S1 direct, read_only role</td>
<td>23 probes</td>
<td>9 probes</td>
<td>direct</td>
</tr>
<tr>
<td>S2 MCP readonly, no role pin</td>
<td>21 probes</td>
<td>3 probes</td>
<td>MCP</td>
</tr>
<tr>
<td>S3 MCP readonly + roleName</td>
<td>18 probes</td>
<td>3 probes</td>
<td>MCP</td>
</tr>
<tr>
<td>S4 MCP read_write role</td>
<td>27 probes</td>
<td>8 probes</td>
<td>write baseline</td>
</tr>
</tbody></table>
<p><em>Bars: probes that ran without an error. Ticks: probes that left a trace (a durable write, a temp table or advisory lock, or a NOTIFY another session received). In S2 to S4, temp-table and advisory-lock traces are inferred from a successful call. Final run, Lakebase Postgres 18, aws-us-east-2.</em></p>
<p>Here is every probe. <code>ok</code> means it ran, <code>ok*</code> means it ran and left a trace, and a code means it was refused with that SQLSTATE. Durable writes and <code>NOTIFY</code> were checked from an owner session in every setup. Temp tables and advisory locks were checked inside the session only on the direct connection (S1). In S2 to S4 those two marks are inferred from a successful call.</p>
<table>
<thead>
<tr>
<th>Probe</th>
<th>S1 direct</th>
<th>S2 no pin</th>
<th>S3 pin</th>
<th>S4 rw</th>
</tr>
</thead>
<tbody><tr>
<td><code>INSERT</code>, <code>UPDATE</code>, <code>DELETE</code> a row</td>
<td>42501</td>
<td>25006</td>
<td>25006</td>
<td>ok</td>
</tr>
<tr>
<td><code>TRUNCATE</code></td>
<td>42501</td>
<td>25006</td>
<td>25006</td>
<td>42501</td>
</tr>
<tr>
<td><code>CREATE TABLE</code>, <code>ALTER TABLE</code>, <code>DROP INDEX</code></td>
<td>42501</td>
<td>25006</td>
<td>25006</td>
<td>42501</td>
</tr>
<tr>
<td><code>nextval()</code></td>
<td>42501</td>
<td>25006</td>
<td>42501</td>
<td>ok</td>
</tr>
<tr>
<td><code>setval()</code></td>
<td>42501</td>
<td>25006</td>
<td>42501</td>
<td>42501</td>
</tr>
<tr>
<td><code>CREATE TEMP TABLE</code> (2 probes)</td>
<td>ok*</td>
<td>25006</td>
<td>25006</td>
<td>ok*</td>
</tr>
<tr>
<td><code>SECURITY DEFINER</code> function that writes</td>
<td>ok*</td>
<td>25006</td>
<td>25006</td>
<td>ok*</td>
</tr>
<tr>
<td>Same, through <code>EXPLAIN ANALYZE</code></td>
<td>ok*</td>
<td>25006</td>
<td>25006</td>
<td>ok*</td>
</tr>
<tr>
<td><code>lo_from_bytea()</code> large object</td>
<td>ok*</td>
<td>25006</td>
<td>25006</td>
<td>ok*</td>
</tr>
<tr>
<td><code>ALTER ROLE agent_ro SET statement_timeout = 0</code></td>
<td>ok*</td>
<td>25006</td>
<td>25006</td>
<td>42501</td>
</tr>
<tr>
<td><code>NOTIFY</code>, <code>pg_notify()</code> (2 probes)</td>
<td>ok*</td>
<td>ok*</td>
<td>ok*</td>
<td>ok*</td>
</tr>
<tr>
<td><code>pg_advisory_lock()</code></td>
<td>ok*</td>
<td>ok*</td>
<td>ok*</td>
<td>ok*</td>
</tr>
<tr>
<td>Other advisory lock calls (3 probes)</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td><code>LOCK TABLE ... ACCESS SHARE</code></td>
<td>25P01</td>
<td>ok</td>
<td>ok</td>
<td>25P01</td>
</tr>
<tr>
<td><code>LOCK TABLE ... ACCESS EXCLUSIVE</code></td>
<td>25P01</td>
<td>ok</td>
<td>42501</td>
<td>25P01</td>
</tr>
<tr>
<td><code>SELECT ... FOR UPDATE</code></td>
<td>42501</td>
<td>25006</td>
<td>25006</td>
<td>ok</td>
</tr>
<tr>
<td>Function body with the signing key (<code>pg_proc.prosrc</code>)</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td>Customer emails, role list, settings (4 probes)</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td>Other sessions' query text</td>
<td>ok, 0 rows</td>
<td>ok, 11 rows</td>
<td>ok, 0 rows</td>
<td>ok, 0 rows</td>
</tr>
<tr>
<td><code>SET statement_timeout = 0</code>, <code>SET work_mem</code>, <code>RESET ALL</code>, <code>LISTEN</code></td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td><code>generate_series</code> sum over 2 million rows</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td><code>SET ROLE neondb_owner</code></td>
<td>42501</td>
<td>ok</td>
<td>42501</td>
<td>42501</td>
</tr>
<tr>
<td><code>pg_read_file()</code>, <code>pg_ls_dir()</code></td>
<td>42501</td>
<td>42501</td>
<td>42501</td>
<td>42501</td>
</tr>
<tr>
<td><code>CREATE EXTENSION</code></td>
<td>42501</td>
<td>25006</td>
<td>25006</td>
<td>42501</td>
</tr>
<tr>
<td><code>REINDEX TABLE</code></td>
<td>42501</td>
<td>ok</td>
<td>42501</td>
<td>42501</td>
</tr>
</tbody></table>
<h3>What the direct connection left open</h3><p>S1 is the case to worry about, because a connection string is the easiest thing to give an agent and the hardest to take back. The role blocked what it was built to block: every row write, every schema change, sequences, <code>SELECT ... FOR UPDATE</code>, and every route to more privilege (<code>SET ROLE</code>, server files, extensions). Most of the 23 probes that ran are reads and session settings that a read-only role is expected to allow.</p>
<p>The nine traces are the interesting part:</p>
<ul>
<li><strong>Four durable writes.</strong> <code>app.record_touch()</code> is a <code>SECURITY DEFINER</code> function owned by the owner, and <code>PUBLIC</code> can execute it. Calling it wrote a row to <code>app.audit_log</code> with no table privilege at all. <code>EXPLAIN ANALYZE</code> runs the statement it explains, so the same call through <code>EXPLAIN</code> wrote a second row. <code>lo_from_bytea()</code> created a large object, which needs no grant by default. And <code>ALTER ROLE agent_ro SET statement_timeout = 0</code> stored a setting on the role in the catalog, where it applies to every future session of that role.</li>
<li><strong>Three session effects.</strong> Two temp tables (one filled with 100 order IDs) and a session-level advisory lock. They disappear when the session ends, but a long-lived agent session can hold an advisory lock that blocks your app's own locks for as long as it likes.</li>
<li><strong>Two signals.</strong> <code>NOTIFY</code> and <code>pg_notify()</code> reached a session that was listening on the channel. If your app uses <code>LISTEN</code> for job queues or cache invalidation, a read-only agent can trigger it.</li>
</ul>
<p>There was also one disclosure without a trace: the function body with <code>signing_key text := 'whsec_demo_...'</code> came back from <code>pg_proc.prosrc</code>. Postgres makes the <code>pg_proc</code> catalog readable by every role by default, so a secret in a function body is readable by any role that can connect. Ours came back in all four setups.</p>
<p>None of this is a Neon bug. Each item is standard Postgres behavior: <code>PUBLIC</code> holds <code>TEMP</code> on the database and <code>EXECUTE</code> on functions by default, a large object needs no grant, the <code>NOTIFY</code> statement needs no privilege, and <code>pg_notify()</code> and the advisory-lock functions run on the <code>EXECUTE</code> that <code>PUBLIC</code> holds on built-in functions. It took a <code>SECURITY DEFINER</code> function in our schema to turn <code>EXECUTE</code> into a write, and many real schemas have one. The role does what it says on the label; it just does not say "sandbox".</p>
<h3>What the MCP read-only mode closed, and what it did not</h3><p>S3 refused all four durable writes and both temp tables with <code>25006</code>. The pinned role kept the privilege checks too: <code>nextval()</code>, <code>ACCESS EXCLUSIVE</code> and <code>SET ROLE</code> failed with <code>42501</code> before the read-only check ran. Three traces remained. The two <code>NOTIFY</code> probes reached our listening session. <code>pg_advisory_lock()</code> succeeded, and since that function returns only once it holds the lock, the lock was taken. We did not observe it from another session, because each <code>run_sql</code> call is a separate request to the MCP Server and we had no way to look while it was held.</p>
<p>The function body with the signing key came back in S3 as well.</p>
<h3>Without a role pin, readonly=true runs as the owner</h3><p>S2 shows why Neon pairs <code>readonly=true</code> with <code>roleName</code>. Without a pin, the server logs in as the database owner, and only the transaction mode stands between the agent and your data. It still refused every <code>INSERT</code>, <code>UPDATE</code>, <code>DELETE</code> and schema change we sent, and all four probes that wrote durably over the direct connection. But as the owner, the agent could run <code>REINDEX TABLE</code>, take an <code>ACCESS EXCLUSIVE</code> lock and <code>SET ROLE</code> to the owner. The owner is also a member of <code>pg_read_all_stats</code>, so <code>pg_stat_activity</code> showed it the query text of every other client session, the app's and the <code>read_only</code> agent's included. Our probe counted 11 visible rows for other sessions, some of them background processes with no query. The direct <code>read_only</code> role saw none. Query text often holds literal values: emails in a <code>WHERE</code> clause, tokens in an <code>INSERT</code>.</p>
<h3>We got 9 predictions wrong</h3><p>Our first runs disagreed with our declared expectations in 9 cells. Each disagreement is a small lesson about Postgres:</p>
<ul>
<li><code>LOCK TABLE</code> needs a transaction block. As a single autocommit statement, on the direct connection or the <code>read_write</code> MCP connection, it fails with <code>25P01</code>. The read-only MCP wrapper supplies the transaction, so there it runs, and the privilege decides.</li>
<li><code>REINDEX</code> is not on the list a <code>READ ONLY</code> transaction blocks, so the owner ran it in S2.</li>
<li>The order of checks differs by statement. With the pinned role, <code>nextval()</code> fails the privilege check (<code>42501</code>) first, while <code>CREATE EXTENSION</code> fails the read-only check (<code>25006</code>) first.</li>
<li><code>SELECT ... FOR UPDATE</code> takes row locks, so <code>READ ONLY</code> refuses it even for the owner.</li>
</ul>
<p>We changed the predictions, never the outcomes, and the reason for each is a comment in <code>lib/probes.mjs</code>. Two other mismatches were harness bugs: a <code>DELETE</code> that hit a foreign key, and an <code>INSERT</code> that collided with a row from the previous run. We fixed the probe and added cleanup, and neither one hid a result.</p>
<h2>Incident one: a read that stalls a migration</h2><p>A read takes an <code>ACCESS SHARE</code> lock on every table it touches and holds it until its transaction ends. <code>ALTER TABLE ... ADD COLUMN</code> needs <code>ACCESS EXCLUSIVE</code>, so it waits for the read. Then Postgres queues every new lock request on that table behind the waiting <code>ALTER</code>, including the app's ordinary inserts. The agent never writes anything, and your app still stops writing.</p>
<p>The setup: four app writers insert into <code>app.orders</code> through the pooled connection string, one insert per writer every 50 ms. At 2 seconds the agent runs a 15-second read on <code>app.orders</code> through <code>run_sql</code> (S3). At 4 seconds the owner runs <code>ALTER TABLE app.orders ADD COLUMN note text</code> without a <code>lock_timeout</code>. A direct owner connection samples <code>pg_locks</code> every 250 ms. We ran three variants three times each:</p>
<table>
<thead>
<tr>
<th>Variant</th>
<th>Migration</th>
<th>Longest app insert</th>
<th>Inserts over 1 s</th>
<th>Backends waiting</th>
<th>Inserts completed</th>
</tr>
</thead>
<tbody><tr>
<td>Agent reads on the primary</td>
<td>11.32 to 12.80 s</td>
<td>11.29 to 12.70 s</td>
<td>4 per run</td>
<td>5</td>
<td>104 to 124</td>
</tr>
<tr>
<td>Agent reads on a read replica</td>
<td>0.14 to 0.16 s</td>
<td>0.17 to 0.51 s</td>
<td>0</td>
<td>0</td>
<td>350 to 378</td>
</tr>
<tr>
<td>Primary, migration with <code>lock_timeout = '2s'</code></td>
<td>gave up at 2.13 s (<code>55P03</code>)</td>
<td>2.02 to 2.10 s</td>
<td>4 per run</td>
<td>5</td>
<td>331 to 357</td>
</tr>
</tbody></table>
<p>Ranges are over three runs of each variant. With nothing in the way, one insert took 128 to 167 ms at the median, which is the round trip from our test machine to aws-us-east-2 through the pooler. "Backends waiting" is the most lock requests on <code>app.orders</code> that <code>pg_locks</code> showed as not yet granted: the <code>ALTER</code> and all four app writers.</p>
<p><strong>Longest app insert while the agent's 15-second read ran</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>migration time</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Agent reads on the primary</td>
<td>12.39s</td>
<td>12.43s</td>
<td>primary</td>
</tr>
<tr>
<td>Migration with lock_timeout 2s</td>
<td>2.09s</td>
<td>2.13s</td>
<td>lock_timeout</td>
</tr>
<tr>
<td>Agent reads on a read replica</td>
<td>0.17s</td>
<td>0.16s</td>
<td>replica</td>
</tr>
</tbody></table>
<p><em>Median of 3 runs per variant. Bars: longest single insert from 4 app writers. Ticks: how long the ALTER TABLE took (or waited before lock_timeout gave up). Lakebase Postgres 18, aws-us-east-2.</em></p>
<p>The replica fix works because a read replica is a separate compute: a lock taken there is not a lock on the primary. It moves the conflict, it does not remove it. The <code>ALTER</code> still has to be replayed on the replica, where it waits for the agent's read or cancels it. In all 3 replica runs, the agent's read finished without being cancelled.</p>
<p><code>lock_timeout</code> is the fix on the migration side. The <code>ALTER</code> gave up after 2.1 seconds with <code>55P03</code> instead of holding the app for 12.8, and the migration tool can retry. The app still waited about 2 seconds, which is the price of finding out. This is the same advice as in <a href="https://devops-daily.com/posts/rehearse-migrations-on-a-neon-branch">rehearsing migrations on a Neon branch</a>: every migration should set a <code>lock_timeout</code>.</p>
<p>Our first version of this test got the incident wrong. The app writers inserted into a side table, not <code>app.orders</code>, so their latency during the stall matched the baseline and the run seemed to show that only the migration waited. The migration numbers were right, the app numbers measured the wrong table. That run is kept in <code>results/discarded/</code> in the repo.</p>
<h2>Incident two: one heavy read is cheap, and nothing bounds a loop</h2><p>The role does not limit how much a read may scan. We gave the agent one legal heavy read: a parallel scan of a 2-million-row, 137 MB table with a chain of 20 <code>md5()</code> calls per row. It ran through the MCP Server as <code>agent_ro</code>, first on the primary, allowed to autoscale from 0.25 to 4 CU, and then on a read replica capped at 1 CU. An owner connection on the same compute read <code>num_cpus()</code> from the <code>neon_utils</code> extension about once a second.</p>
<table>
<thead>
<tr>
<th>Compute</th>
<th>Read time</th>
<th>Peak vCPU</th>
<th>vCPU-seconds, at most</th>
<th>Scale cost, at most</th>
<th>Launch cost, at most</th>
</tr>
</thead>
<tbody><tr>
<td>Primary, 0.25 to 4 CU</td>
<td>25.8 s</td>
<td>2</td>
<td>32.84</td>
<td>$0.00203</td>
<td>$0.00097</td>
</tr>
<tr>
<td>Read replica, 0.25 to 1 CU</td>
<td>29.2 s</td>
<td>1</td>
<td>31.71</td>
<td>$0.00196</td>
<td>$0.00093</td>
</tr>
</tbody></table>
<p>The primary reported 1 vCPU for most of the read and 2 vCPU only in the last four samples, about a second apart. <code>num_cpus()</code> rounds up to a whole vCPU, so a compute at 0.25 CU reports 1. That makes these figures an upper bound: we multiply by Neon's list price of $0.222 per CU-hour on Scale ($0.106 on Launch) and assume every "1" was a full CU. We did not use the consumption API, because it lags a run by minutes.</p>
<p>So one heavy read costs a fraction of a cent, and the capped replica was about 3 seconds slower for about the same cost. That is not the risk. The risk is that nothing in the role bounds how often the agent runs that read or how long a session stays open. The matrix showed that the agent can run <code>SET statement_timeout = 0</code>, and on the direct connection it can store that on its own role. An agent that polls also keeps the compute from suspending; we measured that effect in <a href="https://devops-daily.com/posts/what-keeps-a-neon-compute-awake">what keeps a Neon compute awake</a>. A capped replica for agents bounds the size of the compute, and a role-level <code>statement_timeout</code> bounds the common case, as long as you remember that the session can lower it.</p>
<h2>Drift: when the role stops matching your schema</h2><p>A <code>read_only</code> role is a snapshot of the grants at the moment it was created. Two ordinary changes moved the schema away from it. Both failed closed, which is the safe direction, but both are the kind of thing that ends with someone granting more than they meant to.</p>
<ol>
<li><strong>A table created by another role.</strong> Many teams run migrations as a separate role. We created a <code>migrator</code> role and had it create <code>app.migrator_table</code>. <code>agent_ro</code> got <code>42501 permission denied</code>, because the default privileges from <code>--access read_only</code> are <code>FOR ROLE neondb_owner</code>: they cover the owner's new tables and nobody else's. After <code>ALTER DEFAULT PRIVILEGES FOR ROLE migrator IN SCHEMA app GRANT SELECT ON TABLES TO agent_ro</code>, the migrator's next table was readable. The table it had already created still returned <code>42501</code>, because default privileges apply only to objects created after them. You need both the default privileges for the migration role and a one-time <code>GRANT</code> for what already exists.</li>
<li><strong>A new schema.</strong> <code>app2.secrets</code> returned <code>42501 permission denied for schema app2</code>, because the role was created with <code>--schema app</code>. An agent that cannot see a schema may report that the data does not exist, which is a wrong answer rather than a leak. Pass <code>--schema</code> once per schema, or re-run the grants when you add one.</li>
</ol>
<h2>Hardening, then the re-run</h2><p>This is the hardening we applied as the owner, from <code>schema/harden.sql</code>:</p>
<pre><code class="hljs language-sql" tabindex="0"><span class="hljs-comment">-- TEMP and EXECUTE are PUBLIC defaults; take them back for everyone</span>
<span class="hljs-comment">-- and grant them only to the roles that run the app.</span>
<span class="hljs-keyword">REVOKE</span> TEMPORARY <span class="hljs-keyword">ON</span> DATABASE neondb <span class="hljs-keyword">FROM</span> PUBLIC;
<span class="hljs-keyword">GRANT</span> TEMPORARY <span class="hljs-keyword">ON</span> DATABASE neondb <span class="hljs-keyword">TO</span> neondb_owner, agent_rw;

<span class="hljs-keyword">REVOKE</span> <span class="hljs-keyword">EXECUTE</span> <span class="hljs-keyword">ON</span> <span class="hljs-keyword">ALL</span> FUNCTIONS <span class="hljs-keyword">IN</span> SCHEMA app <span class="hljs-keyword">FROM</span> PUBLIC;
<span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">DEFAULT</span> PRIVILEGES <span class="hljs-keyword">FOR</span> ROLE neondb_owner <span class="hljs-keyword">IN</span> SCHEMA app
  <span class="hljs-keyword">REVOKE</span> <span class="hljs-keyword">EXECUTE</span> <span class="hljs-keyword">ON</span> FUNCTIONS <span class="hljs-keyword">FROM</span> PUBLIC;
<span class="hljs-keyword">GRANT</span> <span class="hljs-keyword">EXECUTE</span> <span class="hljs-keyword">ON</span> <span class="hljs-keyword">FUNCTION</span> app.record_touch(text) <span class="hljs-keyword">TO</span> neondb_owner, agent_rw;

<span class="hljs-comment">-- Bounds for the agent role. These are defaults: a session can still</span>
<span class="hljs-comment">-- change statement_timeout with SET.</span>
<span class="hljs-keyword">ALTER</span> ROLE agent_ro CONNECTION LIMIT <span class="hljs-number">5</span>;
<span class="hljs-keyword">ALTER</span> ROLE agent_ro <span class="hljs-keyword">SET</span> statement_timeout <span class="hljs-operator">=</span> <span class="hljs-string">'30s'</span>;
<span class="hljs-keyword">ALTER</span> ROLE agent_ro <span class="hljs-keyword">SET</span> lock_timeout <span class="hljs-operator">=</span> <span class="hljs-string">'2s'</span>;
<span class="hljs-keyword">ALTER</span> ROLE agent_ro <span class="hljs-keyword">SET</span> idle_in_transaction_session_timeout <span class="hljs-operator">=</span> <span class="hljs-string">'15s'</span>;
</code></pre><p>Revoking <code>TEMPORARY</code> from <code>PUBLIC</code> affects every role that relied on the default, not only the agent, so grant it back to the roles that need it before you run this in production. The same goes for <code>EXECUTE</code>.</p>
<p>Then we ran S1 and S3 again. On the direct connection, four probes flipped from allowed to <code>42501</code>: both temp tables, the <code>SECURITY DEFINER</code> write and its <code>EXPLAIN ANALYZE</code> twin. That took S1 from 23 probes that ran to 19, and from four durable writes to two. The two that stayed open have no grant to revoke: <code>lo_from_bytea()</code> still created a large object, and <code>ALTER ROLE agent_ro SET statement_timeout = 0</code> still ran. <code>NOTIFY</code>, the advisory locks and the function body with the signing key were unchanged too: the <code>NOTIFY</code> statement needs no privilege, the hardening revoked <code>EXECUTE</code> only in the <code>app</code> schema and not on built-in functions such as <code>pg_notify()</code> and <code>pg_advisory_lock()</code>, and <code>pg_proc</code> stays readable.</p>
<p>On the pinned MCP connection, nothing opened or closed. The <code>READ ONLY</code> wrapper already refused everything the hardening addresses, and the only difference was the error: the <code>SECURITY DEFINER</code> calls now failed with <code>42501</code> (no <code>EXECUTE</code>) instead of <code>25006</code>, because the privilege check runs first.</p>
<p>So the hardening matters for the connection-string case, and it is not enough on its own there. The two durable writes it cannot close are why a connection string is weaker than the same role behind the read-only MCP Server. One note on the re-run: the probe runner clears the role's <code>statement_timeout</code> before it starts, so the 30-second default was not in place during the re-run. No probe depends on it.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>One region, one Postgres version, one schema.</strong> Lakebase Postgres 18 in aws-us-east-2, a small fake schema. A schema with more functions, extensions or roles has more <code>PUBLIC</code> defaults to check.</li>
<li><strong>The pooled connection string was not probed.</strong> Our direct probes used the unpooled host. Through the pooler (PgBouncer in transaction mode), session effects such as <code>SET</code>, temp tables and session advisory locks behave differently, because a session does not keep one backend.</li>
<li><strong>No escape attempts.</strong> We probed what single statements can do. We did not try to get around the <code>READ ONLY</code> wrapper or the role pin, beyond checking that <code>run_sql_transaction</code> is gone and that <code>SET ROLE</code> is refused.</li>
<li><strong>No model in the loop.</strong> The probes are what a careless or misled agent could send, not what a given model chooses to send.</li>
<li><strong>Small samples for the incidents.</strong> Three runs per lock variant, one run per compute for cost. The direction is clear, but the exact seconds are not a benchmark. The cost figures are upper bounds because <code>num_cpus()</code> rounds up.</li>
</ul>
<h2>What to do</h2><ol>
<li>Give agents the MCP Server with <code>readonly=true&amp;roleName=&lt;a read_only role&gt;</code>, not a connection string. In our runs that combination stopped every durable write.</li>
<li>If an agent must have a connection string, revoke <code>TEMPORARY</code> and <code>EXECUTE</code> from <code>PUBLIC</code> in the app's schemas first, and keep secrets out of function bodies.</li>
<li>Keep agent reads on a read replica, and set a <code>lock_timeout</code> in every migration. A lock taken on the replica is not a lock on the primary, and <code>lock_timeout</code> makes a migration give up after a set time, so the writes queued behind it wait at most that long. The MCP URL cannot pin a compute, so the replica takes agent instructions that pass its <code>compute_id</code>, or your own layer in front of the database. We did not run the probe matrix on a replica.</li>
<li>Re-check grants after migrations that a different role runs, and pass <code>--schema</code> for every schema the agent should read.</li>
</ol>
<h2>Run it yourself</h2><p><a href="https://github.com/The-DevOps-Daily/neon-read-only-agent" rel="noopener noreferrer">The-DevOps-Daily/neon-read-only-agent on GitHub</a></p>
<p>The repository has one script per stage, and each stage writes JSON lines to <code>results/</code>, stamped with a hash of the code that produced them. <code>npm run verify-stamps</code> checks every file against the commit in <code>results/STAMPED_AT</code>.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">cp</span> .env.example .<span class="hljs-built_in">env</span>      <span class="hljs-comment"># add a Neon API key</span>
npm install
npm <span class="hljs-built_in">test</span>                  <span class="hljs-comment"># offline tests, no database</span>
npm run project           <span class="hljs-comment"># one throwaway project, Postgres 18</span>
npm run seed              <span class="hljs-comment"># fake orders schema and data</span>
npm run roles             <span class="hljs-comment"># neon roles create --access read_only / read_write</span>
npm run matrix            <span class="hljs-comment"># 40 probes x 4 setups</span>
npm run lock-stall        <span class="hljs-comment"># incident one</span>
npm run cost              <span class="hljs-comment"># incident two</span>
npm run drift
npm run harden
npm run rerun             <span class="hljs-comment"># S1 and S3 again after hardening</span>
npm run summarize
npm run verify-stamps
npm run teardown          <span class="hljs-comment"># deletes the project</span>
</code></pre><p>The run is cheap: the incident stages are a few minutes of a 0.25 CU compute, and the cost stage briefly allows 4 CU. Run <code>teardown</code> when you are done, because the read replica and the primary both count toward CU-hours while they are awake.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[EC2 Can Now Tell You Where IMDSv1 Is Still Enabled]]></title>
      <link>https://devops-daily.com/posts/ec2-imdsv1-enabled-metric</link>
      <description><![CDATA[AWS added a configuration metric for IMDSv1 on November 5. Here is how to separate migration gaps from active usage and retire tokenless metadata access safely.]]></description>
      <pubDate>Tue, 10 Nov 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/ec2-imdsv1-enabled-metric</guid>
      <category><![CDATA[AWS]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[AWS]]></category><category><![CDATA[EC2]]></category><category><![CDATA[Security]]></category><category><![CDATA[CloudWatch]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>An EC2 instance can make no tokenless metadata requests all week and still allow them tomorrow. That distinction makes an IMDSv2 migration surprisingly easy to declare finished too early. AWS's new <strong>ImdsV1Enabled</strong> CloudWatch metric gives you a configuration signal to put beside your usage graphs. You can use it to find the remaining exceptions, test one workload at a time, and catch settings that drift back.</p>
<p>AWS <a href="https://aws.amazon.com/about-aws/whats-new/2026/11/ec2-imdsv1enabled-cloudwatch-metric/" rel="noopener noreferrer">announced the metric on November 5, 2026</a>, with availability in every AWS Region and no additional charge for the metric itself. It does not migrate instances for you. The useful change is that your migration dashboard can now answer whether IMDSv1 is still permitted, even when applications happen not to be using it.</p>
<h2>Prerequisites</h2><ul>
<li>AWS CLI v2 configured for the account and Region you want to inspect.</li>
<li>Permission to describe EC2 instances and read CloudWatch metrics.</li>
<li>A staging instance that represents the application, agents, and scheduled jobs you run in production.</li>
<li>A workload owner available before you change metadata settings. The inventory commands below are read-only; the final change command is not.</li>
</ul>
<h2>Three signals, three different questions</h2><p>AWS's <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-transition-to-version-2.html" rel="noopener noreferrer">IMDSv2 transition guide</a> defines the new metric as <code>1</code> when HTTP tokens are optional and <code>0</code> when they are required. Unlike the request counters, it is emitted from configuration rather than metadata access.</p>
<table>
<thead>
<tr>
<th>Signal</th>
<th>Question it answers</th>
<th>How to use it</th>
</tr>
</thead>
<tbody><tr>
<td><code>ImdsV1Enabled</code></td>
<td>Does this instance still allow tokenless access?</td>
<td>Find configuration exceptions.</td>
</tr>
<tr>
<td><code>MetadataNoToken</code></td>
<td>Did a tokenless request succeed?</td>
<td>Find callers that need migration.</td>
</tr>
<tr>
<td><code>MetadataNoTokenRejected</code></td>
<td>Was a tokenless request rejected after enforcement?</td>
<td>Investigate compatibility problems.</td>
</tr>
</tbody></table>
<p>The two request metrics count successful tokenless calls and rejected attempts respectively; the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html" rel="noopener noreferrer">EC2 metrics reference</a> documents both. A quiet request counter does not tell you which configuration is in place.</p>
<p>Consider a backup task that runs once a month. A five-day observation window might never execute it. My recommendation is to keep an explicit exception list with an owner, the workload's longest relevant execution cycle, and the next test date. That makes a migration gap actionable instead of turning it into another permanently yellow dashboard.</p>
<ol>
<li><strong>Find exceptions</strong> Configuration still permits v1</li>
<li><strong>Exercise callers</strong> Boot, jobs, agents, containers</li>
<li><strong>Require tokens</strong> Verify health and rejected calls</li>
</ol>
<h2>Build an inventory before changing anything</h2><p>Start with one account and Region. This example lists running and stopped instances whose metadata options still make tokens optional:</p>
<pre><code class="hljs language-bash" tabindex="0">aws ec2 describe-instances \
  --region eu-west-1 \
  --filters \
    Name=instance-state-name,Values=running,stopped \
    Name=metadata-options.http-tokens,Values=optional \
  --query <span class="hljs-string">'Reservations[].Instances[].{Id:InstanceId,State:State.Name,Endpoint:MetadataOptions.HttpEndpoint,Tokens:MetadataOptions.HttpTokens,HopLimit:MetadataOptions.HttpPutResponseHopLimit}'</span> \
  --output table \
  --no-cli-pager
</code></pre><p>The <a href="https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html" rel="noopener noreferrer">DescribeInstances API</a> supports these filters and returns the metadata options. Keep the endpoint column: <code>HttpEndpoint=disabled</code> is a separate state, not proof of a working IMDSv2 migration. Record those instances separately rather than enabling metadata access just to make every row look alike.</p>
<p>Repeat the inventory for the accounts and Regions in your actual scope. An empty result in one Region is not a fleet-wide result. Include stopped instances in your review because they can return to service later.</p>
<p>Then discover the metric series available to your caller:</p>
<pre><code class="hljs language-bash" tabindex="0">aws cloudwatch list-metrics \
  --region eu-west-1 \
  --metric-name ImdsV1Enabled \
  --output json \
  --no-cli-pager
</code></pre><p>This command lists series and their dimensions, not current values. Use the returned series in CloudWatch to inspect datapoints. AWS documents that <a href="https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html" rel="noopener noreferrer">ListMetrics can take up to 15 minutes to show a new metric</a>, and it omits series that have not reported in two weeks. An empty response is therefore a reason to check account, Region, visibility, and instance configuration, not a green migration result.</p>
<h2>Test the workload, then require IMDSv2</h2><p>Choose one representative staging instance and run its real lifecycle: bootstrapping, service restart, deployment, scheduled maintenance, and credentials refresh. Include monitoring and backup agents. For containers, test from inside the container as well as from the host.</p>
<p>AWS notes that a metadata response hop limit of <code>1</code> <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html" rel="noopener noreferrer">can cause problems in container environments</a>. Do not change hop limits across the fleet just because a host-level test passed; investigate the affected network path separately.</p>
<p>Once the workload is ready, replace the example ID and apply the token requirement to that instance:</p>
<pre><code class="hljs language-bash" tabindex="0">aws ec2 modify-instance-metadata-options \
  --region eu-west-1 \
  --instance-id i-0123456789abcdef0 \
  --http-tokens required \
  --no-cli-pager
</code></pre><p>The <a href="https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-metadata-options.html" rel="noopener noreferrer">modify command</a> can return <code>pending</code> for a running instance. Verify that the setting reaches <code>applied</code>:</p>
<pre><code class="hljs language-bash" tabindex="0">aws ec2 describe-instances \
  --region eu-west-1 \
  --instance-ids i-0123456789abcdef0 \
  --query <span class="hljs-string">'Reservations[].Instances[].MetadataOptions'</span> \
  --output json \
  --no-cli-pager
</code></pre><p>Expect <code>HttpTokens</code> to be <code>required</code> and <code>State</code> to be <code>applied</code>. The commands here are examples, not a transcript from a production account.</p>
<blockquote>
<p><strong>Important</strong></p>
<p>Do not pipe every inventory result straight into the modification command. First check application health and rejected tokenless requests on the staging instance. Keep any rollback scoped to the affected workload and record why it was necessary.</p>
</blockquote>
<h2>Keep replacement instances from undoing the work</h2><p>Changing one running instance leaves a second question: what happens when Auto Scaling replaces it? Update the launch configuration your fleet actually uses, then launch a replacement in staging and inspect its effective metadata options.</p>
<p>AWS's <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html" rel="noopener noreferrer">configuration precedence rules</a> put explicit launch settings ahead of account defaults, then AMI settings. Account defaults also do not rewrite existing instances. Review the launch template version your Auto Scaling group selects rather than assuming a newly edited template is already in use.</p>
<p>For the rollout, keep two views side by side: configuration exceptions and application health. Use <code>MetadataNoToken</code> during preparation and <code>MetadataNoTokenRejected</code> after enforcement. Keep missing telemetry visibly unknown, and assign each remaining exception an owner.</p>
<p>The useful finishing condition is specific: the intended instances require tokens, representative workloads still work, and replacement instances inherit the same setting. The new metric makes the configuration part easier to watch continuously.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Opsville: A Free Pixel-Art DevOps Town in Your Browser]]></title>
      <link>https://devops-daily.com/posts/opsville-free-devops-town-game</link>
      <description><![CDATA[Opsville is a free browser game. Walk a pixel-art town, fix eight DevOps districts with our simulators and quizzes, and see how we built it.]]></description>
      <pubDate>Mon, 09 Nov 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/opsville-free-devops-town-game</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[Games]]></category><category><![CDATA[Learning]]></category><category><![CDATA[TypeScript]]></category><category><![CDATA[Phaser]]></category><category><![CDATA[Cloudflare]]></category>
      <content:encoded><![CDATA[<p>Opsville is a small pixel-art town where every district is broken, and you are the one on call. It is free, it runs in your browser, and it is live today at <a href="https://devops-daily.com/opsville/" rel="noopener noreferrer">devops-daily.com/opsville</a>. Most quests open one of the simulators from our <a href="https://devops-daily.com/simulators">simulators page</a>, then ask you three questions from our quiz bank. Fix all eight districts and the town gets its Launch Day.</p>
<p>This post covers the game first, then how we built it. Here is the 30-second trailer:</p>
<figure>
<video controls preload="none" playsinline width="1280" height="720" poster="/images/posts/opsville-free-devops-town-game/trailer-poster.jpg"><source src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/trailer.mp4" type="video/mp4" /></video>
<figcaption>The Opsville trailer, recorded in the game.</figcaption>
</figure><p><img src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/title-skyline.webp" alt="The Opsville title screen in front of a pixel skyline of the town's buildings" /></p>
<h2>Prerequisites</h2><ul>
<li>A modern browser on a desktop or a phone.</li>
<li>A keyboard, a gamepad or a touch screen.</li>
<li>No account. Your progress saves in your browser.</li>
</ul>
<h2>Walk into town</h2><p>You start in the town square. Mayor Chris has a problem: the town goes live soon, and none of the districts work. That is the <strong>Launch Day</strong> story. Each district is a chapter, with a closing scene once you fix it. The Agent, the robot from our book <a href="https://devops-daily.com/books/devops-survival-guide">The DevOps Survival Guide</a>, waits in the square and comments when you walk into a new district.</p>
<p><img src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/launch-day-town-square.webp" alt="Mayor Chris announces Launch Day in the town square, next to the quest board, the wardrobe and the leaderboard" /></p>
<p>There are 8 districts and 34 quests. The districts open one after another:</p>
<ol>
<li>Linux Lodge</li>
<li>Docker Docks</li>
<li>Git Garden</li>
<li>CI/CD Railway</li>
<li>Kubernetes Castle</li>
<li>Observatory</li>
<li>Security Fort</li>
<li>Neon Data Depot</li>
</ol>
<p>A broken district looks broken. Smoke rises from its landmark, its lamps flicker at night, and its people grumble. When you fix it, lights go up over its sign and its music opens up.</p>
<p><img src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/linux-lodge-snow.webp" alt="Linux Lodge in falling snow, with the Agent's tip from the book: check the disk and the processes before you reboot" /></p>
<h2>How a quest works</h2><p>Walk up to someone and press E, or tap Talk on a phone. They give you a task, such as "The disk is full" in Linux Lodge. The quest opens a simulator from our site in a panel: the Linux terminal, the <a href="https://devops-daily.com/games/fork-bomb-simulator">Fork Bomb Simulator</a>, the <a href="https://devops-daily.com/games/docker-escape">Container Security Challenge</a>, the <a href="https://devops-daily.com/games/k8s-scheduler">Kubernetes Scheduler Challenge</a> and more. When you finish, you answer three questions from our quiz bank. Two right answers pass the quest.</p>
<p>From Docker Docks on, each district ends with a boss. A boss quiz plays as a <strong>live incident</strong>: a status board with three systems that go from "Down" to "Fixed" as you answer.</p>
<p><img src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/boss-incident.webp" alt="A boss quiz as a live incident: the container runtime is fixed while the host kernel and the image registry are still down" /></p>
<p>Each quest gives you XP and a badge. Each cleared district gives you something new: a bike after Linux Lodge, a pet for each of the first five districts (a penguin, a little whale, a cat, a dog and a mini Agent), and new hats.</p>
<h2>Things to do between quests</h2><ul>
<li><strong>The pager.</strong> After you fix Linux Lodge, an incident in an open district pages you now and then. You get two minutes to get there and 30 seconds to fix it. You can snooze it, or turn it off in Help.</li>
<li><strong>The daily quest.</strong> One question a day from the quiz bank. Seven days in a row earns the Streak beanie.</li>
<li><strong>Your house.</strong> You get a log cottage in Linux Lodge with your name on the sign. Inside, there is a cup for every cleared district and a medal for every badge. The garden next to it grows a plant for each day you played in the last four weeks.</li>
<li><strong>The newsstand.</strong> The latest DevOps Daily posts, in the town square.</li>
<li><strong>Photo mode.</strong> Press F, pick a frame and a filter, and save a PNG.</li>
<li><strong>Your look.</strong> A character creator on first start, and a wardrobe in the square for later changes.</li>
<li><strong>The Advent board.</strong> From 1 to 24 December, the town gets snow, lights and a board with one door a day.</li>
</ul>
<p><img src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/house-and-streak-garden.webp" alt="Your log cottage in Linux Lodge, with its name sign, mailbox and the streak garden beside it" /></p>
<p>Twelve lost packets are hidden around town. Find all of them, and an old hatch by the lake opens. Below it is a server room that has not been rebooted in a long time.</p>
<p><img src="https://devops-daily.com/images/posts/opsville-free-devops-town-game/server-room-basement.webp" alt="The secret basement server room, with rows of blinking racks and a terminal labelled opsville-prod-1" /></p>
<h2>Accounts are optional</h2><p>You can play the whole game without an account. If you sign in with a free DevOps Daily account, your save follows you to other devices. You can also join the weekly leaderboard (opt-in, with a board name only), help fill a weekly town goal with everyone else's XP, and race district speedruns.</p>
<p>The Help panel has a larger text switch. You can play with a keyboard, a gamepad or touch controls. If your system asks for reduced motion, the game turns off the moving effects. Sound is off until you turn it on.</p>
<h2>How we built it</h2><h3>Vite, TypeScript and Phaser 3</h3><p>Phaser draws the world: the map, the people, the weather and the effects. The panels, quizzes and HUD are plain HTML on top of the canvas.</p>
<h3>The map is a text file</h3><p>The whole world is one text file: 72 x 50 characters, where one character is one 16 x 16 tile. A build script turns it into Tiled JSON with ground, decor, collision and object layers. Moving a building is a diff you can review. Here are four rows from the top of the map:</p>
<pre><code class="hljs language-text" tabindex="0">P..*.:::.......::::.*..P................ggggg...P..%%%%........%%%%....P
P.v........)..:....v...P................SSDSS...P..HHHH........HHHH.T..P
P..*..T.:::::::.*..T.*.P........[.........:.....P..HHHH....]...HHHH....P
P.*.......:......*.....P########################P..........:...........P
</code></pre><p><code>P</code> is a pine, <code>:</code> a stone path, <code>#</code> a rail track, <code>g</code> a grey roof, <code>S</code> a stone wall and <code>D</code> a door. Characters such as <code>[</code> and <code>)</code> are objects, defined in a list below the grid.</p>
<h3>Static files inside the main site</h3><p>The game builds with the base path <code>/opsville/</code>. A release script copies the build into the site's repo, adds the game's response headers, and opens one pull request. Cloudflare Pages deploys it with the rest of the static site. The game needs no server of its own.</p>
<h3>Simulators in a frame</h3><p>A quest loads <code>/games/&lt;slug&gt;/embed?embed=opsville&amp;run=&lt;id&gt;</code> in a frame. When you finish, the simulator sends a message on the same origin:</p>
<pre><code class="hljs language-typescript" tabindex="0"><span class="hljs-comment">// Sent by the simulator when the player finishes it</span>
{ <span class="hljs-attr">type</span>: <span class="hljs-string">'devops-daily:game-complete'</span>, <span class="hljs-attr">game</span>: <span class="hljs-string">'&lt;slug&gt;'</span>, <span class="hljs-attr">run</span>: <span class="hljs-string">'&lt;id&gt;'</span> }
</code></pre><p>On phones the simulator opens in a new tab, and the same message goes over a <code>BroadcastChannel</code>. If no message arrives, a "Done, take the quiz" button covers it. Either way, the quiz is the real gate.</p>
<ol>
<li><strong>Quest-giver</strong> talk to a local</li>
<li><strong>Simulator</strong> /games embed</li>
<li><strong>game-complete</strong> postMessage</li>
<li><strong>Quiz gate</strong> 2 of 3 to pass</li>
<li><strong>XP and badge</strong> server-checked</li>
</ol>
<h3>XP that the server decides</h3><p>When you are signed in, the game only says "I finished quest X". The learning platform's API looks the quest up in its own list and decides the XP. Each quest counts once per player, two quests must be at least 30 seconds apart, and a player can count at most 24 quests in 24 hours. The rows live in the learning platform's existing Neon database, so the game did not need a new one. A test in the game fails when its quest list drifts from the server's list.</p>
<h3>Art from two sources</h3><p>The tiles and characters come from <a href="https://www.kenney.nl" rel="noopener noreferrer">Kenney's</a> CC0 packs: Tiny Town, Tiny Dungeon and RPG Urban Pack. The landmarks, signs and portraits are generated pixel art. A Python script cuts out their background, scales them to the pixel grid, and maps every colour to one warm palette shared with the tiles. That way both sources look like one game. The music is ours too: chords and patterns played with WebAudio, with no sound files.</p>
<h2>Play it and tell us what breaks</h2><p><strong>Play Opsville free at <a href="https://devops-daily.com/opsville/" rel="noopener noreferrer">devops-daily.com/opsville</a>.</strong> Start in Linux Lodge, and see how far you get before the pager goes off.</p>
<p>Found a bug, or have an idea for a quest or a district? <a href="https://github.com/The-DevOps-Daily/devops-daily/issues/new/choose" rel="noopener noreferrer">Open an issue on our GitHub repo</a> and tell us.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DevOps Weekly Digest - Week 46, 2026]]></title>
      <link>https://devops-daily.com/news/2026-week-46</link>
      <description><![CDATA[⚡ Curated updates from Kubernetes, cloud native tooling, CI/CD, IaC, observability, and security - handpicked for DevOps professionals!]]></description>
      <pubDate>Mon, 09 Nov 2026 00:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/news/2026-week-46</guid>
      <category><![CDATA[DevOps News]]></category>
      <content:encoded><![CDATA[<blockquote>
<p>📌 <strong>Handpicked by DevOps Daily</strong> - Your weekly dose of curated DevOps news and updates!</p>
</blockquote>
<hr />
<h2>⚓ Kubernetes</h2><h3>📄 Donating KubeRay to CNCF</h3><p>The PARK stack (PyTorch, AI models, Ray, and Kubernetes) is emerging as the de facto open source standard for production AI. Within that stack, KubeRay has become the bridge connecting scalable AI app</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/11/09/donating-kuberay-to-cncf/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AWS Nitro Sandboxes: Hardware-enforced workload isolation on Amazon EKS</h3><p>AWS Nitro Sandboxes bring hardware-enforced, per-pod isolation to Amazon EKS. Every sandboxed pod runs in its own lightweight virtual machine with the isolation of a dedicated Amazon EC2 instance and </p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 AWS Containers Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/containers/aws-nitro-sandboxes-hardware-enforced-workload-isolation-on-amazon-eks/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Now generally available: GKE dynamic slicing for higher resilience, and efficiency for TPUs</h3><p>Running large-scale AI workloads on TPU and Google Kubernetes Engine (GKE) requires infrastructure that adapts as fast as your models change. Dynamic Slicing is generally available on GKE, which decou</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/containers-kubernetes/gke-dynamic-slicing-on-tpus/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Cloud Pathways, a distributed ML runtime for massive accelerator fleets, now GA</h3><p>Today, we are thrilled to announce the general availability (GA) of Cloud Pathways (Pathways on Cloud) on Google Kubernetes Engine (GKE) as a core runtime pillar of the Google Cloud AI Hypercomputer. </p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/ai-infrastructure/cloud-pathways-a-distributed-ml-runtime-is-ga/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Scaling OpenTelemetry &amp; Autonomous Kubernetes Ops at KubeCon NA 2026</h3><p>Explore New Relic’s OpenTelemetry, autonomous operations, and Kubernetes observability at KubeCon NA 2026.</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/observability/kubecon-na-2026-opentelemetry-autonomous-operations" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Red Hat OpenShift 5: The intelligent application platform</h3><p>Red Hat OpenShift 5 represents the next evolution of your core application platform. It maintains the foundations you rely on: A consistent operational model built on Kubernetes, supporting containers</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/intelligent-red-hat-openshift-5" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Red Hat OpenShift 5: Bring stronger security to applications and AI</h3><p>Every team wants to get AI models and modern apps into production quickly, but no platform engineer wants to gamble on unvetted code or leaky access boundaries. Red Hat OpenShift 5 is built to give pl</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/red-hat-openshift-5-bring-stronger-security-applications-and-ai" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Monitor VMware vSphere Kubernetes Service (VKS) Clusters with New Relic</h3><p>New Relic is validated on VMware vSphere Kubernetes Service (VKS). See how platform teams on VCF monitor clusters, pods, and apps in one place.</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/infrastructure-monitoring/vmware-vsphere-kubernetes-service-monitoring" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Canonical validates Charmed Apache Spark on VMware vSphere Kubernetes Service</h3><p>Enterprise data teams now have further validations and support around their big data processing workloads deployed using Charmed Apache Spark on VMware vSphere Kubernetes Service (VKS), – the Kubernet</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/canonical-validates-charmed-apache-spark-on-vmware-vsphere-kubernetes" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Kubeflow Trainer v2.4: Bringing Workload-Aware Scheduling to TrainJob</h3><p>Imagine launching an AI/ML distributed training job across 64 GPUs. Two frequent problems happen: Idle Resource Trap: The cluster finds capacity for 63 GPUs and sits idle waiting for the 64th to join,</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Kubeflow Blog</strong></p>
<p><a href="https://blog.kubeflow.org/kubeflow-trainer-v2.4-release/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>☁️ Cloud Native</h2><h3>📄 New research: Virtual machines remain central to cloud native infrastructure as AI workloads evolve</h3><p>A Linux Foundation Research study, sponsored by CNCF and Broadcom, explores how organizations are using virtual machines alongside containers, and the growing importance of workload isolation for AI. </p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/11/09/new-research-virtual-machines-remain-central-to-cloud-native-infrastructure-as-ai-workloads-evolve/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Run Linux applications in hypervisor-isolated micro-VMs with Hyperlight</h3><p>More and more cloud native systems run code nobody on the team wrote: the Python an AI agent generated a second ago, or a plugin a customer just uploaded. Each of them needs a boundary between...</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/11/09/run-linux-applications-in-hypervisor-isolated-micro-vms-with-hyperlight/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Scaling an autonomous agent from one team to fifty: A serverless-to-container decision framework</h3><p>QA teams shipping complex products generate test failures around the clock. Every failure needs a bug ticket with steps, screenshots, The post Scaling an autonomous agent from one team to fifty: A ser</p>
<p><strong>📅 Nov 8, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/scaling-autonomous-agents-framework/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Lima v2.3: Native on Windows, VMnet on Linux, and menu TUI</h3><p>Back in July with v2.2, we added support for Windows guests. On November 6, we shipped v2.3 of Lima and now it runs without extra dependencies on Windows: VMs start through the same Host Compute Syste</p>
<p><strong>📅 Nov 8, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/11/08/lima-v2-3-native-on-windows-vmnet-on-linux-and-menu-tui/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Docker Sandboxes vs. Containers: Choosing the Right Isolation for AI Agents</h3><p>A container shares the host kernel. Docker Sandboxes are microVMs with their own kernel and private Docker Engine. See how they differ and when to use each.</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-sandboxes-vs-containers/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Announcing etcd-operator v0.3.0</h3><p>Today we are proud to announce the release of etcd-operator v0.3.0. This release represents a substantial rewrite of the operator code in order to move from StatefulSet to a custom controller that can</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 etcd Blog</strong></p>
<p><a href="https://etcd.io/blog/2026/announcing-etcd-operator-v0.3.0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Docker and ActiveState Extending Secure Defaults to the Packages Developers Install</h3><p>Docker partners with ActiveState to continue to provide one standard for trusted content across everything you and your teams build on.</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-activestate-trusted-content/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 CNCF Score project improved its security posture by joining the Docker-Sponsored Open Source Program</h3><p>See how CNCF Score improved container security with Docker Scout, Hardened Images, and the Docker-Sponsored Open Source Program.</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/cncf-score-project-improved-its-security-posture-by-joining-the-docker-sponsored-open-source-program/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Red Hat OpenShift is now generally available in Google Cloud’s sovereign regions</h3><p>Data residency rules and sovereignty mandates often determine where a workload is allowed to run. To meet those requirements and keep the platform your teams know and trust, Red Hat is expanding its c</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/red-hat-openshift-now-generally-available-google-clouds-sovereign-regions" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Our Agent Skills for OpenFaaS</h3><p>In this post I’ll explain what Agent Skills we’ve created for OpenFaaS users to move faster with serverless functions. From setup, to development, to tracking down troublesome issues. How we got to co</p>
<p><strong>📅 Nov 3, 2026</strong> • <strong>📰 OpenFaaS Blog</strong></p>
<p><a href="https://www.openfaas.com/blog/openfaas-agent-skills/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔄 CI/CD</h2><h3>📄 How to design a decision-grade pilot for agentic AI</h3><p>Agentic AI changes the nature of a technology evaluation. When an AI system only generates an answer, a person reviews the output and decides whether to use it. An agent goes further: It gathers conte</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/how-to-develop-a-decision-grade-ai-pilot/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Beyond code generation: A developer’s guide to agentic engineering</h3><p>Writing code got faster. The work around it didn't. See how you can plan, write code, review changes, and investigate production issues with GitHub Copilot, and what work still needs your judgment. Th</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/ai-and-ml/github-copilot/beyond-code-generation-a-developers-guide-to-agentic-engineering/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Your Guide to the Developer Tools Track at AWS re:Invent 2026</h3><p>AWS re:Invent 2026 is almost here, and if you’re a developer, the Developer Tools (DVT) track is where you’ll want to spend your week. This year is all about agentic development, from multi-agent work</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 AWS DevOps Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/devops/your-guide-to-the-developer-tools-track-at-aws-reinvent-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Run Maestro tests on real Android devices using AWS Device Farm</h3><p>Learn how ManoMano runs Maestro mobile UI tests on real Android devices using the AWS Device Farm custom test environment. This post covers packaging your YAML flows, configuring the test spec, instal</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 AWS DevOps Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/devops/run-maestro-tests-on-real-android-devices-using-aws-device-farm/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 An expedition for expeditious PR reviews</h3><p>How we’re keeping review standards high, and notifications low, as PR volume grows.</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/an-expedition-for-expeditious-pr-reviews/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Pentagon wants to know who checks code when AI writes it</h3><p>The Department of War now treats AI-generated code as unverified until it’s proven otherwise. DoW Instruction 8430.01, “Accelerated Mission Software,” took effect September 8, 2026, and is the departm</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/department-of-war-instruction-for-ai-generated-code/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Inside GitHub’s new Git storage architecture</h3><p>GitHub's new Git storage architecture is built so thousands of agents can push to a single repository at once. Here's how it works. The post Inside GitHub’s new Git storage architecture appeared first</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/engineering/architecture-optimization/inside-githubs-new-git-storage-architecture/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Release integrity: Carrying intent from PR to production</h3><p>Your software factory can confirm a deployment succeeded, but release integrity asks whether users got the experience you intended.</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/release-integrity-carrying-intent-from-pr-to-production/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Accessibility statements now have a home on GitHub</h3><p>GitHub now highlights ACCESSIBILITY.md files and helps maintainers document accessibility goals, known barriers, supported environments, contributor expectations, and reporting processes. The post Acc</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/open-source/maintainers/accessibility-statements-now-have-a-home-on-github/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Agents write the code. Release Autopilot gets it safely to customers.</h3><p>What LaunchDarkly showed live on the Control Panel: how to safely automate releases at agent speed, from flagging to cleanup.</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/control-panel-recap-release-autopilot/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AI changed how we write software. We must change how we release it.</h3><p>Introducing Release Autopilot.</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/introducing-release-autopilot/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why AI Coding Agents Lose Context After the PR</h3><p>AI coding agents lose context once a PR enters the pipeline. See how code context and delivery context work together across the SDLC | Blog</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/why-ai-coding-agents-lose-context-after-pr" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🏗️ IaC</h2><h3>📄 Pulumi at KubeCon NA 2026: attendee guide</h3><p>KubeCon is in Salt Lake City this year, and we’ve got a full week planned. Here’s where to find us: booth 200, two talks and a panel from our team, a co-located day we’re sponsoring, and a party on We</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/pulumi-at-kubecon-na-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📊 Observability</h2><h3>📄 Any agent anywhere: How to use Grafana Cloud with Claude, Cursor, and any other AI agent</h3><p>AI agents are becoming part of everyday engineering workflows, helping teams write code, troubleshoot issues, and automate tasks. But to be truly useful in production, these agents need access to more</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/any-agent-anywhere-how-to-use-grafana-cloud-with-claude-cursor-and-any-other-ai-agent/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Only include resource detectors you need by building your own Collector</h3><p>There are many reasons for building your own OpenTelemetry Collector distribution. The existing distributions are designed to be broadly useful, which means they include many more components than what</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/collector-with-less-dependencies/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What’s new with Grafana variables: less clutter, easier reuse across dashboards, and more</h3><p>Template variables are one of the most used features in Grafana, providing users with a dropdown to turn one dashboard into hundreds: pick a different cluster, region, or service, and every panel upda</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/whats-new-with-grafana-variables-less-clutter-easier-reuse-across-dashboards-and-more/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Announcing the OpenTelemetry JavaScript SDK 3.0</h3><p>OpenTelemetry JavaScript has released SDK 3.0 (stable v3.0.0 packages and experimental v0.300.0 packages) and API 1.10.0. Migration guide For a detailed description of breaking changes, see the OpenTe</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/otel-js-sdk-3-0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 DevOps Log Management: A Comprehensive Evaluation Guide</h3><p>Discover how effective devops log management helps you quickly troubleshoot incidents, reduce noise, and improve operational visibility with the right data.</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/log/devops-log-management" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Bridging Alloy and OpenTelemetry: the OTel engine is now in Public Preview</h3><p>Many Grafana Cloud customers love what Alloy offers them, but we're also seeing a growing base of users who feel the same way about the OpenTelemetry Collector. We want to make it easy for customers t</p>
<p><strong>📅 Nov 3, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/bridging-alloy-and-opentelemetry-the-otel-engine-is-now-in-public-preview/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Trace linkage: connect the dots between your app traces and database queries</h3><p>When an application request slows down, a trace can show a database call. Finding the database query behind it can be tedious, requiring you to switch between traces and database telemetry, as well as</p>
<p><strong>📅 Nov 3, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/trace-linkage-connect-the-dots-between-your-app-traces-and-database-queries/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Finding performance wins in our own traces with a coding agent</h3><p>How we used a coding agent and the Sentry MCP to find backend performance wins in Sentry's own traces and remove hundreds of millions of queries a week.</p>
<p><strong>📅 Nov 3, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/performance-wins-traces-coding-agent/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔐 Security</h2><h3>📄 Red Hat OpenShift 5: What partners need to know</h3><p>Red Hat OpenShift 5 is the next evolution of the cloud-native application platform you and your customers already trust.For distributors and resellers, Red Hat OpenShift 5 is more than a product relea</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/red-hat-openshift-5-what-partners-need-know" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Kimi K3.1 Pro (Highspeed) now available on Amazon Bedrock</h3><p>Today, Kimi K3.1 Pro from Moonshot AI is available on Amazon Bedrock, expanding our open weight model portfolio with the security and governance that customers rely on. Customers using Amazon Bedrock </p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/11/moonshot-ai-kimi-k3-1-pro-on-amazon-bedrock" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The missing piece: bringing up to 15 years of support to Zephyr for microcontroller-based ECUs</h3><p>We recently announced the launch of Zephyr 26.04 LTS, bringing up to 15 years of security maintenance per LTS and enterprise lifecycle management to microcontroller-grade devices. While central high-p</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/the-missing-piece-bringing-up-to-15-years-of-support-to-zephyr-for-microcontroller-based-ecus" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 5-minute demo: How to manage applications at scale with Red Hat OpenShift</h3><p>Operating cloud-native applications at enterprise scale is rarely a single-cluster affair. As businesses continue to expand across public cloud providers, private datacenters, and edge locations, plat</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/5-minute-demo-how-manage-applications-scale-red-hat-openshift" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Well, That Wasn't Fun: October 16 '26</h3><p>October 16, 2026 began like any other day at Fly.io (meaning, we were anticipating only a medium amount of chaos). By that evening, we had patched and rebooted our entire fleet in response to a Linux </p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Fly.io Blog</strong></p>
<p><a href="https://fly.io/blog/well-that-wasnt-fun/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 October 2026 Node.js vulnerabilities: Azure DevOps security guidance</h3><p>On October 21, 2026, the Node.js project released fixes for 10 vulnerabilities. These include flaws in DNS resolution and TLS certificate validation, along with a Brotli decompression bug that can cor</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Azure DevOps Blog</strong></p>
<p><a href="https://devblogs.microsoft.com/devops/october-2026-node-js-vulnerabilities-azure-devops-security-guidance/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why Higher Ed Needs a New Playbook for Ransomware, AI Risk, and Compliance</h3><p>Discover how higher education institutions use Zero Trust microsegmentation and AI governance to contain ransomware and meet FERPA compliance.</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/2026/nov/higher-ed-needs-new-playbook-ransomware-ai-risk-compliance" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Ubuntu Pro is now available on AWS Deep Learning Amazon Machine Images (DLAMI)</h3><p>In September 2026, Amazon Web Services (AWS) published its first DLAMIs built on Ubuntu Pro in the Base GPU family. By choosing Ubuntu Pro, enterprises launching DLAMIs gain a stronger security mainte</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/ubuntu-pro-available-on-dlami" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 5-minute demo: How to build modern applications with Red Hat OpenShift</h3><p>Building and deploying enterprise applications is no longer just about launching containers—it’s about managing the entire software lifecycle with a focus on security, intelligence, speed, and predict</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/5-minute-demo-how-build-modern-applications-red-hat-openshift" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 CVE-2026-109518 APT vulnerability fixes available</h3><p>Fixes are available in Ubuntu for the CVE-2026-109518 vulnerability affecting APT.</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/apt-vulnerability-fixes-available" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Log Management vs SIEM: Understanding the Key Differences and Use Cases</h3><p>Discover the differences between log management vs SIEM and learn when each tool is right for your team to improve visibility and security.</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/log/log-management-vs-siem" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Snyk Named a Leader in The Forrester Wave™: Agentic Development Security Tools, Q4 2026</h3><p>Snyk is a Leader in The Forrester Wave™: Agentic Development Security Tools, Q4 2026, with the highest score in the strategy category.</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/snyk-named-leader-forrester-wave-agentic-development-security-tools-q4-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>💾 Databases</h2><h3>📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers</h3><p>Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/please-read-me-opportunistic-ransomware-devastating-mysql-servers" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AI Search for Agents: Announcing Automated Embedding in MongoDB Atlas</h3><p>Automated Embedding is now available in Public Preview on MongoDB Atlas. When we introduced Automated Embedding to MongoDB Vector Search in MongoDB Community Edition, we started from a single principl</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 MongoDB Blog</strong></p>
<p><a href="https://www.mongodb.com/company/blog/product-release-announcements/ai-search-for-agents-announcing-automated-embedding-atlas" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Dasha 1.10: Patroni auto-discovery, wait analysis and replication slot forecasts for PostgreSQL</h3><p>Dasha is an open source performance dashboard for PostgreSQL fleets. It connects to your clusters with a read-only role, shows what the databases are doing right now, and explains what to do about it.</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/dasha-110-patroni-auto-discovery-wait-analysis-and-replication-slot-forecasts-for-postgresql-3438/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AgensGraph 2.18 Released: Graph Database on PostgreSQL 18 with Expanded ISO GQL Support</h3><p>The AgensGraph Development Team is pleased to announce the release of AgensGraph v2.18.6.1, the latest version in the 2.18 series. AgensGraph is an open-source graph database, built on PostgreSQL 18.6</p>
<p><strong>📅 Nov 7, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/agensgraph-218-released-graph-database-on-postgresql-18-with-expanded-iso-gql-support-3437/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Brume: FK-safe pseudonymization for PostgreSQL test and debug copies</h3><p>Brume 2.1, an open source (Apache 2.0) tool that creates deterministic, foreign-key-safe pseudonymized copies of PostgreSQL databases for testing, debugging and demos, is now available. Why Brume? Dev</p>
<p><strong>📅 Nov 7, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/brume-fk-safe-pseudonymization-for-postgresql-test-and-debug-copies-3434/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 HeapDeck: an on-call monitor for PostgreSQL on iPhone and iPad</h3><p>HeapDeck is a new native app for engineers on call for PostgreSQL 13 to 19, on the App Store for iPhone and iPad. It opens on one health verdict with ten tiles. The Locks screen draws the tree from pg</p>
<p><strong>📅 Nov 7, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/heapdeck-an-on-call-monitor-for-postgresql-on-iphone-and-ipad-3436/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon Redshift expands SUPER data type to store strings up to 16 million bytes</h3><p>Amazon Redshift now supports individual strings up to 16 million bytes within SUPER data types. Once you opt in through a database parameter, this enhancement applies when ingesting structured or semi</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/11/amazon-redshift-expands-super-to-16-million-bytes/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Managed Postgres on Railway</h3><p>An overview of Postgres on Railway: maintenance, data browsing, query statistics, high availability, backups, recovery, pooling, and upgrades.</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Railway Blog</strong></p>
<p><a href="https://blog.railway.com/p/managed-postgres" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why Actionable Banking Is the Critical Stepping Stone to Full Autonomy</h3><p>Almost nothing is as important to someone today as their financial health. Most of us want to stay on top of what's happening with our bank accounts, like when a deposit clears so we can pay bills, if</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 MongoDB Blog</strong></p>
<p><a href="https://www.mongodb.com/company/blog/innovation/why-actionable-banking-is-critical-stepping-stone-full-autonomy" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Building an Agentic Warehouse Management System—Part 3: Tools, Decisions, and Actions</h3><p>If you haven’t read the first two parts yet, I recommend starting with Part 1: Where AI Agents Add Value and Part 2: Designing and Planning the Agent. There, we introduced the WMS scenario, discussed </p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 MongoDB Blog</strong></p>
<p><a href="https://www.mongodb.com/company/blog/technical/building-an-agentic-warehouse-management-system-part-3-tools-decisions-actions" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing the joint reference architecture for enterprise-grade Redis on VMware Cloud Foundation 9.1</h3><p>Joint reference architecture from VMware by Broadcom and Redis Moving beyond caching to enterprise-grade real-time data Teams building fraud detection, streaming analytics, AI agents, and feature stor</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Redis Blog</strong></p>
<p><a href="https://redis.io/blog/introducing-the-joint-reference-architecture-for-enterprise-grade-redis-on-vmware-cloud-foundation-9-1/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The next chapter of Redis Iris: what agents need to work in the real world</h3><p>The other day, I was chatting with a customer support AI agent dealing with a cancelled flight. The experience was terrible. You could tell the agent realistically only had a few functions, and this w</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 Redis Blog</strong></p>
<p><a href="https://redis.io/blog/the-next-chapter-of-redis-iris-what-agents-need-to-work-in-the-real-world/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🌐 Platforms</h2><h3>📄 The Oracle of Delphi Will Steal Your Credentials</h3><p>Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-oracle-of-delphi-steal-your-credentials" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger</h3><p>In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-nansh0u-campaign-hackers-arsenal-grows-stronger" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Atlas Data Federation and Online Archive Can Now Be Deployed in Azure</h3><p>Exciting developments are on the horizon for users of Microsoft Azure, marking a significant leap in data management capabilities. First off, Atlas Data Federation is now Generally Available on Azure.</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 MongoDB Blog</strong></p>
<p><a href="https://www.mongodb.com/company/blog/product-release-announcements/atlas-data-federation-online-archive-can-now-be-deployed-in-azure" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing TPU Sync: Fast KV cache and weight transfer for inference and RL on TPU</h3><p>As frontier AI evolves from stateless single-turn prompts to million-token agents, disaggregated serving, and continuous reinforcement learning (RL), the primary bottleneck in AI infrastructure has sh</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/ai-infrastructure/introducing-tpu-sync-fast-kv-cache-and-weight-transfer-on-tpu/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Our newest TPUs: 8tx and 8i Max, built for the agentic era</h3><p>Editor’s note: This blog was originally published on News from Google. We first designed TPUs more than a decade ago motivated by Google’s internal demands, including voice recognition and language tr</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/ai-infrastructure/tpu-8tx-and-tpu-8i-max/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Launching the Vite Ecosystem Fund</h3><p>Cloudflare is distributing the first $301,300 of the Vite Ecosystem Fund to open source projects and maintainers. The recipients were chosen by a panel of Vite stack core team members, including maint</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/launching-vite-ecosystem-fund/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Agent Registry is now generally available in AWS GovCloud (US-West) Region</h3><p>AWS Agent Registry, available through Amazon Bedrock AgentCore, is now available in the AWS GovCloud (US-West) Region. AWS Agent Registry is a fully managed discovery service that gives your organizat</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/11/aws-agent-registry-govcloud-us-west/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon MSK Replicator is now available in AWS GovCloud (US-East, US-West) Regions</h3><p>You can now use Amazon MSK Replicator in the AWS GovCloud (US-East, US-West) Regions to replicate streaming data across Amazon MSK clusters and from external Apache Kafka clusters including on-premise</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-msk-replicator-additional-aws-regions" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The post-quantum future of web privacy</h3><p>Cryptographic protocols like Privacy Pass help keep web sites safe from abuse while keeping their users private. This post looks at the quantum threat to these protocols, and some new post-quantum cry</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/vole-mole/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Kotlin Multiplatform Contest Is Open: Win a Trip to KotlinConf</h3><p>The Kotlin Multiplatform Contest is back! Students and recent graduates, turn your idea into a cross-platform project with Kotlin Multiplatform (KMP). The authors of the top three projects will each w</p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/kotlin/2026/11/kotlin-multiplatform-contest-2027/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Agents aren’t humans (but they kinda are)</h3><p>AI agents act for millions, but servers see intermediaries that strip user visibility. This post defines the problem space and introduces a collaborative approach to safely identifying, allowing, and </p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/trusted-agent-delegation/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing DigitalOcean Cuttlefish: Get more done with agents on the go</h3><p>Your agent has been refactoring for two hours. It’s deep in the codebase, making real progress, and then it stops: “Can I run this migration?” Nothing happens until you answer. And you’re not at your </p>
<p><strong>📅 Nov 5, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/introducing-cuttlefish" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📰 Misc</h2><h3>📄 Visual Studio Code 1.146 (Insiders)</h3><p>Learn what's new in Visual Studio Code 1.146 (Insiders) Read the full article</p>
<p><strong>📅 Nov 11, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_146" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why semantic layers will be essential for enterprise AI</h3><p>AI can find your data. It can’t read your mind. AI agents can already reach into your CRM, your warehouse, The post Why semantic layers will be essential for enterprise AI appeared first on The New St</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/semantic-layers-enterprise-ai/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Real-time state has a cost your memory bill won’t show</h3><p>Real-time applications often combine tools with distinct roles. A typical setup might use Kafka to handle events, Flink to process The post Real-time state has a cost your memory bill won’t show appea</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/duplicated-streaming-state-costs/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What’s new in Red Hat OpenShift 5</h3><p>Red Hat OpenShift 5 is now generally available. The Red Hat OpenShift team always tries to bring you the most requested features while, at the same time, pushing the envelope with new opportunities. Y</p>
<p><strong>📅 Nov 9, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/announcing-red-hat-openshift-5" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Moving from AI adoption to an AI operating model: How Docusign engineering went beyond tool rollout</h3><p>Docusign rolled out AI coding tools across its engineering teams, expecting them to help developers write code faster and do The post Moving from AI adoption to an AI operating model: How Docusign eng</p>
<p><strong>📅 Nov 8, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/docusign-ai-engineering-adoption/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Ubuntu Summit 26.10: Less than a week away!</h3><p>Ubuntu Summit 26.10: Less than a week away! Here in London, the long summer has stepped aside for cooler nights and damp, misty mornings. The trees are turning amber and the sunlight is low on the hor</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/ubuntu-summit-26-10-less-than-a-week-away" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 IntelliJ Scala Plugin 2026.3 Is Out!</h3><p>Hello everyone! Scala plugin 2026.3 is here! This release brings more accurate Scala 3 code analysis, substantially reworked Twirl support for Play Framework developers, and more efficient compiler-ba</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/scala/2026/11/06/intellij-scala-plugin-2026-3-is-out/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GoLand’s Profiler Now Works With AI Agents</h3><p>In our practical guide to profiling in Go, we mentioned that some of the Go developers we talked to avoid pprof if they can. For many of them, typical Go services perform well enough without optimizat</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/go/2026/11/06/we-gave-ai-agents-goland-s-profiler-they-found-the-same-bottlenecks-with-half-the-tokens/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Java Annotated Monthly – November 2026</h3><p>October kept the Java world busy, with JDK news, fresh tutorials, Kotlin updates, AI experiments, and plenty more from across the wider ecosystem. We’ve gathered the best of it in one place, so you ca</p>
<p><strong>📅 Nov 6, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/idea/2026/11/java-annotated-monthly-november-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Visual Studio Code 1.145</h3><p>Learn what's new in Visual Studio Code 1.145 Read the full article</p>
<p><strong>📅 Nov 4, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_145" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Designing a better future: your guide to design contribution in Open Source</h3><p>This guide covers why design matters in open source, and how designers can make a bigger impact in open source software.</p>
<p><strong>📅 Nov 3, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/designing-a-better-future-your-guide-to-design-contribution-in-open-source" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Governing Coding-Agent Sprawl with MLflow AI Gateway</h3><p>A demonstration of how to use MLflow AI Gateway to manage coding agent sprawl.</p>
<p><strong>📅 Nov 3, 2026</strong> • <strong>📰 MLflow Blog</strong></p>
<p><a href="https://mlflow.org/blog/govern-coding-agent-sprawl/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Build an Everyday AI Assistant with DigitalOcean Inference]]></title>
      <link>https://devops-daily.com/posts/build-personal-ai-assistant-digitalocean-inference</link>
      <description><![CDATA[Build Daylight, a personal assistant that turns a brain dump into a realistic plan, adapts when your day changes, and saves the changes you accept.]]></description>
      <pubDate>Fri, 06 Nov 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/build-personal-ai-assistant-digitalocean-inference</guid>
      <category><![CDATA[Cloud]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Cloud]]></category><category><![CDATA[DigitalOcean]]></category><category><![CDATA[AI Agents]]></category><category><![CDATA[Inference]]></category><category><![CDATA[Python]]></category><category><![CDATA[FastAPI]]></category>
      <content:encoded><![CDATA[<p>You finish work with groceries to buy, a kitchen to clean, and a workout you meant to do. A friend wants to meet, too. Writing those things into a to-do list takes a minute. Deciding what fits into an evening when you are already tired is the harder part.</p>
<p>That is the problem we are building for. <strong>Daylight</strong> is a small personal AI assistant that helps you make tradeoffs, proposes a manageable plan, and adapts when your circumstances change. Work is one part of the picture; dinner, hobbies, friends, chores, and rest belong in the same conversation.</p>
<p>You can run the app locally with Python and a DigitalOcean Inference key. After getting it working, we will look at how a model proposes structured changes without getting direct control of your saved data. The same request, validation, and persistence pattern is useful in plenty of internal tools.</p>
<h2>Prerequisites</h2><ul>
<li>Python 3.11 or newer, Git, and a terminal with <code>make</code>.</li>
<li>A DigitalOcean account with <a href="https://docs.digitalocean.com/products/inference/how-to/si-overview/" rel="noopener noreferrer">Serverless Inference billing configured</a>.</li>
<li>A model access key scoped to a text model with function calling support.</li>
<li>Node.js 22 or newer only if you want to run the optional browser-rendering tests.</li>
</ul>
<p>The example uses <code>mimo-v2.5-pro</code>. DigitalOcean lists Chat Completions, tool calling, and structured outputs for it in the <a href="https://docs.digitalocean.com/products/inference/details/models/" rel="noopener noreferrer">model catalog</a>. Availability and supported parameters vary by model, so check the catalog when selecting another one.</p>
<p>Each chat submission makes a billable inference request. Completing tasks, accepting plans, deleting tasks, and starting a new day are local application actions.</p>
<h2>Get Daylight running</h2><p>The complete app is in the companion repository:</p>
<p><a href="https://github.com/The-DevOps-Daily/digitalocean-inference-daylight" rel="noopener noreferrer">The-DevOps-Daily/digitalocean-inference-daylight on GitHub</a></p>
<p>Clone it, install the Python dependencies, and copy the configuration template:</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/digitalocean-inference-daylight.git
<span class="hljs-built_in">cd</span> digitalocean-inference-daylight
make install
<span class="hljs-built_in">cp</span> .env.example .<span class="hljs-built_in">env</span>
</code></pre><p>There is no frontend build or separate database service. FastAPI serves the browser interface and API; Python's built-in SQLite support handles persistence.</p>
<h3>Create a model access key</h3><p>Open DigitalOcean's <a href="https://cloud.digitalocean.com/model-studio/manage-keys" rel="noopener noreferrer">Model Access Keys page</a> and create a key for Daylight. Select the model you intend to use. For a request from your laptop, choose <strong>No VPC network</strong>; a VPC-restricted key requires requests from that network. Copy the secret when it is shown.</p>
<p>The <a href="https://docs.digitalocean.com/products/inference/how-to/manage-model-access-keys/" rel="noopener noreferrer">key-management documentation</a> explains model scoping and VPC restrictions. Use a key dedicated to this app rather than sharing credentials between experiments.</p>
<p>Set these values in <code>.env</code>:</p>
<pre><code class="hljs language-text" tabindex="0">DIGITALOCEAN_INFERENCE_KEY=your-model-access-key
DIGITALOCEAN_INFERENCE_MODEL=mimo-v2.5-pro
DAYLIGHT_DATABASE=data/daylight.sqlite3
DIGITALOCEAN_REASONING_EFFORT=none
INFERENCE_TIMEOUT_SECONDS=120
</code></pre><p><code>DIGITALOCEAN_REASONING_EFFORT</code> is optional. Remove it if your selected model does not support that setting. DigitalOcean documents reasoning controls in <a href="https://docs.digitalocean.com/products/inference/how-to/use-reasoning/" rel="noopener noreferrer">Using Reasoning with the Chat Completions and Responses API</a>.</p>
<p>Start the server:</p>
<pre><code class="hljs language-bash" tabindex="0">make run
</code></pre><p>Open <a href="http://127.0.0.1:8082" rel="noopener noreferrer">http://127.0.0.1:8082</a>. Keep the terminal running while you use the app. After changing <code>.env</code>, stop the server with <code>Ctrl+C</code> and run <code>make run</code> again.</p>
<p>The access key stays in the backend process. The browser sends requests to your local FastAPI server, which calls DigitalOcean's <a href="https://docs.digitalocean.com/products/inference/how-to/si-endpoints/" rel="noopener noreferrer">Chat Completions endpoint</a> at <code>https://inference.do-ai.run/v1/chat/completions</code>.</p>
<blockquote>
<p><strong>Note</strong></p>
<p>You control the application code and its local database. Model inference is hosted: chat submissions send recent conversation, your saved plan, preferences, pending proposals, and recently removed task IDs and titles to DigitalOcean and the selected model service. This is not an offline assistant.</p>
</blockquote>
<h2>Give it a day that needs a little rearranging</h2><p>Set the available time to <strong>90 minutes</strong> and energy to <strong>Running low</strong>. Then try:</p>
<blockquote>
<p>I need groceries for dinner, the kitchen needs cleaning, and I wanted to exercise. A friend also wants to meet tonight. Nothing has a hard deadline except eating dinner. I prefer relaxed evenings. Help me make a realistic plan.</p>
</blockquote>
<p>A useful response does more than repeat your list. It considers which activities can shrink, what depends on something else, and what can wait. It might suggest a focused grocery trip, a simple meal, a short kitchen reset, and lighter movement instead of a full workout. It should explain the tradeoffs so you can disagree with them.</p>
<p>In our live test, the first proposal allocated 85 minutes to groceries, dinner, cleanup, and a short walk or stretch. It deferred the full workout and meeting a friend. That is one possible answer, not the correct answer for everyone's evening. If seeing your friend matters more, tell the assistant that and revise the proposal.</p>
<p>The right-hand panel labels the result <strong>Proposed changes</strong>. Select <strong>This works for me</strong> to save it. Until then, the accepted plan remains unchanged.</p>
<p>Now mark groceries complete, change the time control to <strong>20 minutes</strong>, and send:</p>
<blockquote>
<p>The groceries are done. My evening changed and I only have 20 minutes now. Keep dinner as the priority. Suggest a simple meal and defer the cleanup and exercise if necessary. Preserve everything else, including the completed grocery task.</p>
</blockquote>
<p>In our test, Daylight reduced dinner to 20 minutes and moved the remaining activities to Later. Groceries stayed completed. Accepting the revision and reloading preserved that state.</p>
<p><img src="https://devops-daily.com/images/posts/build-personal-ai-assistant-digitalocean-inference/daylight-replanning.png" alt="Daylight showing a saved 20-minute dinner plan alongside the conversation explaining the tradeoffs" /></p>
<p>The screenshot recreates the 20-minute dinner scenario above. Your model's wording and task estimates will vary.</p>
<p>This is the contribution we want from the AI: help someone decide what is doable when time, energy, and priorities collide. The task list gives that reasoning somewhere to land.</p>
<h2>How the assistant proposes changes</h2><p>The implementation is small enough to follow without an agent framework:</p>
<pre><code class="hljs language-text" tabindex="0">app/
├── agent.py       # Prompt, inference request, response validation
├── config.py      # Inference timeout validation
├── main.py        # API routes and user actions
├── models.py      # Request and plan schemas
├── store.py       # SQLite state and revision checks
└── static/        # HTML, CSS, JavaScript, local Markdown renderer
</code></pre><p>Each chat submission includes the current date from the browser, available minutes, energy, saved plan, pending proposal, removed tasks, and the latest 16 conversation messages. Accepted preferences live inside the saved plan rather than depending on an ever-growing transcript.</p>
<ol>
<li><strong>Your situation</strong> message, time, energy</li>
<li><strong>FastAPI + context</strong> saved plan and history</li>
<li><strong>DO Inference</strong> question or tool arguments</li>
<li><strong>Validated proposal</strong> review the tradeoffs</li>
<li><strong>Accept and save</strong> SQLite transaction</li>
</ol>
<p>If an important detail is missing, the model can ask a clarifying question instead of proposing a plan. Otherwise, it can return a <strong>function tool call</strong> named <code>propose_plan</code>.</p>
<p>This is a constrained planning assistant, not a background worker that runs arbitrary tools. There is one model request per submitted message. The tool call is a structured proposal; the app does not dispatch model-generated function names to Python functions or let the model write to SQLite.</p>
<h3>Define what a plan contains</h3><p>In <code>app/models.py</code>, a task has a stable ID, title, time estimate, group, and reason:</p>
<pre><code class="hljs language-python" tabindex="0"><span class="hljs-keyword">class</span> <span class="hljs-title class_">Task</span>(<span class="hljs-title class_ inherited__">StrictModel</span>):
    <span class="hljs-built_in">id</span>: <span class="hljs-built_in">str</span> = Field(pattern=<span class="hljs-string">r"^[A-Za-z0-9_-]{1,80}$"</span>)
    title: <span class="hljs-built_in">str</span> = Field(min_length=<span class="hljs-number">1</span>, max_length=<span class="hljs-number">160</span>)
    minutes: <span class="hljs-built_in">int</span> = Field(ge=<span class="hljs-number">1</span>, le=<span class="hljs-number">1440</span>)
    when: <span class="hljs-type">Literal</span>[<span class="hljs-string">"today"</span>, <span class="hljs-string">"later"</span>, <span class="hljs-string">"done"</span>]
    reason: <span class="hljs-built_in">str</span> = Field(min_length=<span class="hljs-number">1</span>, max_length=<span class="hljs-number">500</span>)
</code></pre><p>The full plan also contains a headline, explanation, task array, and preferences:</p>
<pre><code class="hljs language-python" tabindex="0"><span class="hljs-keyword">class</span> <span class="hljs-title class_">Plan</span>(<span class="hljs-title class_ inherited__">StrictModel</span>):
    headline: <span class="hljs-built_in">str</span> = Field(min_length=<span class="hljs-number">1</span>, max_length=<span class="hljs-number">180</span>)
    explanation: <span class="hljs-built_in">str</span> = Field(min_length=<span class="hljs-number">1</span>, max_length=<span class="hljs-number">1500</span>)
    tasks: <span class="hljs-built_in">list</span>[Task] = Field(max_length=<span class="hljs-number">40</span>)
    memory: <span class="hljs-built_in">str</span> = Field(max_length=<span class="hljs-number">2000</span>)
</code></pre><p><code>StrictModel</code> rejects unknown fields and strips surrounding whitespace. The <code>memory</code> field is a visible note of preferences you have explicitly shared, such as preferring relaxed evenings. The prompt tells the model to preserve those preferences unless corrected and to avoid inventing sensitive traits. You review proposed preferences with the plan before acceptance.</p>
<h3>Put that schema in the inference request</h3><p>The request in <code>app/agent.py</code> uses the Pydantic schema as the function's parameter definition. This excerpt shows the relevant fields; the full implementation also handles configuration and provider errors:</p>
<pre><code class="hljs language-python" tabindex="0">payload = {
    <span class="hljs-string">"model"</span>: os.getenv(<span class="hljs-string">"DIGITALOCEAN_INFERENCE_MODEL"</span>, <span class="hljs-string">"mimo-v2.5-pro"</span>),
    <span class="hljs-string">"messages"</span>: messages,
    <span class="hljs-string">"max_tokens"</span>: <span class="hljs-number">3000</span>,
    <span class="hljs-string">"stream"</span>: <span class="hljs-literal">False</span>,
    <span class="hljs-string">"tools"</span>: [{
        <span class="hljs-string">"type"</span>: <span class="hljs-string">"function"</span>,
        <span class="hljs-string">"function"</span>: {
            <span class="hljs-string">"name"</span>: <span class="hljs-string">"propose_plan"</span>,
            <span class="hljs-string">"description"</span>: <span class="hljs-string">"Propose a complete realistic plan for user review; does not save it."</span>,
            <span class="hljs-string">"parameters"</span>: Plan.model_json_schema(),
        },
    }],
    <span class="hljs-string">"tool_choice"</span>: <span class="hljs-string">"auto"</span>,
}
</code></pre><p>The model sees the request's context and decides whether to ask a question or supply tool arguments. It does not see an API key in the message content; <code>httpx</code> adds authentication as a server-side authorization header.</p>
<h3>Validate the proposal before showing an accept button</h3><p>A function schema tells the model what to produce. The server still needs to check what it actually returned:</p>
<pre><code class="hljs language-python" tabindex="0">proposal = Plan.model_validate_json(
    calls[<span class="hljs-number">0</span>][<span class="hljs-string">"function"</span>][<span class="hljs-string">"arguments"</span>]
).model_dump()

ids = [task[<span class="hljs-string">"id"</span>] <span class="hljs-keyword">for</span> task <span class="hljs-keyword">in</span> proposal[<span class="hljs-string">"tasks"</span>]]
<span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(ids) != <span class="hljs-built_in">len</span>(<span class="hljs-built_in">set</span>(ids)):
    <span class="hljs-keyword">raise</span> ValueError(<span class="hljs-string">"Duplicate task IDs"</span>)

planned_minutes = <span class="hljs-built_in">sum</span>(
    task[<span class="hljs-string">"minutes"</span>]
    <span class="hljs-keyword">for</span> task <span class="hljs-keyword">in</span> proposal[<span class="hljs-string">"tasks"</span>]
    <span class="hljs-keyword">if</span> task[<span class="hljs-string">"when"</span>] == <span class="hljs-string">"today"</span>
)
<span class="hljs-keyword">if</span> planned_minutes &gt; request.minutes:
    <span class="hljs-keyword">raise</span> ValueError(<span class="hljs-string">"Plan exceeds available time"</span>)
</code></pre><p>The code also checks that saved commitments are not silently omitted, completed tasks remain completed, and deleted task IDs do not reappear. If validation fails, it rejects the response without replacing the saved plan.</p>
<p>Accepting arbitrary JSON because the HTTP request returned 200 would skip all of those checks. Passing the arguments through Pydantic is the first step; checking the application's own rules is the second.</p>
<p>These checks do not prove that grocery shopping really takes 25 minutes, that a suggested dinner suits you, or that two tasks represent the same commitment under different IDs. You still review the proposal. The prompt also tells the assistant not to claim it checked a calendar, contacted your friend, or scheduled a reminder: none of those integrations exists in this app.</p>
<h2>Keep control of the plan between conversations</h2><p>The accepted plan stays in <code>data/daylight.sqlite3</code> across page reloads, browser sessions, and server restarts. It has no expiry date and does not reset at midnight.</p>
<p>The usual task actions are ordinary application behavior:</p>
<table>
<thead>
<tr>
<th>Action</th>
<th>What happens</th>
</tr>
</thead>
<tbody><tr>
<td>Complete a task</td>
<td>Moves it to Done and invalidates an older pending proposal.</td>
</tr>
<tr>
<td>Reopen a task</td>
<td>Restores its previous Today or Later group.</td>
</tr>
<tr>
<td>Delete a task</td>
<td>Removes it immediately; the previous plan is available through Undo task deletion until another plan change replaces that snapshot.</td>
</tr>
<tr>
<td>Accept a proposal</td>
<td>Saves the reviewed replacement plan and keeps one previous plan for undo.</td>
</tr>
<tr>
<td>Dismiss a proposal</td>
<td>Keeps the accepted plan.</td>
</tr>
</tbody></table>
<p>A trash button appears beside each saved task's time estimate. Proposed tasks must be accepted before those controls become active.</p>
<p>Deleting a task also adds its ID and title to bounded removed-task context, so the assistant is told not to recreate it from an old conversation. Reusing the deleted ID is rejected. This is not semantic duplicate detection: a model could still invent a different ID for the same activity, so inspect the next proposal. You can explicitly ask to add an activity again with a new ID.</p>
<h3>Clear the conversation or start a new day</h3><p>Both actions are in the <strong>Chat options</strong> menu at the top-right of the conversation panel:</p>
<ul>
<li><strong>Clear conversation</strong> removes the chat and pending proposal while keeping the accepted plan, task progress, and preferences.</li>
<li><strong>Start a new day</strong> clears the chat, removes completed tasks from the current plan, and carries unfinished tasks into Later for review. Preferences remain saved.</li>
</ul>
<p>Both clear the old undo snapshot and ask for confirmation. Starting a new day prepares an editable message; it does not submit an inference request until you press Send. Check your time and energy controls before sending it.</p>
<p>Saved chat is bounded to 60 messages. Inference reads only the latest 16, together with the explicit plan context. That makes the accepted plan the durable record of commitments, rather than hoping that a model can recover them from an unlimited chat log.</p>
<p>SQLite also protects against a less visible problem: stale updates. Every state change increments a revision. A transaction reads the current revision under <code>BEGIN IMMEDIATE</code>, compares it with the request's revision, applies the change, and commits. If a task changes or you clear the conversation while an inference request is running, the old response cannot overwrite the newer state. The API returns a conflict instead.</p>
<h2>Test the behavior, then make it your own</h2><p>Run the backend checks and optional Markdown checks:</p>
<pre><code class="hljs language-bash" tabindex="0">make check
node --<span class="hljs-built_in">test</span> tests/markdown.test.cjs
</code></pre><p>The reviewed version has 39 backend tests and four Markdown tests. They cover plan validation, saved state, completion, deletion and undo, new-day behavior, stale updates, provider failures, host validation, and rendering untrusted model text. They use mocked inference responses and do not incur model charges. The repository also includes GitHub Actions checks for Python 3.11 and 3.14.</p>
<p>Live testing answers a different question. Try your own mix of work and personal activities, accept a plan, reduce the time available, and check that the new proposal both fits the budget and preserves unfinished commitments. Reload after completing an item to verify persistence. A passing schema test alone cannot tell you whether the advice is useful.</p>
<h3>Expect a few operational details</h3><p>We observed planning requests near a minute with MiMo V2.5 Pro, including one diagnostic at 58.7 seconds. Earlier requests exceeded the original 60-second timeout. That is a development observation, not a benchmark or a latency promise. The app now defaults to a configurable 120-second response timeout and a separate 10-second connection timeout. On timeout, it keeps the saved plan and leaves your message in the composer.</p>
<p>If a request fails, identify which layer failed:</p>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>First thing to check</th>
</tr>
</thead>
<tbody><tr>
<td>Local server offline / Failed to fetch</td>
<td>Is <code>make run</code> still running? Start it, then select Reconnect.</td>
</tr>
<tr>
<td>HTTP 401 from DigitalOcean</td>
<td>The key may be invalid or revoked. Replace it in <code>.env</code> and restart.</td>
</tr>
<tr>
<td>HTTP 403</td>
<td>Check model access and any VPC restriction on the key.</td>
</tr>
<tr>
<td>HTTP 429</td>
<td>Wait before retrying; check the account's rate limits.</td>
</tr>
<tr>
<td>Invalid plan</td>
<td>The model returned malformed data or broke an application rule. Clarify the request and retry.</td>
</tr>
<tr>
<td>State conflict</td>
<td>Another action changed the plan while the request was running. Review the latest state before resubmitting.</td>
</tr>
</tbody></table>
<p>Reconnect reloads local state; it does not resend a potentially billable request. Assistant Markdown is rendered with a locally bundled parser, with raw HTML and remote images disabled. User messages remain plain text, and provider error bodies are not echoed into the interface.</p>
<p>Keep this first version on localhost. It has no authentication or separation between users. A shared deployment needs those features, a deliberate host policy, and durable database storage. Notifications, calendars, and unattended actions are separate projects, not hidden capabilities of the model.</p>
<p>A useful next experiment is a small set of scenarios you can rerun when changing the prompt or model: low energy, an urgent errand, a missed day, a completed task, and a new activity you genuinely want to make room for. Judge whether the assistant makes sensible tradeoffs as well as whether its JSON passes validation. If you want a fuller evaluation workflow, our <a href="https://devops-daily.com/posts/build-evaluate-ai-error-explainer-digitalocean-inference">DigitalOcean Inference error-explainer guide</a> walks through one.</p>
<p>You now have a local app whose AI helps turn competing intentions into a plan you can inspect and change. Try it with a real evening, tell it when its priorities are wrong, and see whether the revised day feels more doable.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[cgroup v2 memory.high Throttles, memory.max Kills: Measured on a Host Without Swap]]></title>
      <link>https://devops-daily.com/posts/cgroup-v2-memory-high-vs-memory-max-measured</link>
      <description><![CDATA[We ran one allocating workload under memory.max, memory.high and both on a Linux 6.8 host with no swap. memory.max killed it in under half a second. memory.high did not kill it: new allocations waited, once per 64-page batch at best and once per page next to a busy neighbour on the same CPU. Here are the numbers, the kernel formula behind them, and what systemd, Docker and Kubernetes write.]]></description>
      <pubDate>Fri, 06 Nov 2026 07:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/cgroup-v2-memory-high-vs-memory-max-measured</guid>
      <category><![CDATA[Linux]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Linux]]></category><category><![CDATA[cgroups]]></category><category><![CDATA[Memory]]></category><category><![CDATA[systemd]]></category><category><![CDATA[Docker]]></category><category><![CDATA[Kubernetes]]></category><category><![CDATA[PSI]]></category><category><![CDATA[OOM]]></category>
      <content:encoded><![CDATA[<p>We ran one Python workload that allocates memory in 256 KiB steps, on a Linux 6.8 host with no swap, under cgroup v2 <code>memory.max</code>, <code>memory.high</code>, both and neither. When it asked for 320 MiB with only <code>memory.max</code> at 256 MiB, it ran at full speed and the kernel killed it 0.28 to 0.39 seconds after it started. With only <code>memory.high</code> at 230.4 MiB, the kernel did not kill it. It made new allocations wait instead, and after 150 seconds the workload held 11 to 12 MiB more than <code>memory.high</code>.</p>
<p>The case to worry about is the smaller one. A version of the workload that needs 239 MiB fits under <code>memory.max</code> and runs at full speed there. With <code>memory.high</code> at 90% of that limit, which is what Kubernetes MemoryQoS writes for a 256Mi limit and a zero memory request with the 1.36 default factor, its median allocation took 191.8 ms on a CPU of its own and 6.2 seconds on a CPU it shared with our 100 ms sampler, against 0.26 ms with no limit. It reported no error.</p>
<p>The kernel's throttle formula predicted the length of each sleep to within about 10%. The number of sleeps is what changed: once per 64-page batch, as the code suggests, or once per 4 KiB page. We found in review that our own sampler, on the same CPU, kept the task at once per page. Without it, the count still switched to once per page in phases, while other work ran on the host. And one condition changed both limits: when the program's own code was charged to its cgroup, the kernel reclaimed that code and read it back from disk over and over, and <code>memory.max</code> did not kill it within the 20 seconds we allowed.</p>
<p>This post has the numbers, the kernel formula, the method problems we found on the way, and a check of what systemd, Docker and Kubernetes write to these files.</p>
<h2>TL;DR</h2><ul>
<li><strong><code>memory.max</code> alone killed at full speed.</strong> The 320 MiB workload ran at normal latency (p99 of the last 10% of operations 0.29 to 0.41 ms) and the kernel OOM-killed it 0.28 to 0.39 seconds after it started, in 3 of 3 runs.</li>
<li><strong><code>memory.high</code> alone did not kill within the 150 second cap. It made new allocations wait.</strong> Each sleep was 1.06 to 1.10 times the kernel's formula at the median, <code>64000 x o^2</code> ms with <code>o</code> the share over <code>memory.high</code>. At the 2 s cap the median sleep was 2,047.9 ms, which fits the timer wheel's 64 ms step for a 2,000 ms timeout.</li>
<li><strong>The number of sleeps decides the cost, and a neighbour on the same CPU decides the number.</strong> The kernel's comment says one sleep per 64-page batch. With our sampler on the same CPU, the task slept once per 4 KiB page in 95% of the operations whose sleep was 50 ms or more. Without that neighbour (the workload on another CPU, or the sampler once a second), it switched between about 1 and 64 sleeps per batch in phases of tens of seconds, and spent 46% to 72% of that time in the slow phases.</li>
<li><strong>A program that fits under <code>memory.max</code> but not under <code>memory.high</code> ran much slower and reported nothing.</strong> With <code>memory.high</code> at 90% of a 256 MiB limit (the kubelet's value for a zero memory request and factor 0.9), the median allocation took 191.8 ms on its own CPU and 6,216 ms next to the busy sampler (15 to 17 of its 200 churn operations in 150 s), against 0.257 ms with no limit.</li>
<li><strong>Both limits only delayed the kill.</strong> With <code>memory.high</code> at 90% of <code>memory.max</code>, a leak was OOM-killed after 29.1 minutes next to the busy sampler. With <code>memory.max</code> alone the kill took 0.3 seconds. On a CPU of its own, it was not killed within the 120 second cap in 3 of 3 runs.</li>
<li><strong>Code pages in the cgroup turn both limits into thrashing.</strong> When the program's own binary was charged to its cgroup, the kernel reclaimed it and read it back from disk, 2.4 to 2.6 GB in 25 seconds at our 100 MiB/s disk cap. <code>memory.max</code> did not kill it within 20 seconds in 3 of 3 runs, and memory PSI showed only 8% while io PSI showed 85%.</li>
<li><strong>Page cache is not throttled or killed.</strong> Neither limit killed the file-reading runs or put them to sleep. They capped the cache, so later reads went to disk more often (p99 1.0 to 2.6 ms against 0.13 ms with no limit).</li>
<li><strong>The tools:</strong> systemd writes <code>MemoryHigh=</code> and <code>MemoryMax=</code> to <code>memory.high</code> and <code>memory.max</code> as given. Docker has no flag for <code>memory.high</code>, and on cgroup v2 <code>--memory-reservation</code> becomes <code>memory.low</code>. Kubernetes MemoryQoS writes <code>memory.high</code>; it is alpha and off in 1.36, and beta and on in 1.37 but with no throttling factor set, so no <code>memory.high</code> unless you set one.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Linux machine with cgroup v2 (<code>stat -fc %T /sys/fs/cgroup</code> prints <code>cgroup2fs</code>) and systemd</li>
<li>Root, or a cgroup subtree delegated to your user</li>
<li>Python 3 for the workload and the analysis scripts</li>
<li>Check for swap first: <code>swapon --show</code>. Our results are for a host with no swap. With swap, the kernel can move anonymous memory out, and both limits behave differently</li>
</ul>
<h2>Two limits, two code paths</h2><p>Both files live in a cgroup v2 directory. Both act when the kernel charges a new page to the cgroup. They do different things when the cgroup is over its value.</p>
<ul>
<li><strong><code>memory.max</code></strong> is the hard limit. The kernel docs for 6.8 say: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup." The charge path tries to reclaim first, retries, and calls the OOM killer only when reclaim makes no progress.</li>
<li><strong><code>memory.high</code></strong> is the throttle. The same docs say: "Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached." The charge always succeeds. On the way back to user space, <code>mem_cgroup_handle_over_high()</code> tries to reclaim, and if usage is still over the value it puts the task to sleep.</li>
</ul>
<p>The sleep length comes from <code>calculate_high_delay()</code> in <code>mm/memcontrol.c</code>. On the Ubuntu 6.8.0-147 kernel we ran, with <code>HZ=1000</code>, it works out to:</p>
<pre><code class="hljs language-text" tabindex="0">o       = (usage - memory.high) / memory.high
penalty = 64000 * o^2 ms per charge batch of 64 pages (MEMCG_CHARGE_BATCH)
          no sleep if penalty &lt;= 10 ms; at most 2000 ms (MEMCG_MAX_HIGH_DELAY_JIFFIES = 2 * HZ)
</code></pre><p>So the sleep is 0 until usage is 1.25% over <code>memory.high</code>, 160 ms at 5% over, 640 ms at 10% over, and it hits the 2 s cap at 17.7% over. The comment above the cap reads "Clamp the maximum sleep time per allocation batch to 2 seconds." Each <code>reclaim_high()</code> call adds 1 to the <code>high</code> counter in <code>memory.events</code>. When reclaim frees nothing, the kernel calls it 17 times before it sleeps (once, then <code>MAX_RECLAIM_RETRIES</code> = 16 retries). When reclaim frees something, it keeps retrying and does not sleep. So in a cgroup with nothing to reclaim, the counter rises by 17 per sleep, and that is how we count sleeps from outside the kernel.</p>
<ol>
<li><strong>page fault</strong> charge to the cgroup</li>
<li><strong>over memory.max</strong> before the charge</li>
<li><strong>over memory.high</strong> after the charge</li>
<li><strong>reclaim</strong> direct, in the fault</li>
<li><strong>reclaim_high x17</strong> +17 to memory.events high</li>
<li><strong>OOM kill</strong> nothing reclaimable</li>
<li><strong>sleep</strong> 64000 x o^2 ms, at most 2 s</li>
<li><strong>thrash</strong> reclaim frees the task's own code</li>
</ol>
<p>Connections:</p>
<ul>
<li>page fault -&gt; over memory.max</li>
<li>page fault -&gt; over memory.high</li>
<li>over memory.max -&gt; reclaim</li>
<li>over memory.high -&gt; reclaim_high x17</li>
<li>reclaim -&gt; OOM kill (freed nothing)</li>
<li>reclaim -&gt; thrash (freed file pages)</li>
<li>reclaim_high x17 -&gt; sleep (freed nothing)</li>
<li>reclaim_high x17 -&gt; thrash (freed file pages)</li>
</ul>
<p>Without swap, the kernel has nowhere to put anonymous memory: heaps, stacks, anything a process allocated and wrote. In our runs that memory was never reclaimed (<code>pgsteal</code> stayed 0 in every warm anonymous run). What reclaim could take were file pages: page cache and the mapped code of the program in the cgroup. That decides most of what follows.</p>
<h2>The lab</h2><p>One host: Ubuntu 24.04, kernel 6.8.0-147-generic, systemd 255, 4 vCPU, 7.75 GiB of RAM, no swap device and zswap off, <code>CONFIG_HZ=1000</code>, transparent huge pages on <code>madvise</code> (<code>results/00-host.txt</code>). It is a shared development machine. So every run happened inside one transient systemd scope with <code>Delegate=yes</code> and <code>MemoryMax=1G</code>, every run got its own child cgroup, and no limit went above 1 GiB.</p>
<p>The workload is one script, <code>scripts/lib/workload.py</code>. Each operation maps a 256 KiB private anonymous chunk and writes all of it. That is 64 pages, which is also the kernel's charge batch. Each operation also copies one random chunk it already holds, so the working set stays in use. The phases:</p>
<ol>
<li><strong>Grow</strong> until the workload holds its target.</li>
<li><strong>Churn</strong> 200 times: free one random chunk and allocate a new one, so usage stays flat while the program keeps allocating, like a busy service.</li>
</ol>
<p>The random choices are seeded (mulberry32). Before and after every operation, the workload reads <code>memory.current</code> and the <code>high</code> and <code>max</code> counters in <code>memory.events</code> from open file descriptors, and writes one JSON line with the latency. A driver outside the cgroup samples <code>memory.current</code>, <code>memory.events</code>, <code>memory.pressure</code>, <code>io.pressure</code>, <code>memory.stat</code>, <code>cpu.stat</code> and each process's <code>/proc/&lt;pid&gt;/wchan</code> and <code>/proc/&lt;pid&gt;/stat</code> every 100 ms. The driver pinned itself to CPU 0, and the workload inherited that, so in the main matrix the two shared one CPU. We only saw that in review, and it changed one of the results a lot. The section on how many sleeps a task pays has the runs that measure it.</p>
<p>The two limits are fixed for every run:</p>
<ul>
<li><code>memory.max</code> = 268435456 bytes (256 MiB)</li>
<li><code>memory.high</code> = 241590272 bytes (230.4 MiB). That is the value the kubelet computes with MemoryQoS for <code>limits.memory: 256Mi</code>, <code>requests.memory: 0</code> and the 1.36 default throttling factor of 0.9. It is the lowest <code>memory.high</code> that formula gives for a 256Mi limit. More on that below.</li>
</ul>
<table>
<thead>
<tr>
<th>Run</th>
<th>Workload</th>
<th>What it shows</th>
</tr>
</thead>
<tbody><tr>
<td><code>anon-fit-*</code></td>
<td>grows to 232 MiB (about 239 MiB in <code>memory.current</code>), then churns</td>
<td>a program that fits under <code>memory.max</code> but not under <code>memory.high</code></td>
</tr>
<tr>
<td><code>anon-over-*</code></td>
<td>asks for 320 MiB</td>
<td>a program that needs more than the limit, like a leak</td>
</tr>
<tr>
<td><code>anon-over-4p-high</code></td>
<td>the same 320 MiB split over 4 processes</td>
<td>whether the throttle is per process</td>
</tr>
<tr>
<td><code>anon-fit-high-populate</code></td>
<td><code>mmap</code> with <code>MAP_POPULATE</code></td>
<td>the same bytes, charged in one system call per chunk</td>
</tr>
<tr>
<td><code>anon-clamp-high</code></td>
<td>grows to 300 MiB, then sets its own <code>memory.high</code> so its usage is 33% over it</td>
<td>the 2 s cap</td>
</tr>
<tr>
<td><code>*-owncode</code></td>
<td>a fresh copy of the interpreter binary</td>
<td>a program whose code pages are charged to its own cgroup</td>
</tr>
<tr>
<td><code>file-over-*</code></td>
<td>reads 320 MiB of a 512 MiB file</td>
<td>page cache instead of anonymous memory</td>
</tr>
<tr>
<td><code>anon-over-both-long</code></td>
<td><code>anon-over</code> with both limits, 45 minute cap</td>
<td>how long until the OOM kill</td>
</tr>
</tbody></table>
<p>Each of <code>none</code>, <code>max</code>, <code>high</code> and <code>both</code> ran 3 times, in a fixed order per repetition, with a 150 second cap. The <code>owncode</code> runs thrashed the disk in the smoke tests, so we capped them at 20 seconds. We also capped every run's disk reads and writes at 100 MiB/s with <code>io.max</code> on its own cgroup, so that a thrashing run would not take the shared disk from other work. One run, <code>anon-over-max-owncode-fastdisk</code>, keeps the uncapped disk.</p>
<p><a href="https://github.com/The-DevOps-Daily/cgroup-memory-high-vs-max" rel="noopener noreferrer">The-DevOps-Daily/cgroup-memory-high-vs-max on GitHub</a></p>
<p>We wrote the claim and what would disprove it before the first run and committed it first (<code>CLAIM.md</code>). The smoke runs then changed the design twice, and we added those changes to the same file before the main matrix. Every run directory carries a hash of the run code. The summary script refuses a batch with mixed hashes, or a hash that matches no commit in the repository.</p>
<h2>memory.max: full speed, then the kill</h2><p>With only <code>memory.max</code>, the workload never slowed down. In the three <code>anon-over-max</code> runs the median operation took 0.20 to 0.27 ms, the p99 of the last 10% of operations before the kill was 0.29 to 0.41 ms, and the kernel killed the process 0.28 to 0.39 seconds after it started (<code>results/summary.md</code>). In each run <code>memory.events</code> counted 35 times that a charge hit the limit, then one OOM kill. The kernel log names the cgroup:</p>
<pre><code class="hljs language-text" tabindex="0">oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ddmemhigh-20261105T112606Z.scope,mems_allowed=0,oom_memcg=/ddmemhigh.slice/ddmemhigh-20261105T112606Z.scope/anon-over-max-r1,task_memcg=/ddmemhigh.slice/ddmemhigh-20261105T112606Z.scope/anon-over-max-r1,task=python3.12,pid=142949,uid=0
Memory cgroup out of memory: Killed process 142949 (python3.12) total-vm:276772kB, anon-rss:261504kB, file-rss:7296kB, shmem-rss:0kB, UID:0 pgtables:580kB oom_score_adj:0
</code></pre><p>Nothing in the cgroup could be reclaimed. The cgroup held only the program's anonymous pages, and without swap the kernel cannot free those. So reclaim found nothing, the charge path ran out of retries, and the OOM killer ran. Memory pressure stayed at 0: the task never waited for memory. It was killed.</p>
<p>The <code>anon-fit-max</code> runs are the other half of the same result. That workload needs 239.2 MiB, which fits under 256 MiB. It finished all 200 churn operations with a median of 0.277 ms per churn operation and a <code>max</code> counter of 0. With no limit, the same median was 0.257 ms. If a program fits under <code>memory.max</code>, the limit costs nothing.</p>
<h2>memory.high: slower and slower, not killed</h2><p>With only <code>memory.high</code>, the same 320 MiB workload was not killed in any of the 3 runs within the 150 second cap. The kernel docs say <code>memory.high</code> never invokes the OOM killer, and our driver stopped each run at the cap. Once it was a few MiB over the limit, it moved at the speed of the kernel's sleep timer, not at the speed of a program:</p>
<p><strong>Latency of one 256 KiB allocation vs memory.current</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>226</th>
<th>227</th>
<th>228</th>
<th>229</th>
<th>230</th>
<th>231</th>
<th>232</th>
<th>233</th>
<th>234</th>
<th>235</th>
<th>236</th>
<th>237</th>
<th>238</th>
<th>239</th>
<th>240</th>
<th>241</th>
<th>242</th>
<th>243</th>
<th>244</th>
<th>245</th>
<th>246</th>
<th>247</th>
<th>248</th>
<th>249</th>
<th>250</th>
<th>251</th>
<th>252</th>
<th>253</th>
<th>254</th>
<th>255</th>
<th>256</th>
</tr>
</thead>
<tbody><tr>
<td>no limit</td>
<td>0.248ms</td>
<td>0.252ms</td>
<td>0.249ms</td>
<td>0.251ms</td>
<td>0.246ms</td>
<td>0.253ms</td>
<td>0.242ms</td>
<td>0.26ms</td>
<td>0.24ms</td>
<td>0.252ms</td>
<td>0.25ms</td>
<td>0.256ms</td>
<td>0.243ms</td>
<td>0.24ms</td>
<td>0.213ms</td>
<td>0.226ms</td>
<td>0.238ms</td>
<td>0.242ms</td>
<td>0.235ms</td>
<td>0.215ms</td>
<td>0.223ms</td>
<td>0.224ms</td>
<td>0.225ms</td>
<td>0.214ms</td>
<td>0.253ms</td>
<td>0.234ms</td>
<td>0.235ms</td>
<td>0.215ms</td>
<td>0.239ms</td>
<td>0.218ms</td>
<td>0.222ms</td>
</tr>
<tr>
<td>memory.max only</td>
<td>0.236ms</td>
<td>0.224ms</td>
<td>0.242ms</td>
<td>0.195ms</td>
<td>0.242ms</td>
<td>0.223ms</td>
<td>0.238ms</td>
<td>0.221ms</td>
<td>0.211ms</td>
<td>0.216ms</td>
<td>0.224ms</td>
<td>0.203ms</td>
<td>0.215ms</td>
<td>0.199ms</td>
<td>0.207ms</td>
<td>0.19ms</td>
<td>0.202ms</td>
<td>0.204ms</td>
<td>0.246ms</td>
<td>0.199ms</td>
<td>0.233ms</td>
<td>0.232ms</td>
<td>0.191ms</td>
<td>0.209ms</td>
<td>0.229ms</td>
<td>0.243ms</td>
<td>0.216ms</td>
<td>0.225ms</td>
<td>0.233ms</td>
<td>0.238ms</td>
<td></td>
</tr>
<tr>
<td>memory.high only, sampler on the same CPU</td>
<td>0.226ms</td>
<td>0.216ms</td>
<td>0.2ms</td>
<td>0.206ms</td>
<td>0.235ms</td>
<td>0.228ms</td>
<td>0.22ms</td>
<td>25.621ms</td>
<td>39.782ms</td>
<td>209.811ms</td>
<td>1525.021ms</td>
<td>4583.747ms</td>
<td>5839.866ms</td>
<td>7303.861ms</td>
<td>8839.941ms</td>
<td>10295.852ms</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>memory.high only, own CPU, sampler at 1 s</td>
<td>0.22ms</td>
<td>0.259ms</td>
<td>0.258ms</td>
<td>0.272ms</td>
<td>0.248ms</td>
<td>0.27ms</td>
<td>0.262ms</td>
<td>14.873ms</td>
<td>106.819ms</td>
<td>519.352ms</td>
<td>52.8ms</td>
<td>71.752ms</td>
<td>95.774ms</td>
<td>119.723ms</td>
<td>135.811ms</td>
<td>171.815ms</td>
<td>191.793ms</td>
<td>323.75ms</td>
<td>383.851ms</td>
<td>851.785ms</td>
<td>11251.414ms</td>
<td>379.811ms</td>
<td>1847.768ms</td>
<td>1759.826ms</td>
<td>1135.82ms</td>
<td>1727.849ms</td>
<td>3199.782ms</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>kernel formula, 1 sleep per 64 pages</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>11.243ms</td>
<td>20.243ms</td>
<td>31.243ms</td>
<td>44.243ms</td>
<td>60.243ms</td>
<td>79.243ms</td>
<td>99.243ms</td>
<td>123.243ms</td>
<td>148.243ms</td>
<td>176.243ms</td>
<td>206.243ms</td>
<td>239.243ms</td>
<td>274.243ms</td>
<td>312.243ms</td>
<td>352.243ms</td>
<td>395.243ms</td>
<td>439.243ms</td>
<td>487.243ms</td>
<td>536.243ms</td>
<td>588.243ms</td>
<td>643.243ms</td>
<td>700.243ms</td>
<td>759.243ms</td>
<td>821.243ms</td>
</tr>
<tr>
<td>kernel formula, 1 sleep per page</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>0.243ms</td>
<td>704.243ms</td>
<td>1280.243ms</td>
<td>1984.243ms</td>
<td>2816.243ms</td>
<td>3840.243ms</td>
<td>5056.243ms</td>
<td>6336.243ms</td>
<td>7872.243ms</td>
<td>9472.243ms</td>
<td>11264.243ms</td>
<td>13184.243ms</td>
<td>15296.243ms</td>
<td>17536.243ms</td>
<td>19968.243ms</td>
<td>22528.243ms</td>
<td>25280.243ms</td>
<td>28096.243ms</td>
<td>31168.243ms</td>
<td>34304.243ms</td>
<td>37632.243ms</td>
<td>41152.243ms</td>
<td>44800.243ms</td>
<td>48576.243ms</td>
<td>52544.243ms</td>
</tr>
</tbody></table>
<p><em>1 ms: 1ms</em></p>
<p><em>x axis: memory.current in MiB when the op started (memory.high = 230.4 MiB, memory.max = 256 MiB). Median per 1 MiB over 3 repetitions, warm code, Ubuntu 6.8.0-147, no swap. 'Sampler on the same CPU' is the main matrix (anon-over-high: the workload inherited the driver's CPU 0, sampled every 100 ms); 'own CPU' is diag-over-high-cpu2-1s (workload on CPU 2, sampled every 1 s). memory.max only stays flat up to 256 MiB, then the OOM kill. The formula lines add the no-limit median to the kernel's sleep: once per 64-page op, or once per 4 KiB page. Source: results/runs/anon-over-</em>/ops.jsonl, results/diag/diag-over-high-cpu2-1s-<em>/ops.jsonl, scripts/50-charts.py.</em></p>
<p>In the matrix runs (the orange line), up to 232 MiB an operation took about a quarter of a millisecond, the same as with no limit. Within the next 5 MiB the median went from 0.22 ms to 4.6 seconds, and at 241 MiB it was 10.3 seconds per 256 KiB. After 150 seconds <code>memory.current</code> was 241.5 to 242.2 MiB, 11 to 12 MiB over <code>memory.high</code>, and still growing by 25 to 30 KiB per second. The process used 0.59 to 0.72 seconds of CPU in those 150 seconds. In 98% to 99% of the samples it was in state <code>D</code>, in <code>mem_cgroup_handle_over_high</code>. The purple line is the same workload on a CPU of its own. It stays near the one-sleep-per-batch line for most of the range, with jumps when it falls into a slow phase. The section on how many sleeps a task pays explains both lines.</p>
<p>Here is the same workload through systemd's <code>MemoryMax=</code> and <code>MemoryHigh=</code>, without our driver. The progress line shows the slowest operation in each second:</p>
<p><strong>results/25-demo.txt (no driver, no CPU pinning)</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># commands as scripts/25-demo.sh prints them (it also passes --quiet --slice=ddmemhigh.slice); 320 MiB target</span>
$ systemd-run --scope -p MemoryMax=256M python3 -I scripts/lib/workload.py --kind anon --target-mib 320 --progress
pid 173416 <span class="hljs-keyword">in</span> /sys/fs/cgroup/ddmemhigh.slice/run-r765e1a07143e4aa1be21a9eb2292aaff.scope, memory.high=max memory.max=268435456
   0.0s  held    0 MiB  memory.current    7.0 MiB  ops     1  slowest op     0.33 ms
scripts/25-demo.sh: line 17: 173416 Killed                  systemd-run --scope --quiet --slice=ddmemhigh.slice -p MemoryMax=256M <span class="hljs-variable">$w</span>
<span class="hljs-built_in">exit</span> status 137
$ systemd-run --scope -p MemoryHigh=241590272 -p MemoryMax=256M python3 -I scripts/lib/workload.py --kind anon --target-mib 320 --progress --max-seconds 30
pid 173417 <span class="hljs-keyword">in</span> /sys/fs/cgroup/ddmemhigh.slice/run-rbd28bd3d399c47a9ab104c3b6d4b6eaf.scope, memory.high=241590272 memory.max=268435456
   0.0s  held    0 MiB  memory.current    6.7 MiB  ops     1  slowest op     0.30 ms
   1.0s  held  231 MiB  memory.current  238.2 MiB  ops   924  slowest op    79.86 ms
   7.1s  held  231 MiB  memory.current  238.5 MiB  ops   926  slowest op  5488.05 ms
  12.8s  held  231 MiB  memory.current  238.8 MiB  ops   927  slowest op  5759.67 ms
  19.0s  held  232 MiB  memory.current  239.0 MiB  ops   928  slowest op  6135.79 ms
  25.5s  held  232 MiB  memory.current  239.3 MiB  ops   929  slowest op  6511.84 ms
  27.5s  held  232 MiB  memory.current  239.8 MiB  ops   930  slowest op  1992.06 ms
  28.5s  held  234 MiB  memory.current  241.1 MiB  ops   936  slowest op   359.79 ms
  29.6s  held  234 MiB  memory.current  241.9 MiB  ops   939  slowest op   759.73 ms
end: <span class="hljs-keyword">time</span> after 30.4s, 940 ops, held 235 MiB, memory.current 242.1 MiB
<span class="hljs-built_in">exit</span> status 0
</code></pre><p>The first process was killed before its second progress line, so in less than a second, with exit status 137. The second got 11.7 MiB past <code>memory.high</code> in 30 seconds, and its operations took 5.5 to 6.5 seconds each until they sped up near the end. The next section explains that change.</p>
<p>The <code>anon-fit</code> runs matter more, because nothing in them is wrong. The program needs 239 MiB, fits under its 256 MiB <code>memory.max</code>, and runs at full speed there. With <code>memory.high</code> at 230.4 MiB added, it held about 239 MiB and every churn operation slept. In the matrix, where it shared its CPU with our sampler, the median churn operation took 6,216 ms instead of 0.257 ms, and it finished 15 to 17 of its 200 churn operations before the 150 second cap. That is about 24,000 times slower per operation, with the same memory use and no error anywhere. On a CPU of its own it was about 750 times slower. The next section explains the difference. <code>memory.max</code> was never reached and its counter stayed at 0.</p>
<h2>How many sleeps: it depends on who else runs on the CPU</h2><p>To compare with the kernel, we count sleeps. In the warm runs reclaim never freed anything, so each sleep raised the <code>high</code> counter by 17, and an operation's sleep count is its rise divided by 17. We skip operations whose rise is not a multiple of 17 (11 to 13 per run, all at the edge of the limit, where the formula gives 10 ms or less and the kernel does not sleep). Then the measured latency divided by (sleeps x formula) should be 1.</p>
<p>It was 1.06 to 1.10 at the median in every single-process matrix run, and 97% to 100% of the throttled operations were within 30% of the formula (<code>results/predict.md</code>, section 1). The formula is right about how long each sleep is. The section on the 2 second cap shows where part of the extra 6% to 10% comes from.</p>
<p>The number of sleeps was the surprise. The kernel charges memory in batches of 64 pages and keeps the rest of a batch as a per-CPU precharge (<code>consume_stock()</code>), so the code comment on the cap says "per allocation batch". A 64-page operation should sleep about once. In the matrix, once the sleeps were 50 ms or longer, almost every operation slept 64 times, once per page:</p>
<table>
<thead>
<tr>
<th>Formula sleep length</th>
<th>Operations</th>
<th>Median sleeps per 64 pages</th>
<th>Operations with 64 or more</th>
</tr>
</thead>
<tbody><tr>
<td>11 to 25 ms</td>
<td>83</td>
<td>2</td>
<td>24%</td>
</tr>
<tr>
<td>25 to 50 ms</td>
<td>95</td>
<td>7</td>
<td>23%</td>
</tr>
<tr>
<td>50 to 100 ms</td>
<td>228</td>
<td>64</td>
<td>92%</td>
</tr>
<tr>
<td>100 to 200 ms</td>
<td>73</td>
<td>64</td>
<td>100%</td>
</tr>
<tr>
<td>200 to 400 ms</td>
<td>22</td>
<td>64</td>
<td>100%</td>
</tr>
<tr>
<td>400 to 2000 ms</td>
<td>29</td>
<td>64</td>
<td>100%</td>
</tr>
</tbody></table>
<p>A new charge, and so a new sleep, only happens when the precharge is empty. <code>__refill_stock()</code> empties a CPU's precharge when a different cgroup charges memory on that CPU. In review we found a cgroup that did exactly that in every run: our own driver. It pinned itself to CPU 0 before it started the workload, so every workload in the matrix inherited CPU 0 (the <code>cpu</code> field of every operation says 0). The driver woke up every 100 ms to sample and parsed the files into Python objects, which charges memory to the driver's own cgroup on that CPU. We did not trace each drain, but the runs below show the effect.</p>
<p>So we ran the 320 MiB <code>memory.high</code> workload again, 3 times each, with the workload on CPU 0 or CPU 2 and the sampler every 100 ms or every second (<code>results/diag</code>, 120 second cap):</p>
<table>
<thead>
<tr>
<th>Workload CPU</th>
<th>Sampler</th>
<th>Median sleeps per op (formula sleep 50 ms or more)</th>
<th>Share of that time in ops with 32 or more sleeps</th>
<th><code>memory.current</code> at the cap</th>
</tr>
</thead>
<tbody><tr>
<td>0, with the sampler</td>
<td>every 100 ms</td>
<td>64</td>
<td>100%</td>
<td>241.3 MiB</td>
</tr>
<tr>
<td>0, with the sampler</td>
<td>every 1 s</td>
<td>2</td>
<td>46%</td>
<td>248.8 to 249.8 MiB</td>
</tr>
<tr>
<td>2, alone</td>
<td>every 100 ms</td>
<td>1</td>
<td>53%</td>
<td>250.0 to 252.0 MiB</td>
</tr>
<tr>
<td>2, alone</td>
<td>every 1 s</td>
<td>1</td>
<td>72%</td>
<td>243.5 to 252.4 MiB</td>
</tr>
</tbody></table>
<p>With the sampler on the same CPU every 100 ms, the task slept once per page for the whole run, as in the matrix: a median of 64 sleeps per operation, and 100% of the throttled time in operations with 32 or more sleeps. In the other three settings the median operation slept once or twice. But within each run the count switched between phases of about 1 sleep per operation and phases of 64, each lasting tens of seconds, and the slow phases still took 46% to 72% of the throttled time (<code>results/predict.md</code>, section 1c). That is consistent with other memory activity on the host emptying the precharge in those phases, but we did not trace it. The demo session further down ran with no driver and no pinning at all, and it also paid 5.5 to 6.5 seconds per 256 KiB operation for 25 seconds before it sped up.</p>
<p>So our sampler was not the only cause of the per-page cost. With it on the same CPU, the cost was constant. Read the matrix numbers as a task whose CPU also runs another cgroup that allocates memory every 100 ms. On its own CPU, the same <code>anon-fit</code> workload with <code>memory.high</code> took a median of 191.8 ms per churn operation in all 3 runs (<code>diag-fit-high-cpu2-1s</code>), and finished all 200 churn operations in 2 of the 3 runs within 120 seconds. That is about 750 times the no-limit median, instead of about 24,000 times.</p>
<p><code>MAP_POPULATE</code> is a second control. It faults in the whole chunk inside the <code>mmap()</code> call, so the task returns to user space once per chunk. On CPU 0 with the 100 ms sampler, it slept 2 times in 592 of 600 churn operations and 3 times in the other 8, a median churn operation took 191.8 ms, and all 200 churn operations were done in about 41 seconds (<code>anon-fit-high-populate</code>). How a program touches its memory changed the cost of the same <code>memory.high</code> by a factor of 32 here, with the same neighbour on the CPU.</p>
<h2>The 2 second cap</h2><p>None of the runs above got far enough over <code>memory.high</code> to reach the cap, because the throttle slowed them first. So <code>anon-clamp-high</code> gets there another way: it grows to 300 MiB with no limit, then writes 241590272 to its own <code>memory.high</code>. Its usage of 307.2 MiB is then 33.3% over the limit. The formula then asks for a sleep of about 7,100 ms, and the kernel caps it. Each churn operation then allocates one 4 KiB page.</p>
<table>
<thead>
<tr>
<th>Run</th>
<th>Over <code>memory.high</code></th>
<th>Formula without the cap</th>
<th>Sleeps per operation</th>
<th>Time per sleep, median</th>
<th>min</th>
<th>max</th>
</tr>
</thead>
<tbody><tr>
<td><code>anon-clamp-high-r1</code></td>
<td>33.3%</td>
<td>7,106 ms</td>
<td>1, 2</td>
<td>2,047.9 ms</td>
<td>2,047.3 ms</td>
<td>2,064.9 ms</td>
</tr>
<tr>
<td><code>anon-clamp-high-r2</code></td>
<td>33.3%</td>
<td>7,108 ms</td>
<td>1, 2</td>
<td>2,047.9 ms</td>
<td>2,021.8 ms</td>
<td>2,048.0 ms</td>
</tr>
<tr>
<td><code>anon-clamp-high-r3</code></td>
<td>33.3%</td>
<td>7,107 ms</td>
<td>1, 2</td>
<td>2,047.9 ms</td>
<td>2,011.7 ms</td>
<td>2,047.9 ms</td>
</tr>
</tbody></table>
<p>The median sleep took 2,047.9 ms in all three runs, not 2,000 ms. The cap is 2000 jiffies, and at <code>HZ=1000</code> the kernel's timer wheel keeps timeouts from 512 ms to 4,095 ms in 64 ms steps (<code>kernel/time/timer.c</code>), so a 2,000 ms timeout fires at one of those steps. The median of 2,047.9 ms fits that. Single sleeps ranged from 2,011.7 to 2,064.9 ms, depending on where in a step the timer started. The same rounding, at 8 ms steps from 64 to 511 ms, explains part of the 6% to 10% we saw on the shorter sleeps: with half a step added, the median ratio for sleeps of 64 ms or more drops to 1.030 to 1.055 in the matrix runs. Under 64 ms, where the step is 1 ms, it stays at 1.054 to 1.091, and we did not find the rest (<code>results/predict.md</code>, section 1).</p>
<p>So the cap holds, but it is per sleep, not per allocation. In these runs each 4 KiB operation slept twice after the first one, so it took 4,096 ms. A 256 KiB allocation that sleeps once per page at the cap would take more than two minutes.</p>
<h2>Both limits: high at 90% of max</h2><p>systemd's man page recommends both: "use <code>MemoryHigh=</code> as the main control mechanism and use <code>MemoryMax=</code> as the last line of defense". The idea is that the program slows down before it dies. The <code>anon-over-both</code> runs show what that looks like with our 230.4 MiB and 256 MiB:</p>
<p><strong>memory.current from 10 s to 150 s, the workload asks for 320 MiB</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>10s</th>
<th>20s</th>
<th>30s</th>
<th>40s</th>
<th>50s</th>
<th>60s</th>
<th>70s</th>
<th>80s</th>
<th>90s</th>
<th>100s</th>
<th>110s</th>
<th>120s</th>
<th>130s</th>
<th>140s</th>
<th>150s</th>
</tr>
</thead>
<tbody><tr>
<td>memory.high only, 1 process</td>
<td>237.3 MiB</td>
<td>237.9 MiB</td>
<td>238.4 MiB</td>
<td>238.8 MiB</td>
<td>239.2 MiB</td>
<td>239.6 MiB</td>
<td>239.9 MiB</td>
<td>240.2 MiB</td>
<td>240.5 MiB</td>
<td>240.8 MiB</td>
<td>241.1 MiB</td>
<td>241.4 MiB</td>
<td>241.6 MiB</td>
<td>241.8 MiB</td>
<td>242.1 MiB</td>
</tr>
<tr>
<td>memory.high only, 4 processes</td>
<td>242.3 MiB</td>
<td>243.4 MiB</td>
<td>244.5 MiB</td>
<td>245.6 MiB</td>
<td>246.1 MiB</td>
<td>246.6 MiB</td>
<td>247.1 MiB</td>
<td>247.5 MiB</td>
<td>247.9 MiB</td>
<td>248.3 MiB</td>
<td>248.7 MiB</td>
<td>249.1 MiB</td>
<td>249.4 MiB</td>
<td>249.8 MiB</td>
<td>250.1 MiB</td>
</tr>
<tr>
<td>memory.high + memory.max</td>
<td>236.8 MiB</td>
<td>237.5 MiB</td>
<td>238 MiB</td>
<td>238.5 MiB</td>
<td>238.9 MiB</td>
<td>239.3 MiB</td>
<td>239.7 MiB</td>
<td>240 MiB</td>
<td>240.3 MiB</td>
<td>240.6 MiB</td>
<td>240.9 MiB</td>
<td>241.2 MiB</td>
<td>241.4 MiB</td>
<td>241.7 MiB</td>
<td>241.9 MiB</td>
</tr>
</tbody></table>
<p><em>memory.high: 230.4 MiB</em></p>
<p><em>memory.max: 256 MiB</em></p>
<p><em>Median of 3 repetitions at each 10 s mark, sampled from outside the cgroup every 100 ms. memory.max only reaches 256 MiB and is killed in under 2 s, so it is not drawn. Source: results/runs/anon-over-</em>/samples.jsonl.*</p>
<p>Within the 150 second cap, <code>memory.high</code> plus <code>memory.max</code> behaved like <code>memory.high</code> alone, because the process did not get near <code>memory.max</code>. After 150 seconds all three runs were at 241.7 to 242.0 MiB, 14 MiB under the kill. The growth rate fits the model of one sleep per page: 4 KiB per sleep, at the formula's sleep length for the current usage, predicts 24.7 to 26.1 KiB/s at the end of the runs, and we measured 26.6 to 28.0 KiB/s over the last 60 seconds (<code>results/predict.md</code>, section 5).</p>
<p>To see when the kill comes, we ran the same workload once more with a 45 minute cap (<code>anon-over-both-long</code>, one run, the same setup as the matrix, so on CPU 0 next to the sampler). The kernel killed it after 1,747.7 seconds, 29.1 minutes:</p>
<p><strong>memory.high at 90% of memory.max: the climb to the OOM kill</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>1m</th>
<th>2m</th>
<th>3m</th>
<th>4m</th>
<th>5m</th>
<th>6m</th>
<th>7m</th>
<th>8m</th>
<th>9m</th>
<th>10m</th>
<th>11m</th>
<th>12m</th>
<th>13m</th>
<th>14m</th>
<th>15m</th>
<th>16m</th>
<th>17m</th>
<th>18m</th>
<th>19m</th>
<th>20m</th>
<th>21m</th>
<th>22m</th>
<th>23m</th>
<th>24m</th>
<th>25m</th>
<th>26m</th>
<th>27m</th>
<th>28m</th>
<th>29m</th>
</tr>
</thead>
<tbody><tr>
<td>memory.current</td>
<td>239.7 MiB</td>
<td>241.4 MiB</td>
<td>242.8 MiB</td>
<td>243.9 MiB</td>
<td>244.8 MiB</td>
<td>245.7 MiB</td>
<td>246.4 MiB</td>
<td>247.1 MiB</td>
<td>247.8 MiB</td>
<td>248.4 MiB</td>
<td>248.9 MiB</td>
<td>249.5 MiB</td>
<td>250 MiB</td>
<td>250.5 MiB</td>
<td>250.9 MiB</td>
<td>251.3 MiB</td>
<td>251.7 MiB</td>
<td>252.1 MiB</td>
<td>252.5 MiB</td>
<td>252.8 MiB</td>
<td>253.2 MiB</td>
<td>253.5 MiB</td>
<td>253.9 MiB</td>
<td>254.2 MiB</td>
<td>254.5 MiB</td>
<td>254.8 MiB</td>
<td>255.1 MiB</td>
<td>255.4 MiB</td>
<td>255.7 MiB</td>
</tr>
</tbody></table>
<p><em>memory.high: 230.4 MiB</em></p>
<p><em>memory.max: 256 MiB</em></p>
<p><em>One run, sampled every 100 ms, one point a minute. Source: results/runs/anon-over-both-long-r1/samples.jsonl.</em></p>
<p>In those 29 minutes the process used 1.19 seconds of CPU, its memory pressure <code>some</code> covered 1,742 seconds, and its slowest 256 KiB allocation took 53.2 seconds. From the point where the sleep starts (233.4 MiB, 0.3 seconds in), the formula with one sleep per page predicts 1,720 seconds to reach <code>memory.max</code>, 1.6% less than we measured. With one sleep per 64 pages it would have been 27 seconds (<code>results/predict.md</code>, section 3). Then the OOM kill came with 35 <code>max</code> events, as in the <code>memory.max</code> runs. <code>memory.high</code> did not save the program. It delayed the same OOM kill by half an hour.</p>
<p>On a CPU of its own (<code>diag-over-both-cpu2-1s</code>, 3 runs, 120 second cap), the same workload was not killed within the cap either. It reached 240.8 to 254.4 MiB, depending on how much of the run fell into the one-sleep-per-page phases. So we have two reference points, not a proven range: the formula's 27 seconds if every sleep covers 64 pages, and the 29.1 minutes we measured once next to the sampler.</p>
<h2>The throttle is per process</h2><p><code>memory.high</code> puts each allocating task to sleep on its own. In the matrix, the four processes of <code>anon-over-4p-high</code> all ran on CPU 0 next to the sampler. We cannot split <code>memory.events</code> per process, but their combined growth matches the model of one sleep per page for each process: about four times as fast as one process at the same overage. In <code>anon-over-4p-high</code> the four processes reached 249.1 to 250.8 MiB in 150 seconds, against 241.5 to 242.2 MiB for one process. Over the last 60 seconds they grew 34.0 to 42.3 KiB/s, where the one-sleep-per-page model predicts 31.9 to 38.0 KiB/s for four processes at that usage. A program with many allocating worker processes gets to <code>memory.max</code> sooner than a single process.</p>
<p>The four throttled processes also changed a number most dashboards show. While they slept, the host's load average was 5.33 on 4 vCPUs, and <code>vmstat</code> showed the CPUs 96% and 99% idle in the two one-second samples:</p>
<p><strong>results/observed/loadavg-during-4p-high.txt</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># recorded during anon-over-4p-high-r1: ps, vmstat 1 3, /proc/loadavg, nproc</span>
$ <span class="hljs-built_in">cat</span> results/observed/loadavg-during-4p-high.txt
<span class="hljs-comment"># 2026-11-05T11:39:12Z during anon-over-4p-high-r1 (ps, vmstat 1 3, /proc/loadavg)</span>
 146187 D    mem_cgroup_handle_over_high       0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146188 D    mem_cgroup_handle_over_high       0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146189 D    mem_cgroup_handle_over_high       0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146190 D    mem_cgroup_handle_over_high       0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   <span class="hljs-keyword">in</span>   cs us sy <span class="hljs-built_in">id</span> wa st gu
 0  0      0 1407320  32980 5042700    0    0   835   650  876    2  4  1 94  0  1  0
 0  0      0 1410512  32980 5042720    0    0     0     0 1627 1417  2  3 96  0  0  0
 0  0      0 1413104  32980 5042736    0    0     0    28  621  787  1  0 99  0  0  0
5.33 3.56 2.27 1/347 147320
4
</code></pre><p>The kernel puts the task to sleep with <code>schedule_timeout_killable()</code>. That is a killable sleep: only a fatal signal ends it early. <code>ps</code> shows it as state <code>D</code>, and Linux counts it in the load average like other tasks in state <code>D</code>. A high load average with idle CPUs can mean throttled memory, not only slow disks.</p>
<h2>When the cgroup holds its own code, both limits thrash</h2><p>So far the interpreter's code pages were in the page cache before each run started, charged to a different cgroup. That is the <code>warm</code> setup, and it is the reason the runs above had nothing to reclaim. A container on its first start is different: its binary and libraries are read from its image after it starts, so the page cache for its own code is charged to its own cgroup. The <code>owncode</code> runs copy the interpreter binary to a new file, drop it from the page cache (<code>mincore()</code> found 0 of its pages cached before each run), and run that copy, so the binary is read inside the run cgroup.</p>
<p>That gives the kernel something to reclaim, and it changes both limits:</p>
<table>
<thead>
<tr>
<th>Run (20 s cap)</th>
<th>Outcome</th>
<th><code>memory.current</code></th>
<th>Pages reclaimed</th>
<th>Read from disk</th>
<th>PSI memory <code>some</code></th>
<th>PSI io <code>some</code></th>
</tr>
</thead>
<tbody><tr>
<td><code>memory.high</code> only, 232 MiB target</td>
<td>not killed in 20 s (3 of 3)</td>
<td>230.4 MiB</td>
<td>591,486 to 617,906</td>
<td>2,426 to 2,535 MB</td>
<td>11.4 to 11.9 s</td>
<td>19.9 to 21.2 s</td>
</tr>
<tr>
<td>both, 320 MiB target</td>
<td>not killed in 20 s (3 of 3)</td>
<td>230.2 to 230.4 MiB</td>
<td>633,090 to 634,192</td>
<td>2,593 to 2,598 MB</td>
<td>12.4 to 14.3 s</td>
<td>21.3 to 21.5 s</td>
</tr>
<tr>
<td><code>memory.max</code> only, 320 MiB target</td>
<td>not killed in 20 s (3 of 3)</td>
<td>255.9 MiB</td>
<td>631,197 to 633,258</td>
<td>2,585 to 2,594 MB</td>
<td>1.9 to 2.1 s</td>
<td>21.3 to 21.7 s</td>
</tr>
<tr>
<td><code>memory.max</code> only, uncapped disk, 1 run</td>
<td>not killed in 20 s (1 of 1)</td>
<td>256.0 MiB</td>
<td>2,559,211</td>
<td>10,483 MB</td>
<td>6.2 s</td>
<td>13.6 s</td>
</tr>
</tbody></table>
<p>Under <code>memory.high</code>, reclaim kept succeeding, and what it took was the program's own code. The task faulted it back in from disk, reclaim took it again, and so on. Because reclaim kept freeing pages, the kernel's retry loop never got to the sleep (<code>if (nr_reclaimed || nr_retries--)</code> retries as long as anything was freed). The sampler never found the process in <code>mem_cgroup_handle_over_high</code>, and only 1 of the 108 operations that raised the <code>high</code> counter raised it by a multiple of 17. The process sat at 230.2 to 230.4 MiB, right at <code>memory.high</code>. In 54% to 90% of the samples it was in <code>folio_wait_bit_common</code>, waiting for its code to come back from disk, and in the rest it was running. It never finished its grow phase.</p>
<p>Under <code>memory.max</code> alone, the same thing replaced the OOM kill. The charge path only calls the OOM killer after 16 tries in which reclaim freed nothing, and here reclaim kept finding code pages to free. The process held 255.9 MiB and was not killed in any of the three 20 second runs. <code>memory.events</code> counted 19,140 to 20,019 <code>max</code> events per run, and no OOM. On the uncapped disk, it read 10,483 MB in 25 seconds and was not killed either. In an earlier smoke run with the same setup (older code, in <code>results/_smoke5</code>), the kernel did kill it, 21.7 seconds after it started. So the kill does come in some runs, after a period of thrashing we could not predict. In another smoke run on the uncapped disk (<code>results/_smoke3</code>) it was still thrashing at 70 seconds when our driver stopped it.</p>
<p>Memory PSI under-counts this. In the <code>memory.max</code> runs with the 100 MiB/s cap, memory <code>some</code> was 1.9 to 2.1 seconds out of 25, about 8%, while io <code>some</code> was 21.3 to 21.7 seconds. The kernel counts a wait for a page read as a memory stall only when the page carries the workingset flag (<code>folio_wait_bit_common()</code> in <code>mm/filemap.c</code>), and most of this time went to the disk. A pressure-based killer that watches memory pressure alone would not see this as clearly as the warm <code>memory.high</code> runs, where memory <code>some</code> was 99%.</p>
<h2>Page cache: neither limit throttles or kills</h2><p>The <code>file-over</code> runs read 320 MiB of a file in 256 KiB reads, then read random chunks of it. That is page cache, which the kernel can drop and read again. None of the 12 runs was killed or throttled:</p>
<table>
<thead>
<tr>
<th>Run</th>
<th>Outcome</th>
<th>Peak MiB</th>
<th>Pages reclaimed</th>
<th>PSI memory <code>some</code></th>
<th>p99 per read</th>
</tr>
</thead>
<tbody><tr>
<td>no limit</td>
<td>not killed (3 of 3)</td>
<td>327.7</td>
<td>0</td>
<td>0.00 s</td>
<td>82.1 to 86.5 ms</td>
</tr>
<tr>
<td><code>memory.max</code> only</td>
<td>not killed (3 of 3)</td>
<td>256.0</td>
<td>20,608 to 21,440</td>
<td>0.02 s</td>
<td>83.4 to 88.6 ms</td>
</tr>
<tr>
<td><code>memory.high</code> only</td>
<td>not killed (3 of 3)</td>
<td>230.9</td>
<td>28,352 to 28,864</td>
<td>0.03 s</td>
<td>85.0 to 87.5 ms</td>
</tr>
<tr>
<td>both</td>
<td>not killed (3 of 3)</td>
<td>230.9</td>
<td>28,352 to 28,864</td>
<td>0.02 to 0.03 s</td>
<td>82.9 to 89.9 ms</td>
</tr>
</tbody></table>
<p>Both limits did what the docs say: <code>memory.current</code> stopped at the limit and the kernel reclaimed file pages to stay there. <code>memory.events</code> counted 416 to 430 <code>high</code> events and 322 to 335 <code>max</code> events per run. Every <code>high</code> event was a single reclaim pass with no sleep: the rise per operation was 1 or 2, never 17, because each pass freed enough to get back under the limit. The p99 of 82 to 90 ms is the same with no limit, so it is not the memory limits. The slow reads are in the first pass over the file, read from disk under our 100 MiB/s cap. The limits did cost something later: in the last 10% of operations, random reads that had to go back to disk gave a p99 of 1.0 to 2.6 ms, against 0.13 ms with no limit, where the whole file stayed cached. For a cache-heavy program without swap, both limits are a cache size, not a throttle and not a kill.</p>
<h2>What PSI and the counters show</h2><p>PSI was a reliable witness for the warm throttle. In every warm single-process matrix run with <code>memory.high</code>, the cgroup's <code>memory.pressure</code> <code>some</code> total matched the time the operations spent over their normal latency, at a ratio of 0.998 to 1.068 (<code>results/predict.md</code>, section 4). In the diagnostic runs it was 0.997 to 1.286. PSI also counts the last operation, which was still asleep when the driver stopped the run, and our operation log does not, so the ratio goes above 1 in runs that ended in the middle of a long operation. With one task in the cgroup, <code>full</code> was the same as <code>some</code>, and <code>avg10</code> reached 97% to 99%. The host saw it too. During the long run, the host-wide <code>/proc/pressure/memory</code> read <code>some avg10=96.53</code> and <code>full avg10=95.09</code> (<code>results/observed/host-psi-during-long.txt</code>). Our test cgroup was the one we know was stalled, and the host-wide number was almost as high, so an alert on host memory pressure would fire.</p>
<p>The counters to watch, from these runs:</p>
<ul>
<li><code>memory.events</code> <code>high</code> rising: the cgroup is over <code>memory.high</code>. 17 per sleep when nothing is reclaimable.</li>
<li><code>memory.events</code> <code>max</code> rising without <code>oom_kill</code>: the cgroup is at <code>memory.max</code> and reclaim keeps finding something. In the own-code runs that meant thrashing.</li>
<li><code>memory.stat</code> <code>pgsteal</code> and <code>workingset_refault_file</code> rising together: the cgroup reclaims pages and reads them straight back.</li>
<li><code>memory.pressure</code> <code>some</code> near 100% with almost no CPU use and state <code>D</code>: throttled by <code>memory.high</code>.</li>
<li><code>io.pressure</code> high while <code>memory.pressure</code> is low, in a cgroup at its memory limit: thrashing on file pages.</li>
</ul>
<h2>What systemd, Docker and Kubernetes write</h2><h3>systemd</h3><p>systemd maps its properties one to one. We started transient scopes with each property and read the files back from inside (<code>results/10-systemd-map.txt</code>, systemd 255):</p>
<table>
<thead>
<tr>
<th>Property</th>
<th>File</th>
<th>Value we set</th>
<th>Value in the file</th>
</tr>
</thead>
<tbody><tr>
<td><code>MemoryMin=16M</code></td>
<td><code>memory.min</code></td>
<td>16M</td>
<td>16777216</td>
</tr>
<tr>
<td><code>MemoryLow=64M</code></td>
<td><code>memory.low</code></td>
<td>64M</td>
<td>67108864</td>
</tr>
<tr>
<td><code>MemoryHigh=241590272</code></td>
<td><code>memory.high</code></td>
<td>241590272</td>
<td>241590272</td>
</tr>
<tr>
<td><code>MemoryMax=256M</code></td>
<td><code>memory.max</code></td>
<td>256M</td>
<td>268435456</td>
</tr>
<tr>
<td><code>MemoryHigh=3%</code></td>
<td><code>memory.high</code></td>
<td>3% of RAM</td>
<td>249806848</td>
</tr>
</tbody></table>
<p>A property you do not set stays at the kernel default: <code>max</code> for <code>memory.high</code> and <code>memory.max</code>, 0 for <code>memory.min</code> and <code>memory.low</code>. <code>memory.oom.group</code> stayed 0 in every scope.</p>
<p>The systemd 255 man page says <code>MemoryHigh=</code> is "the main mechanism to control memory usage of a unit", and it recommends <code>MemoryMax=</code> "as the last line of defense". On a host with swap, the kernel can move cold anonymous pages out and hold the unit near the limit. We did not test that. Without swap, our runs show what "main mechanism" means: the unit does not stop at the limit, it slows down, and with <code>MemoryHigh=</code> alone the kernel never kills it.</p>
<p><code>MemoryMin=</code> and <code>MemoryLow=</code> are the other direction. They protect a unit's memory from reclaim when something else needs it. They do not limit or slow the unit itself.</p>
<h3>Docker</h3><p>Docker 29.9.3 has no flag for <code>memory.high</code>. <code>docker run --help</code> on the test host lists <code>--memory</code>, <code>--memory-reservation</code>, <code>--memory-swap</code>, <code>--memory-swappiness</code>, <code>--oom-kill-disable</code> and <code>--oom-score-adj</code> (<code>results/00-host.txt</code>). We followed the two memory flags through the source (<code>results/01-source-check.txt</code>):</p>
<ul>
<li>Docker puts <code>--memory</code> into the OCI spec as <code>memory.limit</code> and <code>--memory-reservation</code> as <code>memory.reservation</code> (<code>getMemoryResources</code> in moby <code>daemon/daemon_unix.go</code>, docker-v29.9.3).</li>
<li>runc 1.5.2, the host's runtime, writes the limit to <code>memory.max</code> and the reservation to <code>memory.low</code> on cgroup v2 (<code>fs2/memory.go</code>). With the systemd cgroup driver it sets the <code>MemoryMax</code> and <code>MemoryLow</code> properties (<code>systemd/v2.go</code>).</li>
</ul>
<p>So on cgroup v2, <code>--memory</code> is a hard limit and <code>--memory-reservation</code> is reclaim protection. The Docker docs call <code>--memory-reservation</code> "a soft limit smaller than <code>--memory</code> which is activated when Docker detects contention or low memory on the host machine." That wording comes from cgroup v1's soft limit. On v2 it is <code>memory.low</code>: it does not slow the container down at any size.</p>
<p>runc does accept <code>memory.high</code> through the OCI spec's <code>unified</code> map, and passes it on as the systemd <code>MemoryHigh</code> property. That is the path the kubelet uses. The Docker CLI does not expose it, so with plain <code>docker run</code> you only get the <code>memory.max</code> behaviour.</p>
<h3>Kubernetes MemoryQoS</h3><p>The kubelet sets <code>memory.max</code> from <code>limits.memory</code> on cgroup v2 in every version. <code>memory.high</code> is a separate feature, <strong>MemoryQoS</strong> (<a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos" rel="noopener noreferrer">KEP-2570</a>), and its status changed in the last two releases. All of this is from <code>results/01-source-check.txt</code>:</p>
<table>
<thead>
<tr>
<th></th>
<th>Kubernetes 1.36 (v1.36.6)</th>
<th>Kubernetes 1.37 (v1.37.2)</th>
</tr>
</thead>
<tbody><tr>
<td><code>MemoryQoS</code> feature gate</td>
<td>alpha, off by default</td>
<td>beta, on by default</td>
</tr>
<tr>
<td><code>memoryThrottlingFactor</code> default</td>
<td>0.9</td>
<td>nil: no <code>memory.high</code> unless you set it</td>
</tr>
<tr>
<td><code>memoryReservationPolicy</code> default</td>
<td><code>None</code>: <code>memory.min</code> and <code>memory.low</code> are 0</td>
<td><code>None</code></td>
</tr>
</tbody></table>
<p>With the gate on and a factor set, the kubelet computes, for each container whose memory request differs from its limit:</p>
<pre><code class="hljs language-text" tabindex="0">memory.high = floor((requests.memory + factor * (limits.memory - requests.memory)) / pageSize) * pageSize
</code></pre><p>With no limit, it uses node allocatable memory instead of <code>limits.memory</code>. It puts the value in the CRI container config's <code>unified</code> map under the key <code>memory.high</code> (<code>kuberuntime_container_linux.go</code>), and the runtime writes it to the container's cgroup. runc passes a unified <code>memory.high</code> on as the systemd <code>MemoryHigh</code> property when it uses the systemd driver. In 1.36, a container whose memory request equals its limit gets no <code>memory.high</code>. 1.37 makes one exception: a container with no request and no limit can still get one when a factor is set.</p>
<p>Our <code>memory.high</code> value is this formula for <code>limits.memory: 256Mi</code>, <code>requests.memory: 0</code> and factor 0.9: 241590272 bytes. Two details matter here. If a pod sets a limit and leaves the request out, the API server sets the request equal to the limit (<code>SetDefaults_Pod</code> in <code>pkg/apis/core/v1/defaults.go</code>), and the kubelet then writes no <code>memory.high</code> at all. And with a request R between 0 and the limit, <code>memory.high</code> is R + 0.9 x (256Mi - R), so it sits between 230.4 MiB and 256 MiB. Our runs are the strictest case: a Burstable container with an explicit zero memory request on a 1.36 node with the MemoryQoS gate turned on. The <code>anon-fit</code> rows are the ones to read for that.</p>
<p>The KEP explains why it took so long. The 1.28 beta was cancelled because a workload "stays stuck indefinitely" at <code>memory.high</code>, and the KEP says kernel 5.9 fixed that livelock. On 6.8 our warm runs did make progress, slowly. The own-code runs barely did. The 1.37 beta turns the gate on but leaves <code>memoryThrottlingFactor</code> unset, so upgrading does not add <code>memory.high</code> to any container. If you set the factor to 0.9, you opt into the <code>both</code> rows above.</p>
<p>The two Kubernetes posts this week cover the other side of the same cgroups: <a href="https://devops-daily.com/posts/kubernetes-empty-pod-resources-besteffort">an empty <code>resources: {}</code> makes pods BestEffort</a> on 1.34 to 1.36 and changes their <code>oom_score_adj</code> and cgroup slice, and <a href="https://devops-daily.com/posts/kubernetes-1-37-garhwal-what-shipped">the 1.37 review</a> lists what else moved in that release.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>One host, one kernel, no swap.</strong> Everything here is Ubuntu 6.8.0-147 on a 4 vCPU machine with no swap. With swap or zswap, reclaim can move anonymous pages out, so <code>memory.high</code> can hold a cgroup near the limit instead of only slowing it, and <code>memory.max</code> reaches the OOM killer later. We did not test that. Newer kernels have changed the memcg charge code, and we did not test them either.</li>
<li><strong>What empties the precharge.</strong> We measured how many sleeps a task paid, and we read in the source what makes a new charge necessary. We did not trace which event emptied the per-CPU precharge each time. Our sampler on the same CPU made it happen on almost every long sleep. Without it, the phases came and went with the host's other work, which we did not control. An idle machine, or one with more CPUs, could pay fewer sleeps.</li>
<li><strong>The rest of the extra per sleep.</strong> The timer wheel's rounding explains about half of the 6% to 10%. We did not split the rest into reclaim time, wake-up delay and our own probes.</li>
<li><strong>When the own-code thrash ends.</strong> Under <code>memory.max</code> it was not killed in 20 seconds in the matrix (one smoke run was killed at 21.7 seconds, another was still running at 70). We stopped these runs early because they read about 100 MB/s from a shared disk, so we cannot say how long a real container would thrash before the kill. The result also depends on disk speed, which we changed with <code>io.max</code>.</li>
<li><strong>Run order and a shared host.</strong> Every repetition ran the configurations in the same order, on a machine with other services. Background load can move a single run either way. The three repetitions agree closely, but we did not randomize the order.</li>
<li><strong>Real programs.</strong> Our workload is a Python loop that touches memory in a fixed pattern. A JVM, a Go program or a database touches memory differently, and the <code>MAP_POPULATE</code> run shows that the touch pattern alone changed the cost 32 times.</li>
<li><strong>The 4-process numbers per operation.</strong> <code>memory.events</code> counts the whole cgroup, so in the 4-process run one operation's rise in <code>high</code> includes the other processes' sleeps. We used that run only for growth, not for sleeps per operation.</li>
<li><strong>Kubernetes on a real node.</strong> We wrote the same <code>memory.high</code> value the kubelet would write, but we did not run a kubelet. The Kubernetes section is from the source at v1.36.6 and v1.37.2.</li>
</ul>
<h2>What to do with this</h2><ul>
<li><strong>Without swap, do not use <code>memory.high</code> as the only limit.</strong> It does not stop growth. It slows new allocations, and a program that keeps allocating stays slow as long as it stays over the limit. The kernel never kills it for <code>memory.high</code>, so add <code>memory.max</code> or a pressure-based killer.</li>
<li><strong>If you set both, leave room.</strong> With <code>memory.high</code> at 90% of <code>memory.max</code>, a program that fits between the two ran about 750 times slower per allocation on a CPU apart from our sampler, and about 24,000 times slower next to it. Apart from the sampler it finished in 2 of 3 runs within 120 seconds; next to it, none finished within 150 seconds. Measure your program's peak first. Put <code>memory.high</code> above the normal working set, not at a fixed fraction of the limit.</li>
<li><strong>Do not plan with the formula alone.</strong> It gives the length of one sleep. How many sleeps a task pays depends on what else allocates memory on the same CPU, by up to 64 times, so test on a host that is as busy as production.</li>
<li><strong>Read <code>memory.events</code> and <code>memory.pressure</code>, not only <code>memory.current</code>.</strong> A throttled cgroup looks healthy by memory use: it sits a few MiB over its limit and uses no CPU. The <code>high</code> counter and memory <code>some</code> near 100% show it at once.</li>
<li><strong>Watch <code>io.pressure</code> and <code>pgsteal</code> on cgroups at their limit.</strong> Thrashing on a program's own code showed up there, not in memory pressure.</li>
<li><strong>On Kubernetes 1.37, MemoryQoS is on but does not throttle until you set <code>memoryThrottlingFactor</code>.</strong> If you set it, test your Burstable pods whose usage sits between the factor and the limit.</li>
</ul>
<p>The scripts, the raw output of every run, and the claim we wrote before the first run are in the repository above.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Alpine OpenSSH Password Bypass (CVE-2026-123158): Check and Fix]]></title>
      <link>https://devops-daily.com/posts/alpine-openssh-password-bypass</link>
      <description><![CDATA[Alpine sshd without PAM could let an attacker into accounts that have no password. Check one host or all your containers, then upgrade and restart sshd.]]></description>
      <pubDate>Thu, 05 Nov 2026 21:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/alpine-openssh-password-bypass</guid>
      <category><![CDATA[Security]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Security]]></category><category><![CDATA[Linux]]></category><category><![CDATA[OpenSSH]]></category><category><![CDATA[SSH]]></category><category><![CDATA[Alpine Linux]]></category><category><![CDATA[Docker]]></category><category><![CDATA[CVE]]></category>
      <content:encoded><![CDATA[<p>If you run sshd on Alpine Linux, upgrade openssh and restart sshd now:</p>
<pre><code class="hljs language-bash" tabindex="0">apk upgrade -U -a &amp;&amp; rc-service sshd restart
</code></pre><p>On 2026-11-05, Alpine published fixed <code>openssh</code> packages for every branch from 3.16 to edge, and OpenSSH released 10.6p2. The bug is <strong>CVE-2026-123158</strong>. On an affected host, a person who can reach the SSH port can log in to an account that has no password, without a password. The cause, in one sentence: when sshd is built without PAM, an error value from musl's <code>crypt()</code> could be treated as a match for accounts that have no password.</p>
<p>This post tells you how to find out if a host or a container is affected, how to fix it, and what to look for in your logs. It does not show how to trigger the bug.</p>
<h2>TLDR</h2><ul>
<li>You are affected only if four conditions are all true. In our tests, a fresh Alpine container with <code>openssh-server</code> failed two of them.</li>
<li>The fix is <code>apk upgrade -U -a</code>, then a restart of sshd. The upgrade alone does not replace the running daemon.</li>
<li>If you cannot upgrade, set <code>PasswordAuthentication no</code> and restart sshd.</li>
<li>Our read-only script checks the four conditions, the package version and the restart for a host or for every running container.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Root on the Alpine host, or access to <code>docker exec</code> for containers</li>
<li>A second SSH session open before you change sshd settings</li>
</ul>
<h2>Who is affected</h2><p>The <a href="https://alpinelinux.org/posts/2026-11-05-openssh-password-authentication-bypass/" rel="noopener noreferrer">Alpine advisory</a> says you are affected only if all of these are true:</p>
<ol>
<li><strong>sshd is built without PAM.</strong> The <code>openssh-server</code> package is. The <code>openssh-server-pam</code> package is affected too if <code>UsePAM</code> is <code>no</code>, which is the OpenSSH default when the config does not set it.</li>
<li><strong><code>PasswordAuthentication yes</code></strong>, which is the OpenSSH default.</li>
<li><strong>An account has <code>*</code> in its password field.</strong> On Alpine base images root is <code>root:*</code>, but the default <code>PermitRootLogin prohibit-password</code> blocks password logins for root. Key-only users, the <code>alpine</code> user on tiny-cloud images, and directory users through <code>musl-nscd</code> (they have no shadow entry) also count.</li>
<li><strong>At least one local account has a <code>$5$</code> or <code>$6$</code> hash.</strong> One user with a password is enough.</li>
</ol>
<p>You are not affected if <code>PasswordAuthentication</code> is <code>no</code>, if you use <code>openssh-server-pam</code> with <code>UsePAM yes</code>, if no account has a <code>$</code>-style hash, or if your containers do not run sshd. Locked accounts (<code>!</code> in the password field) are not affected. Instances where cloud-init sets <code>PasswordAuthentication no</code> are not affected.</p>
<p>These are the fixed <code>openssh</code> versions from the advisory:</p>
<table>
<thead>
<tr>
<th>Branch</th>
<th>Fixed version</th>
<th>Branch</th>
<th>Fixed version</th>
</tr>
</thead>
<tbody><tr>
<td>edge</td>
<td>10.6_p2-r0</td>
<td>3.20</td>
<td>9.7_p1-r6</td>
</tr>
<tr>
<td>3.24</td>
<td>10.3_p1-r2</td>
<td>3.19</td>
<td>9.6_p1-r3</td>
</tr>
<tr>
<td>3.23</td>
<td>10.2_p1-r1</td>
<td>3.18</td>
<td>9.3_p2-r4</td>
</tr>
<tr>
<td>3.22</td>
<td>10.0_p1-r11</td>
<td>3.17</td>
<td>9.1_p1-r7</td>
</tr>
<tr>
<td>3.21</td>
<td>9.9_p2-r1</td>
<td>3.16</td>
<td>9.0_p1-r6</td>
</tr>
</tbody></table>
<h2>Check one host</h2><p>We wrote a small POSIX <code>sh</code> script that reads these facts and changes nothing:</p>
<p><a href="https://github.com/The-DevOps-Daily/alpine-sshd-check" rel="noopener noreferrer">The-DevOps-Daily/alpine-sshd-check on GitHub</a></p>
<p>It reads the branch from <code>/etc/alpine-release</code>, the installed package version from <code>apk</code>, the effective settings from <code>sshd -T</code>, and whether the running sshd is still the sshd binary on disk. From <code>/etc/shadow</code> it reads only the type of each password field: it counts <code>*</code> fields and <code>$5$</code>/<code>$6$</code> fields. It never prints a hash or a user name, and it does not connect to sshd. Because it runs as root, it executes only the packaged sshd binaries in <code>/usr/sbin</code>, and only if root owns them and no one else can write to them. If openssh can come from edge on a stable host (an untagged edge repository, or a pin such as <code>openssh-server@edge</code>), it compares against the edge fix, and it says "cannot tell" when it cannot find where the package came from. Exit codes are <code>0</code> not affected, <code>1</code> affected, <code>2</code> fixed but restart needed, <code>3</code> cannot tell. Add <code>--json</code> for machine-readable output.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">sudo</span> sh check.sh
</code></pre><p>We ran it in throwaway containers on Alpine 3.21, 3.22, 3.23, 3.24 and edge after <code>apk add openssh-server</code>. sshd ran with <code>--network none</code>, so no port was open to anyone. Every branch installed the fixed version, and every run exited <code>0</code>. The run below is the closest one to an affected host: Alpine 3.24 with a key-only user set to <code>*</code> and one user with a random password, so all four conditions are true. The version is the only thing that keeps it safe:</p>
<p><strong>check.sh in the Alpine 3.24 test container</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># all four conditions true, fixed package installed</span>
$ docker <span class="hljs-built_in">exec</span> -i asc-users sh -s &lt; check.sh
alpine-sshd-check: CVE-2026-123158
Alpine release:     3.24.4 (branch 3.24)
sshd binary:        /usr/sbin/sshd (package openssh-server)
Installed openssh:  10.3_p1-r2
Fixed <span class="hljs-keyword">in</span>:           10.3_p1-r2 (installed is fixed: <span class="hljs-built_in">yes</span>)
[1] non-PAM sshd:                  <span class="hljs-built_in">yes</span>
[2] PasswordAuthentication <span class="hljs-built_in">yes</span>:    <span class="hljs-built_in">yes</span>
[3] account with <span class="hljs-string">'*'</span> that can use a password login: <span class="hljs-built_in">yes</span> (2 <span class="hljs-string">'*'</span> accounts, 1 usable; root is <span class="hljs-string">'*'</span>: <span class="hljs-built_in">yes</span>, PermitRootLogin prohibit-password)
[4] <span class="hljs-built_in">local</span> $5$/$6$ <span class="hljs-built_in">hash</span>:            <span class="hljs-built_in">yes</span> (1 found)
sshd running:       <span class="hljs-built_in">yes</span>, started 2026-11-05T21:49:52Z, binary on disk changed 2026-11-05T21:49:35Z, restart needed: no
note: root is <span class="hljs-string">'*'</span> but PermitRootLogin is <span class="hljs-string">'prohibit-password'</span>; root counts only with PermitRootLogin <span class="hljs-built_in">yes</span> (an account that can doas to root still matters)
Result:             NOT AFFECTED: fixed openssh installed (<span class="hljs-built_in">exit</span> 0)
</code></pre><p>Two things we found while we ran it:</p>
<ul>
<li>On 3.24, <code>adduser -D deploy</code> wrote <code>!</code> (locked), not <code>*</code>. A locked account is not affected, but it cannot log in with a key either. If you changed key-only users to <code>*</code> so that key login works, those are the accounts that matter.</li>
<li>OpenSSH 10.6 prints <code>sshd -T</code> keywords in mixed case (<code>PasswordAuthentication yes</code>). Older versions print lower case. The first version of our script matched only lower case and reported "unknown" on edge. Use <code>grep -i</code> in your own checks.</li>
</ul>
<h2>Check containers</h2><p>Containers that do not run sshd are not affected. To find the ones that do, <code>scan-containers.sh</code> lists the running containers, skips those that are not Alpine or have no sshd process, and pipes <code>check.sh</code> into the rest with <code>docker exec -i ... sh -s</code>. Nothing is copied into the container. Because it runs each container's own <code>sh</code> as root inside that container, scan only containers whose images you trust.</p>
<p><strong>scan-containers.sh</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ sh scan-containers.sh
CONTAINER      IMAGE          ALPINE                INSTALLED    FIXED        EXIT  RESULT
asc-users      asc-demo:3.24  3.24.4                10.3_p1-r2   10.3_p1-r2   0     not affected
asc-pa-no      asc-demo:3.24  3.24.4                10.3_p1-r2   10.3_p1-r2   0     not affected
asc-edge       asc-demo:edge  3.25.0_alpha20260805  10.6_p2-r0   10.6_p2-r0   0     not affected
asc-3.24       asc-demo:3.24  3.24.4                10.3_p1-r2   10.3_p1-r2   0     not affected
asc-3.23       asc-demo:3.23  3.23.8                10.2_p1-r1   10.2_p1-r1   0     not affected
asc-3.22       asc-demo:3.22  3.22.8                10.0_p1-r11  10.0_p1-r11  0     not affected
asc-3.21       asc-demo:3.21  3.21.10               9.9_p2-r1    9.9_p2-r1    0     not affected
</code></pre><p>If you build images that install <code>openssh-server</code>, a rebuild can reuse a cached <code>apk add</code> layer with the old package. Rebuild with <code>docker build --pull --no-cache</code>, then replace the running containers. A restart of the old container does not change its package.</p>
<h2>Fix it, and restart sshd</h2><pre><code class="hljs language-bash" tabindex="0">apk upgrade -U -a
rc-service sshd restart
</code></pre><p>The restart is not optional. The package upgrade replaces the files on disk, but it does not restart the sshd that is already running, and the advisory says to restart it after the upgrade. <code>check.sh</code> checks whether the running sshd is still the same file as the sshd binary on disk. An upgrade writes a new file, so a daemon that was not restarted is a different file, and the script exits <code>2</code>. Run it again after the restart.</p>
<p>On edge, the fix comes with OpenSSH 10.6p2. If you jump from an older 10.x, read our post on <a href="https://devops-daily.com/posts/openssh-10-6-domain-usernames-and-compression">what OpenSSH 10.6 changed</a> first: it refuses some usernames and changes <code>ssh -C</code>.</p>
<h2>Workaround: turn off password logins</h2><p>If you cannot upgrade now, turn off password authentication. Either step closes the hole.</p>
<blockquote>
<p><strong>Warning</strong></p>
<p>Make sure that key login works first. Keep your current session open, and test a new key login in a second session before you close it.</p>
</blockquote>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">mkdir</span> -p /etc/ssh/sshd_config.d
<span class="hljs-built_in">echo</span> <span class="hljs-string">'PasswordAuthentication no'</span> &gt; /etc/ssh/sshd_config.d/00-no-passwords.conf
sshd -t &amp;&amp; rc-service sshd restart
sshd -T | grep -i passwordauthentication   <span class="hljs-comment"># must print: passwordauthentication no</span>
</code></pre><p>Alpine's default <code>sshd_config</code> reads <code>/etc/ssh/sshd_config.d/</code> from 3.19 onward. On 3.18 and older, change the line in <code>/etc/ssh/sshd_config</code> itself; the advisory has a <code>sed</code> command for that. A <code>Match</code> block can turn password logins back on for some users or addresses, and <code>sshd -T</code> without <code>-C</code> does not show that, so read your <code>Match</code> blocks too. In our 3.24 test container, the drop-in changed condition 2 to <code>no</code>.</p>
<h2>The tiny-cloud alpine user</h2><p>The official tiny-cloud images (qcow2, VHD and AMI) from 3.18.11 to 3.24.3 have an <code>alpine</code> user with <code>*</code> in the password field and passwordless <code>doas</code>. They are safe on first boot. They become exposed as soon as any <code>$6$</code> hash is on the machine, for example when an admin sets a console password or user-data adds a user with <code>hashed_passwd</code>. Then an attacker can get a shell as <code>alpine</code> and use <code>doas</code> to become root.</p>
<p>Images from 3.21.10, 3.22.8, 3.23.8 and 3.24.4 contain the fixed package. Instances that started from older images do not get the fix from a new image: upgrade openssh on each one, or turn off password logins.</p>
<h2>What to look for in logs</h2><p>Look for password logins to accounts that have no password. On a host with syslog, that is <code>/var/log/messages</code>. For a container that runs <code>sshd -e</code>, it is <code>docker logs</code>.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># accounts with '*' in the password field (names only)</span>
awk -F: <span class="hljs-string">'$2 == "*" { print $1 }'</span> /etc/shadow

<span class="hljs-comment"># password logins; compare the user names with the list above</span>
grep <span class="hljs-string">'sshd.*Accepted password for'</span> /var/log/messages
docker logs &lt;container&gt; 2&gt;&amp;1 | grep <span class="hljs-string">'Accepted password for'</span>
</code></pre><p>Any <code>Accepted password for alpine</code>, for <code>root</code>, or for a key-only user is a bad sign. On exposed machines, also review <code>doas</code> use, new users, new files, and changes to <code>authorized_keys</code> files. If you cannot rule out such a login, treat the machine as compromised and rebuild it.</p>
<h2>What we could not check</h2><ul>
<li><strong>A vulnerable package or a real upgrade.</strong> Alpine replaced the packages on every branch the same day, so each fresh install got the fixed version. Exit codes <code>1</code> and <code>2</code> are simulations in the repo's <code>test-exit-codes.sh</code>: a copy of the script with a raised fixed version, and the same sshd binary renamed over <code>/usr/sbin/sshd</code> while sshd runs. Both gave the expected code.</li>
<li><strong><code>openssh-server-pam</code>, <code>musl-nscd</code> users, tiny-cloud images and branches 3.16 to 3.20.</strong> We did not measure them.</li>
<li><strong>Other musl systems.</strong> The <a href="https://www.cve.org/CVERecord?id=CVE-2026-123158" rel="noopener noreferrer">CVE record</a> covers sshd before 10.6p2 on Linux with musl and without PAM, and names Alpine as the example. We did not check other musl-based distributions or firmware builds.</li>
</ul>
<p>Sources: the <a href="https://alpinelinux.org/posts/2026-11-05-openssh-password-authentication-bypass/" rel="noopener noreferrer">Alpine advisory</a>, the <a href="https://www.openssh.com/releasenotes.html#10.6p2" rel="noopener noreferrer">OpenSSH 10.6p2 release notes</a> and the <a href="https://www.cve.org/CVERecord?id=CVE-2026-123158" rel="noopener noreferrer">CVE record</a>.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[OpenTelemetry Collector Custom Builds: 6.5x Smaller, and What Resource Detectors Cost at Startup]]></title>
      <link>https://devops-daily.com/posts/opentelemetry-collector-custom-build-resource-detectors</link>
      <description><![CDATA[We built a small Collector with ocb and started it next to the contrib release on a droplet, in two network namespaces and on a Raspberry Pi, 5 times per case. The custom build is 6.5x smaller and holds under a quarter of the memory. Cloud detectors for clouds the host is not on held startup for the full 5 s timeout, even on another cloud, and one detector stopped the Collector from starting at all.]]></description>
      <pubDate>Thu, 05 Nov 2026 15:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/opentelemetry-collector-custom-build-resource-detectors</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[OpenTelemetry]]></category><category><![CDATA[Observability]]></category><category><![CDATA[Go]]></category><category><![CDATA[Cloud]]></category><category><![CDATA[Linux]]></category>
      <content:encoded><![CDATA[<p>The contrib release of the OpenTelemetry Collector 0.164.0 is a 420.3 MB binary on linux/amd64. We built a Collector with the <strong>OpenTelemetry Collector Builder (ocb)</strong> that has only the 10 components a VM agent uses and only the <code>env</code> and <code>system</code> resource detectors. It is 64.3 MB, so contrib is 6.5 times larger. After 10 seconds idle, the custom build held 45.1 MiB of resident memory and contrib held 199.8 MiB. The anonymous part (heap, stacks and runtime data) was 8.0 MiB against 34.8 MiB.</p>
<p>The binary adds a little to startup: contrib needed 110 to 160 ms more than the small build on the droplet, and 300 to 500 ms more on the Pi. The detector list in your config can add seconds. We started both builds with cloud detectors for clouds the host was not on, in four network setups. In every list without <code>akamai</code>, each start waited for the slowest detector. When that detector got no answer, it waited for the whole <code>timeout</code>, which is 5 s by default. That happened on a Raspberry Pi with no metadata service. With a list of 13 cloud detectors it also happened on a DigitalOcean droplet, because two of them ask addresses that DigitalOcean does not answer. The detectors run in parallel, so where nothing answered, the list with 3 cloud detectors and the list with 13 cost the same 5 s.</p>
<p><code>timeout: 1s</code> cut the wait to about 1 s in most cases. Two detectors do not obey it. <code>digitalocean</code> ignores the deadline, so where nothing answers it waits its own 2 s. <code>akamai</code> makes its network call while the processor is created. In all four setups we tested, none of them on Akamai, the Collector did not start. It exited at once where the request got an answer or a fast error, and after 30 s where it got no answer.</p>
<p>On 29 October the OpenTelemetry blog showed <a href="https://opentelemetry.io/blog/2026/collector-with-less-dependencies/" rel="noopener noreferrer">how to compile detectors out with ocb</a> and reported a Kubernetes distribution about 20% smaller. This post measures three things that page does not cover. What does the same filter do to a small build? What does the build do to memory at idle? And what does the detector list cost at startup on hosts that are not on the clouds the config names? Every number below comes from a file in the companion repo.</p>
<h2>TL;DR</h2><ul>
<li><strong>Size.</strong> On amd64, contrib is 420.3 MB, the core <code>otelcol</code> release is 183.8 MB, our agent with every detector is 124.1 MB, and the same agent with only <code>env</code> and <code>system</code> is 64.3 MB. On this small build, the detector filter alone removed 48% of the binary and 64 Go modules: the AWS SDK, the Google Cloud clients, the Docker client, the Consul API and four more cloud metadata clients. arm64 gives the same picture.</li>
<li><strong>Memory.</strong> After 10 s idle: contrib 199.8 MiB RSS, core 98.4 MiB, our agent 61.5 MiB, the agent without cloud detectors 45.1 MiB. About four fifths of each number is file-backed memory (<code>RssFile</code>), which grows with the binary. The anonymous part (<code>RssAnon</code>) was 34.8 MiB for contrib and 8.0 MiB for the small build.</li>
<li><strong>Startup with no cloud detectors:</strong> about 190 to 220 ms for contrib and 45 to 85 ms for the small agents on the droplet, and 480 to 610 ms against 110 to 270 ms on the Pi.</li>
<li><strong>Startup with cloud detectors for other clouds:</strong> <code>[env, system, ec2, gcp, azure]</code> held startup for the full 5 s timeout where 169.254.169.254 got no answer. Where the connect failed at once, it held startup for 4.1 s, because the AWS SDK retries the connect. On the droplet those three cost under 4 ms each, but the 13-cloud list still took 5.1 s there, because <code>alibaba_ecs</code> and <code>scaleway</code> ask other addresses.</li>
<li><strong>The timeout setting</strong> bounds the wait: 1 s, 5 s or 10 s plus normal startup. The exceptions are <code>digitalocean</code>, which waits its own 2 s whatever you set when nothing answers, and <code>akamai</code>.</li>
<li><strong>No start at all.</strong> <code>akamai</code> in the list: exit 1 in all four setups, after 30 s in the two with no answer. <code>fail_on_missing_metadata: true</code>: exit 1 after the 5 s timeout. A detector that ocb compiled out but the config still names: <code>validate</code> passes, then a nil pointer panic at start. The processor's name for a detector in the ocb manifest (<code>ec2</code> instead of <code>aws_ec2</code>): the build stops.</li>
<li><strong>What to do.</strong> List only the detectors for the platform the host runs on, and build with the same list in <code>dist.resource.detection.detectors</code>. If one config must cover several clouds, keep the list short, set <code>timeout</code> to 1 or 2 s, and leave out <code>akamai</code>.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>An x86_64 Linux host for the builds, with about 3 GB of free disk. ocb cross-compiles the arm64 binaries there.</li>
<li><code>curl</code>, <code>python3</code> and <code>ip</code> (iproute2). Root only for the two network namespaces.</li>
<li>An arm64 Linux host if you want the second architecture. We used a Raspberry Pi with 4 cores and less than 4 GB of memory.</li>
<li>About an hour for a full run. The cases that fail after 30 s take most of it.</li>
</ul>
<h2>What we built and where we ran it</h2><p>We downloaded the official 0.164.0 releases of <code>otelcol-contrib</code> and <code>otelcol</code> for both architectures and checked each tarball against its published <code>.sha256</code>. Then ocb 0.164.0 built two versions of one small agent. It has the components a VM agent uses:</p>
<ul>
<li>receivers: OTLP, hostmetrics and filelog</li>
<li>processors: batch, memory_limiter and resource_detection</li>
<li>exporters: debug, OTLP and OTLP/HTTP</li>
<li>extension: health_check</li>
</ul>
<p><code>otelcol-agent</code> uses the ocb default, so every detector is compiled in. <code>otelcol-agent-envsys</code> adds the ocb setting that compiles out every detector you do not list:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># manifests/agent-envsys.yaml (excerpt, comment on detectors added)</span>
<span class="hljs-attr">dist:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">otelcol-agent-envsys</span>
  <span class="hljs-attr">version:</span> <span class="hljs-number">0.164</span><span class="hljs-number">.0</span>
  <span class="hljs-attr">build_tags:</span> <span class="hljs-string">'grpcnotrace'</span> <span class="hljs-comment"># same tag as the contrib release manifest</span>
  <span class="hljs-attr">resource:</span>
    <span class="hljs-attr">detection:</span>
      <span class="hljs-attr">detectors:</span> <span class="hljs-comment"># ocb 0.163.1 and later: every detector not listed gets an omit_detector_&lt;name&gt; build tag</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">env:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">system:</span>
</code></pre><p><code>go version -m</code> on the release binaries reports go1.26.10, so we pinned the same Go for our builds. <code>go version -m</code> and <code>file</code> show <code>-trimpath=true</code>, <code>CGO_ENABLED=0</code> and stripped symbols for every binary (<code>results/buildinfo/</code>).</p>
<p>Every start ran the same pipeline: OTLP in, batch, debug out. The harness gave each start a clean environment with only <code>PATH</code> and an empty <code>HOME</code>. No <code>OTEL_*</code> or <code>AWS_*</code> variable and no <code>~/.aws</code> file could change what a detector found. The Pi runs also set <code>TZ=UTC</code>, so their logs use UTC like the droplet's; the droplet runs used an earlier revision of the harness without it. Only the <code>resource_detection</code> block changed between configs. Each case ran 5 times in a shuffled order, after one warm-up start per binary that we left out of the numbers.</p>
<ul>
<li><strong>DigitalOcean droplet</strong> 4 vCPU, 8 GB, amd64<ul>
<li><strong>host network</strong> metadata server answers</li>
<li><strong>blackhole namespace</strong> packets dropped</li>
<li><strong>noroute namespace</strong> connect fails at once</li>
</ul>
</li>
<li><strong>Raspberry Pi</strong> 4 cores, under 4 GB, arm64<ul>
<li><strong>home network</strong> router drops 169.254.169.254</li>
</ul>
</li>
</ul>
<p>The four network setups matter more than the two architectures:</p>
<table>
<thead>
<tr>
<th>Setup</th>
<th>What a request to 169.254.169.254 gets</th>
</tr>
</thead>
<tbody><tr>
<td>Droplet host</td>
<td>DigitalOcean's metadata server answers. The EC2 and Azure paths return 404 in about 1 ms. This is the "host on another cloud" case.</td>
</tr>
<tr>
<td>Blackhole namespace</td>
<td>The default route goes to a dummy device, which discards every packet, so a connect gets no answer. DNS fails at once, because nothing listens on 127.0.0.53 inside the namespace.</td>
</tr>
<tr>
<td>Noroute namespace</td>
<td>Loopback only. A connect fails at once with "network is unreachable".</td>
</tr>
<tr>
<td>Raspberry Pi</td>
<td>The home router drops the packets. DNS works. This is the plain "not on any cloud" host.</td>
</tr>
</tbody></table>
<p><a href="https://github.com/The-DevOps-Daily/otelcol-ocb-vs-contrib" rel="noopener noreferrer">The-DevOps-Daily/otelcol-ocb-vs-contrib on GitHub</a></p>
<h2>Size: what ocb removes</h2><p><strong>Binary size, linux/amd64, OpenTelemetry Collector 0.164.0</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>otelcol-contrib (release)</td>
<td>420.3 MB</td>
<td>release</td>
</tr>
<tr>
<td>otelcol core (release)</td>
<td>183.8 MB</td>
<td>release</td>
</tr>
<tr>
<td>agent, all detectors</td>
<td>124.1 MB</td>
<td>custom</td>
</tr>
<tr>
<td>agent, env + system only</td>
<td>64.3 MB</td>
<td>custom</td>
</tr>
</tbody></table>
<p><em>results/sizes.tsv. Stripped, -trimpath, CGO_ENABLED=0, go1.26.10. The two agents have the same 10 components; only the compiled-in detectors differ.</em></p>
<p>The contrib manifest lists 246 components and 9 config providers, and the binary links 1,088 Go modules. The core release is not small either. Its 30 components include the Kafka and Prometheus receivers and exporters, and it also ships <code>resource_detection</code> with every detector. Our agent with every detector links 289 modules, and the detector filter takes it to 225.</p>
<p>The filter saves more on a small build than on a big one. The OpenTelemetry blog measured the Kubernetes distribution and got 182 MB to 147 MB with 13 fewer dependencies. On our agent it removed 59.8 MB of 124.1 MB (48%) on amd64, 52.9 MB of 112.7 MB (47%) on arm64, and 64 modules. A big distribution needs most of those SDKs for other components, so removing the detectors does not remove the SDKs. In a small agent, nothing else needs them. <code>results/deps/removed-by-detector-filter-amd64.txt</code> lists the 64: 15 <code>aws-sdk-go-v2</code> and <code>smithy-go</code> modules, the Google Cloud compute, auth and API clients, the Docker (moby) client, the Consul API with its HashiCorp libraries, and the Hetzner, Scaleway, Linode and DigitalOcean clients.</p>
<p>For containers, the official <code>otel/opentelemetry-collector-contrib:0.164.0</code> image is 116.1 MB compressed on amd64 (<code>results/images.tsv</code>). We did not build an image of the small agent. Its binary compresses to 20.3 MB with <code>gzip -6</code>, so a <code>FROM scratch</code> image would be near that, but that is an estimate, not a measurement.</p>
<p>One build difference favours our side. The contrib manifest asks for the <code>grpcnotrace</code> build tag and our builds use it, but the release binaries record no build tags in <code>go version -m</code>. We built <code>otelcol-agent-notag</code> to put a size on it: 124.23 MB against 124.11 MB, so the tag is worth 0.1%.</p>
<h2>Memory at idle</h2><p><strong>Resident memory after 10 s idle, droplet, amd64</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>contrib: RssAnon</td>
<td>34.8 MiB</td>
<td>RssAnon</td>
</tr>
<tr>
<td>contrib: RssFile</td>
<td>165.2 MiB</td>
<td>RssFile</td>
</tr>
<tr>
<td>core: RssAnon</td>
<td>18.7 MiB</td>
<td>RssAnon</td>
</tr>
<tr>
<td>core: RssFile</td>
<td>79.5 MiB</td>
<td>RssFile</td>
</tr>
<tr>
<td>agent, all detectors: RssAnon</td>
<td>11.2 MiB</td>
<td>RssAnon</td>
</tr>
<tr>
<td>agent, all detectors: RssFile</td>
<td>50.2 MiB</td>
<td>RssFile</td>
</tr>
<tr>
<td>agent, env + system: RssAnon</td>
<td>8 MiB</td>
<td>RssAnon</td>
</tr>
<tr>
<td>agent, env + system: RssFile</td>
<td>37 MiB</td>
<td>RssFile</td>
</tr>
</tbody></table>
<p><em>Median of 5 starts, /proc//status, config without resource_detection, results/summary.md. RssFile is resident file-backed memory; RssAnon is resident anonymous memory (heap, stacks, runtime).</em></p>
<p>Resident memory follows the binary size, but not one to one. Contrib held 199.8 MiB, 4.4 times the 45.1 MiB of the small agent. About four fifths of every number is <code>RssFile</code>, resident memory backed by files. It grows with the binary: 165.2 MiB for contrib, 37.0 MiB for the small agent. That points at the binary's own pages, but we did not map them one by one. The kernel can drop clean file-backed pages under memory pressure and read them again later. <code>RssAnon</code> is anonymous memory, which it cannot drop without swap. There contrib held 34.8 MiB against 8.0 MiB. A Go program runs the <code>init</code> code of every package it links before <code>main</code>, so a likely part of that difference is setup work for code that this pipeline never uses. We did not profile the heap to prove it.</p>
<p>Detection with <code>[env, system]</code> added little: contrib went to 205.4 MiB and the small agent to 47.8 MiB. The Raspberry Pi gave the same order on arm64: contrib 183.8 MiB with 34.7 MiB anonymous, the small agent 40.6 MiB with 8.0 MiB.</p>
<p>We did not test what happens to <code>RssFile</code> under memory pressure, or how a cgroup memory limit counts it. Read the <code>RssAnon</code> column as the floor and <code>VmRSS</code> as what <code>top</code> shows you.</p>
<h2>Startup: the slowest detector sets the time</h2><p>Median time from exec to the log line "Everything is ready", 5 starts each, from <code>results/summary.md</code>. Each cell is the small agent with every detector, then contrib in brackets:</p>
<table>
<thead>
<tr>
<th>Config</th>
<th>Droplet host</th>
<th>Blackhole namespace</th>
<th>Noroute namespace</th>
<th>Raspberry Pi</th>
</tr>
</thead>
<tbody><tr>
<td><code>[env, system]</code></td>
<td>79 ms (189 ms)</td>
<td>77 ms (210 ms)</td>
<td>83 ms (221 ms)</td>
<td>192 ms (481 ms)</td>
</tr>
<tr>
<td><code>+ ec2, gcp, azure</code></td>
<td>81 ms (196 ms)</td>
<td>5,088 ms (5,203 ms)</td>
<td>4,098 ms (4,242 ms)</td>
<td>5,172 ms (5,486 ms)</td>
</tr>
<tr>
<td><code>+ 13 VM clouds, no akamai</code></td>
<td>5,090 ms (5,213 ms)</td>
<td>5,086 ms (5,203 ms)</td>
<td>4,087 ms (4,227 ms)</td>
<td>5,167 ms (5,393 ms)</td>
</tr>
<tr>
<td><code>+ akamai</code></td>
<td>exit 1 after 0.1 s (0.2 s)</td>
<td>exit 1 after 30.1 s (30.2 s)</td>
<td>exit 1 after 0.1 s (0.2 s)</td>
<td>exit 1 after 30.2 s (30.5 s)</td>
</tr>
</tbody></table>
<p>Without a network wait, contrib needs about 110 to 160 ms more than the small agent to start on the droplet, and about 300 to 500 ms more on the Pi. That is real but small, and it matters only if you start collectors very often, for example one per short CI job.</p>
<p>The wait comes from three things in the processor source (<code>processor/resourcedetectionprocessor</code> at v0.164.1):</p>
<ol>
<li><strong>Detectors run in parallel.</strong> <code>detectResource</code> starts one goroutine per detector and then reads their results in order (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/resourcedetection.go#L203-L222" rel="noopener noreferrer">internal/resourcedetection.go, lines 203 to 222</a>). The wait is the slowest detector, not the sum.</li>
<li><strong>The whole detection has one deadline: <code>timeout</code>.</strong> The default is 5 s (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/factory.go#L153-L157" rel="noopener noreferrer">factory.go, lines 153 to 157</a>). With the default <code>retry.max_elapsed_time: 0</code>, <code>Refresh</code> puts that timeout on the context that every detector gets (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/resourcedetection.go#L159-L168" rel="noopener noreferrer">internal/resourcedetection.go, lines 159 to 168</a>).</li>
<li><strong>Startup waits for it.</strong> The processor's <code>Start</code> runs the first detection and returns its error (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/resourcedetection_processor.go#L36-L40" rel="noopener noreferrer">resourcedetection_processor.go, lines 36 to 40</a>). The Collector logs "Everything is ready" only after every processor has started.</li>
</ol>
<p>The processor also records how long each detector took, in its own <code>otelcol_resourcedetection_detector_duration</code> histogram. We read it from the Collector's Prometheus endpoint after each start. This is the 13-cloud list in the blackhole namespace:</p>
<p><strong>Time per detector, 13 VM cloud detectors, blackhole namespace, timeout 5s</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>ec2</td>
<td>5000.7 ms</td>
<td>waits for timeout</td>
</tr>
<tr>
<td>alibaba_ecs</td>
<td>4999.8 ms</td>
<td>waits for timeout</td>
</tr>
<tr>
<td>azure</td>
<td>4999.7 ms</td>
<td>waits for timeout</td>
</tr>
<tr>
<td>hetzner</td>
<td>5000.2 ms</td>
<td>waits for timeout</td>
</tr>
<tr>
<td>scaleway</td>
<td>4999.6 ms</td>
<td>waits for timeout</td>
</tr>
<tr>
<td>digitalocean</td>
<td>2000.6 ms</td>
<td>own 2 s timeout</td>
</tr>
<tr>
<td>nova</td>
<td>2000.4 ms</td>
<td>own 2 s timeout</td>
</tr>
<tr>
<td>upcloud</td>
<td>2000.7 ms</td>
<td>own 2 s timeout</td>
</tr>
<tr>
<td>vultr</td>
<td>2000.8 ms</td>
<td>own 2 s timeout</td>
</tr>
<tr>
<td>oraclecloud</td>
<td>200.7 ms</td>
<td>fast</td>
</tr>
<tr>
<td>ibmcloud_vpc</td>
<td>1.2 ms</td>
<td>fast</td>
</tr>
<tr>
<td>gcp</td>
<td>0.9 ms</td>
<td>fast</td>
</tr>
<tr>
<td>tencent_cvm</td>
<td>0.7 ms</td>
<td>fast</td>
</tr>
</tbody></table>
<p><em>Median of 5 starts of the small agent with all detectors, from otelcol_resourcedetection_detector_duration in results/runs/droplet-blackhole-extra.jsonl. Every detector reported outcome=success: "not on this cloud" is a success with no attributes.</em></p>
<p>The same histogram tells you which detector held each of the other setups:</p>
<table>
<thead>
<tr>
<th>Setup</th>
<th>Detectors that took the longest</th>
</tr>
</thead>
<tbody><tr>
<td>Droplet host</td>
<td><code>alibaba_ecs</code> and <code>scaleway</code>, 5.0 s each. They ask <code>100.100.100.200</code> and <code>169.254.42.42</code>, and nothing on DigitalOcean answers those. The other 11 finished in 2 to 20 ms (medians).</td>
</tr>
<tr>
<td>Blackhole namespace</td>
<td>Five detectors at 5.0 s and four at 2.0 s (see the chart).</td>
</tr>
<tr>
<td>Noroute namespace</td>
<td><code>ec2</code> alone, 4.0 s. Every other detector finished in under 2 ms.</td>
</tr>
<tr>
<td>Raspberry Pi</td>
<td>Six detectors at 5.0 s: the five from the blackhole namespace and <code>ibmcloud_vpc</code>, which took 1.2 ms in the namespace. We did not trace why. The same four at 2.0 s. With working DNS, <code>gcp</code> took 13.5 ms and <code>tencent_cvm</code> 15.9 ms.</td>
</tr>
</tbody></table>
<p>How each detector probes decides how long it waits:</p>
<ul>
<li><strong>azure</strong> sends one GET with a plain <code>&amp;http.Client{}</code> that has no timeout (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/internal/metadataproviders/azure/metadata.go#L33-L38" rel="noopener noreferrer">internal/metadataproviders/azure/metadata.go, lines 33 to 38</a>). When the packets get no answer, it waits for the processor's deadline. Then it returns an empty resource and no error (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/azure/azure.go#L55-L63" rel="noopener noreferrer">internal/azure/azure.go, lines 55 to 63</a>).</li>
<li><strong>ec2</strong> uses the AWS SDK's IMDS client: a 250 ms dial timeout, a 500 ms response header timeout (<a href="https://github.com/aws/aws-sdk-go-v2/blob/feature/ec2/imds/v1.20.1/feature/ec2/imds/api_client.go#L277-L278" rel="noopener noreferrer">aws-sdk-go-v2 feature/ec2/imds v1.20.1, api_client.go, lines 277 and 278</a>), and up to 3 attempts per request (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/aws/ec2/config.go#L16-L19" rel="noopener noreferrer">internal/aws/ec2/config.go, lines 16 to 19</a>). A connect that fails at once is retried with backoff, which is why <code>ec2</code> took 4.0 s in the noroute namespace. With no answer at all and a 10 s timeout, it gave up by itself after 5.5 s.</li>
<li><strong>gcp</strong> calls <code>metadata.OnGCEWithContext</code>, which races a DNS lookup of <code>metadata.google.internal</code> against an HTTP probe. On a host that does not look like a Google VM, it trusts the first answer (<a href="https://github.com/googleapis/google-cloud-go/blob/compute/metadata/v0.10.0/compute/metadata/metadata.go#L456-L480" rel="noopener noreferrer">cloud.google.com/go/compute/metadata v0.10.0, metadata.go, lines 456 to 480</a>). A fast NXDOMAIN or a fast DNS failure ends it.</li>
<li><strong>digitalocean</strong> ignores the context: its <code>Detect</code> takes <code>_ context.Context</code> (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/digitalocean/digitalocean.go#L51" rel="noopener noreferrer">internal/digitalocean/digitalocean.go, line 51</a>), and the DigitalOcean metadata client has a fixed 2 s timeout (<a href="https://github.com/digitalocean/go-metadata/blob/e3650a3df44b/client.go#L25-L64" rel="noopener noreferrer">digitalocean/go-metadata, client.go, lines 25 and 64</a>).</li>
</ul>
<p>We started from this claim: on a host that is not on EC2, Google Cloud or Azure, each cloud detector adds a wait at startup. Two results would prove it wrong: no wait at all, or the same wait for 3 detectors and for 13. We got the second, so it was half wrong. The detectors do not each add a wait. During detection they overlap, and the slowest one sets the time. But a host on a cloud with a metadata server is not safe either, because not every cloud uses 169.254.169.254.</p>
<h2>The timeout setting</h2><p><strong>Start to "Everything is ready" with [env, system, ec2, gcp, azure]</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>timeout: 1s</th>
<th>timeout: 5s (default)</th>
<th>timeout: 10s</th>
</tr>
</thead>
<tbody><tr>
<td>contrib, blackhole namespace</td>
<td>1238.8 ms</td>
<td>5203.1 ms</td>
<td>10235.4 ms</td>
</tr>
<tr>
<td>agent, blackhole namespace</td>
<td>1077.1 ms</td>
<td>5087.6 ms</td>
<td>10085.4 ms</td>
</tr>
<tr>
<td>contrib, Raspberry Pi</td>
<td>1526.7 ms</td>
<td>5485.5 ms</td>
<td>10710.8 ms</td>
</tr>
<tr>
<td>contrib, noroute namespace</td>
<td>1240.1 ms</td>
<td>4241.6 ms</td>
<td>4229.5 ms</td>
</tr>
<tr>
<td>contrib, droplet host</td>
<td>209.3 ms</td>
<td>196 ms</td>
<td>216 ms</td>
</tr>
</tbody></table>
<p><em>Median of 5 starts per point, results/summary.md. The droplet host answers on 169.254.169.254; the blackhole namespace and the Pi do not. The noroute namespace fails the connect at once.</em></p>
<p>Where nothing answers, the wait is the timeout plus normal startup, on both architectures. In the noroute namespace it stops at 4.1 to 4.2 s, where <code>ec2</code> gives up. <code>timeout: 1s</code> brought every three-cloud case that waited down to 1.1 to 1.2 s.</p>
<p>The 13-cloud list with <code>timeout: 1s</code> did not get there. In the blackhole namespace it started in 2,097 ms (contrib 2,234 ms), and on the Pi in 2,175 ms (contrib 2,410 ms), because <code>digitalocean</code> took its 2 s anyway.</p>
<p>A short timeout has a cost that we could not measure. On a real EC2, Google Cloud or Azure VM, a detector that needs more than 1 s gets cut off. It returns an empty resource with no error, and your telemetry has no cloud attributes. We did not run on those clouds, so we cannot say whether 1 s is enough there. Most examples in the processor README use <code>timeout: 2s</code>.</p>
<h2>Four ways the Collector does not start</h2><ol>
<li><strong>ocb build</strong> unknown detector name: stops</li>
<li><strong>otelcol validate</strong> passes a compiled-out detector</li>
<li><strong>create processor</strong> akamai fetches a token</li>
<li><strong>Start: detection</strong> parallel, one deadline</li>
<li><strong>Everything is ready</strong></li>
</ol>
<h3>1. The akamai detector</h3><p>Of the 14 VM cloud detectors we ran, <code>akamai</code> is the only one that waited or failed while the processor was created. Its <code>NewDetector</code> calls <code>newAkamaiClient(context.Background())</code> (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/akamai/akamai.go#L43-L57" rel="noopener noreferrer">internal/akamai/akamai.go, lines 43 to 57</a>), and the Linode metadata client fetches a token inside <code>NewClient</code> (<a href="https://github.com/linode/go-metadata/blob/v0.3.1/client.go#L88-L91" rel="noopener noreferrer">linode/go-metadata v0.3.1, client.go, lines 88 to 91</a>). This happens before detection, so <code>timeout</code> does not apply, and a failure there is a failure to build the pipeline.</p>
<p>On the droplet, DigitalOcean's metadata server answers the token request, but not with a token, and the Collector exits at once:</p>
<p><strong>results/runs/droplet-host-extra.jsonl, otelcol-contrib, configs/akamai.yaml</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ bin/linux-amd64/otelcol-contrib --config configs/akamai.yaml
Error: failed to build pipelines: failed to create <span class="hljs-string">"resource_detection"</span> processor, <span class="hljs-keyword">in</span> pipeline <span class="hljs-string">"traces"</span>: failed creating detector <span class="hljs-built_in">type</span> <span class="hljs-string">"akamai"</span>: failed to refresh metadata token: failed to generate metadata token: no token returned from API
<span class="hljs-comment"># exit code 1, 0.2 s after exec</span>
</code></pre><p>In the noroute namespace it exited at once with <code>connect: network is unreachable</code>. In the blackhole namespace and on the Pi, the token request first waited for a 30 s dial timeout. The Collector exited after 30.1 s in the namespace and 30.2 s on the Pi (contrib: 30.2 s and 30.5 s). With <code>timeout: 1s</code> it still took 30 s.</p>
<h3>2. fail_on_missing_metadata: true</h3><p>This setting makes a cloud detector return an error when its metadata service is missing, instead of an empty resource. Retry is on by default (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/factory.go#L142-L151" rel="noopener noreferrer">factory.go, lines 142 to 151</a>), so the processor retries the detector with backoff until <code>timeout</code>, and then <code>Start</code> returns the error. With <code>[env, system, ec2, gcp, azure]</code>, the Collector exited with code 1 after 5.1 to 5.3 s in all three droplet setups. The config comment says the setting is for nodes that boot before their metadata service is ready (<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/config.go#L63-L70" rel="noopener noreferrer">config.go, lines 63 to 70</a>). It also turns "not on EC2" into "does not start".</p>
<h3>3. A detector that ocb compiled out</h3><p>This is the trap in the new ocb feature. <code>otelcol-agent-envsys</code> has only <code>env</code> and <code>system</code>. We gave it the three-cloud config:</p>
<p><strong>results/break/amd64-validate.txt and results/break/amd64-start.txt</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ bin/linux-amd64/otelcol-agent-envsys validate --config configs/clouds3.yaml
<span class="hljs-built_in">exit</span> code: 0
$ bin/linux-amd64/otelcol-agent-envsys --config configs/clouds3.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x130a757]

goroutine 57 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor/internal.(*ResourceProvider).detectWithRetry.func2()
	github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.164.1/internal/resourcedetection.go:314 +0xb7
...
<span class="hljs-built_in">exit</span> code: 2
</code></pre><p>In v0.164.1 each <code>*_omit.go</code> stub returns <code>nil, nil</code> from <code>NewDetector</code>, for example <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.164.1/processor/resourcedetectionprocessor/internal/aws/ec2/ec2_omit.go" rel="noopener noreferrer">internal/aws/ec2/ec2_omit.go</a>. The processor keeps the nil detector and calls it. This is <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/52519" rel="noopener noreferrer">issue #52519</a>, open when we wrote this. On <code>main</code>, <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/52063" rel="noopener noreferrer">#52063</a> changes the stubs so they register no factory. It was merged after v0.164.1. With it, the processor returns <code>detector "ec2" is not compiled into this binary</code> with the list of compiled detectors. Until a release has that change, <code>validate</code> does not catch the mismatch, and the first sign is a crash.</p>
<h3>4. The processor's names in the ocb manifest</h3><p>ocb and the processor do not use the same detector names. The processor config says <code>ec2</code>, <code>nova</code> and <code>aks</code>. The ocb manifest wants the build tag names <code>aws_ec2</code>, <code>openstack_nova</code> and <code>azure_aks</code> (<a href="https://github.com/open-telemetry/opentelemetry-collector/blob/cmd/builder/v0.164.0/cmd/builder/internal/builder/config.go#L117-L151" rel="noopener noreferrer">cmd/builder/internal/builder/config.go, lines 117 to 151</a>). This one fails early, which is the good kind of failure:</p>
<p><strong>results/build/agent-badname-amd64.log</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ocb --config manifests/agent-badname.yaml
Error: invalid configuration: ec2: unknown resource detector
</code></pre><h2>What to put in the manifest and the config</h2><p>Make the two lists match, each in its own names:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># ocb manifest: build tag names</span>
<span class="hljs-attr">dist:</span>
  <span class="hljs-attr">resource:</span>
    <span class="hljs-attr">detection:</span>
      <span class="hljs-attr">detectors:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">env:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">system:</span>
        <span class="hljs-bullet">-</span> <span class="hljs-attr">aws_ec2:</span> <span class="hljs-comment"># the processor calls this one "ec2"</span>
</code></pre><pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># Collector config: processor names, the same detectors</span>
<span class="hljs-attr">processors:</span>
  <span class="hljs-attr">resource_detection:</span>
    <span class="hljs-attr">detectors:</span> [<span class="hljs-string">env</span>, <span class="hljs-string">system</span>, <span class="hljs-string">ec2</span>]
    <span class="hljs-attr">timeout:</span> <span class="hljs-string">2s</span> <span class="hljs-comment"># the most a missing metadata service can add to startup</span>
</code></pre><ul>
<li>Use one list per platform. A fleet on EC2 does not need <code>azure</code>, and a fleet on bare metal needs neither.</li>
<li>If one config must cover several clouds, keep the list short and set <code>timeout</code> to 1 or 2 s. Expect <code>digitalocean</code> to add 2 s, whatever the timeout, on hosts where its request gets no answer.</li>
<li>Leave <code>akamai</code> out unless the hosts run on Akamai Cloud.</li>
<li>Keep <code>fail_on_missing_metadata</code> off unless "no metadata" should mean "do not start".</li>
<li>Use the name <code>resource_detection</code>. In 0.164 the old name <code>resourcedetection</code> still works, with a deprecation warning (<code>results/break/amd64-legacy-name.txt</code>).</li>
<li>Until the #52063 change is in a release, start each new custom build once with its real config before you ship it. <code>validate</code> is not enough.</li>
</ul>
<h2>What we could not conclude</h2><ul>
<li><strong>On a real AWS, Google Cloud or Azure VM.</strong> We had no instances there, so we did not measure how long a detector takes when it succeeds, or whether a 1 s timeout cuts it off.</li>
<li><strong>The container image of the small agent.</strong> We report the compressed size of the official image and the gzip size of our binary. We did not build or push an image.</li>
<li><strong>Memory under pressure.</strong> <code>RssFile</code> can be reclaimed. We did not measure how much the kernel takes back, or how a cgroup limit counts it.</li>
<li><strong>Small startup differences.</strong> Each case has 5 starts. The 110 to 160 ms between contrib and the small agent repeats in every droplet setup. The Pi also ran other work, and three of its starts took 4.6 to 22 s. In all three, the time went before the Collector wrote its first log line, which points at reading the binary from the SD card, not at detection. We did not confirm that. They move the medians little, but they are in the min and max columns of <code>results/summary.md</code>.</li>
<li><strong>Slow DNS.</strong> DNS failed at once in the namespaces and answered on the Pi, so <code>gcp</code> and <code>tencent_cvm</code> were fast everywhere. A host whose DNS times out would make them wait too. We did not test that.</li>
<li><strong>Other versions.</strong> Everything here is 0.164.0 and 0.164.1. The akamai and omit behaviour may change in the next release.</li>
</ul>
<p>Related reading:</p>
<ul>
<li><a href="https://devops-daily.com/posts/distributed-tracing-opentelemetry-instrumentation-visualization">Distributed tracing with OpenTelemetry</a> runs a Collector end to end.</li>
<li><a href="https://devops-daily.com/posts/opentelemetry-graduated-what-to-retire-this-quarter">OpenTelemetry graduated</a> covers which vendor agents a Collector can replace.</li>
<li><a href="https://devops-daily.com/posts/trace-context-environment-variables">Trace context in environment variables</a> follows a trace through a CI pipeline.</li>
</ul>
<h2>Summary</h2><p>A custom Collector with only the components you use is 6.5 times smaller than contrib and holds under a quarter of its memory. On a small build, compiling out the unused detectors is half of the size saving. The binary costs tens to hundreds of milliseconds at startup. The detector list can cost seconds. Apart from <code>akamai</code>, the cloud detectors you list for clouds the host is not on share one wait: the slowest probe. Where that probe gets no answer, it usually runs to the full <code>timeout</code>, even on a host on another cloud. <code>digitalocean</code> waits its own 2 s instead, even past a shorter timeout. Keep the list to the platforms you run on, and build with the same list. Set a short timeout if the list must stay long. Start each new binary once with its real config. Leave <code>akamai</code> out unless you run on Akamai.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[An Empty resources: {} Makes Kubernetes 1.34 to 1.36 Pods BestEffort]]></title>
      <link>https://devops-daily.com/posts/kubernetes-empty-pod-resources-besteffort</link>
      <description><![CDATA[An empty pod-level block makes a pod with requests and limits BestEffort on Kubernetes 1.34 to 1.36. We measured what changes on the node, and what does not.]]></description>
      <pubDate>Wed, 04 Nov 2026 15:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/kubernetes-empty-pod-resources-besteffort</guid>
      <category><![CDATA[Kubernetes]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Kubernetes]]></category><category><![CDATA[QoS]]></category><category><![CDATA[GitLab Runner]]></category><category><![CDATA[k3s]]></category><category><![CDATA[cgroups]]></category><category><![CDATA[Eviction]]></category><category><![CDATA[CI/CD]]></category>
      <content:encoded><![CDATA[<p>On Kubernetes 1.34, 1.35 and 1.36, one empty block in a pod spec changes how the node treats the pod. If the pod sets <code>spec.resources: {}</code> at pod level, the kubelet gives it QoS class <strong>BestEffort</strong>. This happens even when every container sets requests and limits, whenever the <code>PodLevelResources</code> feature gate is on, which is the default from 1.34. GitLab Runner's Kubernetes executor builds its job pods with that empty block. Its source does this from 18.10.0 to 19.4.3, unless the runner config sets pod-level values.</p>
<p>We ran the same pods on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. On the three older versions, the empty block moved the pod into the BestEffort cgroup slice. The kubelet set its <code>oom_score_adj</code> to 1000, the highest value the kernel accepts. Next to a busy neighbour, a pod that requested half a CPU got 0.06 to 0.08 cores instead of 0.57 to 0.58. A namespace quota that blocks BestEffort pods rejected it. On 1.37.1 the same pods kept their Burstable and Guaranteed classes.</p>
<p>One thing did not change, and the research we started from got it wrong. It said these pods are first in line for kubelet eviction under memory pressure. They are not. The kubelet's eviction ranking compares memory usage with the memory request. For an empty block, that request still comes from the containers. The QoS class is not part of the ranking, and the kubelet evicted our pods in the same order on every version.</p>
<h2>TL;DR</h2><ul>
<li>On k3s v1.34.12, v1.35.9 and v1.36.5, a pod with container requests and limits plus <code>spec.resources: {}</code> reported <code>qosClass: BestEffort</code>. The Guaranteed variant did too. On v1.37.1 they reported Burstable and Guaranteed. With the new <code>PodLevelResourcesFixKubeletQOSClass</code> feature gate turned off, 1.37.1 went back to BestEffort.</li>
<li>The node settings followed the class. <code>oom_score_adj</code> went from 935 (Burstable) and -997 (Guaranteed) to 1000, and the pod cgroup moved to <code>kubepods-besteffort.slice</code> with <code>cpu.weight</code> 1 and no pod-level <code>memory.max</code>. The container-level limits stayed in place.</li>
<li>A neighbour pod kept both vCPUs busy. Next to it, the BestEffort-classified pod got a median of 0.061 to 0.083 cores on the three affected versions. The same pod without the empty block got 0.573 to 0.580 (so 10.6% to 14.3%). On 1.37.1 the two were equal.</li>
<li>A <code>ResourceQuota</code> with scope <code>BestEffort</code> and <code>pods: "0"</code> rejected the pod on 1.34 to 1.36 and admitted it on 1.37.</li>
<li>The kubelet's memory-pressure eviction order was the same on all four versions, in all 12 repetitions: the true BestEffort pod first, then by memory usage relative to the container request. The empty block did not move a pod forward.</li>
<li>If you run GitLab Runner on Kubernetes, upgrade to 19.5.0 or later. As of 2 November 2026 the latest Kubernetes patch releases for 1.34, 1.35 and 1.36 still have the bug.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A cluster on Kubernetes 1.34, 1.35 or 1.36 (<code>kubectl version</code>), or a machine where you can run the lab</li>
<li><code>kubectl</code> and <code>jq</code></li>
<li>If you run GitLab Runner with the Kubernetes executor: its version (<code>gitlab-runner --version</code>)</li>
<li>To run the lab: a <a href="https://m.do.co/c/2a9bba940f39" rel="sponsored noopener noreferrer">DigitalOcean</a> token (referral link); one <code>s-2vcpu-4gb</code> droplet for about an hour costs a few cents</li>
</ul>
<h2>Where the empty block comes from</h2><p><strong>Pod-level resources</strong> (<a href="https://github.com/kubernetes/enhancements/issues/2837" rel="noopener noreferrer">KEP-2837</a>) let you set CPU and memory for the pod as a whole, not only per container. The feature gate <code>PodLevelResources</code> went alpha in 1.32 and beta, on by default, in 1.34. <a href="https://devops-daily.com/posts/kubernetes-1-36-user-namespaces-pod-resize">Our Kubernetes 1.36 post</a> covers the in-place resize of those values.</p>
<p>The QoS calculation has a shortcut for them. This is <code>ComputePodQOS</code> at v1.34.0, v1.35.0 and v1.36.0, and still at the latest patches, v1.34.13, v1.35.10 and v1.36.6:</p>
<pre><code class="hljs language-go" tabindex="0"><span class="hljs-comment">// pkg/apis/core/v1/helper/qos/qos.go, v1.36.6</span>
<span class="hljs-comment">// When pod-level resources are specified, we use them to determine QoS class.</span>
<span class="hljs-keyword">if</span> utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources) &amp;&amp;
    pod.Spec.Resources != <span class="hljs-literal">nil</span> {
</code></pre><p>An empty block is not nil. It has no requests and no limits, so the function returns BestEffort. The loop over the containers never runs. The issue that reported this, <a href="https://github.com/kubernetes/kubernetes/issues/135082" rel="noopener noreferrer">#135082</a> from November 2025, says that almost everywhere else Kubernetes does not differentiate between nil and empty pod-level resources. The helper for that, <code>IsPodLevelResourcesSet</code>, treats <code>{}</code> as not set.</p>
<p>The fix is <a href="https://github.com/kubernetes/kubernetes/pull/137150" rel="noopener noreferrer">PR #137150</a>, merged on 15 July 2026 for 1.37. It adds the condition <code>IsPodLevelResourcesSet(pod)</code> behind a new gate, <code>PodLevelResourcesFixKubeletQOSClass</code>, beta and on by default from 1.37. It was not backported. The patch releases of 27 October 2026, v1.34.13, v1.35.10 and v1.36.6, still have the old condition. The rest of that release is in <a href="https://devops-daily.com/posts/kubernetes-1-37-garhwal-what-shipped">our Kubernetes 1.37 review</a>; <code>PodLevelResources</code> itself stays beta in 1.37.</p>
<p>GitLab Runner added pod-level resources to its job pods in 18.10.0 (<a href="https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/5922" rel="noopener noreferrer">MR !5922</a>). At every tag we read (18.10.0, 19.3.5 and 19.4.3), the helper that builds the block returns a non-nil object:</p>
<pre><code class="hljs language-go" tabindex="0"><span class="hljs-comment">// executors/kubernetes/kubernetes.go, GitLab Runner v19.4.3</span>
<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(s *executor)</span></span> podResourcesReference() *api.ResourceRequirements {
	resources := api.ResourceRequirements{}

	<span class="hljs-keyword">if</span> s.configurationOverwrites != <span class="hljs-literal">nil</span> {
		resources.Limits = s.configurationOverwrites.podLimits
		resources.Requests = s.configurationOverwrites.podRequests
	}

	<span class="hljs-keyword">return</span> &amp;resources
}
</code></pre><p>If the runner config sets none of <code>pod_cpu_request</code>, <code>pod_memory_request</code>, <code>pod_cpu_limit</code> and <code>pod_memory_limit</code>, both maps are empty. The pod then goes to the API server with <code>resources: {}</code>. The MR that fixed it, <a href="https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7346" rel="noopener noreferrer">!7346</a>, says the same: job pods "were showing <code>qosClass: BestEffort</code> instead of the expected <code>Burstable</code>", and after the fix <code>.spec.resources</code> "should now be empty/absent (was <code>{}</code>)". It merged on 1 October 2026 and shipped in 19.5.0. The 19.4.3 and 19.3.5 tags of 28 October 2026 still have the old function. The class matters only when the runner config gives the job containers requests or limits. Without them, the pods were BestEffort anyway.</p>
<h2>The lab</h2><p>The lab runs on one DigitalOcean droplet: <code>s-2vcpu-4gb</code> in <code>fra1</code>, Ubuntu 24.04, cgroup v2. A cloud firewall allows SSH from one address and nothing else. The Kubernetes API port is never exposed, and <code>kubectl</code> runs on the node over SSH. For each version, a script removes k3s, installs the pinned release, and runs five stages. The kubelet gets an explicit hard eviction threshold, <code>memory.available&lt;2Gi</code>. It also gets a 30 second pressure transition period instead of the default 5 minutes, so repetitions do not wait for the node condition to clear.</p>
<ol>
<li><strong>pod spec</strong> resources: {} + container requests</li>
<li><strong>ComputePodQOS</strong> sees a block: BestEffort</li>
<li><strong>IsPodLevelResourcesSet</strong> sees {}: not set</li>
<li><strong>oom_score_adj</strong> 1000</li>
<li><strong>cgroup slice</strong> besteffort, cpu.weight 1</li>
<li><strong>ResourceQuota scope</strong> BestEffort</li>
<li><strong>eviction ranking</strong> container requests</li>
</ol>
<p>Connections:</p>
<ul>
<li>pod spec -&gt; ComputePodQOS</li>
<li>pod spec -&gt; IsPodLevelResourcesSet</li>
<li>ComputePodQOS -&gt; oom_score_adj</li>
<li>ComputePodQOS -&gt; cgroup slice</li>
<li>ComputePodQOS -&gt; ResourceQuota scope</li>
<li>IsPodLevelResourcesSet -&gt; eviction ranking</li>
</ul>
<p>The pods:</p>
<table>
<thead>
<tr>
<th>Pod</th>
<th>Pod-level block</th>
<th>Container requests</th>
<th>Container limits</th>
</tr>
</thead>
<tbody><tr>
<td><code>be</code></td>
<td>none</td>
<td>none</td>
<td>none</td>
</tr>
<tr>
<td><code>ctl-burstable</code></td>
<td>none</td>
<td>100m CPU, 256Mi</td>
<td>500m CPU, 512Mi</td>
</tr>
<tr>
<td><code>emp-burstable</code></td>
<td><code>{}</code></td>
<td>100m CPU, 256Mi</td>
<td>500m CPU, 512Mi</td>
</tr>
<tr>
<td><code>ctl-guaranteed</code></td>
<td>none</td>
<td>200m CPU, 256Mi</td>
<td>200m CPU, 256Mi</td>
</tr>
<tr>
<td><code>emp-guaranteed</code></td>
<td><code>{}</code></td>
<td>200m CPU, 256Mi</td>
<td>200m CPU, 256Mi</td>
</tr>
<tr>
<td><code>plr-burstable</code></td>
<td>requests 100m/256Mi, limits 500m/512Mi</td>
<td>none</td>
<td>none</td>
</tr>
</tbody></table>
<p><code>plr-burstable</code> is the control for the feature itself: a real pod-level block works as documented on every version. The empty block is the only problem.</p>
<p>We wrote down the claim and what would disprove it before the droplet existed, and committed it first (<code>CLAIM.md</code>). Every recorded file carries a hash of the manifests and node scripts, and the summary script refuses to mix files with different hashes.</p>
<p><a href="https://github.com/The-DevOps-Daily/k8s-empty-pod-resources" rel="noopener noreferrer">The-DevOps-Daily/k8s-empty-pod-resources on GitHub</a></p>
<h2>Finding 1: the class flips on 1.34 to 1.36</h2><p>The API server stored the empty block exactly as written. On v1.34.12, v1.35.9 and v1.36.5 the kubelet then reported BestEffort for both <code>emp-*</code> pods, while their controls kept Burstable and Guaranteed. On v1.37.1 the <code>emp-*</code> pods got the same class as their controls:</p>
<p><strong>results/demo-k3s-v1.36.5.txt and results/demo-k3s-v1.37.1.txt</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># k3s v1.36.5+k3s1</span>
<span class="hljs-comment"># ... kubectl apply and wait omitted; the whole session is in the file</span>
$ kubectl get pods -n demo -o custom-columns=<span class="hljs-string">'NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'</span>
NAME             POD-LEVEL   QOS
ctl-burstable    &lt;none&gt;      Burstable
ctl-guaranteed   &lt;none&gt;      Guaranteed
emp-burstable    map[]       BestEffort
emp-guaranteed   map[]       BestEffort
<span class="hljs-comment"># k3s v1.37.1+k3s1, same manifests</span>
<span class="hljs-comment"># ... kubectl apply and wait omitted</span>
$ kubectl get pods -n demo -o custom-columns=<span class="hljs-string">'NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'</span>
NAME             POD-LEVEL   QOS
ctl-burstable    &lt;none&gt;      Burstable
ctl-guaranteed   &lt;none&gt;      Guaranteed
emp-burstable    map[]       Burstable
emp-guaranteed   map[]       Guaranteed
</code></pre><p>The same result on every version, from <code>results/summary.md</code>:</p>
<table>
<thead>
<tr>
<th>Pod</th>
<th>v1.34.12</th>
<th>v1.35.9</th>
<th>v1.36.5</th>
<th>v1.37.1</th>
<th>v1.37.1, gate off</th>
</tr>
</thead>
<tbody><tr>
<td><code>ctl-burstable</code></td>
<td>Burstable</td>
<td>Burstable</td>
<td>Burstable</td>
<td>Burstable</td>
<td>Burstable</td>
</tr>
<tr>
<td><code>emp-burstable</code></td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>Burstable</td>
<td>BestEffort</td>
</tr>
<tr>
<td><code>ctl-guaranteed</code></td>
<td>Guaranteed</td>
<td>Guaranteed</td>
<td>Guaranteed</td>
<td>Guaranteed</td>
<td>Guaranteed</td>
</tr>
<tr>
<td><code>emp-guaranteed</code></td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>Guaranteed</td>
<td>BestEffort</td>
</tr>
<tr>
<td><code>plr-burstable</code></td>
<td>Burstable</td>
<td>Burstable</td>
<td>Burstable</td>
<td>Burstable</td>
<td>Burstable</td>
</tr>
<tr>
<td><code>be</code></td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>BestEffort</td>
<td>BestEffort</td>
</tr>
</tbody></table>
<p><code>plr-burstable</code>, with real pod-level requests and limits, was Burstable everywhere. Pod-level resources work. Only the empty block breaks the class.</p>
<p>To check that the 1.37 gate is the fix and not something else in the release, we reinstalled v1.37.1 with <code>PodLevelResourcesFixKubeletQOSClass=false</code> on both the API server and the kubelet. Both <code>emp-*</code> pods went back to BestEffort, with <code>oom_score_adj</code> 1000. The API server and kubelet metrics reported the gate as <code>BETA</code>, on by default and off in that run.</p>
<h2>Finding 2: the kernel sees a BestEffort pod</h2><p>QoS is not only a label. The kubelet turns it into numbers that the kernel uses. We read them from <code>/proc</code> and <code>/sys/fs/cgroup</code> on the node, for the main process of each container:</p>
<p><strong>oom_score_adj of the main container process</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>ctl-burstable, every version</td>
<td>935</td>
<td>no pod-level block</td>
</tr>
<tr>
<td>emp-burstable, 1.34 to 1.36</td>
<td>1000</td>
<td>resources: {}, 1.34 to 1.36</td>
</tr>
<tr>
<td>emp-burstable, 1.37.1</td>
<td>935</td>
<td>resources: {}, 1.37.1</td>
</tr>
<tr>
<td>ctl-guaranteed, every version</td>
<td>-997</td>
<td>no pod-level block</td>
</tr>
<tr>
<td>emp-guaranteed, 1.34 to 1.36</td>
<td>1000</td>
<td>resources: {}, 1.34 to 1.36</td>
</tr>
<tr>
<td>emp-guaranteed, 1.37.1</td>
<td>-997</td>
<td>resources: {}, 1.37.1</td>
</tr>
</tbody></table>
<p><em>Read from /proc//oom_score_adj on the node. The 1.34.12, 1.35.9 and 1.36.5 values are identical. The kernel adds this value to a process's badness score when it picks one to kill. Source: results/summary.md.</em></p>
<p>The kernel adds <code>oom_score_adj</code> to each process's badness score before it picks a process to kill, and it accepts values from -1000 to +1000 (<a href="https://github.com/torvalds/linux/blob/v6.8/Documentation/filesystems/proc.rst" rel="noopener noreferrer">proc.rst</a>). A Burstable container gets <code>1000 - 1000 * request / node memory</code>, which is 935 for a 256Mi request on this node. A Guaranteed container gets -997. The empty block gave both 1000, the same as a pod with no resources at all. <code>oom_score</code>, the current OOM killer score the kernel shows for the process, went from 1290 (Burstable) and 4 (Guaranteed) to 1334.</p>
<p>The cgroup changed too. On v1.36.5:</p>
<table>
<thead>
<tr>
<th></th>
<th><code>ctl-burstable</code></th>
<th><code>emp-burstable</code></th>
<th><code>ctl-guaranteed</code></th>
<th><code>emp-guaranteed</code></th>
</tr>
</thead>
<tbody><tr>
<td>QoS slice</td>
<td><code>kubepods-burstable.slice</code></td>
<td><code>kubepods-besteffort.slice</code></td>
<td><code>kubepods.slice</code></td>
<td><code>kubepods-besteffort.slice</code></td>
</tr>
<tr>
<td>pod <code>cpu.weight</code></td>
<td>4</td>
<td>1</td>
<td>8</td>
<td>1</td>
</tr>
<tr>
<td>pod <code>cpu.max</code></td>
<td>0.5 CPU</td>
<td>max (none)</td>
<td>0.2 CPU</td>
<td>max (none)</td>
</tr>
<tr>
<td>pod <code>memory.max</code></td>
<td>512 MiB</td>
<td>max (none)</td>
<td>256 MiB</td>
<td>max (none)</td>
</tr>
<tr>
<td>container <code>cpu.max</code></td>
<td>0.5 CPU</td>
<td>0.5 CPU</td>
<td>0.2 CPU</td>
<td>0.2 CPU</td>
</tr>
<tr>
<td>container <code>memory.max</code></td>
<td>512 MiB</td>
<td>512 MiB</td>
<td>256 MiB</td>
<td>256 MiB</td>
</tr>
<tr>
<td><code>oom_score</code></td>
<td>1290</td>
<td>1334</td>
<td>4</td>
<td>1334</td>
</tr>
</tbody></table>
<p>The container cgroup still has its own limits, so <code>emp-burstable</code> cannot use more than 512 MiB or half a CPU. What it loses is everything the kubelet sets at pod and QoS level. The pod cgroup has no memory limit and no CPU quota of its own. It sits in <code>kubepods-besteffort.slice</code>, where the kubelet sets <code>cpu.weight</code> to 1 for the whole slice.</p>
<h2>Finding 3: half a CPU requested, 0.06 to 0.08 cores received</h2><p><code>cpu.weight</code> 1 matters only when the CPU is busy. To see how much, we ran <code>cpu-neighbour</code>, a Burstable pod that requests 1 CPU and runs two busy processes, so both vCPUs are always taken. Next to it, <code>cpu-ctl</code> and <code>cpu-emp</code> took turns, three times each, alternating which went first. Both request 500m CPU with a 1 CPU limit and run one busy process; <code>cpu-emp</code> adds the empty block. Each pod reads its own cgroup's <code>usage_usec</code> over 20 seconds after a 5 second warm-up.</p>
<p><strong>CPU used next to a busy neighbour, median of 3 runs</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>v1.34.12 cpu-ctl</td>
<td>0.579 cores</td>
<td>cpu-ctl (no block)</td>
</tr>
<tr>
<td>v1.34.12 cpu-emp</td>
<td>0.083 cores</td>
<td>cpu-emp (resources: {})</td>
</tr>
<tr>
<td>v1.35.9 cpu-ctl</td>
<td>0.58 cores</td>
<td>cpu-ctl (no block)</td>
</tr>
<tr>
<td>v1.35.9 cpu-emp</td>
<td>0.07 cores</td>
<td>cpu-emp (resources: {})</td>
</tr>
<tr>
<td>v1.36.5 cpu-ctl</td>
<td>0.573 cores</td>
<td>cpu-ctl (no block)</td>
</tr>
<tr>
<td>v1.36.5 cpu-emp</td>
<td>0.061 cores</td>
<td>cpu-emp (resources: {})</td>
</tr>
<tr>
<td>v1.37.1 cpu-ctl</td>
<td>0.576 cores</td>
<td>cpu-ctl (no block)</td>
</tr>
<tr>
<td>v1.37.1 cpu-emp</td>
<td>0.577 cores</td>
<td>cpu-emp (resources: {})</td>
</tr>
</tbody></table>
<p><em>2 vCPU droplet. Both pods request 500m with a 1 CPU limit and run one busy process; the neighbour requests 1 CPU and runs two. Each pod measured its own cgroup usage_usec over 20 s. Source: results/summary.md.</em></p>
<p>On the three affected versions, <code>cpu-emp</code> got a median of 0.083, 0.070 and 0.061 cores, against 0.579, 0.580 and 0.573 for <code>cpu-ctl</code>. That is 14.3%, 12.1% and 10.6% of the control's CPU, or 7.0 to 9.4 times less. On v1.37.1 the two were the same: 0.576 and 0.577 cores.</p>
<p>The weights explain the direction. <code>cpu-ctl</code>'s pod cgroup has <code>cpu.weight</code> 20 for its 500m request and sits in the burstable slice next to the neighbour. The kubelet sets the burstable slice's weight from the sum of the CPU requests of the Burstable pods in it, which gave 63 here. <code>cpu-emp</code>'s pod cgroup has <code>cpu.weight</code> 1 and sits in the besteffort slice. The kubelet always sets that slice to the minimum, <code>cpu.weight</code> 1. The 500m request of <code>cpu-emp</code> still sets a weight on its container cgroup, but that only counts against other containers in the same pod. On a busy node like this one, a CI build in the BestEffort slice competes with <code>cpu.weight</code> 1 at that level and can get far less CPU than Burstable or Guaranteed pods. We did not run a build.</p>
<p>The runs were short and sequential. Each pod ran three times in a fixed alternating order, and we did not record the neighbour's own usage or other load on the host. On the affected versions, each of the nine <code>cpu-emp</code> runs got less CPU than every <code>cpu-ctl</code> run, including the runs where <code>cpu-emp</code> went first. The exact percentages still belong to this setup.</p>
<h2>Finding 4: a "no BestEffort" quota rejects the pod</h2><p>Some clusters forbid BestEffort pods with a quota like this one, so every workload must declare resources:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">v1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">ResourceQuota</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-literal">no</span><span class="hljs-string">-besteffort</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">hard:</span>
    <span class="hljs-attr">pods:</span> <span class="hljs-string">'0'</span>
  <span class="hljs-attr">scopes:</span> [<span class="hljs-string">'BestEffort'</span>]
</code></pre><p>The API server applies the scope at create time, before any kubelet sees the pod, and it calls the same <code>GetPodQOS</code> function. With no status yet, that function computes the class from the spec. On 1.34 to 1.36 it rejected <code>emp-burstable</code>, a pod whose container sets requests and limits:</p>
<p><strong>quota.py on k3s v1.36.5</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ grep -A3 <span class="hljs-string">' quota$'</span> results/k3s-v1.36.5/run.log
2026-11-02T10:33:20Z quota
be              admitted=False Error from server (Forbidden): error when creating <span class="hljs-string">"/root/k8s-empty-pod-resources/manifests/quota/be.yaml"</span>: pods <span class="hljs-string">"be"</span> is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0
ctl-burstable   admitted=True  pod/ctl-burstable created
emp-burstable   admitted=False Error from server (Forbidden): error when creating <span class="hljs-string">"/root/k8s-empty-pod-resources/manifests/quota/emp-burstable.yaml"</span>: pods <span class="hljs-string">"emp-burstable"</span> is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0
</code></pre><p>On v1.37.1 the API server admitted <code>emp-burstable</code>, and still rejected <code>be</code>. We did not run a runner. But a job pod from 18.10.0 to 19.4.x carries the same empty block, so in a namespace with this quota it should fail the same way on 1.34 to 1.36. The fix discussion names the reverse risk: a pod "could theoretically bypass restricted BestEffort quotas by being classified as Guaranteed during admission, while still behaving as BestEffort at the node level". That can happen when the API server has the fix and the kubelet does not.</p>
<h2>Finding 5: the eviction order did not change</h2><p>The five eviction pods each allocate a fixed amount of memory and then sleep. They are sized so that the two possible rules predict different orders:</p>
<table>
<thead>
<tr>
<th>Pod</th>
<th>Memory request</th>
<th>Allocates</th>
<th>Working set before the hog</th>
<th>Usage against request</th>
</tr>
</thead>
<tbody><tr>
<td><code>be</code></td>
<td>none</td>
<td>50 MiB</td>
<td>54 MiB</td>
<td>above (request is 0)</td>
</tr>
<tr>
<td><code>ctl</code></td>
<td>256Mi</td>
<td>200 MiB</td>
<td>204 MiB</td>
<td>about 52 MiB below</td>
</tr>
<tr>
<td><code>gua-ctl</code></td>
<td>256Mi (= limit)</td>
<td>150 MiB</td>
<td>154 MiB</td>
<td>about 102 MiB below</td>
</tr>
<tr>
<td><code>emp</code> + <code>{}</code></td>
<td>256Mi</td>
<td>100 MiB</td>
<td>104 MiB</td>
<td>about 152 MiB below</td>
</tr>
<tr>
<td><code>gua-emp</code> + <code>{}</code></td>
<td>256Mi (= limit)</td>
<td>50 MiB</td>
<td>54 MiB</td>
<td>about 202 MiB below</td>
</tr>
</tbody></table>
<p>If the kubelet ranked by QoS class, <code>emp</code> and <code>gua-emp</code> would go out together with <code>be</code>, before <code>ctl</code> and <code>gua-ctl</code>. If it ranks by container requests, <code>be</code> goes first because any usage exceeds a request of zero, and the rest follow in order of usage minus request: <code>ctl</code>, <code>gua-ctl</code>, <code>emp</code>, <code>gua-emp</code>.</p>
<p>A host process outside every pod, <code>hog.py</code>, then holds <code>MemAvailable</code> at 1000 MiB. That is below the 2Gi threshold even after all five pods are gone, so the kubelet keeps evicting until nothing evictable is left. The kubelet cannot evict the hog. The pods differ in size on purpose, to separate the two models, so this is not a matched pair. CoreDNS also ran on the node. The kubelet ranked it last in every ranking it logged. In 5 of the 12 logs the window ran long enough to show the next step: the kubelet logged <code>cannot evict a critical pod</code> once only CoreDNS was left.</p>
<p>Every repetition on every version gave the request order. This is the kubelet's own ranking on v1.36.5, while it reported <code>emp</code> and <code>gua-emp</code> as BestEffort:</p>
<p><strong>k3s v1.36.5, repetition 1</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># the class the kubelet reported for each pod</span>
$ jq -c <span class="hljs-string">'select(.rep == 1) | .qos'</span> results/k3s-v1.36.5/40-pressure.jsonl
{<span class="hljs-string">"be"</span>:<span class="hljs-string">"BestEffort"</span>,<span class="hljs-string">"ctl"</span>:<span class="hljs-string">"Burstable"</span>,<span class="hljs-string">"emp"</span>:<span class="hljs-string">"BestEffort"</span>,<span class="hljs-string">"gua-ctl"</span>:<span class="hljs-string">"Guaranteed"</span>,<span class="hljs-string">"gua-emp"</span>:<span class="hljs-string">"BestEffort"</span>}
<span class="hljs-comment"># the kubelet's ranking before each eviction; sed keeps the journald timestamp and the message, and drops the host name, the service PID, the severity letter, the kubelet's own timestamp, the second PID and the source file</span>
$ grep -E <span class="hljs-string">'ranked for eviction|evicted successfully'</span> results/k3s-v1.36.5/40-kubelet-rep1.log | sed -E <span class="hljs-string">'s/^([^ ]+) .*eviction_manager.go:[0-9]+\] /\1 /'</span>
2026-11-02T10:38:17.816980+00:00 <span class="hljs-string">"Eviction manager: pods ranked for eviction"</span> pods=[<span class="hljs-string">"evict-check/be"</span>,<span class="hljs-string">"evict-check/ctl"</span>,<span class="hljs-string">"evict-check/gua-ctl"</span>,<span class="hljs-string">"evict-check/emp"</span>,<span class="hljs-string">"evict-check/gua-emp"</span>,<span class="hljs-string">"kube-system/coredns-7cfb7bc9c7-svb26"</span>]
2026-11-02T10:38:20.108268+00:00 <span class="hljs-string">"Eviction manager: pod is evicted successfully"</span> pod=<span class="hljs-string">"evict-check/be"</span>
2026-11-02T10:38:21.127785+00:00 <span class="hljs-string">"Eviction manager: pods ranked for eviction"</span> pods=[<span class="hljs-string">"evict-check/ctl"</span>,<span class="hljs-string">"evict-check/gua-ctl"</span>,<span class="hljs-string">"evict-check/emp"</span>,<span class="hljs-string">"evict-check/gua-emp"</span>,<span class="hljs-string">"kube-system/coredns-7cfb7bc9c7-svb26"</span>]
2026-11-02T10:38:23.555889+00:00 <span class="hljs-string">"Eviction manager: pod is evicted successfully"</span> pod=<span class="hljs-string">"evict-check/ctl"</span>
2026-11-02T10:38:24.579473+00:00 <span class="hljs-string">"Eviction manager: pods ranked for eviction"</span> pods=[<span class="hljs-string">"evict-check/gua-ctl"</span>,<span class="hljs-string">"evict-check/emp"</span>,<span class="hljs-string">"evict-check/gua-emp"</span>,<span class="hljs-string">"kube-system/coredns-7cfb7bc9c7-svb26"</span>]
2026-11-02T10:38:27.242735+00:00 <span class="hljs-string">"Eviction manager: pod is evicted successfully"</span> pod=<span class="hljs-string">"evict-check/gua-ctl"</span>
2026-11-02T10:38:28.259762+00:00 <span class="hljs-string">"Eviction manager: pods ranked for eviction"</span> pods=[<span class="hljs-string">"evict-check/emp"</span>,<span class="hljs-string">"evict-check/gua-emp"</span>,<span class="hljs-string">"kube-system/coredns-7cfb7bc9c7-svb26"</span>]
2026-11-02T10:38:30.555840+00:00 <span class="hljs-string">"Eviction manager: pod is evicted successfully"</span> pod=<span class="hljs-string">"evict-check/emp"</span>
2026-11-02T10:38:31.572048+00:00 <span class="hljs-string">"Eviction manager: pods ranked for eviction"</span> pods=[<span class="hljs-string">"evict-check/gua-emp"</span>,<span class="hljs-string">"kube-system/coredns-7cfb7bc9c7-svb26"</span>]
2026-11-02T10:38:34.078890+00:00 <span class="hljs-string">"Eviction manager: pod is evicted successfully"</span> pod=<span class="hljs-string">"evict-check/gua-emp"</span>
</code></pre><p>The source explains it. <code>rankMemoryPressure</code> sorts by three keys. First, whether usage exceeds the memory request. Then pod priority. Then usage minus the memory request. The first and third keys get the request from <code>GetResourceRequestQuantity</code>. For an empty block, that helper asks <code>IsPodLevelResourcesSet</code>, gets false, and uses the container requests. Priority does not read requests, and the QoS class is not an input at all. If the kubelet had used a request of zero for <code>emp</code>, <code>emp</code> would have been in the first group with <code>be</code>. It came fourth.</p>
<p><strong>When each pod was evicted, 12 repetitions</strong></p>
<table>
<thead>
<tr>
<th>Series</th>
<th>Samples</th>
<th>Min</th>
<th>Median</th>
<th>p95</th>
<th>Max</th>
</tr>
</thead>
<tbody><tr>
<td>be</td>
<td>12</td>
<td>5.6s</td>
<td>8.350000000000001s</td>
<td>11.5s</td>
<td>11.5s</td>
</tr>
<tr>
<td>ctl</td>
<td>12</td>
<td>9.1s</td>
<td>11.95s</td>
<td>15.3s</td>
<td>15.3s</td>
</tr>
<tr>
<td>gua-ctl</td>
<td>12</td>
<td>12.7s</td>
<td>16s</td>
<td>20.9s</td>
<td>20.9s</td>
</tr>
<tr>
<td>emp</td>
<td>12</td>
<td>16.1s</td>
<td>20.15s</td>
<td>24.7s</td>
<td>24.7s</td>
</tr>
<tr>
<td>gua-emp</td>
<td>12</td>
<td>19.5s</td>
<td>23.6s</td>
<td>29.9s</td>
<td>29.9s</td>
</tr>
</tbody></table>
<p><em>Seconds after the host hog started, from the kubelet's log. 3 repetitions each on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. Every repetition evicted the pods in the same order. Source: results/summary.md.</em></p>
<p>In all 12 repetitions the kubelet started to reclaim 3.0 to 9.2 s after the hog started, and evicted the first pod 2.3 to 3.3 s later. After that it evicted one pod at a time. 41 of the 48 gaps between evictions were 4.0 s or less (median 3.6 s, longest 8.2 s). The last pod was gone 19.5 to 29.9 s after the hog started. It does not wait 10 seconds between evictions: after each one it waits for the pod to be cleaned up and then checks again. Its own <code>available</code> figure at the first eviction was 931 to 1212 MiB, well under the 2Gi threshold and far from empty. The kernel logged no OOM kill in any repetition, so the order is the kubelet's alone.</p>
<p>The two mechanisms are easy to mix up. The kubelet's eviction manager checks every 10 seconds by default when it has nothing to evict, and it ignores the class. The kernel OOM killer acts when memory runs out before the kubelet frees any. It adds <code>oom_score_adj</code> to each process's badness score, and the empty block gives the pod the same 1000 as a pod with no resources at all (Finding 2). We did not trigger a kernel OOM, so we did not watch the kernel make that choice.</p>
<h2>What we got wrong before measuring</h2><p>The research note we started from had the core right and two details wrong.</p>
<ul>
<li><strong>Right:</strong> Kubernetes 1.34 to 1.36 compute BestEffort for an empty pod-level block. GitLab Runner 18.10 to 19.4 build their job pods with that block (from the source and the fix MR). On 1.34 to 1.36 the pods get <code>oom_score_adj</code> 1000. 1.37 fixes the class while its new gate is on, which is the default.</li>
<li><strong>Wrong:</strong> it called #137150 the upstream issue. That is the fix PR. The issue is #135082.</li>
<li><strong>Wrong, and it matters:</strong> it said these pods are first in line for kubelet eviction under memory pressure. Finding 5 shows they are not.</li>
<li><strong>Missing:</strong> the 1.37 fix is a beta feature gate, not a plain code change, and nothing was backported to the 1.34, 1.35 or 1.36 patch releases.</li>
</ul>
<p>We got one prediction wrong ourselves. <code>CLAIM.md</code> said the BestEffort-classified pod would get under 10% of the control's CPU. It got 10.6% to 14.3%. Our falsification line was 50%, so the claim stands, but the number we wrote down was too pessimistic, and we report it as a miss.</p>
<p>We also kept a one-repetition smoke run on v1.36.5 (<code>results/_smoke-k3s-v1.36.5/</code>). It used the same code stamp and gave the same results, but the summary does not count it.</p>
<h2>What to do</h2><ol>
<li><p><strong>Find the pods.</strong> This query lists every pod with an empty pod-level block and the class it got:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl get pods -A -o json \
  | jq -r <span class="hljs-string">'.items[] | select(.spec.resources == {}) | [.metadata.namespace, .metadata.name, .status.qosClass] | @tsv'</span>
</code></pre></li>
<li><p><strong>GitLab Runner 18.10.0 to 19.4.x: upgrade to 19.5.0 or later.</strong> The job pods then go out without a pod-level block, and the class comes from the containers again.</p>
</li>
<li><p><strong>If you cannot upgrade the runner yet, give the block real values.</strong> Set <code>pod_cpu_request</code> and <code>pod_memory_request</code> (and the <code>_limit</code> keys if you want limits) under <code>[runners.kubernetes]</code>. The QoS class then comes from those values, like <code>plr-burstable</code> above. This is a behaviour change, not only a workaround: the pod cgroup then takes its limits from the pod-level values too, as it did for <code>plr-burstable</code>.</p>
</li>
<li><p><strong>Kubernetes 1.37 fixes it by default.</strong> The fix sits behind a beta gate, so check that nobody turned <code>PodLevelResourcesFixKubeletQOSClass</code> off.</p>
</li>
<li><p><strong>Check your own templates.</strong> A Helm chart that renders a pod-level block from a value that defaults to <code>{}</code> produces the same bug. Render the key only when it has content:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># wrong: renders "resources: {}" at pod level when the value is empty</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">resources:</span> {{<span class="hljs-bullet">-</span> <span class="hljs-string">toYaml</span> <span class="hljs-string">.Values.podResources</span> <span class="hljs-string">|</span> <span class="hljs-string">nindent</span> <span class="hljs-number">4</span> }}

<span class="hljs-comment"># right: no key at all when the value is empty</span>
<span class="hljs-attr">spec:</span>
  {{<span class="hljs-bullet">-</span> <span class="hljs-string">with</span> <span class="hljs-string">.Values.podResources</span> }}
  <span class="hljs-attr">resources:</span> {{<span class="hljs-bullet">-</span> <span class="hljs-string">toYaml</span> <span class="hljs-string">.</span> <span class="hljs-string">|</span> <span class="hljs-string">nindent</span> <span class="hljs-number">4</span> }}
  {{<span class="hljs-bullet">-</span> <span class="hljs-string">end</span> }}
</code></pre></li>
</ol>
<p>An empty <code>resources: {}</code> inside a container is harmless. Only the pod-level one under <code>spec</code> changes the class.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>k3s is one distribution.</strong> We ran upstream Kubernetes code as k3s packages it. A managed service can turn the <code>PodLevelResources</code> gate off or carry its own patches. Run the <code>jq</code> query on your own cluster.</li>
<li><strong>No GitLab instance and no runner ran here.</strong> The runner part rests on its source at each tag and on the MR description. The <code>emp-*</code> pods reproduce the field the runner sends, not a full job pod with helper and service containers.</li>
<li><strong>We did not trigger a kernel OOM kill.</strong> The <code>oom_score</code> values are the numbers the kernel uses to choose a victim, read from <code>/proc</code>. We did not watch it choose.</li>
<li><strong>The eviction timing is ours.</strong> It depends on the 2Gi threshold, the hog's 1000 MiB target and how fast it allocates. The order is the finding, not the seconds.</li>
<li><strong>The CPU gap depends on the neighbours.</strong> One neighbour with a 1 CPU request on a 2 vCPU node gave the numbers above. More Burstable or Guaranteed load should make the BestEffort slice's share smaller. On an idle node the weights do not matter, because they only divide contended CPU time; we did not measure that case.</li>
<li><strong>One node, one admission check.</strong> We did not test scheduling across nodes, or any admission rule other than the one quota. The gate-off run on 1.37.1 covered only the class, the kernel-facing numbers and the quota, not CPU or eviction.</li>
<li><strong>Other consumers of the class were not tested.</strong> From the source, the kubelet also rejects new BestEffort pods while the node has the <code>MemoryPressure</code> condition. The static CPU manager gives exclusive CPUs only to Guaranteed pods. We did not measure either.</li>
</ul>
<h2>Summary</h2><p>On Kubernetes 1.34, 1.35 and 1.36, with the <code>PodLevelResources</code> gate at its default (on), <code>spec.resources: {}</code> makes a pod BestEffort no matter what its containers request. On our node that meant <code>oom_score_adj</code> 1000 instead of 935 or -997, and a place in the BestEffort slice with <code>cpu.weight</code> 1 and no pod-level memory or CPU limit. Next to a busy neighbour the pod got about 7 to 9 times less CPU, and a quota that forbids BestEffort pods rejected it. The empty block did not change the kubelet's eviction order. The ranking uses memory requests, and for an empty block those still come from the containers: 12 of 12 repetitions evicted our pods in exactly the order the requests predict. Kubernetes 1.37 fixes the class behind a beta gate that is on by default, and GitLab Runner 19.5.0 stops sending the block. Run the <code>jq</code> query above on your clusters, and upgrade whichever side you can first.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Can a Vision Model Read Your Grafana Panel? 100 Panels on DigitalOcean Inference]]></title>
      <link>https://devops-daily.com/posts/vision-model-grafana-panel-digitalocean-inference</link>
      <description><![CDATA[We drew 100 Grafana-style panels with known answers and asked NVIDIA Nemotron 3.5 Super VL and three other models on DigitalOcean Serverless Inference to read four facts from each. The best model read 65 of 100 fully right, and log axes broke all of them. Here is what a screenshot costs, where the reads go wrong, and a small bot that checks the model against the alert before it posts.]]></description>
      <pubDate>Wed, 04 Nov 2026 07:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/vision-model-grafana-panel-digitalocean-inference</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[AI]]></category><category><![CDATA[LLM]]></category><category><![CDATA[Grafana]]></category><category><![CDATA[Observability]]></category><category><![CDATA[On-call]]></category><category><![CDATA[DigitalOcean]]></category>
      <content:encoded><![CDATA[<p>Grafana can attach a screenshot of the alerting panel to a notification. A bot can send that image to a vision model and post one line next to the page. NVIDIA Nemotron 3.5 Super VL came to DigitalOcean Serverless Inference on 15 October 2026 (<a href="https://docs.digitalocean.com/release-notes/" rel="noopener noreferrer">release notes</a>), so we tested it on that job.</p>
<p>We drew 100 dashboard panels with known answers and asked four questions about each: the peak value, the time of the peak, whether the data crossed the threshold, and which series is highest. Our claim, written before we measured: Nemotron 3.5 Super VL gets all four right on at least 90 of 100 panels, for under $0.001 per screenshot. It got all four right on <strong>48</strong> panels with reasoning off and <strong>54</strong> with reasoning on. The claim failed. The cost limit held only with reasoning off: <strong>$0.00026</strong> per screenshot, against <strong>$0.0024</strong> with reasoning on.</p>
<p>None of the four models we tested came close to 90. Qwen3.8-Max got 65 and Gemma 4 got 64, at <strong>$0.11 per 1,000 screenshots</strong> for Gemma. Log-scale axes broke every model.</p>
<p>So the numbers in an alert message should come from the alert payload, not the model. The more useful result is a small bot that lets the model describe the screenshot and lets rules decide whether to trust it. On Nemotron's 300 reads, the rules caught 63 percent of the wrong reads and blocked no correct read. With the wrong screenshot attached on purpose, the bot refused it 99 times in 100.</p>
<h2>TL;DR</h2><ul>
<li>All four facts right, of 100 Grafana-style panels: Qwen3.8-Max 65, Gemma 4 64, Nemotron 3.5 Super VL 54 with reasoning and 48 without, Llama 4 Maverick 21. Our bar was 90.</li>
<li>Peak time and threshold were easy (86 to 99 percent for every model). Peak value and top series were hard. On log axes, the best model read all four right on 8 of 25 panels.</li>
<li>Reasoning made Nemotron 13 times slower (19 seconds at the median, 82 at worst) and 9 times more expensive. The gain from 48 to 54 is inside the noise.</li>
<li>At temperature 0, Nemotron gave the same four answers in all three runs on 13 of 100 panels. Qwen3.8-Max did on 76.</li>
<li>A cross-check against the alert caught 63 percent of Nemotron's wrong reads (81 percent at a 5 percent tolerance). It cannot check the peak time, the series on a stacked panel, or the words in the summary.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A <a href="https://m.do.co/c/2a9bba940f39" rel="sponsored noopener noreferrer">DigitalOcean account</a> (referral link) with a Serverless Inference model access key</li>
<li>Node.js 22 or later</li>
<li>About $1 of inference if you rerun every model; the reasoning-mode run is most of it</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/grafana-panel-vision" rel="noopener noreferrer">The-DevOps-Daily/grafana-panel-vision on GitHub</a></p>
<h2>The job and the claim</h2><p>A bot that reads the attached panel could save the on-call engineer one click: "p99 went to 1.8 s at 13:55 UTC and stayed under the 2.5 s threshold". The question is whether you can trust that sentence during a night page.</p>
<p>Each model got the same prompt for every panel, with the panel as a PNG, and had to answer in JSON. This is a real answer from Nemotron 3.5 Super VL with reasoning off:</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"peak"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1.8</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"unit"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"s"</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"peakTime"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"13:45"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"thresholdCrossed"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"no"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"topSeries"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"search"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"summary"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Upstream response time stays under 1 second until midday, then spikes sharply to about 1.8 seconds in the early afternoon while remaining below the 2.5 s warning threshold."</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>The truth for that panel is 1.85 s at 13:55 UTC, not crossed, <code>search</code>, so this read passes. The tolerances, set before the first run:</p>
<ul>
<li><strong>Peak value</strong>: within 5 percent of the true peak, after unit conversion. A unit from the wrong family, such as MiB on a latency panel, is wrong. "MB" counts as "MiB", because the axis says MiB.</li>
<li><strong>Peak time</strong>: within 5 percent of the panel's time range. That is 3 minutes on a 1-hour panel, 18 on 6 hours and 72 on 24 hours.</li>
<li><strong>Threshold crossed</strong>: exact. The answer is "yes", "no" or "no threshold".</li>
<li><strong>Top series</strong>: the answer is matched to the nearest legend name, with at most 2 typing errors (1 for short names). A tie, or an answer that names two legend entries, counts as wrong. On a stacked panel, the top series is the one with the thickest band at the peak, not the band drawn on top, and the prompt says so.</li>
</ul>
<p>A panel counts as read only when all four facts are right.</p>
<h2>100 panels with known answers</h2><blockquote>
<p><strong>Warning</strong></p>
<p>These are <strong>Grafana-style panels drawn by our own renderer, not screenshots from Grafana</strong>. Real data has plateaus and near-equal peaks, which our generator rejects so that each question has one answer. Many real dashboards show a legend with Max or Last columns, which would turn the value question into text reading and help the models. A model trained on real Grafana screenshots could read real panels better or worse than ours. We do not know which.</p>
</blockquote>
<p>The renderer writes SVG and converts it to PNG with <code>@resvg/resvg-js</code>. It copies the Grafana dark theme: panel background, Inter font, classic palette, grid, axis labels with units, a list legend, and thresholds as a dashed line, a shaded region or both. There are no tooltips and no legend values. A seeded generator (mulberry32) draws the data, and the truth comes from the same numbers.</p>
<table>
<thead>
<tr>
<th>What varies</th>
<th>Panels</th>
</tr>
</thead>
<tbody><tr>
<td>Unit</td>
<td>ms 29, percent 22, req/s 23, bytes 26</td>
</tr>
<tr>
<td>Series per panel</td>
<td>1: 17, 2: 25, 3: 22, 4: 19, 5: 17</td>
</tr>
<tr>
<td>Log y-axis</td>
<td>25</td>
</tr>
<tr>
<td>Stacked areas</td>
<td>19</td>
</tr>
<tr>
<td>Similar colours (shades of one hue)</td>
<td>27</td>
</tr>
<tr>
<td>Small panel (420x230 or 360x200)</td>
<td>19</td>
</tr>
<tr>
<td>Threshold shown (line, region, both)</td>
<td>65 (28, 17, 20), crossed on 44</td>
</tr>
<tr>
<td>Peak at the right edge</td>
<td>28</td>
</tr>
<tr>
<td>Time range 1 h, 6 h, 24 h (UTC clock)</td>
<td>33, 36, 31</td>
</tr>
<tr>
<td>Noise low, medium, high</td>
<td>32, 42, 26</td>
</tr>
</tbody></table>
<p>In an earlier project, a bug in our own test data made strong models look wrong, so we checked the truth first. A second script reads every panel back from the SVG alone, the way a reader would: the peak from the tick labels, the time from the clock labels, and the top series from the line colours and the legend (on the 19 stacked panels, from the band heights at the peak). It agreed with the truth on all 100 panels, with a largest peak error of 0.011 percent. To prove the check can fail, a third script breaks the renderer in a copy: wrong byte labels gave 38 problems, lines drawn 3 percent too high gave 204, and stacked bands drawn in the wrong colours failed the band check on all 19 stacked panels. We also viewed 14 panels next to their truth records.</p>
<h2>Which models we tested</h2><p>A probe sent the same question to 15 models, with no image and then with the smallest and the largest panel. A model passed if it answered and its prompt token count rose with the image, which proves the image arrived. 11 passed. Mistral 3 14B returned "model does not support image input", and Kimi K2.6 rejected temperature 0. Our account gets HTTP 403, "not available for your subscription tier", for the GPT models, so this test has open-weight models only.</p>
<p>Several vision models reason before they answer, which is too slow for a page. On the two probe panels, Kimi K3 took 28 and 165 seconds, DeepSeek V4.1 Flash 45 and 13, and Qwen 3.5 397B 71 (with no answer) and 42. We kept <strong>Nemotron 3.5 Super VL</strong> (<code>nemotron-3.5-super-vl</code>), with reasoning off and on, and three models that answered in 1.5 to 3 seconds: <strong>Qwen3.8-Max</strong> (<code>qwen3.8-max</code>), <strong>Gemma 4</strong> (<code>gemma-4-31B-it</code>) and <strong>Llama 4 Maverick</strong> (<code>llama-4-maverick</code>). Nemotron Nano 12B v2 VL was also fast; we left it out to keep the test to four models.</p>
<p>Every call in the comparison used temperature 0 and at most 1,024 output tokens. Reasoning mode got 16,384, because the reasoning counts against that limit. Without reasoning, each model read every panel three times. Reasoning calls took about 20 seconds each, so that mode read all 100 panels once and a fixed 34-panel subset two more times.</p>
<p>Nemotron reasons by default. Two request settings turn it off. The <code>/no_think</code> system prompt does not:</p>
<p><strong>stage 0b: turning reasoning off on one 1000x500 panel</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/00b-reasoning-switch.mjs nemotron-3.5-super-vl
nemotron-3.5-super-vl | default                                      | HTTP 200 | 22795 ms | completion 4096 | reasoning chars 13191 | <span class="hljs-variable">$0</span>.00291
nemotron-3.5-super-vl | reasoning_effort=low                         | HTTP 200 | 22271 ms | completion 4096 | reasoning chars 14191 | <span class="hljs-variable">$0</span>.00291
nemotron-3.5-super-vl | reasoning_effort=none                        | HTTP 200 | 1207 ms | completion 99 | reasoning chars 0 | <span class="hljs-variable">$0</span>.00031
nemotron-3.5-super-vl | chat_template_kwargs.enable_thinking=<span class="hljs-literal">false</span>   | HTTP 200 | 1268 ms | completion 92 | reasoning chars 0 | <span class="hljs-variable">$0</span>.00031
nemotron-3.5-super-vl | system prompt /no_think                      | HTTP 200 | 21921 ms | completion 4096 | reasoning chars 14294 | <span class="hljs-variable">$0</span>.00291
</code></pre><p>That probe had a 4,096-token limit, and the reasoning runs used all of it without an answer. In the main run, none of the 168 reasoning calls hit the 16,384 limit. The longest used 11,979 tokens.</p>
<h2>Results</h2><table>
<thead>
<tr>
<th>Model</th>
<th>All four right (95% CI)</th>
<th>Peak value</th>
<th>Peak time</th>
<th>Threshold</th>
<th>Top series</th>
<th>$ per 1,000</th>
<th>Latency p50 / p95</th>
</tr>
</thead>
<tbody><tr>
<td>Qwen3.8-Max</td>
<td>65 (55 to 74)</td>
<td>77</td>
<td>97</td>
<td>97</td>
<td>91</td>
<td>1.47</td>
<td>1.5 s / 2.2 s</td>
</tr>
<tr>
<td>Gemma 4</td>
<td>64 (54 to 73)</td>
<td>77</td>
<td>99</td>
<td>96</td>
<td>86</td>
<td>0.11</td>
<td>2.7 s / 3.4 s</td>
</tr>
<tr>
<td>Nemotron 3.5 Super VL, reasoning on</td>
<td>54 (44 to 63)</td>
<td>71</td>
<td>94</td>
<td>96</td>
<td>80</td>
<td>2.37</td>
<td>19.0 s / 58.5 s</td>
</tr>
<tr>
<td>Nemotron 3.5 Super VL, reasoning off</td>
<td>48 (39 to 58)</td>
<td>69</td>
<td>96</td>
<td>91</td>
<td>75</td>
<td>0.26</td>
<td>1.4 s / 1.8 s</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>21 (14 to 30)</td>
<td>34</td>
<td>89</td>
<td>86</td>
<td>81</td>
<td>0.28</td>
<td>2.7 s / 9.1 s</td>
</tr>
</tbody></table>
<p>Accuracy and cost are from run 1. Latency covers every successful call in all runs, and it is the time of the attempt that succeeded. The parallel runs hit HTTP 429 often (retried attempts: 88 for Nemotron with reasoning off, 7 with reasoning on, 99 for Qwen3.8-Max, 75 for Gemma 4), and those waits are not in the latency figures.</p>
<p><strong>Panels with all four facts right, by panel feature (run 1)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>1 series</td>
<td>88.2%</td>
<td>Nemotron, reasoning off</td>
</tr>
<tr>
<td>1 series</td>
<td>88.2%</td>
<td>Qwen3.8-Max</td>
</tr>
<tr>
<td>1 series</td>
<td>76.5%</td>
<td>Gemma 4</td>
</tr>
<tr>
<td>stacked</td>
<td>31.6%</td>
<td>Nemotron, reasoning off</td>
</tr>
<tr>
<td>stacked</td>
<td>63.2%</td>
<td>Qwen3.8-Max</td>
</tr>
<tr>
<td>stacked</td>
<td>68.4%</td>
<td>Gemma 4</td>
</tr>
<tr>
<td>small panel</td>
<td>42.1%</td>
<td>Nemotron, reasoning off</td>
</tr>
<tr>
<td>small panel</td>
<td>63.2%</td>
<td>Qwen3.8-Max</td>
</tr>
<tr>
<td>small panel</td>
<td>57.9%</td>
<td>Gemma 4</td>
</tr>
<tr>
<td>similar colours</td>
<td>25.9%</td>
<td>Nemotron, reasoning off</td>
</tr>
<tr>
<td>similar colours</td>
<td>55.6%</td>
<td>Qwen3.8-Max</td>
</tr>
<tr>
<td>similar colours</td>
<td>48.1%</td>
<td>Gemma 4</td>
</tr>
<tr>
<td>4-5 series</td>
<td>30.6%</td>
<td>Nemotron, reasoning off</td>
</tr>
<tr>
<td>4-5 series</td>
<td>58.3%</td>
<td>Qwen3.8-Max</td>
</tr>
<tr>
<td>4-5 series</td>
<td>55.6%</td>
<td>Gemma 4</td>
</tr>
<tr>
<td>log y-axis</td>
<td>4%</td>
<td>Nemotron, reasoning off</td>
</tr>
<tr>
<td>log y-axis</td>
<td>8%</td>
<td>Qwen3.8-Max</td>
</tr>
<tr>
<td>log y-axis</td>
<td>32%</td>
<td>Gemma 4</td>
</tr>
</tbody></table>
<p><em>100 Grafana-style panels drawn by our renderer, 31 October 2026. Panels per row: 1 series 17, stacked 19, small 19, similar colours 27, 4-5 series 36, log y-axis 25. Rows this small give wide intervals; read them as directions, not rates.</em></p>
<p>What stands out:</p>
<ul>
<li><strong>Time and threshold are easy.</strong> Every model got the peak time right on 89 to 99 panels and the threshold on 86 to 97.</li>
<li><strong>A simple panel is almost solved.</strong> With one series, Nemotron and Qwen3.8-Max both got 15 of 17 right. Every added feature cost accuracy. Similar colours cost Nemotron the most: 16 of its 25 wrong series names were on the 27 similar-colour panels.</li>
<li><strong>Qwen3.8-Max and Gemma 4 cannot be told apart.</strong> Each got 14 or 13 panels right that the other got wrong. Both beat Nemotron with reasoning off: Qwen3.8-Max on 20 panels against 3, Gemma 4 on 24 against 8 (sign test p = 0.007).</li>
<li><strong>The 5 percent bar hurt Gemma 4 most.</strong> It rounds: 20,000 req/s for a peak of 21,330, 300 ms for 331. Within 10 percent, Gemma read the value right on 93 panels, Qwen3.8-Max on 87 and Nemotron on 82.</li>
</ul>
<p>Repeat runs at temperature 0 showed another difference. Qwen3.8-Max and Gemma 4 gave the same verdict in all three runs on 97 and 98 panels. Nemotron with reasoning off did on 71, and gave the same four answers on only 13. Llama 4 Maverick gave the same four answers on 92 panels and was right in all three runs on only 21. Consistent is not the same as correct.</p>
<h2>Where the reads go wrong</h2><p><strong>Log axes.</strong> The models read a log axis as if it were linear, or round to a tick. Nemotron read the value within 5 percent on 1 of 25 log panels, Qwen3.8-Max on 4 and Gemma 4 on 10.</p>
<p><img src="https://devops-daily.com/images/posts/vision-model-grafana-panel-digitalocean-inference/log-axis-p003.png" alt="A Grafana-style log-scale panel where POST /login spikes to 21.3K req/s between the 10K and 100K ticks" /></p>
<p>The spike peaks at 21.3K req/s at 14:28 UTC. Every model found the series and the time. Qwen3.8-Max and Gemma 4 said 20,000 every time, 6.2 percent low. Nemotron's three runs said 30 req/s, 45,000 req/s and 30K req/s.</p>
<p><strong>The threshold label as the answer.</strong> When a spike crosses a dashed line, the label of that line is the easiest number on the panel to read.</p>
<p><img src="https://devops-daily.com/images/posts/vision-model-grafana-panel-digitalocean-inference/threshold-anchor-p072.png" alt="A log-scale latency panel with two orange series and a dashed threshold at 200 ms; the inventory series rises at the right edge to about 246 ms" /></p>
<p>The rise at the right edge goes to 246 ms, above the 200 ms line. Qwen3.8-Max, Gemma 4 and Llama 4 Maverick answered 200 ms in all three runs: the threshold, not the peak. Nemotron answered 500 ms, 0.3 s and 300 ms. Only Llama 4 Maverick named <code>inventory</code> in every run, and it said the threshold was not crossed.</p>
<p><strong>The band on top is not the biggest.</strong> On a stacked panel, the series drawn last sits on top, even when its own values are small.</p>
<p><img src="https://devops-daily.com/images/posts/vision-model-grafana-panel-digitalocean-inference/stacked-top-band-p020.png" alt="A stacked error-ratio panel; node-a1 is the thick green band at the bottom and node-c3 is the thin band drawn on top" /></p>
<p><code>node-c3</code> is the thin band on top. Qwen3.8-Max and Llama 4 Maverick named it in all three runs and Nemotron in two. Gemma 4 named <code>node-a1</code> every time. This panel also shows the hardest threshold style, a shaded region with no line. The models got a region-only threshold right on 12 to 15 of 17 panels, and a region with a line on 19 or 20 of 20.</p>
<h2>Reasoning on: slower and more expensive, and not clearly better</h2><p>With reasoning on, Nemotron got 54 panels right against 48. Only reasoning got 14 panels right, and only no-reasoning got 8. A sign test gives p = 0.29, so 100 panels cannot say whether reasoning helps. The cost is clear: 3,349 output tokens per call on average in run 1 (8,163 at the 95th percentile), and 19.0 seconds at the median over all 168 calls, 81.8 at worst.</p>
<ul>
<li><strong>The usage block hides the reasoning.</strong> Every reasoning call returned <code>reasoning_tokens: 0</code>, while <code>completion_tokens</code> included the reasoning and the reasoning text came back in the reply. If you cap or alert on <code>reasoning_tokens</code>, you see nothing.</li>
<li><strong>Temperature 0 does not explain it.</strong> Reasoning models are often tuned for a higher temperature, so we ran the 34-panel subset again at 0.6. It got 19 right, against 20 at temperature 0 and 20 with reasoning off.</li>
<li><strong>It is not more repeatable.</strong> On the subset, reasoning gave the same four answers in all three runs on 3 panels, against 7 with reasoning off.</li>
</ul>
<h2>What a screenshot costs</h2><p>DigitalOcean's <a href="https://docs.digitalocean.com/products/inference/details/pricing/" rel="noopener noreferrer">pricing page</a> ("Last verified 30 Oct 2026", read on 31 October) lists no separate image rate for these models. We priced each call from the token counts the API reported, at list input and output prices. That assumes the image tokens inside <code>prompt_tokens</code> are billed at the input rate. The closed October invoice agrees with this. For each of the four models, the change in billed input, cached input and output tokens is equal to the totals the API reported for our 1,522 recorded calls. Each rate on the invoice is the list price, and the invoice has no separate charge for images.</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Image tokens, 360x200</th>
<th>Image tokens, 1000x500</th>
<th>Prompt tokens (cached)</th>
<th>Output tokens</th>
<th>$ per screenshot</th>
</tr>
</thead>
<tbody><tr>
<td>Nemotron 3.5 Super VL, reasoning off</td>
<td>288</td>
<td>514</td>
<td>652 (0)</td>
<td>97</td>
<td>0.00026</td>
</tr>
<tr>
<td>Nemotron 3.5 Super VL, reasoning on</td>
<td>288</td>
<td>514</td>
<td>652 (0)</td>
<td>3,349</td>
<td>0.0024</td>
</tr>
<tr>
<td>Qwen3.8-Max</td>
<td>68</td>
<td>498</td>
<td>594 (262)</td>
<td>126</td>
<td>0.0015</td>
</tr>
<tr>
<td>Gemma 4</td>
<td>266</td>
<td>255</td>
<td>604 (325)</td>
<td>102</td>
<td>0.00011</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>437</td>
<td>1,017</td>
<td>1,096 (271)</td>
<td>72</td>
<td>0.00028</td>
</tr>
</tbody></table>
<p>Image tokens are the prompt tokens with the panel minus those of the same question with no image. Gemma 4 uses about 260 at any size, Qwen3.8-Max scales with the pixels, and Llama 4 Maverick jumps from 437 to 1,017 above 600 pixels wide.</p>
<p>Our first cost numbers were wrong in our favour. Runs 2 and 3 send the same image again, and Qwen3.8-Max, Gemma 4 and Llama 4 Maverick served most of it from the prompt cache: Llama's cached tokens went from 271 in run 1 to 1,044. A bot sees each screenshot once, so we now price run 1 only. That raised Qwen3.8-Max from $1.11 to $1.47 per 1,000 and Gemma 4 from $0.097 to $0.113. Nemotron reported no cached tokens on any call, even for the prompt text that every call shares.</p>
<h2>Rules decide, the model explains</h2><p>The alert already knows the numbers. In our test, each alert comes from a rule on the panel's query, reduced with <code>max</code> over the panel's range, so the payload has the peak value, the series label (unless the rule is on a sum), the threshold and the firing state. If your rule reduces with <code>last</code> or a shorter window, compare against that value instead. The model reads the image blind, with no numbers from the alert, and three rules compare its read with the payload:</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-comment">// lib/bot.mjs (shortened). tol is 10% by default.</span>
<span class="hljs-keyword">export</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">crossCheck</span>(<span class="hljs-params">alert, answer, { tol = <span class="hljs-number">0.1</span> } = {}</span>) {
  <span class="hljs-keyword">if</span> (!answer) <span class="hljs-keyword">return</span> { <span class="hljs-attr">verdict</span>: <span class="hljs-string">'could not confirm'</span>, <span class="hljs-attr">checks</span>: [{ <span class="hljs-attr">fact</span>: <span class="hljs-string">'read'</span>, <span class="hljs-attr">ok</span>: <span class="hljs-literal">false</span> }] };
  <span class="hljs-keyword">const</span> checks = [];
  <span class="hljs-comment">// 1. Peak value against the alert's reduced value.</span>
  <span class="hljs-keyword">const</span> p = <span class="hljs-title function_">readPeak</span>(alert.<span class="hljs-property">unit</span>, answer.<span class="hljs-property">peak</span>);
  checks.<span class="hljs-title function_">push</span>({
    <span class="hljs-attr">fact</span>: <span class="hljs-string">'peak value'</span>,
    <span class="hljs-attr">ok</span>: p.<span class="hljs-property">ok</span> &amp;&amp; <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">abs</span>(p.<span class="hljs-property">base</span> - alert.<span class="hljs-property">values</span>.<span class="hljs-property">B</span>) / alert.<span class="hljs-property">values</span>.<span class="hljs-property">B</span> &lt;= tol,
  });
  <span class="hljs-comment">// 2. Series, when the alert is for one series. A rule on a sum has none.</span>
  <span class="hljs-keyword">if</span> (alert.<span class="hljs-property">labels</span>.<span class="hljs-property">series</span>) {
    checks.<span class="hljs-title function_">push</span>({
      <span class="hljs-attr">fact</span>: <span class="hljs-string">'series'</span>,
      <span class="hljs-attr">ok</span>: <span class="hljs-title function_">matchSeries</span>(answer.<span class="hljs-property">topSeries</span>, [alert.<span class="hljs-property">labels</span>.<span class="hljs-property">series</span>]) === alert.<span class="hljs-property">labels</span>.<span class="hljs-property">series</span>,
    });
  }
  <span class="hljs-comment">// 3. Threshold state, when the model saw a threshold on the panel.</span>
  <span class="hljs-keyword">const</span> th = <span class="hljs-title function_">normThreshold</span>(answer.<span class="hljs-property">thresholdCrossed</span>);
  <span class="hljs-keyword">if</span> (th === <span class="hljs-string">'yes'</span> || th === <span class="hljs-string">'no'</span>) {
    checks.<span class="hljs-title function_">push</span>({ <span class="hljs-attr">fact</span>: <span class="hljs-string">'threshold'</span>, <span class="hljs-attr">ok</span>: th === (alert.<span class="hljs-property">status</span> === <span class="hljs-string">'firing'</span> ? <span class="hljs-string">'yes'</span> : <span class="hljs-string">'no'</span>) });
  }
  <span class="hljs-keyword">return</span> { <span class="hljs-attr">verdict</span>: checks.<span class="hljs-title function_">every</span>(<span class="hljs-function">(<span class="hljs-params">c</span>) =&gt;</span> c.<span class="hljs-property">ok</span>) ? <span class="hljs-string">'confirmed'</span> : <span class="hljs-string">'could not confirm'</span>, checks };
}
</code></pre><p>The message always starts with the alert's own facts. The model's sentence is added only when its read agrees. Here is the bot on three panels, with Nemotron and reasoning off:</p>
<p><strong>the bot on three alerts</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/05b-bot-demo.mjs p034 p090 p003
--- p034 (data/panels/p034.png) -&gt; confirmed, 2033 ms
Container RSS: firing, max 3.19 GiB over the last 6h, threshold 2.51 GiB.
Screenshot (checked against the alert): Container RSS memory usage spikes sharply around 08:15 UTC to over 3 GiB, driven primarily by worker-5c6b-h3t7, <span class="hljs-keyword">then</span> settles back to roughly 1-2 GiB.
--- p090 (data/panels/p090.png) -&gt; confirmed, 1380 ms
Checkout latency p95: firing, max 106.72 ms over the last 24h, threshold 73 ms, series search.
Screenshot (checked against the alert): Checkout latency p95 is stable around 30-40 ms <span class="hljs-keyword">until</span> a spike to about 100 ms near 22:00 UTC, crossing the red 100 ms threshold.
--- p003 (data/panels/p003.png) -&gt; could not confirm, 1311 ms
HTTP requests: firing, max 21.33K req/s over the last 6h, threshold 17K req/s, series POST /login.
Could not confirm the screenshot (the screenshot reads 25K req/s, the alert says 21.33K req/s). Open the dashboard before you trust the image.
</code></pre><p>The p090 message shows the limit. The value, series and threshold state agree, so the bot says "confirmed", but the summary says "the red 100 ms threshold" and the threshold is 73 ms. The rules check the structured answer, not every word. That is why the bot posts the alert's numbers itself.</p>
<p>Over all recorded reads:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Wrong reads</th>
<th>Caught at 10%</th>
<th>Caught at 5%</th>
<th>Correct reads blocked</th>
</tr>
</thead>
<tbody><tr>
<td>Nemotron 3.5 Super VL, reasoning off</td>
<td>155 of 300</td>
<td>97 (63%)</td>
<td>125 (81%)</td>
<td>0</td>
</tr>
<tr>
<td>Qwen3.8-Max</td>
<td>109 of 300</td>
<td>58 (53%)</td>
<td>90 (83%)</td>
<td>0</td>
</tr>
<tr>
<td>Gemma 4</td>
<td>111 of 300</td>
<td>54 (49%)</td>
<td>98 (88%)</td>
<td>0</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>236 of 300</td>
<td>143 (61%)</td>
<td>214 (91%)</td>
<td>0</td>
</tr>
</tbody></table>
<p>"Correct reads blocked: 0" is by construction. Our alert value is the true peak and the bot's checks are looser than the scorer's, so a correct read cannot fail them. A real alert that reduces differently from the panel would add false flags. What the 10 percent check let through for Nemotron:</p>
<ul>
<li>24 reads with only a value error between 5 and 10 percent, such as 300 ms for 331 ms</li>
<li>21 wrong series names, all on stacked panels, where the alert is on the sum and has no series to compare</li>
<li>6 threshold errors where the model said "no threshold" on a panel that shows one; the bot cannot tell that apart from a rule threshold that is not drawn</li>
<li>7 wrong peak times, alone or with a small value error; the payload has no peak time</li>
</ul>
<p>Then we broke it on purpose: each read was checked against the alert of a different panel with the same unit, like a wrong image on the notification. The bot said "could not confirm" for 99 of 100 reads from Nemotron and from Qwen3.8-Max, and for 100 of 100 from Gemma 4 and Llama 4 Maverick. The pair that passed had a wrong alert on a sum (no series check) whose value was 8.6 percent from the read.</p>
<p>We expected the model to copy the alert's numbers if the prompt contained them, which is why the bot reads blind. For Nemotron that was wrong. With the wrong alert in the prompt, its peak matched the image on 81 of 100 panels and the alert on 1, and the cross-check still refused 98. We keep the blind read because it costs nothing, but this test does not show that it is necessary.</p>
<h2>What 100 panels cannot tell you</h2><ul>
<li><strong>The claim failed by a wide margin.</strong> The 95 percent intervals for Nemotron end at 58 and 63 panels, far below 90.</li>
<li><strong>We cannot rank Qwen3.8-Max against Gemma 4,</strong> or Nemotron with reasoning against without. Both gaps are too small for 100 panels.</li>
<li><strong>Feature rows are directions, not rates.</strong> Each covers 17 to 36 panels. Gemma 4's 8 of 25 on log axes has a 95 percent interval of 17 to 52 percent.</li>
<li><strong>The panels are ours.</strong> Real Grafana screenshots are a different input, easier in some ways and harder in others.</li>
<li><strong>One prompt.</strong> We did not try a prompt that warns about log axes. We did not use a JSON response mode, and Qwen3.8-Max wrote invalid JSON twice in 300 calls (<code>"thresholdCrossed": no</code>).</li>
<li><strong>One day of one service.</strong> The models DigitalOcean served on 31 October 2026, at temperature 0. Prices are list prices times reported tokens. The closed October invoice billed the same token totals at the list prices.</li>
</ul>
<h2>Run it yourself</h2><p>Each stage writes a file that the next one reads. Every recorded call is stamped with a hash of the runner code and the inputs. The scorer refuses unstamped, mixed or stale files, and records filed under the wrong configuration. The runner stops before its own count of spend passes a cap ($3.50 by default).</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/grafana-panel-vision
<span class="hljs-built_in">cd</span> grafana-panel-vision
npm ci
<span class="hljs-built_in">cp</span> .env.example .<span class="hljs-built_in">env</span>                 <span class="hljs-comment"># add your model access key</span>

node scripts/01-render-panels.mjs    <span class="hljs-comment"># 100 PNGs + data/truth.jsonl + data/alerts.jsonl</span>
node scripts/01b-check-panels.mjs    <span class="hljs-comment"># read every panel back from the SVG</span>
<span class="hljs-built_in">mv</span> results/raw results/raw-ours      <span class="hljs-comment"># keep our run; yours goes into results/raw</span>
node scripts/02-run-models.mjs --configs nemotron-3.5-super-vl,gemma-4-31B-it --runs 3
node scripts/03-score.mjs            <span class="hljs-comment"># results/summary.md</span>
node scripts/05-crosscheck.mjs       <span class="hljs-comment"># how often the bot catches a wrong read</span>
</code></pre><p>To test your own dashboards, replace the renderer with exports of your real panels and write the truth for each by hand. It is slower, and it is the only way to know how these models read your panels.</p>
<h2>Summary</h2><p>A vision model on DigitalOcean can describe a Grafana panel in about 1.5 seconds for a fraction of a cent. On our 100 panels, no model read the four key facts reliably enough to be the source of truth in a page. Log axes, stacked areas and similar colours caused most errors, and reasoning did not fix them.</p>
<p>What we would ship:</p>
<ul>
<li><strong>Take the numbers from the alert, not from the image.</strong> The payload already has the value, the threshold and the series.</li>
<li><strong>Let the model narrate, and check it first.</strong> Compare its value, series and threshold state with the payload, and say "could not confirm" when they disagree.</li>
<li><strong>Use a fast model with reasoning off.</strong> Gemma 4 and Qwen3.8-Max were the most accurate, and Gemma 4 cost $0.11 per 1,000 screenshots.</li>
<li><strong>Make attached panels easy to read.</strong> Use a linear axis, distinct colours, few series, and a threshold drawn as a line and a region.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Neon Anonymized Branches: Mask the Columns, Then Scan the Rest]]></title>
      <link>https://devops-daily.com/posts/neon-anonymized-branches-what-masking-misses</link>
      <description><![CDATA[We gave a Neon anonymized branch masking rules for every PII column of a 20,000-user table. The masking replaced the values in those columns, and all 20,000 original emails were still readable elsewhere on the branch: in order copies, JSON payloads, support tickets, and an audit table that the masking run itself filled. Here is the scanner we use as a CI gate, and the rules it took to pass it.]]></description>
      <pubDate>Tue, 03 Nov 2026 15:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/neon-anonymized-branches-what-masking-misses</guid>
      <category><![CDATA[Security]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Neon]]></category><category><![CDATA[Postgres]]></category><category><![CDATA[Data Anonymization]]></category><category><![CDATA[PII]]></category><category><![CDATA[Database Branching]]></category><category><![CDATA[Preview Environments]]></category><category><![CDATA[CI/CD]]></category><category><![CDATA[Security]]></category>
      <content:encoded><![CDATA[<p>We created an anonymized branch of a 20,000-user SaaS database on Neon. It had masking rules for the six PII columns of the <code>users</code> table: email, name, phone, street address, date of birth and signup IP. The masking run took 9 seconds. Afterwards, no row in <code>users</code> held its original email, name, phone, address or IP (3 of the 20,000 random dates of birth matched the original by chance). Then we read every text and JSON column on the branch. <strong>All 20,000 original email addresses were still there</strong>, and so was every original name, phone number, street address and IP address.</p>
<p>Most of them were in places nobody writes a rule for: the email copied onto each order at checkout, the JSON payloads of login and checkout events, the text of support tickets. The other 20,000 copies came from the masking run itself. The masking updates each row of <code>users</code>, so the audit trigger on that table fired 20,000 times and saved every original row into <code>audit_log</code>.</p>
<p>Anonymized branches mask the columns you list, and in our runs they did. This post measures what is left in the rest of a realistic schema, and shows the scanner we run as a CI gate before preview branches are made from the masked branch. Everything ran on a real Neon project through the Neon API, and the scripts are public.</p>
<h2>TL;DR</h2><ul>
<li><strong>Masking the listed columns works.</strong> No masked email, name, phone, address or IP kept its original value.</li>
<li><strong>The copies are not masked.</strong> After round 1, 18,776 of the 20,000 original emails (93.9%) were still readable outside the audit table: in orders, event payloads, support tickets and invitations.</li>
<li><strong>The masking run fires your triggers.</strong> <code>audit_log</code> grew from 1,888 to 21,888 rows during masking, and each new row held one user's full original row. The branch refuses connections until the run ends, so you cannot switch the trigger off first.</li>
<li><strong>Free text and JSON need a decision.</strong> We found no rule that edits JSON in place, and <code>regexp_replace</code> removed emails and phone numbers from ticket bodies but not names. Only replacing free text and JSON passed the gate.</li>
<li><strong>Pick generators with care.</strong> <code>anon.dummy_free_email()</code> and <code>anon.dummy_safe_email()</code> made the run fail on a unique column at 20,000 rows, and random IP addresses and phone numbers can be equal to real ones.</li>
<li><strong>Previews from the passing template were ready in 2.3 to 2.5 seconds.</strong> The same rules failed the scan after a migration copied emails into a new table.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Neon account and an API key (Account settings &gt; API keys). Everything here ran on a Scale plan project. Anonymized branches are a beta feature.</li>
<li>Node.js 22 or newer.</li>
<li>Basic Postgres knowledge: triggers, <code>jsonb</code>, materialized views.</li>
</ul>
<h2>What an anonymized branch does</h2><p>Neon's <a href="https://neon.com/docs/workflows/data-anonymization" rel="noopener noreferrer">data anonymization</a> creates a branch of a Lakebase Postgres database from a parent branch, then masks it with <a href="https://postgresql-anonymizer.readthedocs.io/" rel="noopener noreferrer">PostgreSQL Anonymizer</a> (the <code>anon</code> extension, version 2.5.1 on our branches). The masking is static: the run rewrites the rows on the new branch, and the parent does not change. You send the rules with the create call:</p>
<pre><code class="hljs language-bash" tabindex="0">curl -X POST <span class="hljs-string">"https://console.neon.tech/api/v2/projects/<span class="hljs-variable">$PROJECT_ID</span>/branch_anonymized"</span> \
  -H <span class="hljs-string">"Authorization: Bearer <span class="hljs-variable">$NEON_API_KEY</span>"</span> -H <span class="hljs-string">"Content-Type: application/json"</span> \
  -d <span class="hljs-string">'{
    "branch_create": { "branch": { "name": "template", "parent_id": "'</span><span class="hljs-string">"<span class="hljs-variable">$MAIN_ID</span>"</span><span class="hljs-string">'" } },
    "masking_rules": [
      { "database_name": "neondb", "schema_name": "public", "table_name": "users",
        "column_name": "full_name", "masking_function": "anon.dummy_name()" }
    ],
    "start_anonymization": true
  }'</span>
</code></pre><p>Three details matter later:</p>
<ol>
<li>The branch refuses connections until the run ends (<code>Access to this endpoint is temporarily disabled</code>). We saw the same error after each of our failed runs, so in our tests a failed run did not hand out unmasked data.</li>
<li><code>GET /branches/{id}/anonymized_status</code> reports <code>anonymizing</code>, then <code>anonymized</code> or <code>error</code>.</li>
<li>Each rule names one column. Nothing in the API looks at the columns you did not name.</li>
</ol>
<p>That is by design, and Neon's <a href="https://neon.com/blog/pg-redact-content-aware-pii-redaction-with-jev-enforced-in-postgres" rel="noopener noreferrer">pg_redact post</a> makes the same point: column rules cannot reach PII inside free text. We wanted to know how much PII that is in an ordinary schema, where it sits, and how to find it before someone gets a connection string.</p>
<h2>The test database</h2><p>The schema is a small invoicing SaaS. <code>users</code> holds the PII on purpose. Everything else holds it by accident, the way real apps do:</p>
<ol>
<li><strong>users</strong> email, full_name, phone, address, signup_ip</li>
<li><strong>orders</strong> contact_email, shipping_*</li>
<li><strong>support_tickets.body</strong> free text</li>
<li><strong>events.payload</strong> jsonb</li>
<li><strong>invitations</strong> invitee email, name, message</li>
<li><strong>audit_log</strong> trigger: old and new row</li>
<li><strong>customer_summary</strong> materialized view</li>
<li><strong>users.search</strong> generated tsvector</li>
</ol>
<p>Connections:</p>
<ul>
<li>users -&gt; orders (copied)</li>
<li>users -&gt; support_tickets.body (quoted)</li>
<li>users -&gt; events.payload (logged)</li>
<li>users -&gt; invitations (named)</li>
<li>users -&gt; audit_log (every UPDATE)</li>
<li>users -&gt; customer_summary (refresh)</li>
<li>users -&gt; users.search (generated)</li>
</ul>
<p>A seeded generator writes 20,000 users, 40,191 orders, 6,558 tickets, 59,738 events and 2,035 invitations (47.6 MB). Then it changes the email or the address of 1,888 users, so the audit trigger has history with former values. All names are made up, emails use the reserved <code>.example</code> and <code>.test</code> domains, and phone numbers use the fictional 555-01xx range. The generator records every PII value it writes into each row, so we know what each column holds, and it mixes in decoys such as invoice numbers, tracking numbers and the app's own support address.</p>
<h2>The scanner</h2><p><a href="https://github.com/The-DevOps-Daily/neon-anonymized-branches" rel="noopener noreferrer">The-DevOps-Daily/neon-anonymized-branches on GitHub</a></p>
<p><code>scan.mjs</code> takes two branches: the anonymized branch and its parent. It reads the PII columns you list in <code>pii-sources.json</code> on the parent (here five columns of <code>users</code>), then reads every text, JSON, <code>tsvector</code>, <code>inet</code> and <code>xml</code> column of the user tables and materialized views on the anonymized branch (the <code>anon</code> extension's own tables are left out). Per column it reports:</p>
<ul>
<li><strong>known</strong>: rows that still hold a value from the parent, also inside free text and JSON. Phone numbers match on their last 10 digits, and a full name matches as two neighbouring words, so <code>/team/tarek-stratkett</code> in a URL matches "Tarek Stratkett".</li>
<li><strong>pattern</strong>: values shaped like an email, phone number, IP address or street address that are not known values, such as former emails and people who are not users.</li>
<li><strong>row counts</strong> of every table that changed during masking, and a <strong>fingerprint</strong> of the data on the branch (per table, the row count and a sha256 over the sorted md5 of each row).</li>
</ul>
<p>It fails when a column holds a known value, when a column that no rule replaces holds PII-shaped values, when a rule changed no row, when the branch has a database it did not read, when the branch holds data it cannot read (such as a <code>bytea</code> column, large objects or foreign tables), or when the data changed during the scan. Two columns here are computed only from masked columns, a materialized view and a generated search column. They are on a review list: their PII-shaped values go to a person instead of failing the gate, and known values in them still fail.</p>
<p>A pass means the scan found no known value anywhere, and no PII-shaped value outside replaced and review columns. It does not mean the branch is clean. The scanner does not know phone numbers outside the US format, nicknames or encoded values, and it does not read numeric columns.</p>
<p>Against the unmasked parent, the rows it found matched the planted ground truth for all 33 combinations of column and kind. That is not an independent test of recall, because we planted the data in formats the scanner knows. It shows that the scanner counts what it should on this data. A second check, <code>08-predict.mjs</code>, predicts each column from the parent's data and the rules, without Neon. Where the real scan differs from the prediction, the masking run did something on its own.</p>
<h3>The first gate was too clever</h3><p>The first version of the gate raised four false alarms on two templates. Three were values that the fake data generator made by chance and that belong to other users: 2 masked IP addresses and 1 phone number. The fourth was our bug in the <code>tsvector</code> name check. We first excused any known value that also appeared in a masked column. A review showed that this fails open: a real leak with the same value would pass too.</p>
<p>So the gate has no exception for chance matches. Every known value fails it, and a person reads the report. On real data this happens more often than in ours, because <code>anon.dummy_name()</code> returns real first and last names. A false alarm costs a few minutes. A leak costs much more. The old scans are in <code>results/first-gate/</code>.</p>
<h2>Round 1: the columns everyone lists</h2><p>The first rule set masks the PII columns of <code>users</code>, the way the docs and the GitHub Action examples do. For the unique email column we used the Console's Random Unique Email, <code>pg_catalog.concat(anon.dummy_uuidv4(), '@example.com')</code>, which Neon recommends for unique columns.</p>
<p><strong>scan the round 1 template</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/scan.mjs --target template-v1 --known main --evidence --out results/scan-template-v1.json
scanned 26 columns <span class="hljs-keyword">in</span> 7 tables of template-v1 (68.8 s)
  audit_log.old_row                  rule=none      known: email 20938, name 21888, phone 18552, address 20950, ip 21888 pattern: email 950, address 938
  audit_log.new_row                  rule=none      known: email 1888, name 1888, phone 1585, address 1888, ip 1888 pattern: email 20000, phone 17807, address 954, ip 20000
  customer_summary.email             rule=review    known: -                                            pattern: email 20000, phone 227
  events.payload                     rule=none      known: email 19931, name 12816, ip 11888            pattern: ip 8017
  invitations.invitee_email          rule=none      known: email 585                                    pattern: email 1450
  invitations.invitee_name           rule=none      known: name 585                                     pattern: -
  invitations.message                rule=none      known: name 2035                                    pattern: -
  orders.contact_email               rule=none      known: email 40191                                  pattern: -
  orders.shipping_name               rule=none      known: name 36939                                   pattern: -
  orders.shipping_address            rule=none      known: phone 31371, address 36939                   pattern: phone 3252, address 3252
  support_tickets.body               rule=none      known: email 1324, name 1231, phone 1162, address 588 pattern: email 687, phone 622
  users.search                       rule=review    known: -                                            pattern: email 20000, phone 220
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
FAIL: 16 problems
</code></pre><p>These are the rows that still hold a user's original PII:</p>
<table>
<thead>
<tr>
<th>Column</th>
<th>Rows</th>
<th>Rows with a user's original PII</th>
<th>What</th>
</tr>
</thead>
<tbody><tr>
<td><code>audit_log.old_row</code></td>
<td>21,888</td>
<td>21,888</td>
<td>every kind; 20,000 rows were written by the masking run</td>
</tr>
<tr>
<td><code>orders.contact_email</code></td>
<td>40,191</td>
<td>40,191</td>
<td>the email, copied at checkout</td>
</tr>
<tr>
<td><code>orders.shipping_name</code></td>
<td>40,191</td>
<td>36,939</td>
<td>the name</td>
</tr>
<tr>
<td><code>orders.shipping_address</code></td>
<td>40,191</td>
<td>36,939</td>
<td>address and phone, inside <code>jsonb</code></td>
</tr>
<tr>
<td><code>events.payload</code></td>
<td>59,738</td>
<td>34,689</td>
<td>emails, names, IPs, inside <code>jsonb</code></td>
</tr>
<tr>
<td><code>support_tickets.body</code></td>
<td>6,558</td>
<td>2,486</td>
<td>free text</td>
</tr>
<tr>
<td><code>invitations.message</code></td>
<td>2,035</td>
<td>2,035</td>
<td>the inviter's name</td>
</tr>
<tr>
<td><code>audit_log.new_row</code></td>
<td>21,888</td>
<td>1,888</td>
<td>history from before the branch</td>
</tr>
<tr>
<td><code>invitations.invitee_email</code>, <code>invitee_name</code></td>
<td>2,035</td>
<td>585</td>
<td>invitees who are users</td>
</tr>
</tbody></table>
<p>Counted as distinct values, this is how much is still readable on the branch:</p>
<table>
<thead>
<tr>
<th>Kind</th>
<th>Readable anywhere</th>
<th>Readable outside <code>audit_log</code></th>
</tr>
</thead>
<tbody><tr>
<td>Email</td>
<td>20,000 of 20,000</td>
<td>18,776 (93.9%)</td>
</tr>
<tr>
<td>Full name</td>
<td>17,396 of 17,396</td>
<td>16,024 (92.1%)</td>
</tr>
<tr>
<td>Phone</td>
<td>16,967 of 16,967</td>
<td>13,639 (80.4%)</td>
</tr>
<tr>
<td>Street address</td>
<td>20,000 of 20,000</td>
<td>15,907 (79.5%)</td>
</tr>
<tr>
<td>IP address</td>
<td>20,000 of 20,000</td>
<td>9,208 (46.0%)</td>
</tr>
</tbody></table>
<p>The scan also flagged PII of people who are not users, such as 1,450 invitee emails and 8,017 login IPs that are not signup IPs. The prediction matched 21 of the 26 columns, and each difference points to something the masking run did. Two of them are good news: the generated <code>search</code> column and the materialized view <code>customer_summary</code> held no original value, because Postgres computes a generated column again on every <code>UPDATE</code> and Neon refreshes materialized views after the run. The audit trigger caused the others.</p>
<h3>The masking run fires your triggers</h3><p>The prediction for <code>audit_log</code> was 1,888 rows. The branch had 21,888. Static masking <a href="https://postgresql-anonymizer.readthedocs.io/en/latest/static_masking/" rel="noopener noreferrer">updates every row</a> of each table that has a rule, so a row-level <code>AFTER UPDATE</code> trigger fires once for each masked row. Our trigger is a common audit pattern:</p>
<pre><code class="hljs language-sql" tabindex="0"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">FUNCTION</span> audit_users() <span class="hljs-keyword">RETURNS</span> <span class="hljs-keyword">trigger</span> <span class="hljs-keyword">LANGUAGE</span> plpgsql <span class="hljs-keyword">AS</span> $$
<span class="hljs-keyword">BEGIN</span>
  <span class="hljs-keyword">INSERT INTO</span> audit_log (table_name, row_id, action, old_row, new_row)
  <span class="hljs-keyword">VALUES</span> (<span class="hljs-string">'users'</span>, OLD.id, TG_OP, to_jsonb(<span class="hljs-keyword">OLD</span>), <span class="hljs-keyword">CASE</span> <span class="hljs-keyword">WHEN</span> TG_OP <span class="hljs-operator">=</span> <span class="hljs-string">'UPDATE'</span> <span class="hljs-keyword">THEN</span> to_jsonb(<span class="hljs-keyword">NEW</span>) <span class="hljs-keyword">END</span>);
  <span class="hljs-keyword">RETURN</span> <span class="hljs-keyword">NULL</span>;
<span class="hljs-keyword">END</span> $$;
</code></pre><p>So the masking wrote 20,000 new audit rows, and each <code>old_row</code> holds one user's original email, name, phone, address and IP. A rule on <code>users</code> alone made the branch hold a second full copy of <code>users</code>.</p>
<p>There is no clean way to stop this on the branch:</p>
<ul>
<li>The branch refuses connections before the run starts. With <code>start_anonymization: false</code> our client was refused too, so you cannot run <code>ALTER TABLE ... DISABLE TRIGGER</code> first.</li>
<li>The trigger cannot skip the masking run by role. All 21,888 audit rows on the masked branch record <code>neondb_owner</code>, the role the app uses too.</li>
</ul>
<p>What worked was a rule on the audit table itself (<code>old_row</code> and <code>new_row</code> masked with <code>NULL</code>). In our runs Neon masked <code>users</code> first and <code>audit_log</code> after it, so the masking also cleared the rows that its own trigger had written. We put the audit rules first in one rules file (<code>v2-audit-first</code>), and the order did not change. We did not find this order documented, so do not depend on it. The 20,000 extra rows stay on the branch with <code>NULL</code> in both columns, which is why the scanner reports tables whose row count changed.</p>
<h2>Round 2: mask the copies, edit the free text</h2><p>The second rule set (<code>v2-careful</code>) adds a rule for every structured copy the first scan found: the order contact email and shipping name, the shipping address rebuilt with <code>jsonb_build_object</code>, the invitee email and name, and the two audit columns. For the ticket bodies it keeps the text and cuts out emails and phone numbers with the same patterns the scanner uses:</p>
<pre><code class="hljs language-text" tabindex="0">pg_catalog.regexp_replace(
  pg_catalog.regexp_replace(body, '&lt;email pattern&gt;', '[email]', 'g'),
  '&lt;phone pattern&gt;', '[phone]', 'g')
</code></pre><p>Three columns still failed:</p>
<table>
<thead>
<tr>
<th>Column</th>
<th>Rule</th>
<th>Rows with a user's original PII</th>
</tr>
</thead>
<tbody><tr>
<td><code>events.payload</code></td>
<td>none</td>
<td>34,689 (emails 19,931, names 12,816, IPs 11,888)</td>
</tr>
<tr>
<td><code>invitations.message</code></td>
<td>none</td>
<td>2,035 (names)</td>
</tr>
<tr>
<td><code>support_tickets.body</code></td>
<td><code>regexp_replace</code></td>
<td>1,819 (names 1,231, addresses 588)</td>
</tr>
</tbody></table>
<p>The <code>regexp_replace</code> rule did its job. No email or phone number was left in any ticket body, and the JavaScript prediction matched the scan exactly. But regular expressions find shapes, and a name has no shape. "Contact Maeve Ashquist on [phone] before 5pm" is still personal data.</p>
<p>The events table has no rule because we could not write one that keeps the payload. The rule lab (<code>03-rule-lab.mjs</code>) tried 17 expressions, each on its own anonymized branch:</p>
<table>
<thead>
<tr>
<th>Expression</th>
<th>Result</th>
</tr>
</thead>
<tbody><tr>
<td><code>anon.dummy_ipv4()::inet</code>, <code>CAST(... AS inet)</code></td>
<td>rejected with 400: not a valid label</td>
</tr>
<tr>
<td><code>CASE WHEN id % 2 = 0 THEN 'even' ELSE 'odd' END</code></td>
<td>rejected with 400</td>
</tr>
<tr>
<td><code>pg_catalog.jsonb_set(doc, '{email}', ...)</code></td>
<td>rejected with 400</td>
</tr>
<tr>
<td><code>pg_catalog.regexp_replace(doc, ...)</code> on <code>jsonb</code></td>
<td>the run failed: no <code>regexp_replace(jsonb, ...)</code></td>
</tr>
<tr>
<td><code>pg_catalog.jsonb_build_object('email', anon.dummy_safe_email(), ...)</code></td>
<td>masked: the whole document is rebuilt</td>
</tr>
<tr>
<td><code>masking_value: '{}'</code></td>
<td>masked</td>
</tr>
</tbody></table>
<p>The rule parser accepts function calls from the <code>anon</code> schema and 36 <code>pg_catalog</code> functions that the extension labels as trusted, such as <code>concat</code>, <code>regexp_replace</code>, <code>md5</code> and <code>jsonb_build_object</code>. <code>jsonb_set</code> is not one of them, and without casts a rule cannot turn <code>jsonb</code> into text and back. So in our tests a JSON column had two options: rebuild the whole document from fake values, which loses its shape when the shape changes from row to row (our events have four), or replace it with a constant.</p>
<h2>Round 3: replace what you cannot edit</h2><p>The third rule set (<code>v3-strict</code>) replaces what round 2 could only edit: ticket bodies become <code>anon.lorem_ipsum(words := 20)</code>, invitation messages become <code>anon.lorem_ipsum(words := 10)</code>, and event payloads become <code>{}</code>. It also takes IP addresses and phone numbers from ranges that no real value can be in (more on that below). That passed:</p>
<p><strong>scan the round 3 template</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/scan.mjs --target template-v3 --known main --evidence --out results/scan-template-v3.json
scanned 26 columns <span class="hljs-keyword">in</span> 7 tables of template-v3 (41.7 s)
  customer_summary.email             rule=review    known: -                                            pattern: email 20000, phone 212
  users.search                       rule=review    known: -                                            pattern: email 20000, phone 207
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
PASS: no PII from the <span class="hljs-built_in">source</span> columns and no PII-shaped values found outside replaced and review columns (2 to review)
</code></pre><p>It took 37 seconds to mask six tables. This is how much of the original PII each round left readable, counted as distinct values:</p>
<p><strong>Original PII still readable on the masked branch</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>round 1</th>
<th>round 2</th>
<th>round 3</th>
</tr>
</thead>
<tbody><tr>
<td>Email</td>
<td>100%</td>
<td>65.2%</td>
<td>0%</td>
</tr>
<tr>
<td>Full name</td>
<td>100%</td>
<td>59.5%</td>
<td>0%</td>
</tr>
<tr>
<td>Phone</td>
<td>100%</td>
<td>0%</td>
<td>0%</td>
</tr>
<tr>
<td>Street address</td>
<td>100%</td>
<td>2.9%</td>
<td>0%</td>
</tr>
<tr>
<td>IP address</td>
<td>100%</td>
<td>46%</td>
<td>0%</td>
</tr>
</tbody></table>
<p><em>Distinct original values of the 20,000 users found anywhere on each template, as a share of all distinct values of that kind (20,000 emails, 17,396 full names, 16,967 phones, 20,000 addresses, 20,000 IPs). Round 1: 6 rules on users. Round 2: 14 rules, regexp_replace on ticket bodies. Round 3: 16 rules, free text and JSON replaced.</em></p>
<p>The pass has a price. The support inbox in a preview shows lorem ipsum, and the events table has no payloads, so code that reads payloads has nothing to test against. If those features need realistic data, you can generate synthetic tickets and events for the template, or keep the free text and give the branch to fewer people. The regex rule is not a middle ground, because it cannot find names.</p>
<h2>The gate, and preview branches from the template</h2><p>The workflow we tested is: create the anonymized branch, wait for the run, scan it, and only then make preview branches from it.</p>
<ol>
<li><strong>main</strong> production data</li>
<li><strong>anonymized branch</strong> masking rules from git</li>
<li><strong>wait</strong> state: anonymized</li>
<li><strong>scan.mjs</strong> known values + patterns</li>
</ol>
<p>Outcomes:</p>
<ul>
<li><strong>pass: make preview branches</strong></li>
<li><strong>fail: delete the branch</strong></li>
</ul>
<p>The wait matters if you use Neon's GitHub Action. <a href="https://github.com/neondatabase/create-branch-action" rel="noopener noreferrer"><code>neondatabase/create-branch-action</code></a> v6 accepts <code>masking_rules</code> and returns the connection string right away, before the masking run ends. A job that connects next gets <code>Access to this endpoint is temporarily disabled</code>, and after our failed runs the branch stayed closed, so a job that only retries waits until it times out. <code>scan.mjs --wait</code> polls <code>anonymized_status</code> first and fails if the run did not finish. The repository has an example nightly workflow with these steps. We ran the steps through the scripts, not in GitHub Actions.</p>
<p>Two more things to know about the template branch:</p>
<ul>
<li><strong>Do not give it an expiry.</strong> The API refused to create a child from an anonymized branch with <code>expires_at</code>: <code>Branches with an expiration date cannot have child branches</code>. Set <code>expires_at</code> on the previews instead.</li>
<li><strong>Check the data before you make previews.</strong> A scan report is only good for the data it read. <code>04-previews.mjs</code> refuses to branch from the template unless its report is a pass, the branch has no database and no data that the scan did not read, and the data still has the scanned fingerprint. It then checks each preview against the same fingerprint.</li>
</ul>
<p>From the passing template, three previews were ready in 2.3 to 2.5 seconds each, and each one matched the scanned fingerprint. To see the check refuse, the script makes a copy of the template, scans it, and writes one support ticket with a real user's email into it, the way someone might paste a production row into a shared branch:</p>
<p><strong>previews behind the gate</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/04-previews.mjs --template template-v3
gate: template-v3 passed its scan and still has the scanned data (0ff0ef425216dfe8)
preview-pr-101: ready <span class="hljs-keyword">in</span> 2.4 s, checked after 6.1 s, same data as scanned: <span class="hljs-literal">true</span>
preview-pr-102: ready <span class="hljs-keyword">in</span> 2.3 s, checked after 6.0 s, same data as scanned: <span class="hljs-literal">true</span>
preview-pr-103: ready <span class="hljs-keyword">in</span> 2.5 s, checked after 6.1 s, same data as scanned: <span class="hljs-literal">true</span>
reset br-restless-paper-b9i5hgnz to the template <span class="hljs-keyword">in</span> 3.9 s; same data as scanned again: <span class="hljs-literal">true</span>
stale gate: after the scan open; after one INSERT refused: data changed since the scan: public.support_tickets (rows 6558 -&gt; 6559)
</code></pre><p>So treat the template as read-only once it passes: hand out preview branches, never the template's own connection string, and scan again after any change to it.</p>
<p>Then we broke it on purpose. <code>07-break.mjs</code> runs a migration on a branch of main that adds a <code>newsletter_signups</code> table, a copy of 9,956 users' emails and names. The round 3 rules do not know that table, so we made a new template with them and scanned it:</p>
<p><strong>scan after the migration</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/scan.mjs --target template-drift --known main-drift --out results/scan-template-drift.json
scanned 29 columns <span class="hljs-keyword">in</span> 8 tables of template-drift (39.1 s)
  customer_summary.email             rule=review    known: -                                            pattern: email 20000, phone 213
  newsletter_signups.email           rule=none      known: email 9956                                   pattern: -
  newsletter_signups.full_name       rule=none      known: name 9956                                    pattern: -
  users.search                       rule=review    known: -                                            pattern: email 20000, phone 207
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
FAIL: 2 problems
</code></pre><p>This is why the scan reads all the text and JSON columns, and not only the columns in the rules file. Rules describe the schema as it was when someone wrote them. The scan checks the schema as it is now. The new <code>first_name</code> column was not flagged, because a first name alone is not a full name.</p>
<h2>What your tests will notice</h2><p><strong>Unique columns.</strong> Fake data generators repeat themselves. We generated each email function many times and counted the repeats:</p>
<table>
<thead>
<tr>
<th>Function</th>
<th>Repeats in 1,000</th>
<th>Repeats in 20,000</th>
<th>Repeats in 100,000</th>
</tr>
</thead>
<tbody><tr>
<td><code>anon.dummy_free_email()</code></td>
<td>0</td>
<td>180</td>
<td>4,504</td>
</tr>
<tr>
<td><code>anon.dummy_safe_email()</code></td>
<td>57</td>
<td>11,967</td>
<td>90,980</td>
</tr>
<tr>
<td>Random Unique Email (<code>concat(anon.dummy_uuidv4(), '@example.com')</code>)</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td><code>anon.fake_email()</code></td>
<td>378</td>
<td>19,001</td>
<td>99,001</td>
</tr>
</tbody></table>
<p>On the unique <code>users.email</code> column, masking runs with <code>dummy_free_email()</code> and <code>dummy_safe_email()</code> both failed with <code>duplicate key value violates unique constraint "users_email_key"</code>. In the final rule lab run, <code>dummy_free_email()</code> also failed on a 200-row table with a unique column, so a small table is no guarantee. The unmasked data stayed unreachable, but a nightly template job fails.</p>
<p><strong>Where fake emails point.</strong> <code>dummy_free_email()</code> writes addresses at yahoo.com, hotmail.com and gmail.com, so a fake address can belong to a real person. If a preview app sends password resets or receipts, use a function that writes <code>example.com</code> addresses, or switch email off in previews.</p>
<p><strong>Stability.</strong> Two templates made from the same parent with the round 1 rules shared no masked email, name, phone, address or IP, so a test that expects "user 42 is called X" fails on the next template. Values derived from the primary key do not move: with <code>pg_catalog.concat('user-', id, '@example.com')</code> and <code>anon.pseudo_first_name(id)</code>, both templates matched in 20,000 of 20,000 rows. Only use such rules where the input is not PII itself, because anyone with the function can compute the output again.</p>
<p><strong>Foreign keys.</strong> The docs say foreign key columns cannot be masked directly, and that the Console shows an alert instead. Through the API, a rule on <code>orders.user_id</code> was accepted and the run completed. Afterwards, all 40,191 orders belonged to a different user than on the parent, so joins between orders and users no longer match the parent.</p>
<p><strong>Generators that can produce real values.</strong> With the first gate, 2 masked IP addresses from <code>anon.dummy_ipv4()</code> and 1 phone number from <code>anon.dummy_phone_number()</code> were equal to real values of other users. The round 3 rules take IP addresses from the 198.51.100.0/24 documentation range and give phone numbers the area code 000, which no real number has. Names are harder, because <code>anon.dummy_name()</code> returns real names.</p>
<p><strong><code>anon.detect()</code></strong> suggests PII columns from their names. On our schema it missed <code>full_name</code>, <code>street_address</code>, <code>signup_ip</code> and every copy. It is a start for a rules file, not a check.</p>
<h2>How long masking takes</h2><p>The masking run rewrites every row of every table that has a rule. <code>06-timing.mjs</code> masked three columns of one table at three sizes, three runs each, with the compute fixed at 1 CU:</p>
<table>
<thead>
<tr>
<th>Rows</th>
<th>Table size</th>
<th>Masking run (median, min to max)</th>
<th>API call to usable branch (median)</th>
</tr>
</thead>
<tbody><tr>
<td>10,000</td>
<td>2.3 MB</td>
<td>3 s (3 to 4)</td>
<td>13.2 s</td>
</tr>
<tr>
<td>100,000</td>
<td>22.1 MB</td>
<td>6 s (5 to 10)</td>
<td>15.6 s</td>
</tr>
<tr>
<td>1,000,000</td>
<td>226.6 MB</td>
<td>22 s (17 to 32)</td>
<td>33.8 s</td>
</tr>
</tbody></table>
<p>The time grew much less than the row count: 100 times the rows took about 7 times as long. It also varied between runs, from 17 to 32 seconds at 1,000,000 rows, and three runs per size cannot show why. Neon reports the run's start and end in whole seconds. The time to a usable branch also includes creating the branch and starting its compute. For our 47.6 MB database it was 20.3 seconds for round 1 and 47.6 seconds for round 3, which rewrote six tables. We did not measure storage.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>One schema and data we wrote.</strong> The scanner found everything we planted, in the formats we chose. Real free text has misspellings, nicknames and names in other scripts, so its recall on real data will be lower, and we cannot say by how much.</li>
<li><strong>Names of people who are not users are invisible.</strong> In our data that left 5,985 names the scanner could not see by design: gift recipients, colleagues, accountants.</li>
<li><strong>The table order is observed, not documented.</strong> With a different schema Neon may mask the audit table first, and then the trigger's copies would stay unmasked.</li>
<li><strong>Small sizes and one region.</strong> One project in <code>aws-eu-central-1</code>, PostgreSQL 18, tables up to 1,000,000 rows, three runs per size.</li>
<li><strong>Static masking only.</strong> A <a href="https://neon.com/blog/branching-environments-anonymized-pii" rel="noopener noreferrer">Neon blog post</a> says dynamic masking is planned. We tested what is available now.</li>
</ul>
<h2>Summary</h2><p>Anonymized branches masked every column we named. The leaks were all in our schema: copies made at checkout, free text, JSON payloads, a trigger, a new table from a migration. A rules file cannot list what nobody knows is there, so treat the masked branch the way you treat a build: test it before anyone gets a connection string.</p>
<p>What to do:</p>
<ol>
<li>Write rules for the obvious columns, then scan the masked branch to find the copies.</li>
<li>Mask audit and history tables in the same rule set, and check row counts after the run.</li>
<li>Replace JSON columns that hold PII. We found no rule that edits JSON in place.</li>
<li>Decide what to do with free text. <code>regexp_replace</code> removes emails and phone numbers, not names.</li>
<li>Use Random Unique Email (or a value derived from the primary key) for unique email columns, and keep fake emails on <code>example.com</code>.</li>
<li>Gate on the scan, wait for the masking run before you connect, and make previews only from a template that passed. Do not give the template an expiry.</li>
</ol>
<p>Run it against your own project:</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/neon-anonymized-branches
<span class="hljs-built_in">cd</span> neon-anonymized-branches &amp;&amp; <span class="hljs-built_in">cp</span> .env.example .<span class="hljs-built_in">env</span>   <span class="hljs-comment"># add a Neon API key</span>
npm install &amp;&amp; npm run setup &amp;&amp; npm run seed
node scripts/02-template.mjs --rules rules/v1-obvious.json --name template-v1
npm run scan -- --target template-v1 --known main
</code></pre>]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[eMudhra CA Compromise: Two Roots to Remove, and Dropping Comodo AAA Broke 1 of 16,125 Chains]]></title>
      <link>https://devops-daily.com/posts/emudhra-ca-compromise-trust-stores-comodo-aaa</link>
      <description><![CDATA[Browsers blocked the compromised eMudhra TLS intermediates in hours, but images and runtimes read their own trust stores. We tested 34 stores, real curl, Python, Node.js and Go clients, and the chains that the top 20,000 sites send. Debian had to remove two roots because of a Sectigo cross-sign. Removing Comodo AAA broke 1 of 16,125 chains in our OpenSSL test.]]></description>
      <pubDate>Tue, 03 Nov 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/emudhra-ca-compromise-trust-stores-comodo-aaa</guid>
      <category><![CDATA[Security]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Security]]></category><category><![CDATA[TLS]]></category><category><![CDATA[Certificates]]></category><category><![CDATA[Docker]]></category><category><![CDATA[Debian]]></category><category><![CDATA[Java]]></category><category><![CDATA[Node.js]]></category><category><![CDATA[Go]]></category>
      <content:encoded><![CDATA[<p>Firefox and Chrome stopped trusting the six compromised eMudhra TLS intermediate CAs within hours of the report on 17 October 2026. Your containers, JVMs, virtualenvs and Go binaries did not get that update. Each of them reads its own copy of a trust store, and that copy decides whether a forged certificate for any hostname is accepted.</p>
<p>We collected 34 trust stores from package files, published bundles and source trees, and verified the public test chains for the compromised keys against each one. Then we connected real clients to the public test hosts, and checked what Debian's fix does to the chains that 16,348 of the top 20,000 sites send.</p>
<p>Three results matter if you run anything outside a browser:</p>
<ul>
<li><strong>Two roots, not one.</strong> Each compromised TLS key chains to <code>emSign ECC Root CA - G3</code> and, through a Sectigo cross-sign, to <code>AAA Certificate Services</code>. Before the fix, Debian 11, 12 and 13, Ubuntu 24.04 as released, Alpine 3.18 and OpenJDK's <code>cacerts</code> held both. There, removing only the emSign root leaves the Sectigo path open.</li>
<li><strong>The root list predicts everything.</strong> In 560 OpenSSL verifications, the two roots decided every result. The fixed Go, Node.js, Bun and CPython releases block the keys in code. Node.js 24.22.1 and Go 1.26.10 rejected them even with the old Debian 12 bundle.</li>
<li><strong>Removing AAA broke one chain in our test.</strong> Of 16,125 served chains that verified before Debian's fix, one (<code>jisc.ac.uk</code>) failed after it with OpenSSL 3. GnuTLS and Go also failed on that site. 262 more fail only for a client that never looks for another path.</li>
</ul>
<h2>TL;DR</h2><ul>
<li>Update <code>ca-certificates</code> and rebuild images from a fresh base. A digest pinned before 20 October keeps the old bundle.</li>
<li>Debian 11, Alpine 3.18 and 3.19 and Node.js 20 get no fix. Their newest releases still trust the keys.</li>
<li>On Debian and Ubuntu, delete <code>/etc/ssl/certs/java/cacerts</code> and run <code>update-ca-certificates --fresh</code>; the Java keystore is not rebuilt on its own.</li>
<li>Upgrade to runtimes that block the keys in code: Go 1.26.10 or 1.27.3 (then rebuild your Go binaries), Node.js 22.23.4, 24.22.1 or 26.12.1, Bun 1.4.4, Python 3.11.18, 3.12.16, 3.13.18, 3.14.10 or 3.15.2.</li>
<li>If you run a server, do not end your chain at a certificate issued by <code>AAA Certificate Services</code>.</li>
<li>Run <a href="https://github.com/The-DevOps-Daily/emudhra-trust-store-check" rel="noopener noreferrer"><code>scan-tree.py</code></a> on an unpacked image, a virtualenv or a JDK to find what still trusts the keys.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Container images, CI runners or servers that make outbound TLS connections</li>
<li>Python 3.9 or later and the OpenSSL 3 command line tool, to run the companion scripts</li>
<li><code>docker</code> (or any tool that can export an image filesystem) if you want to scan images</li>
</ul>
<h2>What happened, in short</h2><p>On 17 October, a researcher reported to the Apple, Chrome, Microsoft and Mozilla root programs that the private keys of eight eMudhra intermediate CAs could be computed from their public certificates. Mozilla confirmed it and had Firefox blocking the TLS intermediates through OneCRL by 22:59 UTC the same day. Six of the eight CAs issue TLS certificates; two are code signing CAs (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=2083449" rel="noopener noreferrer">Mozilla bug 2083449</a>).</p>
<p>The keys are not theoretical. Between 20 and 21 October, people other than eMudhra logged 107,107 certificates signed with six of the keys to Certificate Transparency. 106,050 of them are proof-of-concept certificates for <code>google.com</code> (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c44" rel="noopener noreferrer">comment 44</a>). Almost none of them carry a revocation URL.</p>
<p>Browsers get blocks through OneCRL, CRLSets and Apple's and Microsoft's trust updates. Almost everything else reads a file. We used the public chains and test hosts of the <a href="https://emudhra.we-have-a-problem.com/" rel="noopener noreferrer">incident tracker</a>, which also lists the patched versions.</p>
<table>
<thead>
<tr>
<th>Date (UTC)</th>
<th>What changed</th>
</tr>
</thead>
<tbody><tr>
<td>17 to 19 Oct</td>
<td>Firefox (OneCRL), Chrome (CRLSets) and Apple block the certificates; eMudhra and Sectigo revoke them</td>
</tr>
<tr>
<td>20 Oct</td>
<td>Go 1.26.10 and 1.27.3; Debian <code>20250419~deb12u2</code> and <code>20250419+deb13u1</code>; Ubuntu <code>20260816ubuntu0.*</code>; certifi 2026.10.19; curl CA bundle 2026-10-20</td>
</tr>
<tr>
<td>21 Oct</td>
<td>Node.js 22.23.4, 24.22.1 and 26.12.1</td>
</tr>
<tr>
<td>22 Oct</td>
<td>Bun 1.4.4</td>
</tr>
<tr>
<td>24 to 27 Oct</td>
<td>OpenJDK removes four roots from its source tree (JDK-8395690); no OpenJDK update release yet</td>
</tr>
<tr>
<td>26 Oct</td>
<td>Python 3.15.2, 3.14.10, 3.13.18, 3.12.16 and 3.11.18</td>
</tr>
</tbody></table>
<h2>One key, two certificates, two roots</h2><p>Each of the six TLS keys appears in two intermediate certificates with the same subject and the same public key. One was issued by <code>emSign Root TLS CA - G3</code>, which is cross-signed by <code>emSign ECC Root CA - G3</code>. The other was cross-signed by Sectigo's <code>AAA Certificate Services</code> (the old Comodo root, called <code>Comodo AAA Services root</code> in Mozilla's data). A leaf signed with the key verifies through either one.</p>
<ol>
<li><strong>Forged leaf</strong> any hostname</li>
<li><strong>EM DV TLS CA - G3B</strong> issued by eMudhra</li>
<li><strong>EM DV TLS CA - G3B</strong> cross-signed by Sectigo</li>
<li><strong>emSign Root TLS CA - G3</strong> cross-signed</li>
<li><strong>AAA Certificate Services</strong> Comodo, 2004</li>
<li><strong>emSign ECC Root CA - G3</strong> removed 20 Oct</li>
</ol>
<p>Connections:</p>
<ul>
<li>Forged leaf -&gt; EM DV TLS CA - G3B</li>
<li>Forged leaf -&gt; EM DV TLS CA - G3B</li>
<li>EM DV TLS CA - G3B -&gt; emSign Root TLS CA - G3</li>
<li>EM DV TLS CA - G3B -&gt; AAA Certificate Services</li>
<li>emSign Root TLS CA - G3 -&gt; emSign ECC Root CA - G3</li>
</ul>
<p>Mozilla stopped trusting <code>AAA Certificate Services</code> for TLS in April 2025. Its root data (NSS builtins) marks the root as trusted for server authentication in version 2.74. Version 2.76, which came with NSS 3.111, does not (<a href="https://github.com/The-DevOps-Daily/emudhra-trust-store-check/blob/main/results/aaa-trust-history.txt" rel="noopener noreferrer"><code>aaa-trust-history.txt</code></a> reads it from the Firefox source). Debian's <code>20250419</code> package is built from 2.74, so Debian 11, 12 and 13 hold AAA, and so do Ubuntu 24.04's release-pocket package and Alpine 3.18. Before the fix, bundles built from later data held only the emSign root. These include Ubuntu's June 2026 update, Alpine 3.19, certifi, curl, Node.js, Bun, RHEL and Temurin. Debian's own August 2026 unstable package, built from 2.90, has no AAA either. So the cause is the age of the data, not Debian's packaging. That is why Debian removed two roots and the others removed one.</p>
<h2>Finding 1: the root list decides it</h2><p>Every script, recorded result and the scanner are in the companion repository. Each stage writes a file the next stage reads, and every number below comes from one of those files.</p>
<p><a href="https://github.com/The-DevOps-Daily/emudhra-trust-store-check" rel="noopener noreferrer">The-DevOps-Daily/emudhra-trust-store-check on GitHub</a></p>
<p>We took each trust store from a fixed file (a package, a published bundle or a source tree), so we know exactly which version we tested. The repository records the URL and SHA-256 of every download. For each store, a script first predicted the result: it accepts the emSign path if it holds <code>emSign ECC Root CA - G3</code>, and the Sectigo path if it holds <code>AAA Certificate Services</code>. Then <code>openssl verify</code> checked the 16 public test chains against each store, with the clock fixed at 30 October 00:00 UTC. There are eight chains per path, including leaves never logged to CT and chains through a rollover certificate.</p>
<p>The result: 560 verifications, 0 that differed from the prediction. A store accepts all eight chains of a path, or none.</p>
<table>
<thead>
<tr>
<th>Trust store</th>
<th>Before the fix</th>
<th>Fixed in</th>
</tr>
</thead>
<tbody><tr>
<td>Debian 11 bullseye</td>
<td>both paths (<code>20250419~deb12u1~deb11u1</code>, the newest)</td>
<td>no fix, LTS ended 31 August 2026</td>
</tr>
<tr>
<td>Debian 12 bookworm</td>
<td>both paths (<code>20250419~deb12u1</code>)</td>
<td><code>20250419~deb12u2</code></td>
</tr>
<tr>
<td>Debian 13 trixie</td>
<td>both paths (<code>20250419</code>)</td>
<td><code>20250419+deb13u1</code></td>
</tr>
<tr>
<td>Ubuntu 24.04</td>
<td>both paths (<code>20240203</code>, release pocket); emSign path (<code>20260601~24.04.1</code>)</td>
<td><code>20260816ubuntu0.24.04.1</code></td>
</tr>
<tr>
<td>Ubuntu 22.04</td>
<td>emSign path (<code>20260601~22.04.1</code>)</td>
<td><code>20260816ubuntu0.22.04.1</code></td>
</tr>
<tr>
<td>Alpine 3.18, 3.19 (EOL)</td>
<td>both paths (3.18), emSign path (3.19)</td>
<td>no fix in either branch</td>
</tr>
<tr>
<td>Alpine 3.22</td>
<td>not tested</td>
<td><code>20261020-r0</code> (also in 3.20, 3.21, 3.23, 3.24 and edge)</td>
</tr>
<tr>
<td>RHEL 9 family (p11-kit, parsed)</td>
<td>emSign path (<code>2025.2.80_v9.0.305-91.el9</code>)</td>
<td><code>2026.2.92_v9.0.319-90.3.el9_8</code></td>
</tr>
<tr>
<td>Python certifi</td>
<td>emSign path (2025.8.3, 2026.7.22)</td>
<td>2026.10.18</td>
</tr>
<tr>
<td>curl CA bundle</td>
<td>emSign path (2026-09-25)</td>
<td>2026-10-20</td>
</tr>
<tr>
<td>Node.js bundled roots</td>
<td>emSign path (20.20.2, 22.23.3, 24.22.0)</td>
<td>22.23.4, 24.22.1</td>
</tr>
<tr>
<td>Bun bundled roots</td>
<td>emSign path (1.4.3)</td>
<td>1.4.4</td>
</tr>
<tr>
<td>OpenJDK source <code>cacerts</code></td>
<td>both paths (jdk21u before JDK-8395690)</td>
<td>in source since 27 October; no update release yet</td>
</tr>
<tr>
<td>Eclipse Temurin JRE <code>cacerts</code></td>
<td>emSign path (21.0.12.1+1; 11.0.32.1+1, the newest 11 build)</td>
<td>21.0.13+9 holds neither root</td>
</tr>
</tbody></table>
<p>Three details from the table are easy to miss:</p>
<ul>
<li><strong>Temurin is not OpenJDK.</strong> OpenJDK's own <code>cacerts</code> holds both roots, and OpenJDK <a href="https://openjdk.org/emudhra" rel="noopener noreferrer">tells you</a> to delete four of them by hand. Temurin builds its <code>cacerts</code> from Mozilla's data, so its 23 October release already holds neither. A JDK from another vendor can be either; check the file, not the version.</li>
<li><strong>certifi over-removed first.</strong> certifi 2026.10.18 removed every emSign root, including <code>emSign Root CA - G1</code>, which Mozilla still trusts. Users of the InCommon certificate service, which many US universities use, reported breakage in <a href="https://github.com/certifi/python-certifi/issues/439" rel="noopener noreferrer">issue #439</a>, and 2026.10.19 put the unaffected roots back. In our scan below, 47 of the served chains end at <code>emSign Root CA - G1</code>.</li>
<li><strong>Removing one root is not enough everywhere.</strong> We built one synthetic store, Debian 12's old bundle minus only <code>emSign ECC Root CA - G3</code>. It rejected all eight emSign-path chains and accepted all eight Sectigo-path chains. That is what you get if you apply the one-root workaround, which is right for Alpine 3.19 and later, to a Debian image.</li>
</ul>
<p><strong>the Sectigo-path chain against three Debian 12 bundles</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># old bundle: holds both roots</span>
$ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile debian-12-before.pem -untrusted aaa-chain.pem aaa-chain.pem
aaa-chain.pem: OK
<span class="hljs-comment"># old bundle minus emSign ECC Root CA - G3 only</span>
$ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile minus-emsign.pem -untrusted aaa-chain.pem aaa-chain.pem
aaa-chain.pem: OK
<span class="hljs-comment"># 20250419~deb12u2: both roots removed</span>
$ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile debian-12-fixed.pem -untrusted aaa-chain.pem aaa-chain.pem
C = NL, O = eMudhra BV, CN = EM DV TLS CA - G3B
error 20 at 1 depth lookup: unable to get <span class="hljs-built_in">local</span> issuer certificate
error aaa-chain.pem: verification failed
</code></pre><p><code>-no-CApath -no-CAstore</code> keeps the machine's own <code>/etc/ssl/certs</code> out of the test, and <code>-attime</code> makes the result repeatable. This command does not check revocation. The 16 intermediate certificates were revoked on 18 and 19 October. In the next section, our client commands did not turn on CRL or OCSP checking either, and with the old bundles they accepted the two public test chains.</p>
<h2>Finding 2: a blocklist in the runtime beats a stale store</h2><p>Four runtimes did not wait for trust stores. They are Go 1.26.10 and 1.27.3 (<a href="https://go.dev/issue/82345" rel="noopener noreferrer">go.dev/issue/82345</a>), Node.js 22.23.4, 24.22.1 and 26.12.1, Bun 1.4.4, and the October 26 CPython releases. Each one compiles a list of the eight compromised public keys into its TLS code. It rejects any chain that contains one of them. Bun's release notes mention only a root update; the list is in <code>blocked_spkis.cpp</code>. Our first stage downloads the four source files from the fix commits. All four hold the same eight SHA-256 hashes of the SubjectPublicKeyInfo. The lists match the key, not a certificate. So they cover both certificates of every key, and the rollover chains.</p>
<p>To see the difference with real clients, we connected curl, Python, Node.js and Go on a Debian 12 machine to two of the public test hosts: one serves the emSign path, the other the Sectigo path. Each client got a trust store from the matrix above. For Go, we built the same small HTTPS client with Go 1.26.9 and 1.26.10. All 24 results matched the prediction:</p>
<table>
<thead>
<tr>
<th>Client</th>
<th>Trust store</th>
<th>emSign path</th>
<th>Sectigo path</th>
</tr>
</thead>
<tbody><tr>
<td>curl 7.88.1 (OpenSSL 3.0)</td>
<td>Debian 12 <code>20250419~deb12u1</code></td>
<td>accepted</td>
<td>accepted</td>
</tr>
<tr>
<td>curl 7.88.1</td>
<td>Ubuntu 24.04 <code>20260601~24.04.1</code></td>
<td>accepted</td>
<td>rejected</td>
</tr>
<tr>
<td>curl 7.88.1</td>
<td>Debian 12 <code>20250419~deb12u2</code></td>
<td>rejected</td>
<td>rejected</td>
</tr>
<tr>
<td>Python 3.11.2 (Debian 12)</td>
<td>Debian 12 <code>20250419~deb12u1</code></td>
<td>accepted</td>
<td>accepted</td>
</tr>
<tr>
<td>Python 3.11.2</td>
<td>certifi 2026.7.22</td>
<td>accepted</td>
<td>rejected</td>
</tr>
<tr>
<td>Python 3.11.2</td>
<td>certifi 2026.10.19</td>
<td>rejected</td>
<td>rejected</td>
</tr>
<tr>
<td>Node.js 24.22.0</td>
<td>bundled roots</td>
<td>accepted</td>
<td>rejected</td>
</tr>
<tr>
<td>Node.js 24.22.0</td>
<td><code>--use-openssl-ca</code>, Debian 12 <code>20250419~deb12u1</code></td>
<td>accepted</td>
<td>accepted</td>
</tr>
<tr>
<td>Node.js 24.22.1</td>
<td>bundled roots</td>
<td>rejected</td>
<td>rejected</td>
</tr>
<tr>
<td>Node.js 24.22.1</td>
<td><code>--use-openssl-ca</code>, Debian 12 <code>20250419~deb12u1</code></td>
<td>rejected</td>
<td>rejected</td>
</tr>
<tr>
<td>Go 1.26.9</td>
<td><code>SSL_CERT_FILE</code>, Debian 12 <code>20250419~deb12u1</code></td>
<td>accepted</td>
<td>accepted</td>
</tr>
<tr>
<td>Go 1.26.10</td>
<td><code>SSL_CERT_FILE</code>, Debian 12 <code>20250419~deb12u1</code></td>
<td>rejected</td>
<td>rejected</td>
</tr>
</tbody></table>
<p>With the old Debian bundle, curl, Python, Node.js 24.22.0 and Go 1.26.9 accept a chain signed with a compromised key on both paths. Node.js 24.22.1 and Go 1.26.10 with the same bundle refuse both. Go 1.26.10 reports <code>x509: no valid chains built</code>, and Node.js reports <code>CERT_REVOKED</code>:</p>
<p><strong>same stale bundle, two Node.js versions (30 October, 11:25 UTC)</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># debian-12-before.pem is ca-certificates 20250419~deb12u1; get.js is in tools/</span>
$ SSL_CERT_FILE=debian-12-before.pem SSL_CERT_DIR=/nonexistent ./node-24.22.0 --use-openssl-ca get.js dv-g3b-aaa-emudhra.we-have-a-problem.com
ACCEPT HTTP 200
$ SSL_CERT_FILE=debian-12-before.pem SSL_CERT_DIR=/nonexistent ./node-24.22.1 --use-openssl-ca get.js dv-g3b-aaa-emudhra.we-have-a-problem.com
reject CERT_REVOKED: certificate revoked
</code></pre><p>Debian 12's own Python (<code>3.11.2-6+deb12u9</code> on our machine) has no denylist, and OpenJDK fixed Java only by removing roots from <code>cacerts</code>. Both are only as safe as the bundle they read. On macOS, Apple <a href="https://support.apple.com/en-us/149555" rel="noopener noreferrer">says</a> that the system <code>curl</code>, <code>git</code> and <code>python3</code> check <code>/etc/ssl/cert.pem</code> and accept these chains until a future update.</p>
<h2>Finding 3: what removing AAA breaks</h2><p>Removing <code>AAA Certificate Services</code> closes the Sectigo path, but other chains still use this 2004 root, and Apple and the incident tracker both warn that removing it can break sites. We measured how many.</p>
<p>We connected once to each of the top 20,000 domains of the Tranco list <code>PYK6J</code>, or to <code>www.</code> if the bare name failed. All connections came from one IPv4 address on 30 October between 11:11 and 11:36 UTC, and we saved the chain each server sent. 16,348 answered. Then a small C program on OpenSSL 3.0.22 verified every chain against Debian 12's bundle before and after the fix. Debian 13's bundles hold the same certificates. The program verified each chain in three ways:</p>
<ul>
<li><strong>modern</strong>: the OpenSSL default since 1.1.0, which looks in the trust store first at each step</li>
<li><strong>nofirst</strong>: trust store first turned off, alternative chain search on</li>
<li><strong>strict</strong>: both turned off, so the verifier follows only the chain the server sent</li>
</ul>
<p>Of the 16,348 chains, 16,125 verified before the fix in modern mode. 1,296 contain a Sectigo, Comodo, USERTrust or AAA certificate, and 283 include a certificate that AAA issued. After the fix:</p>
<table>
<thead>
<tr>
<th>Mode</th>
<th>Chains that verified before the fix</th>
<th>Anchored at AAA before the fix</th>
<th>Broken by the fix</th>
</tr>
</thead>
<tbody><tr>
<td>modern</td>
<td>16,125</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>nofirst</td>
<td>16,124</td>
<td>262</td>
<td>1</td>
</tr>
<tr>
<td>strict</td>
<td>16,111</td>
<td>262</td>
<td>262</td>
</tr>
</tbody></table>
<p>No chain was anchored at <code>emSign ECC Root CA - G3</code>, so every break comes from AAA. Most of the 262 servers send an old copy of a current root, such as <code>USERTrust RSA Certification Authority</code>, cross-signed by AAA. A modern verifier finds the self-signed copy of that root in the store first and never reaches AAA. A verifier that follows only the served chain gets to AAA, does not find it, and fails:</p>
<p><strong>Chains that break only for a client that follows the served chain</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>USERTrust RSA Certification Authority</td>
<td>193</td>
</tr>
<tr>
<td>SSL.com TLS ECC or RSA Root CA 2022</td>
<td>31</td>
</tr>
<tr>
<td>Sectigo Public Server Authentication Root R46</td>
<td>23</td>
</tr>
<tr>
<td>USERTrust ECC Certification Authority</td>
<td>12</td>
</tr>
<tr>
<td>COMODO RSA Certification Authority</td>
<td>2</td>
</tr>
<tr>
<td>SSL.com TLS Issuing ECC CA R2</td>
<td>1</td>
</tr>
</tbody></table>
<p><em>262 of the 16,111 chains that verified in strict mode before the fix, Tranco PYK6J top 20,000, scanned 30 October 2026. Grouped by the certificate issued by AAA Certificate Services that the server sends. In modern mode, OpenSSL 3.0 accepts all but one of them after the fix.</em></p>
<p>The one real break is <code>jisc.ac.uk</code>, and it is not a Sectigo certificate. The server sends its leaf and <code>SSL.com TLS Issuing ECC CA R2</code>, cross-signed by AAA, and nothing else. The same key also has a certificate from <code>SSL.com TLS ECC Root CA 2022</code> (<a href="https://crt.sh/?id=8505503579" rel="noopener noreferrer">crt.sh 8505503579</a>), which Debian holds. The server does not send it, and OpenSSL does not fetch missing intermediates. So on our Debian 12 machine after the fix, curl fails:</p>
<p><strong>jisc.ac.uk before and after Debian removed AAA</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Debian 12, ca-certificates 20250419~deb12u2, 30 October 11:44 UTC</span>
$ curl -sS -o /dev/null --max-time 20 -w <span class="hljs-string">'%{http_code}\n'</span>  https://jisc.ac.uk/
curl: (60) SSL certificate problem: unable to get <span class="hljs-built_in">local</span> issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
000
$ curl -sS -o /dev/null --max-time 20 -w <span class="hljs-string">'%{http_code}\n'</span> --cacert stores/debian-12-before.pem --capath /nonexistent https://jisc.ac.uk/
301
</code></pre><p>To check this with a second TLS stack, we ran <code>git ls-remote</code> with both bundles. On Debian 12, git 2.39.5 uses GnuTLS 3.7.9. The targets were <code>jisc.ac.uk</code> and a seeded random sample of 20 of the other 261 strict-mode breaks. GnuTLS rejected <code>jisc.ac.uk</code> after the fix and accepted the 20 others. Our Go client, built with Go 1.26.9 and 1.26.10, also failed on <code>jisc.ac.uk</code> with the fixed bundle (<code>x509: certificate signed by unknown authority</code>).</p>
<p>So in our tests the AAA removal costs little: one chain in 16,125 for OpenSSL 3, and GnuTLS and Go agreed on the chains we gave them. The risk is in two places. The first is servers that send only an AAA cross-sign of an intermediate, like <code>jisc.ac.uk</code>; the fix is to send the intermediate that the current root issued. The second is clients that cannot search for another path, which we modelled but did not identify.</p>
<h2>Where the old roots hide</h2><p>Updating the host does not update what your services read. Check these first:</p>
<ol>
<li><strong>Images built before 20 October</strong>, and every image pinned by digest. The bundle is a file in a layer. Docker's own <a href="https://www.docker.com/blog/php2shell-and-emudhra-patched/" rel="noopener noreferrer">note</a> puts it plainly: an image "keeps trusting them until it is rebuilt".</li>
<li><strong>Bases and runtimes that will not get a fix.</strong> <code>debian:bullseye</code> and everything built on it, <code>alpine:3.18</code> and <code>alpine:3.19</code>: their newest <code>ca-certificates</code> packages still trust both paths or the emSign path. Node.js 20, end of life since 30 April 2026, still bundles the emSign root and has no blocklist.</li>
<li><strong>The Java keystore on Debian and Ubuntu.</strong> <code>ca-certificates-java</code> does not rebuild <code>/etc/ssl/certs/java/cacerts</code> when <code>ca-certificates</code> changes, so OpenJDK packages keep the old roots after the update (<a href="https://openjdk.org/emudhra#debian" rel="noopener noreferrer">OpenJDK guidance</a>).</li>
<li><strong>Bundles copied by hand.</strong> A <code>COPY --from=build /etc/ssl/certs/ca-certificates.crt</code> into a <code>scratch</code> or distroless image, a <code>certifi</code> pin in a lockfile, or <code>SSL_CERT_FILE</code>, <code>REQUESTS_CA_BUNDLE</code>, <code>PIP_CERT</code> and <code>NODE_EXTRA_CA_CERTS</code> pointing at a file from last year.</li>
<li><strong>Static Go binaries built with Go older than 1.26.10.</strong> The blocklist is compiled in, so only a rebuild adds it. <code>go version -m ./binary</code> prints the Go version that built it.</li>
<li><strong>Runtimes with their own roots.</strong> Node.js before 22.23.4 or 24.22.1 and Bun before 1.4.4 use bundled roots by default, not the system store.</li>
</ol>
<p>To find them, the companion repository has <code>scan-tree.py</code>. It reads a directory tree and reports bundles, keystores and binaries that hold either root, runtimes that carry the blocklist, and Go programs and CPython <code>ssl</code> modules that do not. This is its real output on a test tree with six files: the old Debian 12 bundle, Node.js 24.22.0 and 24.22.1, a fixed certifi bundle (correctly silent), the <code>ssl</code> module of Debian 12's Python, and Temurin 11's <code>cacerts</code>:</p>
<pre><code class="hljs language-text" tabindex="0">TRUSTS   fixture/etc/ssl/certs/ca-certificates.crt
         CA bundle with 150 certificates holds emSign ECC Root CA - G3 and AAA Certificate Services
TRUSTS   fixture/usr/local/bin/node
         embeds emSign ECC Root CA - G3 in its bundled roots
TRUSTS*  fixture/usr/lib/python3.11/lib-dynload/_ssl.cpython-311-aarch64-linux-gnu.so
         CPython ssl module without the eMudhra blocklist: it accepts whatever its trust store accepts
TRUSTS   fixture/usr/lib/jvm/temurin-11/lib/security/cacerts
         CA bundle with 118 certificates holds emSign ECC Root CA - G3
ok       fixture/opt/node-24.22.1/bin/node
         holds all 8 eMudhra blocklist keys (a byte match, not a runtime test)

4 finding(s) that still trust a chain through the compromised keys.
TRUSTS*: safe only if every trust store the program reads at run time is fixed.
No finding is not proof: only the bundle, keystore and binary formats listed in --help are checked.
</code></pre><p><code>TRUSTS*</code> means the program has no blocklist of its own, so it is only as safe as the bundle it reads. An <code>ok</code> is a byte match for all eight keys, not a test of the running program. For a container image, export the filesystem and scan it:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">mkdir</span> rootfs &amp;&amp; docker <span class="hljs-built_in">export</span> <span class="hljs-string">"<span class="hljs-subst">$(docker create IMAGE)</span>"</span> | tar -xf - -C rootfs
python3 scan-tree.py rootfs
</code></pre><p>Then fix what it finds:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Debian and Ubuntu images: upgrade the bundle, then rebuild the Java keystore</span>
apt-get update &amp;&amp; apt-get install --only-upgrade -y ca-certificates
<span class="hljs-built_in">rm</span> -f /etc/ssl/certs/java/cacerts &amp;&amp; update-ca-certificates --fresh

<span class="hljs-comment"># Alpine 3.20 and later (3.18 and 3.19 get no fix: move to a newer base)</span>
apk upgrade --no-cache ca-certificates ca-certificates-bundle

<span class="hljs-comment"># Check what an image actually has</span>
dpkg-query -W ca-certificates   <span class="hljs-comment"># fixed: 20250419~deb12u2, 20250419+deb13u1, 20260816ubuntu0.*</span>
apk info -v ca-certificates-bundle   <span class="hljs-comment"># fixed: 20261020-r0</span>
rpm -q ca-certificates   <span class="hljs-comment"># RHEL 9 fixed: 2026.2.92_v9.0.319-90.3.el9_8</span>
</code></pre><h2>What we could not conclude</h2><ul>
<li><strong>How many systems are still exposed.</strong> The matrix says what each version does, not how many copies of each version are running. We picked old versions on purpose.</li>
<li><strong>Other JDKs and platform verifiers.</strong> We did not check the <code>cacerts</code> of Amazon Corretto, Azul Zulu, Microsoft's build, Oracle JDK or distribution packages. We did not test Windows, macOS or Android, and relied on the vendors for those.</li>
<li><strong>RHEL's real extraction.</strong> We did not run <code>trust extract</code>; we parsed the p11-kit file and kept certificates trusted for server authentication. Our fixed RHEL 9 store has 118 certificates, the same count as the curl bundle of 20 October, which is a cross-check, not proof.</li>
<li><strong>Revocation.</strong> We did not test clients with CRL or OCSP checks turned on. The leaves on the test hosts have no revocation pointers. The eMudhra CRL endpoint also refused some User-Agents (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c44" rel="noopener noreferrer">comment 44</a>), so we would not count on it.</li>
<li><strong>AAA beyond the top 20,000.</strong> One IPv4 address, one morning, the bare name or <code>www.</code> only. Internal services, other host names, mail servers and the long tail can differ, and a CDN can send a different chain to a different network.</li>
<li><strong>Which real clients behave like "strict" mode.</strong> We modelled them with two OpenSSL flags. OpenSSL 3.0 in modern mode handled 261 of the 262 chains, and GnuTLS 3.7.9 the 20 we sampled. We did not test Java, rustls, Windows, macOS or embedded TLS libraries on them, or Go beyond <code>jisc.ac.uk</code>.</li>
</ul>
<h2>Summary</h2><ul>
<li>Each compromised key chains to two roots. Stores built from Mozilla data older than April 2025, including Debian 11, 12 and 13, need both <code>emSign ECC Root CA - G3</code> and <code>AAA Certificate Services</code> removed.</li>
<li>The roots in a store predicted all 560 of our verifications. Check the file, not the version string; <code>scan-tree.py</code> does that for a whole image.</li>
<li>The fixed Go, Node.js, Bun and Python releases block the keys whatever the store says. Debian 11, Alpine 3.18 and 3.19 and Node.js 20 get no fix.</li>
<li>In our OpenSSL test, removing AAA broke 1 of 16,125 served chains (<code>jisc.ac.uk</code>). GnuTLS and Go failed on it too, and the fix for it is on the server.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Docker Build Cache on Ephemeral CI Runners: inline, registry and local, Measured]]></title>
      <link>https://devops-daily.com/posts/docker-build-cache-backends-on-ephemeral-ci-runners</link>
      <description><![CDATA[A fresh CI runner starts with no BuildKit cache. We rebuilt a real multi-stage Node service on a new builder every time, with no cache, inline, registry min and max, and a local cache directory, and measured cold builds, a one-line change, a dependency bump, the export cost and the cache size.]]></description>
      <pubDate>Mon, 02 Nov 2026 19:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/docker-build-cache-backends-on-ephemeral-ci-runners</guid>
      <category><![CDATA[Docker]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Docker]]></category><category><![CDATA[BuildKit]]></category><category><![CDATA[CI/CD]]></category><category><![CDATA[Caching]]></category><category><![CDATA[Docker Buildx]]></category><category><![CDATA[Performance]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>A CI runner that starts empty has no BuildKit cache, so every <code>docker build</code> is a cold build unless you export the cache somewhere and import it on the next run. We rebuilt a real multi-stage Node service, the payment service from Google's Online Boutique demo, on a brand-new BuildKit builder every time, with five setups: no cache, <code>inline</code>, <code>registry</code> with <code>mode=min</code>, <code>registry</code> with <code>mode=max</code>, and a <code>local</code> cache directory.</p>
<p>For a one-line source change, every cache worked. The rebuild went from 79.0 s with no cache to 19.2 to 26.3 s with any registry-backed cache, because BuildKit skipped the whole <code>builder</code> stage, <code>npm install</code> included. For this Dockerfile, the cheapest backends were the best ones. Inline and registry <code>mode=min</code> cost 0.0 and 1.2 s to export. Inline adds only metadata to the image, and <code>mode=min</code> stored a separate 47.7 MB cache with the same layers as the image. <code>mode=max</code> stored 259.9 MB, added 29.8 s of export to every cold build, and made the dependency bump slower, not faster: the cached compiler toolchain saved no time once its layers had to be pulled, and the export added about 20 s. The local directory cache, written fresh each run, was the slowest of all for the one-line change.</p>
<h2>TL;DR</h2><ul>
<li><strong>Source change: all caches skip the <code>builder</code> stage.</strong> Median 19.2 to 26.3 s with inline or registry, against 79.0 s with no cache. In our three runs each, inline and <code>mode=min</code> did as well as <code>mode=max</code>, because BuildKit only loads the cache for the last step that matches.</li>
<li><strong>Export cost is mostly compression.</strong> <code>mode=max</code> spent 27.9 s in <code>preparing build cache for export</code> on a cold build, and about 2 s writing 259.9 MB to the registry. <code>mode=min</code> exported in 1.2 s, inline in 0.0 s.</li>
<li><strong>Dependency bump: <code>mode=max</code> did not help.</strong> It kept the compiler toolchain, but pulling and unpacking the cached base and toolchain layers took 15.0 to 20.6 s per build, about what <code>mode=min</code> spent pulling the base image and running <code>apk add</code> again (13.4 to 19.8 s). Its export then added 18.4 to 24.7 s. Median 132.4 s for <code>mode=max</code> against 110.8 s for <code>mode=min</code>.</li>
<li><strong>Local cache, fresh directory each run:</strong> 74.7 s for a one-line change, because it repeats the full <code>mode=max</code> export every time.</li>
<li><strong>What did not change:</strong> <code>npm install</code> ran on every dependency bump with every backend. A cache saves the steps before the change, never the step that changed.</li>
<li>Our registry ran on the same machine, so these times leave out the network. The sizes carry over: 47.7 MB pulled per warm source build, 259.9 MB per <code>mode=max</code> cache.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Docker with Buildx (we used Docker 29.9.2, Buildx 0.38.1 and BuildKit 0.34.1).</li>
<li>A registry you can push to. We ran <code>registry:2.8.3</code> on the build machine; GHCR, ECR, Docker Hub or your own registry work the same way.</li>
<li>Node.js and npm, only to create the dependency change (<code>npm install --package-lock-only</code>).</li>
<li>If you want to rerun the whole matrix: a 4 vCPU Linux machine and about 80 minutes. We used a droplet on <a href="https://m.do.co/c/2a9bba940f39" rel="sponsored noopener noreferrer">DigitalOcean</a> (referral link).</li>
</ul>
<h2>Why a fresh builder changes everything</h2><p>On a laptop, BuildKit keeps every layer it built in its own store, so the second <code>docker build</code> reuses them without any configuration. A CI runner that starts from a clean VM or container has no store. Unless you tell BuildKit where to import a cache from and where to export it to, every build is a cold build.</p>
<p>BuildKit can export its cache in several ways. <code>inline</code> writes cache metadata into the image you push. <code>registry</code> writes a separate cache manifest to a registry. <code>local</code> writes an OCI layout to a directory that your CI then saves and restores. <code>gha</code> and <code>s3</code> write to GitHub's cache service and to S3. <code>mode=min</code>, the default, exports only the layers of the final image. <code>mode=max</code> also exports the layers of every intermediate stage, such as a <code>builder</code> stage.</p>
<p>The question for a CI pipeline is which of those pays for itself, for the changes that you actually push.</p>
<h2>How we measured</h2><p><strong>The Dockerfile.</strong> We used <code>src/paymentservice</code> from Google's <a href="https://github.com/GoogleCloudPlatform/microservices-demo" rel="noopener noreferrer">Online Boutique</a> demo at commit <code>c7cb6d5b</code>, without changes. It is a real multi-stage Node service, and both base images are pinned by digest. This is the whole Dockerfile without its license header:</p>
<pre><code class="hljs language-dockerfile" tabindex="0"><span class="hljs-comment"># Define a default value so it's not empty if the builder fails to provide it</span>
<span class="hljs-keyword">ARG</span> BUILDPLATFORM=linux/amd64

<span class="hljs-keyword">FROM</span> --platform=$BUILDPLATFORM node:<span class="hljs-number">24.20</span>.<span class="hljs-number">0</span>-alpine@sha256:e67514e5d0f6c46656005e1b693b2ec9d52e80b641307de684d4a015ba7a4eaf AS builder

<span class="hljs-comment"># Some packages (e.g. @google-cloud/profiler) require additional</span>
<span class="hljs-comment"># deps for post-install scripts</span>
<span class="hljs-keyword">RUN</span><span class="language-bash"> apk add --update --no-cache \
    python3 \
    make \
    g++</span>

<span class="hljs-keyword">WORKDIR</span><span class="language-bash"> /usr/src/app</span>

<span class="hljs-keyword">COPY</span><span class="language-bash"> package*.json ./</span>

<span class="hljs-keyword">RUN</span><span class="language-bash"> npm install --only=production</span>

<span class="hljs-keyword">FROM</span> alpine:<span class="hljs-number">3.24</span>.<span class="hljs-number">1</span>@sha256:<span class="hljs-number">28</span>bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b

<span class="hljs-keyword">RUN</span><span class="language-bash"> apk add --no-cache nodejs</span>

<span class="hljs-keyword">WORKDIR</span><span class="language-bash"> /usr/src/app</span>

<span class="hljs-keyword">COPY</span><span class="language-bash"> --from=builder /usr/src/app/node_modules ./node_modules</span>

<span class="hljs-keyword">COPY</span><span class="language-bash"> . .</span>

<span class="hljs-keyword">EXPOSE</span> <span class="hljs-number">50051</span>

<span class="hljs-keyword">ENTRYPOINT</span><span class="language-bash"> [ <span class="hljs-string">"node"</span>, <span class="hljs-string">"index.js"</span> ]</span>
</code></pre><p>The expensive work is in the <code>builder</code> stage: a compiler toolchain from Alpine's mirror, then <code>npm install</code>. The final stage installs Node from Alpine, copies <code>node_modules</code>, and copies the source last.</p>
<p><strong>An ephemeral runner, every time.</strong> Each build got a new BuildKit builder (<code>docker buildx create --driver docker-container --bootstrap</code>), and we removed it after the build. Nothing survived in the builder's store between builds except what the cache backend exported. The base image mirror and the Docker host itself did persist. We recorded builder start-up separately; it is the same for every backend.</p>
<p><strong>Three scenarios:</strong></p>
<ul>
<li><strong>Cold:</strong> an empty cache. The build exports its cache to the backend.</li>
<li><strong>Source change:</strong> one line in <code>server.js</code> (a log message). The build imports the cache that the last cold run wrote.</li>
<li><strong>Dependency change:</strong> <code>@grpc/grpc-js</code> from 1.14.4 to 1.14.7, made with <code>npm install --package-lock-only --save-exact</code>, the size of a typical Dependabot patch bump. The lockfile diff changes only that package (npm also re-sorted two keys in <code>package.json</code>). It imports the same cache.</li>
</ul>
<p>Warm builds import from the cold cache and export to a separate scratch location, so every warm run starts from the same cache and still pays the export cost that a real pipeline pays.</p>
<p><strong>The backends:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Import</th>
<th>Export</th>
</tr>
</thead>
<tbody><tr>
<td>none</td>
<td>nothing</td>
<td>nothing</td>
</tr>
<tr>
<td>inline</td>
<td><code>type=registry,ref=&lt;image&gt;</code></td>
<td><code>type=inline</code> (cache metadata inside the pushed image)</td>
</tr>
<tr>
<td>registry, min</td>
<td><code>type=registry,ref=&lt;cache&gt;</code></td>
<td><code>type=registry,ref=&lt;cache&gt;,mode=min</code></td>
</tr>
<tr>
<td>registry, max</td>
<td><code>type=registry,ref=&lt;cache&gt;</code></td>
<td><code>type=registry,ref=&lt;cache&gt;,mode=max</code></td>
</tr>
<tr>
<td>local</td>
<td><code>type=local,src=&lt;dir&gt;</code></td>
<td><code>type=local,dest=&lt;dir&gt;,mode=max</code></td>
</tr>
</tbody></table>
<p>Every build pushed its image with <code>--provenance=false --sbom=false</code>, so attestations do not add noise.</p>
<p><strong>Repeats.</strong> We planned five runs of every backend and scenario. The shared droplet ran slower than planned and we had a fixed budget, so we got 5 cold runs per backend, 3 source-change runs, and 3 (2 for the local cache) for the dependency bump. Runs went in round-robin order inside each scenario (run 1 of every backend, then run 2, and so on), which spreads slow periods on the shared host across the backends but does not remove order effects (see the caveats). The tables show the median and the number of runs (with 2 runs, the median is their mean). We dropped the page cache before each build and logged the CPU counters around each one, because the droplet's vCPUs are shared and steal time moves build times.</p>
<p><strong>What we recorded.</strong> Wall time of <code>docker buildx build</code>, BuildKit's own step timings from <code>--progress=plain</code> (which steps were <code>CACHED</code>, how long the export took), the cache size from the registry manifest or the cache directory, and the bytes BuildKit pulled from the cache registry, from the registry's access log.</p>
<p><strong>Where the cache lived.</strong> The registry for <code>inline</code> and <code>type=registry</code> was a <code>registry:2.8.3</code> container on the same droplet, at <code>127.0.0.1:5000</code>. We did not have a container registry we could use at no cost, and DigitalOcean's Spaces object storage has a monthly minimum, so there was no remote store that cost nothing. On loopback, writing the 259.9 MB <code>mode=max</code> cache took about 2 s, so <strong>our export and import times are close to the best case.</strong> On a real runner the same bytes cross a network to GHCR, ECR or your own registry, and the time grows with the size. That is why the tables show bytes next to every time: the bytes carry over to your setup, our seconds do not.</p>
<p><strong>Base images.</strong> Every fresh builder resolved <code>node:24.20.0-alpine</code> and <code>alpine:3.24.1</code>, and downloaded their layers whenever a step needed them. We pointed BuildKit at a pull-through mirror (<code>registry:2.8.3</code> with <code>REGISTRY_PROXY_REMOTEURL</code>, also on the droplet), so Docker Hub's rate limits and speed stay out of the numbers. A real ephemeral runner pulls them from Docker Hub or a mirror on every build that needs them, so add that time to the rows where the <code>FROM</code> step ran.</p>
<p><strong>Package downloads were real.</strong> <code>apk add</code> fetched from Alpine's CDN and <code>npm install</code> fetched from the npm registry whenever those steps ran. That is the cost a cache saves, so we left it in.</p>
<p><strong>Why there is no <code>type=s3</code> or <code>type=gha</code>.</strong> <code>type=s3</code> needs an S3 bucket. The only free option here was an S3 emulator on the same droplet, which would measure the emulator, not S3. <code>type=gha</code> works only inside GitHub Actions. We left both out.</p>
<h2>Cold builds and what the export costs</h2><table>
<thead>
<tr>
<th>Backend</th>
<th>Runs</th>
<th>Cold build</th>
<th>Cache export</th>
<th>of which preparing</th>
<th>Cache size</th>
</tr>
</thead>
<tbody><tr>
<td>none</td>
<td>5</td>
<td>79.0 s</td>
<td>-</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td>inline</td>
<td>5</td>
<td>85.4 s</td>
<td>0.0 s</td>
<td>-</td>
<td>in the image</td>
</tr>
<tr>
<td>registry, <code>mode=min</code></td>
<td>5</td>
<td>89.4 s</td>
<td>1.2 s</td>
<td>0.1 s</td>
<td>47.7 MB</td>
</tr>
<tr>
<td>registry, <code>mode=max</code></td>
<td>5</td>
<td>128.0 s</td>
<td>29.8 s</td>
<td>27.9 s</td>
<td>259.9 MB</td>
</tr>
<tr>
<td>local, <code>mode=max</code></td>
<td>5</td>
<td>122.8 s</td>
<td>39.1 s</td>
<td>30.7 s</td>
<td>259.9 MB</td>
</tr>
</tbody></table>
<p>Cold builds: median of 5 runs. Export is the <code>exporting cache</code> step in the BuildKit log; for inline, the <code>preparing layers for inline cache</code> line.</p>
<p>With no cache, a cold build took a median 79.0 s on a fresh builder. <code>npm install</code> was 45.3 s of that, the compiler toolchain (<code>apk add python3 make g++</code>) 6.5 s, and compressing and pushing the image about 9.5 s.</p>
<p>The totals in the table are noisier than the export column. <code>npm install</code> alone took between 29.2 and 70.9 s across the cold runs, because it downloads from the npm registry, and the droplet reported 8 to 27% CPU steal per build. Read the export column for what each backend costs:</p>
<ul>
<li><strong>inline</strong> costs nothing measurable. BuildKit logged <code>preparing layers for inline cache 0.0s done</code>. The cache is metadata in the image config, and its layers are the image's own.</li>
<li><strong>registry, <code>mode=min</code></strong> added 1.2 s for a 47.7 MB cache: the same 5 layers as the final image, stored under the cache tag.</li>
<li><strong>registry, <code>mode=max</code></strong> added 29.8 s for 259.9 MB in 12 layers. 27.9 s of that was one line in the BuildKit log, <code>preparing build cache for export</code>: BuildKit compressing the <code>builder</code> stage's layers (the Node base image, the compiler toolchain and <code>node_modules</code>), which a <code>mode=min</code> export never touches. Writing the layers to the registry took about 2 s, on loopback.</li>
<li><strong>local, <code>mode=max</code></strong> added 39.1 s for the same 259.9 MB, written as an OCI layout on disk. Preparing took 30.7 s here too.</li>
</ul>
<p>So the expensive part of a <code>mode=max</code> export is compression, and it happens before a byte leaves the machine. On a real network, the upload of 259.9 MB comes on top.</p>
<h2>A one-line source change</h2><table>
<thead>
<tr>
<th>Backend</th>
<th>Runs</th>
<th>Source change</th>
<th>Steps cached (of 8)</th>
<th>Pulling cached layers</th>
<th><code>apk add</code> (builder)</th>
<th><code>npm install</code></th>
<th>Pulled from registry</th>
</tr>
</thead>
<tbody><tr>
<td>none (a cold build)</td>
<td>5</td>
<td>79.0 s</td>
<td>0</td>
<td>-</td>
<td>6.5 s</td>
<td>45.3 s</td>
<td>-</td>
</tr>
<tr>
<td>inline</td>
<td>3</td>
<td>26.3 s</td>
<td>7</td>
<td>16.9 s</td>
<td>0.0 s</td>
<td>0.0 s</td>
<td>47.7 MB</td>
</tr>
<tr>
<td>registry, <code>mode=min</code></td>
<td>3</td>
<td>19.2 s</td>
<td>7</td>
<td>12.4 s</td>
<td>0.0 s</td>
<td>0.0 s</td>
<td>47.7 MB</td>
</tr>
<tr>
<td>registry, <code>mode=max</code></td>
<td>3</td>
<td>22.9 s</td>
<td>7</td>
<td>15.7 s</td>
<td>0.0 s</td>
<td>0.0 s</td>
<td>47.7 MB</td>
</tr>
<tr>
<td>local, <code>mode=max</code></td>
<td>3</td>
<td>74.7 s</td>
<td>7</td>
<td>-</td>
<td>0.0 s</td>
<td>0.0 s</td>
<td>-</td>
</tr>
</tbody></table>
<p>The none row is the cold build. Pulling cached layers is the time BuildKit reported on the cached step that had to be unpacked; pulled from registry is from the cache registry's access log.</p>
<p>Every backend skipped the whole <code>builder</code> stage. BuildKit marked 7 of the 8 build steps <code>CACHED</code>, including <code>RUN npm install</code>, and only <code>COPY . .</code> ran. That was true for inline and registry <code>mode=min</code> too, although those caches do not contain a single layer of the <code>builder</code> stage.</p>
<p>That works because of how BuildKit finds a cache hit. A step's cache key comes from its instruction and the keys of the steps before it, and a <code>COPY</code> from the build context adds a checksum of the files it copies. In this graph, that let BuildKit match the final stage's <code>COPY --from=builder</code> step without running the <code>builder</code> stage. Its <a href="https://github.com/moby/buildkit/blob/v0.34.1/docs/dev/solver.md" rel="noopener noreferrer">solver documentation</a> says it "will not require the cache to be loaded or exists for all the vertexes in the graph but will only load it for the final node that is determined to match cache". Here, that node is <code>COPY --from=builder</code>, which is a layer of the final image, and every backend exported it. The <code>builder</code> stage never ran, and none of its layers were downloaded: every registry build pulled the same 47.7 MB, the final image's layers, even from the 259.9 MB <code>mode=max</code> cache.</p>
<p>What a warm build still paid for was pulling and unpacking those cached layers to put <code>COPY . .</code> on top. BuildKit reports that time on the cached <code>COPY --from</code> step: a median of 12.4 to 16.9 s with the registry backends, out of 19.2 to 26.3 s for the whole build.</p>
<p>The local backend was the exception: 74.7 s. Importing from the directory was quick, but exporting to a fresh directory took 52.4 s, because every run wrote the full <code>mode=max</code> cache again. The registry backends exported to a new tag in the same repository, and that took 1.4 s or less, because the layers they referenced were already in the registry. We exported to a new directory on purpose, to stop the cache from growing; the section on the local cache below explains the trade.</p>
<h2>A dependency bump</h2><table>
<thead>
<tr>
<th>Backend</th>
<th>Runs</th>
<th>Dependency bump</th>
<th>Steps cached (of 8)</th>
<th>Pulling cached layers</th>
<th><code>apk add</code> (builder)</th>
<th><code>npm install</code></th>
<th>Pulled from registry</th>
</tr>
</thead>
<tbody><tr>
<td>none (a cold build)</td>
<td>5</td>
<td>79.0 s</td>
<td>0</td>
<td>-</td>
<td>6.5 s</td>
<td>45.3 s</td>
<td>-</td>
</tr>
<tr>
<td>inline</td>
<td>3</td>
<td>105.7 s</td>
<td>2</td>
<td>2.8 s</td>
<td>9.1 s</td>
<td>60.0 s</td>
<td>26.0 MB</td>
</tr>
<tr>
<td>registry, <code>mode=min</code></td>
<td>3</td>
<td>110.8 s</td>
<td>2</td>
<td>2.3 s</td>
<td>9.9 s</td>
<td>61.5 s</td>
<td>26.0 MB</td>
</tr>
<tr>
<td>registry, <code>mode=max</code></td>
<td>3</td>
<td>132.4 s</td>
<td>4</td>
<td>18.0 s</td>
<td>0.0 s</td>
<td>66.9 s</td>
<td>196.8 MB</td>
</tr>
<tr>
<td>local, <code>mode=max</code></td>
<td>2</td>
<td>132.1 s</td>
<td>4</td>
<td>-</td>
<td>0.0 s</td>
<td>51.9 s</td>
<td>-</td>
</tr>
</tbody></table>
<p>A changed lockfile invalidates <code>COPY package*.json</code> in the <code>builder</code> stage, so <code>npm install</code> ran in every build, with every backend. The backends differ only in the step before it, the compiler toolchain:</p>
<ul>
<li><strong>inline and registry <code>mode=min</code></strong> do not hold the <code>builder</code> stage, so BuildKit pulled the Node base image from the mirror (4.3 to 10.2 s) and ran <code>apk add python3 make g++</code> again (a median 9.9 s). Only the final stage's <code>RUN apk add --no-cache nodejs</code> and <code>WORKDIR</code> came from the cache.</li>
<li><strong>registry <code>mode=max</code> and local</strong> held it, and BuildKit marked the toolchain step <code>CACHED</code>. But to run <code>npm install</code> on top of it, BuildKit first had to pull and unpack the cached layers, the Node base image and the toolchain, including one layer of 112.3 MB. With the registry, that took 15.0 to 20.6 s per build, about the same as the 13.4 to 19.8 s that <code>mode=min</code> spent on the base image and <code>apk add</code>. The cache hit saved nothing.</li>
</ul>
<p>We expected <code>mode=max</code> to win this scenario, and wrote that down before the runs. It did not.</p>
<p>Then the export. With <code>mode=max</code>, BuildKit also had to compress the <code>builder</code> stage's new <code>npm install</code> layer, which the image does not contain, and <code>preparing build cache for export</code> took 19.0 s, against 0.7 s for the whole <code>mode=min</code> export.</p>
<p>Put together, the dependency bump took a median 132.4 s with <code>mode=max</code> and 110.8 s with <code>mode=min</code>, and <code>mode=max</code> was slower in each of the three rounds. The export accounts for most of the gap: <code>mode=max</code> took 18.4 to 24.7 s to export, against under 1 s for <code>mode=min</code>, while the toolchain stage cost about the same either way. <code>mode=max</code> also pulled 196.8 MB from the cache registry per build, against 26.0 MB for <code>mode=min</code>. The local cache, also <code>mode=max</code>, took 132.1 s (2 runs). With a remote registry the gap would grow, because those 196.8 MB and the new <code>npm install</code> layer would cross the network.</p>
<p><strong>Build time on a fresh builder (median)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>none</td>
<td>79 s</td>
<td>source change</td>
</tr>
<tr>
<td>none</td>
<td>79 s</td>
<td>dependency bump</td>
</tr>
<tr>
<td>inline</td>
<td>26.3 s</td>
<td>source change</td>
</tr>
<tr>
<td>inline</td>
<td>105.7 s</td>
<td>dependency bump</td>
</tr>
<tr>
<td>registry, mode=min</td>
<td>19.2 s</td>
<td>source change</td>
</tr>
<tr>
<td>registry, mode=min</td>
<td>110.8 s</td>
<td>dependency bump</td>
</tr>
<tr>
<td>registry, mode=max</td>
<td>22.9 s</td>
<td>source change</td>
</tr>
<tr>
<td>registry, mode=max</td>
<td>132.4 s</td>
<td>dependency bump</td>
</tr>
<tr>
<td>local, mode=max</td>
<td>74.7 s</td>
<td>source change</td>
</tr>
<tr>
<td>local, mode=max</td>
<td>132.1 s</td>
<td>dependency bump</td>
</tr>
</tbody></table>
<p><em>One new BuildKit builder per build; cache registry on the same droplet (loopback). Medians; the none bars are the cold build; run counts are in the tables. s-4vcpu-8gb, 2 November 2026.</em></p>
<h2>The local cache: an expensive export</h2><p><code>type=local</code> writes the cache as an OCI layout in a directory, and your CI saves and restores that directory between runs, for example with a cache action. In our runs the import side was cheap: reading the cache manifest took 0.1 s or less, and the cached layers came from local disk.</p>
<p>The export side was the most expensive of all the backends. A cold build spent 39.1 s on it, and a one-line change spent 52.4 s, because each run wrote the full <code>mode=max</code> cache into a new, empty directory. The registry backends wrote their warm cache to a new tag in a registry that already held the layers, and finished in 1.4 s or less.</p>
<p>We wrote to a fresh directory on purpose. When you export to the same directory you import from, BuildKit by default keeps every old blob. Docker's <a href="https://docs.docker.com/build/ci/github-actions/cache/" rel="noopener noreferrer">GitHub Actions cache guide</a> puts it plainly: exporting to a directory that already holds a cache "leaves the blobs of the previous export behind, so the directory grows with every run". BuildKit 0.34.1 documents an export option for this, <code>reset=true</code>, which removes "any blobs in the cache directory that are not referenced by the current manifests in <code>index.json</code>" and is "useful for keeping the local cache directory from growing indefinitely". We planned to measure both the growth and <code>reset=true</code>, and ran out of droplet time first, so we cannot tell you how much either one changes.</p>
<h2>Which backend to use</h2><p>For this Dockerfile, on a runner that starts empty:</p>
<ul>
<li><strong>Use inline or registry <code>mode=min</code>.</strong> They gave the same warm builds as <code>mode=max</code> for a source change, cost 0 to 1.2 s to export, and stored no more than the image's own 47.7 MB of layers. Inline needs no extra tag at all, but it only works when you push the image, and its cache is only as old as the last image you pushed.</li>
<li><strong>Use <code>mode=max</code> only when a <code>builder</code> step is slow to run and cheap to pull.</strong> BuildKit pulls and unpacks a cached layer before it can build on top of it. Here the toolchain step took about 10 s to run, and pulling its cached layers with the base image took about as long as pulling the base image and running it again. A long compile that produces a small output layer is the opposite case, and that is where <code>mode=max</code> earns its export cost.</li>
<li><strong>For a slow Gradle or Bazel step, look past the layer cache.</strong> BuildKit reuses a <code>RUN</code> step whole or runs it again, so a source change before that step reruns the whole build. A tool's own remote build cache works per task and skips the tasks whose inputs did not change. <a href="https://bitrise.io" rel="noopener noreferrer">Bitrise</a>, for example, sells one for Gradle, Bazel, Xcode and React Native that it says works on any CI (checked on its public pages on 2 November 2026). We did not test one here.</li>
<li><strong>Price the export, not just the hit.</strong> <code>mode=max</code> added about 30 s of compression to every cold build and 19.0 s to every dependency bump, before any upload.</li>
<li><strong>Be careful with <code>type=local</code>.</strong> Writing to a fresh directory each run, the older way to keep the cache small, repeats the full <code>mode=max</code> export every time, which took 52.4 s on a one-line change. Docker's guide now reuses the directory with <code>reset=true</code>, which needs Buildx 0.35 or later and removes blobs that the new cache no longer references. We did not get to test it.</li>
</ul>
<p>To check your own Dockerfile, build it twice on a fresh builder with <code>--progress=plain</code> and read three lines per step: whether it says <code>CACHED</code>, how long its <code>DONE</code> takes after that (the pull), and how long <code>preparing build cache for export</code> takes. Those three numbers decide which backend wins.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>The cache registry was on loopback.</strong> Our export and import times leave out the network. With a remote registry, add the transfer of 47.7 MB for a source change with any backend, and of the 259.9 MB <code>mode=max</code> cache on every cold export that writes new layers.</li>
<li><strong>One Dockerfile.</strong> In this one, the expensive work comes before <code>COPY . .</code>, so any cache skips it on a source change. In a Dockerfile whose expensive step comes after <code>COPY . .</code>, no backend can help with a source change. The Go frontend of the same demo is like that: its <code>go build</code> step comes after <code>COPY . .</code> and took 66.0 to 105.8 s in three uncached runs of it that we made on the same droplet for our <a href="https://devops-daily.com/posts/how-much-memory-does-a-ci-build-need">CI build memory post</a>.</li>
<li><strong>No no-cache rerun of the changes.</strong> With no cache, a source change or a dependency bump is a cold build, so we used the cold median (79.0 s) as the reference and did not rerun it with the changed files. The cold builds ran earlier in the session, when steal was lower (8 to 27%, against 15 to 39% for the warm builds), so if anything that reference is a little fast.</li>
<li><strong>Fewer warm runs than planned.</strong> Builds on the shared droplet ran slower than we planned, and we had a fixed budget. Cold builds have 5 runs per backend, source changes 3, and dependency bumps 3 (2 for the local cache). The direction of every comparison we draw held in every round; the exact medians would move with more runs.</li>
<li><strong>Fixed order inside each round.</strong> The backends ran in the same order in every round, so a backend's position could bias its time. The host-wide steal we logged per build does not show a pattern by position, but it rose over the session (from 8% to 39% per build). Dependency round 3 ran in the opposite order, <code>mode=max</code> first, and <code>mode=max</code> was still the slowest registry backend. We point at the step timings because they depend less on any of this.</li>
<li><strong>The local result is about a fresh destination.</strong> We exported each local warm build to a new, empty directory, while the registry backends exported next to layers that were already stored. That shows that a fresh local destination is expensive, not that <code>type=local</code> is slow in general, and we did not measure the cache action's save and restore of the directory.</li>
<li><strong>Untested:</strong> <code>type=gha</code> and <code>type=s3</code>, <code>RUN --mount=type=cache</code> cache mounts, two pipelines writing the same cache tag at once, and the <code>reset=true</code> option for local caches.</li>
</ul>
<h2>Run it yourself</h2><p>Create a 4 vCPU Ubuntu machine with Docker, copy the files below to <code>/root/bench/</code>, and run:</p>
<pre><code class="hljs language-bash" tabindex="0">bash /root/bench/registries.sh
STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh prep   <span class="hljs-comment"># contexts for the source and dependency changes</span>
STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh cold   <span class="hljs-comment"># 5 rounds x 5 backends</span>
STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh warm   <span class="hljs-comment"># 5 source rounds, then dependency rounds while time allows</span>
python3 summarize_p2.py /root/res/p2
</code></pre><p>The script below is the one we ran. Our run did not use <code>warm</code> to the end: to fit the budget we stopped it after three source rounds, ran two dependency rounds with <code>warm2</code>, and a third with <code>dep3</code>, which starts with <code>mode=max</code>. <code>packnow</code> (a tar and zstd test of the local cache) and <code>grow</code> (the same directory in and out, with and without <code>reset=true</code>) are in the script too; we ran out of time before them.</p>
<p>Each build writes its own directory with the BuildKit log, the registry access log for that build, and the sizes, so you can check any single number.</p>
<details>
<summary>registries.sh: cache registry, pull-through mirrors, pinned source</summary><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># The cache registry and the two pull-through mirrors, all on the build machine.</span>
docker pull -q moby/buildkit:v0.34.1
docker pull -q registry:2.8.3
docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3
docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3
docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3
git init -q /root/src/msd
git -C /root/src/msd remote add origin https://github.com/GoogleCloudPlatform/microservices-demo
git -C /root/src/msd fetch -q --depth 1 origin c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712
git -C /root/src/msd checkout -q FETCH_HEAD
</code></pre></details><details>
<summary>buildkitd.toml: base image pulls go to the local mirror</summary><pre><code class="hljs language-toml" tabindex="0"><span class="hljs-section">[registry."docker.io"]</span>
  <span class="hljs-attr">mirrors</span> = [<span class="hljs-string">"127.0.0.1:5001"</span>]
<span class="hljs-section">[registry."gcr.io"]</span>
  <span class="hljs-attr">mirrors</span> = [<span class="hljs-string">"127.0.0.1:5002"</span>]
<span class="hljs-section">[registry."127.0.0.1:5000"]</span>
  <span class="hljs-attr">http</span> = <span class="hljs-literal">true</span>
  <span class="hljs-attr">insecure</span> = <span class="hljs-literal">true</span>
<span class="hljs-section">[registry."127.0.0.1:5001"]</span>
  <span class="hljs-attr">http</span> = <span class="hljs-literal">true</span>
  <span class="hljs-attr">insecure</span> = <span class="hljs-literal">true</span>
<span class="hljs-section">[registry."127.0.0.1:5002"]</span>
  <span class="hljs-attr">http</span> = <span class="hljs-literal">true</span>
  <span class="hljs-attr">insecure</span> = <span class="hljs-literal">true</span>
</code></pre></details><details>
<summary>p2.sh: one fresh builder per build, every backend and scenario</summary><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># Post 2: BuildKit cache backends on a fresh builder per build.</span>
<span class="hljs-comment"># usage: p2.sh prep | smoke | cold | pack | warm | grow | warmnone</span>
<span class="hljs-built_in">set</span> -u
<span class="hljs-built_in">export</span> PATH=/usr/local/bin:<span class="hljs-variable">$PATH</span>
B=/root/bench R=/root/res/p2 C=/root/p2
SVC=/root/src/msd/src/paymentservice
<span class="hljs-comment"># stop starting new builds after this time, to stay inside a machine budget (our last stages used 2026-11-02T18:50:00Z)</span>
STOP_AT=$(<span class="hljs-built_in">date</span> -u -d <span class="hljs-string">"<span class="hljs-variable">${STOP_AT_UTC:-tomorrow}</span>"</span> +%s)
REG=127.0.0.1:5000 BK=moby/buildkit:v0.34.1
<span class="hljs-built_in">mkdir</span> -p <span class="hljs-variable">$R</span> <span class="hljs-variable">$C</span>/cache
<span class="hljs-function"><span class="hljs-title">log</span></span>() { <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-subst">$(date -u +%FT%TZ)</span> $*"</span> | <span class="hljs-built_in">tee</span> -a <span class="hljs-variable">$R</span>/progress.log; }

<span class="hljs-function"><span class="hljs-title">prep</span></span>() {
  <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/ctx-*
  <span class="hljs-built_in">cp</span> -a <span class="hljs-variable">$SVC</span> <span class="hljs-variable">$C</span>/ctx-base
  <span class="hljs-built_in">cp</span> -a <span class="hljs-variable">$SVC</span> <span class="hljs-variable">$C</span>/ctx-src
  sed -i <span class="hljs-string">'s/PaymentService gRPC server started on port/PaymentService gRPC server listening on port/'</span> <span class="hljs-variable">$C</span>/ctx-src/server.js
  <span class="hljs-built_in">cp</span> -a <span class="hljs-variable">$SVC</span> <span class="hljs-variable">$C</span>/ctx-dep
  ( <span class="hljs-built_in">cd</span> <span class="hljs-variable">$C</span>/ctx-dep &amp;&amp; npm install --package-lock-only --save-exact --ignore-scripts --no-audit --no-fund @grpc/grpc-js@1.14.7 )
  <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> 1 2 3 4 5; <span class="hljs-keyword">do</span>
    <span class="hljs-built_in">cp</span> -a <span class="hljs-variable">$SVC</span> <span class="hljs-variable">$C</span>/ctx-grow<span class="hljs-variable">$i</span>
    sed -i <span class="hljs-string">"s/PaymentService gRPC server started on port/PaymentService gRPC server (build <span class="hljs-variable">$i</span>) started on port/"</span> <span class="hljs-variable">$C</span>/ctx-grow<span class="hljs-variable">$i</span>/server.js
  <span class="hljs-keyword">done</span>
  diff -u <span class="hljs-variable">$SVC</span>/server.js <span class="hljs-variable">$C</span>/ctx-src/server.js &gt; <span class="hljs-variable">$R</span>/src-change.diff
  diff -u <span class="hljs-variable">$SVC</span>/package.json <span class="hljs-variable">$C</span>/ctx-dep/package.json &gt; <span class="hljs-variable">$R</span>/dep-change.diff
  diff -u <span class="hljs-variable">$SVC</span>/package-lock.json <span class="hljs-variable">$C</span>/ctx-dep/package-lock.json &gt;&gt; <span class="hljs-variable">$R</span>/dep-change.diff
  grep -c <span class="hljs-string">"listening on port"</span> <span class="hljs-variable">$C</span>/ctx-src/server.js
  <span class="hljs-built_in">head</span> -60 <span class="hljs-variable">$R</span>/dep-change.diff
}

<span class="hljs-comment"># build CFG SCEN RUN [CTX]</span>
<span class="hljs-function"><span class="hljs-title">build</span></span>() {
  <span class="hljs-built_in">local</span> cfg=<span class="hljs-variable">$1</span> scen=<span class="hljs-variable">$2</span> r=<span class="hljs-variable">$3</span> ctx=<span class="hljs-variable">${4:-}</span>
  <span class="hljs-built_in">local</span> out=<span class="hljs-variable">$R</span>/<span class="hljs-variable">$scen</span>/<span class="hljs-variable">$cfg</span>/<span class="hljs-variable">$r</span> name=p2-<span class="hljs-variable">$cfg</span>-<span class="hljs-variable">$scen</span>-<span class="hljs-variable">$r</span>
  <span class="hljs-keyword">if</span> [ $(<span class="hljs-built_in">date</span> +%s) -gt <span class="hljs-variable">$STOP_AT</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-built_in">log</span> <span class="hljs-string">"<span class="hljs-variable">$scen</span> <span class="hljs-variable">$cfg</span> <span class="hljs-variable">$r</span> SKIPPED (time budget)"</span>; <span class="hljs-built_in">return</span>; <span class="hljs-keyword">fi</span>
  <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$out</span>; <span class="hljs-built_in">mkdir</span> -p <span class="hljs-variable">$out</span>
  [ -z <span class="hljs-string">"<span class="hljs-variable">$ctx</span>"</span> ] &amp;&amp; <span class="hljs-keyword">case</span> <span class="hljs-variable">$scen</span> <span class="hljs-keyword">in</span> cold) ctx=<span class="hljs-variable">$C</span>/ctx-base ;; src) ctx=<span class="hljs-variable">$C</span>/ctx-src ;; dep) ctx=<span class="hljs-variable">$C</span>/ctx-dep ;; <span class="hljs-keyword">esac</span>
  <span class="hljs-built_in">local</span> img=<span class="hljs-variable">$REG</span>/paymentservice:<span class="hljs-variable">$cfg</span>-<span class="hljs-variable">$scen</span> from=() to=()
  <span class="hljs-keyword">case</span> <span class="hljs-variable">$cfg</span> <span class="hljs-keyword">in</span>
    none) ;;
    inline)
      to=(--cache-to <span class="hljs-built_in">type</span>=inline)
      <span class="hljs-keyword">if</span> [ <span class="hljs-variable">$scen</span> = cold ]; <span class="hljs-keyword">then</span> img=<span class="hljs-variable">$REG</span>/paymentservice:inline-base
      <span class="hljs-keyword">else</span> from=(--cache-from <span class="hljs-built_in">type</span>=registry,ref=<span class="hljs-variable">$REG</span>/paymentservice:inline-base); <span class="hljs-keyword">fi</span> ;;
    regmin|regmax)
      <span class="hljs-built_in">local</span> m=<span class="hljs-variable">${cfg#reg}</span>
      <span class="hljs-keyword">if</span> [ <span class="hljs-variable">$scen</span> = cold ]; <span class="hljs-keyword">then</span> to=(--cache-to <span class="hljs-built_in">type</span>=registry,ref=<span class="hljs-variable">$REG</span>/cache:<span class="hljs-variable">$m</span>,mode=<span class="hljs-variable">$m</span>)
      <span class="hljs-keyword">else</span> from=(--cache-from <span class="hljs-built_in">type</span>=registry,ref=<span class="hljs-variable">$REG</span>/cache:<span class="hljs-variable">$m</span>); to=(--cache-to <span class="hljs-built_in">type</span>=registry,ref=<span class="hljs-variable">$REG</span>/cache:<span class="hljs-variable">$m</span>-scratch,mode=<span class="hljs-variable">$m</span>); <span class="hljs-keyword">fi</span> ;;
    <span class="hljs-built_in">local</span>)
      <span class="hljs-keyword">if</span> [ <span class="hljs-variable">$scen</span> = cold ]; <span class="hljs-keyword">then</span> <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/cache/local; to=(--cache-to <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,dest=<span class="hljs-variable">$C</span>/cache/local,mode=max)
      <span class="hljs-keyword">else</span> <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/cache/scratch; from=(--cache-from <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,src=<span class="hljs-variable">$C</span>/cache/local); to=(--cache-to <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,dest=<span class="hljs-variable">$C</span>/cache/scratch,mode=max); <span class="hljs-keyword">fi</span> ;;
    grow)
      from=(--cache-from <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,src=<span class="hljs-variable">$C</span>/cache/grow); to=(--cache-to <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,dest=<span class="hljs-variable">$C</span>/cache/grow,mode=max) ;;
    growr)
      from=(--cache-from <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,src=<span class="hljs-variable">$C</span>/cache/growr); to=(--cache-to <span class="hljs-built_in">type</span>=<span class="hljs-built_in">local</span>,dest=<span class="hljs-variable">$C</span>/cache/growr,mode=max,reset=<span class="hljs-literal">true</span>) ;;
  <span class="hljs-keyword">esac</span>
  <span class="hljs-built_in">sync</span>; <span class="hljs-built_in">echo</span> 3 &gt; /proc/sys/vm/drop_caches
  <span class="hljs-built_in">local</span> t0 t1 t2 t3 rc
  t0=$(<span class="hljs-built_in">date</span> +%s.%N)
  docker buildx create --name <span class="hljs-variable">$name</span> --driver docker-container --driver-opt network=host \
    --driver-opt image=<span class="hljs-variable">$BK</span> --buildkitd-config <span class="hljs-variable">$B</span>/buildkitd.toml --bootstrap &gt; <span class="hljs-variable">$out</span>/create.log 2&gt;&amp;1
  t1=$(<span class="hljs-built_in">date</span> +%s.%N)
  <span class="hljs-built_in">local</span> cpu0; cpu0=$(grep <span class="hljs-string">'^cpu '</span> /proc/stat)
  docker buildx build --builder <span class="hljs-variable">$name</span> --progress=plain --provenance=<span class="hljs-literal">false</span> --sbom=<span class="hljs-literal">false</span> \
    --metadata-file <span class="hljs-variable">$out</span>/meta.json <span class="hljs-string">"<span class="hljs-variable">${from[@]}</span>"</span> <span class="hljs-string">"<span class="hljs-variable">${to[@]}</span>"</span> \
    --output <span class="hljs-built_in">type</span>=image,name=<span class="hljs-variable">$img</span>,push=<span class="hljs-literal">true</span>,registry.insecure=<span class="hljs-literal">true</span> <span class="hljs-variable">$ctx</span> &gt; <span class="hljs-variable">$out</span>/build.log 2&gt;&amp;1
  rc=$?
  t2=$(<span class="hljs-built_in">date</span> +%s.%N)
  <span class="hljs-built_in">local</span> cpu1; cpu1=$(grep <span class="hljs-string">'^cpu '</span> /proc/stat)
  docker buildx <span class="hljs-built_in">rm</span> <span class="hljs-variable">$name</span> &gt; /dev/null 2&gt;&amp;1
  t3=$(<span class="hljs-built_in">date</span> +%s.%N)
  docker logs --since <span class="hljs-variable">$t1</span> --<span class="hljs-keyword">until</span> <span class="hljs-variable">$t2</span> reg-cache &gt; <span class="hljs-variable">$out</span>/registry.log 2&gt;&amp;1
  { <span class="hljs-built_in">echo</span> <span class="hljs-string">"rc=<span class="hljs-variable">$rc</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"t0=<span class="hljs-variable">$t0</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"t1=<span class="hljs-variable">$t1</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"t2=<span class="hljs-variable">$t2</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"t3=<span class="hljs-variable">$t3</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"ctx=<span class="hljs-variable">$ctx</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"cpu0=<span class="hljs-variable">$cpu0</span>"</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">"cpu1=<span class="hljs-variable">$cpu1</span>"</span>; } &gt; <span class="hljs-variable">$out</span>/result.env
  <span class="hljs-keyword">case</span> <span class="hljs-variable">$cfg</span> <span class="hljs-keyword">in</span>
    <span class="hljs-built_in">local</span>) <span class="hljs-keyword">if</span> [ <span class="hljs-variable">$scen</span> = cold ]; <span class="hljs-keyword">then</span> d=<span class="hljs-variable">$C</span>/cache/local; <span class="hljs-keyword">else</span> d=<span class="hljs-variable">$C</span>/cache/scratch; <span class="hljs-keyword">fi</span>
           <span class="hljs-built_in">echo</span> <span class="hljs-string">"cache_bytes=<span class="hljs-subst">$(du -sb $d | cut -f1)</span>"</span> &gt;&gt; <span class="hljs-variable">$out</span>/result.env
           <span class="hljs-built_in">echo</span> <span class="hljs-string">"cache_blobs=<span class="hljs-subst">$(ls $d/blobs/sha256 | wc -l)</span>"</span> &gt;&gt; <span class="hljs-variable">$out</span>/result.env
           [ <span class="hljs-variable">$scen</span> = cold ] || <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/cache/scratch ;;
    grow|growr) <span class="hljs-built_in">echo</span> <span class="hljs-string">"cache_bytes=<span class="hljs-subst">$(du -sb $C/cache/$cfg | cut -f1)</span>"</span> &gt;&gt; <span class="hljs-variable">$out</span>/result.env
           <span class="hljs-built_in">echo</span> <span class="hljs-string">"cache_blobs=<span class="hljs-subst">$(ls $C/cache/$cfg/blobs/sha256 | wc -l)</span>"</span> &gt;&gt; <span class="hljs-variable">$out</span>/result.env ;;
  <span class="hljs-keyword">esac</span>
  <span class="hljs-keyword">case</span> <span class="hljs-variable">$cfg</span>-<span class="hljs-variable">$scen</span> <span class="hljs-keyword">in</span>
    regmin-cold) python3 <span class="hljs-variable">$B</span>/regsize.py <span class="hljs-variable">$REG</span> cache min &gt; <span class="hljs-variable">$out</span>/cache-size.json ;;
    regmax-cold) python3 <span class="hljs-variable">$B</span>/regsize.py <span class="hljs-variable">$REG</span> cache max &gt; <span class="hljs-variable">$out</span>/cache-size.json ;;
    regmin-*) python3 <span class="hljs-variable">$B</span>/regsize.py <span class="hljs-variable">$REG</span> cache min-scratch &gt; <span class="hljs-variable">$out</span>/cache-size.json ;;
    regmax-*) python3 <span class="hljs-variable">$B</span>/regsize.py <span class="hljs-variable">$REG</span> cache max-scratch &gt; <span class="hljs-variable">$out</span>/cache-size.json ;;
  <span class="hljs-keyword">esac</span>
  python3 <span class="hljs-variable">$B</span>/regsize.py <span class="hljs-variable">$REG</span> paymentservice <span class="hljs-variable">${img##*:}</span> &gt; <span class="hljs-variable">$out</span>/image-size.json
  <span class="hljs-built_in">log</span> <span class="hljs-string">"<span class="hljs-variable">$scen</span> <span class="hljs-variable">$cfg</span> <span class="hljs-variable">$r</span> rc=<span class="hljs-variable">$rc</span> build=<span class="hljs-subst">$(python3 -c <span class="hljs-string">"print(round(<span class="hljs-variable">$t2</span>-<span class="hljs-variable">$t1</span>,1))"</span>)</span>s"</span>
}

<span class="hljs-function"><span class="hljs-title">pack_test</span></span>() {
    <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3 r4 r5; <span class="hljs-keyword">do</span>
      <span class="hljs-built_in">sync</span>; <span class="hljs-built_in">echo</span> 3 &gt; /proc/sys/vm/drop_caches
      t0=$(<span class="hljs-built_in">date</span> +%s.%N); tar -C <span class="hljs-variable">$C</span>/cache/local -cf - . | zstd -T0 -q -f -o <span class="hljs-variable">$C</span>/local.tar.zst; <span class="hljs-built_in">sync</span>; t1=$(<span class="hljs-built_in">date</span> +%s.%N)
      <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/unpack; <span class="hljs-built_in">mkdir</span> -p <span class="hljs-variable">$C</span>/unpack; <span class="hljs-built_in">sync</span>; <span class="hljs-built_in">echo</span> 3 &gt; /proc/sys/vm/drop_caches
      t2=$(<span class="hljs-built_in">date</span> +%s.%N); zstd -dc -q <span class="hljs-variable">$C</span>/local.tar.zst | tar -C <span class="hljs-variable">$C</span>/unpack -xf -; <span class="hljs-built_in">sync</span>; t3=$(<span class="hljs-built_in">date</span> +%s.%N)
      <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$r</span> pack_s=<span class="hljs-subst">$(python3 -c <span class="hljs-string">"print(round(<span class="hljs-variable">$t1</span>-<span class="hljs-variable">$t0</span>,2))"</span>)</span> unpack_s=<span class="hljs-subst">$(python3 -c <span class="hljs-string">"print(round(<span class="hljs-variable">$t3</span>-<span class="hljs-variable">$t2</span>,2))"</span>)</span> dir_bytes=<span class="hljs-subst">$(du -sb $C/cache/local | cut -f1)</span> tar_zst_bytes=<span class="hljs-subst">$(stat -c %s $C/local.tar.zst)</span>"</span> | <span class="hljs-built_in">tee</span> -a <span class="hljs-variable">$R</span>/pack.txt
    <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/unpack <span class="hljs-variable">$C</span>/local.tar.zst; <span class="hljs-built_in">log</span> PACK-DONE
}

<span class="hljs-keyword">case</span> <span class="hljs-variable">${1:-}</span> <span class="hljs-keyword">in</span>
  prep) prep ;;
  smoke) <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> none inline regmin regmax <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> cold s0; <span class="hljs-keyword">done</span>; <span class="hljs-built_in">log</span> SMOKE-DONE ;;
  cold) <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3 r4 r5; <span class="hljs-keyword">do</span> <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> none inline regmin regmax <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> cold <span class="hljs-variable">$r</span>; <span class="hljs-keyword">done</span>; <span class="hljs-keyword">done</span>; <span class="hljs-built_in">log</span> COLD-DONE ;;
  warm) <span class="hljs-comment"># all source-change rounds first, then whole dependency rounds while they fit before STOP_AT</span>
    <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3 r4 r5; <span class="hljs-keyword">do</span> <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> inline regmin regmax <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> src <span class="hljs-variable">$r</span>; <span class="hljs-keyword">done</span>; <span class="hljs-keyword">done</span>
    last=400
    <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3 r4 r5; <span class="hljs-keyword">do</span>
      t=$(<span class="hljs-built_in">date</span> +%s)
      <span class="hljs-keyword">if</span> [ $((t + last)) -gt <span class="hljs-variable">$STOP_AT</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-built_in">log</span> <span class="hljs-string">"dep <span class="hljs-variable">$r</span> and later skipped: a round needs about <span class="hljs-variable">${last}</span>s"</span>; <span class="hljs-built_in">break</span>; <span class="hljs-keyword">fi</span>
      <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> inline regmin regmax <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> dep <span class="hljs-variable">$r</span>; <span class="hljs-keyword">done</span>
      last=$(( $(date +%s) - t ))
    <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> WARM-DONE ;;
  warm2) <span class="hljs-comment"># switched to after source round 3: dependency rounds first, then the last source rounds, while they fit</span>
    last=400
    <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3 r4 r5; <span class="hljs-keyword">do</span>
      t=$(<span class="hljs-built_in">date</span> +%s)
      <span class="hljs-keyword">if</span> [ $((t + last)) -gt <span class="hljs-variable">$STOP_AT</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-built_in">log</span> <span class="hljs-string">"dep <span class="hljs-variable">$r</span> and later skipped: a round needs about <span class="hljs-variable">${last}</span>s"</span>; <span class="hljs-built_in">break</span>; <span class="hljs-keyword">fi</span>
      <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> inline regmin regmax <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> dep <span class="hljs-variable">$r</span>; <span class="hljs-keyword">done</span>
      last=$(( $(date +%s) - t ))
      [ <span class="hljs-variable">$r</span> = r3 ] &amp;&amp; <span class="hljs-built_in">break</span>
    <span class="hljs-keyword">done</span>
    <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r4 r5; <span class="hljs-keyword">do</span>
      t=$(<span class="hljs-built_in">date</span> +%s)
      <span class="hljs-keyword">if</span> [ $((t + <span class="hljs-number">150</span>)) -gt <span class="hljs-variable">$STOP_AT</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-built_in">log</span> <span class="hljs-string">"src <span class="hljs-variable">$r</span> skipped: time budget"</span>; <span class="hljs-built_in">break</span>; <span class="hljs-keyword">fi</span>
      <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> inline regmin regmax <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> src <span class="hljs-variable">$r</span>; <span class="hljs-keyword">done</span>
    <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> WARM-DONE ;;
  dep3) <span class="hljs-comment"># third dependency round, started after round 2 (time budget)</span>
    <span class="hljs-keyword">for</span> c <span class="hljs-keyword">in</span> regmax regmin inline <span class="hljs-built_in">local</span>; <span class="hljs-keyword">do</span> build <span class="hljs-variable">$c</span> dep r3; <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> WARM-DONE ;;
  packnow) pack_test ;;
  warmnone) <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3 r4 r5; <span class="hljs-keyword">do</span> <span class="hljs-keyword">for</span> s <span class="hljs-keyword">in</span> src dep; <span class="hljs-keyword">do</span> build none <span class="hljs-variable">$s</span> <span class="hljs-variable">$r</span>; <span class="hljs-keyword">done</span>; <span class="hljs-keyword">done</span>; <span class="hljs-built_in">log</span> WARMNONE-DONE ;;
  pack) <span class="hljs-built_in">log</span> <span class="hljs-string">"pack deferred until after the warm builds"</span> ;;
  grow) <span class="hljs-comment"># local cache pack/unpack test, then the same-directory growth test, without and with reset=true</span>
    <span class="hljs-keyword">if</span> [ $(<span class="hljs-built_in">date</span> +%s) -lt $((STOP_AT + <span class="hljs-number">120</span>)) ]; <span class="hljs-keyword">then</span> pack_test; <span class="hljs-keyword">fi</span>
    <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$C</span>/cache/grow <span class="hljs-variable">$C</span>/cache/growr; <span class="hljs-built_in">cp</span> -a <span class="hljs-variable">$C</span>/cache/local <span class="hljs-variable">$C</span>/cache/grow; <span class="hljs-built_in">cp</span> -a <span class="hljs-variable">$C</span>/cache/local <span class="hljs-variable">$C</span>/cache/growr
    <span class="hljs-built_in">echo</span> <span class="hljs-string">"start cache_bytes=<span class="hljs-subst">$(du -sb $C/cache/grow | cut -f1)</span> blobs=<span class="hljs-subst">$(ls $C/cache/grow/blobs/sha256 | wc -l)</span>"</span> &gt; <span class="hljs-variable">$R</span>/grow-start.txt
    <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> 1 2 3 4 5; <span class="hljs-keyword">do</span> build grow src g<span class="hljs-variable">$i</span> <span class="hljs-variable">$C</span>/ctx-grow<span class="hljs-variable">$i</span>; build growr src g<span class="hljs-variable">$i</span> <span class="hljs-variable">$C</span>/ctx-grow<span class="hljs-variable">$i</span>; <span class="hljs-keyword">done</span>; <span class="hljs-built_in">log</span> GROW-DONE ;;
  *) <span class="hljs-built_in">echo</span> <span class="hljs-string">"usage"</span>; <span class="hljs-built_in">exit</span> 2 ;;
<span class="hljs-keyword">esac</span>
</code></pre></details><details>
<summary>regsize.py: compressed bytes behind a registry tag</summary><pre><code class="hljs language-python" tabindex="0"><span class="hljs-comment">#!/usr/bin/env python3</span>
<span class="hljs-string">"""Sum the compressed layer bytes behind a registry tag (image or BuildKit cache)."""</span>
<span class="hljs-keyword">import</span> json, sys, urllib.request

reg, repo, tag = sys.argv[<span class="hljs-number">1</span>:<span class="hljs-number">4</span>]
ACCEPT = <span class="hljs-string">", "</span>.join([
    <span class="hljs-string">"application/vnd.oci.image.index.v1+json"</span>,
    <span class="hljs-string">"application/vnd.oci.image.manifest.v1+json"</span>,
    <span class="hljs-string">"application/vnd.docker.distribution.manifest.list.v2+json"</span>,
    <span class="hljs-string">"application/vnd.docker.distribution.manifest.v2+json"</span>,
])

<span class="hljs-keyword">def</span> <span class="hljs-title function_">get</span>(<span class="hljs-params">ref</span>):
    req = urllib.request.Request(<span class="hljs-string">f"http://<span class="hljs-subst">{reg}</span>/v2/<span class="hljs-subst">{repo}</span>/manifests/<span class="hljs-subst">{ref}</span>"</span>, headers={<span class="hljs-string">"Accept"</span>: ACCEPT})
    <span class="hljs-keyword">with</span> urllib.request.urlopen(req, timeout=<span class="hljs-number">30</span>) <span class="hljs-keyword">as</span> r:
        <span class="hljs-keyword">return</span> json.load(r)

<span class="hljs-keyword">def</span> <span class="hljs-title function_">walk</span>(<span class="hljs-params">m, layers</span>):
    <span class="hljs-keyword">for</span> d <span class="hljs-keyword">in</span> m.get(<span class="hljs-string">"manifests"</span>, []):
        mt = d.get(<span class="hljs-string">"mediaType"</span>, <span class="hljs-string">""</span>)
        <span class="hljs-keyword">if</span> <span class="hljs-string">"manifest"</span> <span class="hljs-keyword">in</span> mt <span class="hljs-keyword">or</span> <span class="hljs-string">"index"</span> <span class="hljs-keyword">in</span> mt:
            walk(get(d[<span class="hljs-string">"digest"</span>]), layers)
        <span class="hljs-keyword">elif</span> <span class="hljs-string">"layer"</span> <span class="hljs-keyword">in</span> mt:
            layers[d[<span class="hljs-string">"digest"</span>]] = d[<span class="hljs-string">"size"</span>]
    <span class="hljs-keyword">for</span> d <span class="hljs-keyword">in</span> m.get(<span class="hljs-string">"layers"</span>, []):
        layers[d[<span class="hljs-string">"digest"</span>]] = d[<span class="hljs-string">"size"</span>]

<span class="hljs-keyword">try</span>:
    top = get(tag)
<span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:
    <span class="hljs-built_in">print</span>(json.dumps({<span class="hljs-string">"error"</span>: <span class="hljs-built_in">str</span>(e)})); sys.exit(<span class="hljs-number">0</span>)
layers = {}
walk(top, layers)
<span class="hljs-built_in">print</span>(json.dumps({<span class="hljs-string">"tag"</span>: <span class="hljs-string">f"<span class="hljs-subst">{repo}</span>:<span class="hljs-subst">{tag}</span>"</span>, <span class="hljs-string">"mediaType"</span>: top.get(<span class="hljs-string">"mediaType"</span>), <span class="hljs-string">"layers"</span>: <span class="hljs-built_in">len</span>(layers), <span class="hljs-string">"bytes"</span>: <span class="hljs-built_in">sum</span>(layers.values()), <span class="hljs-string">"sizes"</span>: <span class="hljs-built_in">sorted</span>(layers.values(), reverse=<span class="hljs-literal">True</span>)}))
</code></pre></details><details>
<summary>summarize_p2.py: parse BuildKit logs, registry logs and sizes into medians</summary><pre><code class="hljs language-python" tabindex="0"><span class="hljs-comment">#!/usr/bin/env python3</span>
<span class="hljs-string">"""Summarize post 2 runs: parse BuildKit plain progress, registry logs and sizes; medians per cell."""</span>
<span class="hljs-keyword">import</span> json, os, re, statistics, sys

root = sys.argv[<span class="hljs-number">1</span>]
VERTEX = re.<span class="hljs-built_in">compile</span>(<span class="hljs-string">r"^#(\d+) (.*)$"</span>)
DONE = re.<span class="hljs-built_in">compile</span>(<span class="hljs-string">r"^DONE ([\d.]+)s$"</span>)
SUBDONE = re.<span class="hljs-built_in">compile</span>(<span class="hljs-string">r"^(preparing layers for inline cache|preparing build cache for export|exporting layers|pushing layers|exporting manifest list|exporting config).*?([\d.]+)s done$"</span>)

<span class="hljs-keyword">def</span> <span class="hljs-title function_">env</span>(<span class="hljs-params">path</span>):
    d = {}
    <span class="hljs-keyword">if</span> os.path.exists(path):
        <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> <span class="hljs-built_in">open</span>(path):
            <span class="hljs-keyword">if</span> <span class="hljs-string">"="</span> <span class="hljs-keyword">in</span> line:
                k, v = line.strip().split(<span class="hljs-string">"="</span>, <span class="hljs-number">1</span>)
                d[k] = v
    <span class="hljs-keyword">return</span> d

<span class="hljs-keyword">def</span> <span class="hljs-title function_">parse_build</span>(<span class="hljs-params">path</span>):
    v = {}
    order = []
    <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> <span class="hljs-built_in">open</span>(path, errors=<span class="hljs-string">"replace"</span>):
        line = line.rstrip(<span class="hljs-string">"\n"</span>)
        m = VERTEX.<span class="hljs-keyword">match</span>(line)
        <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> m:
            <span class="hljs-keyword">continue</span>
        vid, rest = <span class="hljs-built_in">int</span>(m.group(<span class="hljs-number">1</span>)), m.group(<span class="hljs-number">2</span>)
        <span class="hljs-keyword">if</span> vid <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span> v:
            v[vid] = {<span class="hljs-string">"name"</span>: rest, <span class="hljs-string">"done"</span>: <span class="hljs-literal">None</span>, <span class="hljs-string">"cached"</span>: <span class="hljs-literal">False</span>, <span class="hljs-string">"error"</span>: <span class="hljs-literal">False</span>, <span class="hljs-string">"sub"</span>: {}}
            order.append(vid)
            <span class="hljs-keyword">continue</span>
        <span class="hljs-keyword">if</span> rest == <span class="hljs-string">"CACHED"</span>:
            v[vid][<span class="hljs-string">"cached"</span>] = <span class="hljs-literal">True</span>
        <span class="hljs-keyword">elif</span> rest.startswith(<span class="hljs-string">"ERROR"</span>):
            v[vid][<span class="hljs-string">"error"</span>] = <span class="hljs-literal">True</span>
        <span class="hljs-keyword">else</span>:
            d = DONE.<span class="hljs-keyword">match</span>(rest)
            <span class="hljs-keyword">if</span> d:
                v[vid][<span class="hljs-string">"done"</span>] = <span class="hljs-built_in">float</span>(d.group(<span class="hljs-number">1</span>))
            s = SUBDONE.<span class="hljs-keyword">match</span>(rest)
            <span class="hljs-keyword">if</span> s:
                v[vid][<span class="hljs-string">"sub"</span>][s.group(<span class="hljs-number">1</span>)] = v[vid][<span class="hljs-string">"sub"</span>].get(s.group(<span class="hljs-number">1</span>), <span class="hljs-number">0</span>) + <span class="hljs-built_in">float</span>(s.group(<span class="hljs-number">2</span>))
    <span class="hljs-keyword">return</span> [v[i] <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> order]

<span class="hljs-keyword">def</span> <span class="hljs-title function_">registry_bytes</span>(<span class="hljs-params">path</span>):
    got = put = <span class="hljs-number">0</span>
    seen_logrus = <span class="hljs-literal">False</span>
    lines = <span class="hljs-built_in">open</span>(path, errors=<span class="hljs-string">"replace"</span>).read().splitlines() <span class="hljs-keyword">if</span> os.path.exists(path) <span class="hljs-keyword">else</span> []
    <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> lines:
        <span class="hljs-keyword">if</span> <span class="hljs-string">'msg="response completed"'</span> <span class="hljs-keyword">in</span> line:
            seen_logrus = <span class="hljs-literal">True</span>
            meth = re.search(<span class="hljs-string">r"http\.request\.method=(\w+)"</span>, line)
            uri = re.search(<span class="hljs-string">r'http\.request\.uri="([^"]+)"'</span>, line)
            w = re.search(<span class="hljs-string">r"http\.response\.written=(\d+)"</span>, line)
            <span class="hljs-keyword">if</span> meth <span class="hljs-keyword">and</span> uri <span class="hljs-keyword">and</span> w <span class="hljs-keyword">and</span> meth.group(<span class="hljs-number">1</span>) == <span class="hljs-string">"GET"</span> <span class="hljs-keyword">and</span> <span class="hljs-string">"/blobs/"</span> <span class="hljs-keyword">in</span> uri.group(<span class="hljs-number">1</span>):
                got += <span class="hljs-built_in">int</span>(w.group(<span class="hljs-number">1</span>))
    <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> seen_logrus:
        <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> lines:
            m = re.search(<span class="hljs-string">r'"GET (/v2/[^ ]*/blobs/[^ ]*) HTTP/[\d.]+" (\d+) (\d+)'</span>, line)
            <span class="hljs-keyword">if</span> m <span class="hljs-keyword">and</span> m.group(<span class="hljs-number">2</span>) == <span class="hljs-string">"200"</span>:
                got += <span class="hljs-built_in">int</span>(m.group(<span class="hljs-number">3</span>))
    <span class="hljs-keyword">return</span> got

<span class="hljs-keyword">def</span> <span class="hljs-title function_">step_time</span>(<span class="hljs-params">vs, pattern</span>):
    <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> vs:
        <span class="hljs-keyword">if</span> re.search(pattern, x[<span class="hljs-string">"name"</span>]):
            <span class="hljs-keyword">return</span> <span class="hljs-number">0.0</span> <span class="hljs-keyword">if</span> x[<span class="hljs-string">"cached"</span>] <span class="hljs-keyword">else</span> x[<span class="hljs-string">"done"</span>]
    <span class="hljs-keyword">return</span> <span class="hljs-literal">None</span>

<span class="hljs-keyword">def</span> <span class="hljs-title function_">run</span>(<span class="hljs-params">d</span>):
    r = env(os.path.join(d, <span class="hljs-string">"result.env"</span>))
    <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> r:
        <span class="hljs-keyword">return</span> <span class="hljs-literal">None</span>
    vs = parse_build(os.path.join(d, <span class="hljs-string">"build.log"</span>))
    steps = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> vs <span class="hljs-keyword">if</span> re.<span class="hljs-keyword">match</span>(<span class="hljs-string">r"^\[(builder|stage-\d+) \d+/\d+\]"</span>, x[<span class="hljs-string">"name"</span>])]
    exp = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> vs <span class="hljs-keyword">if</span> x[<span class="hljs-string">"name"</span>].startswith(<span class="hljs-string">"exporting cache to"</span>)]
    img = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> vs <span class="hljs-keyword">if</span> x[<span class="hljs-string">"name"</span>] == <span class="hljs-string">"exporting to image"</span>]
    imp = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> vs <span class="hljs-keyword">if</span> x[<span class="hljs-string">"name"</span>].startswith(<span class="hljs-string">"importing cache manifest"</span>)]
    info = {
        <span class="hljs-string">"rc"</span>: <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"rc"</span>]),
        <span class="hljs-string">"bootstrap_s"</span>: <span class="hljs-built_in">round</span>(<span class="hljs-built_in">float</span>(r[<span class="hljs-string">"t1"</span>]) - <span class="hljs-built_in">float</span>(r[<span class="hljs-string">"t0"</span>]), <span class="hljs-number">2</span>),
        <span class="hljs-string">"build_s"</span>: <span class="hljs-built_in">round</span>(<span class="hljs-built_in">float</span>(r[<span class="hljs-string">"t2"</span>]) - <span class="hljs-built_in">float</span>(r[<span class="hljs-string">"t1"</span>]), <span class="hljs-number">2</span>),
        <span class="hljs-string">"steps"</span>: <span class="hljs-built_in">len</span>(steps), <span class="hljs-string">"cached_steps"</span>: <span class="hljs-built_in">sum</span>(<span class="hljs-number">1</span> <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> steps <span class="hljs-keyword">if</span> x[<span class="hljs-string">"cached"</span>]),
        <span class="hljs-string">"cached_names"</span>: [x[<span class="hljs-string">"name"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> steps <span class="hljs-keyword">if</span> x[<span class="hljs-string">"cached"</span>]],
        <span class="hljs-string">"apk_builder_s"</span>: step_time(vs, <span class="hljs-string">r"RUN apk add --update"</span>),
        <span class="hljs-string">"npm_install_s"</span>: step_time(vs, <span class="hljs-string">r"RUN npm install"</span>),
        <span class="hljs-string">"apk_final_s"</span>: step_time(vs, <span class="hljs-string">r"RUN apk add --no-cache nodejs"</span>),
        <span class="hljs-string">"copy_modules_s"</span>: step_time(vs, <span class="hljs-string">r"COPY --from=builder"</span>),
        <span class="hljs-string">"export_cache_s"</span>: <span class="hljs-built_in">sum</span>(x[<span class="hljs-string">"done"</span>] <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> exp) <span class="hljs-keyword">if</span> exp <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"export_image_s"</span>: img[<span class="hljs-number">0</span>][<span class="hljs-string">"done"</span>] <span class="hljs-keyword">if</span> img <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"inline_prep_s"</span>: img[<span class="hljs-number">0</span>][<span class="hljs-string">"sub"</span>].get(<span class="hljs-string">"preparing layers for inline cache"</span>) <span class="hljs-keyword">if</span> img <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"prep_export_s"</span>: <span class="hljs-built_in">sum</span>(x[<span class="hljs-string">"sub"</span>].get(<span class="hljs-string">"preparing build cache for export"</span>, <span class="hljs-number">0</span>) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> exp) <span class="hljs-keyword">if</span> exp <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"import_manifest_s"</span>: <span class="hljs-built_in">sum</span>(x[<span class="hljs-string">"done"</span>] <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> imp) <span class="hljs-keyword">if</span> imp <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"pulled_from_cache_registry_bytes"</span>: registry_bytes(os.path.join(d, <span class="hljs-string">"registry.log"</span>)),
        <span class="hljs-string">"errors"</span>: [x[<span class="hljs-string">"name"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> vs <span class="hljs-keyword">if</span> x[<span class="hljs-string">"error"</span>]],
        <span class="hljs-comment"># a CACHED step still reports DONE time when its layers must be pulled and unpacked for a later step</span>
        <span class="hljs-string">"cached_pull_s"</span>: <span class="hljs-built_in">max</span>([x[<span class="hljs-string">"done"</span>] <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> steps <span class="hljs-keyword">if</span> x[<span class="hljs-string">"cached"</span>]] <span class="hljs-keyword">or</span> [<span class="hljs-number">0</span>]),
    }
    <span class="hljs-keyword">if</span> <span class="hljs-string">"cpu0"</span> <span class="hljs-keyword">in</span> r <span class="hljs-keyword">and</span> <span class="hljs-string">"cpu1"</span> <span class="hljs-keyword">in</span> r:
        a = [<span class="hljs-built_in">int</span>(x) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> r[<span class="hljs-string">"cpu0"</span>].split()[<span class="hljs-number">1</span>:<span class="hljs-number">9</span>]]
        b = [<span class="hljs-built_in">int</span>(x) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> r[<span class="hljs-string">"cpu1"</span>].split()[<span class="hljs-number">1</span>:<span class="hljs-number">9</span>]]
        dd = [y - x <span class="hljs-keyword">for</span> x, y <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(a, b)]
        info[<span class="hljs-string">"steal_pct"</span>] = <span class="hljs-built_in">round</span>(<span class="hljs-number">100</span> * dd[<span class="hljs-number">7</span>] / <span class="hljs-built_in">sum</span>(dd), <span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> <span class="hljs-built_in">sum</span>(dd) <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>
    <span class="hljs-keyword">if</span> <span class="hljs-string">"cache_bytes"</span> <span class="hljs-keyword">in</span> r:
        info[<span class="hljs-string">"cache_bytes"</span>] = <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"cache_bytes"</span>])
        info[<span class="hljs-string">"cache_blobs"</span>] = <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"cache_blobs"</span>])
    <span class="hljs-keyword">for</span> f <span class="hljs-keyword">in</span> (<span class="hljs-string">"cache-size.json"</span>, <span class="hljs-string">"image-size.json"</span>):
        p = os.path.join(d, f)
        <span class="hljs-keyword">if</span> os.path.exists(p):
            <span class="hljs-keyword">try</span>:
                j = json.load(<span class="hljs-built_in">open</span>(p))
                info[f.split(<span class="hljs-string">"."</span>)[<span class="hljs-number">0</span>].replace(<span class="hljs-string">"-"</span>, <span class="hljs-string">"_"</span>) + <span class="hljs-string">"_bytes"</span>] = j.get(<span class="hljs-string">"bytes"</span>)
                info[f.split(<span class="hljs-string">"."</span>)[<span class="hljs-number">0</span>].replace(<span class="hljs-string">"-"</span>, <span class="hljs-string">"_"</span>) + <span class="hljs-string">"_layers"</span>] = j.get(<span class="hljs-string">"layers"</span>)
            <span class="hljs-keyword">except</span> Exception:
                <span class="hljs-keyword">pass</span>
    <span class="hljs-keyword">return</span> info

<span class="hljs-keyword">def</span> <span class="hljs-title function_">med</span>(<span class="hljs-params">xs</span>):
    xs = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> xs <span class="hljs-keyword">if</span> x <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>]
    <span class="hljs-keyword">return</span> <span class="hljs-built_in">round</span>(statistics.median(xs), <span class="hljs-number">2</span>) <span class="hljs-keyword">if</span> xs <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>

out = {}
<span class="hljs-keyword">for</span> scen <span class="hljs-keyword">in</span> (<span class="hljs-string">"cold"</span>, <span class="hljs-string">"src"</span>, <span class="hljs-string">"dep"</span>):
    <span class="hljs-keyword">for</span> cfg <span class="hljs-keyword">in</span> (<span class="hljs-string">"none"</span>, <span class="hljs-string">"inline"</span>, <span class="hljs-string">"regmin"</span>, <span class="hljs-string">"regmax"</span>, <span class="hljs-string">"local"</span>, <span class="hljs-string">"grow"</span>, <span class="hljs-string">"growr"</span>):
        d0 = os.path.join(root, scen, cfg)
        <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> os.path.isdir(d0):
            <span class="hljs-keyword">continue</span>
        runs = {rn: run(os.path.join(d0, rn)) <span class="hljs-keyword">for</span> rn <span class="hljs-keyword">in</span> <span class="hljs-built_in">sorted</span>(os.listdir(d0))}
        runs = {k: v <span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> runs.items() <span class="hljs-keyword">if</span> v}
        meas = [v <span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> runs.items() <span class="hljs-keyword">if</span> k.startswith(<span class="hljs-string">"r"</span>)]
        keys = [<span class="hljs-string">"build_s"</span>, <span class="hljs-string">"bootstrap_s"</span>, <span class="hljs-string">"cached_steps"</span>, <span class="hljs-string">"apk_builder_s"</span>, <span class="hljs-string">"npm_install_s"</span>, <span class="hljs-string">"apk_final_s"</span>, <span class="hljs-string">"copy_modules_s"</span>, <span class="hljs-string">"export_cache_s"</span>, <span class="hljs-string">"export_image_s"</span>,
                <span class="hljs-string">"inline_prep_s"</span>, <span class="hljs-string">"prep_export_s"</span>, <span class="hljs-string">"import_manifest_s"</span>, <span class="hljs-string">"cached_pull_s"</span>, <span class="hljs-string">"pulled_from_cache_registry_bytes"</span>, <span class="hljs-string">"steal_pct"</span>, <span class="hljs-string">"cache_bytes"</span>, <span class="hljs-string">"cache_size_bytes"</span>, <span class="hljs-string">"image_size_bytes"</span>]
        cell = {<span class="hljs-string">"n"</span>: <span class="hljs-built_in">len</span>(meas), <span class="hljs-string">"rcs"</span>: [x[<span class="hljs-string">"rc"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]}
        <span class="hljs-keyword">for</span> k <span class="hljs-keyword">in</span> keys:
            cell[k] = med([x.get(k) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas])
            vals = [x.get(k) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas <span class="hljs-keyword">if</span> x.get(k) <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>]
            <span class="hljs-keyword">if</span> vals <span class="hljs-keyword">and</span> k <span class="hljs-keyword">in</span> (<span class="hljs-string">"build_s"</span>, <span class="hljs-string">"export_cache_s"</span>, <span class="hljs-string">"import_manifest_s"</span>):
                cell[k + <span class="hljs-string">"_range"</span>] = [<span class="hljs-built_in">min</span>(vals), <span class="hljs-built_in">max</span>(vals)]
        cell[<span class="hljs-string">"runs"</span>] = runs
        out[<span class="hljs-string">f"<span class="hljs-subst">{scen}</span>/<span class="hljs-subst">{cfg}</span>"</span>] = cell
json.dump(out, <span class="hljs-built_in">open</span>(os.path.join(root, <span class="hljs-string">"summary.json"</span>), <span class="hljs-string">"w"</span>), indent=<span class="hljs-number">1</span>)
MB = <span class="hljs-number">1e6</span>
<span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{<span class="hljs-string">'cell'</span>:<span class="hljs-number">14</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'n'</span>:&gt;<span class="hljs-number">2</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'build'</span>:&gt;<span class="hljs-number">7</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'range'</span>:&gt;<span class="hljs-number">14</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'cached'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'apkB'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'npm'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'expC'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'expI'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'impM'</span>:&gt;<span class="hljs-number">5</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'pullMB'</span>:&gt;<span class="hljs-number">7</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'cacheMB'</span>:&gt;<span class="hljs-number">8</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'imgMB'</span>:&gt;<span class="hljs-number">6</span>}</span>"</span>)
<span class="hljs-keyword">for</span> k, c <span class="hljs-keyword">in</span> out.items():
    cb = c.get(<span class="hljs-string">"cache_size_bytes"</span>) <span class="hljs-keyword">or</span> c.get(<span class="hljs-string">"cache_bytes"</span>)
    <span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{k:<span class="hljs-number">14</span>}</span> <span class="hljs-subst">{c[<span class="hljs-string">'n'</span>]:&gt;<span class="hljs-number">2</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'build_s'</span>]):&gt;<span class="hljs-number">7</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c.get(<span class="hljs-string">'build_s_range'</span>)):&gt;<span class="hljs-number">14</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'cached_steps'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'apk_builder_s'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'npm_install_s'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'export_cache_s'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'export_image_s'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(c[<span class="hljs-string">'import_manifest_s'</span>]):&gt;<span class="hljs-number">5</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">round</span>((c[<span class="hljs-string">'pulled_from_cache_registry_bytes'</span>] <span class="hljs-keyword">or</span> <span class="hljs-number">0</span>)/MB,<span class="hljs-number">1</span>):&gt;<span class="hljs-number">7</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">round</span>(cb/MB,<span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> cb <span class="hljs-keyword">else</span> <span class="hljs-string">'-'</span>:&gt;<span class="hljs-number">8</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">round</span>(c[<span class="hljs-string">'image_size_bytes'</span>]/MB,<span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> c[<span class="hljs-string">'image_size_bytes'</span>] <span class="hljs-keyword">else</span> <span class="hljs-string">'-'</span>:&gt;<span class="hljs-number">6</span>}</span> rcs=<span class="hljs-subst">{c[<span class="hljs-string">'rcs'</span>]}</span>"</span>)
</code></pre></details><h2>Summary</h2><p>A fresh CI runner has no BuildKit cache, so the backend you pick decides what gets reused. For this multi-stage Node service, the cheapest backends did the most: inline and registry <code>mode=min</code> cut a one-line rebuild from 79.0 s to about 20 s and cost almost nothing to export, because BuildKit could skip the whole <code>builder</code> stage by matching cache keys alone. <code>mode=max</code> stored five times as much, added about 30 s of compression to every cold build, and did not make the dependency bump faster: pulling the cached toolchain took as long as installing it, and the export came on top. Before you switch on <code>mode=max</code> because a guide says so, read <code>--progress=plain</code> for your own build and compare the pull time of each cached step with the time it takes to run.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB]]></title>
      <link>https://devops-daily.com/posts/how-much-memory-does-a-ci-build-need</link>
      <description><![CDATA[We ran five real open-source builds inside 2, 4 and 8 GB memory limits. Four fit in 2 GB. The cgroup counter overstated most of them, and the one that failed was a Vite build that also runs a full type check. Every script is in the post.]]></description>
      <pubDate>Mon, 02 Nov 2026 18:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/how-much-memory-does-a-ci-build-need</guid>
      <category><![CDATA[CI/CD]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[CI/CD]]></category><category><![CDATA[Memory]]></category><category><![CDATA[cgroups]]></category><category><![CDATA[Docker]]></category><category><![CDATA[Gradle]]></category><category><![CDATA[Node.js]]></category><category><![CDATA[Go]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>We ran five real open-source builds (a Vite app build, a TypeScript type check, a Gradle build, a Go build and a Docker image build) inside 2, 4 and 8 GB memory limits on one 4 vCPU machine, with Gradle both with and without its daemon, two or three times at each limit. Four of the five finished inside 2 GB. The one that did not was the Excalidraw web app: its Vite production build also runs a full TypeScript check, together they held 3,248 MiB of anonymous memory, and the kernel killed the build at 2 GB in all 3 runs.</p>
<p>The number that most tools show you was the wrong one for the other four. cgroup v2's <code>memory.peak</code> said the GitHub CLI's Go build peaked at 1,775 MiB, but only 424 MiB of that was anonymous memory, the heaps and stacks of its processes. Most of the rest was page cache, which the kernel drops when it needs the room, plus some kernel memory. A Docker image build looked like 2,072 MiB for the same reason. And a TypeScript type check that held 1,139 MiB with the whole machine available finished inside 2 GB with 970 MiB, because Node sizes its heap limit from the cgroup limit it finds.</p>
<p>This post shows the numbers, the five traps behind them, and every script we used, so you can size a runner from your own builds instead of from a guess.</p>
<h2>TL;DR</h2><ul>
<li><strong>Four of five builds fit in 2 GB.</strong> Only the Excalidraw Vite build needed more: 3,248 MiB of anonymous memory, killed at 2 GB, fine at 4 GB.</li>
<li><strong><code>memory.peak</code> counts page cache.</strong> The Go build and the Docker build showed 1,775 and 2,072 MiB, but their anonymous memory peaked at 424 and 355 MiB. Size from anonymous memory plus a margin, then confirm by running the build inside that limit.</li>
<li><strong>Runtimes size their heaps from the cgroup limit.</strong> Inside 2 GB, Node's default heap limit was 1,120 MiB and the JVM's default maximum heap was 512 MiB. The type check used less memory at 2 GB than at 8 GB and still finished, about 24% slower.</li>
<li><strong>The Gradle daemon keeps its memory after the build.</strong> Five seconds after <code>gradlew</code> exited, the daemon still held 1,048 MiB of anonymous memory. With <code>--no-daemon</code>, that dropped to under 1 MiB, and in our one-command runs the daemon gave no repeatable speed-up.</li>
<li><strong>Limiting the <code>docker</code> CLI did not limit the build.</strong> The CLI's cgroup peaked at 40 MiB while the BuildKit container that did the work peaked at 2,072 MiB.</li>
<li>Wall times on a shared-CPU droplet are noisy: the CPUs reported 23.5% steal time across the runs we tracked. Trust the memory columns more than the time columns.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Linux machine with cgroup v2 and systemd (any recent Ubuntu, Debian or Fedora). Check with <code>stat -fc %T /sys/fs/cgroup</code>, which prints <code>cgroup2fs</code>.</li>
<li>Root access, for <code>systemd-run --scope</code> and for dropping the page cache.</li>
<li>The toolchains for the builds you want to test. Our setup script installs Go, Node, Yarn, a JDK, Rust and Docker.</li>
<li>If you want to rerun our matrix as it is: a 4 vCPU, 8 GB machine and about two hours. We used a droplet on <a href="https://m.do.co/c/2a9bba940f39" rel="sponsored noopener noreferrer">DigitalOcean</a> (referral link).</li>
</ul>
<h2>How we measured</h2><p>We used one DigitalOcean droplet, size <code>s-4vcpu-8gb</code> in <code>fra1</code>, with Ubuntu 24.04 and kernel 6.8. The kernel reports 7,941 MiB of memory and there is no swap. Every build ran in its own transient systemd scope:</p>
<pre><code class="hljs language-bash" tabindex="0">systemd-run --scope --slice=bench.slice --unit=p1-&lt;build&gt;-&lt;<span class="hljs-built_in">limit</span>&gt;-&lt;run&gt; \
  -p MemoryMax=2G -p MemorySwapMax=0 -p AllowedCPUs=0-3 -p OOMPolicy=<span class="hljs-built_in">continue</span> \
  /root/bench/wrap.sh &lt;outdir&gt; &lt;<span class="hljs-built_in">timeout</span>&gt; &lt;build <span class="hljs-built_in">command</span>&gt;
</code></pre><p>These are <strong>cgroup limits on one droplet, not three real machine sizes.</strong> A real 2 GB runner has a smaller kernel footprint, a runner agent and often a Docker daemon in the same memory, so it has less room than our 2 GB scope. The 8 GB limit is above the 7.9 GiB that the droplet has, so the 8 GB column means "the whole machine".</p>
<p>The rest of the setup:</p>
<ul>
<li><strong>CPU.</strong> Every scope was pinned to the same 4 vCPUs with <code>AllowedCPUs=0-3</code>, and <code>nproc</code> printed 4 inside each one. Compilers start about one job per CPU, so a 2 vCPU runner would use less memory and an 8 vCPU runner more. We did not measure other CPU counts.</li>
<li><strong>Cold build, warm dependencies.</strong> We downloaded dependencies once before the runs: <code>yarn install</code>, <code>go mod download</code>, <code>cargo fetch</code>, the Gradle distribution and dependencies, and the base images for the Docker build. Before every run we deleted the build output and the compiler caches (<code>excalidraw-app/build</code>, <code>GOCACHE</code>, <code>target/</code>, Gradle's <code>build/</code> and <code>.gradle/</code>), stopped any Gradle daemon, and dropped the page cache. That is a CI job that restores a dependency cache but not a build cache.</li>
<li><strong>What we recorded.</strong> Wall time, the exit code, <code>memory.peak</code> at the end of the run, <code>memory.events</code> (how often the limit was hit, and OOM kills), and a sampler that read <code>memory.stat</code> every 100 ms for anonymous memory (<code>anon</code>) and page cache (<code>file</code>).</li>
<li><strong>Repeats.</strong> Runs went in round-robin order: run 1 of every cell, then run 2, then run 3, so a slow period on the shared host does not land on one build. Builds ran slower than we planned, so to stay inside our droplet budget the third round covered only the 8 GB and 2 GB limits. That gives <strong>3 runs per cell at 8 GB and 2 GB, and 2 at 4 GB.</strong> The tables show the median of 3, or the mean of 2 at 4 GB. Each build also had one warm-up run at 8 GB that we did not count; it downloaded Gradle and filled the image mirror.</li>
</ul>
<p>The builds, all pinned to exact commits:</p>
<table>
<thead>
<tr>
<th>Build</th>
<th>Command</th>
<th>Source</th>
</tr>
</thead>
<tbody><tr>
<td>Excalidraw app, Vite production build</td>
<td><code>yarn build:app:docker</code></td>
<td>excalidraw/excalidraw <code>8bcdca45</code></td>
</tr>
<tr>
<td>Excalidraw type check, 1,508 files</td>
<td><code>tsc -p tsconfig.json</code> (<code>noEmit</code> is set)</td>
<td>same checkout</td>
</tr>
<tr>
<td>PlantUML, Gradle 9.3.1 on JDK 21</td>
<td><code>./gradlew jar</code>, daemon on and <code>--no-daemon</code></td>
<td>plantuml/plantuml <code>db06d70e</code></td>
</tr>
<tr>
<td>GitHub CLI, Go 1.27.3</td>
<td><code>go build ./cmd/gh</code></td>
<td>cli/cli <code>9490d204</code></td>
</tr>
<tr>
<td>Online Boutique frontend image, BuildKit 0.34.1</td>
<td><code>docker buildx build</code></td>
<td>GoogleCloudPlatform/microservices-demo <code>c7cb6d5b</code></td>
</tr>
<tr>
<td>ripgrep 15.2.0, Rust 1.99.1 (one run)</td>
<td><code>cargo build --release --locked</code></td>
<td>BurntSushi/ripgrep tag <code>15.2.0</code></td>
</tr>
</tbody></table>
<p>Node was 24.22.2 with Yarn 1.22.22, Docker was 29.9.2 with Buildx 0.38.1.</p>
<h2>The results</h2><table>
<thead>
<tr>
<th>Build</th>
<th>Anonymous peak at 8 GB</th>
<th><code>memory.peak</code> at 8 GB</th>
<th>8 GB</th>
<th>4 GB</th>
<th>2 GB</th>
</tr>
</thead>
<tbody><tr>
<td>Excalidraw <code>vite build</code></td>
<td>3,248 MiB</td>
<td>3,578 MiB</td>
<td>ok, 66 s</td>
<td>ok, 83 s</td>
<td>3 of 3 OOM-killed (run 1 from the kernel log)</td>
</tr>
<tr>
<td>Excalidraw <code>tsc</code> type check</td>
<td>1,139 MiB</td>
<td>1,271 MiB</td>
<td>ok, 59 s</td>
<td>ok, 62 s</td>
<td>ok, 73 s</td>
</tr>
<tr>
<td>PlantUML Gradle, daemon</td>
<td>1,143 MiB</td>
<td>1,474 MiB</td>
<td>ok, 124 s</td>
<td>ok, 110 s</td>
<td>ok, 113 s</td>
</tr>
<tr>
<td>PlantUML Gradle, <code>--no-daemon</code></td>
<td>1,144 MiB</td>
<td>1,475 MiB</td>
<td>ok, 88 s</td>
<td>ok, 119 s</td>
<td>ok, 100 s</td>
</tr>
<tr>
<td>GitHub CLI <code>go build</code></td>
<td>424 MiB</td>
<td>1,775 MiB</td>
<td>ok, 141 s</td>
<td>ok, 198 s</td>
<td>ok, 169 s</td>
</tr>
<tr>
<td>Online Boutique frontend, <code>docker buildx build</code></td>
<td>355 MiB</td>
<td>2,072 MiB</td>
<td>ok, 131 s</td>
<td>ok, 129 s</td>
<td>ok, 110 s</td>
</tr>
</tbody></table>
<p>Times are medians of 3 runs at 8 GB and 2 GB, and the mean of 2 runs at 4 GB. Memory columns are from the 8 GB runs.</p>
<p><strong>What the cgroup counter says vs what the build held (8 GB limit, median of 3)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Vite build</td>
<td>3578 MiB</td>
<td>memory.peak</td>
</tr>
<tr>
<td>Vite build</td>
<td>3248 MiB</td>
<td>anonymous memory</td>
</tr>
<tr>
<td>tsc check</td>
<td>1271 MiB</td>
<td>memory.peak</td>
</tr>
<tr>
<td>tsc check</td>
<td>1139 MiB</td>
<td>anonymous memory</td>
</tr>
<tr>
<td>Gradle (daemon)</td>
<td>1474 MiB</td>
<td>memory.peak</td>
</tr>
<tr>
<td>Gradle (daemon)</td>
<td>1143 MiB</td>
<td>anonymous memory</td>
</tr>
<tr>
<td>Gradle (no daemon)</td>
<td>1475 MiB</td>
<td>memory.peak</td>
</tr>
<tr>
<td>Gradle (no daemon)</td>
<td>1144 MiB</td>
<td>anonymous memory</td>
</tr>
<tr>
<td>Go build</td>
<td>1775 MiB</td>
<td>memory.peak</td>
</tr>
<tr>
<td>Go build</td>
<td>424 MiB</td>
<td>anonymous memory</td>
</tr>
<tr>
<td>Docker build</td>
<td>2072 MiB</td>
<td>memory.peak</td>
</tr>
<tr>
<td>Docker build</td>
<td>355 MiB</td>
<td>anonymous memory</td>
</tr>
</tbody></table>
<p><em>memory.peak includes page cache; anonymous memory is the sampled peak of anon in memory.stat (100 ms). One s-4vcpu-8gb droplet, 4 vCPUs, 2 November 2026.</em></p>
<p>Three things stand out. Only one build reached the 2 GB limit at all. The <code>max</code> counter in <code>memory.events</code>, which counts how often a cgroup's usage was about to go over its limit, stayed at 0 for every other run. Memory use at 4 GB and at 8 GB was almost the same for every build, so the extra 4 GB bought nothing. And the gap between <code>memory.peak</code> and anonymous memory is large for exactly the builds that read and write many files: the Go compiler and BuildKit.</p>
<p>The single Rust build, ripgrep 15.2.0 with <code>cargo build --release</code>, fits the same pattern: 896 MiB <code>memory.peak</code>, 572 MiB anonymous, 61.1 s. We ran it once at 8 GB only, because the full matrix did not fit our time budget.</p>
<h2>Trap 1: <code>memory.peak</code> counts page cache</h2><p><code>memory.peak</code> is the highest value of <code>memory.current</code> that the cgroup reached. <code>memory.current</code> charges the cgroup for everything it caused to be in memory, and that includes <strong>page cache</strong>: the source files the compiler read, the object files it wrote, the layers BuildKit unpacked. Page cache is reclaimable. When the cgroup gets close to its limit, the kernel drops clean cache pages and writes back dirty ones before it ever considers killing a process.</p>
<p><strong>Anonymous memory</strong> is the part that cannot go anywhere without swap: heaps, stacks, the compiler's working data. <code>memory.stat</code> reports it as <code>anon</code>. That, plus a smaller amount of kernel memory (37 to 136 MiB in these builds), is what has to fit. Our sampler read it every 100 ms, so it can miss a shorter spike. <code>memory.peak</code> and the <code>max</code> counter cannot miss one, and they show that no build except Vite at 2 GB ever reached its limit. The safe test is still the one we ran: the build inside the limit you plan to buy.</p>
<p>The Go build of the GitHub CLI shows the difference well. Its peak was 1,775 MiB. Anonymous memory never went over 424 MiB; about 1,400 MiB was page cache from the module sources and the build cache that <code>go build</code> writes. At the 2 GB limit, the same build finished with the same peak, because 1,775 MiB is still under 2,048 MiB. With a smaller limit the kernel may reclaim that cache instead of failing the build, but we did not test below 2 GB, so test the lower limit before you rely on it.</p>
<p>So if a dashboard shows that your CI job "peaked at 1.8 GB", that does not mean a 2 GB runner is too small. Look at <code>anon</code> in <code>memory.stat</code> (or RSS in your container metrics), and then add a margin for the runner itself.</p>
<h2>Trap 2: a production build that is also a type check</h2><p>The Excalidraw build needed far more memory than anything else: 3,248 MiB of anonymous memory, against 1,139 MiB for a full type check of the same code. The reason is in its <a href="https://github.com/excalidraw/excalidraw/blob/8bcdca4526867433f99fdfcb55a063668db0c9dc/excalidraw-app/vite.config.mts" rel="noopener noreferrer">Vite config</a>:</p>
<pre><code class="hljs language-ts" tabindex="0"><span class="hljs-title function_">checker</span>({
  <span class="hljs-attr">typescript</span>: <span class="hljs-literal">true</span>,
  <span class="hljs-attr">eslint</span>:
    envVars.<span class="hljs-property">VITE_APP_ENABLE_ESLINT</span> === <span class="hljs-string">"false"</span>
      ? <span class="hljs-literal">undefined</span>
      : { <span class="hljs-attr">lintCommand</span>: <span class="hljs-string">'eslint "./**/*.{js,ts,tsx}"'</span> },
  <span class="hljs-comment">// ...</span>
}),
</code></pre><p><code>vite-plugin-checker</code> runs the TypeScript checker during <code>vite build</code> too: in version 0.7.2, which Excalidraw pins, <code>enableBuild</code> defaults to <code>true</code>. The production <code>.env</code> turns ESLint off but leaves the type check on, and the build also writes source maps. So one <code>vite build</code> is Rollup, source maps and a full <code>tsc</code> run at the same time.</p>
<p>At 2 GB the kernel killed it in all 3 runs, 69 to 122 seconds in, after the log said <code>✓ 1442 modules transformed.</code> The kernel log names the cgroup and the process it chose:</p>
<pre><code class="hljs language-text" tabindex="0">oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=p1-vite-2G-r2.scope,mems_allowed=0,oom_memcg=/bench.slice/p1-vite-2G-r2.scope,task_memcg=/bench.slice/p1-vite-2G-r2.scope,task=MainThread,pid=77242,uid=0
Memory cgroup out of memory: Killed process 77242 (MainThread) total-vm:11451308kB, anon-rss:1138988kB, file-rss:4864kB, shmem-rss:0kB, UID:0 pgtables:29992kB oom_score_adj:0
</code></pre><p>Our wrapper's record of that run shows the kill. <code>memory.events</code> counted 39,825 times that the cgroup was about to go over its limit, and one OOM kill:</p>
<p><strong>vite build, 2 GB limit, run 2</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ <span class="hljs-built_in">cat</span> /root/res/p1/vite/2G/r2/result.env
rc=1
t0=1793637034.083018067
t1=1793637156.285208072
peak=2147483648
current_after=867889152
anon_after=801075200
file_after=38674432
$ <span class="hljs-built_in">cat</span> /root/res/p1/vite/2G/r2/memory.events
low 0
high 0
max 39825
oom 1
oom_kill 1
oom_group_kill 0
</code></pre><p>The process list right after the kill shows the second half of the build. The checker runs <code>tsc</code> as its own child process, and it was still running after Vite had died:</p>
<pre><code class="hljs language-text" tabindex="0">  77265  1808 /bin/sh -c tsc --noEmit
  77266 799108 node /root/src/excalidraw/node_modules/.bin/tsc --noEmit
</code></pre><p>That <code>tsc</code> held 799,108 KiB (780 MiB) of resident memory under the 2 GB limit. Yarn reported <code>Command failed with exit code 1</code>, but the type check kept going until our harness stopped the scope. On a real runner it would keep using memory and CPU until the job ends.</p>
<p>We did not split the 3,248 MiB peak at 8 GB between Rollup and the checker, so we cannot say exactly how much the type check adds. What we can say: if your bundler runs a checker, your build step needs the memory of both, and moving the type check to its own CI job is the first thing to try before you buy a bigger runner.</p>
<h2>Trap 3: runtimes size their heaps from the limit</h2><p>Node and the JVM both read the cgroup limit when they start and size their default heap from it. Our harness asks each runtime what it would use inside each limit:</p>
<p><strong>probe inside a 2 GB scope</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># what Node 24 and JDK 21 pick inside MemoryMax=2G (from p1.sh probe)</span>
$ <span class="hljs-built_in">cat</span> /root/res/p1/probe-2G.txt
<span class="hljs-built_in">nproc</span>=4
memory.max=2147483648
node_heap_size_limit=1174405120
      int ActiveProcessorCount                     = -1                                        {product} {default}
   size_t MaxHeapSize                              = 536870912                                 {product} {ergonomic}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}
</code></pre><p>All four places side by side:</p>
<table>
<thead>
<tr>
<th>Inside</th>
<th>Node 24 <code>heap_size_limit</code></th>
<th>JDK 21 default <code>MaxHeapSize</code></th>
</tr>
</thead>
<tbody><tr>
<td>2 GB scope</td>
<td>1,120 MiB</td>
<td>512 MiB</td>
</tr>
<tr>
<td>4 GB scope</td>
<td>2,240 MiB</td>
<td>1,024 MiB</td>
</tr>
<tr>
<td>8 GB scope</td>
<td>2,240 MiB</td>
<td>2,048 MiB</td>
</tr>
<tr>
<td>whole droplet, no scope</td>
<td>2,240 MiB</td>
<td>1,986 MiB</td>
</tr>
</tbody></table>
<p>Two consequences showed up in the runs.</p>
<p><strong>Heap use follows the limit.</strong> <code>tsc --extendedDiagnostics</code> prints <code>Memory used</code>, which is Node's <code>heapUsed</code> at the end of the run, in units of 1,000 bytes. With the whole machine, the type check of 1,508 files reported <code>Memory used: 1018378K</code> (median of 3, about 971 MiB). Inside 2 GB it reported 886903K (about 846 MiB), anonymous memory fell from 1,139 to 970 MiB, and the check finished about 24% slower (median 72.7 s against 58.8 s). The 2 GB run was 9.5 to 14.2 seconds slower in each of the three rounds. In rounds 2 and 3, where we logged the CPU counters, its steal time was only 4 to 7 points higher, so steal explains only part of the gap. More frequent garbage collection under the smaller heap limit would explain the rest, but we did not record GC activity, so that is a hypothesis. A big heap number at a high limit says little about what the program needs. We expected this check to fail at 2 GB with a heap error, and it did not.</p>
<p><strong>Node stopped at 2,240 MiB even with more memory.</strong> On our 8 GB machine, Node's default heap limit was 2,240 MiB with a 4 GB scope, an 8 GB scope and no scope at all. We did not test a bigger machine, so check what your runner gives you with <code>node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)'</code>. If a build reports <code>JavaScript heap out of memory</code>, that is Node's own limit, not the kernel's. Set <code>NODE_OPTIONS=--max-old-space-size=&lt;MiB&gt;</code> explicitly instead of only buying more memory.</p>
<p>The JVM takes 25% of the limit by default (<code>MaxRAMPercentage=25</code>), so a Maven build or a test JVM without <code>-Xmx</code> gets 512 MiB inside 2 GB. Gradle sets its own daemon heap, which brings us to the next trap.</p>
<h2>Trap 4: the Gradle daemon outlives the build</h2><p>PlantUML's <code>gradle.properties</code> sets <code>org.gradle.jvmargs=-Xmx1g</code>. That cap kept the build inside 2 GB: anonymous memory peaked at about 1,143 MiB with the daemon and 1,144 MiB without it, and neither came near the limit.</p>
<p>The difference is what happens after <code>gradlew</code> exits. With the daemon, which is the default, the JVM stays alive and keeps its heap. Five seconds after the build, the scope still held 1,378 MiB, 1,048 MiB of it anonymous, in one process:</p>
<pre><code class="hljs language-text" tabindex="0">37292 1105976 /usr/lib/jvm/java-21-openjdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED ...
</code></pre><p>With <code>--no-daemon</code>, Gradle printed <code>To honour the JVM settings for this build a single-use Daemon process will be forked</code> and <code>Daemon will be stopped at the end of the build</code>, and five seconds after exit the anonymous memory in the scope was under 1 MiB.</p>
<p><strong>Anonymous memory around the end of the Gradle build (8 GB limit, run 1)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>-55 s</th>
<th>-50 s</th>
<th>-45 s</th>
<th>-40 s</th>
<th>-35 s</th>
<th>-30 s</th>
<th>-25 s</th>
<th>-20 s</th>
<th>-15 s</th>
<th>-10 s</th>
<th>-5 s</th>
<th>0 s</th>
<th>+5 s</th>
</tr>
</thead>
<tbody><tr>
<td>daemon (default)</td>
<td>637 MiB</td>
<td>857 MiB</td>
<td>927 MiB</td>
<td>982 MiB</td>
<td>1053 MiB</td>
<td>1059 MiB</td>
<td>1066 MiB</td>
<td>1073 MiB</td>
<td>1082 MiB</td>
<td>1110 MiB</td>
<td>1115 MiB</td>
<td>1143 MiB</td>
<td>1054 MiB</td>
</tr>
<tr>
<td>--no-daemon</td>
<td>476 MiB</td>
<td>516 MiB</td>
<td>559 MiB</td>
<td>807 MiB</td>
<td>921 MiB</td>
<td>970 MiB</td>
<td>1067 MiB</td>
<td>1072 MiB</td>
<td>1080 MiB</td>
<td>1084 MiB</td>
<td>1112 MiB</td>
<td>1144 MiB</td>
<td>0 MiB</td>
</tr>
</tbody></table>
<p><em>Max of 100 ms samples in each 5 s bin, labelled by the end of the bin; 0 s is the moment gradlew exits, +5 s is the 5 s after. One run each, PlantUML jar task, JDK 21, Gradle 9.3.1.</em></p>
<p>On a CI runner, that leftover daemon sits in the same memory as every later step of the job: the Docker build, the integration tests, the upload. At 8 GB the median time was 123.5 s with the daemon and 87.5 s without (112.7 s and 99.7 s at 2 GB), The ranges overlap and CPU steal was high, so we see no repeatable speed-up from the daemon for one Gradle command per job. The daemon is built to pay off across several Gradle commands in the same job, which we did not test. If a job runs one Gradle command, use <code>--no-daemon</code> (or <code>org.gradle.daemon=false</code>). If it runs several, consider keeping the daemon (we did not test that case), and run <code>./gradlew --stop</code> before the steps that need the memory.</p>
<h2>Trap 5: a limit on the <code>docker</code> CLI does not limit the build</h2><p><code>docker build</code> is a client. In our setup the work ran in a BuildKit container (Buildx's <code>docker-container</code> driver), which has its own cgroup. Other drivers and CI executors lay out cgroups differently, but the CLI is only a client in all of them. We wrapped the <code>docker buildx build</code> command in the same kind of scope as everything else, and at every limit that scope peaked at 39 to 42 MiB: the CLI, streaming logs.</p>
<p>To measure the build, we put the limit on the BuildKit container itself:</p>
<pre><code class="hljs language-bash" tabindex="0">docker buildx create --name p1d8gr1 --driver docker-container \
  --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \
  --driver-opt memory=8g --driver-opt memory-swap=8g --driver-opt cpuset-cpus=0-3 \
  --buildkitd-config buildkitd.toml --bootstrap
</code></pre><p>and read <code>memory.peak</code> from <code>/sys/fs/cgroup/system.slice/docker-&lt;container id&gt;.scope</code>. That container peaked at 2,072 MiB, of which only 355 MiB was anonymous: the Go compiler inside the build. Most of the rest was page cache from the image layers and the build's output, plus 136 MiB of kernel memory.</p>
<p>This matters for runner sizing in two ways. Your CI's memory graph may not show the build at all, depending on whether it measures the job's process tree or the whole machine. And if a self-hosted runner limits only the job's process tree, a Docker build in that job can run outside the limit, so a "2 GB job" can push the host far past 2 GB. Check which cgroup your executor puts the Docker daemon in.</p>
<h2>Two method bugs that cost us a run</h2><p>The first 2 GB Vite run left no result at all. The kernel killed the Vite process, and then systemd stopped the whole scope, including our wrapper script, before it could write anything. The journal said so:</p>
<pre><code class="hljs language-text" tabindex="0">p1-vite-2G-r1.scope: A process of this unit has been killed by the OOM killer.
p1-vite-2G-r1.scope: Failed with result 'oom-kill'.
</code></pre><p>systemd 255 sets <code>DefaultOOMPolicy=stop</code>, and scopes follow it. We set <code>DefaultOOMPolicy=continue</code> before the next 2 GB run (the published <code>p1.sh</code> also passes <code>-p OOMPolicy=continue</code>), and checked with a 64 MiB scope that the wrapper now survives a kill and records <code>oom_kill 1</code>. Runs 2 and 3 at 2 GB have full records.</p>
<p>Then our rerun of run 1 failed too, for a different reason. systemd keeps a failed transient unit loaded, so starting a new scope with the same name was refused:</p>
<pre><code class="hljs language-text" tabindex="0">Failed to start transient scope unit: Unit p1-vite-2G-r1.scope was already loaded or has a fragment file.
</code></pre><p>By the time we saw that, the Docker cache tests for the next post were running on the same droplet, so we did not try again. Run 1 still counts as a kill: its kernel log line reads <code>oom_memcg=/bench.slice/p1-vite-2G-r1.scope,...,task=MainThread,pid=36041</code>, and our sampler, which runs outside the scope, saw the cgroup reach 2,048 MiB, with up to 1,985 MiB anonymous, before its last sample at 70.5 seconds. The published <code>redo</code> mode now runs <code>systemctl reset-failed</code> first.</p>
<p>If you measure builds this way, set <code>OOMPolicy=continue</code> and give each run a unique unit name, or the runs that you care about most, the ones that fail, will be the ones with no data.</p>
<h2>A sizing table</h2><p>From these runs, for a 4 vCPU runner. The "smallest limit that passed" column is measured. The last column is our rule of thumb, not a measurement: a real VM also holds the OS, the runner agent and any service containers, so we pick the smallest size that leaves at least 1 GB above the anonymous peak.</p>
<table>
<thead>
<tr>
<th>Build like ours</th>
<th>Anonymous peak</th>
<th>Smallest limit that passed every run</th>
<th>Runner we would pick</th>
</tr>
</thead>
<tbody><tr>
<td>Bundler that also type-checks (Excalidraw <code>vite build</code>)</td>
<td>3,248 MiB</td>
<td>4 GB</td>
<td>8 GB (moving the type check to its own job may make 4 GB enough; we did not test that)</td>
</tr>
<tr>
<td>Large type check on its own (<code>tsc</code>, 1,508 files)</td>
<td>1,139 MiB at 8 GB, 970 MiB at 2 GB</td>
<td>2 GB</td>
<td>4 GB</td>
</tr>
<tr>
<td>Gradle with a 1 GB daemon heap (PlantUML <code>jar</code>)</td>
<td>1,143 MiB</td>
<td>2 GB</td>
<td>4 GB, with <code>--no-daemon</code></td>
</tr>
<tr>
<td>Go build of a mid-size CLI (<code>gh</code>)</td>
<td>424 MiB</td>
<td>2 GB</td>
<td>2 GB</td>
</tr>
<tr>
<td>Docker build of a Go service</td>
<td>355 MiB inside BuildKit</td>
<td>2 GB</td>
<td>2 GB, plus what the Docker daemon and the other steps need</td>
</tr>
</tbody></table>
<p>Hosted CI services list their machines by vCPU and memory, so you can match a build to a size. <a href="https://bitrise.io" rel="noopener noreferrer">Bitrise</a>, for example, lists every Linux machine type on its pricing page with its vCPU count and memory, such as 2 vCPU with 8 GB, and also sells a remote build cache for Gradle, Bazel and Xcode (checked on its public pages on 2 November 2026). Whatever you use, measure anonymous memory first, then pick the size.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>Three limits on one machine are not three machines.</strong> A real 2 GB runner has less page cache to spare, its own kernel, and a runner agent in the same memory. Leave a margin above the anonymous peak; our sizing table uses 1 GB.</li>
<li><strong>CPU count changes memory.</strong> Every run had 4 vCPUs. The Go compiler, Rollup's workers and Gradle's workers scale with CPUs, so these numbers are not valid for 2 or 16 vCPUs.</li>
<li><strong>Wall times are noisy.</strong> The droplet reported 23.5% CPU steal time across the 33 runs that our CPU counter log covered, and between 3.7% and 35.5% in single runs. The Go build took between 121.3 and 199.6 seconds at limits it never came close to. We do not read anything into a time difference between limits unless it repeats in every round, as the tsc gap did.</li>
<li><strong>Two or three runs per cell is enough to see a failure repeat and a peak hold steady, and that is all.</strong> The anonymous peaks varied by up to 21% for the Go build (353 to 441 MiB at 8 GB) and up to 15% for Gradle, and 6% or less for Vite, tsc and Docker between runs of the same cell.</li>
<li><strong>We did not split shared builds.</strong> We cannot say how much of the Vite build's memory is Rollup and how much is the checker, or how much of the Docker build's page cache is layer extraction.</li>
<li><strong>Rust ran once.</strong> One ripgrep build at 8 GB, so no repeats and no 2 GB or 4 GB data.</li>
</ul>
<h2>Run it yourself</h2><p>Create a 4 vCPU, 8 GB Ubuntu 24.04 machine, copy the files below to <code>/root/bench/</code>, and run:</p>
<pre><code class="hljs language-bash" tabindex="0">bash /root/bench/setup.sh
bash /root/bench/p1.sh warmup   <span class="hljs-comment"># probes the runtimes, one unmeasured run per build</span>
bash /root/bench/p1.sh matrix   <span class="hljs-comment"># 3 rounds x 3 limits x each build (we stopped it after round 2)</span>
bash /root/bench/p1.sh round3   <span class="hljs-comment"># what we ran as round 3: 8 GB and 2 GB only</span>
bash /root/bench/p1.sh redo     <span class="hljs-comment"># reruns any cell that left no result</span>
python3 summarize_p1.py /root/res/p1
</code></pre><p>Each run writes its own directory with <code>result.env</code>, <code>memory.events</code>, <code>samples.json</code> and the build log, so you can read any single run on its own.</p>
<details>
<summary>setup.sh: toolchains, pinned sources, dependencies, registries</summary><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># Droplet setup: toolchains, pinned sources, dependency prefetch, local registries.</span>
<span class="hljs-built_in">set</span> -euxo pipefail
<span class="hljs-built_in">export</span> DEBIAN_FRONTEND=noninteractive
<span class="hljs-built_in">mkdir</span> -p /root/src /root/res
cloud-init status --<span class="hljs-built_in">wait</span> || <span class="hljs-literal">true</span>   <span class="hljs-comment"># first boot runs apt; wait for its lock</span>

<span class="hljs-function"><span class="hljs-title">fetch_sha</span></span>() { git init -q <span class="hljs-string">"<span class="hljs-variable">$1</span>"</span>; git -C <span class="hljs-string">"<span class="hljs-variable">$1</span>"</span> remote add origin <span class="hljs-string">"<span class="hljs-variable">$2</span>"</span>; git -C <span class="hljs-string">"<span class="hljs-variable">$1</span>"</span> fetch -q --depth 1 origin <span class="hljs-string">"<span class="hljs-variable">$3</span>"</span>; git -C <span class="hljs-string">"<span class="hljs-variable">$1</span>"</span> checkout -q FETCH_HEAD; }
fetch_sha /root/src/excalidraw https://github.com/excalidraw/excalidraw 8bcdca4526867433f99fdfcb55a063668db0c9dc
fetch_sha /root/src/plantuml https://github.com/plantuml/plantuml db06d70e18e71839dd5e330ac4086f3845371e3b
fetch_sha /root/src/cli https://github.com/cli/cli 9490d204a2ba95434153dedce99898ce9f281426
git <span class="hljs-built_in">clone</span> -q --depth 1 --branch 15.2.0 https://github.com/BurntSushi/ripgrep /root/src/ripgrep
fetch_sha /root/src/msd https://github.com/GoogleCloudPlatform/microservices-demo c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712

curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz
curl -fsSL https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.xz | tar -C /usr/local --strip-components=1 -xJ
curl -fsSL https://sh.rustup.rs | sh -s -- -y -q --profile minimal --default-toolchain stable
apt-get update -q
apt-get install -y -q openjdk-21-jdk-headless build-essential git jq zstd python3 ca-certificates curl
curl -fsSL https://get.docker.com | sh
<span class="hljs-built_in">export</span> PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:<span class="hljs-variable">$PATH</span> GOTOOLCHAIN=<span class="hljs-built_in">local</span>
npm install -g yarn@1.22.22

docker pull -q moby/buildkit:v0.34.1
docker pull -q registry:2.8.3
docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3
docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3
docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3

<span class="hljs-comment"># dependencies are downloaded once, outside the measured runs</span>
(<span class="hljs-built_in">cd</span> /root/src/excalidraw &amp;&amp; yarn install --frozen-lockfile --network-timeout 600000)
(<span class="hljs-built_in">cd</span> /root/src/cli &amp;&amp; go mod download)
(<span class="hljs-built_in">cd</span> /root/src/ripgrep &amp;&amp; cargo fetch --locked)

<span class="hljs-comment"># systemd stops a scope after an OOM kill by default; keep the wrapper alive to record the result</span>
<span class="hljs-built_in">mkdir</span> -p /etc/systemd/system.conf.d
<span class="hljs-built_in">printf</span> <span class="hljs-string">'[Manager]\nDefaultOOMPolicy=continue\n'</span> &gt; /etc/systemd/system.conf.d/oom.conf
systemctl daemon-reexec
</code></pre></details><details>
<summary>wrap.sh: runs inside the scope and records the result</summary><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># Runs inside the systemd scope. usage: wrap.sh OUTDIR TIMEOUT_S cmd...</span>
out=<span class="hljs-variable">$1</span>; tmo=<span class="hljs-variable">$2</span>; <span class="hljs-built_in">shift</span> 2
cg=/sys/fs/cgroup$(sed -n <span class="hljs-string">'s/^0:://p'</span> /proc/self/cgroup)
<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>"</span> &gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/cgroup"</span>
<span class="hljs-built_in">cat</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.max"</span> &gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/memory.max"</span>
t0=$(<span class="hljs-built_in">date</span> +%s.%N)
<span class="hljs-built_in">timeout</span> -k 15 <span class="hljs-string">"<span class="hljs-variable">$tmo</span>"</span> <span class="hljs-string">"<span class="hljs-variable">$@</span>"</span> &gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/build.log"</span> 2&gt;&amp;1
rc=$?
t1=$(<span class="hljs-built_in">date</span> +%s.%N)
{
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"rc=<span class="hljs-variable">$rc</span>"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"t0=<span class="hljs-variable">$t0</span>"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"t1=<span class="hljs-variable">$t1</span>"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"peak=<span class="hljs-subst">$(cat <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.peak"</span>)</span>"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"current_after=<span class="hljs-subst">$(cat <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.current"</span>)</span>"</span>
  awk <span class="hljs-string">'$1=="anon"||$1=="file"{print $1"_after="$2}'</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.stat"</span>
} &gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/result.env"</span>
<span class="hljs-built_in">cp</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.events"</span> <span class="hljs-string">"<span class="hljs-variable">$out</span>/memory.events"</span>
<span class="hljs-keyword">for</span> p <span class="hljs-keyword">in</span> $(<span class="hljs-built_in">cat</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>/cgroup.procs"</span>); <span class="hljs-keyword">do</span>
  [ <span class="hljs-string">"<span class="hljs-variable">$p</span>"</span> = <span class="hljs-string">"$$"</span> ] &amp;&amp; <span class="hljs-built_in">continue</span>
  ps -o pid=,rss=,args= -p <span class="hljs-string">"<span class="hljs-variable">$p</span>"</span> | <span class="hljs-built_in">cut</span> -c1-400
<span class="hljs-keyword">done</span> &gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/leftover.txt"</span> 2&gt;/dev/null
<span class="hljs-keyword">if</span> [ -n <span class="hljs-string">"<span class="hljs-variable">${LEFTOVER_WAIT:-}</span>"</span> ]; <span class="hljs-keyword">then</span>
  <span class="hljs-built_in">sleep</span> <span class="hljs-string">"<span class="hljs-variable">$LEFTOVER_WAIT</span>"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"current_later=<span class="hljs-subst">$(cat <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.current"</span>)</span>"</span> &gt;&gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/result.env"</span>
  awk <span class="hljs-string">'$1=="anon"{print "anon_later="$2}'</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>/memory.stat"</span> &gt;&gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/result.env"</span>
  <span class="hljs-keyword">for</span> p <span class="hljs-keyword">in</span> $(<span class="hljs-built_in">cat</span> <span class="hljs-string">"<span class="hljs-variable">$cg</span>/cgroup.procs"</span>); <span class="hljs-keyword">do</span>
    [ <span class="hljs-string">"<span class="hljs-variable">$p</span>"</span> = <span class="hljs-string">"$$"</span> ] &amp;&amp; <span class="hljs-built_in">continue</span>
    ps -o pid=,rss=,args= -p <span class="hljs-string">"<span class="hljs-variable">$p</span>"</span> | <span class="hljs-built_in">cut</span> -c1-400
  <span class="hljs-keyword">done</span> &gt; <span class="hljs-string">"<span class="hljs-variable">$out</span>/leftover_later.txt"</span> 2&gt;/dev/null
<span class="hljs-keyword">fi</span>
<span class="hljs-built_in">touch</span> <span class="hljs-string">"<span class="hljs-variable">$out</span>/done"</span>
<span class="hljs-built_in">exit</span> <span class="hljs-variable">$rc</span>
</code></pre></details><details>
<summary>sample.py: polls memory.stat every 100 ms from outside the scope</summary><pre><code class="hljs language-python" tabindex="0"><span class="hljs-comment">#!/usr/bin/env python3</span>
<span class="hljs-string">"""Poll one cgroup v2 directory every 100 ms until a done file appears.

usage: sample.py CGROUP_DIR OUT_JSON DONE_FILE
Records the maximum of anon, file, kernel and memory.current, plus a series.
"""</span>
<span class="hljs-keyword">import</span> json, os, sys, time

cg, out, done = sys.argv[<span class="hljs-number">1</span>:<span class="hljs-number">4</span>]
start = time.time()
<span class="hljs-keyword">while</span> <span class="hljs-keyword">not</span> os.path.isdir(cg):
    <span class="hljs-keyword">if</span> time.time() - start &gt; <span class="hljs-number">60</span> <span class="hljs-keyword">or</span> os.path.exists(done):
        json.dump({<span class="hljs-string">"error"</span>: <span class="hljs-string">"cgroup never appeared"</span>}, <span class="hljs-built_in">open</span>(out, <span class="hljs-string">"w"</span>))
        sys.exit(<span class="hljs-number">0</span>)
    time.sleep(<span class="hljs-number">0.02</span>)

keys = (<span class="hljs-string">"anon"</span>, <span class="hljs-string">"file"</span>, <span class="hljs-string">"kernel"</span>, <span class="hljs-string">"shmem"</span>)
mx = {k: <span class="hljs-number">0</span> <span class="hljs-keyword">for</span> k <span class="hljs-keyword">in</span> keys}
mx[<span class="hljs-string">"current"</span>] = <span class="hljs-number">0</span>
series = []
t0 = time.time()
n = <span class="hljs-number">0</span>
<span class="hljs-keyword">while</span> <span class="hljs-keyword">not</span> os.path.exists(done):
    <span class="hljs-keyword">try</span>:
        <span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(os.path.join(cg, <span class="hljs-string">"memory.stat"</span>)) <span class="hljs-keyword">as</span> f:
            st = {}
            <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> f:
                k, v = line.split()
                <span class="hljs-keyword">if</span> k <span class="hljs-keyword">in</span> keys:
                    st[k] = <span class="hljs-built_in">int</span>(v)
        <span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(os.path.join(cg, <span class="hljs-string">"memory.current"</span>)) <span class="hljs-keyword">as</span> f:
            cur = <span class="hljs-built_in">int</span>(f.read())
    <span class="hljs-keyword">except</span> OSError:
        <span class="hljs-keyword">break</span>
    n += <span class="hljs-number">1</span>
    <span class="hljs-keyword">for</span> k <span class="hljs-keyword">in</span> keys:
        mx[k] = <span class="hljs-built_in">max</span>(mx[k], st.get(k, <span class="hljs-number">0</span>))
    mx[<span class="hljs-string">"current"</span>] = <span class="hljs-built_in">max</span>(mx[<span class="hljs-string">"current"</span>], cur)
    series.append([<span class="hljs-built_in">round</span>(time.time() - t0, <span class="hljs-number">2</span>), st.get(<span class="hljs-string">"anon"</span>, <span class="hljs-number">0</span>), st.get(<span class="hljs-string">"file"</span>, <span class="hljs-number">0</span>), cur])
    time.sleep(<span class="hljs-number">0.1</span>)

json.dump({<span class="hljs-string">"max"</span>: mx, <span class="hljs-string">"samples"</span>: n, <span class="hljs-string">"interval_s"</span>: <span class="hljs-number">0.1</span>, <span class="hljs-string">"series"</span>: series}, <span class="hljs-built_in">open</span>(out, <span class="hljs-string">"w"</span>))
</code></pre></details><details>
<summary>p1-cmds.sh: the builds, and what is deleted before each run</summary><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># Build definitions for post 1. prep_* runs outside the cgroup, cmd_* inside it.</span>
<span class="hljs-built_in">export</span> PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:<span class="hljs-variable">$PATH</span> GOTOOLCHAIN=<span class="hljs-built_in">local</span>
EX=/root/src/excalidraw PU=/root/src/plantuml GH=/root/src/cli RG=/root/src/ripgrep
FRONTEND=/root/src/msd/src/frontend

<span class="hljs-function"><span class="hljs-title">stop_gradle</span></span>() { pkill -f GradleDaemon || <span class="hljs-literal">true</span>; pkill -f GradleWrapperMain || <span class="hljs-literal">true</span>; pkill -f KotlinCompileDaemon || <span class="hljs-literal">true</span>; <span class="hljs-built_in">sleep</span> 1; }

<span class="hljs-function"><span class="hljs-title">prep_vite</span></span>() { <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$EX</span>/excalidraw-app/build <span class="hljs-variable">$EX</span>/node_modules/.vite <span class="hljs-variable">$EX</span>/excalidraw-app/node_modules/.vite; }
<span class="hljs-function"><span class="hljs-title">cmd_vite</span></span>() { <span class="hljs-built_in">cd</span> <span class="hljs-variable">$EX</span> &amp;&amp; yarn build:app:docker; }

<span class="hljs-function"><span class="hljs-title">prep_tsc</span></span>() { find <span class="hljs-variable">$EX</span> -maxdepth 2 -name <span class="hljs-string">'*.tsbuildinfo'</span> -delete; }
<span class="hljs-function"><span class="hljs-title">cmd_tsc</span></span>() { <span class="hljs-built_in">cd</span> <span class="hljs-variable">$EX</span> &amp;&amp; node_modules/.bin/tsc -p tsconfig.json --extendedDiagnostics; }

<span class="hljs-function"><span class="hljs-title">prep_gradle_daemon</span></span>() { stop_gradle; <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$PU</span>/build <span class="hljs-variable">$PU</span>/.gradle; }
<span class="hljs-function"><span class="hljs-title">cmd_gradle_daemon</span></span>() { <span class="hljs-built_in">cd</span> <span class="hljs-variable">$PU</span> &amp;&amp; ./gradlew jar --console=plain --no-build-cache; }

<span class="hljs-function"><span class="hljs-title">prep_gradle_nodaemon</span></span>() { stop_gradle; <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$PU</span>/build <span class="hljs-variable">$PU</span>/.gradle; }
<span class="hljs-function"><span class="hljs-title">cmd_gradle_nodaemon</span></span>() { <span class="hljs-built_in">cd</span> <span class="hljs-variable">$PU</span> &amp;&amp; ./gradlew --no-daemon jar --console=plain --no-build-cache; }

<span class="hljs-function"><span class="hljs-title">prep_go</span></span>() { <span class="hljs-built_in">rm</span> -rf /root/gocache /root/out; <span class="hljs-built_in">mkdir</span> -p /root/out; }
<span class="hljs-function"><span class="hljs-title">cmd_go</span></span>() { <span class="hljs-built_in">cd</span> <span class="hljs-variable">$GH</span> &amp;&amp; GOCACHE=/root/gocache go build -o /root/out/gh ./cmd/gh; }

<span class="hljs-function"><span class="hljs-title">prep_rust</span></span>() { <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$RG</span>/target; }
<span class="hljs-function"><span class="hljs-title">cmd_rust</span></span>() { <span class="hljs-built_in">cd</span> <span class="hljs-variable">$RG</span> &amp;&amp; cargo build --release --locked; }

<span class="hljs-function"><span class="hljs-title">prep_docker</span></span>() { docker buildx <span class="hljs-built_in">ls</span> --format <span class="hljs-string">'{{.Name}}'</span> 2&gt;/dev/null | grep <span class="hljs-string">'^p1d'</span> | xargs -r -n1 docker buildx <span class="hljs-built_in">rm</span> &gt;/dev/null 2&gt;&amp;1 || <span class="hljs-literal">true</span>; }

<span class="hljs-keyword">if</span> [ <span class="hljs-string">"<span class="hljs-variable">${BASH_SOURCE[0]}</span>"</span> = <span class="hljs-string">"<span class="hljs-variable">$0</span>"</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-string">"<span class="hljs-variable">$@</span>"</span>; <span class="hljs-keyword">fi</span>
</code></pre></details><details>
<summary>p1.sh: probe, warm-up, matrix, round 3, redo, and the docker build cell</summary><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-comment"># Post 1 matrix. usage: p1.sh warmup | p1.sh matrix</span>
<span class="hljs-built_in">set</span> -u
B=/root/bench R=/root/res/p1
<span class="hljs-built_in">mkdir</span> -p <span class="hljs-variable">$R</span>
<span class="hljs-built_in">source</span> <span class="hljs-variable">$B</span>/p1-cmds.sh
LIMITS=(8G 4G 2G)
BUILDS=(vite tsc gradle_daemon gradle_nodaemon go rust docker)
<span class="hljs-comment"># rust ran once (warmup) only: the matrix did not fit the droplet time budget</span>
MATRIX_BUILDS=(vite tsc gradle_daemon gradle_nodaemon go docker)
CPUS=0-3
<span class="hljs-function"><span class="hljs-title">log</span></span>() { <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-subst">$(date -u +%FT%TZ)</span> $*"</span> | <span class="hljs-built_in">tee</span> -a <span class="hljs-variable">$R</span>/progress.log; }

<span class="hljs-function"><span class="hljs-title">scope_run</span></span>() { <span class="hljs-comment"># unit limit out tmo cmd...</span>
  <span class="hljs-built_in">local</span> unit=<span class="hljs-variable">$1</span> lim=<span class="hljs-variable">$2</span> out=<span class="hljs-variable">$3</span> tmo=<span class="hljs-variable">$4</span>; <span class="hljs-built_in">shift</span> 4
  python3 <span class="hljs-variable">$B</span>/sample.py /sys/fs/cgroup/bench.slice/<span class="hljs-variable">$unit</span>.scope <span class="hljs-variable">$out</span>/samples.json <span class="hljs-variable">$out</span>/done &amp;
  <span class="hljs-built_in">local</span> sp=$!
  systemd-run --scope --quiet --slice=bench.slice --unit=<span class="hljs-variable">$unit</span> \
    -p MemoryMax=<span class="hljs-variable">$lim</span> -p MemorySwapMax=0 -p AllowedCPUs=<span class="hljs-variable">$CPUS</span> -p OOMPolicy=<span class="hljs-built_in">continue</span> \
    <span class="hljs-variable">$B</span>/wrap.sh <span class="hljs-variable">$out</span> <span class="hljs-variable">$tmo</span> <span class="hljs-string">"<span class="hljs-variable">$@</span>"</span>
  <span class="hljs-built_in">wait</span> <span class="hljs-variable">$sp</span>
  systemctl stop <span class="hljs-variable">$unit</span>.scope 2&gt;/dev/null || <span class="hljs-literal">true</span>
}

<span class="hljs-function"><span class="hljs-title">docker_cell</span></span>() { <span class="hljs-comment"># lim out tmo tag</span>
  <span class="hljs-built_in">local</span> lim=<span class="hljs-variable">$1</span> out=<span class="hljs-variable">$2</span> tmo=<span class="hljs-variable">$3</span> tag=<span class="hljs-variable">$4</span>
  <span class="hljs-built_in">local</span> name=$(<span class="hljs-built_in">echo</span> p1d<span class="hljs-variable">$tag</span> | <span class="hljs-built_in">tr</span> A-Z a-z) m=$(<span class="hljs-built_in">echo</span> <span class="hljs-variable">$1</span> | <span class="hljs-built_in">tr</span> G g)
  docker buildx create --name <span class="hljs-variable">$name</span> --driver docker-container \
    --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \
    --driver-opt memory=<span class="hljs-variable">$m</span> --driver-opt memory-swap=<span class="hljs-variable">$m</span> --driver-opt cpuset-cpus=<span class="hljs-variable">$CPUS</span> \
    --buildkitd-config <span class="hljs-variable">$B</span>/buildkitd.toml --bootstrap &gt; <span class="hljs-variable">$out</span>/create.log 2&gt;&amp;1
  <span class="hljs-built_in">local</span> cid; cid=$(docker inspect -f <span class="hljs-string">'{{.Id}}'</span> buildx_buildkit_<span class="hljs-variable">${name}</span>0)
  <span class="hljs-built_in">local</span> cg=/sys/fs/cgroup/system.slice/docker-<span class="hljs-variable">$cid</span>.scope
  <span class="hljs-built_in">cat</span> <span class="hljs-variable">$cg</span>/memory.max &gt; <span class="hljs-variable">$out</span>/container_memory.max
  <span class="hljs-built_in">mkdir</span> -p <span class="hljs-variable">$out</span>/cli
  python3 <span class="hljs-variable">$B</span>/sample.py <span class="hljs-variable">$cg</span> <span class="hljs-variable">$out</span>/container_samples.json <span class="hljs-variable">$out</span>/cli/done &amp;
  <span class="hljs-built_in">local</span> sp=$!
  scope_run p1-docker-<span class="hljs-variable">$tag</span> <span class="hljs-variable">$lim</span> <span class="hljs-variable">$out</span>/cli <span class="hljs-variable">$tmo</span> \
    docker buildx build --builder <span class="hljs-variable">$name</span> --progress=plain --provenance=<span class="hljs-literal">false</span> --output <span class="hljs-built_in">type</span>=cacheonly <span class="hljs-variable">$FRONTEND</span>
  <span class="hljs-built_in">wait</span> <span class="hljs-variable">$sp</span>
  { <span class="hljs-built_in">echo</span> <span class="hljs-string">"container_peak=<span class="hljs-subst">$(cat $cg/memory.peak)</span>"</span>; <span class="hljs-built_in">cat</span> <span class="hljs-variable">$out</span>/cli/result.env; } &gt; <span class="hljs-variable">$out</span>/result.env
  <span class="hljs-built_in">cp</span> <span class="hljs-variable">$cg</span>/memory.events <span class="hljs-variable">$out</span>/container_memory.events
  <span class="hljs-built_in">cp</span> <span class="hljs-variable">$out</span>/cli/build.log <span class="hljs-variable">$out</span>/build.log
  docker buildx <span class="hljs-built_in">rm</span> <span class="hljs-variable">$name</span> &gt; /dev/null 2&gt;&amp;1
}

<span class="hljs-function"><span class="hljs-title">run_cell</span></span>() { <span class="hljs-comment"># build limit run tmo</span>
  <span class="hljs-built_in">local</span> b=<span class="hljs-variable">$1</span> lim=<span class="hljs-variable">$2</span> r=<span class="hljs-variable">$3</span> tmo=<span class="hljs-variable">$4</span>
  <span class="hljs-built_in">local</span> tag=<span class="hljs-variable">$b</span>-<span class="hljs-variable">$lim</span>-<span class="hljs-variable">$r</span> out=<span class="hljs-variable">$R</span>/<span class="hljs-variable">$b</span>/<span class="hljs-variable">$lim</span>/<span class="hljs-variable">$r</span>
  <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$out</span>; <span class="hljs-built_in">mkdir</span> -p <span class="hljs-variable">$out</span>
  prep_<span class="hljs-variable">$b</span>
  <span class="hljs-built_in">sync</span>; <span class="hljs-built_in">echo</span> 3 &gt; /proc/sys/vm/drop_caches; <span class="hljs-built_in">sleep</span> 2
  <span class="hljs-keyword">if</span> [ <span class="hljs-variable">$b</span> = docker ]; <span class="hljs-keyword">then</span>
    docker_cell <span class="hljs-variable">$lim</span> <span class="hljs-variable">$out</span> <span class="hljs-variable">$tmo</span> <span class="hljs-variable">${lim}</span><span class="hljs-variable">$r</span>
  <span class="hljs-keyword">else</span>
    <span class="hljs-keyword">case</span> <span class="hljs-variable">$b</span> <span class="hljs-keyword">in</span> gradle_*) <span class="hljs-built_in">export</span> LEFTOVER_WAIT=5 ;; <span class="hljs-keyword">esac</span>
    scope_run p1-<span class="hljs-variable">$tag</span> <span class="hljs-variable">$lim</span> <span class="hljs-variable">$out</span> <span class="hljs-variable">$tmo</span> bash <span class="hljs-variable">$B</span>/p1-cmds.sh cmd_<span class="hljs-variable">$b</span>
    <span class="hljs-built_in">unset</span> LEFTOVER_WAIT
  <span class="hljs-keyword">fi</span>
  [ <span class="hljs-variable">$b</span> = gradle_daemon ] || [ <span class="hljs-variable">$b</span> = gradle_nodaemon ] &amp;&amp; stop_gradle
  <span class="hljs-built_in">log</span> <span class="hljs-string">"<span class="hljs-variable">$tag</span> <span class="hljs-subst">$(tr '\n' ' ' &lt; $out/result.env)</span>"</span>
}

<span class="hljs-function"><span class="hljs-title">probe</span></span>() { <span class="hljs-comment"># record what the runtimes see inside each limit</span>
  <span class="hljs-keyword">for</span> lim <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${LIMITS[@]}</span>"</span>; <span class="hljs-keyword">do</span>
    systemd-run --scope --quiet --slice=bench.slice --unit=probe-<span class="hljs-variable">$lim</span> -p MemoryMax=<span class="hljs-variable">$lim</span> -p MemorySwapMax=0 -p AllowedCPUs=<span class="hljs-variable">$CPUS</span> \
      bash -c <span class="hljs-string">'echo "nproc=$(nproc)"; echo "memory.max=$(cat /sys/fs/cgroup$(sed -n "s/^0:://p" /proc/self/cgroup)/memory.max)";
        echo "node_heap_size_limit=$(node -e "console.log(require(\"v8\").getHeapStatistics().heap_size_limit)")";
        java -XX:+PrintFlagsFinal -version 2&gt;/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage|ActiveProcessorCount) "'</span> &gt; <span class="hljs-variable">$R</span>/probe-<span class="hljs-variable">$lim</span>.txt 2&gt;&amp;1
  <span class="hljs-keyword">done</span>
  { <span class="hljs-built_in">echo</span> <span class="hljs-string">"nproc=<span class="hljs-subst">$(nproc)</span>"</span>; free -b | <span class="hljs-built_in">head</span> -2; <span class="hljs-built_in">echo</span> <span class="hljs-string">"node_heap_size_limit=<span class="hljs-subst">$(node -e 'console.log(require(<span class="hljs-string">"v8"</span>)</span>.getHeapStatistics().heap_size_limit)')"</span>;
    java -XX:+PrintFlagsFinal -version 2&gt;/dev/null | grep -E <span class="hljs-string">" (MaxHeapSize|MaxRAMPercentage) "</span>; } &gt; <span class="hljs-variable">$R</span>/probe-host.txt 2&gt;&amp;1
}

<span class="hljs-keyword">case</span> <span class="hljs-variable">${1:-}</span> <span class="hljs-keyword">in</span>
  warmup)
    probe
    <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${BUILDS[@]}</span>"</span>; <span class="hljs-keyword">do</span> run_cell <span class="hljs-variable">$b</span> 8G r0 900; <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> WARMUP-DONE ;;
  matrix)
    <span class="hljs-built_in">declare</span> -A TMO
    <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${MATRIX_BUILDS[@]}</span>"</span>; <span class="hljs-keyword">do</span>
      . &lt;(grep -E <span class="hljs-string">'^t[01]='</span> <span class="hljs-variable">$R</span>/<span class="hljs-variable">$b</span>/8G/r0/result.env)
      w=$(python3 -c <span class="hljs-string">"print(int(float('<span class="hljs-variable">$t1</span>')-float('<span class="hljs-variable">$t0</span>')))"</span>)
      t=$((<span class="hljs-number">3</span> * w + <span class="hljs-number">60</span>)); [ <span class="hljs-variable">$t</span> -gt 600 ] &amp;&amp; t=600; [ <span class="hljs-variable">$t</span> -lt 120 ] &amp;&amp; t=120
      TMO[<span class="hljs-variable">$b</span>]=<span class="hljs-variable">$t</span>
    <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> <span class="hljs-string">"timeouts: <span class="hljs-subst">$(for b in <span class="hljs-string">"<span class="hljs-variable">${MATRIX_BUILDS[@]}</span>"</span>; do printf '%s=%s ' $b ${TMO[$b]}; done)</span>"</span>
    <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3; <span class="hljs-keyword">do</span>
      <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${MATRIX_BUILDS[@]}</span>"</span>; <span class="hljs-keyword">do</span>
        <span class="hljs-keyword">for</span> lim <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${LIMITS[@]}</span>"</span>; <span class="hljs-keyword">do</span> run_cell <span class="hljs-variable">$b</span> <span class="hljs-variable">$lim</span> <span class="hljs-variable">$r</span> <span class="hljs-variable">${TMO[$b]}</span>; <span class="hljs-keyword">done</span>
      <span class="hljs-keyword">done</span>
    <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> MATRIX-DONE ;;
  round3) <span class="hljs-comment"># third run at 8G and 2G only: the full round did not fit the droplet time budget</span>
    <span class="hljs-built_in">declare</span> -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351)
    <span class="hljs-built_in">log</span> <span class="hljs-string">"round 3 at 8G and 2G only"</span>
    <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${MATRIX_BUILDS[@]}</span>"</span>; <span class="hljs-keyword">do</span>
      <span class="hljs-keyword">for</span> lim <span class="hljs-keyword">in</span> 8G 2G; <span class="hljs-keyword">do</span> run_cell <span class="hljs-variable">$b</span> <span class="hljs-variable">$lim</span> r3 <span class="hljs-variable">${TMO[$b]}</span>; <span class="hljs-keyword">done</span>
    <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> MATRIX-DONE ;;
  redo) <span class="hljs-comment"># rerun cells that left no result (systemd stopped the scope after an OOM kill)</span>
    <span class="hljs-built_in">declare</span> -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351)
    <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${MATRIX_BUILDS[@]}</span>"</span>; <span class="hljs-keyword">do</span> <span class="hljs-keyword">for</span> lim <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${LIMITS[@]}</span>"</span>; <span class="hljs-keyword">do</span> <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> r1 r2 r3; <span class="hljs-keyword">do</span>
      d=<span class="hljs-variable">$R</span>/<span class="hljs-variable">$b</span>/<span class="hljs-variable">$lim</span>/<span class="hljs-variable">$r</span>
      <span class="hljs-keyword">if</span> [ -d <span class="hljs-variable">$d</span> ] &amp;&amp; ! grep -q <span class="hljs-string">'^rc='</span> <span class="hljs-variable">$d</span>/result.env 2&gt;/dev/null; <span class="hljs-keyword">then</span>
        <span class="hljs-built_in">rm</span> -rf <span class="hljs-variable">$d</span>.lost; <span class="hljs-built_in">mv</span> <span class="hljs-variable">$d</span> <span class="hljs-variable">$d</span>.lost; <span class="hljs-built_in">log</span> <span class="hljs-string">"redo <span class="hljs-variable">$b</span> <span class="hljs-variable">$lim</span> <span class="hljs-variable">$r</span>"</span>
        systemctl reset-failed p1-<span class="hljs-variable">$b</span>-<span class="hljs-variable">$lim</span>-<span class="hljs-variable">$r</span>.scope 2&gt;/dev/null  <span class="hljs-comment"># added after our run: a failed scope blocks its name</span>
        run_cell <span class="hljs-variable">$b</span> <span class="hljs-variable">$lim</span> <span class="hljs-variable">$r</span> <span class="hljs-variable">${TMO[$b]}</span>
      <span class="hljs-keyword">fi</span>
    <span class="hljs-keyword">done</span>; <span class="hljs-keyword">done</span>; <span class="hljs-keyword">done</span>
    <span class="hljs-built_in">log</span> REDO-DONE ;;
  *) <span class="hljs-built_in">echo</span> <span class="hljs-string">"usage: <span class="hljs-variable">$0</span> warmup|matrix|round3|redo"</span>; <span class="hljs-built_in">exit</span> 2 ;;
<span class="hljs-keyword">esac</span>
</code></pre></details><details>
<summary>buildkitd.toml: send base image pulls to the local mirror</summary><pre><code class="hljs language-toml" tabindex="0"><span class="hljs-section">[registry."docker.io"]</span>
  <span class="hljs-attr">mirrors</span> = [<span class="hljs-string">"127.0.0.1:5001"</span>]
<span class="hljs-section">[registry."gcr.io"]</span>
  <span class="hljs-attr">mirrors</span> = [<span class="hljs-string">"127.0.0.1:5002"</span>]
<span class="hljs-section">[registry."127.0.0.1:5000"]</span>
  <span class="hljs-attr">http</span> = <span class="hljs-literal">true</span>
  <span class="hljs-attr">insecure</span> = <span class="hljs-literal">true</span>
<span class="hljs-section">[registry."127.0.0.1:5001"]</span>
  <span class="hljs-attr">http</span> = <span class="hljs-literal">true</span>
  <span class="hljs-attr">insecure</span> = <span class="hljs-literal">true</span>
<span class="hljs-section">[registry."127.0.0.1:5002"]</span>
  <span class="hljs-attr">http</span> = <span class="hljs-literal">true</span>
  <span class="hljs-attr">insecure</span> = <span class="hljs-literal">true</span>
</code></pre></details><details>
<summary>summarize_p1.py: medians per build and limit</summary><pre><code class="hljs language-python" tabindex="0"><span class="hljs-comment">#!/usr/bin/env python3</span>
<span class="hljs-string">"""Summarize post 1 runs: medians per build and limit over the measured runs (r1..r3)."""</span>
<span class="hljs-keyword">import</span> json, os, re, statistics, sys

root = sys.argv[<span class="hljs-number">1</span>]
BUILDS = [<span class="hljs-string">"vite"</span>, <span class="hljs-string">"tsc"</span>, <span class="hljs-string">"gradle_daemon"</span>, <span class="hljs-string">"gradle_nodaemon"</span>, <span class="hljs-string">"go"</span>, <span class="hljs-string">"rust"</span>, <span class="hljs-string">"docker"</span>]
LIMITS = [<span class="hljs-string">"8G"</span>, <span class="hljs-string">"4G"</span>, <span class="hljs-string">"2G"</span>]
MiB = <span class="hljs-number">2</span> ** <span class="hljs-number">20</span>

<span class="hljs-keyword">def</span> <span class="hljs-title function_">env</span>(<span class="hljs-params">path</span>):
    d = {}
    <span class="hljs-keyword">if</span> os.path.exists(path):
        <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> <span class="hljs-built_in">open</span>(path):
            <span class="hljs-keyword">if</span> <span class="hljs-string">"="</span> <span class="hljs-keyword">in</span> line:
                k, v = line.strip().split(<span class="hljs-string">"="</span>, <span class="hljs-number">1</span>)
                d[k] = v
    <span class="hljs-keyword">return</span> d

<span class="hljs-keyword">def</span> <span class="hljs-title function_">events</span>(<span class="hljs-params">path</span>):
    d = {}
    <span class="hljs-keyword">if</span> os.path.exists(path):
        <span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> <span class="hljs-built_in">open</span>(path):
            k, v = line.split()
            d[k] = <span class="hljs-built_in">int</span>(v)
    <span class="hljs-keyword">return</span> d

<span class="hljs-keyword">def</span> <span class="hljs-title function_">run_info</span>(<span class="hljs-params">d, build</span>):
    r = env(os.path.join(d, <span class="hljs-string">"result.env"</span>))
    <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> r <span class="hljs-keyword">or</span> <span class="hljs-string">"rc"</span> <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span> r:
        <span class="hljs-keyword">return</span> <span class="hljs-literal">None</span>
    log = <span class="hljs-built_in">open</span>(os.path.join(d, <span class="hljs-string">"build.log"</span>), errors=<span class="hljs-string">"replace"</span>).read() <span class="hljs-keyword">if</span> os.path.exists(os.path.join(d, <span class="hljs-string">"build.log"</span>)) <span class="hljs-keyword">else</span> <span class="hljs-string">""</span>
    <span class="hljs-keyword">if</span> build == <span class="hljs-string">"docker"</span>:
        <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> r.get(<span class="hljs-string">"container_peak"</span>):
            <span class="hljs-keyword">return</span> <span class="hljs-literal">None</span>  <span class="hljs-comment"># warm-up run 0 recorded no container numbers (builder name case bug)</span>
        ev = events(os.path.join(d, <span class="hljs-string">"container_memory.events"</span>))
        smp = json.load(<span class="hljs-built_in">open</span>(os.path.join(d, <span class="hljs-string">"container_samples.json"</span>)))
        peak = <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"container_peak"</span>])
        cli_peak = <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"peak"</span>])
    <span class="hljs-keyword">else</span>:
        ev = events(os.path.join(d, <span class="hljs-string">"memory.events"</span>))
        smp = json.load(<span class="hljs-built_in">open</span>(os.path.join(d, <span class="hljs-string">"samples.json"</span>)))
        peak = <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"peak"</span>])
        cli_peak = <span class="hljs-literal">None</span>
    rc = <span class="hljs-built_in">int</span>(r[<span class="hljs-string">"rc"</span>])
    wall = <span class="hljs-built_in">float</span>(r[<span class="hljs-string">"t1"</span>]) - <span class="hljs-built_in">float</span>(r[<span class="hljs-string">"t0"</span>])
    <span class="hljs-keyword">if</span> rc == <span class="hljs-number">0</span>:
        outcome = <span class="hljs-string">"ok"</span>
    <span class="hljs-keyword">elif</span> ev.get(<span class="hljs-string">"oom_kill"</span>, <span class="hljs-number">0</span>) &gt; <span class="hljs-number">0</span>:
        outcome = <span class="hljs-string">"oom_kill"</span>
    <span class="hljs-keyword">elif</span> <span class="hljs-string">"heap out of memory"</span> <span class="hljs-keyword">in</span> log <span class="hljs-keyword">or</span> <span class="hljs-string">"Reached heap limit"</span> <span class="hljs-keyword">in</span> log:
        outcome = <span class="hljs-string">"node_heap"</span>
    <span class="hljs-keyword">elif</span> <span class="hljs-string">"OutOfMemoryError"</span> <span class="hljs-keyword">in</span> log:
        outcome = <span class="hljs-string">"java_heap"</span>
    <span class="hljs-keyword">elif</span> rc <span class="hljs-keyword">in</span> (<span class="hljs-number">124</span>, <span class="hljs-number">137</span>):
        outcome = <span class="hljs-string">"timeout"</span>
    <span class="hljs-keyword">else</span>:
        outcome = <span class="hljs-string">f"fail_rc<span class="hljs-subst">{rc}</span>"</span>
    mx = smp.get(<span class="hljs-string">"max"</span>, {})
    <span class="hljs-keyword">return</span> {
        <span class="hljs-string">"rc"</span>: rc, <span class="hljs-string">"outcome"</span>: outcome, <span class="hljs-string">"wall_s"</span>: <span class="hljs-built_in">round</span>(wall, <span class="hljs-number">1</span>),
        <span class="hljs-string">"peak_mib"</span>: <span class="hljs-built_in">round</span>(peak / MiB), <span class="hljs-string">"anon_max_mib"</span>: <span class="hljs-built_in">round</span>(mx.get(<span class="hljs-string">"anon"</span>, <span class="hljs-number">0</span>) / MiB),
        <span class="hljs-string">"file_max_mib"</span>: <span class="hljs-built_in">round</span>(mx.get(<span class="hljs-string">"file"</span>, <span class="hljs-number">0</span>) / MiB), <span class="hljs-string">"kernel_max_mib"</span>: <span class="hljs-built_in">round</span>(mx.get(<span class="hljs-string">"kernel"</span>, <span class="hljs-number">0</span>) / MiB),
        <span class="hljs-string">"samples"</span>: smp.get(<span class="hljs-string">"samples"</span>),
        <span class="hljs-string">"oom_kill"</span>: ev.get(<span class="hljs-string">"oom_kill"</span>, <span class="hljs-number">0</span>), <span class="hljs-string">"max_events"</span>: ev.get(<span class="hljs-string">"max"</span>, <span class="hljs-number">0</span>), <span class="hljs-string">"high_events"</span>: ev.get(<span class="hljs-string">"high"</span>, <span class="hljs-number">0</span>),
        <span class="hljs-string">"after_mib"</span>: <span class="hljs-built_in">round</span>(<span class="hljs-built_in">int</span>(r.get(<span class="hljs-string">"current_after"</span>, <span class="hljs-number">0</span>)) / MiB), <span class="hljs-string">"anon_after_mib"</span>: <span class="hljs-built_in">round</span>(<span class="hljs-built_in">int</span>(r.get(<span class="hljs-string">"anon_after"</span>, <span class="hljs-number">0</span>)) / MiB),
        <span class="hljs-string">"cli_peak_mib"</span>: <span class="hljs-built_in">round</span>(cli_peak / MiB) <span class="hljs-keyword">if</span> cli_peak <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"later_mib"</span>: <span class="hljs-built_in">round</span>(<span class="hljs-built_in">int</span>(r[<span class="hljs-string">"current_later"</span>]) / MiB) <span class="hljs-keyword">if</span> <span class="hljs-string">"current_later"</span> <span class="hljs-keyword">in</span> r <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"anon_later_mib"</span>: <span class="hljs-built_in">round</span>(<span class="hljs-built_in">int</span>(r[<span class="hljs-string">"anon_later"</span>]) / MiB) <span class="hljs-keyword">if</span> <span class="hljs-string">"anon_later"</span> <span class="hljs-keyword">in</span> r <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"leftover_later"</span>: <span class="hljs-built_in">open</span>(os.path.join(d, <span class="hljs-string">"leftover_later.txt"</span>)).read().strip() <span class="hljs-keyword">if</span> os.path.exists(os.path.join(d, <span class="hljs-string">"leftover_later.txt"</span>)) <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
        <span class="hljs-string">"leftover"</span>: <span class="hljs-built_in">open</span>(os.path.join(d, <span class="hljs-string">"leftover.txt"</span>)).read().strip() <span class="hljs-keyword">if</span> os.path.exists(os.path.join(d, <span class="hljs-string">"leftover.txt"</span>)) <span class="hljs-keyword">else</span> <span class="hljs-string">""</span>,
    }

<span class="hljs-keyword">def</span> <span class="hljs-title function_">med</span>(<span class="hljs-params">xs</span>):
    xs = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> xs <span class="hljs-keyword">if</span> x <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>]
    <span class="hljs-keyword">return</span> <span class="hljs-built_in">round</span>(statistics.median(xs), <span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> xs <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>

out = {}
<span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> BUILDS:
    <span class="hljs-keyword">for</span> lim <span class="hljs-keyword">in</span> LIMITS:
        runs = {}
        <span class="hljs-keyword">for</span> rn <span class="hljs-keyword">in</span> (<span class="hljs-string">"r0"</span>, <span class="hljs-string">"r1"</span>, <span class="hljs-string">"r2"</span>, <span class="hljs-string">"r3"</span>):
            d = os.path.join(root, b, lim, rn)
            <span class="hljs-keyword">if</span> os.path.isdir(d):
                info = run_info(d, b)
                <span class="hljs-keyword">if</span> info:
                    runs[rn] = info
        meas = [runs[k] <span class="hljs-keyword">for</span> k <span class="hljs-keyword">in</span> (<span class="hljs-string">"r1"</span>, <span class="hljs-string">"r2"</span>, <span class="hljs-string">"r3"</span>) <span class="hljs-keyword">if</span> k <span class="hljs-keyword">in</span> runs]
        ok = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas <span class="hljs-keyword">if</span> x[<span class="hljs-string">"outcome"</span>] == <span class="hljs-string">"ok"</span>]
        out[<span class="hljs-string">f"<span class="hljs-subst">{b}</span>/<span class="hljs-subst">{lim}</span>"</span>] = {
            <span class="hljs-string">"runs"</span>: runs, <span class="hljs-string">"n"</span>: <span class="hljs-built_in">len</span>(meas), <span class="hljs-string">"n_ok"</span>: <span class="hljs-built_in">len</span>(ok),
            <span class="hljs-string">"outcomes"</span>: [x[<span class="hljs-string">"outcome"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas],
            <span class="hljs-string">"wall_med_ok"</span>: med([x[<span class="hljs-string">"wall_s"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> ok]),
            <span class="hljs-string">"wall_med_all"</span>: med([x[<span class="hljs-string">"wall_s"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"peak_med"</span>: med([x[<span class="hljs-string">"peak_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"anon_med"</span>: med([x[<span class="hljs-string">"anon_max_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"file_med"</span>: med([x[<span class="hljs-string">"file_max_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"peak_range"</span>: [<span class="hljs-built_in">min</span>(x[<span class="hljs-string">"peak_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas), <span class="hljs-built_in">max</span>(x[<span class="hljs-string">"peak_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas)] <span class="hljs-keyword">if</span> meas <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
            <span class="hljs-string">"anon_range"</span>: [<span class="hljs-built_in">min</span>(x[<span class="hljs-string">"anon_max_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas), <span class="hljs-built_in">max</span>(x[<span class="hljs-string">"anon_max_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas)] <span class="hljs-keyword">if</span> meas <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
            <span class="hljs-string">"wall_range"</span>: [<span class="hljs-built_in">min</span>(x[<span class="hljs-string">"wall_s"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas), <span class="hljs-built_in">max</span>(x[<span class="hljs-string">"wall_s"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas)] <span class="hljs-keyword">if</span> meas <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span>,
            <span class="hljs-string">"after_med"</span>: med([x[<span class="hljs-string">"after_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"anon_after_med"</span>: med([x[<span class="hljs-string">"anon_after_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"cli_peak_med"</span>: med([x[<span class="hljs-string">"cli_peak_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"later_med"</span>: med([x[<span class="hljs-string">"later_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"anon_later_med"</span>: med([x[<span class="hljs-string">"anon_later_mib"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
            <span class="hljs-string">"max_events_med"</span>: med([x[<span class="hljs-string">"max_events"</span>] <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> meas]),
        }
json.dump(out, <span class="hljs-built_in">open</span>(os.path.join(root, <span class="hljs-string">"summary.json"</span>), <span class="hljs-string">"w"</span>), indent=<span class="hljs-number">1</span>)
<span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{<span class="hljs-string">'cell'</span>:<span class="hljs-number">26</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'n'</span>:&gt;<span class="hljs-number">2</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'ok'</span>:&gt;<span class="hljs-number">2</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'wall_ok'</span>:&gt;<span class="hljs-number">8</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'peak'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'anon'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'file'</span>:&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-string">'after'</span>:&gt;<span class="hljs-number">6</span>}</span> outcomes / ranges"</span>)
<span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> out.items():
    <span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{k:<span class="hljs-number">26</span>}</span> <span class="hljs-subst">{v[<span class="hljs-string">'n'</span>]:&gt;<span class="hljs-number">2</span>}</span> <span class="hljs-subst">{v[<span class="hljs-string">'n_ok'</span>]:&gt;<span class="hljs-number">2</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(v[<span class="hljs-string">'wall_med_ok'</span>]):&gt;<span class="hljs-number">8</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(v[<span class="hljs-string">'peak_med'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(v[<span class="hljs-string">'anon_med'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(v[<span class="hljs-string">'file_med'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{<span class="hljs-built_in">str</span>(v[<span class="hljs-string">'after_med'</span>]):&gt;<span class="hljs-number">6</span>}</span> <span class="hljs-subst">{v[<span class="hljs-string">'outcomes'</span>]}</span> wall<span class="hljs-subst">{v[<span class="hljs-string">'wall_range'</span>]}</span> anon<span class="hljs-subst">{v[<span class="hljs-string">'anon_range'</span>]}</span> peak<span class="hljs-subst">{v[<span class="hljs-string">'peak_range'</span>]}</span> maxev=<span class="hljs-subst">{v[<span class="hljs-string">'max_events_med'</span>]}</span> cli=<span class="hljs-subst">{v[<span class="hljs-string">'cli_peak_med'</span>]}</span> later=<span class="hljs-subst">{v[<span class="hljs-string">'later_med'</span>]}</span>/<span class="hljs-subst">{v[<span class="hljs-string">'anon_later_med'</span>]}</span>"</span>)
</code></pre></details><h2>Summary</h2><p>Most CI builds need less memory than the cgroup counter suggests, because <code>memory.peak</code> includes page cache that the kernel gives back under pressure. Measure anonymous memory, run the build inside the limit you want to buy, and check <code>memory.events</code> for <code>max</code> and <code>oom_kill</code>. Then look for the traps: a bundler that runs a type checker, a runtime that sizes its heap from the limit, a Gradle daemon that stays after the build, and a Docker build that runs outside your cgroup. In our runs, fixing any of those was worth more than doubling the runner's memory.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DevOps Weekly Digest - Week 45, 2026]]></title>
      <link>https://devops-daily.com/news/2026-week-45</link>
      <description><![CDATA[⚡ Curated updates from Kubernetes, cloud native tooling, CI/CD, IaC, observability, and security - handpicked for DevOps professionals!]]></description>
      <pubDate>Mon, 02 Nov 2026 00:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/news/2026-week-45</guid>
      <category><![CDATA[DevOps News]]></category>
      <content:encoded><![CDATA[<blockquote>
<p>📌 <strong>Handpicked by DevOps Daily</strong> - Your weekly dose of curated DevOps news and updates!</p>
</blockquote>
<hr />
<h2>⚓ Kubernetes</h2><h3>📄 From Working Group to SIG Architecture: spotlight on AI Conformance</h3><p>In this SIG Architecture spotlight we talked with Janet Kuo (Google) and Yuan Tang (TensorScale AI) about the journey of AI Conformance in Kubernetes, from the initial Working Group to the integration</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 Kubernetes Blog</strong></p>
<p><a href="https://kubernetes.io/blog/2026/10/29/sig-arch-ai-conformance-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Announcing the Workload Conformance Working Group</h3><p>We are excited to announce that the Kubernetes community now has a Workload Conformance Working Group. The working group is writing a shared standard for how workloads behave at runtime on Kubernetes.</p>
<p><strong>📅 Oct 27, 2026</strong> • <strong>📰 Kubernetes Blog</strong></p>
<p><a href="https://kubernetes.io/blog/2026/10/27/announcing-workload-conformance-wg/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>☁️ Cloud Native</h2><h3>📄 Keycloak's observability journey: a guide for project maintainers</h3><p>As part of an ongoing series, the Developer Experience SIG interviews organizations about their real-world OpenTelemetry deployments to share practical lessons with the broader community. This post fe</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/devex-keycloak/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Ecosystem is the Answer: Docker, the Blueprint Alliance, and the Importance of Collaboration in the Agentic Era</h3><p>Docker is a founding member of the Blueprint Alliance, a 12-company reference architecture for securing AI agents, built around four governance questions.</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-blueprint-alliance-agent-baseline/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Where should your code live in a Docker Sandbox?</h3><p>Docker Sandboxes support three ways to organize an agent’s code: shared mounts, Git worktrees, and clone mode. Here’s how to choose between them.</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-sandbox-code-checkout-workflows/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Linkerd Story: The people Behind The Service Mesh — A new documentary premieres at KubeCon + CloudNativeCon North America 2026</h3><p>Join CNCF and Buoyant on November 9 in Salt Lake City for the premiere of a new documentary celebrating a decade of Linkerd, the evolution of service mesh technology, and the people who helped make it</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/11/02/the-linkerd-story-the-people-behind-the-service-mesh-a-new-documentary-premieres-at-kubecon-cloudnativecon-north-america-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 KubeCon + CloudNativeCon NA 2026: End User TAB recommendations and tips</h3><p>KubeCon + CloudNativeCon NA 2026 in Salt Lake City will bring the cloud native community together with projects, vendors and end users in what is one of the largest open source events. In this post, w</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/29/kubecon-cloudnativecon-na-2026-end-user-tab-recommendations-and-tips/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Coding Agent Horror Stories: The CVE That Did Not Exist</h3><p>An AI code review agent invented a CVE and recommended dangerous fixes. Learn how Docker Sandboxes contain agents and why their output needs verification.</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/coding-agent-horror-stories-the-cve-that-did-not-exist/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Volcano v1.16 Released: NamespaceQueue, fairshare, and a broader HyperNode ecosystem</h3><p>Volcano v1.16.0 is available . v1.16.0 introduces NamespaceQueue and the fairshare plugin. Tenants can use hierarchical queues within their own namespaces to organize quotas and queue hierarchies with</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/28/volcano-v1-16-released-namespacequeue-fairshare-and-a-broader-hypernode-ecosystem/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Win the Agentic Era, Together: Introducing the Docker Navigator Partner Program</h3><p>Docker is launching the Navigator Partner Program, with three tiers, co-selling benefits, and a new incentive for every registered AI governance deal.</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-navigator-partner-program/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Hardening Tetragon’s gRPC API with Tetragon</h3><p>Tetragon versions before v1.7 expose an unauthenticated gRPC admin API on , and any pod running with on the same node can reach it. This…</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 Cilium Blog</strong></p>
<p><a href="https://cilium.io/blog/2026/10/28/tetragon-grpc" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔄 CI/CD</h2><h3>📄 10 questions to ask any software vendor about sovereignty</h3><p>Sovereignty is moving from buzzword to buying criterion. The European Commission’s proposed Cloud and AI Development Act (CADA) would set assurance levels for the cloud and AI services public authorit</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/10-questions-to-ask-any-software-vendor-about-sovereignty/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Unified API Inventory: Accelerate Patching with Cross-Source</h3><p>Cross-source API correlation connects runtime findings to their code, documentation, and fixes for improved defense against machine-speed attacks. | Blog</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/unified-api-inventory-reduces-the-time-to-patch-with-cross-source-correlation" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Game Off 2026: The theme is SHIFT</h3><p>GitHub’s annual month-long game jam is back for its 14th year! This November, make a game inspired by the theme SHIFT. Change gears, swap perspectives, bend reality, or just press a certain key. The p</p>
<p><strong>📅 Nov 1, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/open-source/gaming/game-off-2026-the-theme-is-shift/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What is release automation? From deploy pipelines to automated releases</h3><p>Release automation moves code from build to production without manual steps. Learn how it works, where AI fits, and how to automate the release step.</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/what-is-release-automation/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitLab Orbit is GA: Higher agent accuracy, lower token cost</h3><p>An agent’s true cost isn’t only the tokens it burns. It’s the review that calls your code “safe to merge” after seeing the diff, but without any context on the pipeline. We call that artificial confid</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/gitlab-orbit-ga/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitLab Orbit: Find vulnerability blast radius without the manual hunt</h3><p>A critical vulnerability needs a fix and a map. The fix is usually fast. Finding where the vulnerable component lives, who owns it, and whether it's exposed to the Internet is what costs you time. Mos</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/find-vulnerability-blast-radius-with-gitlab-orbit/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitHub Universe 2026: Evolving every layer of the stack</h3><p>Today, we laid out our vision for GitHub as the most comprehensive, end-to-end software delivery system in the world. The post GitHub Universe 2026: Evolving every layer of the stack appeared first on</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/news-insights/company-news/github-universe-2026-evolving-every-layer-of-the-stack/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Octoverse: Commits, pushes, reach new highs in 2026 as developers build with agents</h3><p>See how a record year of growth reached every industry, every field, and every kind of builder in the 2026 Octoverse report. The post Octoverse: Commits, pushes, reach new highs in 2026 as developers </p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Measuring developer experience on an Amazon EKS developer platform</h3><p>Most platform teams can't prove their internal developer platform is improving software delivery. This post shows how to automatically collect the four DORA metrics (deployment frequency, lead time fo</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 AWS Containers Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/containers/measuring-developer-experience-on-an-amazon-eks-developer-platform/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🏗️ IaC</h2><h3>📄 Amazon Connect Customer automatically saves in-progress emails as drafts</h3><p>Amazon Connect Customer automatically saves in-progress emails as drafts, helping agents resume work without losing progress when connectivity is interrupted or when their browser is accidentally clos</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-connect-email-draft-save/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Friday Five — October 30, 2026 | Red Hat</h3><p>Fierce Network - Telcos can cash in on AI, not just pay for it: Red HatRed Hat's Joe Fernandes discusses how open models can help telcos move from renting AI capacity to owning it and selling inferenc</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/friday-five-october-30-2026-red-hat" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Tracing failures across Terraform state boundaries with AWS DevOps Agent</h3><p>When a change in one Terraform stack silently breaks an application in another, the team seeing the symptoms often doesn't own the stack that caused it. This post shows how AWS DevOps Agent traces the</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 AWS DevOps Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/devops/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📊 Observability</h2><h3>📄 Instrumentation Hub in Grafana Cloud: start easier, stay in control</h3><p>It's no secret that getting started with observability can be much harder than it should be. That's why, at ObservabilityCon 2025, we introduced Instrumentation Hub: a guided, scalable way to roll out</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/instrumentation-hub-start-your-observability-journey-easier-and-still-stay-in-control/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 E-Maj 5.0.1 : Major Security Update.</h3><p>We are very glad to announce the E-Maj 5.0.1 version. E-Maj is a PostgreSQL extension which enables fine-grained write logging and time travel on subsets of the database. This version mainly fixes a v</p>
<p><strong>📅 Oct 31, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/e-maj-501-major-security-update-3425/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What’s new with Google Cloud</h3><p>Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/inside-google-cloud/whats-new-google-cloud/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 OpenTelemetry eBPF Instrumentation reaches v1.0.0</h3><p>We’re excited to announce v1.0.0 of OpenTelemetry eBPF Instrumentation—OBI! OBI brings OpenTelemetry observability to running Linux applications without requiring changes to their source code. It uses</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/obi-v1-release/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Announcing the 2026 Governance Committee Election Results</h3><p>We are delighted to announce the results of the 2026 OpenTelemetry Governance Committee Election! The OpenTelemetry project continues to bring together an exceptionally vibrant community, who are shap</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/gc-elections-results/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Only include resource detectors you need by building your own Collector</h3><p>There are many reasons for building your own OpenTelemetry Collector distribution. The existing distributions are designed to be broadly useful, which means they include many more components than what</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/collector-with-less-dependencies/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Grafana 13.3 release: manage variables across dashboards, share findings with Notebooks, a UI refresh, and more</h3><p>Grafana 13.3 is here, bringing more consistency and control to how you work across Grafana. Cross-dashboard variables help you manage shared dashboard controls from one place, while Notebooks let you </p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/grafana-13-3-release-all-the-latest-features/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Turo + New Relic: Turn AI code volume into production reliability</h3><p>See how Turo uses the New Relic MCP server to give AI agents live production context, slash incident triage to 60 seconds, and scale releases safely.</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/ai/turo-new-relic-turn-ai-code-volume-into-production-reliability" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Measuring soft navigation web vitals</h3><p>The Sentry JavaScript SDK v11 reports LCP, CLS, and INP for soft navigations. Here's what that reveals about your SPA, with interactive demos.</p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/measuring-soft-navigation-web-vitals/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 fluent-package v6.1.0 has been released</h3><p>Hi users! We have released fluent-package v6.1.0 on 2026-10-29. Fluent Package is a stable distribution package of Fluentd. (successor of td-agent) This is a new release of the v6.1 series. Note that </p>
<p><strong>📅 Oct 29, 2026</strong> • <strong>📰 Fluentd Blog</strong></p>
<p><a href="https://www.fluentd.org/blog/fluent-package-v6.1.0-has-been-released" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 DigitalOcean Signals: Observability for AI Agents and Inference</h3><p>You ship an agent. It runs in production, sessions start and finish, and the dashboards stay green. But then a user mentions that they spent ten minutes retrying the same failing command with slightly</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/digitalocean-signals-observability-ai-agents" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1</h3><p>When evaluating AI models for observability, generating a plausible query or explanation isn't enough. Teams need to know whether an agent's conclusions are supported by telemetry, whether its changes</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-reaches-v1-1/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔐 Security</h2><h3>📄 New Bug-Fix Releases for MPS Versions 2026.2 RC2, 2026.1.2, 2025.3.3, 2025.2.5, and 2025.1.5</h3><p>We’ve released updates for multiple MPS versions that fix a critical security issue. MPS 2026.1.2 and 2026.2 RC2 additionally bring a substantial improvement to the Projectional Agent Toolkit plugin. </p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/mps/2026/11/mps-2026-1-2-released/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Gitea 28.3.0 is released</h3><p>We are excited to announce the release of <strong>Gitea 28.3.0</strong>, the third maintenance release for the 28 series. It contains a large number of security fixes, mostly around token scopes, and bug fixes and</p>
<p><strong>📅 Nov 1, 2026</strong> • <strong>📰 Gitea Blog</strong></p>
<p><a href="https://blog.gitea.com/release-of-28.3.0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Open-source security in the AI exploit era: Patch without changing a thing</h3><p>Here’s how a vulnerability in a popular open-source library plays out in 2026: A maintainer merges a fix and publishes The post Open-source security in the AI exploit era: Patch without changing a thi</p>
<p><strong>📅 Nov 1, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/patchmageddon-open-source-security22842502/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pgAdmin 4 v9.19 Released</h3><p>The pgAdmin Development Team is pleased to announce pgAdmin 4 version 9.19. This release of pgAdmin 4 includes 136 bug fixes and new features, including fixes for 15 security vulnerabilities (CVE-2026</p>
<p><strong>📅 Oct 31, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pgadmin-4-v919-released-3426/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Daybreak Red from OpenAI now available to eligible customers on Amazon Bedrock in AWS GovCloud (US-East)</h3><p>Security teams need powerful tools to find vulnerabilities, test exploits, and build defenses. Daybreak Red from OpenAI is now available to eligible customers on Amazon Bedrock in AWS GovCloud (US-Eas</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/openai-daybreak-red-govcloud-east/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why embargoes matter for open source security</h3><p>Open source is built on a foundation of transparency, collaboration, and speed. We share code, we share ideas, and together, we build better software. But when it comes to discovering and patching sec</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/why-embargoes-matter-for-open-source-security" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Blog: Falco 0.45.2: security updates and a look back at 0.45.1</h3><p>Hey Falco community, Falco 0.45.2 is available! This patch brings certificate trust updates and newer security dependencies, alongside the reliability fixes delivered in 0.45.1. Since we did not publi</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 Falco Blog</strong></p>
<p><a href="https://falco.org/blog/falco-0-45-2/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 PHP2Shell Won't Be the Last. Here's Where the Next One Will Show Up First.</h3><p>PHP2Shell showed how quickly an engine-level flaw can put internet-facing applications at risk. Learn why outside-in inventory, rapid testing, and repeat checks matter before the next disclosure.</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/php2shell-where-next-one-will-show-up-first/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What happens when vulnerabilities are discovered faster than you can fix them?</h3><p>We're entering a new era of application security. Frontier AI models are uncovering complex flaws that traditional tools overlooked, and attackers are using the same models to exploit them faster. Rem</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/security-at-machine-speed/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Announcing SOC 1, FedRAMP 20x Class A, PCI, and HIPAA</h3><p>Explore Harness' latest compliance milestones: SOC 1, FedRAMP 20x Class A, PCI DSS, and HIPAA assessments across its platform and runtime security. | Blog</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/announcing-soc-1-fedramp-pci-and-hipaa" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Snyk UNFILTERED: Inside Our First-Ever Customer Summit</h3><p>At Snyk’s first customer summit, security leaders shared candid perspectives on AI risk, vulnerability remediation, and governing autonomous agents. Five themes reveal how teams are adapting security </p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/snyk-unfiltered-customer-summit/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Azure DevOps Server RC (26H2) Is Now Available</h3><p>Today we’re releasing the release candidate (RC) of the next Azure DevOps Server release. It brings new features previously available in our hosted version, along with security and quality improvement</p>
<p><strong>📅 Oct 27, 2026</strong> • <strong>📰 Azure DevOps Blog</strong></p>
<p><a href="https://devblogs.microsoft.com/devops/azure-devops-server-rc-26h2-is-now-available/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>💾 Databases</h2><h3>📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers</h3><p>Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/please-read-me-opportunistic-ransomware-devastating-mysql-servers" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Beyond Migration: Using Database Migration Service for disk shrinking</h3><p>When database engineers hear “database migration", they typically think of a one-time onboarding pathway—moving databases from on-premises environments to the cloud, or between different cloud provide</p>
<p><strong>📅 Nov 2, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/databases/use-dms-to-shrink-cloud-sql-for-sql-server-disk/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How to Build a LlamaIndex Vector Store With TiDB</h3><p>A LlamaIndex vector store is where your index keeps node embeddings, usually alongside the text chunks and metadata, so queries can find similar content. The default store lives in memory, so a restar</p>
<p><strong>📅 Oct 30, 2026</strong> • <strong>📰 TiDB Blog</strong></p>
<p><a href="https://www.pingcap.com/blog/llamaindex-vector-store/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why Real-Time AI at Scale is So Hard</h3><pre><code class="hljs language-html" tabindex="0">
**📅 Oct 29, 2026** • **📰 ScyllaDB Blog**

[**🔗 Read more**](https://www.scylladb.com/2026/10/29/why-real-time-ai-at-scale-is-so-hard/)

### 📄 Connecting coding agents to the right documentation with a navigable context layer

Ask a question about Redis and the job seems simple: find the right documentation and explain it. But documentation questions come in many shapes. “What does XACK do?” is a precise command question. “

**📅 Oct 29, 2026** • **📰 Redis Blog**

[**🔗 Read more**](https://redis.io/blog/navigable-context-layer-for-coding-agents/)

### 📄 PostgreSQL@SCaLE24x CfP closing soon on Nov 1, 2027

Hi folks! We're hosting a PostgreSQL track at SCaLE again — come join us in sunny Pasadena! PostgreSQL@SCaLE is a two-day event on Thursday–Friday, April 1–2, 2027, at the Pasadena Convention Center, 

**📅 Oct 28, 2026** • **📰 PostgreSQL News**

[**🔗 Read more**](https://www.postgresql.org/about/news/postgresqlscale24x-cfp-closing-soon-on-nov-1-2027-3363/)

### 📄 Estuary Expands PostgreSQL CDC with Targeted Backfills and Scalable Discovery

Estuary has released several updates to its PostgreSQL change data capture (CDC) connector focused on backfill recovery, discovery at scale, and schema-change detection. Targeted Backfills PostgreSQL 

**📅 Oct 28, 2026** • **📰 PostgreSQL News**

[**🔗 Read more**](https://www.postgresql.org/about/news/estuary-expands-postgresql-cdc-with-targeted-backfills-and-scalable-discovery-3419/)

### 📄 P99 CONF 2026 Recap: A Tail of Two Days

Our Aussie host’s take on P99 CONF around the clock and (now) anytime on demand

**📅 Oct 27, 2026** • **📰 ScyllaDB Blog**

[**🔗 Read more**](https://www.scylladb.com/2026/10/27/p99-conf-2026-recap/)

### 📄 Introducing Semantic Operator: Governed Data Access for AI Agents

Semantic Operator is a new Kubeflow subproject that gives AI agents trusted answers from business data. It works in three steps: Define metrics. Data teams describe certified metrics once in an Apache

**📅 Oct 27, 2026** • **📰 Kubeflow Blog**

[**🔗 Read more**](https://blog.kubeflow.org/introducing-semantic-operator/)

### 📄 Three layers, all config: how DigitalOcean Managed Agents enforce NVIDIA OpenShell-aligned boundaries

A few weeks ago, we handed an AI agent one of the scariest jobs in infrastructure: moving huffl’s production backend from Supabase to DigitalOcean. huffl.ai is a real product. Underneath it sits 31 ta

**📅 Oct 26, 2026** • **📰 DigitalOcean Blog**

[**🔗 Read more**](https://www.digitalocean.com/blog/managed-agents-enforce-nvidia-openshell)

---

## 🌐 Platforms

### 📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

**📅 Nov 2, 2026** • **📰 Linode Blog**

[**🔗 Read more**](https://www.akamai.com/blog/security/the-oracle-of-delphi-steal-your-credentials)

### 📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

**📅 Nov 2, 2026** • **📰 Linode Blog**

[**🔗 Read more**](https://www.akamai.com/blog/security/the-nansh0u-campaign-hackers-arsenal-grows-stronger)

### 📄 The Rise of LLMJacking: How Threat Actors Monetize Compromised Cloud and Model Resources

Written by: Christopher Romano, Codi Starks, Garrett Mahoney, John Wolfram, Josh Murchie, K Singh, Mathew Potaczek, Mike Lombardi, Rik Dolfing Introduction Mandiant responded to multiple incidents in 

**📅 Nov 2, 2026** • **📰 Google Cloud Blog**

[**🔗 Read more**](https://cloud.google.com/blog/topics/threat-intelligence/llmjacking-monetizing-compromised-cloud-models/)

### 📄 Shutting the Door on the XCPE Botnet

Introduction Last week, Google and industry partners dismantled the XCPE backdoor which enrolled more than 1.3 million set-top boxes into a botnet. We assess with high confidence that the backdoor and

**📅 Nov 2, 2026** • **📰 Google Cloud Blog**

[**🔗 Read more**](https://cloud.google.com/blog/topics/threat-intelligence/shutting-the-door-on-the-xcpe-botnet/)

### 📄 How we halved DNS build delay and cut memory use by 85%

Zone setting changes used to rebuild entire DNS zones, even ones with tens of millions of records. Here is how we built a planner that picks the smallest safe build, cutting full builds by 40%, peak m

**📅 Nov 2, 2026** • **📰 Cloudflare Blog**

[**🔗 Read more**](https://blog.cloudflare.com/dns-targeted-zone-builds/)

### 📄 Amazon Bedrock expands API support and introduces Cross-Region Inferencing for OpenAI models in AWS GovCloud (US)

Amazon Bedrock now supports the OpenAI GPT-5.6 models (Terra and Luna) on the bedrock-runtime endpoint, with support for the Responses, Converse, and Chat Completions APIs in AWS GovCloud (US). It als

**📅 Oct 30, 2026** • **📰 CloudFormation Updates**

[**🔗 Read more**](https://aws.amazon.com/about-aws/whats-new/2026/10/bedrock-openai-runtime-govcloud/)

### 📄 Web Search now available on Kiro in AWS GovCloud (US)

Kiro web search gives the agent real-time access to current information on the internet, so its answers draw on up-to-date search results and page content. The agent uses web search automatically when

**📅 Oct 30, 2026** • **📰 CloudFormation Updates**

[**🔗 Read more**](https://aws.amazon.com/about-aws/whats-new/2026/10/kiro-web-search-aws-govcloud-us/)

### 📄 Frontier AI–Accelerated Attacks: Securing the Agentic Web and MCP Surface

**📅 Oct 30, 2026** • **📰 Linode Blog**

[**🔗 Read more**](https://www.akamai.com/blog/ai/2026/oct/frontier-ai-accelerated-attacks-agentic-web-mcp-surface)

### 📄 4 real? Tracking exploitation and patching of eMudhra’s weak keys

Four, just four: that’s the key to break into any unpatched connection on the Internet that is using eMudhra certificates. Here’s how we quickly patched Cloudflare, what evidence we can see of attempt

**📅 Oct 30, 2026** • **📰 Cloudflare Blog**

[**🔗 Read more**](https://blog.cloudflare.com/emudhra-patching/)

### 📄 Cloudflare services for private origins is now generally available

Public traffic to private origins, with WAF, bot management, caching, and Workers, is now available to every customer with Cloudflare One connectivity.

**📅 Oct 30, 2026** • **📰 Cloudflare Blog**

[**🔗 Read more**](https://blog.cloudflare.com/private-origins-public-to-private/)

### 📄 Direct VM networking in AWS: Introducing BGP cloud connector on Red Hat OpenShift Service on AWS

When migrating virtual machines (VMs) to the cloud, virtualization engineers hit a familiar wall: how do you route traffic directly to a Red Hat OpenShift Virtualization guest VM using that VM’s own I

**📅 Oct 30, 2026** • **📰 OpenShift Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/direct-vm-networking-aws-introducing-bgp-cloud-connector-red-hat-openshift-service-aws)

### 📄 The boundary problem in agentic AI is a platform problem

An autonomous agent will do exactly what its tools let it do, and no more. This sounds like a constraint. In practice it is the whole risk, because the set of things an agent’s tools let...

**📅 Oct 29, 2026** • **📰 CNCF Blog**

[**🔗 Read more**](https://www.cncf.io/blog/2026/10/29/the-boundary-problem-in-agentic-ai-is-a-platform-problem/)

---

## 📰 Misc

### 📄 Visual Studio Code 1.145 (Insiders)

Learn what is new in Visual Studio Code 1.145 (Insiders). Read the full article

**📅 Nov 4, 2026** • **📰 VS Code Blog**

[**🔗 Read more**](https://code.visualstudio.com/updates/v1_145)

### 📄 Governing Coding-Agent Sprawl with MLflow AI Gateway

A demonstration of how to use MLflow AI Gateway to manage coding agent sprawl.

**📅 Nov 3, 2026** • **📰 MLflow Blog**

[**🔗 Read more**](https://mlflow.org/blog/govern-coding-agent-sprawl/)

### 📄 OpenAI’s new shared usage limit pits ChatGPT against Codex

OpenAI has revealed plans to combine usage allowances across ChatGPT and Codex, as part of a broader effort to integrate The post OpenAI’s new shared usage limit pits ChatGPT against Codex appeared fi

**📅 Nov 2, 2026** • **📰 The New Stack**

[**🔗 Read more**](https://thenewstack.io/openai-chatgpt-codex-shared-limits/)

### 📄 JetBrains Earns 180 G2 Badges in Fall 2026, Recognized for Enterprise Leadership, Usability, and Developer Experience

Discover how JetBrains products are recognized for enterprise leadership, developer experience, and ease of adoption, and why this matters for engineering teams looking to improve productivity and sca

**📅 Nov 2, 2026** • **📰 JetBrains Blog**

[**🔗 Read more**](https://blog.jetbrains.com/blog/2026/11/02/jetbrains-180-g2-badges/)

### 📄 Livestream: Scaling Agentic Development With Air Governance

Join us on November 18, from 4:00–5:00 pm CET, for a livestream with Viktor Kiselev and Guillaume Garnier from the JetBrains Air product team. Your developers want to use the best AI agents for the jo

**📅 Nov 2, 2026** • **📰 JetBrains Blog**

[**🔗 Read more**](https://blog.jetbrains.com/ai/2026/11/livestream-scaling-agentic-development-with-air-governance/)

### 📄 What’s new in Red Hat OpenShift migration advisor: Clearer reports and smarter planning

Red Hat OpenShift migration advisor helps you assess your VMware inventory of virtual machines (VMs) and plan migrations to Red Hat OpenShift Virtualization. Over the past few months, we’ve focused on

**📅 Nov 2, 2026** • **📰 Red Hat Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/whats-new-red-hat-openshift-migration-advisor-clearer-reports-and-smarter-planning)

### 📄 Sovereignty in the AI era: Choice, control, and autonomy

For the past few years, AI felt like a high-stakes exhibition game. Teams across every industry were on the pitch experimenting with large language models, running rapid proofs-of-concept, and testing

**📅 Nov 2, 2026** • **📰 Red Hat Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/sovereignty-ai-era-choice-control-and-autonomy)

### 📄 A step-by-step guide for how to kill the line-by-line code review

Agents verify; people decide. Most teams running AI code review today have it the other way around. They point a The post A step-by-step guide for how to kill the line-by-line code review appeared fir

**📅 Nov 1, 2026** • **📰 The New Stack**

[**🔗 Read more**](https://thenewstack.io/ai-code-review-invariants/)

### 📄 The context problem: How Docusign built agents that understand the business behind contracts

Ask a general-purpose language model to summarize a contract, and it will probably do a reasonable job. Ask it to The post The context problem: How Docusign built agents that understand the business b

**📅 Oct 31, 2026** • **📰 The New Stack**

[**🔗 Read more**](https://thenewstack.io/docusign-contract-ai-agents/)

### 📄 IntelliJ IDEA 2026.2.4 Is Out!

IntelliJ IDEA 2026.2.4 is now available! You can update to this version from inside the IDE, using the Toolbox App, or using snaps if you are a Ubuntu user. You can also download it from our website. 

**📅 Oct 30, 2026** • **📰 JetBrains Blog**

[**🔗 Read more**](https://blog.jetbrains.com/idea/2026/10/intellij-idea-2026-2-4/)

### 📄 How parental leave helped 2 new parents find their rhythm

Becoming a parent brings so much excitement, and can have its fair share of challenges. At Red Hat, parental leave is designed so work isn't one of those worries. In 2025, we set an ambitious goal to 

**📅 Oct 30, 2026** • **📰 Red Hat Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/how-parental-leave-helped-2-new-parents-find-their-rhythm)

### 📄 Visual Studio Code 1.144

Learn what's new in Visual Studio Code 1.144 Read the full article

**📅 Oct 28, 2026** • **📰 VS Code Blog**

[**🔗 Read more**](https://code.visualstudio.com/updates/v1_144)
</code></pre>]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Terraform 1.17's -minimal-refresh: 202 API Calls Down to Zero, and the Drift Goes Quiet Too]]></title>
      <link>https://devops-daily.com/posts/terraform-1-17-minimal-refresh-measured</link>
      <description><![CDATA[Terraform 1.17 can skip the refresh for every resource you did not change. We measured it on 202 real DigitalOcean resources: API calls went from 202 to 0 and plan time halved, and then we broke three records behind its back to see what it stops noticing.]]></description>
      <pubDate>Sat, 31 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/terraform-1-17-minimal-refresh-measured</guid>
      <category><![CDATA[Terraform]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Terraform]]></category><category><![CDATA[IaC]]></category><category><![CDATA[Drift Detection]]></category><category><![CDATA[DigitalOcean]]></category><category><![CDATA[Performance]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>Terraform 1.17.0 shipped on October 21 with a new plan option, <code>-minimal-refresh</code>. It refreshes only the resources that your configuration is about to change, and skips the API reads for everything else. We pointed it at 202 real resources on DigitalOcean. A normal plan made 202 API requests and took 9.5 seconds. With <code>-minimal-refresh</code> it made none and took 4.3 seconds.</p>
<p>Then we deleted one record, changed the TTL of two others behind Terraform's back, and planned again. A normal plan found all three. With <code>-minimal-refresh</code>, Terraform made no API requests and reported no changes.</p>
<p>That trade is the whole story of this flag. It is a real speed-up, and it is also a switch that turns off drift detection for most of your state. This post shows both, with the numbers, so you can decide where it belongs in your pipeline.</p>
<h2>TL;DR</h2><ul>
<li>On 200 DNS records plus a zone and a CNAME (202 resources), a full refresh made 202 GET requests. <code>-minimal-refresh</code> made 0 when nothing changed in config and 1 when one record changed. Median plan time fell from 9.5 s to 4.3 s, about the same as <code>-refresh=false</code> (4.5 s).</li>
<li>A second full plan within a minute of the first took 60 s instead of 9.5 s, because two full plans went over DigitalOcean's limit of 250 API requests per minute. The provider's request log showed only <code>200</code>s; the 429s appeared only in the HTTP client's retry messages (<code>retrying in 51s</code>).</li>
<li>After we deleted one record and changed the TTL of two others out of band, a full plan found all three and <code>-minimal-refresh</code> found none. With a config change on <code>r[0]</code>, it refreshed only <code>r[0]</code> and still hid the drift on the CNAME that depends on it.</li>
<li>A destroy plan with <code>-minimal-refresh</code> skips the refresh of managed resources, so it plans to delete resources that are already gone. The DigitalOcean provider treats the 404 on delete as already deleted, so our destroy still succeeded; a provider that returns an error there would fail.</li>
<li>Use it for fast merge request plans on large states, refresh fully before you apply, and keep a scheduled <code>terraform plan -refresh-only</code> (or a normal plan) so drift still has somewhere to show up.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Terraform 1.17.0 or later (<code>terraform version</code>)</li>
<li>A DigitalOcean API token with domain read, create and delete scopes, if you want to run the lab</li>
<li>A zone name under a domain you control; the lab never delegates it, so no real DNS changes</li>
</ul>
<h2>What the flag does</h2><p>A refresh reads each resource in the state from the provider's API and records what it finds. <a href="https://devops-daily.com/posts/who-owns-the-terraform-state-file#h3-what-does-terraform-refresh-do">What does terraform refresh do?</a> covers the basics. <code>-minimal-refresh</code> changes which resources get that read.</p>
<p>The <a href="https://developer.hashicorp.com/terraform/cli/commands/plan" rel="noopener noreferrer">plan command docs</a> describe it in one line: it "reduces the number of remote API requests by refreshing only resource instances that Terraform plans to change". The implementation is in <a href="https://github.com/hashicorp/terraform/pull/38953" rel="noopener noreferrer">PR #38953</a>, which spells out the details:</p>
<ul>
<li>For each resource instance, Terraform first plans against the state it already has, without reading the remote object.</li>
<li>If that plan has no change, the instance is not refreshed at all.</li>
<li>If it has a change, Terraform throws that plan away, refreshes the instance, and plans it again.</li>
<li>Each instance decides on its own. A refresh of one resource does not automatically refresh the resources that depend on it.</li>
<li>Destroy plans, and deposed or orphaned instances, skip the refresh with this flag.</li>
</ul>
<ol>
<li><strong>resource in state</strong></li>
<li><strong>plan against state</strong> no API read</li>
</ol>
<p>Outcomes:</p>
<ul>
<li><strong>no change: skip the refresh</strong></li>
<li><strong>change: refresh, plan again</strong></li>
</ul>
<p>The docs also say what that costs: the option "causes Terraform to ignore external changes to resources that have no planned changes". The rest of this post measures both sides of that sentence.</p>
<h2>The lab</h2><p>The <a href="https://github.com/The-DevOps-Daily/terraform-minimal-refresh-check" rel="noopener noreferrer">companion repository</a> creates 202 resources on DigitalOcean: one DNS zone, 200 TXT records and one CNAME. DigitalOcean DNS costs nothing and needs no droplets, and a record refresh is one GET request, so counting requests is simple.</p>
<pre><code class="hljs language-hcl" tabindex="0"><span class="hljs-keyword">resource</span> <span class="hljs-string">"digitalocean_record"</span> <span class="hljs-string">"r"</span> {
  count  = var.records
  domain = digitalocean_domain.lab.id
  type   = <span class="hljs-string">"TXT"</span>
  name   = format(<span class="hljs-string">"r%03d"</span>, count.index)
  value  = count.index == var.changed ? <span class="hljs-string">"v2"</span> : <span class="hljs-string">"v1"</span> <span class="hljs-comment"># -var changed=7 changes one record in config</span>
  ttl    = <span class="hljs-number">1800</span>
}

<span class="hljs-comment"># Depends on r[0], so a config change to r[0] touches it in the graph.</span>
<span class="hljs-keyword">resource</span> <span class="hljs-string">"digitalocean_record"</span> <span class="hljs-string">"pointer"</span> {
  domain = digitalocean_domain.lab.id
  type   = <span class="hljs-string">"CNAME"</span>
  name   = <span class="hljs-string">"pointer"</span>
  value  = <span class="hljs-string">"<span class="hljs-variable">${digitalocean_record.r[0].fqdn}</span>."</span>
  ttl    = <span class="hljs-number">1800</span>
}
</code></pre><p>Each plan in the clean runs uses <code>TF_LOG=DEBUG</code>, because the DigitalOcean provider prints its API requests only at that level, and <code>plan.sh</code> counts the logged request lines and reads the planned changes from <code>terraform show -json</code>. The plans ran on a Raspberry Pi 3 B+ with 1 GB of RAM, so the absolute times are slower than on a laptop. The request counts do not depend on the machine. They are the requests the provider's request log records; a retry below that logger (Finding 2) is not counted again. We wrote down what we expected before each measured run and committed it to the repository first. <code>SOURCES.md</code> in the repository keeps the exact excerpts from the docs, the PR and the provider source that this post quotes.</p>
<p><a href="https://github.com/The-DevOps-Daily/terraform-minimal-refresh-check" rel="noopener noreferrer">The-DevOps-Daily/terraform-minimal-refresh-check on GitHub</a></p>
<h2>Finding 1: 202 requests down to 0, and half the plan time</h2><p>With nothing changed in the configuration, a full plan read every resource. <code>-minimal-refresh</code> read none. With one record changed in config, it read exactly that one record:</p>
<p><strong>bench2.sh, round 1, then three full plans back to back</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ <span class="hljs-built_in">cut</span> -f1-5 results/bench2-20261029T114834Z.tsv | grep -E <span class="hljs-string">'^(label|quiet/[a-z]+/[a-z-]+/1|burst)'</span>
label	seconds	gets	requests	429s
quiet/full/no-change/1	9.5	202	202	0
quiet/minimal/no-change/1	4.3	0	0	0
quiet/minimal/one-change/1	4.9	1	1	0
quiet/norefresh/no-change/1	4.5	0	0	0
burst/full/no-change/1	9.4	202	202	0
burst/full/no-change/2	60.4	202	202	0
burst/full/no-change/3	60.6	202	202	0
</code></pre><p>Across three rounds, the median times were:</p>
<p><strong>Median plan time, 202 DigitalOcean resources</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>full refresh (202 GETs)</td>
<td>9.5s</td>
<td>refresh</td>
</tr>
<tr>
<td>-minimal-refresh, no change (0 GETs)</td>
<td>4.3s</td>
<td>minimal</td>
</tr>
<tr>
<td>-minimal-refresh, 1 change (1 GET)</td>
<td>4.5s</td>
<td>minimal</td>
</tr>
<tr>
<td>-refresh=false (0 GETs)</td>
<td>4.5s</td>
<td>off</td>
</tr>
</tbody></table>
<p><em>Three runs each, every full-refresh plan after a 65 s pause; no 429 was logged. Raspberry Pi 3 B+, Terraform 1.17.0, provider digitalocean 2.106.0. Source: results/bench2-20261029T114834Z.tsv.</em></p>
<p>Two things stand out. First, <code>-minimal-refresh</code> is as fast as <code>-refresh=false</code> here, because with nothing changed it does the same work. Second, a plan without any refresh still took about 4.3 s on this machine. Refreshing all 202 resources added about 5 s on top, roughly 25 ms of wall-clock time per resource with Terraform's default parallelism of 10. We did not split that into network time and Terraform or provider work, so expect both numbers to be different on your hardware.</p>
<h2>Finding 2: a second full plan within a minute took six times longer</h2><p>Our first benchmark looked wrong. Identical full plans took anything from 9 to 62 seconds. The cause was our own script: it ran every plan twice in a row, once to time it and once with logging, so each measurement meant about 404 requests across two consecutive full plans. DigitalOcean <a href="https://github.com/digitalocean/openapi/blob/79e5c87f46291271df945e1094817144cac2f827/specification/description.yml" rel="noopener noreferrer">allows 5,000 requests per hour and 250 per minute</a> per token. We marked that run as confounded in the repository and measured the effect on purpose instead: three full plans back to back.</p>
<p><strong>Three full plans back to back</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>1st plan</td>
<td>9.4s</td>
</tr>
<tr>
<td>2nd plan</td>
<td>60.4s</td>
</tr>
<tr>
<td>3rd plan</td>
<td>60.6s</td>
</tr>
</tbody></table>
<p><em>Each plan sends 202 GET requests; DigitalOcean allows 250 per minute. Source: results/bench2-20261029T114834Z.tsv.</em></p>
<p>The odd part was the log. Every slow plan still showed exactly 202 GET requests and no <code>429</code> status line. A rerun of two back-to-back full plans that kept the whole log (<code>burst-log.sh</code>) found the 429s somewhere else:</p>
<p><strong>burst-log.sh, second of two full plans back to back</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ sed -n <span class="hljs-string">'/^burst\/full\/2/,/^# ended/p'</span> results/burst-log-20261029T132653Z.txt | grep -E <span class="hljs-string">'^burst|status lines|retry:'</span>
burst/full/2	59.7	202	202	0	none	none
  all status lines: {<span class="hljs-string">'200'</span>: 202}; retry messages: 10; longest silent gap <span class="hljs-keyword">in</span> the <span class="hljs-built_in">log</span>: 50.0 s
  retry: digitalocean.com/v2/domains/tf-refresh-lab.devops-daily.com/records/1836867618 (status: 429): retrying <span class="hljs-keyword">in</span> 51s (4 left)
  retry: digitalocean.com/v2/domains/tf-refresh-lab.devops-daily.com/records/1836867807 (status: 429): retrying <span class="hljs-keyword">in</span> 51s (4 left)
  retry: digitalocean.com/v2/domains/tf-refresh-lab.devops-daily.com/records/1836867744 (status: 429): retrying <span class="hljs-keyword">in</span> 51s (4 left)
</code></pre><p>In that rerun, every HTTP status line in the request log was a <code>200</code>, but the HTTP client printed ten retry messages. The three the script kept each show a <code>429 Too Many Requests</code> and a retry scheduled 51 seconds later. The log was silent for 50 seconds, and the second plan took 59.7 s.</p>
<p>The 429s never show up as status lines because of how the provider builds its client (<a href="https://github.com/digitalocean/terraform-provider-digitalocean/blob/v2.106.0/digitalocean/config/config.go" rel="noopener noreferrer">config.go</a>). It creates an HTTP client with retries (4 by default, 1 to 30 s apart; a 51 s wait is longer than that maximum, which suggests it came from DigitalOcean's <code>retry-after</code> header) and then wraps its request logging around that client. The request log sees one request going in and the final <code>200</code> coming out. The <code>429</code> exists only in the retry library's own message. If a plan suddenly takes a minute longer, search the <code>TF_LOG</code> output for <code>retrying in</code> or <code>status: 429</code>; the HTTP status lines alone hide the retries.</p>
<p>This is where <code>-minimal-refresh</code> earns its keep. A CI system that runs a plan for every push to every open pull request can easily send several full refreshes a minute against the same token. With the flag, an unchanged resource costs no requests at all.</p>
<h2>Finding 3: the drift it stops seeing</h2><p><code>drift.sh</code> changes three records through the DigitalOcean API, behind Terraform's back:</p>
<ul>
<li><code>r[42]</code> is deleted</li>
<li><code>r[43]</code> gets a TTL of 60 instead of 1800</li>
<li><code>pointer</code>, the CNAME that depends on <code>r[0]</code>, gets a TTL of 60</li>
</ul>
<p>Then it plans in several ways. Each full-refresh plan waits a minute first, and no <code>429</code> was logged in any of them.</p>
<table>
<thead>
<tr>
<th>Plan</th>
<th>API reads</th>
<th>Changes it shows</th>
</tr>
</thead>
<tbody><tr>
<td>Full refresh</td>
<td>202</td>
<td><code>pointer</code> update, <code>r[42]</code> create, <code>r[43]</code> update</td>
</tr>
<tr>
<td><code>-minimal-refresh</code></td>
<td>0</td>
<td>none</td>
</tr>
<tr>
<td>Full refresh, <code>r[0]</code> changed in config</td>
<td>202</td>
<td><code>r[0]</code> update plus the three above</td>
</tr>
<tr>
<td><code>-minimal-refresh</code>, <code>r[0]</code> changed in config</td>
<td>1</td>
<td><code>r[0]</code> update only</td>
</tr>
<tr>
<td><code>-refresh-only</code></td>
<td>202</td>
<td>no infrastructure actions; drift reported on <code>pointer</code>, <code>r[42]</code> and <code>r[43]</code> (applying it would update the state)</td>
</tr>
</tbody></table>
<p>Source: <code>results/drift-20261029T124036Z.tsv</code>. Full-refresh plans had a 65-second pause before them; the <code>-minimal-refresh</code> plans ran right after.</p>
<p>The second row is the one to remember. One record is gone and two have the wrong TTL, and the plan says there is nothing to do. It will keep saying that on every pull request until someone changes those resources in code or runs a plan that refreshes them.</p>
<p>The fourth row shows the rule from the PR in practice: each resource decides on its own whether to refresh. <code>pointer</code> depends on <code>r[0]</code>, and <code>r[0]</code> was refreshed because its config changed, but <code>pointer</code> had no planned change of its own, so its TTL drift stayed hidden. If you expected a change to one resource to pull its dependents into the refresh, it does not do that automatically: a dependent is refreshed only when it has a planned change of its own.</p>
<p><code>-refresh-only</code> caught all three, which is what it is for. The docs say you cannot combine it with <code>-minimal-refresh</code>, so it cannot be made cheap the same way.</p>
<h2>Finding 4: destroy plans skip the refresh completely</h2><p>With <code>-minimal-refresh</code>, a destroy plan skips the refresh of managed resources, as the PR describes. After <code>r[42]</code> was deleted out of band, a normal destroy plan refreshed, noticed it was gone, and planned 201 deletions. The <code>-minimal-refresh</code> destroy plan made no API requests and planned 202, including the record that no longer existed.</p>
<p>We then destroyed the lab with <code>terraform apply -destroy -minimal-refresh</code>. It exited 0 and reported 202 resources destroyed, one of which no longer existed (<code>results/teardown-20261029T124712Z.txt</code>). That worked only because the DigitalOcean provider's record delete <a href="https://github.com/digitalocean/terraform-provider-digitalocean/blob/v2.106.0/digitalocean/domain/resource_record.go" rel="noopener noreferrer">treats a 404 as already gone</a>. A provider that returns an error on a missing object would fail, possibly after some deletions have already completed. For destroy, let Terraform refresh.</p>
<h2>What went wrong while we measured</h2><p>Three mistakes, all caught before writing, all kept in the repository:</p>
<ul>
<li>The first benchmark ran every plan twice back to back, so each full plan meant about 404 requests in quick succession, more than the 250 a minute DigitalOcean allows. Its request counts are right; its timings mix in rate-limit waits. The file says so on its second line, and <code>bench2.sh</code> has the clean timings.</li>
<li>Our first drift run asked DigitalOcean to change the CNAME's TTL with a PATCH that sent only the TTL. The API answered <code>422: Data needs to end with a dot</code>, even though it stores the value without one, so the dependent-resource case did not run. The fixed script sends the value with the trailing dot.</li>
<li><code>plan.sh</code> first read only <code>resource_changes</code> from the plan JSON. A refresh-only plan reports what it found under <code>resource_drift</code>, so it looked like it found nothing. The script now records both.</li>
</ul>
<p>A fourth check failed in a harmless way: <code>bench2.sh</code> tried to measure the cost of debug logging by unsetting <code>TF_LOG</code>, but the log was still written, so both rows were logged runs. So the cost of debug logging is unmeasured. All the clean timings in this post come from logged runs, so they compare fairly with each other.</p>
<h2>When to use it</h2><p><code>-minimal-refresh</code> fits the fast plan that runs on every merge request, if your state is large or your provider's rate limit is tight. The resources you changed are still refreshed, so their part of the plan starts from the real remote state. Treat it as early feedback, though: resources without a planned change of their own are planned from state, and a refresh of one resource does not automatically refresh the ones that depend on it, as Finding 3 shows. Run a fully refreshed plan before you apply.</p>
<p>It is the wrong tool anywhere you rely on plans to surface drift. Keep at least one plan that refreshes everything:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># Assumes terraform init, the backend and provider credentials are set up</span>
<span class="hljs-comment"># in before_script or the image, and a pipeline schedule exists for drift-check.</span>

<span class="hljs-comment"># Fast plan for every merge request: only changed resources are read.</span>
<span class="hljs-attr">mr-plan:</span>
  <span class="hljs-attr">rules:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">if:</span> <span class="hljs-string">'$CI_PIPELINE_SOURCE == "merge_request_event"'</span>
  <span class="hljs-attr">script:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-string">terraform</span> <span class="hljs-string">plan</span> <span class="hljs-string">-minimal-refresh</span>

<span class="hljs-comment"># Nightly drift check: reads every resource and fails when anything moved.</span>
<span class="hljs-attr">drift-check:</span>
  <span class="hljs-attr">rules:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">if:</span> <span class="hljs-string">'$CI_PIPELINE_SOURCE == "schedule"'</span>
  <span class="hljs-attr">script:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-string">terraform</span> <span class="hljs-string">plan</span> <span class="hljs-string">-refresh-only</span> <span class="hljs-string">-detailed-exitcode</span> <span class="hljs-comment"># exit code 2 means drift</span>
</code></pre><p>The docs add two more rules: do not use it, or <code>-refresh=false</code>, when you upgrade provider versions, and you cannot combine it with <code>-refresh-only</code> or <code>-refresh=false</code>. From our results we would add two more: do not use it for destroy plans you want to be accurate, and do not apply a plan made with it without a full refresh first.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>One provider, two resource types, all DNS.</strong> A zone and its records are cheap to read. Providers with slow reads (databases, Kubernetes objects, large IAM policies) should save more time per skipped refresh; we did not measure them.</li>
<li><strong>One slow machine.</strong> The 4.3 s without a refresh was measured on a Raspberry Pi 3 B+. On other hardware both the baseline and the cost of the refresh will be different.</li>
<li><strong>Other providers' destroy behaviour.</strong> The DigitalOcean provider treats a 404 on delete as already deleted. A provider that returns an error there would fail the destroy instead.</li>
<li><strong>Large states.</strong> We stopped at 202 resources to stay inside DigitalOcean's 5,000 requests an hour, and our repeated runs still took the token down to 4 remaining requests once.</li>
</ul>
<h2>Summary</h2><p><code>-minimal-refresh</code> does what it says. On 202 DigitalOcean resources it cut API requests from 202 to 0 or 1 and plan time from 9.5 s to 4.3 s, about the same as <code>-refresh=false</code>, while still refreshing the resources you change. It also stays clear of the slowdown we saw when a second full plan ran within a minute of the first (60 s), when requests hit DigitalOcean's per-minute limit and were retried 51 s later. The price is drift: with no config change it reported none of the three records we changed behind its back, and a change to one resource did not refresh the resource that depends on it. Use it for fast merge request plans, refresh fully before you apply, keep a scheduled full or refresh-only plan for drift, and do not use it for destroy.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Base64 Explained: 3 Bytes In, 4 Characters Out, Zero Secrecy]]></title>
      <link>https://devops-daily.com/posts/base64-explained</link>
      <description><![CDATA[Base64 turns 3 bytes into 4 characters and hides nothing. See it in Kubernetes Secrets and JWTs, measure the 33% overhead, and fix the bugs it causes.]]></description>
      <pubDate>Fri, 30 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/base64-explained</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[Base64]]></category><category><![CDATA[Kubernetes]]></category><category><![CDATA[Security]]></category><category><![CDATA[JWT]]></category><category><![CDATA[Linux]]></category><category><![CDATA[Bash]]></category>
      <content:encoded><![CDATA[<p>Base64 is in more of your config than you think. Kubernetes Secrets store their values in it, a signed JWT in an <code>Authorization</code> header is three base64url strings joined by dots, kubeconfig files carry certificates in it, and CI systems ask you to paste base64 when a file has to fit in a variable. Most of the time it works. When it does not, the bug is usually small, like an extra newline or a value encoded twice, and it takes an afternoon to find because the broken value looks almost the same as the right one.</p>
<p>This post shows how base64 works, why its output is a third bigger, and what it does not protect. Then it walks through the bugs that keep coming back. Every command and output below comes from a real run.</p>
<h2>TLDR</h2><ul>
<li>Base64 maps every <strong>3 bytes to 4 characters</strong> from a 64-character alphabet. Anyone can reverse it with one command. It is an <strong>encoding, not encryption</strong>.</li>
<li>A Kubernetes Secret's <code>data</code> field is plain base64. <code>kubectl get secret -o yaml</code> plus <code>base64 -d</code> shows the value to anyone with read access.</li>
<li>The output is <strong>33.3% bigger</strong> than the input. GNU <code>base64</code> also adds a newline every 76 characters, which makes it 35.1%. Gzip wins most of it back on the wire.</li>
<li>The common bugs: a <strong>trailing newline</strong> from <code>echo</code>, <strong>wrapped output</strong> pasted into YAML or an env var, <strong>double encoding</strong> with <code>--from-literal</code> or <code>stringData</code>, and the <strong>URL-safe alphabet without padding</strong> in JWTs.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Linux shell with GNU coreutils for <code>base64</code> and <code>basenc</code>. We used coreutils 9.1 on Debian 12.</li>
<li><code>openssl</code> and <code>python3</code> for the JWT and comparison examples.</li>
<li><code>kubectl</code> for the Secret examples. Every <code>kubectl</code> command here uses <code>--dry-run=client</code>, so you do not need a cluster. We used kubectl v1.37.1.</li>
</ul>
<h2>How base64 turns 3 bytes into 4 characters</h2><p>Base64 uses 64 symbols, and 64 is 2 to the power of 6, so each output character carries exactly 6 bits. Three bytes are 24 bits, and 24 bits split cleanly into four 6-bit groups. That is the whole algorithm: take 3 bytes, cut them into four 6-bit numbers, and look each number up in the alphabet.</p>
<p>This script prints each step for the string <code>k8s</code>:</p>
<pre><code class="hljs language-python" tabindex="0"><span class="hljs-keyword">import</span> base64

data = <span class="hljs-string">b"k8s"</span>
bits = <span class="hljs-string">""</span>.join(<span class="hljs-string">f"<span class="hljs-subst">{b:08b}</span>"</span> <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> data)
groups = [bits[i:i + <span class="hljs-number">6</span>] <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-built_in">len</span>(bits), <span class="hljs-number">6</span>)]

<span class="hljs-built_in">print</span>(<span class="hljs-string">"bytes   "</span>, <span class="hljs-string">" "</span>.join(<span class="hljs-string">f"<span class="hljs-subst">{b:08b}</span>"</span> <span class="hljs-keyword">for</span> b <span class="hljs-keyword">in</span> data))
<span class="hljs-built_in">print</span>(<span class="hljs-string">"6 bits  "</span>, <span class="hljs-string">" "</span>.join(groups))
<span class="hljs-built_in">print</span>(<span class="hljs-string">"values  "</span>, <span class="hljs-string">" "</span>.join(<span class="hljs-built_in">str</span>(<span class="hljs-built_in">int</span>(g, <span class="hljs-number">2</span>)) <span class="hljs-keyword">for</span> g <span class="hljs-keyword">in</span> groups))
<span class="hljs-built_in">print</span>(<span class="hljs-string">"base64  "</span>, base64.b64encode(data).decode())
</code></pre><pre><code class="hljs language-text" tabindex="0">bytes    01101011 00111000 01110011
6 bits   011010 110011 100001 110011
values   26 51 33 51
base64   azhz
</code></pre><p>The standard alphabet from <a href="https://www.rfc-editor.org/rfc/rfc4648" rel="noopener noreferrer">RFC 4648</a> is:</p>
<pre><code class="hljs language-text" tabindex="0"> 0-25   A B C ... Z
26-51   a b c ... z
52-61   0 1 2 ... 9
62      +
63      /
</code></pre><p>So 26 is <code>a</code>, 51 is <code>z</code> and 33 is <code>h</code>, and <code>k8s</code> becomes <code>azhz</code>. Decoding runs the same steps backwards. There is no key and nothing to guess, which is why base64 is not a way to hide anything.</p>
<h2>Padding: what the = signs mean</h2><p>When the input length is not a multiple of 3, the last group is short. Base64 fills the missing bits with zeros and adds <code>=</code> so the output length stays a multiple of 4:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">printf</span> <span class="hljs-string">'k'</span> | <span class="hljs-built_in">base64</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'k8'</span> | <span class="hljs-built_in">base64</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'k8s'</span> | <span class="hljs-built_in">base64</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">aw==
azg=
azhz
</code></pre><p>One leftover byte gives 2 characters plus <code>==</code>. Two leftover bytes give 3 characters plus <code>=</code>. Padding carries no data, which is why some formats, JWT among them, drop it. That saves a few bytes and causes a bug we will get to below.</p>
<p>Because the mapping is fixed, common inputs have recognizable prefixes. JSON that starts with <code>{"</code> followed by a letter encodes to a string starting with <code>eyJ</code>, so <code>eyJ...</code> is almost always JSON, often a JWT. Base64-encoded PEM text (<code>-----BEGIN ...</code>) starts with <code>LS0tLS1CRUdJTi</code>.</p>
<h2>The 33% overhead, measured</h2><p>Every 3 input bytes become 4 output characters, so the encoded size is <code>4 × ceil(n / 3)</code>. For large inputs that is 4/3 of the original, an overhead of 33.3%. GNU <code>base64</code> also wraps its output at 76 characters by default, which adds one newline for every 76 characters.</p>
<p>We encoded 1 MiB of random data to check the math:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">head</span> -c 1048576 /dev/urandom &gt; blob.bin
<span class="hljs-built_in">wc</span> -c &lt; blob.bin
<span class="hljs-built_in">base64</span> blob.bin | <span class="hljs-built_in">wc</span> -c        <span class="hljs-comment"># default: wrapped at 76 characters</span>
<span class="hljs-built_in">base64</span> -w0 blob.bin | <span class="hljs-built_in">wc</span> -c    <span class="hljs-comment"># -w0: no wrapping</span>
<span class="hljs-built_in">base64</span> blob.bin | <span class="hljs-built_in">wc</span> -l
</code></pre><pre><code class="hljs language-text" tabindex="0">1048576
1416501
1398104
18397
</code></pre><p>The formula gives the same numbers: <code>4 × ceil(1048576 / 3)</code> is 1,398,104 characters, and <code>ceil(1398104 / 76)</code> is 18,397 lines, each ending in a newline, for 1,416,501 bytes in total. That is 1.3333 times the input without wrapping and 1.3509 times with it.</p>
<p><strong>1 MiB of random data, before and after base64</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>raw</td>
<td>1048576 bytes</td>
<td>raw</td>
</tr>
<tr>
<td>base64 -w0</td>
<td>1398104 bytes</td>
<td>base64</td>
</tr>
<tr>
<td>base64 (wrapped)</td>
<td>1416501 bytes</td>
<td>base64</td>
</tr>
<tr>
<td>base64 -w0 | gzip -9</td>
<td>1058004 bytes</td>
<td>compressed</td>
</tr>
</tbody></table>
<p><em>Measured with wc -c on Debian 12, GNU coreutils 9.1 and gzip -9. The base64 overhead is 33.3% (35.1% with line wrapping); after gzip it is 0.9% on the wire.</em></p>
<p>On the wire, compression gets most of it back. Base64 only uses 6 of every 8 bits, and gzip notices:</p>
<pre><code class="hljs language-bash" tabindex="0">gzip -9 -c blob.bin | <span class="hljs-built_in">wc</span> -c
<span class="hljs-built_in">base64</span> -w0 blob.bin | gzip -9 | <span class="hljs-built_in">wc</span> -c
</code></pre><pre><code class="hljs language-text" tabindex="0">1048763
1058004
</code></pre><p>For this sample, gzip cut the overhead from 33.3% to 0.9%. The overhead still counts wherever the encoded size is what matters: memory, environment variables, cookies and request headers. A JWT is sent in a header on every call, and header size limits apply to its encoded form: each claim costs 4/3 of its size, and nginx's default <code>large_client_header_buffers</code> allows 8 KB per header line.</p>
<h2>Kubernetes Secrets are base64, not encryption</h2><p>A Secret's <code>data</code> field holds base64 strings. Create one locally to see what the API server would store:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl create secret generic db-creds \
  --from-literal=username=app \
  --from-literal=password=<span class="hljs-string">'hunter2-prod'</span> \
  --dry-run=client -o yaml
</code></pre><pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">apiVersion:</span> <span class="hljs-string">v1</span>
<span class="hljs-attr">data:</span>
  <span class="hljs-attr">password:</span> <span class="hljs-string">aHVudGVyMi1wcm9k</span>
  <span class="hljs-attr">username:</span> <span class="hljs-string">YXBw</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">Secret</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">db-creds</span>
</code></pre><p>Getting the value back takes one pipe. On a cluster you would use <code>kubectl get secret db-creds -o jsonpath='{.data.password}'</code>; here we read the same field from the dry run:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl create secret generic db-creds --from-literal=password=<span class="hljs-string">'hunter2-prod'</span> \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.password}'</span> | <span class="hljs-built_in">base64</span> -d; <span class="hljs-built_in">echo</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">hunter2-prod
</code></pre><p>Base64 is there so binary data such as keystores and TLS keys fits inside JSON and YAML. It does nothing for secrecy.</p>
<blockquote>
<p><strong>Warning</strong></p>
<p>Anyone who can <code>get</code> or <code>list</code> Secrets in a namespace can read every value in it, and so can anyone who can create a pod there that mounts them. A Secret manifest committed to Git is the password in plain sight. What protects Secrets is RBAC, <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/" rel="noopener noreferrer">encryption at rest</a> for etcd, and keeping the values out of Git with a tool such as Sealed Secrets, SOPS or an external secret store.</p>
</blockquote>
<p>For a longer walkthrough, see <a href="https://devops-daily.com/guides/introduction-to-kubernetes/06-configmaps-and-secrets#h3-decode-every-key-of-a-secret">how to decode every key of a Secret</a> in the ConfigMaps and Secrets part of our Kubernetes guide.</p>
<h2>Bug 1: the newline from echo</h2><p><code>echo</code> adds a newline, and base64 encodes it like any other byte:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">echo</span> <span class="hljs-string">'hunter2-prod'</span> | <span class="hljs-built_in">base64</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">'hunter2-prod'</span> | <span class="hljs-built_in">base64</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">aHVudGVyMi1wcm9kCg==
aHVudGVyMi1wcm9k
</code></pre><p>The first value ends in <code>Cg==</code>, which is the encoded <code>\n</code>. Decode it and look at the bytes:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">echo</span> <span class="hljs-string">'aHVudGVyMi1wcm9kCg=='</span> | <span class="hljs-built_in">base64</span> -d | <span class="hljs-built_in">od</span> -c
</code></pre><pre><code class="hljs language-text" tabindex="0">0000000   h   u   n   t   e   r   2   -   p   r   o   d  \n
0000015
</code></pre><p>Put that value in a Secret and the password is now <code>hunter2-prod</code> plus a newline. The app sends 13 bytes, the database expects 12, and you get an authentication error for a password that looks right in every log line.</p>
<p>The same thing happens with files. A file written with <code>echo</code> ends in a newline, and <code>--from-file</code> keeps it:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">echo</span> <span class="hljs-string">'hunter2-prod'</span> &gt; password.txt
kubectl create secret generic db-creds --from-file=password=password.txt \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.password}'</span>; <span class="hljs-built_in">echo</span>

<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">'hunter2-prod'</span> &gt; password.txt
kubectl create secret generic db-creds --from-file=password=password.txt \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.password}'</span>; <span class="hljs-built_in">echo</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">aHVudGVyMi1wcm9kCg==
aHVudGVyMi1wcm9k
</code></pre><p>Use <code>printf '%s'</code> whenever the exact bytes matter, and pipe a suspect value through <code>base64 -d | od -c</code> to see the last byte.</p>
<h2>Bug 2: wrapped output in YAML and env vars</h2><p>GNU <code>base64</code> breaks its output into 76-character lines. A database URL is long enough to hit that:</p>
<pre><code class="hljs language-bash" tabindex="0">DB_URL=<span class="hljs-string">'postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require'</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$DB_URL</span>"</span> | <span class="hljs-built_in">wc</span> -c
<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$DB_URL</span>"</span> | <span class="hljs-built_in">base64</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">69
cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9vcmRlcnM/c3Ns
bW9kZT1yZXF1aXJl
</code></pre><p>Now put it into a manifest the way many scripts do, with a <a href="https://devops-daily.com/posts/how-cat-eof-works-in-bash">heredoc</a>:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">cat</span> &lt;&lt;<span class="hljs-string">EOF &gt; secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-url
data:
  url: $(printf '%s' "$DB_URL" | base64)
EOF</span>
kubectl create --dry-run=client -f secret.yaml -o yaml
</code></pre><pre><code class="hljs language-text" tabindex="0">error: error parsing secret.yaml: error converting YAML to JSON: yaml: line 8: could not find expected ':'
</code></pre><p>The second line of the value landed at column 0, and YAML read it as a new key. In an env var or a CI variable the newline does not fail this loudly: it ends up inside the value. Turn wrapping off with <code>-w0</code>:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">cat</span> &lt;&lt;<span class="hljs-string">EOF &gt; secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-url
data:
  url: $(printf '%s' "$DB_URL" | base64 -w0)
EOF</span>
sed -n <span class="hljs-string">'s/^  url: //p'</span> secret.yaml | <span class="hljs-built_in">base64</span> -d
</code></pre><pre><code class="hljs language-text" tabindex="0">postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require
</code></pre><p>Even simpler: let <code>kubectl create secret generic db-url --from-literal=url="$DB_URL" --dry-run=client -o yaml</code> write the manifest, and the encoding is never your problem.</p>
<h3>macOS and openssl</h3><p>macOS ships a different <code>base64</code>. According to its <a href="https://keith.github.io/xcode-man-pages/bintrans.1" rel="noopener noreferrer">man page</a>, it does not wrap by default (<code>-b</code> sets a line length, and <code>-w</code> is accepted for GNU compatibility). Older macOS releases decode only with <code>-D</code>, and on them <code>-d</code> is a debug flag. A script that has to run on both should encode with a pipe that works everywhere:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$DB_URL</span>"</span> | <span class="hljs-built_in">base64</span> | <span class="hljs-built_in">tr</span> -d <span class="hljs-string">'\n'</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9vcmRlcnM/c3NsbW9kZT1yZXF1aXJl
</code></pre><p><code>openssl base64</code> is another option, but it wraps at 64 characters and has a sharper edge when decoding:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$DB_URL</span>"</span> | openssl <span class="hljs-built_in">base64</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$DB_URL</span>"</span> | <span class="hljs-built_in">base64</span> -w0 | openssl <span class="hljs-built_in">base64</span> -d | <span class="hljs-built_in">wc</span> -c; <span class="hljs-built_in">echo</span> <span class="hljs-string">"rc=<span class="hljs-variable">${PIPESTATUS[2]}</span>"</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$DB_URL</span>"</span> | <span class="hljs-built_in">base64</span> -w0 | openssl <span class="hljs-built_in">base64</span> -d -A; <span class="hljs-built_in">echo</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9v
cmRlcnM/c3NsbW9kZT1yZXF1aXJl
0
rc=0
postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require
</code></pre><p>Without <code>-A</code>, OpenSSL 3.0 decoded our 92-character single line to <strong>zero bytes and exit code 0</strong>. No error, just an empty value. If you use <code>openssl base64</code> for single-line data, pass <code>-A</code> both ways.</p>
<h2>Bug 3: double encoding</h2><p><code>--from-literal</code>, <code>--from-file</code> and <code>stringData</code> all take plain text and encode it for you. Give them a value that is already encoded and it gets encoded again:</p>
<pre><code class="hljs language-bash" tabindex="0">kubectl create secret generic db-creds \
  --from-literal=password=<span class="hljs-string">"<span class="hljs-subst">$(printf '%s' 'hunter2-prod' | base64)</span>"</span> \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.password}'</span>; <span class="hljs-built_in">echo</span>

kubectl create secret generic db-creds \
  --from-literal=password=<span class="hljs-string">"<span class="hljs-subst">$(printf '%s' 'hunter2-prod' | base64)</span>"</span> \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.password}'</span> | <span class="hljs-built_in">base64</span> -d; <span class="hljs-built_in">echo</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">YUhWdWRHVnlNaTF3Y205aw==
aHVudGVyMi1wcm9k
</code></pre><p>The pod now gets <code>aHVudGVyMi1wcm9k</code> as its password. The giveaway is a first decode that looks like base64 instead of the value you stored. Decode it again to confirm:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">echo</span> <span class="hljs-string">'YUhWdWRHVnlNaTF3Y205aw=='</span> | <span class="hljs-built_in">base64</span> -d | <span class="hljs-built_in">base64</span> -d; <span class="hljs-built_in">echo</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">hunter2-prod
</code></pre><p>The rule: <code>data</code> takes base64, and <code>stringData</code>, <code>--from-literal</code> and <code>--from-file</code> take the raw value. In a Helm chart, <code>b64enc</code> belongs in <code>data:</code>, never in <code>stringData:</code>.</p>
<p>Not every double encoding is a bug. PEM is itself base64 with header lines, so a TLS Secret's <code>tls.crt</code> field is encoded twice on purpose:</p>
<pre><code class="hljs language-bash" tabindex="0">openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout tls.key -out tls.crt -days 30 -subj <span class="hljs-string">'/CN=orders.internal'</span> 2&gt;/dev/null
kubectl create secret tls orders-tls --cert=tls.crt --key=tls.key \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.tls\.crt}'</span> | <span class="hljs-built_in">cut</span> -c1-40
kubectl create secret tls orders-tls --cert=tls.crt --key=tls.key \
  --dry-run=client -o jsonpath=<span class="hljs-string">'{.data.tls\.crt}'</span> | <span class="hljs-built_in">base64</span> -d | <span class="hljs-built_in">head</span> -2
</code></pre><pre><code class="hljs language-text" tabindex="0">LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J
-----BEGIN CERTIFICATE-----
MIIBijCCAS+gAwIBAgIUazfeROKqRj7HLheO41+kH88CZagwCgYIKoZIzj0EAwIw
</code></pre><p>That <code>LS0tLS1CRUdJTi</code> prefix is the <code>-----BEGIN</code> from earlier. The <code>certificate-authority-data</code> field in a kubeconfig works the same way.</p>
<h2>base64url and JWTs</h2><p>Two characters of the standard alphabet cause trouble outside of files. <code>+</code> means a space in a form-encoded query string, and <code>/</code> is a path separator. RFC 4648 defines a second alphabet, <strong>base64url</strong>, that uses <code>-</code> and <code>_</code> instead:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">printf</span> <span class="hljs-string">'\xfb\xff\xbf'</span> | <span class="hljs-built_in">base64</span>
<span class="hljs-built_in">printf</span> <span class="hljs-string">'\xfb\xff\xbf'</span> | basenc --base64url
</code></pre><pre><code class="hljs language-text" tabindex="0">+/+/
-_-_
</code></pre><p>This is not only about odd bytes. The database URL from earlier encodes to <code>...cnM/c3Ns...</code> in standard base64 and <code>...cnM_c3Ns...</code> in base64url. In a URL path, that <code>/</code> splits the value in two.</p>
<p>JWTs use base64url and also drop the <code>=</code> padding (<a href="https://www.rfc-editor.org/rfc/rfc7515#section-2" rel="noopener noreferrer">RFC 7515</a>). This script, saved as <code>make-jwt.sh</code>, builds a real HS256 token with shell tools:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-meta">#!/usr/bin/env bash</span>
<span class="hljs-built_in">set</span> -euo pipefail

<span class="hljs-function"><span class="hljs-title">b64url</span></span>() { basenc --base64url -w0 | <span class="hljs-built_in">tr</span> -d <span class="hljs-string">'='</span>; }

header=$(<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">'{"alg":"HS256","typ":"JWT"}'</span> | b64url)
payload=$(<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">'{"sub":"deploy-bot","role":"admin","exp":1793318400}'</span> | b64url)
signature=$(<span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$header</span>.<span class="hljs-variable">$payload</span>"</span> \
  | openssl dgst -sha256 -hmac <span class="hljs-string">'not-a-real-key'</span> -binary | b64url)

<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$header</span>.<span class="hljs-variable">$payload</span>.<span class="hljs-variable">$signature</span>"</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXBsb3ktYm90Iiwicm9sZSI6ImFkbWluIiwiZXhwIjoxNzkzMzE4NDAwfQ.mirpAmeTzGcS59Fo6JImIz4IBPSFSqpDJy1qltzzzuA
</code></pre><p>Both JSON parts start with <code>eyJ</code>. Now try to read the payload the obvious way:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">chmod</span> +x make-jwt.sh
TOKEN=$(./make-jwt.sh)
<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$TOKEN</span>"</span> | <span class="hljs-built_in">cut</span> -d. -f2 | <span class="hljs-built_in">base64</span> -d; <span class="hljs-built_in">echo</span> <span class="hljs-string">" rc=$?"</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">{"sub":"deploy-bot","role":"admin","exp":1793318400}base64: invalid input
 rc=1
</code></pre><p>The payload part is 70 characters, not a multiple of 4, so GNU <code>base64</code> prints what it could decode and then exits with 1. The JSON looks complete, which makes this easy to miss: under <code>set -e</code> the script stops here, and without it the script carries on with output that may be cut short. <code>basenc --base64url -d</code> fails the same way, because it also expects padding. Translate the alphabet and put the padding back first:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-function"><span class="hljs-title">b64url_decode</span></span>() {
  <span class="hljs-built_in">local</span> s
  s=$(<span class="hljs-built_in">tr</span> <span class="hljs-string">'_-'</span> <span class="hljs-string">'/+'</span>)               <span class="hljs-comment"># back to the standard alphabet</span>
  <span class="hljs-keyword">case</span> $(( <span class="hljs-variable">${#s}</span> % <span class="hljs-number">4</span> )) <span class="hljs-keyword">in</span>        <span class="hljs-comment"># put the stripped padding back</span>
    2) s=<span class="hljs-string">"<span class="hljs-variable">$s</span>=="</span> ;;
    3) s=<span class="hljs-string">"<span class="hljs-variable">$s</span>="</span> ;;
  <span class="hljs-keyword">esac</span>
  <span class="hljs-built_in">printf</span> <span class="hljs-string">'%s'</span> <span class="hljs-string">"<span class="hljs-variable">$s</span>"</span> | <span class="hljs-built_in">base64</span> -d
}

<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$TOKEN</span>"</span> | <span class="hljs-built_in">cut</span> -d. -f2 | b64url_decode; <span class="hljs-built_in">echo</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$TOKEN</span>"</span> | <span class="hljs-built_in">cut</span> -d. -f1 | b64url_decode; <span class="hljs-built_in">echo</span>
</code></pre><pre><code class="hljs language-text" tabindex="0">{"sub":"deploy-bot","role":"admin","exp":1793318400}
{"alg":"HS256","typ":"JWT"}
</code></pre><p>In Python, <code>base64.urlsafe_b64decode(part + "=" * (-len(part) % 4))</code> does the same in one line.</p>
<p>Anyone holding a signed JWT like this one can read its claims, because they are only base64url. The signature stops people from changing them, not from reading them, so never put a secret in a JWT payload. (Encrypted JWTs, JWE, are a different format with five parts.) For a quick look at a token, our <a href="https://devops-daily.com/tools/jwt-decoder">JWT decoder</a> runs in the browser, and the <a href="https://devops-daily.com/tools/base64">base64 tool</a> does the same for plain values.</p>
<h2>Data URIs</h2><p>A data URI puts a whole file inside a URL: <code>data:&lt;media type&gt;;base64,&lt;data&gt;</code>. Browsers then skip the extra request. We built one from this site's 180x180 Apple touch icon, saved as <code>icon.png</code>:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-built_in">wc</span> -c &lt; icon.png
<span class="hljs-built_in">printf</span> <span class="hljs-string">'data:image/png;base64,%s'</span> <span class="hljs-string">"<span class="hljs-subst">$(base64 -w0 icon.png)</span>"</span> | <span class="hljs-built_in">wc</span> -c
<span class="hljs-built_in">printf</span> <span class="hljs-string">'data:image/png;base64,%s'</span> <span class="hljs-string">"<span class="hljs-subst">$(base64 -w0 icon.png)</span>"</span> | <span class="hljs-built_in">cut</span> -c1-60
</code></pre><pre><code class="hljs language-text" tabindex="0">1167
1578
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAA
</code></pre><p>1,167 bytes became 1,578 characters: the 33% plus the 22-character prefix. Use <code>-w0</code> again so the URI stays on one line in HTML or CSS. The image also loses its own cache entry and is downloaded again with every page or stylesheet that inlines it. That is fine for a few small icons and bad for anything large or shared across pages.</p>
<p>For SVG, which is already text, base64 is not always the best choice. Our 110-byte SVG made a 174-character base64 data URI and a 143-character URL-encoded one (<code>data:image/svg+xml,%3Csvg ...</code>, with single quotes inside the SVG).</p>
<h2>Summary</h2><p>Base64 is a fixed mapping from bytes to 64 safe characters. It makes binary data fit in text formats, costs a third more space, and hides nothing. The bugs come from a short list of mistakes:</p>
<ul>
<li><strong>Base64 does not protect Secrets.</strong> RBAC, encryption at rest and keeping values out of Git do.</li>
<li><strong><code>echo</code> adds a newline.</strong> Use <code>printf '%s'</code> and check suspect values with <code>base64 -d | od -c</code>.</li>
<li><strong>GNU <code>base64</code> wraps at 76 characters.</strong> Use <code>-w0</code>, or <code>| tr -d '\n'</code> where macOS is involved. With <code>openssl base64</code>, pass <code>-A</code> both ways.</li>
<li><strong><code>data</code> takes base64; <code>stringData</code>, <code>--from-literal</code> and <code>--from-file</code> take raw values.</strong> If the first decode still looks like base64, check for double encoding.</li>
<li><strong>JWTs use base64url without padding.</strong> Swap <code>-_</code> back to <code>+/</code> and restore the <code>=</code> before decoding.</li>
<li><strong>Check the exit code of <code>base64 -d</code>.</strong> GNU prints partial output before it fails, and the Rust coreutils in Ubuntu 26.10 <a href="https://devops-daily.com/posts/ubuntu-26-10-rust-coreutils-what-changes">behave differently on bad input</a>.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[GitLab.com's 60-Requests-an-Hour Limit Is Live, and the Headers Won't Warn You]]></title>
      <link>https://devops-daily.com/posts/gitlab-anonymous-rate-limit-60-per-hour</link>
      <description><![CDATA[We sent anonymous requests to GitLab.com until the new per-IP limit blocked us. It depended on which project we asked for, the successful responses never mentioned it, and Retry-After was wrong in both directions. Here is what we measured and how to find the anonymous calls in your CI before GitLab does.]]></description>
      <pubDate>Thu, 29 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/gitlab-anonymous-rate-limit-60-per-hour</guid>
      <category><![CDATA[CI/CD]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[GitLab]]></category><category><![CDATA[CI/CD]]></category><category><![CDATA[Rate Limiting]]></category><category><![CDATA[API]]></category><category><![CDATA[Automation]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>GitLab.com now limits requests without credentials to 60 per hour per IP address, at least for part of its traffic. On the evening of October 28 we sent 140 anonymous API requests to GitLab's own <code>gitlab-runner</code> project in under ten minutes, and every one of them came back <code>200</code>. Then we pointed the same loop at a public project in a personal namespace. After 58 requests, the 59th came back <code>429 Too Many Requests</code>, and our IP stayed blocked for more than 40 minutes.</p>
<p>Three things from that evening matter if you run CI or automation against GitLab.com:</p>
<ul>
<li><strong>What counts depends on what you ask for.</strong> While we were blocked, requests to projects in the <code>gitlab-org</code> and <code>inkscape</code> groups still worked, and requests to personal projects and to <code>/api/v4/version</code> did not. If you test against a big open source project, you may conclude that nothing has changed.</li>
<li><strong>Successful responses do not show the new limit.</strong> The last <code>200</code> before the block carried <code>RateLimit-Remaining: 476</code>. That number belongs to the old 500-per-minute throttle. The new limit only appears in the <code>429</code> itself.</li>
<li><strong><code>Retry-After</code> follows the clock, the block does not.</strong> Every <code>Retry-After</code> pointed to the end of the current clock hour. The block ended about an hour after the first request that counted. Our first explanation for that was wrong, and a test with predictions written down in advance showed it.</li>
</ul>
<p>Everything below comes from one IP in one overnight session, October 28 to 29 UTC, with the scripts and raw results in a public repository, so you can run the same checks from your own network.</p>
<h2>TL;DR</h2><ul>
<li>GitLab announced the Free and unauthenticated limits for October 19, 2026, and the limits for Premium and Ultimate for January 2027. Anonymous traffic gets 60 requests per hour per IP.</li>
<li>From our IP, the budget was exactly 60 requests. The count started with the first request and reset about an hour later. Requests to the <code>gitlab-org</code> group did not count, <code>401</code> responses did, and blocked requests did not extend the block. The <code>429</code> carried <code>RateLimit-Name: unauthenticated_traffic_per_ip</code>.</li>
<li>Nothing in a successful response warns you. Track your own request counts, and alert on <code>429</code> responses with that <code>RateLimit-Name</code>.</li>
<li>Do not plan around <code>Retry-After</code> from this throttle. It points to the end of the clock hour, and the block ends about an hour after the first request of the current count.</li>
<li>A separate limit, most likely at Cloudflare, stopped bursts of <code>401</code> responses after 25 requests, with a <code>429</code> that looks nothing like GitLab's.</li>
<li>Behind one NAT address, every runner shares those 60 requests. Find anonymous <code>curl</code>, <code>wget</code> and Dockerfile <code>ADD</code> calls to gitlab.com and give them a token. <code>CI_JOB_TOKEN</code> covers raw files, releases, packages, artifacts and archives.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A GitLab.com account or group, and CI jobs or scripts that talk to gitlab.com</li>
<li><code>bash</code>, <code>curl</code> and Python 3.9 or later to run the scripts in the companion repository</li>
<li>Access to your runners' egress IP addresses, if you want to reason about shared limits</li>
</ul>
<h2>What GitLab announced</h2><p>GitLab's <a href="https://about.gitlab.com/blog/rate-limit-change-2026/" rel="noopener noreferrer">announcement</a> ties GitLab.com rate limits to the subscription plan. The numbers are on the <a href="https://docs.gitlab.com/user/gitlab_com/rate_limits/" rel="noopener noreferrer">rate limits page</a>, which still called them proposed when we read it on October 28:</p>
<table>
<thead>
<tr>
<th>Caller</th>
<th>Sustained limit</th>
<th>Burst limit (API)</th>
<th>Starts</th>
</tr>
</thead>
<tbody><tr>
<td>No credentials, per IP</td>
<td>60 per hour</td>
<td>none listed</td>
<td>October 19, 2026</td>
</tr>
<tr>
<td>Free, per signed-in user</td>
<td>5,000 per hour</td>
<td>100 per minute</td>
<td>October 19, 2026</td>
</tr>
<tr>
<td>Premium, per signed-in user</td>
<td>15,000 per hour</td>
<td>1,250 per minute</td>
<td>January 2027</td>
</tr>
<tr>
<td>Ultimate, per signed-in user</td>
<td>25,000 per hour</td>
<td>2,000 per minute</td>
<td>January 2027</td>
</tr>
</tbody></table>
<p>Free also has a separate 200-per-minute limit for web traffic. Personal access tokens, OAuth tokens and CI/CD job tokens all move a request from the anonymous limit to your plan's limit. Git over HTTPS is not counted against the anonymous limit. GitLab Self-Managed and GitLab Dedicated are not affected. The old per-minute limits stay in place, and GitLab applies the lower of the two.</p>
<p>The hourly limit matters more than the burst limit. At full burst speed, you reach the hourly cap long before the hour is over:</p>
<p><strong>Minutes at full burst speed until the hourly limit</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Free (5,000/h, 100/min)</td>
<td>50 min</td>
</tr>
<tr>
<td>Premium (15,000/h, 1,250/min)</td>
<td>12 min</td>
</tr>
<tr>
<td>Ultimate (25,000/h, 2,000/min)</td>
<td>12.5 min</td>
</tr>
</tbody></table>
<p><em>Sustained limit divided by burst limit, from GitLab's rate limits page, retrieved 2026-10-28. Per signed-in user, API traffic.</em></p>
<h2>How we measured</h2><p>The companion repository holds the scripts and every result file:</p>
<p><a href="https://github.com/The-DevOps-Daily/gitlab-rate-limit-check" rel="noopener noreferrer">The-DevOps-Daily/gitlab-rate-limit-check on GitHub</a></p>
<p><code>probe.sh</code> sends requests with a one-second pause between them and writes the status code, the cache status and the rate-limit headers of each response to a TSV file stamped with the script's hash. All requests came from one residential IP address over IPv4, with no token, on October 28 between 22:38 and 23:38 UTC and on October 29 between 01:00 and 04:03 UTC.</p>
<p><strong>probe.sh against a project in a personal namespace</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ PROJECT=dwt1%2Fdotfiles ./probe.sh api 70
results/api-anon-20261028T225259Z.tsv
     58 200	BYPASS	throttle_unauthenticated_api	500
     12 429	BYPASS	unauthenticated_traffic_per_ip
$ sed -n <span class="hljs-string">'58,61p'</span> results/api-anon-20261028T225259Z.tsv
56	22:54:21	200	BYPASS	throttle_unauthenticated_api	500	478	
57	22:54:23	200	BYPASS	throttle_unauthenticated_api	500	477	
58	22:54:24	200	BYPASS	throttle_unauthenticated_api	500	476	
59	22:54:26	429	BYPASS	unauthenticated_traffic_per_ip			334
</code></pre><p>The columns are request number, UTC time, status, Cloudflare cache status, <code>RateLimit-Name</code>, <code>RateLimit-Limit</code>, <code>RateLimit-Remaining</code> and <code>Retry-After</code>.</p>
<h2>Finding 1: the limit depends on the project you request</h2><p>Before that run we had already sent 140 anonymous API requests to <code>gitlab-org/gitlab-runner</code> and 70 cache-busted raw-file requests to the same project, in just over ten minutes. All 210 returned <code>200</code>, and the only throttle in the headers was the old <code>throttle_unauthenticated_api</code> or <code>throttle_unauthenticated_web</code>, at 500 requests per minute.</p>
<p>The run against <code>dwt1/dotfiles</code>, a popular public project in a personal namespace, was blocked after 58 requests. That hour we had also made two requests to endpoints outside any project, <code>/api/v4/version</code> and a <code>/api/v4/projects</code> search. If those count, the block came after exactly 60 requests, which also suggests nothing else on that address was sending anonymous requests to GitLab.com. While the IP was blocked, we sent one request to each of six targets:</p>
<table>
<thead>
<tr>
<th>Target</th>
<th>Status while blocked</th>
</tr>
</thead>
<tbody><tr>
<td><code>gitlab-org/gitlab-runner</code></td>
<td>200</td>
</tr>
<tr>
<td><code>gitlab-org/gitlab</code></td>
<td>200</td>
</tr>
<tr>
<td><code>inkscape/inkscape</code></td>
<td>200</td>
</tr>
<tr>
<td><code>dwt1/dotfiles</code> (personal namespace)</td>
<td>429</td>
</tr>
<tr>
<td><code>CalcProgrammer1/OpenRGB</code> (personal namespace)</td>
<td>429</td>
</tr>
<tr>
<td><code>/api/v4/version</code> (no project)</td>
<td>429</td>
</tr>
</tbody></table>
<p>Our reading is that the new anonymous limit depends on the namespace you request, most likely on its plan: <code>gitlab-org</code> is GitLab's own group, Inkscape is a large open source group, and requests to both went through. We cannot see a namespace's plan without credentials, so this is an inference from the pattern, not a fact we could check.</p>
<p>To test that reading, we waited until the IP had been quiet for more than an hour and ran <code>count-test.sh</code> at 01:00:45 UTC on October 29. It sends 10 requests to <code>gitlab-org/gitlab-runner</code>, then 10 to <code>/api/v4/version</code>, then requests to <code>dwt1/dotfiles</code> until the first <code>429</code>. If the <code>gitlab-org</code> requests do not count and the others do, the last step gets 50 successes before the block. It got exactly 50:</p>
<p><strong>count-test.sh from a clean start</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./count-test.sh
results/count-test-20261029T010045Z.tsv
10 paid 200
50 personal 200
1 personal 429
10 unscoped 401
personal-namespace successes before the first 429: 50 (prediction: 50)
</code></pre><p><code>paid</code> is the script's label for the <code>gitlab-org</code> step, after our guess about why it is not counted. The same total would come out if the <code>gitlab-org</code> requests counted and the <code>/api/v4/version</code> requests did not, but the first evening rules that out: more than 200 <code>gitlab-org</code> requests in the same hour as the 58 personal ones did not trigger a block. The <code>/api/v4/version</code> requests all returned <code>401 Unauthorized</code>, so a request that fails still uses up the budget.</p>
<p>So if you want to know whether your automation will be throttled, test it against the projects it really uses. A quick check against a large open source project tells you nothing.</p>
<h2>Finding 2: successful responses do not show the new limit</h2><p>GitLab's advice is to watch <code>RateLimit-Remaining</code> and slow down as it drops. For this limit, that does not work. Every successful response in our runs described only the old per-minute throttle. Request 58 got <code>RateLimit-Remaining: 476</code>, and request 59 got a <code>429</code>.</p>
<p>The <code>429</code> was the first response that mentioned the new limit. It carried <code>RateLimit-Name: unauthenticated_traffic_per_ip</code> and a <code>Retry-After</code>, but no <code>RateLimit-Limit</code> or <code>RateLimit-Remaining</code>.</p>
<p>So you have two options. Count your own anonymous requests, or alert on the first <code>429</code> with that <code>RateLimit-Name</code>. The repository's <code>glcurl</code> wrapper logs one JSON line per request without the token, and <code>budget.py</code> reads those logs and reports each caller's busiest minute and busiest hour against the plan limits.</p>
<h2>Finding 3: the count resets an hour after it starts, not at the top of the hour</h2><p>The first <code>429</code>, at 22:54:26 UTC, said <code>Retry-After: 334</code>. That points to 23:00:00, the top of the hour. We waited until 23:00:45 and tried again. Requests to <code>gitlab-org</code> still worked, but the first request that counts was blocked, and this time <code>Retry-After</code> was 3,540 seconds, pointing to midnight. Then we sent one request every three minutes to see when the block really ended:</p>
<table>
<thead>
<tr>
<th>UTC</th>
<th>Status</th>
<th><code>Retry-After</code></th>
</tr>
</thead>
<tbody><tr>
<td>22:54:26</td>
<td>429 (first)</td>
<td>334</td>
</tr>
<tr>
<td>23:01:01</td>
<td>429</td>
<td>3,540</td>
</tr>
<tr>
<td>23:02:07</td>
<td>429</td>
<td>3,473</td>
</tr>
<tr>
<td>23:20:10</td>
<td>429</td>
<td>2,390</td>
</tr>
<tr>
<td>23:35:12</td>
<td>429</td>
<td>1,488</td>
</tr>
<tr>
<td>23:38:12</td>
<td>not limited</td>
<td>none</td>
</tr>
</tbody></table>
<p>Every <code>Retry-After</code> was the time until the end of the current clock hour, but the block ended between 23:35 and 23:38. The first request that counted that evening was a one-off <code>/api/v4/version</code> call. It has no result file, but our terminal log puts it between 22:38:02 and 22:38:03, and an hour after that falls inside the three minutes in which the block ended.</p>
<p>Our first explanation was a sliding window: count the requests of the last 3,600 seconds, and let the caller back in as the oldest one drops out. We tested it a few hours later. The count test had blocked the IP at 01:02:29, and its first counted request was at 01:01:01. <code>predict-window.py</code> worked out what three counting models expect, and we committed its predictions before <code>window-test.sh</code> ran:</p>
<table>
<thead>
<tr>
<th>Request</th>
<th>Clock hour</th>
<th>Sliding 60 minutes</th>
<th>Weighted estimate</th>
<th>GitLab</th>
</tr>
</thead>
<tbody><tr>
<td>One at 01:50:00</td>
<td>429</td>
<td>429</td>
<td>429</td>
<td>429, <code>Retry-After: 600</code></td>
</tr>
<tr>
<td>One at 02:00:30</td>
<td>allowed</td>
<td>429</td>
<td>allowed</td>
<td>429, <code>Retry-After: 3570</code></td>
</tr>
<tr>
<td>Back to back from 02:01:45</td>
<td>59 allowed</td>
<td>35 to 46 allowed</td>
<td>1 allowed</td>
<td>25 allowed, then a different 429</td>
</tr>
</tbody></table>
<p>The weighted estimate is a common shortcut in rate limiters: the previous hour's count, weighted by the part of it still inside the last 3,600 seconds, plus the current hour's count. Half a minute into the new clock hour we were still blocked, and <code>Retry-After</code> now pointed to 03:00:00. A minute later, 25 requests went through, and then the burst was stopped by the limit in the next section. The sliding window matched, and we wrote it up that way.</p>
<p>The second review of this post found the hole. A counter that starts with the first request and resets completely 3,600 seconds later, the way a Redis <code>INCR</code> with an <code>EXPIRE</code> on the first increment works, fits the same three rows. The two models only disagree while old requests are leaving a sliding window, and our first test of that was cut off at request 26 by the same limit. So we ran another, with requests that return <code>200</code> and do not trip that limit. At 03:04:00 the sliding window predicted 25 more requests, and the resetting counter 35. GitLab allowed 35:</p>
<p><strong>series-test.sh against a personal project, predictions committed at 03:03:46</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ URL=<span class="hljs-string">'https://gitlab.com/api/v4/projects/dwt1%2Fdotfiles'</span> START=$(<span class="hljs-built_in">date</span> -u -d 03:04:00 +%s) ./series-test.sh
results/series-test-20261029T030400Z.tsv
     35 200	throttle_unauthenticated_api
      1 429	unauthenticated_traffic_per_ip
</code></pre><p>So the sliding window was wrong. The first test could have told the two apart, but the other limit stopped it after 25 requests. What matched every response from the hourly limit: a count that starts with the first request that counts, allows 60, and resets about an hour after that first request.</p>
<p>We tested that once more, with the predictions committed first. The count that series ran into started with a request sent at 03:02:00.5, so it should reset at about 04:02:00, not at 04:00:00, where <code>Retry-After</code> pointed:</p>
<p><strong>reset-test.sh, predictions committed at 03:06:45</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ TIMES=<span class="hljs-string">"04:00:30 04:01:50 04:02:10"</span> BURST_AT=04:02:15 ./reset-test.sh
results/reset-test-20261029T030649Z.tsv
2026-10-29T04:00:30.666Z 429 unauthenticated_traffic_per_ip 3569
2026-10-29T04:01:50.122Z 429 unauthenticated_traffic_per_ip 3490
2026-10-29T04:02:10.586Z 200 throttle_unauthenticated_api
1 x 429 unauthenticated_traffic_per_ip
59 x 200 throttle_unauthenticated_api
</code></pre><p>Both blocked responses now pointed <code>Retry-After</code> to 05:00:00, about 59 and 58 minutes away. Less than 21 seconds later the block was gone, and all 60 came back at once: one at 04:02:10 and 59 in the burst that followed. <code>predict-window.py --compare</code> checks each response against what each model expects, given the requests GitLab had accepted before it. Only the resetting counter matched all 63 responses in this run. The clock hour matched 61, because it expected the two requests after 04:00:00 to go through, and its forecast before the run had expected 57 in the burst, not 59.</p>
<p>Two more things follow from the data. Blocked requests did not extend the block: we got 38 blocked responses the first evening, and it still ended about an hour after 22:38. And the budget is exactly 60. After the 25-request burst at 02:01:45, <code>burst-test.sh</code> sent 25 more back to back, waited 75 seconds, and then sent requests with a 1.5-second pause between them. With 50 used, it should get 10 more, and it got exactly 10:</p>
<p><strong>burst-test.sh: 25 + 25 + 10 = 60</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./burst-test.sh
results/burst-test-20261029T020328Z.tsv
     25 C	401	throttle_unauthenticated_api
      1 C	429	
     10 D	401	throttle_unauthenticated_api
      1 D	429	unauthenticated_traffic_per_ip
</code></pre><p>So <code>Retry-After</code> from this throttle is wrong in both directions. At 22:54 it pointed about 35 to 38 minutes too early. At 04:01:50 it said 3,490 seconds, and the block ended less than 21 seconds later. If you keep a log of your anonymous requests, the block ends about an hour after the first request of the current count, and then all 60 come back at once. Since blocked requests did not extend the block, checking again every few minutes is safe. The better fix is to stop sending anonymous requests from that address and switch to a token.</p>
<p>For the other limits, <code>backoff.py</code> in the repository waits <code>Retry-After</code>, uses capped exponential backoff with full jitter when a <code>429</code> has no headers, and slows down when <code>RateLimit-Remaining</code> gets low.</p>
<h2>Finding 4: a second limit we could not pin down</h2><p>Every run of requests to <code>/api/v4/version</code> that got that far stopped at request 26 with a different <code>429</code>: no <code>RateLimit-Name</code>, and <code>Retry-After: 60</code>. Two were back-to-back bursts that took about ten seconds, and one was a slower series that took 48. We saved one of these responses in full. Like every gitlab.com response it carried <code>server: cloudflare</code>, but none of the headers that GitLab's own <code>429</code> had, such as <code>gitlab-lb</code>, <code>gitlab-sv</code>, <code>x-runtime</code> and the <code>RateLimit-*</code> headers. So it most likely came from Cloudflare, not from GitLab. The body said:</p>
<pre><code class="hljs language-text" tabindex="0">The service is temporarily unavailable due to elevated error rates. Please retry after a short wait.
</code></pre><p>Every request in those runs had returned <code>401</code>, because <code>/api/v4/version</code> needs a token. The message suggests a limit on errors, but 60 back-to-back <code>404</code> responses for a missing file in <code>gitlab-org/gitlab-runner</code>, at the pace of the fast bursts, did not trip it. Neither did 60 <code>200</code> responses sent four at a time in about eight seconds. So it does not apply one request rate or one error count to every route. It may count <code>401</code> responses, or apply only to that endpoint, or leave <code>gitlab-org</code> alone. Our tests cannot tell these apart.</p>
<p>It matters for anything that retries quickly on authentication errors. Our runs got this <code>429</code> after 25 failed requests, and it carries none of the headers your GitLab rate-limit handling looks for. When a <code>429</code> has no <code>RateLimit-Name</code>, log its body before you decide how long to wait.</p>
<h2>A method bug: cached raw files never reach the limiter</h2><p>Our first raw-file run looked strange. All 70 requests returned <code>200</code>, and <code>RateLimit-Remaining</code> stayed at 494 for the whole run. The probe did not record cache headers yet, so two minutes later we checked by hand:</p>
<pre><code class="hljs language-text" tabindex="0">cf-cache-status: HIT
age: 48
cache-control: max-age=60, public, must-revalidate, stale-while-revalidate=60, stale-if-error=300, s-maxage=60
ratelimit-remaining: 494
ratelimit-reset: 1792878180
</code></pre><p>That is an excerpt of one response, saved in the repository. Cloudflare answered from its cache, so the rate-limit headers were a frozen copy: that <code>ratelimit-reset</code> is 21:43 UTC on October 24, four days before our run. Those requests most likely never reached GitLab's limiter. With a unique query string on each request, all 70 responses were <code>cf-cache-status: MISS</code> and the counter moved again. <code>probe.sh</code> now adds that query string by default and records the cache status of every response.</p>
<p>The API responses all said <code>cf-cache-status: BYPASS</code>, so API calls reach GitLab every time. Do not count on the cache for raw files either: you cannot tell in advance which requests it will absorb.</p>
<h2>Where it hurts: many runners behind one address</h2><p>60 requests per hour is plenty for a laptop. It is not much for a fleet of self-hosted runners that leave the network through one NAT gateway, because every anonymous request from every runner counts against the same IP.</p>
<p>Here is a worked example, not a measurement. Twelve parallel jobs per pipeline, a pipeline every ten minutes, and each job fetches an install script and makes two API calls without a token. <code>examples/make_nat_log.py</code> writes that hour as a log, and <code>budget.py</code> compares it with the limits. It groups anonymous requests by the egress address they leave from, and token requests by the GitLab user behind the token:</p>
<pre><code class="hljs language-text" tabindex="0">$ python3 budget.py examples/runner-nat.jsonl
caller                       requests  max/min  limit  max/hour  limit    use
anon@runners-nat                  216       36    500       216     60   360%  &lt;-- over
token:release-bot                  90       90    100        90   5000    90%  &lt;-- near
</code></pre><p>The busiest minute, 36 requests, is far below the old 500-per-minute limit. Each pipeline makes 36 anonymous requests, so if all of them count and the hour starts with an unused budget, the second pipeline takes the total past 60. The release bot with a token is fine on the hourly limit, but 90 requests in 45 seconds is close to the Free burst limit of 100 per minute.</p>
<p>The usual sources of anonymous traffic in CI are:</p>
<ul>
<li>install scripts fetched with <code>curl ... | sh</code> from a <code>/-/raw/</code> URL</li>
<li>release binaries downloaded from <code>/-/releases/</code> links</li>
<li>source archives from <code>/-/archive/</code></li>
<li><code>ADD https://gitlab.com/...</code> in a Dockerfile, which runs on every uncached build</li>
<li>API calls through <code>$CI_API_V4_URL</code> without a <code>JOB-TOKEN</code> header</li>
</ul>
<h2>Find the anonymous calls before GitLab does</h2><p><code>find_anon_calls.py</code> scans CI files, shell and PowerShell scripts, Dockerfiles and Makefiles for these calls and tells you what to use instead. Git over HTTPS is skipped, because it is not counted against the anonymous limit. Run against three of the repository's test fixtures:</p>
<pre><code class="hljs language-text" tabindex="0">$ python3 find_anon_calls.py tests/fixtures/.gitlab-ci.yml tests/fixtures/Dockerfile tests/fixtures/install.sh
tests/fixtures/.gitlab-ci.yml:8: [raw] anonymous call to https://gitlab.com/acme/tools/-/raw/main/install.sh
    fix: use GET /api/v4/projects/:id/repository/files/:path/raw?ref=:ref with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/.gitlab-ci.yml:12: [api] anonymous call to $CI_API_V4_URL/projects/42/releases
    fix: add a token header: JOB-TOKEN: $CI_JOB_TOKEN in CI (packages, releases, raw files, artifacts, archives) or a project/group access token for other endpoints
tests/fixtures/.gitlab-ci.yml:20: [archive] anonymous call to https://gitlab.com/acme/lib/-/archive/v2.0/lib-v2.0.tar.gz
    fix: use GET /api/v4/projects/:id/repository/archive with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/Dockerfile:3: [raw] anonymous call to https://gitlab.com/acme/tools/-/raw/main/healthcheck.sh
    fix: use GET /api/v4/projects/:id/repository/files/:path/raw?ref=:ref with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/Dockerfile:5: [raw] anonymous call to https://gitlab.com/acme/tools/-/raw/main/motd
    fix: use GET /api/v4/projects/:id/repository/files/:path/raw?ref=:ref with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/install.sh:3: [download] anonymous call to https://gitlab.com/acme/cli/-/releases/v1.4.0/downloads/cli-linux-amd64
    fix: download through the API (releases, job artifacts, packages) with a token
6 anonymous GitLab call(s) found
</code></pre><p>It exits with status 1 when it finds anything, so you can add it as a CI job. It decides per command, so a token on one command does not cover the next one on the same line. It is still a pattern matcher, not a parser: it misses URLs built at runtime and calls in languages it does not read.</p>
<h2>The fix: send a token</h2><p>In a GitLab CI job you already have one. According to GitLab's <a href="https://docs.gitlab.com/ci/jobs/ci_job_token/" rel="noopener noreferrer">job token documentation</a>, <code>CI_JOB_TOKEN</code> can read raw files, releases and release links, packages, job artifacts and repository archives. For a project outside your own, add your project to the target project's job token allowlist. Public and internal projects allow some of these reads without it.</p>
<p>The most common change is the install script:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Before: anonymous, counts against the runner's IP</span>
curl -fsSL https://gitlab.com/acme/tools/-/raw/main/install.sh | sh

<span class="hljs-comment"># After: authenticated with the job token, counts against your plan's limits.</span>
<span class="hljs-comment"># 1234 is the numeric ID of the acme/tools project; the file path is URL-encoded.</span>
curl -fsSL --header <span class="hljs-string">"JOB-TOKEN: <span class="hljs-variable">$CI_JOB_TOKEN</span>"</span> \
  <span class="hljs-string">"<span class="hljs-variable">$CI_API_V4_URL</span>/projects/1234/repository/files/install.sh/raw?ref=main"</span> | sh
</code></pre><p>For endpoints the job token cannot reach, such as reading project or group details, use a project or group access token with the <code>read_api</code> scope, stored as a masked CI/CD variable. Outside CI, use a personal access token with the narrowest scope that works. And if a third-party tool cannot authenticate at all, GitLab asks you to contact <code>limits@gitlab.com</code>.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>One IP, one night.</strong> GitLab may enforce the limit differently in other regions, at other times, or for other address ranges.</li>
<li><strong>Namespace plans.</strong> We inferred that paid groups are not counted yet. We cannot see a namespace's plan without credentials.</li>
<li><strong>The exact rule.</strong> A counter that resets about an hour after its first request matched every response from the hourly limit, including two runs predicted in advance. Close variants fit the same data, such as a reset an hour after the start of that request's minute, or a few seconds short of an hour. A clock-hour window, a weighted estimate and a sliding window each failed at least one prediction. Other implementations could fit the same data, and GitLab can change it at any time.</li>
<li><strong>The second limit.</strong> We could trigger it only with <code>401</code> responses from <code>/api/v4/version</code>. We do not know what it counts, and that it runs at Cloudflare is an inference from the headers.</li>
<li><strong>Authenticated limits.</strong> We had no GitLab token on the test machine, so we did not test the per-user Free limits of 5,000 per hour and 100 per minute.</li>
<li><strong>January.</strong> We could not test the Premium and Ultimate limits, which start in January 2027, or whether requests to paid groups start counting against the anonymous limit then.</li>
</ul>
<h2>Summary</h2><p>The new anonymous limit is live on GitLab.com for at least some of the traffic, and it is easy to miss. It did not apply to the projects we tested first. Successful responses carried no sign of it, and the <code>Retry-After</code> on the <code>429</code> did not say when the block would end.</p>
<p>Before it shows up as a failed pipeline:</p>
<ol>
<li>Run <code>find_anon_calls.py</code> over your CI files, scripts and Dockerfiles.</li>
<li>Give every gitlab.com call a token, starting with <code>CI_JOB_TOKEN</code> where it is allowed.</li>
<li>Count requests per runner egress IP, not per job.</li>
<li>Alert on <code>429</code> responses with <code>RateLimit-Name: unauthenticated_traffic_per_ip</code>. Do not wait for their <code>Retry-After</code>: in our tests the block ended about an hour after the first request of the current count from that address.</li>
<li>Before January, check your busiest hour against the Premium or Ultimate limits with <code>budget.py</code>.</li>
</ol>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[15 Minutes to 5: What Happens When You Move iOS CI From M1 to M4]]></title>
      <link>https://devops-daily.com/posts/ios-ci-m1-to-m4-15-minutes-to-5</link>
      <description><![CDATA[The Wikipedia iOS app's build and 503 unit tests took a median 15.4 minutes on GitHub's M1 runner and 5.6 minutes on an M4. We split the time by phase, priced each run, and checked whether a bigger Mac is worth it.]]></description>
      <pubDate>Thu, 29 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/ios-ci-m1-to-m4-15-minutes-to-5</guid>
      <category><![CDATA[CI/CD]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[CI/CD]]></category><category><![CDATA[iOS]]></category><category><![CDATA[Xcode]]></category><category><![CDATA[macOS]]></category><category><![CDATA[GitHub Actions]]></category><category><![CDATA[Bitrise]]></category>
      <content:encoded><![CDATA[<p>The Wikipedia iOS app's build and its 503 unit tests took a median <strong>922 seconds (15.4 minutes)</strong> on GitHub's standard <code>macos-26</code> runner, an Apple M1 with 3 vCPU. The same job, with the same Xcode, simulator and <code>xcodebuild</code> commands, took a median <strong>337 seconds (5.6 minutes)</strong> on a Bitrise Build Hub M4 Large with 5 vCPU. That is 2.7 times faster, over 17 runs on each side, every run cold.</p>
<p>In our <a href="https://devops-daily.com/posts/github-actions-bill-keeps-growing">first article</a> this was one row in a pipeline benchmark. Here we look only at the iOS job: which phase gets faster, what a job costs at list price, whether a bigger Mac helps, and what you change in a workflow to move it.</p>
<p><a href="https://github.com/The-DevOps-Daily/gha-runner-benchmark" rel="noopener noreferrer">The-DevOps-Daily/gha-runner-benchmark on GitHub</a></p>
<h2>TLDR</h2><ul>
<li><strong>The build gained the most minutes:</strong> 538 s on the M1 against 145 s on the M4 Large (3.7x). The unit tests went from 370 s to 86 s (4.3x).</li>
<li><strong>Cost per job at list price:</strong> a median $0.99 on GitHub (16 billed minutes at $0.062) and $0.21 on the M4 Large (7 billed minutes at $0.03).</li>
<li><strong>A bigger M4 adds little:</strong> the 10 vCPU M4 XLarge took 9% less time than the M4 Large and costs twice as much per minute. The 14 vCPU M4 Pro XLarge took 23% less time, and it is on the Enterprise plan only.</li>
<li><strong>Machines start on demand:</strong> iOS jobs waited a median 38 seconds for a Build Hub machine, against 8 seconds on GitHub.</li>
<li><strong>The workflow change is the <code>runs-on</code> line:</strong> it points at a Build Hub pool. The setup around it is a GitHub App, a pool and an image with your Xcode. GitHub Actions still schedules the job and keeps the logs and secrets.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>An iOS project that already builds in GitHub Actions with <code>xcodebuild</code></li>
<li>A way to read job times: the Actions tab, or the <code>gh</code> CLI for many runs</li>
<li>For Build Hub: a Bitrise workspace and the Build Hub GitHub App installed on the repository</li>
</ul>
<h2>The setup</h2><table>
<thead>
<tr>
<th>Side</th>
<th>Runner</th>
<th>Machine</th>
<th>List price</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub-hosted</td>
<td><code>macos-26</code></td>
<td>Apple M1 (virtual), 3 vCPU, 7 GB</td>
<td>$0.062 a minute</td>
</tr>
<tr>
<td>Build Hub, self-serve Pro Tier 1</td>
<td>M4 Large</td>
<td>Apple M4 (virtual), 5 vCPU, 14 GB</td>
<td>$0.03 a minute</td>
</tr>
</tbody></table>
<p>These are not same-size machines: the M4 Large has more cores and twice the memory. We compared the standard macOS runner on GitHub with the smallest Mac on Build Hub: the default choice on each side.</p>
<p><strong>The workload.</strong> The <a href="https://github.com/wikimedia/wikipedia-ios" rel="noopener noreferrer">Wikipedia iOS app</a> at commit <code>ca8aaeb9</code>, its own "Run Unit Tests" job: resolve Swift packages, <code>xcodebuild build-for-testing</code>, then <code>test-without-building</code> on an iPhone 17 simulator with iOS 26.5 and Xcode 26.5. In the separate runs below, all 503 tests passed every time, and in the pipeline runs every iOS job succeeded. Every run started cold, with no DerivedData or package cache on either side. Cold runs compare the whole environment, not only the chip: the images, macOS patch versions, storage and network also differ.</p>
<p><strong>How we ran it.</strong> Bitrise provided the Build Hub workspace for these tests. The 17 runs per side come from the pipeline benchmark on 16 and 17 October 2026, dispatched in pairs. The phase split and the bigger Macs come from separate runs of the same job on 15 and 16 October: 7 on GitHub and 3 on each Build Hub machine. Those are fewer samples, so compare their rows with each other, not with the 17-run numbers. All runs and the raw results are in the <a href="https://github.com/The-DevOps-Daily/gha-runner-benchmark" rel="noopener noreferrer">benchmark repository</a>.</p>
<h2>Where the minutes go</h2><p><strong>Wikipedia iOS job by phase, median seconds</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Resolve packages</td>
<td>64s</td>
<td>GitHub M1 3 vCPU</td>
</tr>
<tr>
<td>Resolve packages</td>
<td>27s</td>
<td>Build Hub M4 Large</td>
</tr>
<tr>
<td>Build for testing</td>
<td>538s</td>
<td>GitHub M1 3 vCPU</td>
</tr>
<tr>
<td>Build for testing</td>
<td>145s</td>
<td>Build Hub M4 Large</td>
</tr>
<tr>
<td>Unit tests</td>
<td>370s</td>
<td>GitHub M1 3 vCPU</td>
</tr>
<tr>
<td>Unit tests</td>
<td>86s</td>
<td>Build Hub M4 Large</td>
</tr>
</tbody></table>
<p><em>Separate runs of the iOS job on 15 and 16 October 2026: GitHub macos-26 7 runs, Build Hub M4 Large 3 runs. Cold, no cache. Source: results/ios-wikipedia-unit-tests.json.</em></p>
<table>
<thead>
<tr>
<th>Phase</th>
<th>GitHub M1, 3 vCPU</th>
<th>Build Hub M4 Large, 5 vCPU</th>
<th>Faster by</th>
</tr>
</thead>
<tbody><tr>
<td>Resolve packages</td>
<td>64 s</td>
<td>27 s</td>
<td>2.4x</td>
</tr>
<tr>
<td>Build for testing</td>
<td>538 s</td>
<td>145 s</td>
<td>3.7x</td>
</tr>
<tr>
<td>Unit tests</td>
<td>370 s</td>
<td>86 s</td>
<td>4.3x</td>
</tr>
<tr>
<td>Whole job, same runs</td>
<td>1,026 s</td>
<td>323 s</td>
<td>3.2x</td>
</tr>
</tbody></table>
<p>The build is where most of the minutes are, and it is where most of them were saved: 393 seconds per job. Compiling Swift is CPU work, and the M4 Large has a newer chip and two more cores. The unit tests gained the most in relative terms. They run in the iOS simulator, which needs CPU and memory too, and the M1 runner has 7 GB for Xcode, the simulator and the test host together. We did not measure memory pressure, so that is a possible factor, not a finding.</p>
<p>Package resolution gained least. It downloads the same packages on both sides, and we did not time the network part separately. These are the separate runs, so the whole job took 1,026 seconds on GitHub and 323 seconds on the M4 Large. Checkout, Xcode selection and the other steps took the rest: a median 87 and 69 seconds, measured run by run. Phase medians do not add up to the job median.</p>
<h2>What a job costs</h2><p>At list price, each job's billed time is rounded up to a whole minute and multiplied by the per-minute price. Build Hub bills per started minute and includes the machine's start-up, which the job's timestamps do not show, so we estimated it from the wait before each job, as in the first article. That is why the M4 Large bills 7 minutes for a 337 second job. Over the 17 pipeline runs per side:</p>
<table>
<thead>
<tr>
<th>Side</th>
<th>Median job time</th>
<th>Billed minutes</th>
<th>Cost per job</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub <code>macos-26</code></td>
<td>922 s</td>
<td>16</td>
<td>$0.99</td>
</tr>
<tr>
<td>Build Hub M4 Large</td>
<td>337 s</td>
<td>7</td>
<td>$0.21</td>
</tr>
</tbody></table>
<p>Two things to keep in mind:</p>
<ul>
<li><strong>On GitHub, public repositories pay nothing</strong> for standard runners. In a private repository, standard macOS minutes come out of your plan's included usage first; the $0.99 is the price after it is used up.</li>
<li><strong>Build Hub is sold as a monthly tier.</strong> If you do not use the budget you pay for, the real price per job is higher than $0.21.</li>
</ul>
<p>A team that runs the iOS job 20 times a day would, at these list prices, spend about $20 a day on GitHub and about $4 on Build Hub, and use about 3 hours less job time. That is total job time, not the time any one person waits: jobs run in parallel, and other jobs may be slower. Check your own run count before you trust that number: rounding makes short jobs cost more than they look.</p>
<h2>Is a bigger Mac worth it?</h2><p>We ran the same job on the larger Build Hub Macs too.</p>
<table>
<thead>
<tr>
<th>Machine</th>
<th>Plan</th>
<th>Runs</th>
<th>Median job time</th>
<th>Cost per job</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub <code>macos-26</code>: M1, 3 vCPU, 7 GB</td>
<td>GitHub</td>
<td>7</td>
<td>1,026 s (805 to 1,305)</td>
<td>$1.12 (18 billed minutes)</td>
</tr>
<tr>
<td>Build Hub M4 Large: M4, 5 vCPU, 14 GB</td>
<td>Self-serve</td>
<td>3</td>
<td>323 s (314 to 335)</td>
<td>$0.18 (6 billed minutes)</td>
</tr>
<tr>
<td>Build Hub M4 XLarge: M4, 10 vCPU, 28 GB</td>
<td>Self-serve</td>
<td>3</td>
<td>295 s (293 to 301)</td>
<td>$0.36 (6 billed minutes)</td>
</tr>
<tr>
<td>Build Hub M4 Pro XLarge: M4 Pro, 14 vCPU, 54 GB</td>
<td>Enterprise</td>
<td>3</td>
<td>250 s (246 to 251)</td>
<td>No per-minute list price</td>
</tr>
</tbody></table>
<p>Twice the cores and twice the memory, from the M4 Large to the M4 XLarge, saved 28 seconds (9%) and doubled the cost per job. For this app, the Large gave most of the gain at half the XLarge's cost per job. The M4 Pro XLarge, with the M4 Pro chip and 14 vCPU, took 73 seconds less than the Large (23%). It is on the Enterprise plan, which Bitrise sells by concurrency and not by minutes, so we cannot give a list price per job for it. Billed minutes for the Build Hub rows include the estimated machine start-up, as above.</p>
<p>Your app may be different. A project with many independent modules can use more cores than the Wikipedia app did. Run your own job three times on each size before you pick one.</p>
<h2>Moving one job</h2><p>GitHub Actions still runs the workflow: it schedules the job, streams the logs and passes the secrets. The work is in the setup, and the workflow itself changes little.</p>
<ol>
<li>Install the Build Hub GitHub App on the repository, and create a pool in your Bitrise workspace with the machine type and an image that has your Xcode version.</li>
<li>Point <code>runs-on</code> at the pool's label. Ours was <code>bitrise-runner-ios-m4-large</code>.</li>
<li>Select the Xcode version in the job, so that both sides use the same one.</li>
</ol>
<p>Images name the Xcode app differently (<code>Xcode_26.5.app</code>, <code>Xcode-26.5.app</code> or <code>Xcode.app</code>), so our job picks the app by the version it reports, and fails if it is not there:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">jobs:</span>
  <span class="hljs-attr">ios-tests:</span>
    <span class="hljs-attr">runs-on:</span> [<span class="hljs-string">bitrise-runner-ios-m4-large</span>] <span class="hljs-comment"># was: macos-26</span>
    <span class="hljs-attr">env:</span>
      <span class="hljs-attr">XCODE_VERSION:</span> <span class="hljs-string">'26.5'</span>
    <span class="hljs-attr">steps:</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Select</span> <span class="hljs-string">Xcode</span>
        <span class="hljs-attr">run:</span> <span class="hljs-string">|
          for app in /Applications/Xcode*.app; do
            if [ "$(DEVELOPER_DIR="$app/Contents/Developer" xcodebuild -version 2&gt;/dev/null | head -1)" = "Xcode $XCODE_VERSION" ]; then
              sudo xcode-select -s "$app/Contents/Developer"
              break
            fi
          done
          test "$(xcodebuild -version | head -1)" = "Xcode $XCODE_VERSION"</span>
</code></pre><p>The <code>xcodebuild</code> steps did not change at all. Two checks are worth adding before the first real run:</p>
<ul>
<li><strong>The simulator exists.</strong> <code>xcrun simctl list devices available</code> should list the device and the iOS version in your <code>-destination</code>. If the image lacks the runtime, the job fails, so check it before the first real run.</li>
<li><strong>Expect a short wait for a machine.</strong> A self-serve pool starts a machine when a job arrives. Our iOS jobs waited a median 38 seconds before they started, which is not in the job times above.</li>
</ul>
<h2>What we could not conclude</h2><ul>
<li><strong>Chip or cores.</strong> The M1 runner has 3 vCPU and the M4 Large has 5. We did not run a 3 vCPU M4, so we cannot say how much of the 2.7x comes from the chip and how much from the cores and the memory. Going from 5 to 10 M4 cores, with twice the memory, saved only 9%. The 14 vCPU M4 Pro saved 23%, but it is a different chip, so we cannot say how much came from the cores.</li>
<li><strong>Warm builds.</strong> Every run was cold. With a DerivedData or package cache, both sides get faster, and the gap may shrink. We did not measure that.</li>
<li><strong>Other apps.</strong> This is one app at one commit. A smaller app saves fewer minutes, and a larger one may save more.</li>
<li><strong>GitHub's larger Mac runners.</strong> GitHub also sells larger macOS runners on paid plans. We compared the standard one, which every team gets.</li>
</ul>
<h2>Summary</h2><p>Moving the Wikipedia iOS job from GitHub's M1 runner to a Build Hub M4 Large took it from 15.4 to 5.6 minutes and from $0.99 to $0.21 per job at list price, with the same workflow steps. Most of the time came out of the build and the simulator tests. A bigger M4 added little; the M4 Pro XLarge took 23% less time than the M4 Large. If your iOS job is the slowest one in your pipeline, measure it on one larger machine before you change anything else, and keep the Xcode version pinned so the comparison stays fair.</p>
<p>The workflows, the dispatch scripts and every result are in the <a href="https://github.com/The-DevOps-Daily/gha-runner-benchmark" rel="noopener noreferrer">benchmark repository</a>. The setup for Build Hub is in <a href="https://bitrise.io/platform/build-hub" rel="noopener noreferrer">Bitrise's documentation</a>.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Moving Authorization Out of Your App Code: If-Statements vs Cerbos vs SpiceDB, Measured]]></title>
      <link>https://devops-daily.com/posts/authorization-if-statements-vs-cerbos-vs-spicedb</link>
      <description><![CDATA[We built one permission model as if-statements, as Cerbos policies and as a SpiceDB relationship graph, grew it through roles, teams, sharing and a late contractor rule, and checked 84 hand-written decisions at every step. Here is what each design cost in latency, in list queries and in lines of code.]]></description>
      <pubDate>Tue, 27 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/authorization-if-statements-vs-cerbos-vs-spicedb</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Authorization]]></category><category><![CDATA[Security]]></category><category><![CDATA[Architecture]]></category><category><![CDATA[Cerbos]]></category><category><![CDATA[SpiceDB]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>Many products rebuild their permissions three times: first roles, then teams, then per-document sharing. Somewhere after that, a rule arrives that nobody planned for. The usual advice is to move authorization out of the application code before that happens, into a policy engine or a Zanzibar-style relationship graph. We wanted to know what that move costs and what it buys, so we built the same permission model three ways and grew it in four steps:</p>
<ol>
<li><strong>if-statements</strong> in the application, with a SQL query for lists.</li>
<li><strong>Cerbos</strong> policy files, evaluated by a local Cerbos policy decision point (PDP).</li>
<li><strong>SpiceDB</strong>, an open source implementation of Google's <a href="https://research.google/pubs/zanzibar-googles-consistent-global-authorization-system/" rel="noopener noreferrer">Zanzibar</a> design: a schema plus relationships.</li>
</ol>
<p>A hand-written table of 84 expected decisions checked all three at every step. Then we timed them on a Raspberry Pi 4 with 10,000 documents. The numbers describe this setup: a Pi, one request at a time, the app database in the same process. Both the milliseconds and the ratios between the designs belong to this setup; a network, a different database or other hardware can change either.</p>
<p>What we found:</p>
<ul>
<li><strong>A single check is cheap only in-process.</strong> The if-statements answered in 0.14 ms. Cerbos took 3.8 ms and SpiceDB 10.8 ms (median p50). On this Pi a call to either server that does no authorization work already costs about 2.4 ms, and Cerbos's own log puts its handling time at 0.63 ms. By the bar we set before measuring (p99 under 5 ms), a check through either engine is not cheap on this machine.</li>
<li><strong>Listing is where they really differ.</strong> "Which of these 10,000 documents can this member view?" took 4.4 ms as SQL, 11.9 to 31.6 ms through a Cerbos query plan turned into SQL (depending on two lines of our own adapter), and 404 ms through SpiceDB when its cache was cold (77 ms warm). For an admin who can view all 10,000, SpiceDB took 3.3 seconds cold.</li>
<li><strong>The late rule cost little code in every design:</strong> 22 changed lines in one file for the if-statements, 15 in two files for Cerbos, 13 in two files for SpiceDB. It never stayed out of the application: Cerbos needed a new attribute in every request, and SpiceDB needed a new relationship synced from the database. In SpiceDB it also made a cold list about 3.5 times slower.</li>
<li><strong>Leave out part of the late rule and every design fails open.</strong> The table caught each of the three faults we injected, and it also caught SpiceDB returning duplicate documents in a list.</li>
</ul>
<h2>TLDR</h2><ul>
<li>Same model, three designs, four steps, one table of 84 hand-written decisions. All three matched it at every step.</li>
<li>Single check, median p50 and p99: if-statements 0.14 / 0.24 ms, Cerbos 3.8 / 7.9 ms, SpiceDB 10.8 / 19.4 ms on new checks and 2.8 ms p50 on repeated ones, because SpiceDB caches results.</li>
<li>List for a typical member (1,020 of 10,000 visible): SQL 4.4 ms, Cerbos 11.9 to 31.6 ms, SpiceDB 77 ms warm to 404 ms cold. SpiceDB's own docs suggest a different pattern above about 10,000 results, and for an admin's first page it took 18 ms.</li>
<li>Lines changed over the four steps: 83 for if-statements, 211 for Cerbos, 187 for SpiceDB. The engines cost more up front; by step 4 a new rule cost them fewer lines than the if-statements (15 and 13 against 22).</li>
<li>The engines did not remove application changes. Every new rule needed data, and the data lives in your app.</li>
<li>Keep the if-statements while one app owns the rules and you can test them as a table. Move out when several services need the same answer or the relationships nest deeply.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Node.js 22.13 or later (the demo uses the built-in <code>node:sqlite</code>)</li>
<li>Linux on arm64 or amd64. The demo downloads pinned Cerbos and SpiceDB release binaries and checks their SHA-256. No Docker.</li>
<li>Basic familiarity with role-based access control and SQL joins</li>
<li>The demo repository, which contains every commit, script and recorded run in this post:</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/authz-three-ways" rel="noopener noreferrer">The-DevOps-Daily/authz-three-ways on GitHub</a></p>
<h2>The model and the four steps</h2><p>The app has users, teams, documents and shares in SQLite. The actions are <code>view</code>, <code>edit</code> and <code>delete</code>. We wrote the rules for all four steps into <code>CLAIM.md</code> before writing any code, together with what result would prove our claim wrong:</p>
<table>
<thead>
<tr>
<th>Step</th>
<th>Rule</th>
</tr>
</thead>
<tbody><tr>
<td>1. Roles</td>
<td>Each user has one org role. <code>viewer</code> can view every document, <code>editor</code> can view and edit, <code>admin</code> can view, edit and delete.</td>
</tr>
<tr>
<td>2. Teams</td>
<td>Each document belongs to a team. <code>viewer</code> and <code>editor</code> now apply only to documents of the user's own teams. <code>admin</code> is unchanged.</td>
</tr>
<tr>
<td>3. Sharing</td>
<td>A document can be shared with one user as <code>view</code> or <code>edit</code>, whatever that user's teams are. A share never grants delete.</td>
</tr>
<tr>
<td>4. Late rule</td>
<td>Contractors get no access at all to documents outside their own teams. This overrides every earlier grant, including <code>admin</code> and shares.</td>
</tr>
</tbody></table>
<p>We picked the late rule before implementing anything. It is a realistic one: security asks for it after an audit, and it cuts across every grant that already exists, so it tests whether a design can say "no" over the top of many "yes" paths.</p>
<p>The expected-decisions table, <code>fixture/expected.csv</code>, has seven users, two teams, four documents and four shares, and one row per (user, action, document) with the expected answer at each step:</p>
<pre><code class="hljs language-text" tabindex="0">user,action,document,step1,step2,step3,step4
cora,view,d4,allow,deny,allow,deny
kai,delete,d1,allow,allow,allow,deny
nell,view,d2,allow,deny,allow,allow
</code></pre><p>Cora is an editor in <code>eng</code> and a contractor, with an edit share on <code>d4</code>, a <code>sales</code> document. Kai is an admin and a contractor in <code>sales</code>. Nell is a viewer with no team and a view share on <code>d2</code>. <code>harness/correctness.js</code> runs every row against each design at each step, and also asks each design for the full list of documents each user can view, which must match the table's <code>view</code> rows.</p>
<h2>Three designs</h2><ul>
<li><strong>if-statements</strong> everything in the app<ul>
<li><strong>check()</strong> JavaScript, up to 4 queries</li>
<li><strong>list()</strong> one SQL query</li>
<li><strong>SQLite</strong> the app's own tables</li>
</ul>
</li>
<li><strong>Cerbos</strong> stateless policy engine<ul>
<li><strong>App</strong> reads attributes from SQLite</li>
<li><strong>Cerbos PDP</strong> policy YAML, gRPC</li>
<li><strong>Plan to SQL</strong> PlanResources, your adapter</li>
</ul>
</li>
<li><strong>SpiceDB</strong> Zanzibar-style graph<ul>
<li><strong>App</strong> syncs rows to relationships</li>
<li><strong>SpiceDB</strong> schema, gRPC</li>
<li><strong>Relationships</strong> SpiceDB's own copy</li>
</ul>
</li>
</ul>
<p>The three designs put the rule and the data in different places, and that difference explains almost every number below.</p>
<ul>
<li><strong>if-statements</strong> (<code>approaches/ifs/index.js</code>): one <code>check()</code> function that reads the user, the document, the team membership and any share from SQLite, and one <code>list()</code> function that is a SQL query. The rule is written twice: once in JavaScript and once in SQL.</li>
<li><strong>Cerbos</strong> (<code>approaches/cerbos/</code>): the rules live in YAML policies, with derived roles such as <code>team_editor</code> and <code>shared_viewer</code>. Cerbos stores no application data, so for every check the app reads the same rows from SQLite and sends them as principal and resource attributes. For lists, Cerbos's PlanResources API returns a condition tree, and our <code>plan-to-sql.js</code> turns it into a SQL <code>WHERE</code>.</li>
<li><strong>SpiceDB</strong> (<code>approaches/spicedb/</code>): the rules live in a schema, and the data lives in SpiceDB as relationships such as <code>document:d1#team@team:eng</code>. <code>sync.js</code> copies the app's rows into SpiceDB. A check is one CheckPermission call with no database read in the app. A list is LookupResources.</li>
</ul>
<p>Both engines ran on 127.0.0.1 and were called over gRPC through their official Node clients, one client each for the whole run, without TLS: Cerbos 0.57.0 with policies on disk, and SpiceDB 1.57.0 with its in-memory datastore, which its <a href="https://authzed.com/docs/spicedb/concepts/datastores" rel="noopener noreferrer">docs</a> describe as intended for testing integrations. Each design has one commit per step in the repo, tagged <code>step&lt;N&gt;-ifs</code>, <code>step&lt;N&gt;-cerbos</code> and <code>step&lt;N&gt;-spicedb</code>, so every line count below comes from <code>git diff --numstat</code>.</p>
<h2>The same rule, three ways</h2><p>Here is the late rule, "contractors get nothing outside their own teams", as each design wrote it. These are excerpts from the step 4 commits; <code>// ...</code> marks lines left out.</p>
<p><strong>Step 4: contractors get nothing outside their own teams</strong></p>
<p><strong>if-statements: check()</strong></p>
<pre><code class="hljs language-javascript," tabindex="0">const inTeam = Boolean(isMember.get(doc.team_id, userId));
// Contractors get nothing outside their own teams, whatever else grants it.
if (user.contractor &amp;&amp; !inTeam) return false;
// A share grants access whatever the user's role and teams are.
const share = getShare.get(docId, userId);
if (share &amp;&amp; SHARE_ACTIONS[share.level].includes(action)) return true;
// Roles other than admin apply only to documents of the user's teams.
if (user.role !== 'admin' &amp;&amp; !inTeam) return false;
return ROLE_ACTIONS[user.role].includes(action);
</code></pre><p><strong>if-statements: list()</strong></p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-comment">// Contractors see only documents of their own teams.</span>
<span class="hljs-keyword">const</span> contractorFilter = <span class="hljs-string">`(:contractor = 0 OR d.team_id IN
  (SELECT team_id FROM team_members WHERE user_id = :user))`</span>;
<span class="hljs-keyword">const</span> allDocs = db.<span class="hljs-title function_">prepare</span>(<span class="hljs-string">`SELECT d.id FROM documents d WHERE <span class="hljs-subst">${contractorFilter}</span>`</span>);
<span class="hljs-keyword">const</span> visibleDocs = db.<span class="hljs-title function_">prepare</span>(<span class="hljs-string">`
  SELECT d.id FROM documents d
  JOIN team_members m ON m.team_id = d.team_id
  WHERE m.user_id = :user
  UNION
  SELECT s.document_id FROM shares s
  JOIN documents d ON d.id = s.document_id
  WHERE s.user_id = :user AND <span class="hljs-subst">${contractorFilter}</span>`</span>);
</code></pre><p><strong>Cerbos: policy</strong></p>
<pre><code class="hljs language-yaml," tabindex="0"># Contractors get nothing outside their own teams, whatever else grants it.
- actions: ["*"]
  effect: EFFECT_DENY
  roles: ["*"]
  condition:
    match:
      all:
        of:
          - expr: request.principal.attr.contractor == true
          - expr: "!(request.resource.attr.team in request.principal.attr.teams)"
</code></pre><p><strong>Cerbos: app</strong></p>
<pre><code class="hljs language-javascript," tabindex="0">const getUser = db.prepare('SELECT id, role, contractor FROM users WHERE id = ?');
// ...
return { id: user.id, roles: [user.role], attr: { teams, contractor: user.contractor === 1 } };
</code></pre><p><strong>SpiceDB: schema</strong></p>
<pre><code class="hljs language-zed," tabindex="0">definition org {
    // ...
    relation contractor: user
}

definition document {
    // ...
    // Contractors get nothing outside their own teams, whatever else grants it.
    permission outside_contractor = org-&gt;contractor - team-&gt;member

    permission view = (org-&gt;admin + team-&gt;view_docs + viewer + editor) - outside_contractor
    permission edit = (org-&gt;admin + team-&gt;edit_docs + editor) - outside_contractor
    permission delete = org-&gt;delete_docs - outside_contractor
}
</code></pre><p><strong>SpiceDB: sync</strong></p>
<pre><code class="hljs language-javascript," tabindex="0">for (const u of db.prepare('SELECT id, role, contractor FROM users').all()) {
  out.push(touch('org', ORG, u.role, 'user', u.id));
  if (u.contractor) out.push(touch('org', ORG, 'contractor', 'user', u.id));
}
</code></pre><p>Three things stand out:</p>
<ul>
<li><strong>The if-statements wrote the rule more than once:</strong> in <code>check()</code>, and again in each list query that can return a document from outside the user's teams (the admin query and the shares branch). Miss one and the list leaks while every check passes.</li>
<li><strong>Cerbos needed no change to the plan adapter.</strong> The planner folds the principal's side of the condition into constants, so for Cora the deny rule becomes an <code>eq</code> on <code>request.resource.attr.team</code> with the value <code>eng</code>, ANDed with her grants, which the adapter already handled (<code>results/final/plans-step4.json</code>). It did need one more attribute from the app.</li>
<li><strong>SpiceDB's rule is one line of schema,</strong> an exclusion: <code>org-&gt;contractor - team-&gt;member</code>. But SpiceDB knows only what the app writes into it, so <code>sync.js</code> needed a new relationship type too.</li>
</ul>
<h2>What each step cost in code</h2><p><strong>Lines changed in each design, per step</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>1: roles</th>
<th>2: teams</th>
<th>3: sharing</th>
<th>4: contractors</th>
</tr>
</thead>
<tbody><tr>
<td>if-statements</td>
<td>29 lines</td>
<td>12 lines</td>
<td>20 lines</td>
<td>22 lines</td>
</tr>
<tr>
<td>Cerbos</td>
<td>73 lines</td>
<td>82 lines</td>
<td>41 lines</td>
<td>15 lines</td>
</tr>
<tr>
<td>SpiceDB</td>
<td>133 lines</td>
<td>31 lines</td>
<td>10 lines</td>
<td>13 lines</td>
</tr>
</tbody></table>
<p><em>Lines added plus deleted in each design's own commit, from git diff --numstat. The shared data-model commits (29, 26, 9, 5 lines) are the same for all three and not counted. Step 2 for SpiceDB includes the de-duplication fix the table forced (8 of its 31 lines).</em></p>
<table>
<thead>
<tr>
<th>Step</th>
<th>if-statements</th>
<th>Cerbos</th>
<th>SpiceDB</th>
</tr>
</thead>
<tbody><tr>
<td>1. Roles</td>
<td>1 file, 29 lines</td>
<td>3 files, 73 lines</td>
<td>3 files, 133 lines</td>
</tr>
<tr>
<td>2. Teams</td>
<td>1 file, 12 lines</td>
<td>4 files, 82 lines</td>
<td>3 files, 31 lines</td>
</tr>
<tr>
<td>3. Sharing</td>
<td>1 file, 20 lines</td>
<td>4 files, 41 lines</td>
<td>2 files, 10 lines</td>
</tr>
<tr>
<td>4. Late rule</td>
<td>1 file, 22 lines</td>
<td>2 files, 15 lines</td>
<td>2 files, 13 lines</td>
</tr>
<tr>
<td>Total</td>
<td>83 lines</td>
<td>211 lines</td>
<td>187 lines</td>
</tr>
</tbody></table>
<p>The shapes differ more than the totals. The if-statements were cheapest to start, and from step 2 on each rule cost more than the last (12, 20, 22 lines), because every rule touched both <code>check()</code> and the list query. Cerbos paid most at step 2, when the first condition appeared and we had to write the 43-line plan-to-SQL adapter. SpiceDB paid most at step 1: client setup, paging for LookupResources and the sync code. After that, sharing cost SpiceDB 10 lines.</p>
<p>Two caveats keep these numbers honest. Line counts depend on style, so read the diffs in the repo rather than trusting a total. And our if-statements are the best case: one module with one <code>check()</code>. If your checks are spread over many handlers, the late rule is many edits.</p>
<h2>A single check</h2><p><strong>One permission check, 10,000 documents (median p50, tick at p99)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>p99</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>if-statements, in-process</td>
<td>0.14ms</td>
<td>0.24ms</td>
<td>if-statements</td>
</tr>
<tr>
<td>Cerbos over gRPC</td>
<td>3.83ms</td>
<td>7.91ms</td>
<td>Cerbos</td>
</tr>
<tr>
<td>SpiceDB over gRPC, new checks</td>
<td>10.78ms</td>
<td>19.37ms</td>
<td>SpiceDB</td>
</tr>
<tr>
<td>SpiceDB over gRPC, repeated checks</td>
<td>2.83ms</td>
<td>15.62ms</td>
<td>SpiceDB</td>
</tr>
<tr>
<td>No-op call to Cerbos</td>
<td>2.4ms</td>
<td>6ms</td>
<td>No-op call</td>
</tr>
<tr>
<td>No-op call to SpiceDB</td>
<td>2.44ms</td>
<td>6.31ms</td>
<td>No-op call</td>
</tr>
</tbody></table>
<p><em>Raspberry Pi 4, step 4 model, concurrency 1. Checks: median across 3 runs of 10,000 each. No-op calls: one run of 5,000 each. "New checks" are random (user, action, document) triples; "repeated checks" draw from 100 triples, so SpiceDB's cache answers most of them. A no-op call does no authorization work. Absolute numbers are from a Pi: compare the bars with each other.</em></p>
<p>We timed 10,000 random (user, action, document) checks per run, three runs, one at a time, with the three designs interleaved on every sample so that anything else on the Pi slowed all of them equally. All three gave the same answer on every sample that completed.</p>
<p>At the median, the in-process check (0.144 ms) was about 27 times faster than Cerbos (3.83 ms) and about 75 times faster than SpiceDB (10.8 ms). Where that time goes:</p>
<ul>
<li>A request to either server that does no authorization work (Cerbos <code>ServerInfo</code>, SpiceDB <code>ReadSchema</code>) took 2.40 and 2.44 ms at the median on this Pi. A Node-to-Node TCP echo between two processes took 0.33 ms, so most of that is request handling in the gRPC client and server, not the loopback network.</li>
<li>Cerbos logs its own handling time for each request. Its median was 0.63 ms, against 3.8 ms seen by the client.</li>
<li>So for Cerbos, a request that does nothing already costs most of what a check costs. For SpiceDB on new checks, it is about a quarter of the 10.8 ms. We did not measure where the rest goes inside SpiceDB, and a <code>ReadSchema</code> call is only a reference point, not the same work as a check minus the graph.</li>
<li>SpiceDB's cache removes most of that difference. When the same 100 checks repeated, as they do when people reopen the same documents, its p50 dropped to 2.8 ms, close to the no-op call, while Cerbos stayed at 3.8 ms.</li>
</ul>
<p>One thing the chart hides: the if-statements' check runs up to four queries against SQLite in the same process. With Postgres across a network, each of those queries is a network round trip (the <a href="https://devops-daily.com/comparisons/sqlite-vs-postgresql">SQLite vs PostgreSQL comparison</a> covers this difference in architecture). The Cerbos design pays the same queries and the PDP call on top. The SpiceDB design reads nothing from the app database for a check. In that layout the gap between the in-process design and the engines could be smaller than it is here. We did not measure it.</p>
<p>One of 60,000 SpiceDB checks failed with <code>UNAVAILABLE: Stream closed with error code NGHTTP2_REFUSED_STREAM</code>. SpiceDB closes client connections after 30 seconds by default (<code>--grpc-max-conn-age</code>), which is our best explanation, and the Node client did not retry it. We counted the failure and left it out of the timings. In production, give the client a retry policy for <code>UNAVAILABLE</code>.</p>
<h2>Listing what a user can see</h2><p>This is the question every list page asks, and the designs answer it in completely different ways:</p>
<ul>
<li><strong>if-statements:</strong> a SQL query with a <code>UNION</code> of team documents and shared documents, filtered for contractors.</li>
<li><strong>Cerbos:</strong> PlanResources returns a condition tree for this user, our adapter turns it into a <code>WHERE</code>, and SQLite runs it.</li>
<li><strong>SpiceDB:</strong> LookupResources walks the graph backwards from the user and streams document IDs. The app then fetches those rows from its own database, a step we did not time.</li>
</ul>
<p><strong>List the documents a typical member can view (1,020 of 10,000)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>if-statements: SQL</td>
<td>4.4ms</td>
<td>if-statements</td>
</tr>
<tr>
<td>Cerbos: plan to SQL, IN subquery</td>
<td>11.9ms</td>
<td>Cerbos</td>
</tr>
<tr>
<td>Cerbos: plan to SQL, as committed</td>
<td>31.6ms</td>
<td>Cerbos</td>
</tr>
<tr>
<td>SpiceDB: LookupResources, warm</td>
<td>77.2ms</td>
<td>SpiceDB</td>
</tr>
<tr>
<td>SpiceDB: LookupResources, cold</td>
<td>403.9ms</td>
<td>SpiceDB</td>
</tr>
</tbody></table>
<p><em>Median ms on a Raspberry Pi 4, step 4 model. "As committed" turns each share condition into a correlated EXISTS; the IN-subquery variant changes only those two lines. SpiceDB bars come from a separate cache experiment with one unlimited LookupResources call: "cold" is the first call in a new 5 s revision window, "warm" repeats it at the same revision.</em></p>
<table>
<thead>
<tr>
<th>10,000 documents</th>
<th>Visible</th>
<th>if-statements</th>
<th>Cerbos (as committed)</th>
<th>SpiceDB, warm</th>
<th>SpiceDB, cold</th>
</tr>
</thead>
<tbody><tr>
<td>Admin</td>
<td>10,000</td>
<td>25.3 ms</td>
<td>29.9 ms</td>
<td>691 ms</td>
<td>3,279 ms</td>
</tr>
<tr>
<td>Member</td>
<td>1,020</td>
<td>4.4 ms</td>
<td>31.6 ms</td>
<td>77 ms</td>
<td>404 ms</td>
</tr>
<tr>
<td>Contractor</td>
<td>510</td>
<td>2.4 ms</td>
<td>9.6 ms</td>
<td>41 ms</td>
<td>247 ms</td>
</tr>
</tbody></table>
<p>The SQL and Cerbos columns are medians of the run medians from <code>results/final/list-*.json</code>. The SpiceDB columns come from a separate cache experiment, <code>harness/spicedb-list-cache.js</code>, which separates the two cases on purpose, because SpiceDB's list time depends on its cache more than on anything else. It differs from our adapter's list call in two ways: it makes one LookupResources call with no limit instead of pages of 1,000, and it reads with <code>minimize_latency</code> and then <code>at_exact_snapshot</code> instead of <code>at_least_as_fresh</code>. Timed through the adapter itself, the member's list had a median of 75 ms and the admin's 803 ms, mostly warm calls; see below.</p>
<p><strong>SpiceDB LookupResources, step 4, 10,000 documents</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ <span class="hljs-built_in">nice</span> -n 19 node harness/spicedb-list-cache.js --tree=.steps/step4 --out=results/final/spicedb-list-cache.json
admin      visible 10000  first <span class="hljs-keyword">in</span> a new 5 s window: median 3278.544 ms  same revision again: median 691.44 ms
member     visible 1020  first <span class="hljs-keyword">in</span> a new 5 s window: median 403.896 ms  same revision again: median 77.2 ms
contractor visible 510  first <span class="hljs-keyword">in</span> a new 5 s window: median 246.668 ms  same revision again: median 41.139 ms
problems (wrong answers or a reused revision): 0
</code></pre><p>SpiceDB caches results per revision. By default it reads at a revision rounded to a 5-second window (<code>--datastore-revision-quantization-interval</code>), and the in-memory datastore's revisions are timestamps, so every 5 seconds the read revision moves and the cache starts cold, even with no writes. Which case a request gets depends on timing. In our runs of 20 back-to-back admin lists (<code>results/final/list-*.json</code>), most found a warm cache, with a median of about 800 ms, and the slowest took about 3.7 s.</p>
<p>Four more things we measured about lists:</p>
<ul>
<li><strong>Our Cerbos adapter was the slow part, not Cerbos.</strong> The committed adapter turns "this user is in the document's <code>viewers</code>" into a correlated <code>EXISTS</code>, which SQLite runs once per document. Changing those two adapter lines to an uncorrelated <code>d.id IN (SELECT document_id FROM shares WHERE ...)</code> cut the member list from 31.6 ms to 11.9 ms, with the same answers (<code>scripts/variant.js</code>). The plan is only as fast as the SQL you turn it into, and that SQL is yours to write. Cerbos publishes <a href="https://docs.cerbos.dev/cerbos/latest/api/" rel="noopener noreferrer">reference adapters</a> for Prisma, Drizzle, SQLAlchemy and others; we used none, because we used plain SQLite.</li>
<li><strong>Page size matters for LookupResources.</strong> Our list code asks LookupResources for pages of 1,000, the default of SpiceDB's <code>--max-lookup-resources-limit</code> flag, and follows the cursor. For the admin that took 0.8 seconds over 11 calls. Pages of 100 took 7.2 seconds over 106 calls, and a single call with no limit took 0.7 seconds (<code>results/final/lookup-pages.json</code>).</li>
<li><strong>SpiceDB's docs already say this.</strong> Its guide to <a href="https://authzed.com/docs/spicedb/modeling/protecting-a-list-endpoint" rel="noopener noreferrer">protecting a list endpoint</a> suggests LookupResources when a user can see fewer than about 10,000 resources, and otherwise fetching a page of candidates from your own database and filtering them with CheckBulkPermissions. We timed that for "the first 50 documents in ID order", against our adapter listing every visible ID with LookupResources, sorting them and keeping 50 (<code>harness/spicedb-page.js</code>). For the admin the bulk checks took 18 ms with one call, against 786 ms for the full LookupResources list. For the contractor, who can see 5% of the documents, it took 167 ms over 11 bulk calls, against 37 ms. So the right pattern depends on how much of the data each user can see.</li>
<li><strong>The late rule made SpiceDB's cold lists about 3.5 times slower.</strong> We ran the cold and warm list test against the code of every step (<code>results/steps/step&lt;N&gt;-spicedb-list-cache.json</code>). From step 3 to step 4, the admin's cold list went from 928 to 3,204 ms, the member's from 108 to 390 ms and the contractor's from 64 to 227 ms. The warm lists did not move (686 to 696, 73 to 70, 42 to 38 ms). Thirteen lines of schema and sync were cheap to write; the exclusion they added is not cheap to evaluate.</li>
</ul>
<h2>What the table caught</h2><p>The 84-row table was the most useful file in the repo. It caught two things we would otherwise have shipped.</p>
<p><strong>SpiceDB returns duplicates.</strong> At step 2, Kai, an admin who is also a member of <code>sales</code>, got <code>d3</code> and <code>d4</code> twice from our adapter's paged LookupResources calls: once through the admin path and once through the team path. This is expected behaviour: an <a href="https://github.com/authzed/spicedb/issues/3190" rel="noopener noreferrer">issue in the SpiceDB repository</a> explains that two paths to the same resource can produce two results, calls duplicates a known limitation of cursored calls, and recommends de-duplicating on the client. We added a <code>Set</code> and recorded the failing run in <code>results/caught/</code>.</p>
<p><strong>Every design fails open when part of the late rule is missing.</strong> <code>scripts/break.js</code> leaves out one part of the step 4 change in each design and runs the table again:</p>
<p><strong>leave out part of the late rule</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/break.js &gt; results/breaks.txt
$ grep -v <span class="hljs-string">'^     '</span> results/breaks.txt
ifs-forgot-the-list-query      mismatches 2 (list) caught
cerbos-forgot-the-attribute    mismatches 10 (check, list) caught
spicedb-forgot-the-sync        mismatches 10 (check, list) caught
</code></pre><ul>
<li><strong>if-statements:</strong> <code>check()</code> has the rule, but <code>list()</code> kept its step 3 query. Every check passes, and Cora's list still shows the <code>sales</code> document shared with her.</li>
<li><strong>Cerbos:</strong> the policy has the deny rule, but the app does not send <code>contractor</code>. The condition cannot match a missing attribute, so the deny never fires. Cerbos can <a href="https://docs.cerbos.dev/cerbos/latest/policies/schemas" rel="noopener noreferrer">validate attributes</a> against a JSON schema; with <code>reject</code> enforcement, its docs say every action of an invalid request is denied. With <code>contractor</code> marked as required, that would turn this failure from open to closed. We did not test it.</li>
<li><strong>SpiceDB:</strong> the schema has the exclusion, but <code>sync.js</code> does not write <code>org#contractor</code>. With no relationship there is nothing to exclude.</li>
</ul>
<p>None of the three designs made these omissions safe. The table of expected decisions caught all three, which is why we would run it against every change.</p>
<h2>The costs this benchmark leaves out</h2><ul>
<li><strong>Keeping SpiceDB in sync.</strong> <code>sync.js</code> copies the whole database at start. A real app must write every change, such as a new share or a new contractor, to both its database and SpiceDB, and a crash between the two writes leaves them disagreeing. AuthZed calls this <a href="https://authzed.com/blog/the-dual-write-problem" rel="noopener noreferrer">the dual-write problem</a> and lists ways to handle it, from a transactional outbox to making SpiceDB the source of truth. Cerbos and the if-statements read the app's data at request time, so they do not have this problem.</li>
<li><strong>Running a server.</strong> Both engines are another service to deploy, monitor, upgrade and keep available, or a hosted product to pay for.</li>
<li><strong>A real datastore.</strong> SpiceDB on Postgres or CockroachDB will have different numbers from the in-memory datastore.</li>
<li><strong>Fetching the rows.</strong> SpiceDB's list returns IDs. A list page then needs a second query for titles and dates.</li>
</ul>
<h2>When the if-statements are the right answer</h2><p>In this repo the if-statements were the fastest design on every timing we took for all three, and the smallest in total code. They are the right answer more often than vendor pages suggest. Keep them when:</p>
<ul>
<li><strong>One application owns the rules.</strong> Everything lives in one codebase and one database, and one team changes both. A policy engine earns its keep when several services must answer the same question the same way.</li>
<li><strong>Your rules are mostly filters on your own tables.</strong> Teams, owners and shares are joins. SQL already plans them, and your list endpoint keeps pagination, sorting and search.</li>
<li><strong>You can keep the checks in one module.</strong> Our <code>ifs</code> design is one file with one <code>check()</code> and one <code>list()</code>. That is the best case. If your checks are spread over fifty handlers, the late rule is fifty edits, and our line counts do not apply to you.</li>
<li><strong>You test the rules as data.</strong> The expected-decisions table is what made each step safe, and it works for if-statements as well as for policies.</li>
</ul>
<p>The if-statements stop being enough when:</p>
<ul>
<li><strong>More than one service or language needs the same answer.</strong> Copying the rules into each one is how they drift apart.</li>
<li><strong>People outside the team need to read or change the rules,</strong> for example a security review of what contractors can see. A separate policy file or schema may be easier for them to review than a SQL <code>UNION</code> inside a JavaScript function, depending on what they already read.</li>
<li><strong>You need a record of decisions.</strong> Cerbos can write <a href="https://docs.cerbos.dev/cerbos/latest/configuration/audit" rel="noopener noreferrer">decision logs</a> (off by default, and off in our runs). Homemade code has none unless you build it.</li>
<li><strong>The relationships nest,</strong> such as folders inside folders or groups inside groups. Walking a graph is what SpiceDB is built for, and it is awkward to write and to index in SQL. Our model has no nesting, so this test does not cover it.</li>
</ul>
<h2>The hosted options</h2><p>Several companies offer managed authorization services or control planes, some built on these engines and some on others. Some run the decision service for you; others manage policies while evaluation stays in your environment. We did not test any of them, so this is only what their own documentation says they offer:</p>
<ul>
<li><strong>Cerbos Hub</strong> is a control plane for the open source Cerbos PDP. Per its <a href="https://docs.cerbos.dev/cerbos-hub/" rel="noopener noreferrer">docs</a>, it tests, signs and distributes policy bundles to your PDPs, which keep running in your own network, and it offers an embedded PDP compiled to WebAssembly for evaluation in browsers, edge functions and other JavaScript environments.</li>
<li><strong>AuthZed</strong> sells hosted SpiceDB. Per its <a href="https://authzed.com/docs/authzed/guides/picking-a-product" rel="noopener noreferrer">docs</a>, AuthZed Cloud is a self-service platform with usage-based pricing, and AuthZed Dedicated is a private deployment in the cloud provider and regions you choose. For lists too large for LookupResources, its <a href="https://authzed.com/docs/spicedb/modeling/protecting-a-list-endpoint" rel="noopener noreferrer">list guide</a> describes AuthZed Materialize: its Event Streams feature, in early access for Dedicated customers at the time of writing, lets your service keep a local table of permissions so that filtering becomes a JOIN.</li>
<li><strong>Oso Cloud</strong> is Oso's authorization service, with policies written in the Polar language. Its <a href="https://www.osohq.com/docs/authorization-data/local-authorization" rel="noopener noreferrer">local authorization</a> mode returns SQL, including conditions that filter a list query, which your application runs against its own PostgreSQL or MySQL database; Oso does not read that data.</li>
<li><strong>Permit.io</strong> offers a managed cloud PDP, or a PDP container that you run next to your services and that evaluates policies with OPA or Cedar, per its <a href="https://docs.permit.io/" rel="noopener noreferrer">docs</a>. It lists RBAC, ABAC and ReBAC models, and bulk checks for lists and tables.</li>
<li><strong>WorkOS FGA</strong> extends WorkOS RBAC with roles scoped to resources in a parent-child hierarchy, per its <a href="https://workos.com/docs/fga" rel="noopener noreferrer">docs</a>. Your app registers resources with WorkOS, and the API checks access and lists the resources a user can access.</li>
</ul>
<p>The trade we measured carries over to each of them: wherever the rule lives, it has to meet the data it needs. Either the data travels with every request, or someone keeps a copy in sync, or, as Oso's local authorization does, the rule becomes a query that runs where the data already is.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>Production latency.</strong> Everything ran on one Raspberry Pi 4, at concurrency 1, with the app database in the same process. Neither the milliseconds nor the ratios predict a production system; they show how the designs differ under the same conditions.</li>
<li><strong>Tuned servers.</strong> We did not tune either engine. Both ran with their defaults, except that we turned off telemetry and SpiceDB's metrics endpoint and set SpiceDB's log level to <code>warn</code>. Cerbos, at its default <code>info</code> level, writes two log lines per request; we did not measure what that costs it. Everything, servers included, ran at the lowest CPU priority (<code>nice -n 19</code>) on a Pi shared with other work, and each timed run first waited for a quiet machine: a 1-minute load average under 1.25 for the step 4 runs and under 1.5 for the per-step runs, giving up after 10 and 15 minutes. Each result file records the load average it started at.</li>
<li><strong>Throughput.</strong> We measured one request at a time. We did not measure how many checks per second each design sustains.</li>
<li><strong>Deep hierarchies.</strong> Our model has no nested folders or groups, which is where a relationship graph should do best and recursive SQL is hardest. This test does not cover it.</li>
<li><strong>Why the exclusion costs SpiceDB so much on a cold list.</strong> We measured that it does, not where the time goes inside SpiceDB.</li>
<li><strong>The cause of the one failed SpiceDB call.</strong> The 30-second connection age is our best explanation; we did not confirm it.</li>
</ul>
<h2>Summary</h2><p>We set out to show that a single check is cheap in all three designs and that listing and the late rule separate them. By the rules we wrote in <code>CLAIM.md</code> before measuring, that claim is wrong, because its first half failed. A single check is cheap only in-process: through either engine it cost a network call, and on this Pi that call alone was about 2.4 ms. The second half held. For the member, a Cerbos plan turned into SQL took 2.7 to 7 times as long as the hand-written SQL, and SpiceDB's LookupResources 18 to 92 times as long, warm to cold. The cache, the page size and our own SQL adapter moved those numbers as much as the engines did. The late rule cost every design 13 to 22 lines and still needed a change in the application, because every rule needs data and the data lives in your app.</p>
<p>If one app owns the rules, keep them in code, keep them in one module, and test them with a table like <code>fixture/expected.csv</code>. When you do move them out, plan the data path first: what each request must carry for Cerbos, what must be kept in sync for SpiceDB, and how your list pages will get their answers.</p>
<p>For deny-by-default routes and a test for every access check, see <a href="https://devops-daily.com/guides/owasp-top-10/01-broken-access-control">A01:2025 Broken Access Control</a> in our OWASP Top 10 guide.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[LiteLLM 1.107.0 Was Built From a Lookalike Fork: What PyPI Provenance Can and Cannot Tell You]]></title>
      <link>https://devops-daily.com/posts/litellm-1-107-0-compromise-pypi-provenance</link>
      <description><![CDATA[The poisoned LiteLLM release came out of the real release pipeline, fed from a fork one hyphen away. We measured what a PyPI attestation would have shown, what the Docker image provenance showed, and how many of the 1,000 most downloaded PyPI projects publish attestations at all: 39%.]]></description>
      <pubDate>Tue, 27 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/litellm-1-107-0-compromise-pypi-provenance</guid>
      <category><![CDATA[Python]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Python]]></category><category><![CDATA[Security]]></category><category><![CDATA[Supply Chain]]></category><category><![CDATA[PyPI]]></category><category><![CDATA[LiteLLM]]></category><category><![CDATA[Provenance]]></category>
      <content:encoded><![CDATA[<p>On October 21, 2026, <code>litellm==1.107.0</code> was on PyPI for about 1 hour 43 minutes, with a credential stealer and a remote-code loader in all seven wheels. Reading <code>github.com/BerriAI/litellm</code> would not have shown you anything: LiteLLM's own release pipeline built the release from a lookalike fork, <code>Berri-AI/litellm</code>, one hyphen away from the real name.</p>
<p>So we asked a narrower question: which record could have told you where the artifact came from? We checked PyPI, the container registries and the 1,000 most downloaded PyPI projects. The short answer:</p>
<ul>
<li>The <strong>Docker images' build provenance</strong> named the fork. It was there to read.</li>
<li><strong>PyPI had nothing to check.</strong> litellm published PEP 740 attestations on 15 releases in March and April 2026 and on none of the 214 releases since. Those 15 name the private release repository, <code>BerriAI/project-releaser</code>, not the source repository. That is the repository whose release workflows the attacker ran, so an attestation on 1.107.0 would most likely have named the same publisher as every legitimate release.</li>
<li>Across the 1,000 most downloaded PyPI projects, <strong>39.0% of latest releases carry an attestation</strong>, and <strong>33.4%</strong> carry one from the repository their own metadata names.</li>
</ul>
<p>This post covers what happened, how to check your environments, lock files, caches and images, what to rotate, the provenance gap with the numbers, and what you can enforce today. Every number we measured comes from a file in the companion repo. Incident facts, such as the timeline, come from the sources we link, and the repo's <code>SOURCES.md</code> lists each one.</p>
<p><a href="https://github.com/The-DevOps-Daily/pypi-provenance-check" rel="noopener noreferrer">The-DevOps-Daily/pypi-provenance-check on GitHub</a></p>
<h2>TLDR</h2><ul>
<li><strong>What shipped:</strong> 7 wheels (<code>cp310-abi3</code>, Linux, macOS, Windows) with <code>cook.pth</code>, <code>cook.py</code> and <code>sitecustomize.py</code>, plus container images under <code>v1.107.0</code>, <code>latest</code>, <code>main-latest</code> and <code>main-stable</code> on GHCR and Docker Hub. Clean versions: 1.106.2, 1.107.0rc1 and 1.107.0rc2.</li>
<li><strong>Check without starting the suspect Python.</strong> The payload runs whenever an interpreter starts with that site-packages on its path, and that includes <code>pip</code>. Our checker runs under <code>python3 -I -S</code> and reads files only. It finds the payload in venvs, lock files, pip and uv caches and image tarballs.</li>
<li><strong>Rotate</strong> everything the host could reach: cloud role credentials, every provider key stored in the LiteLLM proxy database, the master and virtual keys, <code>DATABASE_URL</code>, CI and registry tokens, SSH and TLS keys.</li>
<li><strong>The provenance gap:</strong> a PyPI publish attestation names the workflow that uploaded the file, not the source it built. <code>pypi-attestations verify</code> against <code>BerriAI/litellm</code> fails even on a legitimate litellm release. In the files we sampled from the 396 attested projects in our survey, we found no SLSA provenance attestation, the PyPI type that can record a source repository.</li>
<li><strong>23 projects stopped attesting.</strong> They attested an earlier release and not their latest one. Today, 12 of them upload with <code>uv publish</code>, which uploads attestations but does not create them.</li>
<li><strong>What works today:</strong> a release cooldown (<code>exclude-newer = "7 days"</code> in uv, <code>--uploaded-prior-to P7D</code> in pip 26.1+), hash-locked installs, image digests plus a provenance source check, and publisher pins for packages that attest. As of October 2026, neither pip nor uv verifies attestations at install time.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Python 3.11 or later on the hosts you want to check (the scripts use only the standard library)</li>
<li>Read access to your virtual environments, lock files, pip and uv caches, CI logs and container images</li>
<li>Optional: <code>uv</code>, to run the <code>pypi-attestations</code> verifier, and <code>docker</code> or <code>podman</code>, to export images for scanning</li>
</ul>
<h2>What happened</h2><p>LiteLLM is a Python SDK and proxy that puts many LLM providers behind one OpenAI-style API. The proxy usually stores every upstream provider key in its database, which made it a valuable target. The timeline below comes from the maintainers' <a href="https://docs.litellm.ai/blog/security-incident-oct-21-2026" rel="noopener noreferrer">incident post</a> and their <a href="https://docs.litellm.ai/blog/security-incident-oct-21-2026-attack-chain" rel="noopener noreferrer">attack-chain post</a>, which they call the canonical account.</p>
<table>
<thead>
<tr>
<th>Time (UTC, Oct 21, 2026)</th>
<th>Event</th>
</tr>
</thead>
<tbody><tr>
<td>10:08</td>
<td>Branch protection on the private release repo <code>BerriAI/project-releaser</code> rejects commit <code>9cf97e9</code> ("ci: release from a fork source repository"). A stolen token of a GitHub organization owner disables the protection and force-pushes it.</td>
</tr>
<tr>
<td>10:46:39</td>
<td>A staging run starts with source <code>Berri-AI/litellm</code> at commit <code>20535d8f</code>, which "exists only on the attacker's fork".</td>
</tr>
<tr>
<td>11:11:18</td>
<td>A second stolen maintainer token approves the production gate.</td>
</tr>
<tr>
<td>11:39:51 to 11:40:13</td>
<td>Seven <code>litellm==1.107.0</code> wheels are uploaded to PyPI.</td>
</tr>
<tr>
<td>11:46:56</td>
<td>Images are pushed under <code>v1.107.0</code>, <code>latest</code>, <code>main-latest</code> and <code>main-stable</code>.</td>
</tr>
<tr>
<td>11:53:49 to 11:54:01</td>
<td>A third stolen token deletes the workflow run records.</td>
</tr>
<tr>
<td>12:06:41</td>
<td>Upwind opens <a href="https://github.com/BerriAI/litellm/issues/47826" rel="noopener noreferrer">issue #47826</a>.</td>
</tr>
<tr>
<td>13:23</td>
<td>PyPI removes 1.107.0.</td>
</tr>
<tr>
<td>13:51</td>
<td>The GHCR tags are gone.</td>
</tr>
<tr>
<td>14:20</td>
<td>The Docker Hub tags are removed or repointed to clean images.</td>
</tr>
</tbody></table>
<p>The way in was ordinary. On May 27, a config file with the master key of the LiteLLM Agent Platform was committed to a public repository. The commit was force-pushed over, but the key was not revoked. On October 10, the attacker used it to read a GitHub CLI OAuth token that belonged to an organization owner. More maintainer tokens were taken from repository secrets the same day. A maintainer summed it up in the issue as "use of insufficiently scoped keys combined with failure to revoke after accidental exposure."</p>
<p>The payload sits in <code>litellm-1.107.0.data/purelib/</code>, so the installer puts it in the root of site-packages:</p>
<ul>
<li><code>cook.pth</code> contains <code>import cook</code>, so <code>cook.py</code> runs every time that interpreter starts. We covered this startup trick in detail in our post on the <a href="https://devops-daily.com/posts/shai-hulud-hades-pypi-wave-june-2026">Shai-Hulud Hades wave on PyPI</a>.</li>
<li><code>cook.py</code> is a stealer that calls itself <code>privcollect</code>. It encrypts what it collects and POSTs it to <code>http://www.litellm.dev/healthcheck</code>, a lookalike domain registered on October 13.</li>
<li><code>sitecustomize.py</code> forks a daemon that holds <code>/tmp/remote_code_loader.lock</code>. Every hour, it reads a domain from <code>raw.githubusercontent.com/letters9813/pythonclient-test/main/domain.txt</code> and runs the code at <code>http://&lt;domain&gt;/codepy</code>. The issue reports that the file has been empty since October 20, so this stage was dormant.</li>
</ul>
<p>Early reports said the <code>v1.107.0</code> git tag was clean. The maintainers later wrote that "there is no official v.1.107.0 source tag". The tag list we pulled from <code>BerriAI/litellm</code> on October 26 has <code>v1.107.0-dev.1</code> to <code>-dev.3</code> and <code>v1.107.0-rc.1</code> and <code>-rc.2</code>, and no <code>v1.107.0</code>. Either way, the malicious lines were never in <code>BerriAI/litellm</code>. The lookalike repository now returns 404.</p>
<p>This is the second LiteLLM poisoning this year. In March, versions 1.82.7 and 1.82.8 were published with a stolen PyPI token, which <a href="https://www.chainguard.dev/unchained/litellm-attacked-again-malicious-1-107-0-wheels-on-pypi-and-docker-hub-october-2026" rel="noopener noreferrer">Chainguard</a> attributes to TeamPCP, the group in our <a href="https://devops-daily.com/posts/mini-shai-hulud-pytorch-lightning-supply-chain-attack">PyTorch Lightning post</a>. There is no evidence that links the two incidents.</p>
<h2>Check if you were hit</h2><p>You are exposed if any of these happened:</p>
<ul>
<li>A <code>pip install</code> or <code>uv</code> resolve picked <code>litellm==1.107.0</code> between 11:39:51 and 13:23 UTC. This includes unpinned <code>litellm</code> and <code>litellm[proxy]</code> requirements, and lock files that were refreshed in that window.</li>
<li>You ran an image pulled by one of the four tags between 11:46:56 and 13:51 UTC (GHCR) or 14:20 UTC (Docker Hub), or you built an image <code>FROM</code> one of those tags in that window.</li>
</ul>
<p>Pulling or unpacking an image does not start Python, so it does not run the payload. Any start of the image's Python with normal site initialization does: a container whose entrypoint runs it, a <code>RUN</code> step in a derived build that runs it (<code>pip install</code> counts), or <code>docker exec ... python</code>.</p>
<p>The most important rule comes from the issue: <strong>do not start with <code>pip install</code>.</strong> Any Python process that starts in the affected environment runs <code>cook.pth</code> and <code>sitecustomize.py</code> again, and pip is a Python process.</p>
<h3>With nothing but a shell</h3><pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># The payload files in any site-packages or dist-packages on this host (all mounted filesystems)</span>
find / \( -path /proc -o -path /sys \) -prune -o \
  \( -path <span class="hljs-string">'*/site-packages/*'</span> -o -path <span class="hljs-string">'*/dist-packages/*'</span> \) \
  \( -name <span class="hljs-string">'cook.pth'</span> -o -name <span class="hljs-string">'cook.py'</span> -o -name <span class="hljs-string">'litellm-1.107.0.dist-info'</span> \) -<span class="hljs-built_in">print</span> 2&gt;/dev/null

<span class="hljs-comment"># Host markers. The first one is written only after a successful exfiltration POST.</span>
<span class="hljs-built_in">ls</span> -la /tmp/.pc_once_72129118c59f5636c0bb /tmp/remote_code_loader.lock /tmp/.pc_sq_* /tmp/.pc_e_* 2&gt;/dev/null

<span class="hljs-comment"># Literal 1.107.0 pins in requirements-style files (skips rc, dev, post and local versions)</span>
grep -rniE <span class="hljs-string">'^[[:space:]]*litellm(\[[^]]*\])?[[:space:]]*===?[[:space:]]*1\.107\.0([[:space:];,#\\]|$)'</span> \
  --include=<span class="hljs-string">'*.txt'</span> --include=<span class="hljs-string">'*.in'</span> --include=<span class="hljs-string">'*.lock'</span> .
</code></pre><p>A missing marker file proves nothing. Containers and <code>/tmp</code> cleanup remove it. The <code>find</code> runs without <code>-xdev</code> so it covers separately mounted <code>/home</code>, <code>/srv</code> and volume paths; errors are hidden, so run it as root. The <code>grep</code> only sees the literal spelling <code>1.107.0</code>.</p>
<h3>With the checker</h3><p><code>ioc_check.py</code> is one standard-library file. Run it with <code>-I -S</code> so that the interpreter running it does not process <code>.pth</code> files or import <code>sitecustomize</code> from its own site-packages:</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/pypi-provenance-check
<span class="hljs-built_in">cd</span> pypi-provenance-check

python3 -I -S ioc_check.py                                    <span class="hljs-comment"># envs, pip/uv caches, host markers</span>
python3 -I -S ioc_check.py lock requirements.txt uv.lock pylock.toml
docker save litellm/litellm:main-stable -o img.tar &amp;&amp; python3 -I -S ioc_check.py scan img.tar
</code></pre><p>With no arguments, it searches <code>/usr/lib</code>, <code>/usr/local/lib</code>, <code>/opt</code>, <code>/srv</code>, <code>/app</code> and your home directory for site-packages, scans the pip, uv, Poetry and PDM caches, and checks the host markers. On Linux it also looks for processes with <code>PRIVCOLLECT_ACTIVE=1</code> in their environment or holding the loader's lock file. Run it as root. As a normal user, a directory, file or archive it cannot read is reported as incomplete, so a run that finds nothing else exits 4 instead of 0. Processes it cannot inspect are only listed as a note.</p>
<p>We did not download or run the real wheel. The tests use a harmless stand-in built at test time: its <code>cook.py</code> and <code>sitecustomize.py</code> are comments, and its <code>cook.pth</code> is <code>import cook</code>. That 12-byte file has the same sha256 as the real <code>cook.pth</code> from the wheel's RECORD, which confirms the published hash. Here is the checker on a venv with the stand-in installed:</p>
<pre><code class="hljs language-text" tabindex="0">$ python3 -I -S ioc_check.py env fx/infected-venv
HIT        installed        fx/infected-venv/lib/python3.12/site-packages/litellm-1.107.0.dist-info
                           litellm 1.107.0 is installed
HIT        payload-file     fx/infected-venv/lib/python3.12/site-packages/cook.pth
                           cook.pth in site-packages root (sha256 matches the published payload); owner: litellm-1.107.0
HIT        payload-file     fx/infected-venv/lib/python3.12/site-packages/cook.py
                           cook.py in site-packages root; owner: litellm-1.107.0
HIT        payload-file     fx/infected-venv/lib/python3.12/site-packages/sitecustomize.py
                           sitecustomize.py installed by litellm-1.107.0
INFO       searched         fx/infected-venv
                           1 site-packages directories checked

INDICATORS FOUND: treat this host as compromised and rotate what it could reach (exit 1)
</code></pre><p>Exit codes: <code>0</code> nothing found, <code>1</code> an indicator of the payload, <code>3</code> something suspicious that is not the payload, <code>4</code> incomplete, because a path, directory or archive could not be read. The <code>lock</code> check also catches equivalent spellings of the version, such as <code>==1.107</code> and <code>==1.107.0.0</code>.</p>
<p>A <code>sitecustomize.py</code> or a <code>.pth</code> file with <code>import</code> lines is suspicious only when nothing owns it, so the checker reads every <code>RECORD</code> in the site-packages directory. That was not enough on our Debian 12 based test machine: <code>/usr/lib/python3/dist-packages</code> holds <code>distutils-precedence.pth</code> and a <code>logilab_common</code> namespace file, both with <code>import</code> lines, and no <code>RECORD</code> lists them, because Debian packages installed them. The checker also reads dpkg's file lists, so it reports them as installed by <code>python3-setuptools</code> and <code>python3-logilab-common</code> and exits 0 (<code>examples/debian-dist-packages.txt</code>). Without that, any Debian or Ubuntu host with such packages installed would get exit 3.</p>
<h3>pip and uv caches</h3><p>A cache can hold the wheel long after the venv is gone, and a CI cache restores it into the next job. We put the stand-in wheel through uv 0.13.0, pip 23.0.1 and pip 26.2.2 (served from a local TLS server, because pip only caches responses from secure origins) and recorded what each one left behind:</p>
<table>
<thead>
<tr>
<th>Tool</th>
<th>What it left behind (path in the cache)</th>
</tr>
</thead>
<tbody><tr>
<td>uv 0.13.0</td>
<td>the unpacked wheel, <code>archive-v0/&lt;id&gt;/litellm-1.107.0.data/purelib/cook.pth</code>, and an index entry, <code>wheels-v7/index/&lt;hash&gt;/litellm/1.107.0-py3-none-any</code></td>
</tr>
<tr>
<td>pip 23.0.1</td>
<td>one file under <code>http/</code>, the HTTP response with the wheel inside it</td>
</tr>
<tr>
<td>pip 26.2.2</td>
<td><code>http-v2/.../&lt;hash&gt;.body</code>, the raw wheel</td>
</tr>
</tbody></table>
<p><code>scan</code> found the payload in all three caches (<code>examples/cache-layout-demo.txt</code> in the repo). Wheel member names are stored uncompressed in a zip file, so the checker also searches raw bytes for <code>litellm-1.107.0.data/purelib/cook</code>, which catches the older pip format. pip 26's <code>.body</code> file is the wheel byte for byte (same sha256 in our run), so on a real cache the checker also matches it against the seven published wheel hashes.</p>
<h3>CI runners</h3><ul>
<li><strong>Hosted runners</strong> are gone after the job. What remains is the job log. Search the logs from October 21 for <code>litellm-1.107.0</code> (pip prints <code>Downloading litellm-1.107.0-...</code> and <code>Successfully installed ... litellm-1.107.0</code>) and <code>litellm==1.107.0</code> (uv prints <code> + litellm==1.107.0</code>). We checked these lines with pip 23.0.1, pip 26.2.2 and uv 0.13.0 on the stand-in wheel.</li>
<li><strong>Restored dependency caches</strong> can carry the wheel into later jobs. Scan the cache directory, or delete the cache entries created that day.</li>
<li><strong>Persistent and self-hosted runners</strong> are hosts. Run the full check on them.</li>
</ul>
<h3>Images</h3><p>Scan a saved image: <code>docker save</code> or <code>podman save</code> output, an OCI archive, or a flat <code>docker export</code> tar of a container's filesystem. The checker reads gzip layers and reports zstd layers as unreadable. To compare digests, the platform manifests of the bad <code>litellm</code> image are <code>sha256:bd1aeee4...</code> (amd64) and <code>sha256:56b74573...</code> (arm64), under index <code>sha256:1bbcb144...</code>. The checker carries the full list, including the <code>-database</code> and <code>-non_root</code> variants.</p>
<h2>Rotate what</h2><p>The stealer does not need the remote loader: it runs in a background thread as soon as the interpreter starts. The <a href="https://github.com/BerriAI/litellm/issues/47826" rel="noopener noreferrer">issue</a> lists what it collects. Rotate in roughly this order:</p>
<ol>
<li><strong>Clean up first, from the shell.</strong> Delete <code>cook.pth</code>, <code>cook.py</code>, <code>sitecustomize.py</code> and the matching <code>__pycache__/cook*.pyc</code> and <code>sitecustomize*.pyc</code> from every affected site-packages. Kill the detached Python daemon that holds <code>/tmp/remote_code_loader.lock</code>. Then rebuild the venv or image from scratch with <code>litellm==1.106.2</code>.</li>
<li><strong>Cloud credentials.</strong> The stealer reads AWS, Azure and gcloud files and instance metadata (AWS IMDSv2, ECS task credentials, GCP and Azure metadata), and validates AWS credentials with STS <code>GetCallerIdentity</code>. Rotate the instance or task role, then search CloudTrail for <code>GetCallerIdentity</code> from that role around install time and for use of its credentials from outside your network.</li>
<li><strong>The LiteLLM proxy itself.</strong> If <code>DATABASE_URL</code> is set, the stealer dumps <code>LiteLLM_CredentialsTable</code> and <code>LiteLLM_ProxyModelTable</code>, which hold every upstream provider key. Rotate all of them, the master key, the virtual keys and the database password.</li>
<li><strong>Build and publish credentials.</strong> Kubernetes service account tokens and kubeconfigs, <code>/run/secrets</code>, CI secrets (GitLab Runner, Jenkins), Vault and Terraform credentials, <code>.git-credentials</code>, <code>.npmrc</code>, <code>.pypirc</code>, Maven, Gradle, Cargo, RubyGems and Composer tokens, <code>.env</code> files.</li>
<li><strong>Keys and the rest.</strong> SSH, GPG and TLS private keys (including Let's Encrypt), Hugging Face tokens, AI coding tool configs (Codex, Gemini, Cursor, Windsurf, VS Code MCP settings and others), browser-saved passwords and cookies, and anything in the environment of any process on the host. The stealer reads <code>/proc/*/environ</code> for every process it can.</li>
</ol>
<p>If 1.107.0 ran on a host with outbound internet access, assume the exfiltration succeeded.</p>
<h2>The provenance gap</h2><p>The release passed a source review because it was not built from the reviewed source. The question is which record names the source an artifact was actually built from.</p>
<ol>
<li><strong>BerriAI/litellm</strong> the repo you review</li>
<li><strong>Berri-AI/litellm</strong> lookalike fork</li>
<li><strong>BerriAI/project-releaser</strong> private release workflows</li>
<li><strong>PyPI wheels</strong> publish attestation</li>
<li><strong>Container images</strong> BuildKit provenance</li>
</ol>
<p>Connections:</p>
<ul>
<li>BerriAI/litellm -&gt; BerriAI/project-releaser (normal release)</li>
<li>Berri-AI/litellm -&gt; BerriAI/project-releaser (Oct 21)</li>
<li>BerriAI/project-releaser -&gt; PyPI wheels (upload)</li>
<li>BerriAI/project-releaser -&gt; Container images (push)</li>
</ul>
<h3>What PyPI records</h3><p>A PEP 740 publish attestation is a Sigstore signature made with the identity of the CI workflow that uploads the file: repository, workflow file and environment. Trusted Publishing accepts an upload only from a publisher registered for the project on PyPI. PyPI then <a href="https://github.com/pypi/warehouse/blob/main/warehouse/attestations/services.py" rel="noopener noreferrer">verifies each attestation</a> against the identity of the publisher that authenticated the upload, and records that publisher in the file's provenance. So the attestation answers "which workflow uploaded this file". It does not say which source that workflow built.</p>
<p>We pulled litellm's whole history from PyPI's Simple API and read the attestations through the <a href="https://docs.pypi.org/api/integrity/" rel="noopener noreferrer">Integrity API</a>:</p>
<p><strong>litellm on PyPI, checked 2026-10-26 23:17 UTC</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ python3 -I scripts/litellm_history.py
litellm: 956 releases on PyPI, checked 2026-10-26T23:17:57+00:00
  no attest.  727 releases  0.1.0 (2023-07-27) .. 1.82.6 (2026-03-22)
  attested     15 releases  1.83.0 (2026-03-31) .. 1.83.14 (2026-04-26)
  no attest.  214 releases  1.84.0.dev1 (2026-04-29) .. 1.106.4 (2026-10-25)
  publisher on 15 of 15 attested releases (1.83.0 .. 1.83.14): GitHub repository=BerriAI/project-releaser workflow=publish-litellm.yml environment=pypi-publish
  1.107.0: not on PyPI (removed on 21 Oct 2026)
wrote results/litellm-attestation-history-20261026T231757Z.json
<span class="hljs-comment"># verify the last attested wheel against the repo litellm's metadata names, then against its publisher</span>
$ pypi-attestations verify pypi --repository https://github.com/BerriAI/litellm pypi:litellm-1.83.14-py3-none-any.whl
Verification failed: provenance was signed by repository <span class="hljs-string">"BerriAI/project-releaser"</span>, expected <span class="hljs-string">"BerriAI/litellm"</span>
$ pypi-attestations verify pypi --repository https://github.com/BerriAI/project-releaser pypi:litellm-1.83.14-py3-none-any.whl
OK: litellm-1.83.14-py3-none-any.whl
</code></pre><p>Three things follow from that output.</p>
<ol>
<li><strong>There was nothing to check on PyPI.</strong> 1.107.0 is gone from PyPI now (the JSON and Integrity APIs return 404), so we cannot read its record directly. The reporters wrote that litellm "has never published attestations". That is not quite right, because 15 releases had them. But 1.106.2, which the same pipeline built and uploaded an hour before 1.107.0, has none, and neither do the 213 other releases since April 29. The attested run lasted four weeks. It began with 1.83.0 on March 31, after the March compromise, when the maintainers moved releases to the private repo, and ended with 1.83.14 on April 26.</li>
<li><strong>A check against the source repository fails on the good releases too.</strong> The last legitimate attested wheel fails <code>--repository https://github.com/BerriAI/litellm</code>, the repository that litellm's metadata names. A policy that compares the attestation with the metadata would have flagged every attested litellm release and taught everyone to ignore it.</li>
<li><strong>A check against the real publisher would most likely have passed 1.107.0.</strong> The maintainers write that the compromised workflow "signed and published the poisoned artifacts", and an upload through Trusted Publishing can only come from a publisher registered for the project. So an attestation on 1.107.0 would have named a registered publisher. The one on all 15 attested releases is <code>BerriAI/project-releaser</code>, <code>publish-litellm.yml</code>, <code>pypi-publish</code>. If 1.107.0 went out through that identity, a check pinned to it would have passed. If it went out through another registered identity, a pin on the full identity would have flagged the change. A check on the repository alone would have flagged it only if that identity used a different repository. The signing certificate also records the release repository's commit, but that repository is private, so nobody outside could have told a good commit from a bad one.</li>
</ol>
<p>This is an inference: we do not have the 1.107.0 upload record. But the publish attestation has no field for the repository that a workflow checked out. PyPI does support a second attestation type, <a href="https://docs.pypi.org/attestations/" rel="noopener noreferrer">SLSA provenance</a>, which can record a source. We come back to it in the survey.</p>
<h3>What the image recorded</h3><p>BuildKit, the builder behind <code>docker buildx</code>, can attach a SLSA provenance document to an image. In it, <code>vcs:source</code> is the git remote of the build context, so it names the repository that the build checked out. We read it for the clean v1.106.2 images from the registry API on October 26 at 23:08 UTC, without pulling a layer (excerpt):</p>
<pre><code class="hljs language-text" tabindex="0">$ python3 -I scripts/image_provenance.py ghcr.io/berriai/litellm:v1.106.2 ... ghcr.io/berriai/litellm:v1.107.0 ...
ghcr.io/berriai/litellm:v1.106.2  index sha256:7294b90f868c...
  linux/arm64  vcs:source   https://github.com/BerriAI/litellm
               vcs:revision 49943cd52ea46aacf8db2e5cb2b890d580fe9bb4
               builder      https://github.com/BerriAI/project-releaser/actions/runs/39574962801/attempts/1
               workflow     BerriAI/project-releaser/.github/workflows/stage.yml@refs/heads/main (d687b69)
               inputs       allow_republish=false, dry_run=false, git_ref=49943cd52ea46aacf8db2e5cb2b890d580fe9bb4, release_type=stable, repo=litellm
  ...
ghcr.io/berriai/litellm:v1.107.0  HTTP 404
ghcr.io/berriai/litellm@sha256:bd1aeee419ea36da9aa2b657288da4101887be468f020ac63d22ba6ed84a3ea5  HTTP 404
</code></pre><p>Commit <code>49943cd</code> is the v1.106.2 tag. The same source, revision and workflow appear on every v1.106.2 image we read: three variants on GHCR with two platforms each, and Docker Hub's <code>litellm/litellm:v1.106.2</code>, which has the same index digest as GHCR's. The 1.107.0 tag and the bad platform digests return 404 on GHCR. The 1.107.0 images are deleted, so we could not read theirs. Upwind and <a href="https://socket.dev/blog/litellm-compromise" rel="noopener noreferrer">Socket</a> read them before removal and recorded <code>vcs:source https://github.com/Berri-AI/litellm</code> at <code>20535d8f</code>, with a dispatch input that set <code>source_owner</code> to <code>Berri-AI</code>. With Docker, <code>docker buildx imagetools inspect &lt;image&gt; --format '{{ json .Provenance.SLSA }}'</code> shows the same document.</p>
<p>This record named the fork, and the reason is a little uncomfortable. BuildKit writes <code>vcs:source</code> inside the build, and the attacker controlled the build. Nothing outside the build vouches for the field: the maintainers say the same release workflow signed the images, and it signed the bad ones too. It named the fork because the build left the remote URL as it was. It is good evidence, but it is not a guarantee.</p>
<h3>How many PyPI projects publish an attestation at all</h3><p>A provenance check needs a record to read. We surveyed the latest release of the 1,000 most downloaded projects (30-day downloads to October 1, 2026, from <a href="https://github.com/hugovk/top-pypi-packages" rel="noopener noreferrer">hugovk/top-pypi-packages</a>, saved in the repo so the set is fixed) plus a list of 44 AI-stack projects. For each project, the survey records how many files of the latest release have provenance, and who published them. The run started on October 26 at 22:36 UTC: 1,022 projects, 2,925 requests, no errors.</p>
<p><strong>Latest PyPI releases with an attestation</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>repo match</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Top 1-100</td>
<td>60%</td>
<td>56%</td>
<td>Top</td>
</tr>
<tr>
<td>Top 101-500</td>
<td>40.5%</td>
<td>35%</td>
<td>Top</td>
</tr>
<tr>
<td>Top 501-1000</td>
<td>33.6%</td>
<td>27.6%</td>
<td>Top</td>
</tr>
<tr>
<td>All top 1,000</td>
<td>39%</td>
<td>33.4%</td>
<td>All</td>
</tr>
<tr>
<td>AI-stack list (44)</td>
<td>29.5%</td>
<td>25%</td>
<td>AI</td>
</tr>
</tbody></table>
<p><em>Bar: latest release has an attestation on at least one file. Tick: attested by the repository the release's metadata names, counting 5 renamed or transferred repositories as matches. Top projects by 30-day downloads to Oct 1, 2026; surveyed Oct 26, 2026, 22:36 UTC. Source: results/summary-20261026T223625Z.md in the demo repo.</em></p>
<p>The 390 attested top-1,000 projects break down like this:</p>
<table>
<thead>
<tr>
<th>Attested by</th>
<th>Projects</th>
</tr>
</thead>
<tbody><tr>
<td>The repository the metadata names</td>
<td>329</td>
</tr>
<tr>
<td>The same repository after a rename or transfer (filelock, pendulum, dulwich, cramjam, dbt-extractor)</td>
<td>5</td>
</tr>
<tr>
<td>A different public repository (numpy, scipy, scikit-learn and matplotlib publish from <code>*-release</code> repos)</td>
<td>5</td>
</tr>
<tr>
<td>A repository that is not publicly visible (mlflow, spaCy and seven other Explosion packages, semgrep, modal, dbt-protos)</td>
<td>14</td>
</tr>
<tr>
<td>A GitHub repository, but the metadata names none (numba, selenium, thrift and 4 more)</td>
<td>7</td>
</tr>
<tr>
<td>A Google Cloud service account, with no repository (30 Google client libraries)</td>
<td>30</td>
</tr>
<tr>
<td>No attestation on the latest release</td>
<td>610</td>
</tr>
</tbody></table>
<p>Some limits on these numbers. A project counts as attested if any file of its latest release is attested (3 were partial, such as sqlalchemy with 64 of 72 files). Restricting to the 864 projects whose latest release was uploaded on or after November 14, 2024, the day PyPI announced attestations, gives 45.0% attested and 38.5% repository matches. To check the method, a second script asked the Integrity API file by file (up to 80 files per release) about 40 randomly chosen projects and the 23 projects described below. All 63 agreed with the survey.</p>
<p>Three findings stand out.</p>
<p><strong>The publisher is often not the source repository, on purpose.</strong> numpy, scipy, scikit-learn, matplotlib, mlflow and spaCy publish from a separate release repository, like litellm. That can be a reasonable design: it keeps the publishing rights in a repository that fewer people can change. But it means a "publisher equals metadata repository" rule fails for some of the most installed packages in Python. What you can pin is the publisher that you observe.</p>
<p><strong>No sampled file carries the attestation that records a source.</strong> We read up to three attested files per project. For all 396 attested projects in the survey (390 from the top 1,000 and 6 more from the AI list), every attestation in those files is a PyPI publish attestation (<code>https://docs.pypi.org/attestations/publish/v1</code>). None is the SLSA provenance type, which PyPI describes as recording "a package's original source location".</p>
<p><strong>23 projects stopped attesting.</strong> They attested an earlier release and not their latest one: litellm, fastapi, typer, sqlmodel, langchain-core, fastmcp, langfuse, watchfiles, databricks-sdk, the Supabase client family and others. We read their publish workflows as they are today. 12 of them upload with <code>uv publish</code>. The <a href="https://docs.astral.sh/uv/guides/package/" rel="noopener noreferrer">uv docs</a> say it "does not currently generate attestations; attestations must be created separately before publishing". 2 (langchain-core and dspy) set <code>attestations: false</code> on the PyPA publish action. For 7, we found no upload step in a public workflow, and 2 (databricks-sdk and rich-toolkit) name no GitHub repository in their metadata, so we read nothing. Several packages share one repository (typer and typer-slim, httpx2 and httpcore2, the Supabase family), so this is evidence about repositories more than about each package. This shows how those projects publish now, not why they stopped.</p>
<p>In the AI-stack list, 13 of 44 latest releases are attested. openai, mcp, transformers, huggingface-hub, tokenizers, mistralai, ollama, pinecone, pydantic-ai, openai-agents and haystack-ai are attested by the repository their metadata names. google-genai is attested by a Google service account, and autogen-agentchat by <code>microsoft/autogen</code>, which its metadata does not name. Without an attestation: anthropic, langchain, langgraph, llama-index, torch, vllm, tiktoken, chromadb, crewai and 22 more. No attestation does not mean a project publishes carelessly. It means that a provenance check has nothing to read.</p>
<p>The same pattern holds inside one dependency tree. We locked <code>litellm==1.106.2</code> with hashes for Linux x86_64 and Python 3.12 and ran the per-file check on the 58 packages, which reads the provenance of every hashed file (about 3 minutes):</p>
<pre><code class="hljs language-text" tabindex="0">$ python3 -I -S provenance_check.py --lock examples/litellm-1.106.2-lock/requirements.txt    (excerpt, 5 of 58 package lines)
aiohttp==3.14.5                      attested-match       GitHub repository=aio-libs/aiohttp workflow=ci-cd.yml environment=pypi
...
filelock==3.32.7                     attested-other-repo  GitHub repository=tox-dev/filelock workflow=release.yaml environment=release  (metadata: github.com/tox-dev/py-filelock)
...
litellm==1.106.2                     no-attestation
openai==2.54.0                       attested-match       GitHub repository=openai/openai-python workflow=create-releases.yml environment=publish
pydantic-core==2.50.1                no-attestation
...

attested-match: 33, attested-other-repo: 1, no-attestation: 24
</code></pre><p>The one <code>attested-other-repo</code> is filelock, whose repository was renamed. The per-file checker does not resolve renames; the survey does.</p>
<h2>What you can enforce today</h2><p><strong>1. A release cooldown.</strong> 1.107.0 was on PyPI for about 1 hour 43 minutes. Any rule that ignores files younger than two hours would have skipped it. Malware scanners flagged it within that window: Socket reports 7 minutes after upload, Chainguard 16. A cooldown gives detection time to work.</p>
<p><strong>Skip releases younger than 7 days</strong></p>
<p><strong>uv (pyproject.toml)</strong></p>
<pre><code class="hljs language-toml" tabindex="0"><span class="hljs-section">[tool.uv]</span>
<span class="hljs-attr">exclude-newer</span> = <span class="hljs-string">"7 days"</span>

<span class="hljs-comment"># let a security fix through on purpose</span>
<span class="hljs-attr">exclude-newer-package</span> = { cryptography = <span class="hljs-literal">false</span> }
</code></pre><p><strong>uv pip</strong></p>
<pre><code class="hljs language-bash" tabindex="0">uv pip install --exclude-newer <span class="hljs-string">"7 days"</span> -r requirements.txt
</code></pre><p><strong>pip 26.1+</strong></p>
<pre><code class="hljs language-bash" tabindex="0">pip install --uploaded-prior-to P7D -r requirements.txt
</code></pre><p>uv compares each file's upload time and accepts friendly durations (<code>24 hours</code>, <code>1 week</code>, <code>30 days</code>) and ISO 8601 durations (<code>P7D</code>). pip added <code>--uploaded-prior-to</code> in 26.0 and the <code>P3D</code> day form in 26.1. It works only with indexes that publish upload times, and PyPI does. The cost is real: a cooldown delays security fixes by the same amount, so exempt the packages you patch urgently.</p>
<p><strong>2. Hash-locked installs, relocked on purpose.</strong> A <code>uv.lock</code>, a <code>pylock.toml</code> or a <code>pip-compile --generate-hashes</code> file made before 11:39 UTC on October 21 kept you on 1.106.2 or earlier. Install with <code>uv sync --locked</code> or <code>pip install --require-hashes</code>, and treat a lock refresh as a change to review.</p>
<p><strong>3. Image digests plus a source check.</strong> Pin images by digest, and before you promote a new digest, check that its BuildKit provenance names the repository you expect. A digest pin alone would have pinned the bad image if you had updated it during the window. Remember that the source field is written by the build, so it can be forged.</p>
<p><strong>4. Publisher pins for packages that attest.</strong> Verify against the publisher you observed, not the repository in the metadata:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># One file, cryptographically, with the PyPA verifier (pypi-attestations 0.0.30)</span>
uvx --from pypi-attestations==0.0.30 pypi-attestations verify pypi \
  --repository https://github.com/psf/requests pypi:requests-2.34.2-py3-none-any.whl

<span class="hljs-comment"># Every installed or locked file: record publishers once, alert when one changes or disappears</span>
python3 -I -S provenance_check.py --<span class="hljs-built_in">env</span> .venv --write-pins pins.json
python3 -I -S provenance_check.py --lock uv.lock --pins pins.json
</code></pre><p><code>provenance_check.py</code> reads PyPI's record of the attestation and does not repeat the Sigstore verification. PyPI checks the signature at upload, and <code>pypi-attestations</code> checks it again on your side. For a lock, every hash must match a file of the release. For an installed wheel, the checker matches the wheel tags and compares the installed <code>METADATA</code> with the sha256 that PyPI publishes for that wheel's metadata (pip and uv both install it unchanged). That ties the install to a PyPI wheel's metadata, not to every installed byte. Be honest about what a pin buys you. Pinned in April, litellm's publisher would have "disappeared" on every release from April 29 on. That is 206 alerts before the one that mattered, and that is how checks like this get turned off.</p>
<p><strong>What pip and uv do not do.</strong> As of October 2026, neither installer verifies PEP 740 attestations. The pip changelog up to 26.2.2 (October 20) and the uv changelog up to 0.13.5 (October 21) list no such option. pip checks hashes (<code>--require-hashes</code>) and upload times. uv checks hashes and upload times, and <code>uv publish</code> uploads attestation files that another step created.</p>
<p><strong>5. If you maintain a package.</strong> Keep attestations on. The PyPA publish action creates them by default when you use Trusted Publishing. If you switch to <code>uv publish</code>, add a step that creates them: uv's <a href="https://docs.astral.sh/uv/guides/integration/github/" rel="noopener noreferrer">GitHub Actions guide</a> runs <code>astral-sh/attest-action</code> before <code>uv publish</code>. If you publish from a separate release repository, say so in your README, so that users can pin the right publisher.</p>
<p><strong>6. If you run the LiteLLM proxy.</strong> In the issue thread, a LiteLLM maintainer called default-deny egress "a very good way to run LiteLLM". A proxy that can reach only its model providers could not have sent anything to <code>www.litellm.dev</code> or fetched a second stage.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>We could not see 1.107.0 on PyPI.</strong> It was removed before we started, and we found no archived copy of its JSON or provenance. The statement that it had no attestation comes from the reporters and from its neighbors: 1.106.2, from the same pipeline an hour earlier, has none.</li>
<li><strong>What a 1.107.0 attestation would have said is an inference</strong> from the Trusted Publishing rules and the 15 attested releases. We do not know which workflow file uploaded 1.107.0.</li>
<li><strong>We did not read the 1.107.0 image provenance ourselves.</strong> The images return 404 by tag and by digest. <code>Berri-AI/litellm</code> at <code>20535d8f</code> comes from Upwind and Socket.</li>
<li><strong>We do not know why litellm and the 22 other projects stopped attesting.</strong> PyPI's JSON API does not say which client uploaded a file, and the workflows we read are today's versions.</li>
<li><strong>The survey counts projects, not installs.</strong> It looks at the latest release only, and your pins may point to older releases with different records.</li>
<li><strong>We did not test forging BuildKit provenance.</strong> We say it can be forged because whoever runs the build writes it, not because we did it.</li>
<li><strong>Nothing here catches an attacker who runs your real pipeline on your real source.</strong> That is what happened to <a href="https://devops-daily.com/posts/tanstack-npm-worm-dead-mans-switch">TanStack on npm</a>, where the malicious versions carried valid provenance.</li>
</ul>
<h2>Summary</h2><ul>
<li>LiteLLM 1.107.0 came from the real release pipeline, fed from a lookalike fork. Reviewing <code>BerriAI/litellm</code> could not have caught it.</li>
<li>Check without starting the suspect Python: <code>python3 -I -S ioc_check.py</code> covers venvs, lock files, pip and uv caches, image tarballs and host markers. Then rotate everything the host could reach, starting with cloud roles and the proxy's provider keys.</li>
<li>The Docker image's BuildKit provenance named the fork. PyPI had no attestation to check, and a PyPI publish attestation names the uploading workflow, not the source it built.</li>
<li>39.0% of the top 1,000 PyPI projects attest their latest release. 33.4% are attested by the repository their metadata names. 23 projects in our set stopped attesting, and we found no SLSA provenance attestation on any of the 396 attested projects.</li>
<li>What works today is a cooldown, hash locks, digest pins with a source check, and publisher pins where they exist. As of October 2026, pip and uv do not verify attestations for you.</li>
</ul>
<p>To attest a build and check it against a pinned repository and workflow with <code>gh attestation verify</code>, read <a href="https://devops-daily.com/guides/owasp-top-10/08-software-integrity-failures#h3-build-provenance">A08:2025 Software or Data Integrity Failures</a> in our OWASP Top 10 guide.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Why Your GitHub Actions Bill Keeps Growing, and What Each Fix Saves]]></title>
      <link>https://devops-daily.com/posts/github-actions-bill-keeps-growing</link>
      <description><![CDATA[Per-job rounding, macOS minutes at ten times the Linux price, re-runs and storage add up quietly. We measured where one repository's minutes went in 30 days, then ran a mobile pull request pipeline 34 times on GitHub-hosted runners and on Bitrise Build Hub to see what faster machines change in time and money.]]></description>
      <pubDate>Mon, 26 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/github-actions-bill-keeps-growing</guid>
      <category><![CDATA[CI/CD]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[CI/CD]]></category><category><![CDATA[GitHub Actions]]></category><category><![CDATA[FinOps]]></category><category><![CDATA[iOS]]></category><category><![CDATA[Android]]></category><category><![CDATA[Benchmarks]]></category><category><![CDATA[Bitrise]]></category>
      <content:encoded><![CDATA[<p>A GitHub Actions bill rarely grows because of one decision. It grows a minute at a time: every job is rounded up to a whole minute, every matrix leg pays for its own checkout and setup, macOS minutes cost about ten times as much as Linux minutes, and artifacts stay in storage for 90 days unless you say otherwise. For a mobile team, the most expensive job is often the iOS build. In the pipeline we tested, it was a median 57% of GitHub's cost per run, although the Android build took longer.</p>
<p>We measured both halves of the problem. First, a read-only script that shows where one repository's minutes went in the last 30 days, and the free changes that act on what it finds. Then the runner question. We ran one mobile pull request pipeline (the iOS build and unit tests, the Android build and the Android unit tests, all started together) 17 times on GitHub-hosted runners and 17 times on <a href="https://bitrise.io/platform/build-hub" rel="noopener noreferrer">Bitrise Build Hub</a>, which runs GitHub Actions jobs on its own M4 Macs and AMD Linux machines.</p>
<p><strong>About the same cost at list price, less than half the wait.</strong> With the Build Hub Linux pool allowed to run two machines at once, the pipeline finished in a median 18.5 minutes on Build Hub over 7 runs, against 43.6 minutes on GitHub over 17 runs. At list price, a pipeline run cost a median $1.77 on Build Hub and $1.79 on GitHub, over 17 runs each, and the ranges overlap. The iOS job is where the money moves: it ran 2.7 times faster on a 5 vCPU M4 and cost about a fifth as much per job. The two Android jobs ran about twice as fast on a 16 vCPU Linux machine and cost about twice as much per job.</p>
<h2>TLDR</h2><ul>
<li>Run the audit script in this post on your busiest repository first. On ours, rounding each job up to a whole minute added a quarter to the billed minutes, and re-runs added almost nothing. Your mix will differ, and the script tells you which fix matters.</li>
<li>Free fixes inside GitHub Actions: cancel superseded runs, filter by path inside the job (not at the workflow level, or required checks hang), set <code>timeout-minutes</code>, merge tiny jobs, and set <code>retention-days</code> on artifacts.</li>
<li>macOS jobs are the clearest case for a different runner. The Wikipedia iOS app's build and tests took a median 922 s on GitHub's <code>macos-26</code> (3 vCPU M1) and 337 s on a Build Hub M4 Large (5 vCPU M4): $0.99 against $0.21 per job at list price.</li>
<li>Heavy Linux jobs got faster on a bigger machine, not cheaper. The Thunderbird Android build took 2,604 s on GitHub's 4 vCPU runner and 1,244 s on Build Hub's 16 vCPU Linux XLarge, at $0.53 against $1.10 per job.</li>
<li>Set the machine pool maximum to at least the number of jobs that start together. With a Linux pool maximum of 1, our second Android job waited a median 15 minutes for the first one, and the pipeline took a median 35.3 minutes.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A GitHub repository with some Actions history, and the <a href="https://cli.github.com/" rel="noopener noreferrer">GitHub CLI</a> logged in with read access to it.</li>
<li>Node.js 22 or later for the audit script.</li>
<li>To repeat the runner test: a Bitrise Build Hub workspace, and an organization owner who can install the Build Hub GitHub App.</li>
</ul>
<h2>Where the minutes go</h2><p>GitHub bills hosted runners per job, per minute, at a rate set by the machine. These are the mechanisms that make a bill grow without anyone deciding to spend more. Prices are GitHub's list prices, <a href="https://docs.github.com/en/billing/reference/actions-runner-pricing" rel="noopener noreferrer">checked on 18 October 2026</a>.</p>
<table>
<thead>
<tr>
<th>Mechanism</th>
<th>What it does to the bill</th>
</tr>
</thead>
<tbody><tr>
<td>macOS rate</td>
<td>The standard macOS runner (3 vCPU M1) costs $0.062 a minute. A 2-core Linux runner costs $0.006, so a macOS minute costs about 10 times as much. The 5 vCPU M2 Pro (<code>-xlarge</code>) costs $0.102.</td>
</tr>
<tr>
<td>Per-job rounding</td>
<td>"GitHub rounds the minutes and partial minutes each job uses up to the nearest whole minute." A 20-leg matrix of 35-second jobs bills 20 minutes for 11.7 minutes of work.</td>
</tr>
<tr>
<td>Matrix fan-out</td>
<td>Each leg repeats checkout, toolchain setup and cache restore, and pays its own rounding.</td>
</tr>
<tr>
<td>Larger runners</td>
<td>On Team and Enterprise Cloud plans, they can use the plan's included usage first. After that, every minute is billed. They are never free, even in public repositories.</td>
</tr>
<tr>
<td>Re-runs</td>
<td>"Re-run all jobs" pays for every job again. "Re-run failed jobs" pays only for the failed jobs and the jobs that depend on them.</td>
</tr>
<tr>
<td>Artifact storage</td>
<td>$0.25 per GB-month, accrued hourly, with a 90-day default retention. Deleting an artifact does not refund what it has already accrued.</td>
</tr>
<tr>
<td>Cache</td>
<td>10 GB per repository is free, and entries not used for 7 days are evicted. Since <a href="https://github.blog/changelog/2025-11-20-github-actions-cache-size-can-now-exceed-10-gb-per-repository/" rel="noopener noreferrer">20 November 2025</a> you can pay for more at $0.07 per GB-month. A cache that is too small keeps evicting itself, and every miss is a cold build.</td>
</tr>
<tr>
<td>AI review</td>
<td>Copilot code review <a href="https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/" rel="noopener noreferrer">uses Actions minutes</a> in private repositories since 1 June 2026.</td>
</tr>
<tr>
<td>Queueing</td>
<td>Free, Pro and Team plans run at most 5 macOS jobs at the same time. Waiting does not cost minutes, but it costs the people who wait.</td>
</tr>
</tbody></table>
<p>Flaky tests make re-runs a habit. GitHub's own engineering team <a href="https://github.blog/engineering/engineering-principles/reducing-flaky-builds-by-18x/" rel="noopener noreferrer">reported in 2020</a> that "1 in 11 commits had at least one red build caused by a flaky test". Google <a href="https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html" rel="noopener noreferrer">reported in 2016</a> "about 1.5% of all test runs reporting a 'flaky' result". Every one of those red builds is a full re-run if someone presses the wrong button.</p>
<h2>Measure your own bill</h2><p>The list above says what can happen. Your run history says what did. This script reads the last 30 days of runs and jobs for one repository with <code>gh api</code>, rounds each job the way GitHub does, and prints where the minutes went. It only reads, and it runs fine on a laptop or a Raspberry Pi.</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-meta">#!/usr/bin/env node</span>
<span class="hljs-comment">// Where did this repository's GitHub Actions minutes go? Reads the run and job</span>
<span class="hljs-comment">// history with the gh CLI (read-only) and prints the billed minutes, the</span>
<span class="hljs-comment">// minutes added by per-job rounding, the minutes spent on re-runs, queue time,</span>
<span class="hljs-comment">// and the workflows that bill the most.</span>
<span class="hljs-comment">//</span>
<span class="hljs-comment">//   node audit.mjs owner/repo [days]      # default: the 30 full days before today</span>
<span class="hljs-keyword">import</span> { execFile } <span class="hljs-keyword">from</span> <span class="hljs-string">'node:child_process'</span>;
<span class="hljs-keyword">import</span> { promisify } <span class="hljs-keyword">from</span> <span class="hljs-string">'node:util'</span>;

<span class="hljs-keyword">const</span> [repo, days = <span class="hljs-string">'30'</span>] = process.<span class="hljs-property">argv</span>.<span class="hljs-title function_">slice</span>(<span class="hljs-number">2</span>);
<span class="hljs-keyword">if</span> (!repo) <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Error</span>(<span class="hljs-string">'usage: node audit.mjs owner/repo [days]'</span>);
<span class="hljs-keyword">const</span> run = <span class="hljs-title function_">promisify</span>(execFile);
<span class="hljs-keyword">const</span> <span class="hljs-title function_">gh</span> = <span class="hljs-keyword">async</span> (<span class="hljs-params">path</span>) =&gt;
  <span class="hljs-title class_">JSON</span>.<span class="hljs-title function_">parse</span>(
    (<span class="hljs-keyword">await</span> <span class="hljs-title function_">run</span>(<span class="hljs-string">'gh'</span>, [<span class="hljs-string">'api'</span>, <span class="hljs-string">'--paginate'</span>, <span class="hljs-string">'--slurp'</span>, path], { <span class="hljs-attr">maxBuffer</span>: <span class="hljs-number">1</span> &lt;&lt; <span class="hljs-number">28</span> })).<span class="hljs-property">stdout</span>
  );

<span class="hljs-comment">// One query per day: the runs endpoint returns at most 1,000 runs per search.</span>
<span class="hljs-keyword">const</span> runs = [];
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> d = <span class="hljs-number">1</span>; d &lt;= days; d++) {
  <span class="hljs-keyword">const</span> day = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Date</span>(<span class="hljs-title class_">Date</span>.<span class="hljs-title function_">now</span>() - d * <span class="hljs-number">864e5</span>).<span class="hljs-title function_">toISOString</span>().<span class="hljs-title function_">slice</span>(<span class="hljs-number">0</span>, <span class="hljs-number">10</span>);
  <span class="hljs-keyword">const</span> pages = <span class="hljs-keyword">await</span> <span class="hljs-title function_">gh</span>(<span class="hljs-string">`repos/<span class="hljs-subst">${repo}</span>/actions/runs?created=<span class="hljs-subst">${day}</span>&amp;per_page=100`</span>);
  runs.<span class="hljs-title function_">push</span>(...pages.<span class="hljs-title function_">flatMap</span>(<span class="hljs-function">(<span class="hljs-params">p</span>) =&gt;</span> p.<span class="hljs-property">workflow_runs</span>).<span class="hljs-title function_">filter</span>(<span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">status</span> === <span class="hljs-string">'completed'</span>));
}

<span class="hljs-comment">// Jobs of every attempt (filter=all), eight requests at a time.</span>
<span class="hljs-keyword">const</span> jobs = [];
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> i = <span class="hljs-number">0</span>; i &lt; runs.<span class="hljs-property">length</span>; i += <span class="hljs-number">8</span>) {
  <span class="hljs-keyword">const</span> batch = runs.<span class="hljs-title function_">slice</span>(i, i + <span class="hljs-number">8</span>).<span class="hljs-title function_">map</span>(<span class="hljs-title function_">async</span> (r) =&gt; {
    <span class="hljs-keyword">const</span> js = (
      <span class="hljs-keyword">await</span> <span class="hljs-title function_">gh</span>(<span class="hljs-string">`repos/<span class="hljs-subst">${repo}</span>/actions/runs/<span class="hljs-subst">${r.id}</span>/jobs?filter=all&amp;per_page=100`</span>)
    ).<span class="hljs-title function_">flatMap</span>(<span class="hljs-function">(<span class="hljs-params">p</span>) =&gt;</span> p.<span class="hljs-property">jobs</span>);
    <span class="hljs-keyword">return</span> js.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">j</span>) =&gt;</span> ({ ...j, <span class="hljs-attr">workflow</span>: r.<span class="hljs-property">name</span> }));
  });
  jobs.<span class="hljs-title function_">push</span>(...(<span class="hljs-keyword">await</span> <span class="hljs-title class_">Promise</span>.<span class="hljs-title function_">all</span>(batch)).<span class="hljs-title function_">flat</span>());
}

<span class="hljs-keyword">const</span> <span class="hljs-title function_">min</span> = (<span class="hljs-params">a, b</span>) =&gt; (<span class="hljs-title class_">Date</span>.<span class="hljs-title function_">parse</span>(b) - <span class="hljs-title class_">Date</span>.<span class="hljs-title function_">parse</span>(a)) / <span class="hljs-number">60000</span>;
<span class="hljs-keyword">const</span> ran = jobs.<span class="hljs-title function_">filter</span>(
  <span class="hljs-function">(<span class="hljs-params">j</span>) =&gt;</span>
    j.<span class="hljs-property">started_at</span> &amp;&amp;
    j.<span class="hljs-property">completed_at</span> &amp;&amp;
    j.<span class="hljs-property">conclusion</span> !== <span class="hljs-string">'skipped'</span> &amp;&amp;
    <span class="hljs-title function_">min</span>(j.<span class="hljs-property">started_at</span>, j.<span class="hljs-property">completed_at</span>) &gt; <span class="hljs-number">0</span>
);
<span class="hljs-keyword">const</span> rows = ran.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">j</span>) =&gt;</span> {
  <span class="hljs-keyword">const</span> used = <span class="hljs-title function_">min</span>(j.<span class="hljs-property">started_at</span>, j.<span class="hljs-property">completed_at</span>);
  <span class="hljs-keyword">return</span> {
    <span class="hljs-attr">workflow</span>: j.<span class="hljs-property">workflow</span>,
    used,
    <span class="hljs-attr">billed</span>: <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">ceil</span>(used),
    <span class="hljs-attr">rerun</span>: j.<span class="hljs-property">run_attempt</span> &gt; <span class="hljs-number">1</span>,
    <span class="hljs-attr">queue</span>: <span class="hljs-title function_">min</span>(j.<span class="hljs-property">created_at</span>, j.<span class="hljs-property">started_at</span>) * <span class="hljs-number">60</span>,
    <span class="hljs-attr">os</span>: j.<span class="hljs-property">labels</span>.<span class="hljs-title function_">join</span>(<span class="hljs-string">' '</span>).<span class="hljs-title function_">includes</span>(<span class="hljs-string">'macos'</span>) ? <span class="hljs-string">'macOS'</span> : <span class="hljs-string">'other'</span>,
  };
});

<span class="hljs-keyword">if</span> (rows.<span class="hljs-property">length</span> === <span class="hljs-number">0</span>) {
  <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`<span class="hljs-subst">${repo}</span>: no jobs ran in the last <span class="hljs-subst">${days}</span> days`</span>);
  process.<span class="hljs-title function_">exit</span>(<span class="hljs-number">0</span>);
}

<span class="hljs-keyword">const</span> <span class="hljs-title function_">sum</span> = (<span class="hljs-params">xs, f</span>) =&gt; xs.<span class="hljs-title function_">reduce</span>(<span class="hljs-function">(<span class="hljs-params">s, x</span>) =&gt;</span> s + <span class="hljs-title function_">f</span>(x), <span class="hljs-number">0</span>);
<span class="hljs-keyword">const</span> <span class="hljs-title function_">pct</span> = (<span class="hljs-params">q, xs</span>) =&gt;
  [...xs].<span class="hljs-title function_">sort</span>(<span class="hljs-function">(<span class="hljs-params">a, b</span>) =&gt;</span> a - b)[<span class="hljs-title class_">Math</span>.<span class="hljs-title function_">min</span>(xs.<span class="hljs-property">length</span> - <span class="hljs-number">1</span>, <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">floor</span>(q * xs.<span class="hljs-property">length</span>))];
<span class="hljs-keyword">const</span> queues = rows.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">queue</span>);
<span class="hljs-keyword">const</span> used = <span class="hljs-title function_">sum</span>(rows, <span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">used</span>);
<span class="hljs-keyword">const</span> billed = <span class="hljs-title function_">sum</span>(rows, <span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">billed</span>);
<span class="hljs-keyword">const</span> rerun = <span class="hljs-title function_">sum</span>(
  rows.<span class="hljs-title function_">filter</span>(<span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">rerun</span>),
  <span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">billed</span>
);
<span class="hljs-keyword">const</span> short = rows.<span class="hljs-title function_">filter</span>(<span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">used</span> &lt; <span class="hljs-number">1</span>).<span class="hljs-property">length</span>;

<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`<span class="hljs-subst">${repo}</span>, last <span class="hljs-subst">${days}</span> days: <span class="hljs-subst">${runs.length}</span> runs, <span class="hljs-subst">${rows.length}</span> jobs that ran`</span>);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`minutes used   <span class="hljs-subst">${used.toFixed(<span class="hljs-number">0</span>)}</span>`</span>);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(
  <span class="hljs-string">`minutes billed <span class="hljs-subst">${billed}</span>  (+<span class="hljs-subst">${(billed - used).toFixed(<span class="hljs-number">0</span>)}</span> from rounding each job up, <span class="hljs-subst">${((billed / used - <span class="hljs-number">1</span>) * <span class="hljs-number">100</span>).toFixed(<span class="hljs-number">0</span>)}</span>%)`</span>
);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(
  <span class="hljs-string">`jobs under 1 minute: <span class="hljs-subst">${short}</span> of <span class="hljs-subst">${rows.length}</span> (<span class="hljs-subst">${((short / rows.length) * <span class="hljs-number">100</span>).toFixed(<span class="hljs-number">0</span>)}</span>%), each billed as 1`</span>
);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(
  <span class="hljs-string">`re-run attempts: <span class="hljs-subst">${rows.filter((r) =&gt; r.rerun).length}</span> jobs, <span class="hljs-subst">${rerun}</span> billed minutes (<span class="hljs-subst">${((rerun / billed) * <span class="hljs-number">100</span>).toFixed(<span class="hljs-number">1</span>)}</span>%)`</span>
);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`queue: median <span class="hljs-subst">${pct(<span class="hljs-number">0.5</span>, queues).toFixed(<span class="hljs-number">0</span>)}</span> s, p90 <span class="hljs-subst">${pct(<span class="hljs-number">0.9</span>, queues).toFixed(<span class="hljs-number">0</span>)}</span> s`</span>);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`macOS jobs: <span class="hljs-subst">${rows.filter((r) =&gt; r.os === <span class="hljs-string">'macOS'</span>).length}</span>`</span>);
<span class="hljs-keyword">const</span> byWorkflow = <span class="hljs-title class_">Object</span>.<span class="hljs-title function_">entries</span>(<span class="hljs-title class_">Object</span>.<span class="hljs-title function_">groupBy</span>(rows, <span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">workflow</span>))
  .<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">[name, rs]</span>) =&gt;</span> [name, <span class="hljs-title function_">sum</span>(rs, <span class="hljs-function">(<span class="hljs-params">r</span>) =&gt;</span> r.<span class="hljs-property">billed</span>), rs.<span class="hljs-property">length</span>])
  .<span class="hljs-title function_">sort</span>(<span class="hljs-function">(<span class="hljs-params">a, b</span>) =&gt;</span> b[<span class="hljs-number">1</span>] - a[<span class="hljs-number">1</span>]);
<span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">'top workflows by billed minutes:'</span>);
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">const</span> [name, b, n] <span class="hljs-keyword">of</span> byWorkflow.<span class="hljs-title function_">slice</span>(<span class="hljs-number">0</span>, <span class="hljs-number">5</span>))
  <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`  <span class="hljs-subst">${<span class="hljs-built_in">String</span>(b).padStart(<span class="hljs-number">6</span>)}</span>  <span class="hljs-subst">${name.slice(<span class="hljs-number">0</span>, <span class="hljs-number">40</span>)}</span> (<span class="hljs-subst">${n}</span> jobs)`</span>);
</code></pre><p>We ran it on the repository behind this site on 18 October 2026:</p>
<p><strong>audit.mjs</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node audit.mjs The-DevOps-Daily/devops-daily 30
The-DevOps-Daily/devops-daily, last 30 days: 1120 runs, 906 <span class="hljs-built_in">jobs</span> that ran
minutes used   1796
minutes billed 2234  (+438 from rounding each job up, 24%)
<span class="hljs-built_in">jobs</span> under 1 minute: 125 of 906 (14%), each billed as 1
re-run attempts: 2 <span class="hljs-built_in">jobs</span>, 6 billed minutes (0.3%)
queue: median 3 s, p90 4 s
macOS <span class="hljs-built_in">jobs</span>: 0
top workflows by billed minutes:
    1077  Tests (377 <span class="hljs-built_in">jobs</span>)
     813  Build Test (376 <span class="hljs-built_in">jobs</span>)
     110  IndexNow Submission (93 <span class="hljs-built_in">jobs</span>)
      26  npm_and_yarn <span class="hljs-keyword">in</span> / <span class="hljs-keyword">for</span> @next/third-partie (2 <span class="hljs-built_in">jobs</span>)
      22  npm_and_yarn <span class="hljs-keyword">in</span> / <span class="hljs-keyword">for</span> @next/third-partie (2 <span class="hljs-built_in">jobs</span>)
</code></pre><p>The repository is public and runs only Linux jobs, so all of this was free, but the shape is common. Rounding added 438 minutes, a 24% markup on the 1,796 minutes the jobs used. The jobs are short: just under 2 minutes each on average, and 125 of them finished in under a minute and billed a full one. Re-runs were 0.3% of the billed minutes, and jobs waited 3 seconds for a runner. Both workflows that bill the most already cancel superseded runs. So of the things this script measures, rounding is the one worth fixing here, with fewer and longer jobs. Billed at the 2-core private-repository rate of $0.006, the rounding would be $2.63 of a $13.40 month. That is small money, but a repository with the same mix of short jobs and a bill a hundred times larger pays the same 24%.</p>
<p>Some limits of the script. It uses the job's start and end times, which is what GitHub bills, but it cannot see your plan's included usage or discounts, so treat its numbers as list-price minutes. It selects runs by the day they were created, so a re-run attempt counts on the day of the original run, and a day with more than 1,000 runs is cut at 1,000. It does not look at artifact storage or cache hit rates. And it counts runs on self-hosted runners as if they were billed; filter them out by label if you have any.</p>
<h2>Free fixes inside GitHub Actions</h2><p>None of these needs a new vendor. Start with the one your audit points at.</p>
<p><strong>Cancel runs that a newer push makes useless.</strong> Every push to a pull request starts a full run. Without a concurrency group, the old run keeps going and bills to the end.</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">concurrency:</span>
  <span class="hljs-attr">group:</span> <span class="hljs-string">${{</span> <span class="hljs-string">github.workflow</span> <span class="hljs-string">}}-${{</span> <span class="hljs-string">github.ref</span> <span class="hljs-string">}}</span>
  <span class="hljs-comment"># Cancel superseded pull request runs, but let runs on main finish.</span>
  <span class="hljs-attr">cancel-in-progress:</span> <span class="hljs-string">${{</span> <span class="hljs-string">github.event_name</span> <span class="hljs-string">==</span> <span class="hljs-string">'pull_request'</span> <span class="hljs-string">}}</span>
</code></pre><p><strong>Filter by path inside the job, not at the workflow level.</strong> A workflow-level <code>paths:</code> filter skips the whole workflow, and then <a href="https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax" rel="noopener noreferrer">its checks stay "Pending"</a>, which blocks a pull request that requires them. A job skipped by its <code>if:</code> condition reports success instead, so put the decision in a small first job:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># In a workflow that runs on pull_request.</span>
<span class="hljs-attr">permissions:</span>
  <span class="hljs-attr">contents:</span> <span class="hljs-string">read</span>
  <span class="hljs-attr">pull-requests:</span> <span class="hljs-string">read</span>

<span class="hljs-attr">jobs:</span>
  <span class="hljs-attr">changes:</span>
    <span class="hljs-attr">runs-on:</span> <span class="hljs-string">ubuntu-24.04</span>
    <span class="hljs-attr">outputs:</span>
      <span class="hljs-attr">ios:</span> <span class="hljs-string">${{</span> <span class="hljs-string">steps.diff.outputs.ios</span> <span class="hljs-string">}}</span>
    <span class="hljs-attr">steps:</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">diff</span>
        <span class="hljs-attr">env:</span>
          <span class="hljs-attr">GH_TOKEN:</span> <span class="hljs-string">${{</span> <span class="hljs-string">github.token</span> <span class="hljs-string">}}</span>
        <span class="hljs-attr">run:</span> <span class="hljs-string">|
          # List the files this pull request changes, without a checkout.
          # A failed API call fails this job, so make it a required check too.
          files=$(gh pr diff ${{ github.event.pull_request.number }} -R ${{ github.repository }} --name-only)
          if grep -q '^ios/' &lt;&lt;&lt; "$files"; then
            echo "ios=true" &gt;&gt; "$GITHUB_OUTPUT"
          fi
</span>
  <span class="hljs-attr">ios:</span>
    <span class="hljs-attr">needs:</span> <span class="hljs-string">changes</span>
    <span class="hljs-attr">if:</span> <span class="hljs-string">needs.changes.outputs.ios</span> <span class="hljs-string">==</span> <span class="hljs-string">'true'</span>
    <span class="hljs-attr">runs-on:</span> <span class="hljs-string">macos-26</span>
    <span class="hljs-attr">timeout-minutes:</span> <span class="hljs-number">45</span>
    <span class="hljs-attr">steps:</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v7</span>
      <span class="hljs-comment"># ... build and test</span>
</code></pre><p>Make both <code>changes</code> and <code>ios</code> required checks. If <code>changes</code> fails, <code>ios</code> is skipped, and a skipped job counts as passed. The <code>changes</code> job bills one Linux minute. Skipping one standard macOS job of 15 minutes saves $0.93.</p>
<p><strong>Set <code>timeout-minutes</code> on every job.</strong> The default limit is 6 hours. One hung macOS job that runs to the limit bills 360 minutes, which is $22.32 at $0.062.</p>
<p><strong>Merge tiny jobs.</strong> Three 20-second lint jobs bill 3 minutes. One job that runs all three bills 1. The same applies to matrix legs that only differ in a flag.</p>
<p><strong>Put light automation on the smallest runner.</strong> <code>ubuntu-slim</code> is a 1 vCPU runner in a container at $0.002 a minute, with a 15-minute job limit and no Docker. It suits labelers, triage bots and other jobs that wait on an API. arm64 Linux runners are also cheaper than x64 at the same size: $0.005 against $0.006 for 2 cores, and $0.008 against $0.012 for 4 cores.</p>
<p><strong>Keep artifacts for days, not months.</strong></p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-bullet">-</span> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/upload-artifact@v7</span>
  <span class="hljs-attr">with:</span>
    <span class="hljs-attr">name:</span> <span class="hljs-string">test-results</span>
    <span class="hljs-attr">path:</span> <span class="hljs-string">build/test-results</span>
    <span class="hljs-attr">retention-days:</span> <span class="hljs-number">5</span>
</code></pre><p>At the 90-day default, an artifact accrues 18 times the GB-hours it accrues at 5 days.</p>
<p><strong>Re-run failed jobs, not all jobs.</strong> And fix the flaky test that made you re-run. The audit's re-run line tells you whether this is worth your time.</p>
<h2>Moving the slow jobs to a faster runner</h2><p>Some jobs are slow because the machine is small, and no workflow change fixes that. GitHub's standard macOS runner is a 3 vCPU M1 with 7 GB of memory. Its largest Apple silicon runner is a 5 vCPU M2 Pro with 14 GB.</p>
<p>Bitrise Build Hub keeps GitHub Actions as the orchestrator and supplies the machines. The jobs still appear in the Actions tab, with the same logs, secrets and triggers. Setup has three parts:</p>
<ol>
<li>An organization owner installs the Bitrise Build Hub GitHub App, preferably only on the repositories that need it.</li>
<li>In the Bitrise workspace, you create a machine pool: an image, a machine type, and a maximum number of machines. Bitrise registers the machines with GitHub as runners in a runner group.</li>
<li>You change <code>runs-on</code> for the jobs you want to move. Bitrise adds a <code>bitrise-runner-&lt;pool name&gt;</code> label to each pool:</li>
</ol>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">jobs:</span>
  <span class="hljs-attr">ios:</span>
    <span class="hljs-comment"># Was: runs-on: macos-26</span>
    <span class="hljs-attr">runs-on:</span> [<span class="hljs-string">bitrise-runner-ios-m4-large</span>]
</code></pre><p><code>runs-on</code> is set per job, so you can move the iOS job and keep everything else where it is.</p>
<p>Three things are different from GitHub-hosted runners, and all three mattered in our test:</p>
<ul>
<li><strong>The images are Bitrise stacks</strong>, not GitHub's runner images. <a href="https://bitrise.io/stacks" rel="noopener noreferrer">bitrise.io/stacks</a> lists what each macOS and Linux stack includes, so check it before you move a job. The Xcode 26.5 stack had the same Xcode build (17F42) and the same iPhone 17 simulator that we used on GitHub. The Ubuntu 24.04 Linux image did not have the Android SDK command-line tools or the OpenGL and X11 libraries that Thunderbird's Compose UI tests load, so our workflow installs them when they are missing: 2 to 7 seconds for the SDK tools and 9 to 14 seconds for four packages. Both steps run inside the job, so their time is in our results and in the cost.</li>
<li><strong>The pool maximum is a setting you own.</strong> A pool starts machines on demand, up to its maximum. With the Android pool at a maximum of 1, the two Android jobs of each pipeline ran one after the other, and the second one waited a median 15 minutes for the first (more on this below).</li>
<li><strong>Billing is a monthly budget.</strong> A Build Hub tier fee becomes your compute budget, and jobs use it at a per-minute rate per machine type. Build Hub bills per started minute, and the billed time includes the machine's start-up. By default, jobs stop running on Build Hub when the budget is used up. You can <a href="https://bitrise.io/changelog/build-hub-pay-as-you-go-limit-setting/26370" rel="noopener noreferrer">turn on pay-as-you-go</a> to keep them running, at twice the standard rate per minute and with a spending cap you set. Size the tier from your run count, and watch the balance.</li>
</ul>
<p>The machines register like self-hosted runners, so the usual hygiene applies. Restrict the runner group to the repositories that need it, and require approval before workflows from forks run.</p>
<h2>The benchmark: one mobile pull request, 34 runs</h2><p>We wanted a test that looks like a real team's pull request, not a synthetic compile.</p>
<ol>
<li><strong>Pull request</strong> one pipeline run</li>
<li><strong>iOS build + tests</strong> Wikipedia app</li>
<li><strong>Android build</strong> Thunderbird</li>
<li><strong>Android tests</strong> Thunderbird</li>
<li><strong>Pipeline done</strong> last job finished</li>
</ol>
<p>Connections:</p>
<ul>
<li>Pull request -&gt; iOS build + tests</li>
<li>Pull request -&gt; Android build</li>
<li>Pull request -&gt; Android tests</li>
<li>iOS build + tests -&gt; Pipeline done</li>
<li>Android build -&gt; Pipeline done</li>
<li>Android tests -&gt; Pipeline done</li>
</ul>
<p><strong>The workloads.</strong> Both apps are open source and pinned by full commit SHA, and every run starts cold, with no cache on either side.</p>
<ul>
<li>iOS: the <a href="https://github.com/wikimedia/wikipedia-ios" rel="noopener noreferrer">Wikipedia iOS app</a> at <code>ca8aaeb9</code>, its "Run Unit Tests" job: resolve packages, <code>xcodebuild build-for-testing</code>, then <code>test-without-building</code> on an iPhone 17 simulator with iOS 26.5 and Xcode 26.5. All 503 unit tests pass on both sides.</li>
<li>Android: <a href="https://github.com/thunderbird/thunderbird-android" rel="noopener noreferrer">Thunderbird for Android</a> at <code>cf54b1fd</code>, with the commands of its own CI. The build job runs <code>./gradlew :app-thunderbird:assemble -Pci=true</code>. The test job runs <code>./gradlew testsOnCi --parallel</code>. Both use the project's CI Gradle settings, with one change: Gradle gets one worker per CPU (upstream pins 4, the size of a GitHub standard runner), and machines with more than 20 GB of memory give the Gradle daemon half of it.</li>
</ul>
<p><strong>The machines.</strong></p>
<table>
<thead>
<tr>
<th>Side</th>
<th>iOS job</th>
<th>Android jobs</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub-hosted</td>
<td><code>macos-26</code>: Apple M1 (virtual), 3 vCPU, 7 GB. $0.062 a minute</td>
<td><code>ubuntu-24.04</code>: 4 vCPU, 16 GB. $0.012 a minute (the 4-core rate in a private repository)</td>
</tr>
<tr>
<td>Build Hub, self-serve Pro Tier 1</td>
<td>M4 Large: Apple M4 (virtual), 5 vCPU, 14 GB. $0.03 a minute</td>
<td>Linux XLarge: AMD EPYC Genoa, 16 vCPU, 64 GB. $0.05 a minute</td>
</tr>
</tbody></table>
<p>These are not same-size pairs. On GitHub we used the standard macOS runner and the standard Linux runner. In a public repository like ours, it has 4 vCPU. In a private repository it has 2 vCPU, and a 4 vCPU runner needs a Team or Enterprise Cloud plan. On Build Hub we used the smallest M4 and the largest self-serve Linux machine. The Build Hub machines have more CPU and memory, and the results hold for these configurations only. Cost per run shows what the size difference costs. Build Hub prices are the rates at which jobs use the plan budget, read from <a href="https://bitrise.io/pricing" rel="noopener noreferrer">bitrise.io/pricing</a> on 18 October 2026, unchanged since we first read them on 10 October.</p>
<p><strong>How we ran it.</strong> Bitrise provided the Build Hub workspace for these tests. The workflows and every run are public in the <a href="https://github.com/The-DevOps-Daily/gha-runner-benchmark" rel="noopener noreferrer">benchmark repository</a>. One workflow, <code>bench-mobile-pipeline.yml</code>, starts the three jobs together. We dispatched the GitHub and Build Hub runs in pairs, most of them seconds apart: 10 pairs from 16 October 15:24 to 17 October 12:15 UTC, then 7 more from 17 October 15:44 to 21:25 UTC. Every iOS job ran all 503 tests with no failures, and every Android build succeeded. One Android test task on GitHub (run 38985488530) failed on a network error: Robolectric could not download a dependency, and Gradle stopped after about 10 minutes. Actions still shows that job as green, because our workflow records a failed Gradle task in its result file instead of failing the job. So 16 of 17 GitHub test jobs and all 17 Build Hub test jobs ran the full suite. We kept that run. Leaving it out makes GitHub's medians slightly worse (45.1 minutes and $1.80 per pipeline run), so keeping it is the conservative choice. Pipeline time is from the moment the run is created to the end of its last job, so queueing and machine start-up count, as they do for the developer who waits. Cost is each job's billed time, rounded up to a whole minute, times the list price, added up per run. On GitHub, the billed time is the job's own run time. Build Hub also bills the machine's start-up, which the job's timestamps do not show, so on that side we counted from the moment the job was queued (or, when it waited for a busy pool, from the moment the pool's machine was free) to the end of the job. That adds a median 38 seconds per job on macOS and 34 seconds on Linux, and it may include a few seconds of scheduling, so if anything it makes Build Hub's cost slightly high. Three tiny planning jobs run on GitHub's runner on both sides and are left out of the cost.</p>
<p><a href="https://github.com/The-DevOps-Daily/gha-runner-benchmark" rel="noopener noreferrer">The-DevOps-Daily/gha-runner-benchmark on GitHub</a></p>
<h3>Pipeline time</h3><p><strong>Mobile pipeline, run created to last job done</strong></p>
<table>
<thead>
<tr>
<th>Series</th>
<th>Samples</th>
<th>Min</th>
<th>Median</th>
<th>p95</th>
<th>Max</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub (17 runs)</td>
<td>17</td>
<td>32.8 min</td>
<td>43.6 min</td>
<td>60.8 min</td>
<td>60.8 min</td>
</tr>
<tr>
<td>Build Hub, pool max 2 (7 runs)</td>
<td>7</td>
<td>17 min</td>
<td>18.5 min</td>
<td>29.4 min</td>
<td>29.4 min</td>
</tr>
<tr>
<td>Build Hub, pool max 1 (10 runs)</td>
<td>10</td>
<td>26.2 min</td>
<td>35.3 min</td>
<td>52.1 min</td>
<td>52.1 min</td>
</tr>
</tbody></table>
<p><em>Every run. GitHub: macos-26 + ubuntu-24.04, 17 runs, 16 to 17 Oct 2026. Build Hub: M4 Large + Linux XLarge, 10 runs with the Android pool maximum at 1, then 7 at 2. Source: results/mobile-pipeline.json.</em></p>
<p>With the Android pool at a maximum of 2, the pipeline took a median 18.5 minutes (17.0 to 29.4) on Build Hub, against 43.6 minutes (32.8 to 60.8) on GitHub. That is about 2.4 times faster. We compare with all 17 GitHub runs because it is the conservative choice: the 7 GitHub runs paired with the pool-maximum-2 runs had a median of 49.8 minutes.</p>
<h3>The pool maximum mistake</h3><p>Our first 10 Build Hub runs had the Android pool maximum at 1, a setting we chose and did not check. The first Android job started after a median 34 seconds. The second waited for the first to finish: a median 886 seconds, and up to 2,646 seconds. The pipeline took a median 35.3 minutes. After we set the maximum to 2, both Android jobs started after 32 to 34 seconds in every run. Not all of the drop to 18.5 minutes came from the setting: the Android builds were also faster in the later runs, a median 1,070 seconds against 1,592.</p>
<p>The rule: set each pool's maximum to at least the number of its jobs that start at the same time, and remember that a matrix starts all of its legs at once.</p>
<h3>Each job</h3><table>
<thead>
<tr>
<th>Job</th>
<th>GitHub, median (range), 17 runs</th>
<th>Build Hub, median (range), 17 runs</th>
<th>Faster by</th>
</tr>
</thead>
<tbody><tr>
<td>iOS build + tests</td>
<td>922 s (597 to 1,173)</td>
<td>337 s (314 to 471)</td>
<td>2.7x</td>
</tr>
<tr>
<td>Android build</td>
<td>2,604 s (1,956 to 3,638)</td>
<td>1,244 s (981 to 2,581)</td>
<td>2.1x</td>
</tr>
<tr>
<td>Android tests</td>
<td>967 s (710 to 1,186)</td>
<td>467 s (400 to 587)</td>
<td>2.1x</td>
</tr>
</tbody></table>
<p>The job times are from all 17 runs per side. The pool maximum changes how long a job waits, not how long it runs, so it does not affect these numbers. Queue time is separate: on GitHub, jobs started after a median 3 seconds on Linux and 8 seconds on macOS. On Build Hub, where each pool starts a machine when a job arrives, they started after a median 33 to 34 seconds on Linux (at pool maximum 2) and 38 seconds on macOS.</p>
<p>The GitHub Android times move a lot. In the 17 runs, GitHub's 4 vCPU runner reported three different CPU models for the build and five for the tests. The two builds on an AMD EPYC 9V45 host took 1,780 and 1,892 seconds for the Gradle task, and the twelve on an EPYC 7763 took 1,979 to 3,426 seconds, so the host explains part of the spread but not all of it. Build Hub's Android build varied too, from 981 to 2,581 seconds for the job, on machines that all reported the same CPU.</p>
<h3>More Mac sizes, including Enterprise</h3><p>Before the pipeline runs, on 15 and 16 October, we ran the same iOS job on its own (<code>bench-ios-app.yml</code>) on bigger Macs too. These are separate runs with fewer samples than the pipeline, so compare the rows with each other, not with the tables above.</p>
<table>
<thead>
<tr>
<th>Machine</th>
<th>Plan</th>
<th>Runs</th>
<th>Job time, median (range)</th>
<th>Faster by</th>
<th>List price per job</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub <code>macos-26</code>: M1, 3 vCPU, 7 GB</td>
<td>GitHub-hosted</td>
<td>7</td>
<td>1,026 s (805 to 1,305)</td>
<td>baseline</td>
<td>$1.12 (18 billed minutes)</td>
</tr>
<tr>
<td>Build Hub M4 Large: M4, 5 vCPU, 14 GB</td>
<td>Self-serve</td>
<td>3</td>
<td>323 s (314 to 335)</td>
<td>3.2x</td>
<td>$0.18 (6 billed minutes)</td>
</tr>
<tr>
<td>Build Hub M4 XLarge: M4, 10 vCPU, 28 GB</td>
<td>Self-serve</td>
<td>3</td>
<td>295 s (293 to 301)</td>
<td>3.5x</td>
<td>$0.36 (6 billed minutes)</td>
</tr>
<tr>
<td>Build Hub M4 Pro XLarge: M4 Pro, 14 vCPU, 54 GB</td>
<td>Enterprise</td>
<td>3</td>
<td>250 s (246 to 251)</td>
<td>4.1x</td>
<td>None: Enterprise is not billed per minute</td>
</tr>
</tbody></table>
<p>All 503 tests passed in every run. The M4 Pro XLarge was the fastest Mac we tested, with 23% less job time than the M4 Large. It is on the Enterprise plan only, which Bitrise sells at an "All-You-Can-Eat" price limited by concurrency, not by minutes, so there is no cost per job to compare. Enterprise pools can also keep idle machines running, and ours did: the two jobs we started after the pool was set up began within 2 seconds. On the self-serve plans, the M4 Large stays the value pick: the M4 XLarge took 9% less time at twice the price per minute.</p>
<h3>Cost per run</h3><p><strong>List-price cost of each pipeline run</strong></p>
<table>
<thead>
<tr>
<th>Series</th>
<th>Samples</th>
<th>Min</th>
<th>Median</th>
<th>p95</th>
<th>Max</th>
</tr>
</thead>
<tbody><tr>
<td>GitHub</td>
<td>17</td>
<td>1.41 USD</td>
<td>1.79 USD</td>
<td>2.03 USD</td>
<td>2.03 USD</td>
</tr>
<tr>
<td>Build Hub</td>
<td>17</td>
<td>1.51 USD</td>
<td>1.77 USD</td>
<td>2.83 USD</td>
<td>2.83 USD</td>
</tr>
</tbody></table>
<p><em>Each job's billed time rounded up to a whole minute, on Build Hub including machine start-up, times the list price: GitHub $0.062 macOS, $0.012 Linux 4-core; Build Hub $0.03 M4 Large, $0.05 Linux XLarge (budget rates, 18 Oct 2026). All 17 runs per side.</em></p>
<p>A pipeline run cost a median $1.77 on Build Hub (range $1.51 to $2.83) and $1.79 on GitHub ($1.41 to $2.03). The means point the other way, $1.91 on Build Hub against $1.74 on GitHub, because of a few slow Android builds on Build Hub: the $2.83 run had a 2,581-second build. So "about the same" is the right summary, not "cheaper".</p>
<p>These are list prices for the compute each run used, not two invoices. Build Hub is sold as a monthly tier, so budget you do not use makes the real price per run higher. Our GitHub runs cost nothing, because the repository is public. In a private repository, both runners can use your plan's included usage first. GitHub gives it as a dollar amount, $12 to $300 a month by plan. The 4-core Linux runner is a larger runner, so it needs a Team or Enterprise Cloud plan.</p>
<p>We used all 17 Build Hub runs for cost because the pool maximum changes queue time, not billed minutes. The 7 runs at pool maximum 2 alone had a median cost of $1.54, mostly because their Android builds were faster (a median 1,070 seconds, against 1,592 seconds in the first 10 runs). Using that number would flatter Build Hub.</p>
<p>The cost per job shows where the money goes:</p>
<table>
<thead>
<tr>
<th>Job</th>
<th>GitHub, median per job</th>
<th>Build Hub, median per job</th>
</tr>
</thead>
<tbody><tr>
<td>iOS build + tests</td>
<td>$0.99 (16 billed minutes at $0.062)</td>
<td>$0.21 (7 billed minutes at $0.03)</td>
</tr>
<tr>
<td>Android build</td>
<td>$0.53 (44 at $0.012)</td>
<td>$1.10 (22 at $0.05)</td>
</tr>
<tr>
<td>Android tests</td>
<td>$0.20 (17 at $0.012)</td>
<td>$0.45 (9 at $0.05)</td>
</tr>
</tbody></table>
<p>The Macs save money and the 16 vCPU Linux machines cost more per job, and in this pipeline the two roughly cancel out. Each column is a median of its own, so the rows do not add up to the per-run medians above. A run used a median 79 billed minutes on GitHub and 39 on Build Hub: fewer minutes, at a higher price per Linux minute.</p>
<p>For budget sizing, use the mean, not the median: at $1.91 per pipeline run, Pro Tier 1's $210 monthly budget covers about 110 runs of this pipeline. Leave headroom: by default, jobs stop running on Build Hub when the budget is used up, and pay-as-you-go minutes cost twice the standard rate.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>The macOS pair is not equal in size.</strong> We did not run GitHub's 5 vCPU M2 Pro (the <code>-xlarge</code> macOS runner), because larger runners need a Team or Enterprise Cloud plan, which our test organization does not have. Its price makes the cost question simple, though: at $0.102 a minute it would have to finish this job within two billed minutes to cost less than Build Hub's $0.21.</li>
<li><strong>The Android pair is not equal in size either.</strong> We did not run GitHub's 16-core Linux larger runner. Per vCPU, its $0.042 a minute ($0.0026 per vCPU-minute) is cheaper than Build Hub's Linux XLarge at $0.05 ($0.0031). We do not know how fast it is on this build.</li>
<li><strong>An 8 vCPU machine might be the better buy for Android.</strong> Build Hub's Linux Large costs half the XLarge rate. We did not test it.</li>
<li><strong>Cold builds only.</strong> Both sides ran without a Gradle or Xcode cache. Teams with a warm cache will see shorter jobs on both sides, and we did not measure how the gap changes.</li>
<li><strong>Our repository is public.</strong> GitHub runs in public repositories are free, and its standard Linux runner there has 4 vCPU. In a private repository, <code>ubuntu-latest</code> has 2 vCPU at $0.006. A 4 vCPU runner there is a larger runner at $0.012, and only Team and Enterprise Cloud plans have larger runners. We priced the GitHub side at the 4-core rate, the same size we ran.</li>
<li><strong>Build Hub start-up time is our estimate.</strong> Bitrise bills per started minute and includes the machine's start-up, which the job's timestamps do not show. We used the wait before each job started, minus any time it waited for another job to free the pool's machine, as the start-up time, so the Build Hub costs may be slightly high. Self-serve pools do not keep idle machines between jobs; that option is on the Enterprise plan, which is not billed per minute.</li>
<li><strong>Seven runs at pool maximum 2, in one evening.</strong> The 18.5-minute median comes from 7 runs between 15:44 and 21:25 UTC on 17 October. The job times come from all 17 runs, spread over two days.</li>
<li><strong>One app per platform.</strong> Wikipedia and Thunderbird are real, busy projects, but two apps are not every app. Run the benchmark on your own slowest job before you move it.</li>
</ul>
<h2>Which jobs to move</h2><p>The decision is per job, and <code>runs-on</code> makes it a one-line change.</p>
<ul>
<li><strong>iOS and other macOS jobs: move them first.</strong> This is where the test was both faster and cheaper: 2.7 times faster and about a fifth of the cost per job, on a newer and larger Mac than GitHub's standard runner. Start with the M4 Large. On the Enterprise plan, the M4 Pro XLarge was faster still.</li>
<li><strong>Heavy Linux jobs, such as Android Gradle builds: move them if waiting costs you more than the minutes.</strong> On a 16 vCPU machine they ran about twice as fast at about twice the cost per job. Measure an 8 vCPU machine too.</li>
<li><strong>Light Linux jobs: either side works.</strong> Build Hub's smallest Linux machine is now the 2Small (2 vCPU, 8 GB) at $0.00625 a minute, close to GitHub's 2-core runner at $0.006. For the smallest jobs, such as a labeler, <code>ubuntu-slim</code> at $0.002 is still the cheapest.</li>
<li><strong>Before any of that, run the audit.</strong> Rounding, re-runs and storage are free to fix, and they shrink whatever runner bill you end up with.</li>
</ul>
<p>For the full feature-by-feature view, including concurrency limits, images, caching and what happens when the budget runs out, see the <a href="https://devops-daily.com/comparisons/bitrise-build-hub-vs-github-hosted-runners">Bitrise Build Hub vs GitHub-hosted runners comparison</a>.</p>
<h2>Summary</h2><p>A GitHub Actions bill grows from small, measurable things: whole-minute rounding, runs nobody needed, artifacts kept for 90 days, and macOS minutes at about ten times the Linux price. The audit script shows what rounding and re-runs cost your repository, and most of the fixes are a few lines of YAML.</p>
<p>When the slow job is slow because the machine is small, a faster runner changes the picture. In our mobile pipeline, Build Hub cut the wait from a median 43.6 minutes to 18.5 minutes at about the same list-price cost per run. The iOS job was cheaper and faster. The Android jobs were faster and more expensive. Set the pool maximum to match the jobs that start together, measure your own slowest job on both sides, and move the jobs where the numbers say so.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DevOps Weekly Digest - Week 44, 2026]]></title>
      <link>https://devops-daily.com/news/2026-week-44</link>
      <description><![CDATA[⚡ Curated updates from Kubernetes, cloud native tooling, CI/CD, IaC, observability, and security - handpicked for DevOps professionals!]]></description>
      <pubDate>Mon, 26 Oct 2026 00:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/news/2026-week-44</guid>
      <category><![CDATA[DevOps News]]></category>
      <content:encoded><![CDATA[<blockquote>
<p>📌 <strong>Handpicked by DevOps Daily</strong> - Your weekly dose of curated DevOps news and updates!</p>
</blockquote>
<hr />
<h2>⚓ Kubernetes</h2><h3>📄 Is Kubernetes ready for AI agents?</h3><p>Kubernetes was built around a fairly simple assumption. It’s that software does what its developers told it to do. AI The post Is Kubernetes ready for AI agents? appeared first on The New Stack.</p>
<p><strong>📅 Oct 25, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/kubernetes-ready-ai-agents/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Multi-tenant GPU Kubernetes on simulated hardware with Kamaji and HAMi</h3><p>GPU teams usually face a bad trade-off. Give every team its own Kubernetes cluster and you pay for a control plane per team. Put everyone in one shared cluster and you get weak isolation. Either way,.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/23/multi-tenant-gpu-kubernetes-on-simulated-hardware-with-kamaji-and-hami/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Inside Agent Compute Engine: Building the compute foundation for the agentic era</h3><p>Every decade or so, a shift in software forces us to rethink computer architecture from first principles. After the first boom of web-scale traffic, Google pioneered warehouse-scale computing and Linu</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/ai-infrastructure/agent-compute-engine-deep-dive/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Before you evaluate an AI SRE, give your existing agent context</h3><p>If you’re evaluating AI SRE products, you probably own most of one already. The coding agent your engineers use, whether that’s Claude Code, Codex or Cursor, can already investigate a Kubernetes clust</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/21/before-you-evaluate-an-ai-sre-give-your-existing-agent-context/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Simplify your enterprise kubernetes and govern your AI agents with Canonical</h3><p>Discover how Canonical is delivering simpler kubernetes to scale enterprise AI at Kubecon North America (NA) 2026 Enterprises have a clear mandate: scale AI initiatives securely and without delay. Yet</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/simplify-your-enterprise-kubernetes-and-govern-your-ai-agents-with-canonical" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Red Hat OpenShift Virtualization achieves NVIDIA-Certified Hypervisor status</h3><p>Red Hat OpenShift Virtualization 4.22 has achieved NVIDIA-Certified Hypervisor status for NVIDIA HGX systems, validating near bare-metal GPU performance for representative AI and accelerated computing</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/red-hat-openshift-virtualization-achieves-nvidia-certified-hypervisor-status" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Securing AI agents in the enterprise AI factory: NVIDIA BlueField and DOCA Argus on Red Hat OpenShift</h3><p>The agent problem at scaleAutonomous agents are powerful because they act. But when instructions are ambiguous, when tools don't behave as expected, or when a complex task takes an unexpected turn, th</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/securing-ai-agents-enterprise-ai-factory-nvidia-bluefield-doca-argus-red-hat-openshift" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>☁️ Cloud Native</h2><h3>📄 PHP2Shell and eMudhra Patched</h3><p>PHP2Shell and the eMudhra certificate compromise landed in the same week. Docker published a fixed PHP image within 24 hours and rebuilt the trust store across the catalog.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/php2shell-and-emudhra-patched/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Istio at KubeCon North America 2026: Let’s Connect in Salt Lake City!</h3><p>Get ready for a packed agenda of Istio activities at KubeCon + CloudNativeCon North America 2026, including Istio Day sessions in the Cloud Native Theater on AI-aware traffic management, multicluster </p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Istio Blog</strong></p>
<p><a href="https://istio.io/latest/blog/2026/kubecon-na/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Security Update on PHP2Shell: Impact for Docker Customers</h3><p>On Tuesday, October 20, 2026, Docker began tracking two critical vulnerabilities in PHP, CVE-2026-109618 and CVE-2026-109619, commonly known as PHP2Shell.</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/security-update-on-php2shell-impact-for-docker-customers/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 ActiveState Partners with Docker to Secure the Open Source in Enterprise Containers</h3><p>Container security spans the image, build, and runtime. See the practical steps that shrink your attack surface and keep vulnerabilities out of production.</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-activestate-partnership/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Filtering and reducing AI agent telemetry with the OpenTelemetry Collector</h3><p>Turn on an AI coding agent’s telemetry and it throws a wall of text at you. It’s expensive to store and what’s worse, you can’t make sense of it. It’s all OpenTelemetry native, though. So you...</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/22/filtering-and-reducing-ai-agent-telemetry-with-the-opentelemetry-collector/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 etcd Patch Releases: v3.7.3, v3.6.16, v3.5.35</h3><p>SIG-etcd has published patch releases for all three supported versions of etcd. These updates fix multiple public security vulnerabilities, as well as several other operational issues. Users should up</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 etcd Blog</strong></p>
<p><a href="https://etcd.io/blog/2026/oct-22-patch-release/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Your Agent Runs in a Docker Sandbox. Now It Can Join a Team.</h3><p>Run AI agents in Docker Sandboxes with BAND AI. Learn how to connect isolated agents, collaborate across environments, and manage access securely.</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/band-ai-agents-docker-sandboxes/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Humans of OpenTelemetry - KubeCon Japan 2026</h3><p>We’re back with our fifth edition of Humans of OpenTelemetry, and for the first time, from Japan! At KubeCon + CloudNativeCon Japan 2026 in Yokohama, I interviewed OpenTelemetry end users and practiti</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/humans-of-otel-japan/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Running Pulumi Neo in a Docker Sandbox: workshop recap</h3><p>The more an agent can do on its own, the more infrastructure work you can hand it, as long as its limits hold when it gets something wrong. In April, they didn’t hold for PocketOS. A coding agent on a</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/running-pulumi-neo-in-a-docker-sandbox/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔄 CI/CD</h2><h3>📄 What is an AI software factory?</h3><p>In an AI software factory, agents build and engineers decide what ships. Learn why code changes and agent behavior both need control in production.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/what-is-an-ai-software-factory/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 gt, the new Gitea CLI</h3><p>Gitea's command line tool, <code>tea</code>, has been rewritten from scratch. It has a new name, <code>gt</code>, and a new home, <a href="https://gitea.com/gitea/cli" rel="noopener noreferrer">gitea.com/gitea/cli</a>. Today, we are releasing its first versio</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Gitea Blog</strong></p>
<p><a href="https://blog.gitea.com/release-of-gitea-cli-0.1.0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Bazel releases addressing the eMudhra CA compromise</h3><p>The Bazel team has published new releases of Bazel and Bazelisk to address the recent disclosure of the eMudhra intermediate CA private key compromise. The following Bazel releases remove the distrust</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Bazel Blog</strong></p>
<p><a href="https://blog.bazel.build/2026/10/23/emudhra-ca-compromise.html" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Platform Engineering Tools: A Buyer's Guide to the Internal Developer Platform Stack</h3><p>A category-by-category guide to platform engineering tools — developer portals, orchestration, IaC, CI/CD, and observability — for teams building an IDP.</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/observability/platform-engineering-tools" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Ubuntu 26.04 runner images are available</h3><p>The official <a href="https://gitea.com/gitea/runner-images" rel="noopener noreferrer">runner images</a> for Gitea Actions now support <strong>Ubuntu 26.04</strong>. The default images are now built directly from <code>buildpack-deps</code> instead of the unmai</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Gitea Blog</strong></p>
<p><a href="https://blog.gitea.com/runner-images-ubuntu-26.04/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 PHP2Shell: GitLab helps find and remediate critical PHP flaws</h3><p>GitLab is not affected by PHP2Shell (CVE-2026-109618 and CVE-2026-109619). The GitLab application is not written in PHP and does not run it to serve web requests, and our Security team has found no ev</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/gitlab-helps-remediate-critical-php-flaws/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Scaling Source Control and CI for AI Coding Agents</h3><p>AI agents are overloading source control and CI. See how 7x growth in PRs and builds at Harness shaped Agent-Ready SCM. | Blog</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/when-code-stops-being-the-bottleneck" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Bring Your Own IdP to Harness MCP | OAuth Proxy Mode</h3><p>Learn how Harness MCP OAuth proxy mode enables enterprise developers to use their own identity provider without static tokens or shared credentials. | Blog</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/bring-your-own-identity-provider-to-harness-mcp" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitLab Critical Patch Release: 19.5.1, 19.4.3, 19.3.6, 19.2.9, 19.1.9, 19.0.10, 18.11.13</h3><p><strong>📅 Oct 21, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-5-1-released/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 php2shell: the flaw hiding all across the web</h3><p>Discover how php2shell vulnerabilities expose PHP servers to remote code execution and how Harness virtual patching protects endpoints while you deploy fixes. | Blog</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/php2shell-the-flaw-hiding-all-across-the-web" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The frontier loves feature flags</h3><p>Two ideas stand out: using classification to drive automation, and treating feature flags as the last line of defense.</p>
<p><strong>📅 Oct 20, 2026</strong> • <strong>📰 LaunchDarkly Blog</strong></p>
<p><a href="https://launchdarkly.com/blog/the-frontier-loves-feature-flags/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How 3 Java Champions are using GitHub Copilot for increased productivity</h3><p>Three Java Champions share how they use GitHub Copilot to solve problems, accelerate development, and keep humans responsible for the results. The post How 3 Java Champions are using GitHub Copilot fo</p>
<p><strong>📅 Oct 20, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/ai-and-ml/github-copilot/how-3-java-champions-are-using-github-copilot-for-increased-productivity/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🏗️ IaC</h2><h3>📄 Web Search on Amazon Bedrock now supports domain filtering and user location controls</h3><p>Web Search on Amazon Bedrock now supports two new controls that give you more precision over the results that ground a model's response: domain filtering and user location. Both are configured directl</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/bedrock-web-search-domain-user-location/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AI infrastructure on tap with the Pulumi AICR provider</h3><p>Recently, some members of the Pulumi team spent time in San Francisco attending the CoreWeave: Fully Connected 2026 conference, where we had a chance to talk to everyone from solo developers to engine</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/self-hosted-ai-infrastructure-pulumi-aicr/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📊 Observability</h2><h3>📄 The sandbox wasn’t enough: Why AI agent governance requires runtime observability</h3><p>Models running an OpenAI cybersecurity evaluation reached the public Internet from an environment designed to be isolated. Package installation through The post The sandbox wasn’t enough: Why AI agent</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/ai-agent-runtime-observability/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The AI observability race will be won on interoperability</h3><p>The AI race is often measured by model quality, speed, and token cost. For business leaders, those measures contribute to The post The AI observability race will be won on interoperability appeared fi</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/ai-observability-through-interoperability/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Fluentd v1.20.1 has been released</h3><p>Hi users! We released v1.20.1 on October 26, 2026. ChangeLog is here. This release is a maintenance release for the v1.20 series. It removes yajl-ruby from the runtime dependencies of Fluentd, so that</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Fluentd Blog</strong></p>
<p><a href="https://www.fluentd.org/blog/fluentd-v1.20.1-has-been-released" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Help us understand how you use OpenTelemetry's ecosystem lists</h3><p>Do you use OpenTelemetry’s registry or ecosystem lists? Comment on the issue and tell us which pages you need and why. We are reviewing the future of these pages. Understanding how people use them wil</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/ecosystem-lists-user-feedback/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Datadog vs. Grafana: Comparing Two Paths to Observability (2026)</h3><p>Datadog vs. Grafana compared on cost predictability, operational overhead, and observability scope, and where a unified platform changes the calculus.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/observability/datadog-vs-grafana" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Datadog vs. Splunk: Which Fits Your Observability Stack in 2026?</h3><p>See how Datadog and Splunk compare on monitoring, log analytics, and cost — and why more teams are consolidating onto one platform instead of two.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/observability/datadog-vs-splunk" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Fluentd v1.20.0 has been released</h3><p>Hi users! We released v1.20.0 on October 23, 2026. ChangeLog is here. Fluentd v1.20.0 is the first release in the v1.20 series. It enables Ruby JIT by default and adds options to limit the size of eva</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Fluentd Blog</strong></p>
<p><a href="https://www.fluentd.org/blog/fluentd-v1.20.0-has-been-released" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Meeting the Moment: Growing Our Sales Team in the Silicon South</h3><p>Build your tech sales career at New Relic Atlanta with hands-on AI tools, a collaborative culture, and real opportunities to grow.</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/news/meeting-the-moment-growing-our-sales-team-in-the-silicon-south" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Should you use a log or a metric?</h3><p>If you already log it, you can probably just count the logs. Here's when a question deserves its own metric, shown with one signup handler.</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/when-you-need-a-metric/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Observability patterns for AI agents on Amazon EKS</h3><p>Traditional monitoring shows an AI agent's endpoint returned HTTP 200, but not why it picked the wrong tool, burned through your token budget, or looped. This post implements two OpenTelemetry observa</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 AWS Containers Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/containers/observability-patterns-for-ai-agents-on-amazon-eks/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Spend less time on operational busywork and resolve incidents faster with Grafana Workflows</h3><p>You're investigating checkout errors when someone asks who's coordinating the incident. Support needs an update, and another responder is looking for the runbook. You stop what you're doing to get eve</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/spend-less-time-on-operational-busywork-and-resolve-incidents-faster-with-grafana-workflows/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 We just gave a million dollars to Open Source maintainers</h3><p>Sentry raised its Open Source budget from $750,000 to $1,000,000 this year. Here's why, and how we split it among maintainers and foundations.</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/another-year-another-1-000-000-to-open-source-maintainers/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔐 Security</h2><h3>📄 Threats Making WAVs - Incident Response to a Cryptomining Attack</h3><p>Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/threats-making-wavs-incident-reponse-cryptomining-attack" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Isolation. Security. Scale.</h3><p>Without these three things, Agents cannot be properly implemented in a controlled environment, they will not have policy enforcement to ensure they do what they’re supposed to do, and there’s zero way</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/26/isolation-security-scale/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 PHP Patches Two Critical Unauthenticated RCEs (CVE-2026-109618, CVE-2026-109619):</h3><p>Two critical flaws in PHP’s request lifecycle can enable unauthenticated remote code execution. Upgrade to a fixed release or install a vendor backport for older versions.</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/php-patches-critical-unauthenticated-rces/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Mobile Application Security Testing: Methods, Tools</h3><p>Mobile application security testing finds flaws in your app and its SDKs before release. Learn MAST methods, iOS/Android risks, and tool criteria. | Blog</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/mobile-application-security-testing" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AWS says its AI security agent fixes 89% of bugs — but there’s a catch</h3><p>AWS made Continuum for code vulnerabilities generally available on Thursday. The service finds and validates bugs in your repositories, checks The post AWS says its AI security agent fixes 89% of bugs</p>
<p><strong>📅 Oct 24, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/aws-says-its-ai-security-agent-fixes-89-of-bugs-but-theres-a-catch/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Security Disclosure Recommendation: Resolving Login Issues</h3><p>Commits pushed while Railway paused deployments on October 21 were not deployed. Redeploy the latest commit, reconnect GitHub, and recreate CLI and CI credentials to get your services deploying again.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Railway Blog</strong></p>
<p><a href="https://blog.railway.com/p/security-disclosure-recommendation-recover-account-access" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Security Disclosure Recommendation: Recover Failing Deploys</h3><p>Commits pushed while Railway paused deployments on October 21 were not deployed. Redeploy the latest commit, reconnect GitHub, and recreate CLI and CI credentials to get your services deploying again.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Railway Blog</strong></p>
<p><a href="https://blog.railway.com/p/security-disclosure-recommendation-recover-failing-deploys" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon Corretto October 2026 Quarterly Updates</h3><p>On October 22, 2026 Amazon announced quarterly security and critical updates for Amazon Corretto Long-Term Supported (LTS) versions of OpenJDK. Corretto 27.0.1.0.10, 25.0.5.0.9, 21.0.13.0.10, 17.0.21.</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-corretto-october-2026-quarterly-updates" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Preliminary Report: Oct 21st Security Incident</h3><p>Preliminary report detailing Railway's October 21st security incident.</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Railway Blog</strong></p>
<p><a href="https://blog.railway.com/p/preliminary-report-october-security-incident" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How bug bounty sharpens one researcher’s security skills</h3><p>As Cybersecurity Awareness Month continues, the GitHub Bug Bounty team spotlights @r31n, diving into their research approach, favorite bug classes, and advice for aspiring hackers. The post How bug bo</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/security/how-bug-bounty-sharpens-one-researchers-security-skills/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Strengthening OpenTelemetry security while preserving API stability</h3><p>OpenTelemetry has updated its client versioning policy to make room for security fixes that need to change behavior. Stable SDKs and Contrib components, such as instrumentation libraries and exporters</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/security-versioning-policy/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Localizing compliance at scale: achieving South Korea’s ISMS certification</h3><p>Cloudflare has achieved South Korea's ISMS certification following an audit across 80 controls. Learn how we localized user consent flows, adapted social sign-up paths at the edge, and validated our g</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/k-isms/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>💾 Databases</h2><h3>📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers</h3><p>Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/please-read-me-opportunistic-ransomware-devastating-mysql-servers" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How to Send YugabyteDB Anywhere Alerts to Microsoft Teams Workflows</h3><p>Connecting YugabyteDB Anywhere to Microsoft Teams Workflows allows operations and database engineers to deliver alerts directly into their primary communication channel. Routing notifications to a sha</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Yugabyte Blog</strong></p>
<p><a href="https://www.yugabyte.com/blog/yugabytedb-anywhere-alerts-to-microsoft-teams-workflows/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How organizations can get more value from open source</h3><p>An organization might first encounter open source when it adopts an open source library, operating system, database, or application. But robust open source engagement goes much further than just using</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/how-organizations-can-get-more-value-open-source" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pgrecon: open-source Oracle to PostgreSQL migration assessment and schema conversion</h3><p>pgrecon, an Apache-2.0 licensed Oracle to PostgreSQL migration assessment and schema conversion tool, is now public. The current release is 0.7.3. pgrecon works entirely offline. A reviewable, read-on</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pgrecon-open-source-oracle-to-postgresql-migration-assessment-and-schema-conversion-3410/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 SafeGrd: PostgreSQL backups that are restored and checked on a schedule</h3><p>SafeGrd is a new backup service for PostgreSQL 13 and later that restores every backup on a schedule and checks the row counts. It takes a backup, restores it later into memory or a throwaway database</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/safegrd-postgresql-backups-that-are-restored-and-checked-on-a-schedule-3414/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 BackupDrill: open-source Supabase Postgres backups with automated restore drills</h3><p>BackupDrill backs up Supabase Postgres databases to a bucket you own, then restores each backup into a throwaway PostgreSQL instance to check it restores. Most teams find out whether their pg_dump bac</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/backupdrill-open-source-supabase-postgres-backups-with-automated-restore-drills-3413/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pgCluu v5.0 has been released</h3><p>Antananarivo, Madagascar - October 20, 2026 pgCluu PostgreSQL Cluster Utilization pgCluu is a Perl program used to perform a full audit of a PostgreSQL Cluster performances. It is divided in two parts</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pgcluu-v50-has-been-released-3417/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🌐 Platforms</h2><h3>📄 The Oracle of Delphi Will Steal Your Credentials</h3><p>Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-oracle-of-delphi-steal-your-credentials" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger</h3><p>In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-nansh0u-campaign-hackers-arsenal-grows-stronger" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 PHP2Shell and Azure DevOps: what to check</h3><p>On October 21, 2026, PHP disclosed two critical vulnerabilities that can allow remote code execution. Neither Azure DevOps Services, our cloud-hosted offering, nor Azure DevOps Server, our on-premises</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Azure DevOps Blog</strong></p>
<p><a href="https://devblogs.microsoft.com/devops/php2shell-and-azure-devops-what-to-check/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Whats is Batch Inference? (2027)</h3><p>Last week’s support tickets, a 50,000-item product catalog, and a backlog of contracts to summarize. What do these jobs have in common? None of them needs a response in two seconds. But if you run the</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/what-is-batch-inference" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Origin Range Requests: Serving large files without range-request tuning</h3><p>Naive byte-range caching can waste gigabytes of origin bandwidth on small requests. We rebuilt our cache architecture in Pingora to coordinate partial fetches automatically, cutting R2 p99 latency in </p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/origin-range-requests/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Bringing post-quantum cryptography to geographically fenced key management</h3><p>Cloudflare's new post-quantum GeoKey Manager protects geographically fenced decryption keys from harvest now, decrypt later attacks. Learn how we redesigned our cryptography stack to support strict da</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/post-quantum-geo-key-manager/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon Connect Customer Global Resiliency now supports persistent chat across linked AWS Regions</h3><p>Amazon Connect Customer Global Resiliency now provides customers a mechanism to deliver long, persistent chat experiences across linked AWS Regions. With this capability, end users can resume previous</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-connect-global-resiliency-persistent-chat" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend</h3><p>Prompt caching is now generally available on DigitalOcean Serverless Inference, across supported models in our open model catalog, with no changes to your request code required. Cached input is billed</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/now-available-prompt-caching" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 A new way to upgrade to Ubuntu Pro on AWS</h3><p>Upgrading existing Ubuntu LTS instances to Ubuntu Pro using AWS License Manager just got better with a new destination license model. We are happy to announce a new way to upgrade existing Ubuntu LTS </p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/upgrade-ubuntu-lts-instances-to-ubuntu-pro-in-aws" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AWS Transit Gateway and AWS Cloud WAN now support 1:1 NAT</h3><p>AWS Transit Gateway and AWS Cloud WAN now provide managed one-to-one network address translation (NAT) directly on their attachments, through integration with Regional Private NAT Gateway. This enable</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/aws-transit-gateway-cloud-wan-1-1-nat/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Feng Shui for Robots: The Fly MCP Server</h3><p>In Databricks’ State of AI Agents 2026 report, there’s a chart from Neon’s telemetry that’s somehow both mind-blowing and unsurprising. In October 2023, AI agents created 0.1% of the databases on Neon</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Fly.io Blog</strong></p>
<p><a href="https://fly.io/blog/feng-shui-for-robots/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Expanding Claude availability and deployment choice on Google Cloud</h3><p>Every organization has unique requirements for building and scaling generative AI. At Google Cloud, we are committed to giving developers model choice and architectural flexibility to run workloads ho</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/ai-machine-learning/expanding-claude-availability-on-google-cloud/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📰 Misc</h2><h3>📄 Visual Studio Code 1.144 (Insiders)</h3><p>Learn what is new in Visual Studio Code 1.144 (Insiders) Read the full article</p>
<p><strong>📅 Oct 28, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_144" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AI on Your Terms: How We See AI in YouTrack</h3><p>AI on your terms When we published our YouTrack roadmap at the beginning of the year, we only briefly touched on our plans for AI. Now it’s time to share more details about how we see its role within </p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/youtrack/2026/10/ai-on-your-terms-how-we-see-ai-in-youtrack/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Enterprise AI needs a data strategy, not just a model strategy</h3><p>Many enterprise AI conversations start with the same question: Which model should we use? With models becoming more and more commoditized, the big differentiator is whether your data is ready for any </p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/enterprise-ai-needs-data-strategy-not-just-model-strategy" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Responsible disclosure in the age of Vulnpocalypse</h3><p>New tools can find potential flaws faster than teams can assess and fix them. When a serious flaw is discovered, the central challenge is ensuring maintainers, distributors, and operators receive the </p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/responsible-disclosure-age-vulnpocalypse" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 MLOps for edge AI: Keeping models accurate over time</h3><p>MLOps for edge AI: Keeping models accurate over timeIf you followed Part 1, Part 2, and Part 3 of this edge AI machine learning operations (MLOps) series, you've seen how a model goes from training an</p>
<p><strong>📅 Oct 26, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/mlops-edge-ai-keeping-models-accurate-over-time" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Busy Plugin Developers Newsletter – Q3 2026</h3><p>Your quarterly dose of plugin dev news, tools, and tips from JetBrains ⚡JetBrains Plugin Developer Conf 2026 Is Coming! November 10, 2026 | Online | FreeJoin our annual free virtual event on developin</p>
<p><strong>📅 Oct 25, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/platform/2026/10/busy-plugin-developers-newsletter-q3-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 From personas to archetypes: a living, shared understanding of the people we build for (Part 2 of 2)</h3><p>This is the second of two posts on our archetypes, shared, machine-readable descriptions of the people we build for. The first covers why we needed them and what they are. This one is about how we bui</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/from-personas-to-archetypes-part-2" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 JetBrains GameDev Day 2026: Watch the Recordings</h3><p>Missed JetBrains GameDev Day 2026, or want to revisit your favorite sessions? All the recordings are now available on YouTube! On September 28, we brought together game developers, engineers, and indu</p>
<p><strong>📅 Oct 23, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/dotnet/2026/10/23/jetbrains-gamedev-day-2026-watch-the-recordings/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Air Governance: Manage AI Agents, Access, and Spend Throughout Your Organization</h3><p>Today, JetBrains Air Governance becomes generally available. Formerly known as JetBrains Central, this is the organization layer of JetBrains Air, a system of products for agentic development. Air Gov</p>
<p><strong>📅 Oct 22, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/blog/2026/10/22/air-governance-manage-ai-agents-access-and-spend-throughout-your-organization/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Visual Studio Code 1.143</h3><p>Learn what's new in Visual Studio Code 1.143 Read the full article</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_143" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 From personas to archetypes: a living, shared understanding of the people we build for (Part 1 of 2)</h3><p>Personas are easy to create but often ignored. This is the first of two posts about how we developed archetypes to keep our personas fresh at Canonical. These are shared, machine-readable descriptions</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/from-personas-to-archetypes" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Node.js October 2026 Security Release: Finding Every Node You Actually Run]]></title>
      <link>https://devops-daily.com/posts/node-october-2026-security-release-find-every-node</link>
      <description><![CDATA[Node.js 22.23.4, 24.22.1 and 26.12.1 fix a Brotli decoder bug that the project says permits remote code execution, plus TLS and DNS bugs. The version bump is one line. pm2 entries, the pm2 daemon, systemd units, cron jobs and containers each keep their own Node, so here is how to find every one and prove it moved.]]></description>
      <pubDate>Thu, 22 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/node-october-2026-security-release-find-every-node</guid>
      <category><![CDATA[Linux]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Linux]]></category><category><![CDATA[Node.js]]></category><category><![CDATA[Security]]></category><category><![CDATA[pm2]]></category><category><![CDATA[Patch Management]]></category><category><![CDATA[CVE]]></category><category><![CDATA[Docker]]></category>
      <content:encoded><![CDATA[<p>On 21 October 2026 the Node.js project released 22.23.4, 24.22.1 and 26.12.1. The headline fix is CVE-2026-111497, an out-of-bounds write in the Brotli decoder that the advisory says "permits remote code execution". The same release fixes two TLS certificate hostname-check bypasses, a DNS bug that truncates names at a NUL character, and updates the bundled CA store. End-of-life lines get no fix.</p>
<p>Patching Node is a one-line version bump. Finding every Node your servers actually run is the work. When we patched a small production box on release day, six Node apps under pm2 installed with nvm, installing the new version was the easy part. Then we found pm2 entries with the old binary written into them, a systemd unit with the old <code>PATH</code> hard-coded, child processes we had not thought about, and cron jobs on two other boxes that pointed at the old install by absolute path. None of them moved when the default version changed.</p>
<p>This post covers what to do now, who is exposed (including services that call <code>fetch()</code> on a URL a user controls), the upgrade steps for nvm, pm2, systemd and containers with the traps we hit, how to verify with <code>/proc</code>, and a read-only audit script with its real output from one of our machines.</p>
<h2>What to do now</h2><ol>
<li><strong>Upgrade every Node</strong> to 22.23.4, 24.22.1 or 26.12.1 or later. Node 20, 25 and older are end-of-life and get no fixed release. Move them to 22, 24 or 26.</li>
<li><strong>List every place Node runs or will start</strong>: running processes, pm2 entries (live and saved), the pm2 daemon, systemd units, crontabs and helper scripts, and containers. Changing the nvm default moves none of them.</li>
<li><strong>Move each one</strong> with the steps below, then check the real binary of every process in <code>/proc</code>. Process names do not help: on our build machine <code>pgrep node</code> found 0 of 5 Node processes.</li>
<li><strong>Rebuild images</strong> with <code>--pull</code> and redeploy. The official <code>node:22</code>, <code>node:24</code> and <code>node:26</code> tags have the fix. <code>node:20</code>, <code>node:25</code> and <code>node:18</code> will not get it.</li>
<li><strong>If a service cannot be upgraded today</strong> and accepts Brotli data from strangers, close that path in front of it (details below). For outbound <code>fetch()</code> the fix is the upgrade; until then, turning off features that fetch user-supplied URLs is the way to close that path.</li>
<li><strong>Run the audit script</strong> on every host and keep the output.</li>
</ol>
<h2>Prerequisites</h2><ul>
<li>Shell access to the hosts that run Node, as the user that owns the Node processes. Root is not needed.</li>
<li>Python 3.8 or newer for the audit script. It uses only the standard library and only reads.</li>
<li>The <code>docker</code> or <code>podman</code> CLI on hosts that run Node in containers.</li>
</ul>
<h2>What was fixed</h2><p>The <a href="https://nodejs.org/en/blog/vulnerability/october-2026-security-releases" rel="noopener noreferrer">advisory</a> lists ten vulnerabilities. The four rated high:</p>
<table>
<thead>
<tr>
<th>CVE</th>
<th>What the advisory calls it</th>
<th>Lines</th>
</tr>
</thead>
<tbody><tr>
<td>CVE-2026-111497</td>
<td>Brotli decoder out-of-bounds write permits remote code execution</td>
<td>22, 24, 26</td>
</tr>
<tr>
<td>CVE-2026-105483</td>
<td>TLS certificate hostname bypass when the IP is written in fullwidth digits</td>
<td>24, 26</td>
</tr>
<tr>
<td>CVE-2026-106069</td>
<td>TLS certificate hostname bypass when the host loosely resolves to an IP address</td>
<td>22, 24, 26</td>
</tr>
<tr>
<td>CVE-2026-105482</td>
<td><code>dns.resolve</code> truncates names containing a NUL character</td>
<td>22, 24, 26</td>
</tr>
</tbody></table>
<p>The medium ones are a keep-alive socket that processes a response written before the request was sent (CVE-2026-105481), DNS SANs that can authenticate IP endpoints written in a non-canonical IPv4 form (CVE-2026-106072), and <code>crypto.randomUUID()</code> and <code>randomInt()</code> caches saved into startup snapshots and single executable applications, so that every process repeats the same values (CVE-2026-106074). The low ones are two Permission Model bypasses and a ZIP header issue on 26.x.</p>
<p>The release also updates the bundled root certificates to NSS 3.131.1 and makes Node reject certificate chains that contain blocked public keys. The advisory lists this as the "ca-cert (eMudhra Intermediate CA Compromise)" update. The bundled undici, c-ares and OpenSSL (22 and 24) are updated too.</p>
<p>Two things the advisory does not give you yet. There is no technical detail on any of the bugs, and on the morning of 22 October all ten CVE IDs were still in the RESERVED state at CVE.org, with no NVD record. So there is no CVSS score; "high" is the Node.js project's rating.</p>
<p>The timeline in UTC, from the release metadata:</p>
<ul>
<li><strong>21 Oct, 10:32 to 10:36</strong>: the fixed binaries on nodejs.org/dist are dated in this window. The GitHub releases were published at 10:38.</li>
<li><strong>10:48</strong>: the update to the official Docker images was committed.</li>
<li><strong>about 11:35</strong>: the Debian-based official images were built. Docker Hub shows the amd64 and arm64 images as pushed at about 14:40.</li>
<li><strong>19:27</strong>: the Brotli project released 1.2.1, 1.1.1 and 1.0.10, each with "CVE-2026-111497: out-of-bounds write permits remote code execution" in the notes. So the bug is in the Brotli decoder itself, and Node shipped the upstream fix as a cherry-picked commit.</li>
<li><strong>22:58 to 00:08</strong>: the Alpine-based official images were built, about 12 to 13.5 hours after the release.</li>
</ul>
<h2>Who is exposed</h2><h3>Brotli: any Node process that decodes Brotli someone else chose</h3><p>The decoder is compiled into the <code>node</code> binary and sits behind <code>node:zlib</code>. Every way into it counts, and three are common:</p>
<ol>
<li><strong>Your code decompresses Brotli directly</strong>: <code>zlib.brotliDecompress()</code>, <code>brotliDecompressSync()</code> or <code>createBrotliDecompress()</code> on uploads, archives, cache entries or messages.</li>
<li><strong>Your service fetches URLs.</strong> <code>fetch()</code> in Node is the bundled undici, and it decodes responses that say <code>Content-Encoding: br</code> <a href="https://github.com/nodejs/node/blob/v24.22.1/deps/undici/src/lib/web/fetch/index.js#L2256-L2260" rel="noopener noreferrer">with <code>zlib.createBrotliDecompress</code></a>. It goes by the response's header, whatever the request asked for, so setting <code>Accept-Encoding: identity</code> does not keep Brotli away from a hostile server. axios in Node <a href="https://github.com/axios/axios/blob/v1.20.0/lib/adapters/http.js#L1179-L1209" rel="noopener noreferrer">does the same</a> unless you pass <code>decompress: false</code>. Link previews, webhook testers, feed readers, "import from URL" features and AI agents that browse are all services that fetch a URL a user chose.</li>
<li><strong>Your service accepts compressed request bodies.</strong> body-parser 2 <a href="https://github.com/expressjs/body-parser/blob/v2.3.0/lib/read.js#L190-L224" rel="noopener noreferrer">decompresses <code>br</code> request bodies</a> that it parses, and its <code>inflate</code> option defaults to <code>true</code>. Express 5 ships body-parser 2, so <code>express.json()</code> on Express 5 decodes Brotli bodies sent with a JSON content type, from any client. Express 4 uses body-parser 1.x, which handles only gzip and deflate.</li>
</ol>
<p>Reaching the decoder is not the same as being exploitable, and the advisory does not say what input triggers the bug. These are the paths to upgrade or close first. Because the decoder is in the binary, no npm update fixes it. Upgrading the <code>undici</code> package in your <code>package.json</code>, for example, still decodes through the old Node's zlib.</p>
<p>If you cannot upgrade a service today, the interim options close only the paths you control:</p>
<pre><code class="hljs language-js" tabindex="0"><span class="hljs-comment">// Express 5: refuse compressed request bodies (415) until Node is upgraded.</span>
<span class="hljs-comment">// This also refuses gzip and deflate request bodies.</span>
app.<span class="hljs-title function_">use</span>(express.<span class="hljs-title function_">json</span>({ <span class="hljs-attr">inflate</span>: <span class="hljs-literal">false</span> }));
</code></pre><pre><code class="hljs language-nginx" tabindex="0"><span class="hljs-comment"># In front of any Node service: refuse Brotli request bodies.</span>
<span class="hljs-attribute">if</span> (<span class="hljs-variable">$http_content_encoding</span> <span class="hljs-regexp">~* "br")</span> {
    <span class="hljs-attribute">return</span> <span class="hljs-number">415</span>;
}
</code></pre><p>Distribution builds are different. Debian's <code>nodejs</code> package (we checked the trixie build of 20.19.2) is built with <code>--shared-brotli</code> against the system <code>libbrotli</code>, so on Debian the Brotli fix comes from the distribution's brotli package, not from the Node version number. At the time we checked, neither Debian's security tracker nor Ubuntu's notices listed CVE-2026-111497 yet. The audit script reports such builds as unknown rather than guessing.</p>
<h3>TLS hostname checks and DNS</h3><p>When Node connects over TLS it checks the server's certificate against the host you asked for. Both high-severity TLS bugs are in that check when the host is, or loosely resolves to, an IP address written in a non-standard form: fullwidth digits (24 and 26 only) or a form that "loosely resolves to an IP address" (all lines). The fix makes Node reject non-canonical IP addresses. The advisory gives no more detail than that. Our reading is that the services to look at first are the ones where someone else chooses the host string: webhook senders, URL fetchers and integrations that connect to customer endpoints.</p>
<p>The DNS bug truncated names at a NUL byte in <code>dns.resolve*()</code>, and the fix rejects such input. Code that resolves names it got from users is where this matters.</p>
<h3>End-of-life lines</h3><p>"End-of-Life versions are always affected when a security release occurs," says the advisory. Node 20 reached end of life on 30 April 2026 and Node 25 on 1 June 2026. Neither gets a fixed release, and the official <code>node:20</code> and <code>node:25</code> image tags stay on 20.20.2 and 25.9.0.</p>
<h2>Why changing the default moves nothing</h2><p>nvm installs the new version next to the old one, which is what you want: the old one stays for rollback. <code>nvm alias default 24.22.1</code> then changes what <strong>new shells</strong> get. Nothing that is already running reads that alias, and nothing that stored a path to the old install reads it either.</p>
<ol>
<li><strong>nvm install 24.22.1</strong> next to the old version</li>
<li><strong>nvm alias default</strong> new shells get it</li>
<li><strong>What runs the new Node?</strong> check /proc, not config</li>
</ol>
<p>Outcomes:</p>
<ul>
<li><strong>New shells, and pm2 apps restarted with --update-env from one</strong></li>
<li><strong>pm2 daemon, cluster workers, absolute interpreters, systemd unit, cron, images: still old</strong></li>
</ul>
<p>On the production box, each of these needed its own step:</p>
<ul>
<li><strong>pm2 entries with an absolute interpreter.</strong> Several entries had the old Node binary written into them as the interpreter, and the ones started as <code>npm</code> or <code>npx</code> had their script path under the old version's <code>bin</code> directory. Changing the default did not move them. They had to be deleted and recreated with the new interpreter.</li>
<li><strong>pm2 entries with a bare <code>node</code> interpreter</strong> took the new binary only after <code>pm2 restart &lt;name&gt; --update-env</code> from a shell where the new Node came first in <code>PATH</code>.</li>
<li><strong>The pm2 systemd unit</strong> that <code>pm2 startup</code> wrote had the old version's <code>PATH</code> and the old pm2 path hard-coded. After editing it, <code>pm2 update</code> moved the pm2 daemon itself, which restarts every app at once. The sites on that box were down for under a minute while the Next.js apps started.</li>
<li><strong>Child processes.</strong> Checking <code>/proc/&lt;pid&gt;/exe</code> for every process turned up chains such as <code>npx</code> to <code>tsx</code> to <code>node</code>, and <code>npm start</code> to <code>next-server</code>. Each process in a chain has its own binary.</li>
<li><strong>Cron jobs and helper scripts</strong> on two other boxes had absolute paths to the old Node.</li>
</ul>
<h2>What pm2 really does on an upgrade</h2><p>To check each of those behaviors on its own, <code>lab/pm2-pin-lab.sh</code> in the repo starts four throwaway apps in a separate <code>PM2_HOME</code> from a shell where Node v24.14.0 comes first in <code>PATH</code>: <code>bare</code> (<code>pm2 start server.js</code>), <code>via-npm</code> (<code>pm2 start npm -- start</code>, where the start script is <code>node server.js</code>), <code>abs</code> (<code>--interpreter</code> set to the old binary) and <code>clustered</code> (<code>-i 1</code>, cluster mode). Then it puts v24.22.1 first in <code>PATH</code>, as a changed nvm default does, and records the real binary of every process after each step. A plain <code>pm2 restart</code> left all four on v24.14.0. This is the state after <code>pm2 restart &lt;name&gt; --update-env</code> for each app:</p>
<p><strong>pm2-pin-lab.sh, step B2</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># excerpt of results/pm2-lab-20261022T063002Z.txt (2026-10-22T06:30:02Z, aarch64, pm2 6.0.14)</span>
<span class="hljs-comment"># old: v24.14.0, new: v24.22.1 first in PATH; a plain restart left all four apps on v24.14.0</span>
$ lab/pm2-pin-lab.sh &lt;v24.14.0 bin&gt; &lt;v24.22.1 bin&gt;
$ pm2 restart bare --update-env
$ pm2 restart via-npm --update-env
$ pm2 restart abs --update-env
$ pm2 restart clustered --update-env

== B2. pm2 restart --update-env from the new shell
daemon     pid 1937570 node v24.14.0   &lt;v24.14.0&gt;/bin/node
bare       interpreter=node  script=&lt;app&gt;/server.js
           saved PATH finds node <span class="hljs-keyword">in</span> &lt;v24.22.1&gt;/bin
  pid 1938284 node v24.22.1    &lt;v24.22.1&gt;/bin/node                          server.js
via-npm    interpreter=&lt;v24.14.0&gt;/bin/node  script=&lt;v24.14.0&gt;/bin/npm
           saved PATH finds node <span class="hljs-keyword">in</span> &lt;v24.22.1&gt;/bin
  pid 1938375 node v24.14.0    &lt;v24.14.0&gt;/bin/node                          npm start
    pid 1938462 dash (not node)  /usr/bin/dash                                sh -c node server.js
      pid 1938468 node v24.22.1    &lt;v24.22.1&gt;/bin/node                          node server.js
abs        interpreter=&lt;v24.14.0&gt;/bin/node  script=&lt;app&gt;/server.js
           saved PATH finds node <span class="hljs-keyword">in</span> &lt;v24.22.1&gt;/bin
  pid 1938455 node v24.14.0    &lt;v24.14.0&gt;/bin/node                          server.js
clustered  interpreter=node  script=&lt;app&gt;/server.js  (cluster mode)
           saved PATH finds node <span class="hljs-keyword">in</span> &lt;v24.22.1&gt;/bin
  pid 1938518 node v24.14.0    &lt;v24.14.0&gt;/bin/node                          server.js
</code></pre><p><code>bare</code> moved. <code>via-npm</code> is split: npm itself still runs on the old Node because pm2 stored the old binary as its interpreter, while its child <code>node server.js</code> got the new one from <code>PATH</code>. <code>abs</code> did not move. <code>clustered</code> did not move either, even though its saved <code>PATH</code> now finds v24.22.1: in cluster mode the daemon forks the workers with its own Node binary, so they move only when the daemon moves. All four steps, from <code>results/pm2-lab-20261022T063002Z.txt</code>:</p>
<table>
<thead>
<tr>
<th>App</th>
<th>After <code>restart</code></th>
<th>After <code>restart --update-env</code></th>
<th>After delete and recreate</th>
<th>After <code>pm2 update</code></th>
</tr>
</thead>
<tbody><tr>
<td><code>bare</code> (bare <code>node</code>)</td>
<td>old</td>
<td>new</td>
<td>not recreated: new</td>
<td>new</td>
</tr>
<tr>
<td><code>via-npm</code> (npm path stored)</td>
<td>old</td>
<td>npm old, app new</td>
<td>new</td>
<td>new</td>
</tr>
<tr>
<td><code>abs</code> (absolute interpreter)</td>
<td>old</td>
<td>old</td>
<td>new</td>
<td>new</td>
</tr>
<tr>
<td><code>clustered</code> (cluster mode)</td>
<td>old</td>
<td>old</td>
<td>not recreated: old</td>
<td>new</td>
</tr>
<tr>
<td>pm2 daemon</td>
<td>old</td>
<td>old</td>
<td>old</td>
<td>new</td>
</tr>
</tbody></table>
<p><code>pm2 update</code> restarted every app in one command: all four PIDs changed. On the production box that was the step where the sites went down. In the lab the command took 2.3 seconds; that measures the command, not how long the apps were unavailable, and real apps take as long as they take to start.</p>
<h2>Upgrade steps: nvm, pm2 and systemd</h2><p>Run these as the user that owns the pm2 daemon.</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># 1. Install next to the old version and bring global packages (pm2) along.</span>
nvm install 24.22.1 --reinstall-packages-from=24.14.0
nvm <span class="hljs-built_in">alias</span> default 24.22.1

<span class="hljs-comment"># 2. Work from a shell where the new Node comes first in PATH.</span>
nvm use 24.22.1
<span class="hljs-built_in">command</span> -v node &amp;&amp; node --version

<span class="hljs-comment"># 3. See what each pm2 entry is pinned to.</span>
pm2 jlist | python3 -c <span class="hljs-string">'import json, sys
for a in json.load(sys.stdin):
    e = a["pm2_env"]
    print(a["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("pm_exec_path"), sep="  ")'</span>
</code></pre><p>Then move each entry according to what step 3 shows:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Fork mode, interpreter "node": pick up the new PATH.</span>
pm2 restart api --update-env

<span class="hljs-comment"># Interpreter is an absolute path to the old node, or the script is</span>
<span class="hljs-comment"># .../v24.14.0/bin/npm or npx: delete and start it again from this shell</span>
<span class="hljs-comment"># (or from your ecosystem file with the same name and arguments).</span>
pm2 delete web
pm2 start npm --name web -- start

<span class="hljs-comment"># Save the list that a reboot brings back.</span>
pm2 save

<span class="hljs-comment"># Regenerate the startup unit: the old one hard-codes the old PATH and pm2 path.</span>
pm2 unstartup     <span class="hljs-comment"># prints a sudo command; run it</span>
pm2 startup       <span class="hljs-comment"># prints a sudo command; run it</span>

<span class="hljs-comment"># Move the daemon, and every cluster-mode app with it.</span>
<span class="hljs-comment"># This restarts ALL apps at once: do it in a quiet window.</span>
pm2 update
</code></pre><p>The pm2 docs ask for the <a href="https://pm2.keymetrics.io/docs/usage/startup/" rel="noopener noreferrer"><code>unstartup</code> and <code>startup</code> pair</a> after every Node upgrade. The new unit is pinned again, this time to the new version. That is fine, as long as something checks it at the next upgrade.</p>
<p>Two details are easy to miss. <code>pm2 save</code> writes the saved list, including each entry's environment, so if you save before <code>--update-env</code> the reboot copy keeps the old <code>PATH</code>. And the daemon only reads the systemd unit when systemd starts it. If someone started pm2 by hand, the fixed unit applies at the next boot, not now.</p>
<h2>Cron jobs and helper scripts</h2><p>Cron runs with a minimal <code>PATH</code>, so jobs that need Node usually name it by absolute path, and that path names one install. The audit script below reads only your own <code>crontab -l</code>, so search the system crontabs and your scripts on every host too:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Paths inside a versioned nvm, fnm, n, volta or asdf install</span>
<span class="hljs-built_in">sudo</span> grep -rnE <span class="hljs-string">'/(\.nvm/versions/node|node-versions|n/versions/node|\.volta/tools/image/node|installs/nodejs)/v?[0-9]+\.[0-9]+\.[0-9]+'</span> \
  /etc/crontab /etc/cron.d /var/spool/cron /etc/systemd/system /usr/local/bin ~/bin 2&gt;/dev/null
</code></pre><p>Update the path in each match. A job that points at a version you later uninstall stops running, which at least fails loudly; one that points at the old version you kept for rollback keeps running vulnerable and says nothing.</p>
<h2>Containers</h2><p>A container keeps the Node it was built with. Rebuild with <code>--pull</code> so the builder fetches the new base image instead of a cached one, deploy by the new tag or digest, and check what is running:</p>
<pre><code class="hljs language-bash" tabindex="0">docker build --pull -t registry.example.com/app:2026-10-22 .

docker ps --format <span class="hljs-string">'{{.ID}} {{.Names}}'</span> | <span class="hljs-keyword">while</span> <span class="hljs-built_in">read</span> -r <span class="hljs-built_in">id</span> name; <span class="hljs-keyword">do</span>
  <span class="hljs-built_in">printf</span> <span class="hljs-string">'%-30s %s\n'</span> <span class="hljs-string">"<span class="hljs-variable">$name</span>"</span> <span class="hljs-string">"<span class="hljs-subst">$(docker exec -e NODE_OPTIONS= <span class="hljs-string">"<span class="hljs-variable">$id</span>"</span> node --version 2&gt;&amp;1)</span>"</span>
<span class="hljs-keyword">done</span>
</code></pre><p><code>node --version</code> inside a container reports the <code>node</code> on the container's <code>PATH</code>. If an image carries more than one Node, <code>docker exec &lt;id&gt; readlink /proc/1/exe</code> shows which binary the container's main process runs (when the image has <code>readlink</code>).</p>
<p>The repo's <code>registry_scan.py</code> read the Node version of 24 official <code>node</code> tags for amd64 and arm64 from registry metadata only (manifest and image config, no layers) at 06:23 UTC on 22 October. Every 22, 24 and 26 tag we sampled, and the <code>latest</code>, <code>lts</code> and <code>alpine</code> aliases, shipped the fixed version. How long they took:</p>
<p><strong>Official node images: hours from the release to build and to push</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Debian-based amd64: built</td>
<td>1 h</td>
<td>built</td>
</tr>
<tr>
<td>Debian-based amd64: pushed</td>
<td>4.1 h</td>
<td>pushed</td>
</tr>
<tr>
<td>Debian-based arm64: built</td>
<td>1 h</td>
<td>built</td>
</tr>
<tr>
<td>Debian-based arm64: pushed</td>
<td>4.1 h</td>
<td>pushed</td>
</tr>
<tr>
<td>Alpine amd64: built</td>
<td>12.4 h</td>
<td>built</td>
</tr>
<tr>
<td>Alpine amd64: pushed</td>
<td>13.1 h</td>
<td>pushed</td>
</tr>
<tr>
<td>Alpine arm64: built</td>
<td>13.5 h</td>
<td>built</td>
</tr>
<tr>
<td>Alpine arm64: pushed</td>
<td>16.1 h</td>
<td>pushed</td>
</tr>
</tbody></table>
<p><em>Reference: fixed binaries on nodejs.org/dist by 10:36 UTC on 21 Oct 2026. Built: the image config's created time. Pushed: Docker Hub's last_pushed time for that platform image, which is metadata, not proof of when a tag first served it. Latest value in each group of tags. Source: results/registry-20261022T062338Z-summary.txt.</em></p>
<p>If you rebuilt an Alpine-based image on the afternoon of 21 October, check it again: by this metadata the fixed Alpine images were built 12 to 13.5 hours after the release. The end-of-life tags will stay where they are: <code>node:18</code> (18.20.8, built March 2025), <code>node:20</code> (20.20.2, built April 2026) and <code>node:25</code> (25.9.0, built April and May 2026).</p>
<h2>Verify with /proc, not with names</h2><p>pm2 renames its daemon to <code>PM2 v6.0.14: God Daemon</code> and Next.js renames its server to <code>next-server</code>, and on our build machine the plain <code>node</code> processes showed up as <code>MainThread</code>. Those names are what <code>pgrep</code> matches and what <code>top</code> shows. On that machine <code>pgrep -c node</code> returned 0 while five processes ran a <code>node</code> binary (<code>results/pgrep-vs-proc-20261022T063334Z.txt</code>). The kernel's link to the executable does not lie:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Every process whose binary is node, with the binary it really runs (root sees all users)</span>
<span class="hljs-keyword">for</span> d <span class="hljs-keyword">in</span> /proc/[0-9]*; <span class="hljs-keyword">do</span>
  exe=$(<span class="hljs-built_in">readlink</span> <span class="hljs-string">"<span class="hljs-variable">$d</span>/exe"</span> 2&gt;/dev/null) || <span class="hljs-built_in">continue</span>
  <span class="hljs-keyword">case</span> <span class="hljs-string">"<span class="hljs-variable">${exe% (deleted)}</span>"</span> <span class="hljs-keyword">in</span> */node|*/nodejs) ;; *) <span class="hljs-built_in">continue</span> ;; <span class="hljs-keyword">esac</span>
  <span class="hljs-built_in">printf</span> <span class="hljs-string">'%-8s %-70s %s\n'</span> <span class="hljs-string">"<span class="hljs-variable">${d#/proc/}</span>"</span> <span class="hljs-string">"<span class="hljs-variable">$exe</span>"</span> <span class="hljs-string">"<span class="hljs-subst">$(tr '\0' ' ' &lt; <span class="hljs-string">"<span class="hljs-variable">$d</span>/cmdline"</span> | cut -c1-50)</span>"</span>
<span class="hljs-keyword">done</span>

<span class="hljs-comment"># The version of one of them, read from the binary without running it.</span>
<span class="hljs-comment"># Official builds compile in their headers URL. Do not run another user's</span>
<span class="hljs-comment"># binary with --version as root: that runs their code as root.</span>
<span class="hljs-built_in">sudo</span> grep -a -o -m1 -E <span class="hljs-string">'node-v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?-headers\.tar\.gz'</span> /proc/12345/exe
<span class="hljs-comment"># node-v24.14.0-headers.tar.gz</span>

<span class="hljs-comment"># Its children, which have their own binaries</span>
ps -o pid,args --forest -g <span class="hljs-string">"<span class="hljs-subst">$(ps -o sid= -p 12345 | tr -d ' ')</span>"</span>
</code></pre><p>An exe that ends in <code>(deleted)</code> means the file was replaced on disk after the process started, so the process still runs the old code until it restarts. For more on that pattern, and on proving a runtime is patched when the version string can mislead, see <a href="https://devops-daily.com/posts/php2shell-check-every-php-is-patched">PHP2Shell: How to Check That Every PHP You Run Is Actually Patched</a>.</p>
<h2>Run the audit script</h2><p><a href="https://github.com/The-DevOps-Daily/node-runtime-audit" rel="noopener noreferrer">The-DevOps-Daily/node-runtime-audit on GitHub</a></p>
<p><code>node_runtime_audit.py</code> is one file of standard-library Python. Its sources are <code>/proc</code>, the output of <code>pm2 jlist</code> and <code>crontab -l</code>, systemd unit files, and <code>node --version</code> inside running containers; besides those it only runs <code>systemctl show</code> for pm2 units and <code>docker</code> or <code>podman</code> <code>ps</code> and <code>inspect</code>, and looks at file metadata. For every running process it reads the real binary through <code>/proc/&lt;pid&gt;/exe</code> and the version compiled into that file, which works for deleted binaries and for processes inside containers. By default it never runs a binary it finds on the host, and it does not open the Node binaries that pm2 entries, crontabs and units name: their version comes from a running process that uses the same file, or from the path as a hint, where a path that names an unfixed version counts as vulnerable and one that names a fixed version stays unknown. Files are pinned with <code>O_PATH</code> and checked before they are read, so a planted FIFO or device cannot stall it. Command lines and container names are printed with control characters escaped. Run it as the user that owns the Node processes; root is not needed.</p>
<p>This is the real output on the Raspberry Pi we build on, run as the user that owns the Node processes, with <code>--redact</code>, which hides user names and app names. By the time of this run, the fnm default, the crontab and the pm2 systemd unit had already been moved to v24.22.1:</p>
<p><strong>node_runtime_audit.py on the Pi</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># 2026-10-22T07:10:33Z Raspberry Pi (Debian 12, aarch64), Node via fnm, pm2 6.0.14, run as the user that owns the Node processes</span>
$ python3 node_runtime_audit.py --redact
node-runtime-audit: Node.js security release of 2026-10-21, fixed releases v22.23.4, v24.22.1, v26.12.1
lines without a fix (end-of-life) count as vulnerable
note: covers the processes, crontab and pm2 of USER; run it as each user that owns Node processes

Running Node processes (real binary from /proc/&lt;pid&gt;/exe)
  [vulnerable]  pid 1883303  v24.14.0  USER       /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: PM2 v6.0.14: God Daemon (/home/USER/.pm2)
      parents: systemd(1)
      why: v24.14.0 &lt; v24.22.1
  [vulnerable]  pid 1883371  v24.14.0  USER       /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: node /home/USER/projects/app-1/node_modules/.bin/next dev
      parents: bun(1883370) &lt;- node(1883303) &lt;- systemd(1)
      why: v24.14.0 &lt; v24.22.1
  [vulnerable]  pid 1883390  v24.14.0  USER       /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: next-server (v16.2.6)
      parents: node(1883371) &lt;- bun(1883370) &lt;- node(1883303) &lt;- systemd(1)
      why: v24.14.0 &lt; v24.22.1
  [vulnerable]  pid 1884634  v24.14.0  USER       /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: node /home/USER/projects/app-1/.next/dev/build/postcss.js 38963
      parents: node(1883390) &lt;- node(1883371) &lt;- bun(1883370) &lt;- node(1883303)
      why: v24.14.0 &lt; v24.22.1
  183 processes of other <span class="hljs-built_in">users</span> not inspected: run the script as their owner

pm2 live list (pm2 jlist, PM2_HOME=/home/USER/.pm2)
  [unknown]     pm2 CLI /home/USER/.bun/bin/pm2 is writable by group or others (mode 777): not run, live list skipped
  [vulnerable]  daemon pid 1883303 v24.14.0 /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node

Installed Node versions (version managers)
  fnm <span class="hljs-keyword">in</span> /home/USER/.local/share/fnm
      installed: v24.14.0 (running), v24.14.1 (from path), v24.22.1 (from path)
      default <span class="hljs-built_in">alias</span> -&gt; v24.22.1 (from path)
  `node` on this shell<span class="hljs-string">'s PATH -&gt; v24.22.1 (from path) /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node
  for information only: these do not change the exit code

Crontab: absolute Node paths
  [unknown]     crontab -l (USER):33
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node -&gt; v24.22.1 (from path)  (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)
  [unknown]     crontab -l (USER):53
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node -&gt; v24.22.1 (from path)  (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)

systemd units: absolute Node paths
  [unknown]     /etc/systemd/system/pm2-USER.service:12
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin -&gt; v24.22.1 (from path)  (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)
      unit: pm2-USER.service ActiveState=inactive MainPID=0
      note: the unit is not what runs pm2 now; its PATH applies at the next boot

Containers
  no docker or podman CLI found

result: VULNERABLE: something runs or will start an unfixed Node (exit 1)
$ echo $?
1</span>
</code></pre><p>The fnm default, the shell's <code>node</code>, both cron jobs and the systemd unit point at the v24.22.1 install. The script reports the cron and unit paths as unknown rather than patched, because no running process uses that binary and it does not open it. The four processes that run are v24.14.0. The pm2 daemon is not running under the systemd unit (ActiveState=inactive), so the unit we had fixed is not what runs it now. The one pm2 app is started by <code>bun</code>, and the Node processes under it run the app's <code>node_modules/.bin/next</code> script. A separate check (<code>scripts/pm2-path-evidence.py</code>, <code>results/pm2-path-20261022T070504Z.txt</code>) shows that the process the pm2 daemon started and the saved pm2 list both have the v24.14.0 install first on <code>PATH</code>, which is where those Node processes come from. If the app is resurrected from that saved list after a reboot, it gets the same <code>PATH</code>; we did not reboot to confirm. The fix here is <code>pm2 restart &lt;app&gt; --update-env</code> from a fixed shell, then <code>pm2 save</code>, then moving the daemon.</p>
<p>The run found one more thing: the <code>pm2</code> CLI on that machine is world-writable (mode 777). The script will not run a CLI that someone else could have changed, so it skipped the live pm2 list and reported it as unknown. Check the permissions of your Node installs and global CLIs while you are in there.</p>
<p>The exit code is 1 when anything runs or will start a vulnerable Node, otherwise 2 when anything could not be checked, and 0 when it found nothing vulnerable or unknown within what it covers, including no Node at all. Unknown never counts as patched: a pre-release build, a distribution build linked to the system Brotli, a binary without an embedded version, a unit it could not read, a failed <code>pm2 jlist</code>, <code>crontab -l</code> or <code>docker ps</code>, a config path that only names a fixed version, or a cron path to a Node that no longer exists all give 2. It only sees what the user running it can see, so run it as each user that owns Node processes. To run it across hosts without copying it first:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-keyword">for</span> h <span class="hljs-keyword">in</span> deploy@web1 deploy@web2 deploy@web3; <span class="hljs-keyword">do</span>
  ssh <span class="hljs-string">"<span class="hljs-variable">$h</span>"</span> <span class="hljs-string">'python3 - --json'</span> &lt; node_runtime_audit.py &gt; <span class="hljs-string">"node-<span class="hljs-variable">${h#*@}</span>.json"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$h</span> exit $?"</span>
<span class="hljs-keyword">done</span>
</code></pre><p>The tests build fake hosts with the cases a version check gets wrong: a fake <code>node</code> that would leave a marker file if the script ever ran it, a binary replaced while it runs, a container binary, a Debian-style build, a pm2 entry whose config is fixed but whose process was never restarted, npm pinned to the old install while its child gets the new one, a cluster-mode app, an old binary in a directory named after a fixed version, end-of-life and pre-release versions, a cron job on a removed version, a shell wrapper that mentions a fixed version, units owned by other users, and FIFOs, devices, hard links and symlinks planted where a file is expected. One test starts a real process, deletes its binary and checks that the script still reads the running file.</p>
<h2>Rollback</h2><p>Keep the old version installed until every check passes. Rolling back is the same work in reverse: <code>nvm alias default 24.14.0</code>, then the same <code>--update-env</code>, delete-and-recreate, <code>pm2 save</code>, startup unit and <code>pm2 update</code> steps from a shell where the old version comes first. For containers, redeploy the previous image digest. A rollback puts the vulnerable version back, so treat it as a few hours to fix a regression, not as a resting state.</p>
<p>Once the audit shows nothing on the old version, <code>nvm uninstall 24.14.0</code>. Anything still pinned to it will then fail to start instead of quietly running the old code, and the audit script reports those paths as missing.</p>
<h2>What we could not check</h2><ul>
<li><strong>The bugs themselves.</strong> The advisory gives no technical detail, and we did not test whether any code path reaches the Brotli flaw. The exposure section is based on which code calls the decoder.</li>
<li><strong>Scores.</strong> The CVE records were still reserved when we checked, so there is no CVSS score.</li>
<li><strong>Distribution packages.</strong> Debian and Ubuntu had not published anything for CVE-2026-111497 when we checked. Check your distribution's <code>nodejs</code> and <code>brotli</code> advisories.</li>
<li><strong>Containers on a real host.</strong> The Pi has no Docker, so the container checks are covered by the test fixtures only. The registry scan read real image metadata.</li>
<li><strong>The downtime number.</strong> "Under a minute" comes from the production upgrade. The lab's <code>pm2 update</code> took 2.3 seconds because its apps do nothing.</li>
<li><strong>Bare <code>node</code> in cron jobs and units.</strong> The script checks absolute paths and <code>PATH=</code> lines; it does not resolve a bare <code>node</code> through the job's <code>PATH</code>. It reads only your own <code>crontab -l</code>, not the system crontabs or the saved pm2 list, and it does not follow <code>EnvironmentFile=</code>.</li>
<li><strong>Other platforms.</strong> The registry scan covers linux/amd64 and linux/arm64 only. The script needs Linux and <code>/proc</code>, and does not cover Kubernetes, stopped containers, or Node binaries renamed to something other than <code>node</code> (single executable applications, Electron).</li>
</ul>
<h2>Checklist</h2><ul>
<li><input disabled type="checkbox" /> Every host: Node 22.23.4, 24.22.1 or 26.12.1 or later installed; Node 20, 25 and older replaced</li>
<li><input disabled type="checkbox" /> Every pm2 entry: absolute interpreters and npm or npx script paths recreated, bare <code>node</code> entries restarted with <code>--update-env</code>, then <code>pm2 save</code></li>
<li><input disabled type="checkbox" /> pm2 startup unit regenerated (<code>pm2 unstartup</code>, <code>pm2 startup</code>) and the daemon moved with <code>pm2 update</code> in a quiet window</li>
<li><input disabled type="checkbox" /> Cluster-mode apps checked after the daemon moved</li>
<li><input disabled type="checkbox" /> Crontabs, systemd units and helper scripts searched for absolute paths to the old install</li>
<li><input disabled type="checkbox" /> Images rebuilt with <code>--pull</code> (Alpine-based ones checked after the late rebuild) and redeployed by new tag or digest</li>
<li><input disabled type="checkbox" /> Services that fetch user-supplied URLs or accept compressed bodies upgraded first; interim measures where they cannot be</li>
<li><input disabled type="checkbox" /> Every Node process checked by <code>/proc/&lt;pid&gt;/exe</code>, not by name, including child processes</li>
<li><input disabled type="checkbox" /> Node installs and global CLIs (pm2, npm) writable only by their owner</li>
<li><input disabled type="checkbox" /> Audit script output saved per host, and the old version uninstalled once it is clean</li>
</ul>
<p>Every fact in this post is listed with its source in the repo's <code>SOURCES.md</code>.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[PHP2Shell: How to Check That Every PHP You Run Is Actually Patched]]></title>
      <link>https://devops-daily.com/posts/php2shell-check-every-php-is-patched</link>
      <description><![CDATA[PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 fix two critical remote code execution bugs. Installing the update is the easy part. A patched Ubuntu PHP still says 8.3.6, running workers keep the old code, and 4.4 hours after the release 8 of 39 popular images still shipped a vulnerable PHP. Here is how to find the PHP you run and prove it is fixed.]]></description>
      <pubDate>Thu, 22 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/php2shell-check-every-php-is-patched</guid>
      <category><![CDATA[Linux]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Linux]]></category><category><![CDATA[PHP]]></category><category><![CDATA[Security]]></category><category><![CDATA[Docker]]></category><category><![CDATA[Patch Management]]></category><category><![CDATA[CVE]]></category>
      <content:encoded><![CDATA[<p>On 21 October 2026 the PHP project released PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 to fix CVE-2026-109618 and CVE-2026-109619, the pair now called PHP2Shell. Both are rated critical, CVSS v4.0 9.2. Unauthenticated remote code execution has been demonstrated against Apache with mod_php before any application code runs, and PHP says every installation that serves web requests is at risk.</p>
<p>So update. The trouble is that "we updated" can be true while the server is still vulnerable. Ubuntu and RHEL fix PHP without changing the version number, so <code>php -v</code> says the same thing before and after. PHP-FPM and Apache keep running the old code until they restart. A fixed base image fixes nothing until your image is rebuilt and the old containers are gone. This post shows how to find every PHP you run, how to prove each one is fixed, and what the interim settings do and do not block. It comes with a read-only check script, and a registry scan of 39 popular PHP images: by their own metadata, 8 still shipped a vulnerable PHP 4.4 hours after the scheduled release, and 3 still did at 5.5 hours.</p>
<h2>What to do now</h2><ol>
<li><strong>Update PHP everywhere</strong> to 8.5.12, 8.4.27, 8.3.36 or 8.2.35, or to your distribution's fixed package. On Ubuntu and RHEL the fixed package keeps the old upstream version, so check the package version, not <code>php -v</code>.</li>
<li><strong>Restart every PHP process</strong>: PHP-FPM, Apache with mod_php, php-cgi, LiteSpeed and FrankenPHP. Then confirm nothing still runs the replaced binary.</li>
<li><strong>Rebuild and redeploy containers</strong>, then check what the running containers contain, not what the registry has.</li>
<li><strong>If you cannot update today</strong>, set <code>session.upload_progress.enabled=0</code> (and the other settings below) in php.ini or the FPM pool, restart, and treat it as partial. PHP says only the update is a complete fix.</li>
<li><strong>PHP 8.1 and older get no upstream release.</strong> Use a vendor backport or move to 8.2 or newer.</li>
<li><strong>Run the check</strong> on every host and keep the output as your evidence.</li>
</ol>
<h2>Prerequisites</h2><ul>
<li>Shell access to the hosts that run PHP, and root if you want to see processes owned by other users.</li>
<li>Python 3.6 or newer for the check script. It uses only the standard library.</li>
<li>The docker or podman CLI on hosts that run PHP in containers, and <code>kubectl</code> for clusters.</li>
</ul>
<h2>What was fixed</h2><p>PHP published two advisories:</p>
<ul>
<li><strong>CVE-2026-109618</strong>, <a href="https://github.com/php/php-src/security/advisories/GHSA-w8hw-w9fx-j79w" rel="noopener noreferrer">GHSA-w8hw-w9fx-j79w</a>, "RCE via OOM corruption". Out-of-memory handling can interrupt PHP while request data is only partly initialized, and request cleanup can then treat that data as valid.</li>
<li><strong>CVE-2026-109619</strong>, <a href="https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp" rel="noopener noreferrer">GHSA-6wmf-4rv5-p3rp</a>, "RCE via stale global state". State owned by one request can survive after its memory is freed, and a later request handled by the same worker can use it.</li>
</ul>
<table>
<thead>
<tr>
<th>Branch</th>
<th>Fixed in</th>
<th>Note</th>
</tr>
</thead>
<tbody><tr>
<td>8.5</td>
<td>8.5.12</td>
<td></td>
</tr>
<tr>
<td>8.4</td>
<td>8.4.27</td>
<td></td>
</tr>
<tr>
<td>8.3</td>
<td>8.3.36</td>
<td></td>
</tr>
<tr>
<td>8.2</td>
<td>8.2.35</td>
<td>End of life on 31 December 2026</td>
</tr>
<tr>
<td>8.1 and older</td>
<td>no upstream release</td>
<td>Source backports for 7.4, 8.0 and 8.1 only</td>
</tr>
</tbody></table>
<p>The timeline, all on 21 October unless noted (UTC):</p>
<ul>
<li><strong>20 October</strong>: PHP announced the release for 21 October at 14:00 UTC and <a href="https://www.php.net/archive/2026.php#2026-10-20-1" rel="noopener noreferrer">reserved the right to release earlier</a> if details became public.</li>
<li><strong>02:29</strong>: VirusTotal saw the first submission of a <a href="https://blog.virustotal.com/2026/10/sizing-php-exposure-window-for-cve-2026.html" rel="noopener noreferrer">leaked archive</a> with technical details.</li>
<li><strong>13:34</strong>: the GitHub advisories were published. Ubuntu's <a href="https://ubuntu.com/security/notices/USN-8948-1" rel="noopener noreferrer">USN-8948-1</a> followed at 13:56.</li>
<li><strong>14:00</strong>: the scheduled release time. The php.net <a href="https://www.php.net/archive/2026.php#2026-10-21-5" rel="noopener noreferrer">announcements</a> carry no time of day; NVD published both CVEs at 14:17.</li>
<li><strong>16:03</strong>: the last of the 16 official <code>php</code> tags we scanned for 8.2 to 8.5 (linux/amd64) was built with a fixed PHP. Docker Hub showed the new version tags as pushed between 16:48 and 17:31.</li>
</ul>
<p>The first report took about ten weeks to reach a release. <a href="https://calif.io/php2shell/" rel="noopener noreferrer">Calif</a>, which credits the discovery to Calif, OpenAI and Wiz, dates discovery to 1 August 2026, the first report to PHP to 11 August and the second to 21 August. Calif traces the earliest affected code path to PHP 5.4.0, and says a related out-of-memory issue found in a 2024 audit was left unfixed because exploitation was not proven. In OpenAI's account, Greg Brockman <a href="https://x.com/gdb/status/2112921529650688312" rel="noopener noreferrer">wrote</a> that OpenAI had spent the past two months "leading a coalition to mitigate" the bug, found as part of its Patch the Planet initiative. OpenAI's own write-up is <a href="https://openai.com/index/coordinated-disclosure-of-critical-vulnerabilities/" rel="noopener noreferrer">here</a>.</p>
<h2>Who is really exposed</h2><p>PHP's <a href="https://www.php.net/archive/2026.php#2026-10-21-5" rel="noopener noreferrer">advisory</a> lists "all PHP installations serving web requests, including PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed" and puts mod_php at the top. The GitHub advisories go into more detail per server API (SAPI):</p>
<table>
<thead>
<tr>
<th>SAPI</th>
<th>What the PHP advisories say</th>
</tr>
</thead>
<tbody><tr>
<td>Apache mod_php</td>
<td>Highest risk. Unauthenticated RCE validated over HTTP with default limits, before any application code runs.</td>
</tr>
<tr>
<td>PHP-FPM</td>
<td>At risk. The CVE-2026-109619 session path was validated against FPM, but it needs a suitable application operation. For CVE-2026-109618, source analysis says FPM is susceptible, but no complete FPM exploit was shown. The advisory also names per-request settings sent by the web server (<code>PHP_VALUE</code> or <code>PHP_ADMIN_VALUE</code>) that produce a warning as a risk for FPM.</td>
</tr>
<tr>
<td>php-cgi</td>
<td>Listed as at risk by php.net. The advisories do not discuss it separately.</td>
</tr>
<tr>
<td>FrankenPHP</td>
<td>Listed. Source analysis says that, like mod_php, it is likely susceptible to CVE-2026-109618.</td>
</tr>
<tr>
<td>LiteSpeed (LSAPI)</td>
<td>Listed. Application-specific exploits are described as likely but not validated.</td>
</tr>
<tr>
<td>CLI</td>
<td>Not named. The bugs are in web request handling. But <code>php -S</code> serves HTTP from the CLI binary, and the CLI usually comes from the same source package as FPM and mod_php, so an old CLI is a sign of what else on the host is old.</td>
</tr>
</tbody></table>
<p>Red Hat <a href="https://security.access.redhat.com/data/csaf/v2/vex/2026/cve-2026-109618.json" rel="noopener noreferrer">rates both bugs Important, not Critical</a>, because "Standard Red Hat Enterprise Linux deployments use PHP-FPM rather than mod_php." That is a statement about the default, not about your server. PHP names mod_php as the highest-risk setup, so if you run mod_php on RHEL, put it first even though Red Hat's rating is Important.</p>
<blockquote>
<p><strong>Note</strong></p>
<p><strong>The sources disagree on the affected range.</strong> PHP's advisories and the CVE.org records list only 8.2 to 8.5. NVD marks every version below 8.2.35 as vulnerable, with no lower bound. Google Cloud says "All versions of PHP from 5.4 through 8.5.11", and PHP itself shipped source backports for 7.4, 8.0 and 8.1. Treat every PHP older than the fixed releases as affected, including end-of-life branches.</p>
</blockquote>
<h2>Why <code>php -v</code> does not tell you</h2><p>Debian ships the new upstream release, so on Debian <code>php -v</code> changes when you are fixed: bookworm gets 8.2.35 and trixie gets 8.4.27. Ubuntu and RHEL backport the patch into the version they already ship. Ubuntu 24.04's fixed php8.3 still reports 8.3.6. So does the vulnerable one.</p>
<p>To show this with real packages, the repo's <code>scripts/ubuntu-backport-demo.sh</code> downloads <code>php8.3-fpm</code> and <code>php8.3-common</code> from the Ubuntu archive, from before the fix (<code>.12</code>) and with the fix (<code>.13</code>). It extracts them without installing or running anything, reads the PHP version compiled into <code>php-fpm8.3</code>, and searches the package changelog for the CVE IDs:</p>
<p><strong>ubuntu-backport-demo.sh</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ scripts/ubuntu-backport-demo.sh
<span class="hljs-comment"># 2026-10-21T18:26:56Z Ubuntu 24.04 php8.3 packages from http://archive.ubuntu.com/ubuntu/pool</span>
sha256 60187aae56db2056e916d99ba4854415993429f12732ace1b5eb5225bb33f584  php8.3-fpm_8.3.6-0ubuntu0.24.04.12_amd64.deb
sha256 85e4fc03ff84fef7dfed923c99d1a2e0e6f49147a50bcd5eaca4e7cd05a835fa  php8.3-common_8.3.6-0ubuntu0.24.04.12_amd64.deb
sha256 ae395e2c15c3b96da56da807a204f771a589eb5bf54a3e5e239b1f50ade2815b  php8.3-fpm_8.3.6-0ubuntu0.24.04.13_amd64.deb
sha256 801cae4e842e1b9e9d60a542bf18ba831bf504b2f3070404892bf3f8fb943ea4  php8.3-common_8.3.6-0ubuntu0.24.04.13_amd64.deb

package 8.3.6-0ubuntu0.24.04.12
  php-fpm8.3 embeds: PHP/8.3.6
  changelog top:     php8.3 (8.3.6-0ubuntu0.24.04.12) noble-security; urgency=medium
  PHP2Shell IDs:     none
package 8.3.6-0ubuntu0.24.04.13
  php-fpm8.3 embeds: PHP/8.3.6
  changelog top:     php8.3 (8.3.6-0ubuntu0.24.04.13) noble-security; urgency=medium
  PHP2Shell IDs:     CVE-2026-109618 CVE-2026-109619 
</code></pre><p>Both binaries carry the same version. Only the package version and its changelog say which one has the fix. This is what the fixed packages looked like on the evening of 21 October:</p>
<table>
<thead>
<tr>
<th>System</th>
<th>Fixed package</th>
<th>Does <code>php -v</code> change?</th>
</tr>
</thead>
<tbody><tr>
<td>Debian 12 bookworm</td>
<td>php8.2 8.2.35-1~deb12u1 (<a href="https://security-tracker.debian.org/tracker/CVE-2026-109618" rel="noopener noreferrer">DLA-4848-1</a>)</td>
<td>Yes, to 8.2.35</td>
</tr>
<tr>
<td>Debian 13 trixie</td>
<td>php8.4 8.4.27-1~deb13u1 (DSA-6574-1)</td>
<td>Yes, to 8.4.27</td>
</tr>
<tr>
<td>Ubuntu 26.04</td>
<td>php8.5 8.5.4-0ubuntu1.5 (<a href="https://ubuntu.com/security/notices/USN-8948-1" rel="noopener noreferrer">USN-8948-1</a>)</td>
<td>No, stays 8.5.4</td>
</tr>
<tr>
<td>Ubuntu 24.04</td>
<td>php8.3 8.3.6-0ubuntu0.24.04.13</td>
<td>No, stays 8.3.6</td>
</tr>
<tr>
<td>Ubuntu 22.04</td>
<td>php8.1 8.1.2-1ubuntu2.28</td>
<td>No, stays 8.1.2</td>
</tr>
<tr>
<td>Ubuntu 14.04 to 20.04 (ESM)</td>
<td>php5, php7.0, php7.2, php7.4 (<a href="https://ubuntu.com/security/notices/USN-8948-2" rel="noopener noreferrer">USN-8948-2</a>)</td>
<td>No</td>
</tr>
<tr>
<td>RHEL 8.10 EUS, php:7.4 module</td>
<td>php-7.4.33-8.module+el8.10.0+25117+30337cca (RHSA-2026:87080)</td>
<td>No, stays 7.4.33</td>
</tr>
<tr>
<td>RHEL 8 php:8.2, RHEL 9, RHEL 10</td>
<td>No fix listed at 16:49 UTC</td>
<td></td>
</tr>
<tr>
<td>packages.sury.org (Debian 12, arm64)</td>
<td>Still at 8.2.34, 8.3.35, 8.4.26 and 8.5.11 at about 18:10 UTC</td>
<td>Yes, when it lands</td>
</tr>
<tr>
<td>rpms.remirepo.net</td>
<td>php.net: "The repo will not have updates today"</td>
<td>Yes, when it lands</td>
</tr>
</tbody></table>
<p>The same trap works in the other direction. A scanner that compares only the upstream version reports fixed Ubuntu and RHEL packages as vulnerable.</p>
<p>The right check compares the package version with your distribution's fixed version for that package, or finds both CVE IDs in the changelog of the package that owns the binary. The examples use Ubuntu 24.04's php8.3-fpm; take the fixed version for your release and package from its USN, DSA or RHSA:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Ubuntu 24.04, php8.3-fpm from the Ubuntu archive (fixed in USN-8948-1)</span>
dpkg --compare-versions <span class="hljs-string">"<span class="hljs-subst">$(dpkg-query -W -f='${Version}' php8.3-fpm)</span>"</span> ge 8.3.6-0ubuntu0.24.04.13 \
  &amp;&amp; <span class="hljs-built_in">echo</span> fixed || <span class="hljs-built_in">echo</span> NOT fixed

<span class="hljs-comment"># Which package owns the binary that actually runs, and what its changelog says</span>
dpkg-query -S <span class="hljs-string">"<span class="hljs-subst">$(readlink -f /usr/sbin/php-fpm8.3)</span>"</span>
zcat /usr/share/doc/php8.3-fpm/changelog.Debian.gz | <span class="hljs-built_in">head</span> -1
<span class="hljs-keyword">for</span> <span class="hljs-built_in">id</span> <span class="hljs-keyword">in</span> CVE-2026-109618 CVE-2026-109619; <span class="hljs-keyword">do</span>
  zgrep -q <span class="hljs-string">"<span class="hljs-variable">$id</span>"</span> /usr/share/doc/php8.3-fpm/changelog.Debian.gz &amp;&amp; <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$id</span> listed"</span> || <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$id</span> NOT listed"</span>
<span class="hljs-keyword">done</span>

<span class="hljs-comment"># RHEL, AlmaLinux, Rocky Linux: ask about the package that owns the binary</span>
rpm -qf <span class="hljs-string">"<span class="hljs-subst">$(readlink -f /usr/sbin/php-fpm)</span>"</span>
<span class="hljs-keyword">for</span> <span class="hljs-built_in">id</span> <span class="hljs-keyword">in</span> CVE-2026-109618 CVE-2026-109619; <span class="hljs-keyword">do</span>
  rpm -qf --changelog <span class="hljs-string">"<span class="hljs-subst">$(readlink -f /usr/sbin/php-fpm)</span>"</span> | grep -q <span class="hljs-string">"<span class="hljs-variable">$id</span>"</span> &amp;&amp; <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$id</span> listed"</span> || <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$id</span> NOT listed"</span>
<span class="hljs-keyword">done</span>

<span class="hljs-comment"># Alpine packages carry the upstream version</span>
apk info -v | grep -E <span class="hljs-string">'^php[0-9]+(-fpm|-cgi|-apache2|-embed)?-[0-9]'</span>
</code></pre><p>Two details make the changelog check less simple than it looks:</p>
<ul>
<li><strong>On Debian and Ubuntu, <code>/usr/share/doc/php8.3-fpm</code> is usually a symlink</strong> to <code>php8.3-common</code>'s docs. If an upgrade was partial and <code>php8.3-fpm</code> was held back, you read the newer package's changelog and conclude the old FPM binary is fixed. Check that the first line of the changelog names the version that <code>dpkg-query -W php8.3-fpm</code> reports. The check script refuses a changelog whose top entry does not match the installed package.</li>
<li><strong>Container images often strip <code>/usr/share/doc</code>.</strong> Then there is no changelog to read, and the package version against the distribution's tracker is the only evidence.</li>
<li><strong>A changelog line is a mention, not proof.</strong> A fix that was later reverted would still be listed. Treat it as supporting evidence next to the package version.</li>
</ul>
<h2>Restart, or the old code keeps running</h2><p>Upgrading the package replaces the files on disk. The PHP-FPM master and its workers, and every Apache child with mod_php loaded, still have the old binary mapped in memory. Linux marks a mapped file that has been replaced as <code>(deleted)</code>, which gives you a direct check:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Run as root, or processes owned by other users are not visible.</span>
<span class="hljs-comment"># PHP-FPM, php-cgi, LiteSpeed and FrankenPHP processes on a replaced binary</span>
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">ls</span> -l /proc/[0-9]*/exe 2&gt;/dev/null | grep -E <span class="hljs-string">'(php|frankenphp).*\(deleted\)'</span>

<span class="hljs-comment"># Apache processes that still map a replaced libphp</span>
<span class="hljs-built_in">sudo</span> grep -l <span class="hljs-string">'libphp.*(deleted)'</span> /proc/[0-9]*/maps 2&gt;/dev/null
</code></pre><p>Restart the service (<code>systemctl restart php8.3-fpm</code>, <code>systemctl restart apache2</code>), run both commands again until they print nothing, and then check the version or package of the binary the service runs. An empty result only says no replaced file is still mapped; it does not say the file on disk is the fixed one. <code>needrestart</code> on Debian and Ubuntu and <code>dnf needs-restarting</code> on RHEL also list processes that need a restart after an upgrade. PHP's own advice for the interim settings is to "restart every worker" as well, because a setting changed in php.ini does not reach a worker that is already running.</p>
<h2>Containers: rebuild, redeploy, then check what is running</h2><p>A container keeps the PHP it was built with. Google's <a href="https://cloud.google.com/blog/products/identity-security/responding-to-core-php-vulnerabilities-cve-2026-109618-cve-2026-109619" rel="noopener noreferrer">guidance for Cloud Run</a> says it plainly: services without automatic base image updates "must be rebuilt and redeployed", and "Container images you build yourself are your responsibility."</p>
<p>Rebuild with <code>--pull</code>, so the builder fetches the new base instead of a cached one:</p>
<pre><code class="hljs language-bash" tabindex="0">docker build --pull -t registry.example.com/app:2026-10-21 .
</code></pre><p>If your Dockerfile pins a digest (<code>FROM php:8.4-fpm@sha256:...</code>), update the pin. If it pins a version tag, move to <code>8.4.27-fpm</code> or later. Then deploy the new tag or digest. On Kubernetes, <code>kubectl rollout restart</code> with a reused tag and <code>imagePullPolicy: IfNotPresent</code> starts the new pods from the image already cached on the node, which is the old one.</p>
<p>Then check the containers that are running, not the image you meant to deploy:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Docker hosts (use podman the same way): the PHP CLI version in each running container</span>
docker ps --format <span class="hljs-string">'{{.ID}} {{.Image}} {{.Names}}'</span> | <span class="hljs-keyword">while</span> <span class="hljs-built_in">read</span> -r <span class="hljs-built_in">id</span> image name; <span class="hljs-keyword">do</span>
  <span class="hljs-built_in">printf</span> <span class="hljs-string">'%-30s %-40s '</span> <span class="hljs-string">"<span class="hljs-variable">$name</span>"</span> <span class="hljs-string">"<span class="hljs-variable">$image</span>"</span>
  docker <span class="hljs-built_in">exec</span> <span class="hljs-string">"<span class="hljs-variable">$id</span>"</span> php -r <span class="hljs-string">'echo PHP_VERSION, PHP_EOL;'</span> 2&gt;&amp;1 || <span class="hljs-built_in">echo</span> <span class="hljs-string">"PHP CLI check failed"</span>
<span class="hljs-keyword">done</span>

<span class="hljs-comment"># Kubernetes: the image digests your pods are actually running</span>
kubectl get pods -A -o jsonpath=<span class="hljs-string">'{..imageID}'</span> | <span class="hljs-built_in">tr</span> <span class="hljs-string">' '</span> <span class="hljs-string">'\n'</span> | <span class="hljs-built_in">sort</span> -u
</code></pre><p>The CLI version is a first look, not proof for the web server: a container can have a fixed CLI and an older FPM or mod_php, so check the binary that serves requests (<code>php-fpm -v</code>, or its package) as well. The repo's <code>registry_scan.py</code> takes repository-qualified digests (<code>registry/repository@sha256:...</code>) as arguments and reads the PHP version from each image's metadata without pulling it. Some runtimes report <code>imageID</code> in other forms, such as a bare digest, which you need to map back to a repository first. That works for images built <code>FROM php</code>, which carry <code>ENV PHP_VERSION</code>. For images that install PHP from distribution packages, look inside a running container, because the version is not in the metadata.</p>
<h2>How fast the popular images were rebuilt</h2><p>We wanted to know how long "pull the new image" takes to become true, so <code>registry_scan.py</code> read the PHP version of 39 popular images from registry metadata only: the manifest and the image config, a few KB each, checked against their digests, never a layer. The version is the one the image declares in <code>ENV PHP_VERSION</code> (or Bitnami's <code>APP_VERSION</code>), which images built <code>FROM php</code> inherit. An image could replace PHP after that, so this is what the images report, not a look at the binary. The list is the official <code>php</code> image for 8.1 to 8.5, the official application images built on it, and widely used community images. <code>images.txt</code> in the repo says why each one is there. All results are for linux/amd64.</p>
<p>We ran the scan at 18:21 and 19:32 UTC on 21 October, 4.4 and 5.5 hours after the scheduled release, and again at 18:27 UTC on 22 October, about 24 hours later:</p>
<p><strong>39 popular PHP images by the PHP version their metadata reports</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>21 Oct 18:21: vulnerable</td>
<td>8</td>
<td>vulnerable</td>
</tr>
<tr>
<td>21 Oct 18:21: unknown</td>
<td>2</td>
<td>unknown</td>
</tr>
<tr>
<td>21 Oct 18:21: patched</td>
<td>29</td>
<td>patched</td>
</tr>
<tr>
<td>21 Oct 19:32: vulnerable</td>
<td>3</td>
<td>vulnerable</td>
</tr>
<tr>
<td>21 Oct 19:32: unknown</td>
<td>2</td>
<td>unknown</td>
</tr>
<tr>
<td>21 Oct 19:32: patched</td>
<td>34</td>
<td>patched</td>
</tr>
<tr>
<td>22 Oct 18:27: vulnerable</td>
<td>3</td>
<td>vulnerable</td>
</tr>
<tr>
<td>22 Oct 18:27: unknown</td>
<td>2</td>
<td>unknown</td>
</tr>
<tr>
<td>22 Oct 18:27: patched</td>
<td>34</td>
<td>patched</td>
</tr>
</tbody></table>
<p><em>linux/amd64. PHP version from ENV PHP_VERSION (or Bitnami APP_VERSION) in each image config, read without pulling layers. Fixed: 8.5.12, 8.4.27, 8.3.36, 8.2.35; 8.1 and older count as vulnerable. Unknown: no full version in the metadata. Files: results/scan-20261021T182129Z.json, results/scan-20261021T193200Z.json, results/scan-20261022T182715Z.json.</em></p>
<ul>
<li><strong>All 16 official <code>php</code> tags we scanned for 8.2 to 8.5 reported a fixed PHP</strong> in every run, built 1.8 to 2.1 hours after the scheduled release.</li>
<li><strong>At 18:21, 8 images still shipped a vulnerable PHP</strong>: <code>php:8.1-fpm</code> and <code>php:8.1-apache</code> (8.1.34, last built in December 2025; there is no fixed 8.1 release to rebuild with), <code>joomla:latest</code> (8.4.26), <code>nextcloud:latest</code> and <code>nextcloud:fpm</code> (8.5.11), <code>phpmyadmin:latest</code> (8.3.35), <code>yourls:latest</code> (8.5.11) and <code>laravelsail/php84-composer:latest</code> (8.4.1, built in November 2024). The Laravel documentation uses that image to run Composer, not to serve traffic.</li>
<li><strong>At 19:32, 3 were still vulnerable</strong>: the two <code>php:8.1</code> tags and <code>laravelsail/php84-composer:latest</code>. <code>joomla</code>, both <code>nextcloud</code> tags, <code>phpmyadmin</code> and <code>yourls</code> had fixed images by then.</li>
<li><strong>About 24 hours later, at 18:27 UTC on 22 October, the same 3 were still vulnerable</strong>: <code>php:8.1-fpm</code>, <code>php:8.1-apache</code> and <code>laravelsail/php84-composer:latest</code>. None of them had a new build.</li>
<li><strong>2 images could not be judged from metadata</strong> in any run: <code>thecodingmachine/php:8.4-v5-fpm</code> sets <code>PHP_VERSION=8.4</code> without the patch level, and <code>linuxserver/nextcloud</code> installs PHP from Alpine packages and records no PHP version at all.</li>
</ul>
<p><strong>Hours from the scheduled 14:00 UTC release to the image build</strong></p>
<table>
<thead>
<tr>
<th>Series</th>
<th>Samples</th>
<th>Min</th>
<th>Median</th>
<th>p95</th>
<th>Max</th>
</tr>
</thead>
<tbody><tr>
<td>Official php 8.2 to 8.5 (16)</td>
<td>16</td>
<td>1.83 h</td>
<td>1.935 h</td>
<td>2.06 h</td>
<td>2.06 h</td>
</tr>
<tr>
<td>Official app images (13)</td>
<td>13</td>
<td>2.47 h</td>
<td>2.53 h</td>
<td>4.08 h</td>
<td>4.08 h</td>
</tr>
<tr>
<td>Community images (5)</td>
<td>5</td>
<td>2.23 h</td>
<td>3.68 h</td>
<td>4.05 h</td>
<td>4.05 h</td>
</tr>
</tbody></table>
<p><em>Patched images only, by the 'created' time in each linux/amd64 image config, from results/scan-20261021T193200Z.json. This is when the image was built, not when the registry started serving it. Images still on a vulnerable PHP are not on this chart.</em></p>
<p>One result changes how you should read a registry. Docker Hub listed <code>joomla:latest</code> as pushed at 15:46 UTC, after the release. Its amd64 image was built at 01:52 UTC with PHP 8.4.26. So a fresh "last pushed" time is not evidence that the image you pull has a fixed PHP. The build time is not either: the fixed <code>joomla</code>, <code>phpmyadmin</code> and <code>yourls</code> images that the registry served at 19:32 were built between 16:28 and 16:36 UTC, but at 18:30 the registry still served the old ones for amd64. <code>nextcloud:latest</code> and <code>nextcloud:fpm</code> looked the same at 18:21; by 18:30 their amd64 images had moved to 8.5.12. To check the method, at 18:30 UTC we compared the amd64 digest from the Docker Hub API with the registry's own index for those three images and two others: they agreed for all five (<code>results/method-check-20261021T183037Z.txt</code>).</p>
<h2>The interim settings and their limits</h2><p>If you cannot update today, PHP lists <a href="https://www.php.net/archive/2026.php#2026-10-21-5" rel="noopener noreferrer">settings that reduce exposure</a> to specific paths. Its own words: they "reduce exposure to specific attack paths but do not make an installation safe."</p>
<p>For CVE-2026-109619, turn off session upload progress, and on PHP 8.5 fatal error backtraces. Set them where they apply before PHP parses the request: php.ini, the FPM pool, or the Apache config. PHP says not to rely on <code>ini_set()</code> or <code>.user.ini</code>, which apply too late.</p>
<pre><code class="hljs language-ini" tabindex="0"><span class="hljs-comment">; php.ini (and the php.ini of every SAPI: cli, fpm, apache2, cgi)</span>
<span class="hljs-attr">session.upload_progress.enabled</span> = <span class="hljs-number">0</span>
<span class="hljs-comment">; PHP 8.5 only</span>
<span class="hljs-attr">fatal_error_backtraces</span> = <span class="hljs-number">0</span>
</code></pre><pre><code class="hljs language-ini" tabindex="0"><span class="hljs-comment">; PHP-FPM pool, for example /etc/php/8.3/fpm/pool.d/www.conf</span>
php_admin_flag<span class="hljs-section">[session.upload_progress.enabled]</span> = off
</code></pre><pre><code class="hljs language-apache" tabindex="0"><span class="hljs-comment"># Apache with mod_php</span>
<span class="hljs-attribute">php_admin_flag</span> session.upload_progress.enabled <span class="hljs-literal">off</span>
</code></pre><p>Then restart every worker. What this does and does not do, per the <a href="https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp" rel="noopener noreferrer">advisory</a>:</p>
<ul>
<li>It "prevents the demonstrated early session producing and consuming operations. It does not mitigate the other lifecycle paths." <code>fatal_error_backtraces=0</code> "blocks the confirmed fatal backtrace path only."</li>
<li>PHP lists it under CVE-2026-109619 only. For CVE-2026-109618 PHP suggests enforcing <code>memory_limit=-1</code> so applications cannot override it, and warns that an underlying allocation failure can still reach the bug. With no PHP memory limit, use a container or cgroup limit as the availability boundary, and do not use <code>RLIMIT_AS</code> or <code>RLIMIT_DATA</code>, which can cause the failure you are trying to avoid.</li>
<li>Hiding warnings does not help. Changing <code>error_reporting</code>, <code>display_errors</code> or <code>log_errors</code>, or using <code>@</code>, does not protect against CVE-2026-109619. Instead, remove per-request settings that produce warnings or deprecation notices, including <code>PHP_VALUE</code> or <code>PHP_ADMIN_VALUE</code> lines your web server passes to FPM.</li>
<li>If your application does not use multipart form data, PHP suggests blocking it at the frontend. It also suggests a low <code>post_max_size</code> (8M for applications that do not need file uploads, 32M for those that do) and proxy limits that also cover requests without a usable Content-Length.</li>
<li>Edge rules help but do not replace the update. <a href="https://www.helpnetsecurity.com/2026/10/21/php2shell-php-vulnerabilities-patched-cve-2026-109618-cve-2026-109619/" rel="noopener noreferrer">Help Net Security</a> reports that Cloudflare deployed WAF rules for the known exploit vectors to all customers, including Free, and <a href="https://blog.virustotal.com/2026/10/sizing-php-exposure-window-for-cve-2026.html" rel="noopener noreferrer">VirusTotal</a> reports its CTO saying that only updating PHP fully protects a site.</li>
</ul>
<p>PHP also suggests keeping <code>session.upload_progress.enabled=0</code> after you patch if you do not use upload progress bars, "as more attackers try to find flaws in this hardly-used and hardly-tested feature."</p>
<h2>PHP 8.1 and older</h2><p>The PHP project does not release fixes for 8.1 or older. It published minimal source backports for 7.4, 8.0 and 8.1 for people who build from source, and points everyone else to their vendor. Some vendors have shipped them: Ubuntu for 22.04's PHP 8.1 and, under ESM, for 14.04 to 20.04, and Red Hat for the RHEL 8.10 EUS php:7.4 module.</p>
<p>The official <code>php:8.1</code> tags we scanned still reported 8.1.34, built in December 2025. Google's <a href="https://cloud.google.com/blog/products/identity-security/responding-to-core-php-vulnerabilities-cve-2026-109618-cve-2026-109619" rel="noopener noreferrer">guidance</a> says App Engine applications on PHP 5.5, 7.2, 7.3, 7.4 and 8.1 remain vulnerable and must move to 8.2 or newer. VirusTotal <a href="https://blog.virustotal.com/2026/10/sizing-php-exposure-window-for-cve-2026.html" rel="noopener noreferrer">counted</a> about 758,000 internet-facing hosts that announce an end-of-life PHP version from 5.4 to 8.1, out of about 1.1 million that announce a version at all. That counts exposed PHP, not confirmed vulnerable hosts, but it shows how much of the long tail has no simple update.</p>
<p>If you are on 8.2, the same applies on 31 December 2026, when it reaches end of life.</p>
<h2>Run the check</h2><p><a href="https://github.com/The-DevOps-Daily/php2shell-check" rel="noopener noreferrer">The-DevOps-Daily/php2shell-check on GitHub</a></p>
<p><code>php2shell_check.py</code> is one file of standard-library Python, so you can copy it to a server and run it. For each PHP it finds, it reports the SAPI, the version compiled into the binary or Apache module (so mod_php is checked without running Apache; <code>-v</code> is the fallback for binaries without a readable string), the package that owns the file, and whether that package's changelog names both vulnerabilities by CVE or GHSA ID. A changelog only counts if it belongs to the installed package and <code>dpkg --verify</code> or <code>rpm -V</code> confirms the binary still matches that package. It also reports the interim settings from a fresh <code>php -i</code>, the FPM pool files, or the parsed php.ini files for mod_php, plus host web server lines that set them or pass <code>PHP_VALUE</code>. A running service started with other options can differ. Then it lists running PHP processes and marks any that still run a replaced binary. Through the first container CLI it finds (docker, then podman), it checks the PHP binaries inside each running container that has a PHP process, and adds an <code>unknown</code> result when no checked binary covers the kind of PHP process running there (for example an FPM process but only a CLI binary found). The interim settings are reported only: they never turn a vulnerable result into a pass.</p>
<p>This is the real output on the Raspberry Pi we build on, which has the PHP 8.4.23 CLI from packages.sury.org and no web server:</p>
<p><strong>php2shell-check on the Pi</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># 2026-10-21T18:45:30Z Raspberry Pi, Debian 12, PHP CLI from packages.sury.org, no web server, run as a normal user</span>
$ python3 php2shell_check.py
php2shell-check: fixed releases 8.2.35, 8.3.36, 8.4.27, 8.5.12

PHP on disk
  [vulnerable]   /usr/bin/php8.4                    cli
      version: 8.4.23 (embedded string)
      exposure: CLI: not a web SAPI unless it runs `php -S`; same package line often ships FPM/mod_php
      also as: /bin/php, /bin/php8.4, /usr/bin/php
      package: php8.4-cli 8.4.23-1+0~20260703.52+debian12~1.gbp817a0c (dpkg); CVE IDs <span class="hljs-keyword">in</span> changelog: none
      why: upstream 8.4.23 &lt; 8.4.27
      session.upload_progress.enabled: On  [php8.4 -i]
      memory_limit=-1  post_max_size=8M

Running PHP processes
  none

Containers
  none checked
note: no docker or podman CLI found; container processes are judged by the version <span class="hljs-keyword">in</span> /proc

result: VULNERABLE or restart needed (<span class="hljs-built_in">exit</span> 1)
$ <span class="hljs-built_in">echo</span> $?
1
</code></pre><p>The exit code is 1 when anything is vulnerable or needs a restart. Otherwise it is 2 when anything could not be checked, and 0 when every PHP it found is patched or it found no PHP at all (it says which). Anything left <code>unknown</code>, such as a changelog that lists only one of the two CVEs on an older version, a pre-release build, or an unreadable process, gives 2 rather than 0, unless something else is vulnerable, which gives 1. It only knows the paths and names it searches, so a 0 means "nothing vulnerable found", not proof that no other PHP exists. To run it across hosts without copying it first:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-keyword">for</span> h <span class="hljs-keyword">in</span> web1 web2 web3; <span class="hljs-keyword">do</span>
  ssh <span class="hljs-string">"<span class="hljs-variable">$h</span>"</span> <span class="hljs-string">'sudo python3 - --json'</span> &lt; php2shell_check.py &gt; <span class="hljs-string">"php2shell-<span class="hljs-variable">$h</span>.json"</span>
  <span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$h</span> exit $?"</span>
<span class="hljs-keyword">done</span>
</code></pre><p>The test suite builds fake hosts with the cases a version check gets wrong: an Ubuntu backport, the same version without the update, a partial upgrade whose symlinked changelog belongs to another package, an old binary copied over a patched package, a changelog with only one of the two CVEs, a source build where FPM was left on the old version, a process running a deleted binary, and containers on old and new images.</p>
<h2>How to prove you are fixed</h2><p>For each host, keep evidence of all of these, not just the first:</p>
<ol>
<li><strong>Package</strong>: the upstream version is at or above the fixed release, or the package version is at or above your distribution's fixed version.</li>
<li><strong>Processes</strong>: no PHP-FPM, php-cgi, LiteSpeed or FrankenPHP process runs a <code>(deleted)</code> binary, and no Apache process maps a <code>(deleted)</code> libphp.</li>
<li><strong>Containers</strong>: the binary that serves requests inside each running container (<code>php-fpm</code>, mod_php, FrankenPHP) has a fixed version or a fixed package, not just the CLI. The image digest you deployed is the one running.</li>
<li><strong>Images</strong>: every image you build FROM a PHP base has been rebuilt since the fix. The registry scan of its digest is a quick inventory check; the container check above is the proof.</li>
<li><strong>Exposure while vulnerable</strong>: if a host was internet-facing and unpatched, check it for compromise. The <a href="https://www.ncsc.gov.uk/news/critical-security-issue-affecting-php" rel="noopener noreferrer">NCSC</a> asks for this, and <a href="https://www.wiz.io/blog/php2shell-php-rce" rel="noopener noreferrer">Wiz</a> suggests looking for unusual multipart requests followed by memory-limit errors or repeated HTTP 500 responses, unexpectedly large or malformed <code>PHPSESSID</code> response cookies, and unexplained worker churn.</li>
</ol>
<h2>What we could not check</h2><ul>
<li><strong>Containers on a real host.</strong> The Pi we build on has no Docker, so the container checks are covered by the test fixtures only. The registry scan read real images.</li>
<li><strong>Web servers.</strong> The Pi runs PHP from the command line only. mod_php and FPM detection are covered by the tests.</li>
<li><strong>Whether the interim settings block anything.</strong> We did not test any part of the bugs, and the tool does not. PHP's own text is the source for what the settings cover.</li>
<li><strong>Later distribution updates.</strong> The distribution table is as of the evening of 21 October. At 16:49 UTC Red Hat listed a fix only for the RHEL 8.10 EUS php:7.4 module, the Debian 12 arm64 index on packages.sury.org still had the old versions at about 18:10 UTC, and PHP said Remi would not have updates that day.</li>
<li><strong>Other architectures.</strong> The registry scan reads linux/amd64 only. We did not measure arm64 images.</li>
<li><strong>Unusual install paths.</strong> The script searches distribution paths, Remi, cPanel, Plesk, CloudLinux alt-php, LiteSpeed, aaPanel and <code>$PATH</code>. PHP elsewhere, <code>.htaccess</code> overrides and stopped containers are not covered.</li>
</ul>
<h2>Checklist</h2><ul>
<li><input disabled type="checkbox" /> Every host: PHP package updated, checked against the distribution's fixed version, not <code>php -v</code></li>
<li><input disabled type="checkbox" /> Every PHP-FPM pool, Apache, php-cgi, LiteSpeed and FrankenPHP service restarted, with no <code>(deleted)</code> PHP binaries left in <code>/proc</code></li>
<li><input disabled type="checkbox" /> Every image built FROM a PHP base rebuilt with <code>--pull</code> and redeployed by new tag or digest</li>
<li><input disabled type="checkbox" /> Running containers checked inside, including the binary that serves requests, and their image digests matched to what you deployed</li>
<li><input disabled type="checkbox" /> PHP 8.1 and older: vendor backport installed or upgrade to 8.2+ scheduled, with interim settings in place until then</li>
<li><input disabled type="checkbox" /> Interim settings applied where updates wait, in php.ini or the pool, with workers restarted</li>
<li><input disabled type="checkbox" /> <code>session.upload_progress.enabled=0</code> kept where upload progress bars are not used</li>
<li><input disabled type="checkbox" /> Internet-facing hosts that were unpatched checked for signs of compromise</li>
<li><input disabled type="checkbox" /> Check script output saved per host as evidence</li>
</ul>
<p>Every fact in this post is listed with its source in the repo's <code>SOURCES.md</code>, together with the points where the sources disagree.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[A Service Catalog Nobody Updates Can Be Worse Than None: We Checked 217 Entries Against GitHub]]></title>
      <link>https://devops-daily.com/posts/service-catalog-drift</link>
      <description><![CDATA[We checked 217 public Backstage catalog entries from 30 organizations against GitHub. One in ten had a record GitHub contradicts, three owners in four could not be checked from outside, and where the catalog and CODEOWNERS both named an owner, they agreed only half the time. Here is the checker, the method, and how to keep a catalog honest.]]></description>
      <pubDate>Thu, 22 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/service-catalog-drift</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[Backstage]]></category><category><![CDATA[Service Catalog]]></category><category><![CDATA[Platform Engineering]]></category><category><![CDATA[CODEOWNERS]]></category><category><![CDATA[GitHub]]></category><category><![CDATA[Internal Developer Portal]]></category>
      <content:encoded><![CDATA[<p>We took 217 service catalog entries that 30 organizations publish on GitHub, in the Backstage <code>catalog-info.yaml</code> format, and checked each one against what GitHub says today. For one in ten (21 entries), GitHub contradicted a record: in 9 the catalog entry itself was wrong (a repository renamed while the catalog kept the old name, a repository reference without its owner, a dead link, no owner at all), and in 12 the CODEOWNERS file next to it named people or teams GitHub no longer accepts. That is a floor, not a total. For three entries in four, the owner could not be checked at all from outside the organization. And where the catalog and CODEOWNERS both named an owner for the same file, the two agreed, with every owner resolving, only half the time.</p>
<p>That matters most at the worst moment. During an incident, someone opens the catalog to find out who owns the failing service. With no entry, they go and ask. With a wrong entry, they can page a team that was reorganized away, or a person who left, and do it with confidence because the catalog said so. We did not measure that happening; it is the reason the measurement is worth doing. This post shows the checker we built, how we picked the sample, what it found, the bugs two reviews found in our own method, and the habits that keep a catalog true.</p>
<h2>TL;DR</h2><ul>
<li>217 Component entries from 214 public repositories in 30 GitHub organizations, checked on 2026-10-22: <strong>21 (9.7%) had at least one record GitHub contradicts</strong>, 9 of them in the catalog entry itself and 12 only in the CODEOWNERS file next to it. 62 (28.6%) had only warnings, and 134 (61.8%) had nothing we could find.</li>
<li>Ownership is where the records disagree. Of 73 entries whose catalog file is also covered by a CODEOWNERS rule, 37 agree and every owner resolves. 14 name owners that GitHub cannot resolve, and 22 name different owners (in 7 of those, one team name contains the other, which may be the same team).</li>
<li>159 entries (73.3%) have an owner nobody outside the org can check: GitHub teams are only visible to members, and some owners come from other directories. With a token that can read your org, the script checks teams and members directly.</li>
<li>Links barely rotted: 2 of 152 were dead. But only 70 entries had any link, and none was titled as a runbook or a dashboard. We could not test the links that probably matter most during an incident.</li>
<li>We expected older catalog files to drift more. Drift was 8.0% for files changed in the last year and 11.4% for older ones, a gap this sample cannot tell apart from chance.</li>
<li>The fixes are boring and they work: derive owners from the source of truth, check <code>catalog-info.yaml</code> in CI, use GitHub's own CODEOWNERS validation, and retire entries nobody claims.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Python 3.10 or later (the checker uses only the standard library)</li>
<li>A GitHub token in <code>GITHUB_TOKEN</code>, or a logged-in <code>gh</code> CLI. To check owners inside your own org, the token needs to read the org's teams and members</li>
<li>A catalog in the Backstage <code>catalog-info.yaml</code> format, or the public sample in the repository</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/service-catalog-drift" rel="noopener noreferrer">The-DevOps-Daily/service-catalog-drift on GitHub</a></p>
<h2>Why a catalog drifts</h2><p>A catalog entry is usually written once, by a scaffolder template, on the day the repository is created. After that, teams get reorganized and renamed, people move on, repositories are renamed or transferred, and documentation moves from one wiki to the next. None of those changes touch <code>catalog-info.yaml</code>.</p>
<p>None of them break anything either. Backstage's own <a href="https://backstage.io/docs/features/software-catalog/faq" rel="noopener noreferrer">FAQ</a>, answering whether a plugin should reject entities that fail validation, says: "Never throw errors due to "soft" errors, in particular relations not matching an existing target". So an owner that points at a team that no longer exists loads and shows up like any other. GitHub <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/renaming-a-repository" rel="noopener noreferrer">redirects a renamed repository</a>, so a catalog that still uses the old name keeps working, and nobody has a reason to look. The catalog stays green while it drifts away from the truth.</p>
<p>That is why a stale catalog can be worse than none. An empty owner field sends you looking. A wrong one can send you to the wrong place, quickly.</p>
<h2>What the checker compares</h2><p><code>drift.py</code> reads Backstage <code>Component</code> entities and checks five things against GitHub, all read-only:</p>
<table>
<thead>
<tr>
<th>Check</th>
<th>Drift (GitHub contradicts it)</th>
<th>Warning (look at it)</th>
<th>Cannot tell from outside</th>
</tr>
</thead>
<tbody><tr>
<td>owner</td>
<td>no <code>spec.owner</code>; with org access, a team that does not exist or has no members</td>
<td>a team GitHub cannot resolve in the CODEOWNERS files that name it; a <code>user:</code> with no GitHub account of that name</td>
<td>a team no CODEOWNERS file names; a user who is not a public member; a group in another directory</td>
</tr>
<tr>
<td>repo</td>
<td>the slug redirects (renamed or transferred); a <code>github.com/project-slug</code> that is not <code>owner/repo</code>; archived while the catalog does not say <code>deprecated</code></td>
<td>no human commit on the default branch in a year; a root catalog file that describes a different repository</td>
<td>a repository that is not visible (deleted or private)</td>
</tr>
<tr>
<td>codeowners</td>
<td>the CODEOWNERS rule for the catalog file names owners GitHub's validation cannot resolve</td>
<td>CODEOWNERS and the catalog name different owners</td>
<td>GitHub's validation not readable</td>
</tr>
<tr>
<td>links</td>
<td>404 or 410; a GitHub file missing at a ref that exists</td>
<td></td>
<td>401, 403, 429, 5xx, timeouts, internal hostnames, pages below a GitHub repository</td>
</tr>
<tr>
<td>lifecycle</td>
<td></td>
<td><code>production</code> or <code>experimental</code> with no human commit in a year; no lifecycle</td>
<td></td>
</tr>
</tbody></table>
<ol>
<li><strong>catalog-info.yaml</strong> Component entities</li>
<li><strong>Owner</strong> teams API or CODEOWNERS errors</li>
<li><strong>Repository</strong> redirects, archive, commits</li>
<li><strong>Links</strong> public addresses only</li>
<li><strong>Report</strong> reasons and exit code</li>
</ol>
<p>A drift on the CODEOWNERS check is about the file that routes code reviews, not about the catalog entry itself. We count both because both are ownership records people rely on, but we report them separately below. A warning is a fact worth a look: a library that works may not need a commit for a year. "Cannot tell" is its own status and never counted as either.</p>
<h3>Checking an owner from outside the org</h3><p>Inside your org, checking an owner is a lookup: does the team exist, does it have members, is this user a member. From outside, GitHub hides teams. But GitHub validates CODEOWNERS files, and it publishes the result. For a public repository, anyone can call <a href="https://docs.github.com/en/rest/repos/repos#list-codeowners-errors-for-a-repository" rel="noopener noreferrer"><code>GET /repos/{owner}/{repo}/codeowners/errors</code></a>, and it lists every owner GitHub cannot resolve, with messages such as:</p>
<pre><code class="hljs language-text" tabindex="0">Unknown owner on line 3: make sure the team @org/team exists, is publicly visible, and has write access to the repository
</code></pre><p>So if any CODEOWNERS file in the org names the catalog's owner team, GitHub can tell us whether that team resolves. The checker looks in the entry's own repository first, then runs one code search for the team in the org's other CODEOWNERS files. A team that resolves exists. A team that does not resolve might be deleted, renamed, secret, or only lack write access there, so on the owner check it is a warning, not drift. A team that no CODEOWNERS file names stays "cannot tell". All of this assumes Backstage group names are GitHub team slugs, which is what Backstage's GitHub org provider produces; groups from other namespaces are not mapped.</p>
<h3>Links come from strangers</h3><p>The links in a catalog file are written by whoever wrote the file, so link checks are opt-in (<code>--check-links</code>) and careful. The token only ever goes to <code>api.github.com</code>, and github.com links are checked through the API with every path segment validated and quoted. Everything else gets a <code>HEAD</code>, then a <code>GET</code>, with no credentials and no cookies, at most one request per second per host. The host name is resolved once, every address must be public, and the request goes to that pinned address, so a second DNS answer cannot point it at an internal service. Every redirect hop is checked the same way.</p>
<h2>The sample</h2><p>We wanted catalogs that real organizations maintain, not tutorials. The selection, all recorded in the repository:</p>
<ol>
<li>GitHub code search for <code>filename:catalog-info.yaml path:/</code> (files up to 20,000 bytes) on 2026-10-22, between 07:08 and 07:23 UTC. Code search returns at most 1,000 results per query, so <code>sample.py</code> splits the query by file size into 16 ranges. GitHub reported 8,864 matches; we fetched 9,761 unique repositories (code search counts are estimates).</li>
<li>3,579 of those repositories belong to personal accounts and were dropped. The other 6,182 belong to 984 organizations. Code search returned no forks.</li>
<li>53 organizations had a root <code>catalog-info.yaml</code> in 20 or more repositories. Many turned out to be test, demo or QA organizations for developer portals, so we kept the 30 that have a website on their GitHub profile and at least one repository with 10 or more stars. That rule keeps out obvious test orgs; it does not prove that every sampled repository is a real service, and a few are demos or samples. The excluded orgs are stored as hashes, because many are named after a person.</li>
<li>From each of the 30, 8 repositories at random (seed 20261022): 240 files. One was a scaffolder template, one was not valid YAML, and 24 held no Component entity, only other kinds. That leaves <strong>217 Component entries from 214 repositories</strong>.</li>
</ol>
<p>The 30 organizations are a mix of companies, public bodies, research institutes and open-source projects: bcgov, broadinstitute, cds-snc, Clever, contentful, cultureamp, DEFRA, dell, DiamondLightSource, dwp, eduNEXT, elastic, equinor, grafana, hmcts, Invoca, Kajabi, kartverket, open-dxp, openedx, pantheon-systems, prime-radiant-inc, radiorabe, reearth, sendbird, snyk, sourcefuse, spotify, Unity-Technologies and vtex-apps.</p>
<p>Every organization contributed 8 repositories, but a repository can hold several entries or none, so organizations contribute between 1 and 10 entries. The shares in this post are pooled over entries. Averaged per organization instead, the drift share is 10.7% rather than 9.7%.</p>
<p>So the population this describes is narrow: root-level <code>catalog-info.yaml</code> files in public repositories of organizations that use them widely. Catalog files elsewhere in a repository, catalogs kept in a separate repository, and private catalogs are not in it.</p>
<h2>The run</h2><p>One pass makes about 1,700 GitHub API requests, cached on disk per token for 24 hours. <code>--redact</code> keeps people out of the output: user logins become <code>user:&lt;redacted&gt;</code>, and owner and team names become a hash with a key that is random per run, which is why you see <code>group:&lt;56b023&gt;</code> below and different hashes in the committed results. The results name organizations, repositories and services, never a person.</p>
<p><strong>drift.py on the public sample</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ python3 drift.py --sample sample/sample-2026-10-22.json --check-links --redact --quiet &gt; report.txt 2&gt;/dev/null
$ <span class="hljs-built_in">echo</span> $?
1
$ <span class="hljs-built_in">tail</span> -7 report.txt
217 entries from 238 files (1 templates, 1 unparsed, 0 unreadable skipped)
  drift 21 (9.7%), warn 62, ok 134
  owner      drift 1, ok 53, unverifiable 159, warn 4
  repo       drift 6, ok 168, warn 43
  codeowners drift 14, n/a 144, ok 37, warn 22
  links      drift 2, n/a 147, ok 63, unverifiable 5
  lifecycle  n/a 38, ok 148, warn 31
$ awk <span class="hljs-string">'/^(DRIFT|WARN) /{p=($2=="com.unity.services.samples.relationships"||$2=="changelog-github"||$2=="kaizen-design-system"||$2=="loopback4-microservice-catalog")} p'</span> report.txt
DRIFT  com.unity.services.samples.relationships  (Unity-Technologies/com.unity.services.samples.friends:catalog-info.yaml)  owner group:&lt;56b023&gt;
         repo       drift        Unity-Technologies/com.unity.services.samples.relationships moved to Unity-Technologies/com.unity.services.samples.friends
         codeowners drift        CODEOWNERS line 2 names owners GitHub cannot resolve: @Unity-Technologies/&lt;072a85&gt;
DRIFT  changelog-github  (cultureamp/changelog-github:catalog-info.yaml)  owner group:&lt;ffb463&gt;
         repo       drift        github.com/project-slug is not a valid owner/repo
DRIFT  kaizen-design-system  (cultureamp/kaizen-design-system:catalog-info.yaml)  owner group:&lt;159434&gt;
         codeowners drift        CODEOWNERS line 4 names owners GitHub cannot resolve: @cultureamp/&lt;15544f&gt;
         links      drift        1 of 5 links dead: other https://cultureamp.design/&lt;path 642625a6c6&gt; (HTTP 404)
DRIFT  loopback4-microservice-catalog  (sourcefuse/loopback4-microservice-catalog:catalog-info.yaml)  owner group:&lt;046fce&gt;
         owner      warn         GitHub cannot resolve @sourcefuse/&lt;046fce&gt; <span class="hljs-keyword">in</span> the CODEOWNERS files that name it (deleted, renamed, secret, or without write access)
         codeowners drift        .github/CODEOWNERS line 5 names owners GitHub cannot resolve: @sourcefuse/&lt;046fce&gt;
</code></pre><p>The exit code is 1 because drift was found, so the same command can fail a CI job. The four entries above show the main kinds. The Unity sample was renamed from <code>relationships</code> to <code>friends</code>, its catalog entry still uses the old name, and line 2 of its CODEOWNERS names a team GitHub cannot resolve. cultureamp's changelog tool sets <code>github.com/project-slug</code> to the repository name without its owner, which Backstage's GitHub integrations cannot use. In cultureamp's design system, a CODEOWNERS team does not resolve and the Storybook link answers 404. In sourcefuse's microservice catalog, the catalog's own owner team is the one GitHub cannot resolve in the same repository's CODEOWNERS: drift for the CODEOWNERS file, a warning for the owner.</p>
<h2>What it found</h2><p><strong>What 217 public catalog entries got wrong</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>CODEOWNERS names an owner GitHub cannot resolve (14)</td>
<td>6.5%</td>
<td>Drift</td>
</tr>
<tr>
<td>Repository renamed, catalog has the old name (4)</td>
<td>1.8%</td>
<td>Drift</td>
</tr>
<tr>
<td>project-slug is not owner/repo (2)</td>
<td>0.9%</td>
<td>Drift</td>
</tr>
<tr>
<td>Dead link (2)</td>
<td>0.9%</td>
<td>Drift</td>
</tr>
<tr>
<td>No owner at all (1)</td>
<td>0.5%</td>
<td>Drift</td>
</tr>
<tr>
<td>No human commit in a year (39)</td>
<td>18%</td>
<td>Warning</td>
</tr>
<tr>
<td>Says production or experimental, idle a year (28)</td>
<td>12.9%</td>
<td>Warning</td>
</tr>
<tr>
<td>Catalog and CODEOWNERS name different owners (22)</td>
<td>10.1%</td>
<td>Warning</td>
</tr>
<tr>
<td>Owner team does not resolve where it is named (3)</td>
<td>1.4%</td>
<td>Warning</td>
</tr>
</tbody></table>
<p><em>Share of all 217 entries with each finding; an entry can have more than one. 30 GitHub organizations, 8 random repositories each, checked 2026-10-22. Source: results/summary-20261022T091627Z.json in the demo repository.</em></p>
<h3>Ownership is where the records disagree</h3><p>Most findings are about who owns what. The clearest view is the 73 entries whose catalog file is also covered by a CODEOWNERS rule, so there are two records of the owner to compare:</p>
<p><strong>summarize.py</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ python3 summarize.py results/run-20261022T091627Z.json | sed -n <span class="hljs-string">'/^catalog vs CODEOWNERS/,/Fisher/p'</span>
catalog vs CODEOWNERS: 73 entries with both; agree and resolve 37 (50.7%); unknown owners 14 (teams 8, <span class="hljs-built_in">users</span> 6)
  disagree:   4  catalog names a team, CODEOWNERS people
  disagree:  11  different team names
  disagree:   7  one team name contains the other
drift by file age: under 1 year {<span class="hljs-string">'entries'</span>: 112, <span class="hljs-string">'drift'</span>: 9, <span class="hljs-string">'pct'</span>: 8.0}, 1 year or more {<span class="hljs-string">'entries'</span>: 105, <span class="hljs-string">'drift'</span>: 12, <span class="hljs-string">'pct'</span>: 11.4}, Fisher p = 0.493
</code></pre><p><strong>73 entries with both a catalog owner and a CODEOWNERS rule</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Agree, every owner resolves</td>
<td>37</td>
<td>Agree</td>
</tr>
<tr>
<td>One team name contains the other</td>
<td>7</td>
<td>Disagree</td>
</tr>
<tr>
<td>Different team names</td>
<td>11</td>
<td>Disagree</td>
</tr>
<tr>
<td>Catalog names a team, CODEOWNERS people</td>
<td>4</td>
<td>Disagree</td>
</tr>
<tr>
<td>CODEOWNERS owner GitHub cannot resolve</td>
<td>14</td>
<td>Cannot resolve</td>
</tr>
</tbody></table>
<p><em>"One name contains the other" is, for example, a short catalog name inside a longer GitHub team slug; it may be the same team, but the data cannot show it. Unresolvable owners are what GitHub's CODEOWNERS validation reports for the rule that covers the catalog file.</em></p>
<p>37 agree. The other 36, almost half, either name different owners or name owners GitHub cannot resolve. In 7 of them one team name contains the other: in four snyk repositories, for example, the catalog uses a short team name that is part of the longer GitHub team slug in CODEOWNERS. Those may well be the same team. Leave them out and 29 of 73 (40%) still disagree or cannot be resolved. Of the 14 with unresolvable owners, 8 involve a team and 6 involve individual users.</p>
<p>Two caveats keep this from being a verdict on the catalog. CODEOWNERS says who reviews code, which is not always who owns the service, so a disagreement means one record is stale or the two mean different things. And GitHub reports a team as unknown when it was deleted or renamed, is secret, or has no write access to the repository; any of those means the team does not review that code, but only the first two mean it is gone.</p>
<p>Unresolvable owners repeat. In vtex-apps, the same unknown team sits on line 3 of the CODEOWNERS in three of the eight sampled repositories, which looks like a file copied from a template and never updated. Four organizations hold 14 of the 21 drifted entries, and 20 of the 30 had none.</p>
<p>The owner field itself could be checked for only 58 entries (27%): 37 resolved in the entry's own CODEOWNERS, 15 in another repository's, 3 named a team GitHub could not resolve in any CODEOWNERS file that names it, 2 were checked through GitHub's user API, and 1 had no owner at all. Backstage <a href="https://backstage.io/docs/features/software-catalog/descriptor-format" rel="noopener noreferrer">requires</a> <code>spec.owner</code> on a Component, so that entry probably never loaded into anyone's catalog. The 159 we could not check are 142 groups and 17 users: hidden teams, groups that do not look like GitHub teams, users who are not public members of the org, and user IDs that are not GitHub logins.</p>
<h3>Repositories that moved</h3><p>4 entries (1.8%) point at a repository by a name it no longer has. GitHub redirects the old name, so links and API calls keep working. But GitHub's documentation warns that if anyone creates a new repository with the old name, "redirects to the renamed repository will no longer work", and the catalog entry then points at someone else's code. 2 more entries set <code>github.com/project-slug</code> to a bare repository name without the owner, and 2 live in one repository but describe another, which is what a catalog file copied from a sibling repository and never edited looks like.</p>
<p>We found no archived repositories. That is a gap in the sample, not good news: none of the 211 repositories the checker read was archived (each entry records the flag), a code search we tried with <code>archived:true</code> returned nothing (so archived repositories are probably missing from the index), and Backstage's <a href="https://backstage.io/docs/integrations/github/discovery" rel="noopener noreferrer">GitHub discovery</a> skips them by default (<code>allowArchived</code> defaults to <code>false</code>).</p>
<h3>Idle, but still "production"</h3><p>39 entries (18.0%) live in repositories with no human commit on the default branch for over a year, and 2 more where the last 30 commits are all from bots. We do not count commits authored by Dependabot, Renovate or other bots as activity. 28 of the 39 still say <code>production</code> (19) or <code>experimental</code> (9). A stable library can go a year without a change, so this is a warning, not drift. An <code>experimental</code> service with no human commit for a year is a decision nobody made.</p>
<h3>Links</h3><p>Only 70 of 217 entries (32%) had any link at all. Of 152 links, 132 answered, 2 were dead, and 18 could not be checked: 12 pointed at hostnames that do not resolve in public DNS (internal tools, most likely), 5 answered 403, and 1 was a page below a GitHub repository that we do not check. The two dead ones were a Storybook page and a vulnerability report page on a vendor's site.</p>
<p>None of the 152 links was titled as a runbook, an on-call schedule or a dashboard. Public catalogs may simply not publish them, or may title them differently; either way we could not test those links, and they are the ones you would want to work during an incident. If you run this on your own catalog, the link check is where we would expect your numbers to differ most from ours.</p>
<h3>Does a catalog decay with age?</h3><p><strong>Drift by when the catalog file last changed</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Under 6 months (81)</td>
<td>9.9%</td>
</tr>
<tr>
<td>6 to 12 months (31)</td>
<td>3.2%</td>
</tr>
<tr>
<td>1 to 2 years (48)</td>
<td>12.5%</td>
</tr>
<tr>
<td>Over 2 years (57)</td>
<td>10.5%</td>
</tr>
</tbody></table>
<p><em>Share of entries with at least one drift, by the date of the last commit that touched catalog-info.yaml. Under one year: 9 of 112. One year or more: 12 of 105. Fisher's exact test p = 0.49, which also treats entries as independent although entries from one organization share habits.</em></p>
<p>We expected a clear slope and did not find one: 8.0% for files changed in the last year against 11.4% for older files. Fisher's exact test gives p = 0.49, and it assumes the entries are independent, which they are not quite: entries from one organization share templates and habits. So this sample cannot tell the difference from chance. Drift and warnings together are far more common in older files (24.7% and 19.4% under a year, 54.2% and 54.4% over), but mostly by construction: a repository with no commits for a year also has a catalog file nobody touched for a year. The median catalog file in the sample was last changed 352 days before we checked it.</p>
<p>A single snapshot also cannot see the worst decay. Entries that rotted badly enough to be noticed may have been deleted, and deleted entries are not in a sample of files that still exist.</p>
<h2>The bugs two reviews found in our checker</h2><p>The first version of the checker and of this post had problems, found by our own runs and by a second review of the code and the draft. Several changed numbers you just read, and some of the corrections made the result look worse, others better.</p>
<ol>
<li><strong>An unresolvable team was called a missing owner.</strong> The first version reported a catalog owner as drift when GitHub could not resolve the team in a CODEOWNERS file, including one in another repository. But a team can exist and simply lack write access there. That is now a warning on the owner, and only the broken CODEOWNERS file counts as drift. Three entries moved from owner drift to a warning; two of them had no other drift.</li>
<li><strong>An invalid repository reference was quietly replaced.</strong> Two entries set <code>github.com/project-slug</code> to a repository name without its owner. The first version fell back to the repository the file lives in and reported nothing. It now reports the annotation as drift, which added two drifted entries just as the first fix removed two. The headline stayed at 21 by coincidence.</li>
<li><strong>A user ID that looks like a GitHub login.</strong> One organization names users by an ID from its own directory, three letters and five digits, which is also a valid GitHub login. GitHub had no account by that name, and the first version reported a departed owner. A missing account is now a warning, unless you pass <code>--users-are-github-logins</code> for a catalog where you know users are GitHub logins.</li>
<li><strong>GitHub's two error formats.</strong> For a user, the CODEOWNERS error says <code>make sure @user exists</code>; for a team, <code>make sure the team @org/team exists, is publicly visible, ...</code>. The first version only parsed the user form, and for teams it fell back to marking every owner on the line as unknown. It changed no result in this sample, but on another catalog it would have.</li>
<li><strong>A YAML reader that was too forgiving.</strong> The checker has its own small YAML reader to stay dependency-free. One sampled file had <code>description:</code> at the wrong indentation, which swallowed the <code>annotations:</code> block below it. PyYAML rejects that file, and our reader accepted it. It now raises the same error, and a cross-check against PyYAML on all 240 files agrees on every field the checker reads, with the one invalid file rejected by both. The review also found that it decoded some escapes in quoted strings wrongly; it now handles them or refuses the file.</li>
<li><strong>Too generous with links and CODEOWNERS.</strong> Any page below an existing GitHub repository counted as a working link, a redirect loop counted as an answer, <code>docs/*</code> in CODEOWNERS also matched files in subdirectories (GitHub's own docs say it does not), and an unreadable CODEOWNERS validation could still report agreement. Each now reports "cannot tell" or matches GitHub's rules.</li>
<li><strong>People in the output.</strong> A bare owner such as <code>owner: someone</code> is treated as a team by Backstage, but it is sometimes a person's login, and one owner was an email address. An early run printed them. That file was deleted, and owner and team names are now hashed with a key that is never written down.</li>
</ol>
<p>Smaller fixes: a link with a space in it stopped an early run; cached answers now expire after a day and are kept per token; CODEOWNERS lines with invalid syntax are skipped, as GitHub skips them; a file link is not called dead when its ref might contain a slash; and an error in one entry, or a 401 or 5xx from GitHub, makes the run exit with 2 instead of stopping or passing. None of these changed a count in this sample. Every superseded run that left a file is kept in <code>results/</code> with a note on what changed.</p>
<h2>False positives, and what we could not check</h2><ul>
<li><strong>A team GitHub cannot resolve is not always deleted.</strong> GitHub's message covers deleted, renamed, secret, and teams without write access to that repository.</li>
<li><strong>Different names can be the same people.</strong> A parent team and a child team, or a Backstage group name that differs from the GitHub team slug, show up as a disagreement. That is why disagreement is a warning.</li>
<li><strong>A link that answers is not always alive.</strong> A 200 can be a login page, or a "page not found" that answers 200. We count both as working. A 403 from a site that blocks scripts is "cannot tell".</li>
<li><strong>Bot detection is by commit author.</strong> A merge commit by a person counts as human activity even when it merges a bot's change, so "no human commit in a year" can undercount idle repositories.</li>
<li><strong>Owner coverage is uneven.</strong> We could check 27% of owners, mostly those named in some CODEOWNERS file. Those may be better or worse kept than the rest; we cannot tell which.</li>
<li><strong>Private catalogs are not in the sample.</strong> Public repositories may be kept better or worse than internal ones, and internal catalogs carry the runbook, on-call and dashboard links we could not test.</li>
</ul>
<h2>How to keep a catalog honest</h2><h3>Derive owners, do not type them</h3><p>Many of the disagreements above look like two hand-typed names for the same thing. Load groups from the source of truth instead: Backstage's <a href="https://backstage.io/docs/integrations/github/org" rel="noopener noreferrer">GitHub org provider</a> mirrors GitHub teams as Groups on a schedule, and the catalog products sync from identity providers. Then the owner in <code>catalog-info.yaml</code> and the handle in CODEOWNERS are the same string, and a machine can compare them:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-comment"># catalog-info.yaml</span>
<span class="hljs-attr">apiVersion:</span> <span class="hljs-string">backstage.io/v1alpha1</span>
<span class="hljs-attr">kind:</span> <span class="hljs-string">Component</span>
<span class="hljs-attr">metadata:</span>
  <span class="hljs-attr">name:</span> <span class="hljs-string">payments-api</span>
  <span class="hljs-attr">annotations:</span>
    <span class="hljs-attr">github.com/project-slug:</span> <span class="hljs-string">acme/payments-api</span> <span class="hljs-comment"># owner/repo, not just the repository name</span>
<span class="hljs-attr">spec:</span>
  <span class="hljs-attr">type:</span> <span class="hljs-string">service</span>
  <span class="hljs-attr">lifecycle:</span> <span class="hljs-string">production</span>
  <span class="hljs-attr">owner:</span> <span class="hljs-string">group:default/payments</span> <span class="hljs-comment"># the GitHub team slug, not a display name</span>
</code></pre><pre><code class="hljs language-text" tabindex="0"># .github/CODEOWNERS: the same team, so the two can be checked against each other
*    @acme/payments
</code></pre><h3>Check the catalog file in CI</h3><p>Drift arrives two ways: someone edits the file badly, or the world changes while nobody touches it. A pull request check catches the first, a weekly scheduled run catches the second. The repository ships this workflow in <code>examples/</code>:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">name:</span> <span class="hljs-string">catalog</span> <span class="hljs-string">drift</span>

<span class="hljs-attr">on:</span>
  <span class="hljs-attr">pull_request:</span>
    <span class="hljs-attr">paths:</span> [<span class="hljs-string">'catalog-info.yaml'</span>, <span class="hljs-string">'.github/CODEOWNERS'</span>, <span class="hljs-string">'CODEOWNERS'</span>, <span class="hljs-string">'docs/CODEOWNERS'</span>]
  <span class="hljs-attr">schedule:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">cron:</span> <span class="hljs-string">'17 6 * * 1'</span>
  <span class="hljs-attr">workflow_dispatch:</span>

<span class="hljs-attr">permissions:</span>
  <span class="hljs-attr">contents:</span> <span class="hljs-string">read</span>

<span class="hljs-attr">jobs:</span>
  <span class="hljs-attr">drift:</span>
    <span class="hljs-attr">runs-on:</span> <span class="hljs-string">ubuntu-latest</span>
    <span class="hljs-attr">steps:</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v5</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v5</span>
        <span class="hljs-attr">with:</span>
          <span class="hljs-attr">repository:</span> <span class="hljs-string">The-DevOps-Daily/service-catalog-drift</span>
          <span class="hljs-attr">ref:</span> <span class="hljs-string">main</span> <span class="hljs-comment"># pin this to a commit SHA you have read</span>
          <span class="hljs-attr">path:</span> <span class="hljs-string">.catalog-drift</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/setup-python@v6</span>
        <span class="hljs-attr">with:</span>
          <span class="hljs-attr">python-version:</span> <span class="hljs-string">'3.13'</span>
      <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">check</span> <span class="hljs-string">catalog-info.yaml</span>
        <span class="hljs-attr">env:</span>
          <span class="hljs-comment"># The default token cannot read org teams; use a token with Members: read for full owner checks</span>
          <span class="hljs-attr">GITHUB_TOKEN:</span> <span class="hljs-string">${{</span> <span class="hljs-string">secrets.CATALOG_DRIFT_TOKEN</span> <span class="hljs-string">||</span> <span class="hljs-string">github.token</span> <span class="hljs-string">}}</span>
          <span class="hljs-attr">REPO:</span> <span class="hljs-string">${{</span> <span class="hljs-string">github.repository</span> <span class="hljs-string">}}</span>
          <span class="hljs-comment"># on a pull request, read CODEOWNERS as the pull request changes it</span>
          <span class="hljs-attr">REF:</span> <span class="hljs-string">${{</span> <span class="hljs-string">github.event.pull_request.head.sha</span> <span class="hljs-string">||</span> <span class="hljs-string">github.sha</span> <span class="hljs-string">}}</span>
        <span class="hljs-attr">run:</span> <span class="hljs-string">python</span> <span class="hljs-string">.catalog-drift/drift.py</span> <span class="hljs-string">catalog-info.yaml</span> <span class="hljs-string">--repo</span> <span class="hljs-string">"$REPO"</span> <span class="hljs-string">--ref</span> <span class="hljs-string">"$REF"</span> <span class="hljs-string">--check-links</span> <span class="hljs-string">--no-org-search</span> <span class="hljs-string">--cache-hours</span> <span class="hljs-number">0</span>
</code></pre><p>It exits with 1 on drift or on a catalog file that does not parse, and with 2 when it could not check what it was given, so a broken run never passes quietly.</p>
<h3>Use the check GitHub already runs</h3><p>You do not need our script to find the worst cases. GitHub already validates every CODEOWNERS file, and one call per repository lists what it cannot resolve:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Owners GitHub cannot resolve in this repository's CODEOWNERS</span>
gh api repos/OWNER/REPO/codeowners/errors \
  --jq <span class="hljs-string">'.errors[] | "line \(.line): \(.message | split("\n")[0])"'</span>
</code></pre><p>Loop it over every repository in your org once a month. Each line it prints is an owner or a rule GitHub cannot use, which can leave reviews without the reviewer you meant, and is often a sign that the catalog owner is stale too.</p>
<h3>Give the scorecard consequences</h3><p>Every catalog product has scorecards, and so does Backstage through the Tech Insights plugin. The useful checks are the ones in this post: the owner resolves, the runbook link answers, the lifecycle matches the activity. Put the result where people already work, such as a comment on the pull request or a weekly list sent to each owning team, and not only on a dashboard.</p>
<h3>Retire what nobody claims</h3><p>An entry whose owner does not resolve and whose repository has been idle for a year may have nobody to fix it. One workflow we would suggest: open an issue for the last known owner, and if nobody confirms ownership within 30 days, mark the entry <code>deprecated</code>; remove it 30 days later unless someone claims it. A missing entry sends people asking. A wrong one can send them to the wrong place.</p>
<h2>Build or buy</h2><p>Every catalog product handles freshness with some mix of the same three tools: sync from a source of truth, automatic discovery, and scorecards. Here is what each one's public documentation says, in alphabetical order, with one documented limit each where we found one:</p>
<ul>
<li><strong>Backstage</strong> (open source) mirrors GitHub teams and users with its <a href="https://backstage.io/docs/integrations/github/org" rel="noopener noreferrer">org provider</a>, finds <code>catalog-info.yaml</code> files with <a href="https://backstage.io/docs/integrations/github/discovery" rel="noopener noreferrer">GitHub discovery</a>, and removes <a href="https://backstage.io/docs/features/software-catalog/life-of-an-entity" rel="noopener noreferrer">orphaned entities</a>. Checks on catalog data, including the owner field, come from the <a href="https://github.com/backstage/community-plugins/tree/main/workspaces/tech-insights/plugins/tech-insights-backend" rel="noopener noreferrer">Tech Insights</a> community plugin. Limit: as quoted above, relations that point at nothing are accepted.</li>
<li><strong>Cortex</strong> has <a href="https://docs.cortex.io/scorecards/scorecard-examples" rel="noopener noreferrer">scorecard rules</a> such as "has an owner" and "has a runbook", syncs <a href="https://docs.cortex.io/ingesting-data-into-cortex/entities/ownership" rel="noopener noreferrer">ownership</a> daily from identity providers, Git hosts and HR systems, and <a href="https://docs.cortex.io/ingesting-data-into-cortex/entities-overview/entities/archiving-entities/auto-archive" rel="noopener noreferrer">auto-archives</a> entities whose repository was deleted or archived. Limit: in GitOps mode, a YAML file that is moved rather than deleted is not archived.</li>
<li><strong>Humanitec</strong> says in its <a href="https://developer.humanitec.com/platform-orchestrator/docs/humanitec-vs-others/overview/" rel="noopener noreferrer">own docs</a> that it is not an internal developer portal: the Platform Orchestrator is the backend of a platform and <a href="https://developer.humanitec.com/platform-orchestrator/docs/humanitec-vs-others/backstage-port-cortex-etc./" rel="noopener noreferrer">complements</a> portals such as Backstage, Port and Cortex.</li>
<li><strong>OpsLevel</strong> has a <a href="https://docs.opslevel.com/docs/getting-started-with-rubrics" rel="noopener noreferrer">component ownership check</a> in its rubric, <a href="https://docs.opslevel.com/docs/github-teams-sync" rel="noopener noreferrer">syncs GitHub teams</a> every 12 hours, <a href="https://docs.opslevel.com/docs/service-detection" rel="noopener noreferrer">detects services</a> from repositories and suggests owners from commit authors, and <a href="https://docs.opslevel.com/docs/repositories" rel="noopener noreferrer">removes repositories</a> deleted at the Git provider. Limit: the team sync page says deleted GitHub teams and users are not synced.</li>
<li><strong>Port</strong> has <a href="https://docs.port.io/governance/standards-and-compliance/examples/scorecard-use-cases/" rel="noopener noreferrer">scorecards</a> with a "has a team" rule, <a href="https://docs.port.io/context-lake/business-context/ownership/" rel="noopener noreferrer">syncs users and teams</a> from SSO and Git providers, deletes entities that are gone from the source on a <a href="https://docs.port.io/context-lake/ingestion/ingest-data-into-port/native-integrations/sync-mechanisms/full-resync/" rel="noopener noreferrer">full resync</a>, and has a <a href="https://www.docs.port.io/guides/all/connect-github-codeowners-with-service-team-and-user" rel="noopener noreferrer">guide for mapping CODEOWNERS</a> to teams. Limit: entities of a resource type removed from the mapping are <a href="https://docs.port.io/context-lake/ingestion/configure-mapping/entity-cleanup/" rel="noopener noreferrer">not deleted automatically</a>.</li>
<li><strong>Roadie</strong> is managed Backstage: <code>catalog-info.yaml</code> is its <a href="https://roadie.io/docs/catalog/overview" rel="noopener noreferrer">native format</a>, it discovers files from Git providers, runs <a href="https://roadie.io/docs/tech-insights/introduction/" rel="noopener noreferrer">Tech Insights</a> scorecards, and its Catalog Completeness beta <a href="https://roadie.io/docs/catalog/repositories/" rel="noopener noreferrer">suggests owners</a> for repositories. Limit: deleting a location <a href="https://roadie.io/docs/catalog/location-management/" rel="noopener noreferrer">does not remove</a> the entities found through it.</li>
</ul>
<p>If you already run Backstage, the checks in this post fit in a standard-library Python script that runs in CI. What the products add is the plumbing around them: syncing from HR and identity systems, a place where scores are shown, and someone else keeping it running. Whichever way you go, test it the way we tested ours. The repository has a fixture catalog with one of each kind of drift; point your tool at something like it and confirm it fails on each one. A tool you have only seen report "all good" has not shown you that it can find anything.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>Whether catalogs decay with age.</strong> The difference we saw (8.0% against 11.4%) is within chance for this sample, and one snapshot cannot measure a rate.</li>
<li><strong>Whether three owners in four are right.</strong> 159 owners could not be checked from outside. Among the 58 we could check, 1 entry had no owner and 4 drew a warning, and that rate does not carry over to the rest.</li>
<li><strong>Whether a disagreement means a wrong owner.</strong> CODEOWNERS and the catalog can mean different things by "owner".</li>
<li><strong>Whether runbooks and on-call links rot.</strong> None were identified in public catalogs.</li>
<li><strong>How private catalogs compare.</strong> Every entry here is in a public repository.</li>
<li><strong>Whether a wrong entry ever caused a bad page.</strong> "Can be worse than none" is an argument about how people use catalogs during incidents, not something this data measures.</li>
</ul>
<h2>Summary</h2><ul>
<li>Check a catalog against the systems it describes, not against itself. Ours found 21 entries out of 217 where GitHub contradicts the entry or its CODEOWNERS, all of which looked fine in their own files.</li>
<li>Ownership is where the records disagree. Where two records of the owner existed, half disagreed or named an owner GitHub could not resolve.</li>
<li>From outside an org, most owners cannot be checked at all. Run the check where the token can see your teams, and map directory groups to GitHub teams explicitly.</li>
<li>Make owners the same string everywhere by deriving them from GitHub teams or your identity provider, then compare catalog and CODEOWNERS in CI.</li>
<li>Run the check on a schedule too: a team deleted in a reorg changes no file.</li>
<li>Retire entries nobody claims. A gap sends people asking; a wrong answer can send them to the wrong place.</li>
</ul>
<p>The checker, the sample, every result and the tests are in <a href="https://github.com/The-DevOps-Daily/service-catalog-drift" rel="noopener noreferrer">The-DevOps-Daily/service-catalog-drift</a>. Run it against your own catalog, with a token that can see your teams, and see how your numbers compare.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[What Keeps a Neon Compute Awake, and What It Costs]]></title>
      <link>https://devops-daily.com/posts/what-keeps-a-neon-compute-awake</link>
      <description><![CDATA[We watched Neon computes through the API to see which clients stop scale to zero. Idle connections do not. An open transaction, a monitoring agent and a health check every 5 minutes do, and that health check keeps the smallest compute running 180 CU-hours a month.]]></description>
      <pubDate>Wed, 21 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/what-keeps-a-neon-compute-awake</guid>
      <category><![CDATA[FinOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Neon]]></category><category><![CDATA[Postgres]]></category><category><![CDATA[Scale to Zero]]></category><category><![CDATA[Serverless]]></category><category><![CDATA[FinOps]]></category><category><![CDATA[Connection Pooling]]></category><category><![CDATA[Monitoring]]></category>
      <content:encoded><![CDATA[<p>We pointed a health check at a Neon compute every 5 minutes, the same as Neon's default scale-to-zero timeout. The compute never suspended. In 40 minutes it took 8 checks, woke once for the first one, and stayed awake the whole time. Over 30 days that is 180 CU-hours at the smallest compute size: about $19 on the Launch plan, $40 on Scale, and on the Free plan it uses up the 100 CU-hour allowance by day 17.</p>
<p>The suspect most people reach for first, an idle connection left open by the app, does not keep a compute awake. We held idle connections, direct and pooled, and the compute suspended every time, a few seconds later than with nothing open. What the suspend did do was cut those connections, including the pooled ones.</p>
<p>This post measures which ordinary clients keep a Neon compute running and which do not, how long a compute takes to suspend, what your app sees when it does, and what each pattern costs per month. Everything runs against real computes in one Neon project, watched through the Neon API, and the scripts are public so you can run them on your own account.</p>
<h2>TL;DR</h2><ul>
<li><strong>Idle connections do not stop scale to zero.</strong> With a 60-second timeout the compute suspended 66 to 75 seconds after the last query with nothing open, and 81 to 83 seconds with an idle direct or pooled connection held. The suspend dropped every held connection, pooled ones included.</li>
<li><strong>An open transaction keeps it awake for as long as it stays open.</strong> A session left <code>idle in transaction</code>, direct or through the pooler, kept the compute up for the whole 6-minute window, 4 runs out of 4. Setting <code>idle_in_transaction_session_timeout = '30s'</code> let it suspend again.</li>
<li><strong>A monitoring agent that queries every 20 seconds keeps it awake</strong>, even when it only reads <code>pg_stat_database</code> on the <code>postgres</code> database.</li>
<li><strong>A <code>LISTEN</code> connection does not keep it awake.</strong> It is dropped on suspend, and a notification sent after the suspend never reached it (2 of 2 runs).</li>
<li><strong>Health checks are the expensive one.</strong> With the default 5-minute timeout, a check every 5 minutes kept the compute awake for the whole 40-minute run. In our runs the suspend came 312 to 327 seconds after a wake, so each check arrived before it. A check every 6 minutes still kept it awake 90.6% of the time, and every 10 minutes 52.1%.</li>
<li><strong>A wake adds roughly half a second to a second.</strong> End-to-end checks (connect, authenticate, query, close) that woke the compute had medians of 847 to 1,134 ms per run; checks on an awake compute took 308 to 339 ms, from a client 48 ms from the region.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Neon account and an API key (Account settings &gt; API keys). The 60-second timeout used in part of the test needs a plan that lets you change it (Scale); everything with the default 5-minute timeout works on any plan.</li>
<li>Node.js 22 or newer.</li>
<li>Basic Postgres knowledge: sessions, transactions, <code>pg_stat_activity</code>.</li>
</ul>
<h2>How Neon decides a compute is idle</h2><p>Neon's documentation says a compute scales to zero after 5 minutes of inactivity, that you can change the timeout on paid plans, and that a connected logical replication subscriber prevents the suspend. It does not say what counts as activity.</p>
<p>The answer is in the compute's own code. Every compute runs <code>compute_ctl</code>, and its <a href="https://github.com/neondatabase/neon/blob/f3a0e4f255/compute_tools/src/monitor.rs#L205-L350" rel="noopener noreferrer">activity monitor</a> checks Postgres every 500 ms. It moves the "last active" time forward when:</p>
<ol>
<li>The totals in <code>pg_stat_database</code> change: the summed <code>active_time</code> (time spent running queries) or the number of <code>sessions</code>, across all databases except <code>postgres</code> and the templates. This catches a query that started and finished between two checks, and every new connection. The code calls this path experimental but says it is on for everyone.</li>
<li>Any client backend is in a state other than <code>idle</code>. That includes <code>active</code> (a query is running) and <code>idle in transaction</code> (a transaction is open but doing nothing). Backends of Neon's own <code>cloud_admin</code> role are left out here.</li>
<li>An idle backend changed state recently: the latest <code>state_change</code> among idle backends counts as the last activity (<a href="https://github.com/neondatabase/neon/blob/f3a0e4f255/compute_tools/src/monitor.rs#L444-L497" rel="noopener noreferrer">the backend check</a>). A connection that ran a query 20 seconds ago and went idle moves the clock to then. Connecting moves it once too, but a connection that then just sits idle does not keep moving it.</li>
<li>A logical replication walsender exists, an active logical replication subscription exists, or an autovacuum worker is running.</li>
</ol>
<p>The control plane reads the last activity time and suspends the compute once the timeout has passed. This is the open-source code at commit <code>f3a0e4f255</code>; the version running in production may differ, which is one more reason to measure.</p>
<p>So the code predicts that an idle connection should not keep a compute awake, and an open transaction should. The rest of this post checks those predictions and puts numbers on them.</p>
<h2>How we measured</h2><p><a href="https://github.com/The-DevOps-Daily/neon-keep-awake" rel="noopener noreferrer">The-DevOps-Daily/neon-keep-awake on GitHub</a></p>
<p>One Neon project in <code>aws-eu-central-1</code> running Lakebase Postgres (PostgreSQL 17), every compute fixed at 0.25 CU so CU-hours are easy to read off. Each health-check configuration gets its own branch, and each branch has its own compute, so configurations running side by side do not affect each other. The idle-client scenarios ran one after another on a single branch.</p>
<p>The compute's state comes from <code>GET /projects/{id}/endpoints/{id}</code>, which returns <code>current_state</code> and <code>suspended_at</code>. That is a control-plane call: it never touches the compute, so watching cannot keep it awake. The scripts read it every 2 seconds for the idle clients, every 3 seconds for the health checks, and every 15 seconds for the long unattended watch.</p>
<p>Awake time is an estimate. An awake period starts when a check that found the compute suspended began connecting, and ends at the <code>suspended_at</code> that Neon reports, which is in whole seconds. We checked it against Neon's own record: the project's operations log lists every <code>start_compute</code> and <code>suspend_compute</code>. For the health-check runs, the awake time from the log and from our polling differed by at most half a percentage point. That is a cross-check of the timing, not of the bill.</p>
<p>That check caught one bug of ours. A poll taken a moment after a check woke the compute can still show the previous suspend, and our first summary ended the awake period there. The fix is to ignore a poll whose <code>suspended_at</code> is older than the wake, and even that went wrong at first: Neon's timestamps are whole seconds (<code>17:57:46Z</code>) and ours have milliseconds (<code>17:57:46.649Z</code>), and compared as strings <code>46Z</code> sorts after <code>46.649Z</code>. The bug cost two runs 3 to 4 seconds of awake time each, small but in the direction that makes the bill look smaller. The summary now compares timestamps as dates. A review of the draft found a second one: the runner could fire one extra check right at the end of a window. The summary now counts only checks scheduled inside the window, which took one check off two of the runs.</p>
<p>The scripts are split by stage, and each writes JSON lines stamped with a hash of the code that wrote it:</p>
<p><strong>neon-keep-awake</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ npm run setup
project odd-poetry-65586042, main branch br-icy-recipe-b91wrumq, compute ep-weathered-shadow-b91petoo
<span class="hljs-comment"># stage 1: idle clients, 60 s timeout</span>
$ npm run idle-clients
none <span class="hljs-comment">#1: suspended at 2026-10-18T17:52:15Z (wake 694 ms)</span>
none <span class="hljs-comment">#2: suspended at 2026-10-18T17:53:30Z (wake 947 ms)</span>
none <span class="hljs-comment">#3: suspended at 2026-10-18T17:54:46Z (wake 907 ms)</span>
idle-direct <span class="hljs-comment">#1: suspended at 2026-10-18T17:56:16Z (wake 815 ms)</span>
idle-direct <span class="hljs-comment">#2: suspended at 2026-10-18T17:57:46Z (wake 805 ms)</span>
idle-direct <span class="hljs-comment">#3: suspended at 2026-10-18T17:59:16Z (wake 793 ms)</span>
idle-pooled <span class="hljs-comment">#1: suspended at 2026-10-18T18:00:45Z (wake 774 ms)</span>
idle-pooled <span class="hljs-comment">#2: suspended at 2026-10-18T18:02:17Z (wake 787 ms)</span>
idle-pooled <span class="hljs-comment">#3: suspended at 2026-10-18T18:03:46Z (wake 804 ms)</span>
idle-in-transaction-direct <span class="hljs-comment">#1: still active after 360s (wake 815 ms)</span>
idle-in-transaction-direct <span class="hljs-comment">#2: still active after 360s (wake 1020 ms)</span>
idle-in-transaction-pooled <span class="hljs-comment">#1: still active after 360s (wake 940 ms)</span>
idle-in-transaction-pooled <span class="hljs-comment">#2: still active after 360s (wake 792 ms)</span>
idle-in-transaction-with-timeout <span class="hljs-comment">#1: suspended at 2026-10-18T18:34:31Z (wake 1080 ms)</span>
idle-in-transaction-with-timeout <span class="hljs-comment">#2: suspended at 2026-10-18T18:36:16Z (wake 895 ms)</span>
listen-direct <span class="hljs-comment">#1: suspended at 2026-10-18T18:37:46Z (wake 876 ms)</span>
listen-direct <span class="hljs-comment">#2: suspended at 2026-10-18T18:39:16Z (wake 296 ms)</span>
monitor-postgres-db-20s <span class="hljs-comment">#1: still active after 360s (wake 263 ms)</span>
</code></pre><p>The <code>wake</code> figure in that log is the time to connect and run <code>SELECT 1</code> at the start of each run. Most runs started right after a suspend, so it is usually a cold start. The last two started on a warm compute: the previous <code>LISTEN</code> run had just woken it to send its test notification, which is why they are near 300 ms.</p>
<h2>Idle connections do not keep it awake, and they do not survive the suspend</h2><p>Stage 1 sets the suspend timeout to 60 seconds, wakes the compute, leaves one kind of client open, and times the suspend from our last query.</p>
<table>
<thead>
<tr>
<th>What was left open</th>
<th>Runs</th>
<th>Suspended</th>
<th>Seconds from the last query to the suspend</th>
</tr>
</thead>
<tbody><tr>
<td>Nothing</td>
<td>3</td>
<td>3 of 3</td>
<td>75, 68, 66</td>
</tr>
<tr>
<td>One idle direct connection</td>
<td>3</td>
<td>3 of 3</td>
<td>82, 81, 83</td>
</tr>
<tr>
<td>One idle pooled connection (PgBouncer)</td>
<td>3</td>
<td>3 of 3</td>
<td>81, 83, 81</td>
</tr>
<tr>
<td>A <code>LISTEN</code> connection</td>
<td>2</td>
<td>2 of 2</td>
<td>83, 79</td>
</tr>
</tbody></table>
<p>An idle connection did not stop the suspend. The suspend did come later: the median was 82 seconds with an idle direct connection and 81 with a pooled one, against 68 with nothing open. With three runs per row, all on the same compute one after another, that is an observation we cannot explain, not an established effect. Either way, a compute with a 60-second timeout went to sleep in under a minute and a half with connections still open.</p>
<p>Look at what happened to those connections. Five seconds after each suspend, the script ran <code>SELECT 1</code> on the connection it had held:</p>
<pre><code class="hljs language-text" tabindex="0">idle direct:  'error' event: Connection terminated unexpectedly
              next query:    Client has encountered a connection error and is not queryable
idle pooled:  'error' event: Connection terminated unexpectedly
              next query:    Client has encountered a connection error and is not queryable
</code></pre><p>Every held connection was dead, 6 out of 6, pooled ones included, and the <code>LISTEN</code> connections in the last row died the same way. The pooler did not keep its client connections open across the suspend, so a pooled connection string does not protect your app from one.</p>
<p>For most apps the fix is to make sure a dropped connection is handled, not fatal. With <code>node-postgres</code>, use <code>Pool</code>, not a single long-lived <code>Client</code>, and handle the pool's <code>error</code> event. Without a listener, an idle client that loses its socket throws an unhandled error and can crash the process:</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-keyword">import</span> pg <span class="hljs-keyword">from</span> <span class="hljs-string">'pg'</span>;

<span class="hljs-keyword">const</span> pool = <span class="hljs-keyword">new</span> pg.<span class="hljs-title class_">Pool</span>({
  <span class="hljs-attr">connectionString</span>: process.<span class="hljs-property">env</span>.<span class="hljs-property">DATABASE_URL</span>,
  <span class="hljs-comment">// Close idle clients before the compute's suspend timeout closes them for you.</span>
  <span class="hljs-attr">idleTimeoutMillis</span>: <span class="hljs-number">30_000</span>,
});

<span class="hljs-comment">// An idle client whose socket closed (for example, the compute suspended).</span>
<span class="hljs-comment">// The pool removes it; this listener stops it from crashing the process.</span>
pool.<span class="hljs-title function_">on</span>(<span class="hljs-string">'error'</span>, <span class="hljs-function">(<span class="hljs-params">err</span>) =&gt;</span> <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">warn</span>(<span class="hljs-string">'idle client dropped:'</span>, err.<span class="hljs-property">message</span>));

<span class="hljs-comment">// Retry once on a dropped connection, but only when the caller says the</span>
<span class="hljs-comment">// statement is safe to run twice.</span>
<span class="hljs-keyword">export</span> <span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">query</span>(<span class="hljs-params">sql, params, { retry = <span class="hljs-literal">false</span> } = {}</span>) {
  <span class="hljs-keyword">try</span> {
    <span class="hljs-keyword">return</span> <span class="hljs-keyword">await</span> pool.<span class="hljs-title function_">query</span>(sql, params);
  } <span class="hljs-keyword">catch</span> (err) {
    <span class="hljs-keyword">if</span> (retry &amp;&amp; <span class="hljs-regexp">/Connection terminated/</span>.<span class="hljs-title function_">test</span>(err.<span class="hljs-property">message</span>)) <span class="hljs-keyword">return</span> pool.<span class="hljs-title function_">query</span>(sql, params);
    <span class="hljs-keyword">throw</span> err;
  }
}
</code></pre><p>The retry is off by default for a reason. When a connection drops, you cannot always tell whether the statement ran before it did, so only retry statements with no side effects or with an idempotency key. A plain read qualifies; an <code>INSERT</code> without a key does not, and neither does a <code>SELECT</code> that calls a function with side effects. The error check also catches only this one message, so treat it as a starting point, not complete coverage.</p>
<h3>LISTEN does not keep it awake either</h3><p>A connection waiting on <code>LISTEN</code> sits in the <code>idle</code> state, so the monitor treats it like any idle connection. Both runs suspended on schedule. After each suspend the script opened a new connection, which woke the compute, and sent <code>NOTIFY keep_awake, 'after-suspend'</code>. The listener never got it, in either run, because its connection had been closed by the suspend.</p>
<p>If you use <code>LISTEN/NOTIFY</code> for cache invalidation or live updates on a compute that scales to zero, the listener must reconnect and run <code>LISTEN</code> again, and it must not assume it saw every notification sent while it was gone. Read the current state after each reconnect.</p>
<h2>An open transaction keeps it awake for as long as it stays open</h2><p>The monitor counts any backend that is not <code>idle</code> as activity, and a session that ran <code>BEGIN</code> and then stopped is <code>idle in transaction</code>. That is the classic leaked transaction: a request handler that checks out a client, starts a transaction, and returns on an error path without <code>COMMIT</code> or <code>ROLLBACK</code>.</p>
<table>
<thead>
<tr>
<th>What was left open</th>
<th>Runs</th>
<th>Suspended within 6 minutes</th>
<th>After the transaction ended</th>
</tr>
</thead>
<tbody><tr>
<td><code>BEGIN</code> on a direct connection, then nothing</td>
<td>2</td>
<td>0 of 2</td>
<td>suspended 65 and 72 s later</td>
</tr>
<tr>
<td><code>BEGIN</code> on a pooled connection, then nothing</td>
<td>2</td>
<td>0 of 2</td>
<td>suspended 68 and 72 s later</td>
</tr>
<tr>
<td>The same, with <code>idle_in_transaction_session_timeout = '30s'</code></td>
<td>2</td>
<td>2 of 2, 102 and 98 s after the last query</td>
<td></td>
</tr>
</tbody></table>
<p>Through the pooler it behaves the same way. In transaction mode, PgBouncer pins a server connection to the client for as long as the transaction is open, so Postgres still sees a backend <code>idle in transaction</code>.</p>
<p>With the timeout set, Postgres ended the leaked session after 30 seconds, and the compute suspended about 70 seconds after that, the same delay as with nothing open. That setting is the one line worth adding to every Neon project that scales to zero, at the database or role level so it applies to every connection:</p>
<pre><code class="hljs language-sql" tabindex="0"><span class="hljs-comment">-- Ends sessions that hold a transaction open with nothing running.</span>
<span class="hljs-keyword">ALTER</span> DATABASE neondb <span class="hljs-keyword">SET</span> idle_in_transaction_session_timeout <span class="hljs-operator">=</span> <span class="hljs-string">'60s'</span>;
</code></pre><p>We checked this form too, with <code>30s</code>: the database owner role that Neon creates is allowed to run it, a new connection reported <code>30s</code> without setting anything itself, and a transaction leaked on that connection no longer kept the compute up. It suspended 101 seconds after the last query (<code>scripts/07-database-timeout.mjs</code>).</p>
<p>Pick a value longer than any legitimate pause inside your transactions. A leaked transaction is a bug anyway: it holds locks and stops vacuum from cleaning up. On a compute that scales to zero, it also bills you for every hour it stays open.</p>
<h2>A monitoring agent keeps it awake</h2><p>Monitoring agents such as <code>postgres_exporter</code> keep a connection open and run queries on a timer. We ran the same pattern: one connection to the <code>postgres</code> database, reading <code>pg_stat_database</code> every 20 seconds.</p>
<p>The compute stayed awake for the whole 6-minute window and suspended 63 seconds after the agent stopped. One run only, but it matches the code: each query moves the connection's <code>state_change</code>, and the backend check counts that whichever database the query runs in. Only the <code>pg_stat_database</code> totals leave the <code>postgres</code> database out.</p>
<p>So a database scrape every 15 or 30 seconds keeps the compute awake around the clock. If you want monitoring and scale to zero together, scrape less often than the suspend timeout and accept gaps, or read compute state from the Neon API, which does not wake the compute. That is how we watched compute state for this post without adding database activity.</p>
<h2>Health checks: the interval decides everything</h2><p>Uptime monitors, load balancer probes and Kubernetes readiness probes often run a query to prove the database is reachable. Stage 2 runs exactly that: every N seconds, open a new pooled connection, run <code>SELECT 1</code>, close it. Each configuration gets its own compute, starts from a suspended state, and runs for 12 to 60 minutes while the script records when the compute is awake.</p>
<table>
<thead>
<tr>
<th>Suspend timeout</th>
<th>Check every</th>
<th>Minutes</th>
<th>Checks</th>
<th>Checks that woke it</th>
<th>Awake</th>
<th>CU-hours per 30 days at 0.25 CU</th>
</tr>
</thead>
<tbody><tr>
<td>60 s</td>
<td>30 s</td>
<td>12</td>
<td>24</td>
<td>1</td>
<td>100%</td>
<td>180</td>
</tr>
<tr>
<td>60 s</td>
<td>90 s</td>
<td>15</td>
<td>10</td>
<td>10</td>
<td>82.2%</td>
<td>148</td>
</tr>
<tr>
<td>60 s</td>
<td>120 s</td>
<td>16</td>
<td>8</td>
<td>8</td>
<td>70.2%</td>
<td>126</td>
</tr>
<tr>
<td>60 s</td>
<td>180 s</td>
<td>18</td>
<td>6</td>
<td>6</td>
<td>40.5%</td>
<td>73</td>
</tr>
<tr>
<td>300 s</td>
<td>60 s</td>
<td>30</td>
<td>30</td>
<td>1</td>
<td>100%</td>
<td>180</td>
</tr>
<tr>
<td>300 s</td>
<td>300 s</td>
<td>40</td>
<td>8</td>
<td>1</td>
<td>100%</td>
<td>180</td>
</tr>
<tr>
<td>300 s</td>
<td>360 s</td>
<td>42</td>
<td>7</td>
<td>7</td>
<td>90.6%</td>
<td>163</td>
</tr>
<tr>
<td>300 s</td>
<td>420 s</td>
<td>42</td>
<td>6</td>
<td>6</td>
<td>74.7%</td>
<td>135</td>
</tr>
<tr>
<td>300 s</td>
<td>600 s</td>
<td>60</td>
<td>6</td>
<td>6</td>
<td>52.1%</td>
<td>94</td>
</tr>
</tbody></table>
<p>"Checks that woke it" counts the checks for which the last API reading before the check showed the compute suspended. When that number is 1, only the very first check of the run had to wake the compute, and every later check found it already awake.</p>
<p><strong>Share of time the compute was awake, by health-check interval</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>60 s timeout, check every 30 s</td>
<td>100%</td>
<td>60 s suspend timeout</td>
</tr>
<tr>
<td>60 s timeout, check every 90 s</td>
<td>82.2%</td>
<td>60 s suspend timeout</td>
</tr>
<tr>
<td>60 s timeout, check every 120 s</td>
<td>70.2%</td>
<td>60 s suspend timeout</td>
</tr>
<tr>
<td>60 s timeout, check every 180 s</td>
<td>40.5%</td>
<td>60 s suspend timeout</td>
</tr>
<tr>
<td>5 min timeout, check every 1 min</td>
<td>100%</td>
<td>5 min suspend timeout (default)</td>
</tr>
<tr>
<td>5 min timeout, check every 5 min</td>
<td>100%</td>
<td>5 min suspend timeout (default)</td>
</tr>
<tr>
<td>5 min timeout, check every 6 min</td>
<td>90.6%</td>
<td>5 min suspend timeout (default)</td>
</tr>
<tr>
<td>5 min timeout, check every 7 min</td>
<td>74.7%</td>
<td>5 min suspend timeout (default)</td>
</tr>
<tr>
<td>5 min timeout, check every 10 min</td>
<td>52.1%</td>
<td>5 min suspend timeout (default)</td>
</tr>
</tbody></table>
<p><em>A new pooled connection and SELECT 1 every N seconds, one 0.25 CU compute per row, 12 to 60 minutes each, one run per row. Estimated by polling the Neon API; matches Neon's operations log within half a percentage point. Data: results/summary.json in The-DevOps-Daily/neon-keep-awake.</em></p>
<p>The reason a 5-minute check beats a 5-minute timeout is the time between a wake and the suspend that follows it. Where the compute suspended between checks, we can estimate that cycle, from the moment the check started connecting to Neon's <code>suspended_at</code>:</p>
<ul>
<li>With the 60-second timeout, a compute woken by a single check suspended 71 to 87 seconds later (24 cycles, median 74 s).</li>
<li>With the 5-minute timeout, it suspended 312 to 327 seconds later (19 cycles on three computes, median 314 s).</li>
</ul>
<p>So in our runs, with the default timeout, a check that came within 312 seconds of the previous one always found the compute still up and started the clock again, and a check every 300 seconds kept it up for the whole 40-minute run. That is what we observed, not a guaranteed grace period: the suspend came 12 to 27 seconds after the timeout here, and Neon does not document that gap.</p>
<p>Make the interval a little longer and the compute does get to sleep between checks, but not for long. Every check pays for a full cycle of about 5 minutes and 15 seconds, so every 6 minutes it was awake 90.6% of the time and every 7 minutes 74.7%. A check every 10 minutes woke the compute 6 times out of 6, and each wake kept it up 312 to 313 seconds for a query that took about a second. That is 52% of the hour, for a database that did nothing else.</p>
<h3>What a wake costs your users</h3><p>Letting the compute sleep has a price too: the first request after a quiet stretch waits for the compute to start.</p>
<table>
<thead>
<tr>
<th></th>
<th>Median</th>
<th>Max</th>
<th>Checks</th>
</tr>
</thead>
<tbody><tr>
<td>Check that found the compute awake</td>
<td>308 to 339 ms per run</td>
<td></td>
<td>59 checks, 3 runs</td>
</tr>
<tr>
<td>Check that had to wake it</td>
<td>847 to 1,134 ms per run</td>
<td>1,878 ms</td>
<td>46 wakes, 9 runs</td>
</tr>
<tr>
<td>Plain TCP connect to the region (no TLS, no query)</td>
<td>48 ms</td>
<td></td>
<td>10</td>
</tr>
</tbody></table>
<p>These are end-to-end times for the whole check: open a new TLS connection, authenticate, run the query and close. That is why even a warm check takes about 300 ms from a client 48 ms away. In three of the runs only the first check woke the compute, so their wake figure is a single sample. Comparing across runs rather than pair by pair, waking the compute added roughly 0.5 to 0.8 seconds, and the slowest wake took 1.9 seconds. None of the 105 checks failed. For context, our <a href="https://devops-daily.com/posts/neon-vs-supabase-free-tier-benchmarks" rel="noopener noreferrer">Neon free tier benchmarks</a> measured cold starts with a median of 568 ms and a p95 of 1.06 s from a client in the same region, with a different method.</p>
<p>For a health check, an extra second does not matter. For the first user request of the morning it might, and that is the real trade-off you are making when you decide how long the compute may sleep.</p>
<h2>What it costs</h2><p>Neon bills compute in CU-hours: compute size times the hours it runs. A compute kept awake by a light pattern like these sits at its minimum size, so the cost is the minimum CU times the hours awake (real load can scale it up, which costs more). With list prices from <a href="https://neon.com/pricing" rel="noopener noreferrer">Neon's pricing page</a> ($0.106 per CU-hour on Launch, $0.222 on Scale) and the Free plan's 100 CU-hours per project per month, for one 0.25 CU compute over 30 days with nothing else using it:</p>
<table>
<thead>
<tr>
<th>What keeps it awake</th>
<th>Awake</th>
<th>CU-hours per 30 days at 0.25 CU</th>
<th>Launch</th>
<th>Scale</th>
<th>Free plan</th>
</tr>
</thead>
<tbody><tr>
<td>Health check every 1 or 5 min, default timeout</td>
<td>100%</td>
<td>180</td>
<td>$19.08</td>
<td>$39.96</td>
<td>allowance used up after 400 hours, about day 17</td>
</tr>
<tr>
<td>Health check every 6 min, default timeout</td>
<td>90.6%</td>
<td>163</td>
<td>$17.28</td>
<td>$36.19</td>
<td>used up around day 19</td>
</tr>
<tr>
<td>Health check every 10 min, default timeout</td>
<td>52.1%</td>
<td>94</td>
<td>$9.94</td>
<td>$20.82</td>
<td>fits, with 6 CU-hours left for real traffic</td>
</tr>
<tr>
<td>Leaked transaction, monitoring agent every 20 s</td>
<td>100% while it lasts</td>
<td>up to 180</td>
<td>up to $19.08</td>
<td>up to $39.96</td>
<td>as above</td>
</tr>
<tr>
<td>Idle connections, <code>LISTEN</code></td>
<td>suspended normally</td>
<td>no sustained cost observed</td>
<td>none seen</td>
<td>none seen</td>
<td>none seen</td>
</tr>
</tbody></table>
<p>On the Free plan, Neon's plans page says that when a project runs out of CU-hours, its compute is suspended until the next billing period. A 5-minute health check on a Free project, starting the month with the full allowance and nothing else using it, does not cost money; it takes the database down for the last 13 days of a 30-day month.</p>
<p>Two things make these numbers bigger in practice:</p>
<ul>
<li><strong>Minimum size.</strong> We fixed every compute at 0.25 CU. If your compute's autoscaling minimum is 1 CU, multiply by four: an always-awake 1 CU compute is 720 CU-hours a month, $76 on Launch and $160 on Scale.</li>
<li><strong>Branches.</strong> Every branch with a compute has its own suspend timer. A preview environment per pull request, each with its own health check, keeps each preview branch's compute awake. Ten open previews with a 5-minute check are ten always-on computes.</li>
</ul>
<h2>Does Neon wake it on its own?</h2><p>A May 2026 thread in Neon's GitHub discussions, <a href="https://github.com/neondatabase/neon/discussions/12900" rel="noopener noreferrer">neondatabase/neon#12900</a>, describes Launch projects at 0.25 CU that each used about 6 CU-hours a day whatever their traffic, which is a compute awake around the clock. One reply blamed <code>check_availability</code>, an operation that Neon's control plane runs to <a href="https://neon.com/docs/manage/operations" rel="noopener noreferrer">check that a compute can start on a branch</a>. We looked for it in two places.</p>
<ul>
<li><strong>Our test project.</strong> Stage 3 watched the main branch's compute, which nothing ever connected to, for 3.9 hours after it suspended, reading its state every 15 seconds. It never woke. The project's operations log, read at the end, shows no <code>check_availability</code> and no <code>start_compute</code> for it after the first suspend, and Neon's usage counter for that branch read 368 active seconds, about the six minutes the compute ran after we created the project.</li>
<li><strong>An older project of ours.</strong> Its main compute suspended on 23 September, and we stopped using the project that evening. Its operations log shows one <code>check_availability</code> for that compute on 24 September, 929 ms long, and no <code>start_compute</code> for it afterwards. The compute's <code>suspended_at</code> still reads 23 September. Two weeks later Neon archived the idle branch.</li>
</ul>
<p>That matches Neon's own description. The operations page says the check verifies that a compute <em>can</em> start, and a <a href="https://neon.com/docs/changelog/2023-08-28" rel="noopener noreferrer">2023 changelog entry</a> says the availability checker runs on 0.25 CU computes and can use pre-started ones. So in these two records, no availability check restarted a suspended compute. That does not rule out other behavior on other projects, and we cannot tell what kept the computes in that thread awake. The patterns that kept ours awake were open transactions, frequent monitoring queries and frequent health checks.</p>
<h2>What to change</h2><ol>
<li><strong>Keep the database out of liveness checks.</strong> A liveness probe should check that the process is up, not that Postgres answers. If you must check the database, do it well below the suspend timeout's rate: with the default 5-minute timeout, every 10 minutes still kept the compute awake half the time, so think in hours, not minutes.</li>
<li><strong>Check compute state through the API instead of with a query.</strong> <code>GET /projects/{project_id}/endpoints/{endpoint_id}</code> returns <code>current_state</code> without waking anything. A monitor that alerts on the API's view of the compute, and only queries the database when a user is actually using it, keeps scale to zero working.</li>
<li><strong>Set <code>idle_in_transaction_session_timeout</code> at the database level.</strong> It ends leaked transactions, which otherwise keep the compute awake for as long as they stay open, and they are a bug anyway.</li>
<li><strong>Scrape metrics less often than the timeout, or not from the database.</strong> A monitoring agent that queries every 15 to 30 seconds turns scale to zero off.</li>
<li><strong>Expect connections to drop.</strong> Idle and <code>LISTEN</code> connections, direct or pooled, were closed when the compute suspended. Use a pool, handle its <code>error</code> event, retry only statements that are safe to repeat, and re-run <code>LISTEN</code> after a reconnect.</li>
<li><strong>If you want it awake, say so.</strong> On paid plans you can raise the suspend timeout or turn scale to zero off. For a production database with steady traffic, that is often the right call. The compute bill is the same either way, but a setting you chose is explicit and predictable, and a health check that happens to keep the compute up is neither.</li>
</ol>
<h2>What we could not conclude</h2><ul>
<li><strong>Sample sizes are small.</strong> Three runs for each idle-connection row, two for the transaction and <code>LISTEN</code> rows, one for the monitoring agent, one for the database-level timeout check, and one run per health-check row. The results agree with the activity-monitor code, which is why we trust them, but the later suspend we saw with idle connections (median 81 to 82 seconds against 68) is not explained, and those scenarios ran one after another on the same compute.</li>
<li><strong>Open transactions were watched for 6 minutes.</strong> The code says such a session counts as activity for as long as it stays open, and all four runs stayed awake for the whole window; we did not run one for hours.</li>
<li><strong>One region, one compute size.</strong> Everything ran in <code>aws-eu-central-1</code> at 0.25 CU. The time from a wake to the suspend (71 to 87 seconds with a 60-second timeout, 312 to 327 seconds with 5 minutes) may differ in other regions or under load. It also differed between computes in the same project: with the 5-minute timeout, one compute's cycles took 325 to 327 seconds every time while the others took 312 to 314. We do not know why.</li>
<li><strong>Cost is computed, not billed.</strong> Neon's per-branch usage counters (<code>active_time_seconds</code>, <code>compute_time_seconds</code>) read 0 for the short runs when we deleted their branches, and for the longer runs they were well behind our own measurement (for example 988 active seconds against our 1,877), so they seem to update with a delay. We could not check CU-hours against Neon's meter. We checked the awake time against Neon's operations log instead, and converted with list prices.</li>
<li><strong>Health checks were on a precise schedule, and the gap after the timeout is not documented.</strong> A real uptime service has some jitter, and Neon may suspend sooner after the timeout on other computes or in other regions. Either could let a 5-minute check miss now and then; we did not test that.</li>
<li><strong>We did not test logical replication or autovacuum.</strong> The code says both keep a compute awake, and Neon's docs say so for replication subscribers.</li>
</ul>
<h2>Summary</h2><p>Scale to zero on Neon worked the way the compute's own activity monitor says it should. Idle connections, pooled or direct, and <code>LISTEN</code> connections did not hold a compute up; they were cut when it suspended. Activity did hold it up: a leaked transaction, a monitoring agent, a health check every 5 minutes with the default timeout. That last one is the expensive surprise. It costs a 0.25 CU compute 180 CU-hours a month, which is about $19 on Launch, $40 on Scale, or almost two weeks of downtime on Free.</p>
<p>Run the scripts against your own project to see which of these you have:</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/neon-keep-awake
<span class="hljs-built_in">cd</span> neon-keep-awake &amp;&amp; <span class="hljs-built_in">cp</span> .env.example .<span class="hljs-built_in">env</span>   <span class="hljs-comment"># add a Neon API key</span>
npm install &amp;&amp; npm run setup &amp;&amp; npm run health-checks
</code></pre>]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Prompt Caching on DigitalOcean Inference, Measured: What Breaks It and How Long It Lasts]]></title>
      <link>https://devops-daily.com/posts/digitalocean-inference-prompt-caching-measured</link>
      <description><![CDATA[We sent 867 requests with a long on-call runbook as the system prompt to three DigitalOcean-hosted models. With the runbook first and unchanged, the cache cut the input bill about 5 times on Llama 4 Maverick and more than 12 times on DeepSeek V4.1 Flash. One timestamp on the first line turned it off on all three. Here is what else turns it off, how long a cached prefix lasted, and what prompt_cache_key does.]]></description>
      <pubDate>Mon, 19 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/digitalocean-inference-prompt-caching-measured</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[AI]]></category><category><![CDATA[LLM]]></category><category><![CDATA[Caching]]></category><category><![CDATA[FinOps]]></category><category><![CDATA[DigitalOcean]]></category>
      <content:encoded><![CDATA[<p>Most LLM features send the same long block of text on every request: a system prompt, a policy, a runbook, a list of tools. DigitalOcean Serverless Inference caches that repeated prefix automatically on the models it hosts, bills the cached part at a discount, and starts streaming sooner. You do not turn it on. You can turn it off by accident, and the request itself does not tell you.</p>
<p>We sent 867 requests to Llama 4 Maverick, DeepSeek V4.1 Flash and <code>gpt-oss-120b</code>, with an on-call runbook of up to about 67,000 tokens as the system prompt. With an 8,000-token runbook first and unchanged, the cache served <strong>99.7 percent</strong> of the prompt on Llama 4 Maverick after the first request and cut the input cost about <strong>5 times</strong>. On DeepSeek V4.1 Flash it cut it more than <strong>12 times</strong>. Then we put the current time on the first line of the system prompt, the way a lot of prompt templates do, and <strong>every request after the first missed</strong>, on all three models.</p>
<p>This post covers what we measured: which prompt layouts keep the cache and which break it, what a hit does to time to first token as the prompt grows, how long a cached prefix lasted on each model, what <code>prompt_cache_key</code> changes, and why <code>gpt-oss-120b</code> was the hardest to keep warm. Every token and cache number comes from the usage block the API returned, timings come from our client, and the harness and raw results are public.</p>
<p><a href="https://github.com/The-DevOps-Daily/do-inference-prompt-cache" rel="noopener noreferrer">The-DevOps-Daily/do-inference-prompt-cache on GitHub</a></p>
<h2>TL;DR</h2><ul>
<li>With a stable 8k-token prefix, input cost per 1,000 requests went from $1.92 to $0.39 on Llama 4 Maverick and from $2.40 to $0.19 on DeepSeek V4.1 Flash. <code>gpt-oss-120b</code> went from $0.78 to $0.35 in one run and to $0.55 in another; the share of its prompt served from cache swung between 35 and 68 percent.</li>
<li>A timestamp on the first line of the system prompt: all 57 requests after the first missed, across the three models. The same timestamp after the runbook: back to 99.5, 93.7 and 63.2 percent of tokens from cache. A user's name a quarter of the way in: only the part before it, 27 to 28 percent of the prompt, was cached.</li>
<li>At 64k tokens, a cache hit cut the time to the first streamed token from 5.0 to 1.4 seconds on Llama 4 Maverick and from 3.1 to 0.9 seconds on DeepSeek V4.1 Flash. <code>gpt-oss-120b</code> never hit at 64k in 5 tries.</li>
<li>How long a prefix stayed usable depended on the model. Llama 4 Maverick hit after a nominal 5 idle minutes and missed after 15. DeepSeek V4.1 Flash missed every prefix it had seen only once, and hit every prefix it had seen twice, up to 30 minutes later. <code>gpt-oss-120b</code> missed all 9 probes after a minute or more.</li>
<li><code>prompt_cache_key</code> kept caches apart: a second key on the same prefix missed 12 of 12 times. It did not change hit rates in our runs.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A DigitalOcean account with a Serverless Inference <a href="https://docs.digitalocean.com/products/inference/how-to/manage-model-access-keys/" rel="noopener noreferrer">model access key</a></li>
<li>Node.js 22 or later (the harness has no dependencies)</li>
<li>Optional: an API token with billing read access, for the invoice check in the repo</li>
<li>About $2 of inference credit to rerun everything; our recorded requests cost about $1.06 at list prices</li>
</ul>
<h2>How the cache works on DigitalOcean-hosted models</h2><p>The <a href="https://docs.digitalocean.com/products/inference/how-to/use-prompt-caching/" rel="noopener noreferrer">prompt caching guide</a> covers three families. Anthropic and OpenAI models have their own controls (<code>cache_control</code>, <code>prompt_cache_retention</code>). The open-weight models DigitalOcean hosts itself, which is what we tested, cache with no code change at all:</p>
<ul>
<li>The cache matches an exact token prefix, in blocks of up to 256 tokens depending on the model.</li>
<li>Writing to the cache is free. Cached input tokens are billed at the model's cache read rate.</li>
<li>Entries have no fixed expiry. They are evicted least-recently-used first, "and sooner under heavy load".</li>
<li>The cache lives on the replica that served the earlier request. The platform routes matching prefixes back to that replica, but under very high load it may send a request elsewhere, which is a miss.</li>
<li>Every response reports the cached part in <code>usage.prompt_tokens_details.cached_tokens</code> (and <code>cache_read_input_tokens</code>).</li>
</ul>
<p>The discount differs a lot between models. These are the rates for the three we tested, from the <a href="https://docs.digitalocean.com/products/inference/details/pricing/" rel="noopener noreferrer">pricing page</a> (last verified 15 October 2026) and our own invoice:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Input, per 1M tokens</th>
<th>Cached input, per 1M</th>
<th>Discount</th>
</tr>
</thead>
<tbody><tr>
<td>Llama 4 Maverick</td>
<td>$0.25</td>
<td>$0.05</td>
<td>80%</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash</td>
<td>$0.30</td>
<td>$0.006</td>
<td>98%</td>
</tr>
<tr>
<td><code>gpt-oss-120b</code></td>
<td>$0.10</td>
<td>$0.02</td>
<td>80%</td>
</tr>
</tbody></table>
<p>The pricing page lists no cache read rate for <code>gpt-oss-120b</code>; the $0.02 is the rate on the <code>Cache Read Input</code> line of our invoice preview. The caching guide says cached tokens are "discounted by up to 90%", but the pricing page and our invoice both show DeepSeek V4.1 Flash at 98 percent off.</p>
<h2>The test</h2><p>The prompt is an on-call runbook for a made-up platform: 64 services, each with an owner, an SLO, its alerts and numbered first steps with real-looking commands (<code>deployctl rollback checkout-api</code>, <code>featurectl set checkout-api.degraded true</code>). A seeded script cuts it at section boundaries to nominal sizes of 1k, 4k, 8k, 16k and 64k tokens, at about four characters per token. Each model's tokenizer counts the same text a little differently: the 8k runbook is 7,700 to 8,000 tokens, and the 64k one 64,000 to 67,000. The runbook is the system prompt, and the user message is one of 40 on-call questions ("checkout-indexer is returning 5xx after a deploy 10 minutes ago. What is the first command to run?").</p>
<p>Every request is streamed with <code>stream_options: { include_usage: true }</code>, <code>temperature</code> 0 and <code>max_tokens</code> 48. We record the time to the first streamed token, total time and the usage block. For DeepSeek V4.1 Flash and <code>gpt-oss-120b</code> that first token is usually reasoning, not answer text, because both reason before they answer. Each experiment starts from a prefix the cache has never seen: a random revision id on the first line, the same for all requests in that experiment.</p>
<p>Two definitions matter for reading the numbers:</p>
<ul>
<li>A <strong>hit</strong> is a request where at least half the prompt came from the cache, and a <strong>miss</strong> is one where under 10 percent did. <code>gpt-oss-120b</code> reports 64 cached tokens on many requests that share nothing with an earlier one, which looks like its chat template, so "cached tokens above zero" does not mean a hit.</li>
<li>Llama 4 Maverick answered <code>529 Platform overloaded</code> 55 times during the run. The harness retried each one, counted it and kept it out of the timings. The other two models refused nothing. So the 867 requests are the ones that succeeded: 922 attempts in all, plus a first run of 20 requests we discarded because it did not record its retries.</li>
</ul>
<ol>
<li><strong>00-build-prompts</strong> runbook 1k to 64k</li>
<li><strong>02-run</strong> one experiment, one model</li>
<li><strong>results/raw</strong> every usage block</li>
<li><strong>03-score</strong> hits, TTFT, cost</li>
</ol>
<h2>A stable prefix: what you get</h2><p>The first experiment is the best case: the same 8k-token runbook on 20 requests in a row, a different question each time. This chart shows how much of each request's prompt came from the cache.</p>
<p><strong>Share of the prompt served from cache, 20 requests in a row</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>1</th>
<th>2</th>
<th>3</th>
<th>4</th>
<th>5</th>
<th>6</th>
<th>7</th>
<th>8</th>
<th>9</th>
<th>10</th>
<th>11</th>
<th>12</th>
<th>13</th>
<th>14</th>
<th>15</th>
<th>16</th>
<th>17</th>
<th>18</th>
<th>19</th>
<th>20</th>
</tr>
</thead>
<tbody><tr>
<td>Llama 4 Maverick</td>
<td>0%</td>
<td>99.6%</td>
<td>99.7%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.6%</td>
<td>99.6%</td>
<td>99.7%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.8%</td>
<td>99.6%</td>
<td>99.6%</td>
<td>99.7%</td>
<td>99.8%</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash</td>
<td>0%</td>
<td>0%</td>
<td>99.2%</td>
<td>99.3%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.1%</td>
<td>99.1%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.2%</td>
<td>99.1%</td>
<td>99.1%</td>
<td>99.2%</td>
<td>99.2%</td>
</tr>
<tr>
<td>gpt-oss-120b</td>
<td>0.8%</td>
<td>98.9%</td>
<td>0%</td>
<td>99.1%</td>
<td>0.8%</td>
<td>99.1%</td>
<td>99.1%</td>
<td>0%</td>
<td>0.8%</td>
<td>98.9%</td>
<td>99%</td>
<td>99.1%</td>
<td>99.1%</td>
<td>0%</td>
<td>99.1%</td>
<td>99.1%</td>
<td>99%</td>
<td>98.9%</td>
<td>0.8%</td>
<td>99.1%</td>
</tr>
</tbody></table>
<p><em>8k-token runbook as the system prompt, one new question per request, sent one after another. cached_tokens / prompt_tokens from each response. Run on 19 October 2026.</em></p>
<p>Three different behaviors from one feature:</p>
<ul>
<li><strong>Llama 4 Maverick</strong> hit from the second request on and never missed again.</li>
<li><strong>DeepSeek V4.1 Flash</strong> missed the first two requests and hit every one after that. That was common, not universal. Counting every experiment that sent a new prefix twice within seconds, the second request missed 24 times out of 30, and the third request on the same prefix hit all 26 times we sent one.</li>
<li><strong><code>gpt-oss-120b</code></strong> missed 6 of the 19 requests after the first, scattered through the run. In a later run with the same layout, it hit only 10 of 28. A cached prefix lives on one replica, and the guide says requests can land on one that does not hold it. That fits what we saw, but the response does not say which replica served it, so we cannot confirm it.</li>
</ul>
<p>Here is what that does to the input bill, per 1,000 requests at list prices:</p>
<p><strong>Input cost per 1,000 requests, 8k-token prompt</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Llama 4 Maverick, no cache</td>
<td>1.92$</td>
<td>Without the cache</td>
</tr>
<tr>
<td>Llama 4 Maverick, measured</td>
<td>0.39$</td>
<td>Measured</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, no cache</td>
<td>2.4$</td>
<td>Without the cache</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, measured</td>
<td>0.19$</td>
<td>Measured</td>
</tr>
<tr>
<td>gpt-oss-120b, no cache</td>
<td>0.78$</td>
<td>Without the cache</td>
</tr>
<tr>
<td>gpt-oss-120b, stable run</td>
<td>0.35$</td>
<td>Measured</td>
</tr>
<tr>
<td>gpt-oss-120b, later run</td>
<td>0.55$</td>
<td>Measured</td>
</tr>
</tbody></table>
<p><em>Average over requests 2 to 20 of the stable run, scaled to 1,000, so DeepSeek's second-request miss is included. Counting the first request too gives $0.47, $0.30 and $0.37. The gpt-oss-120b later run is the key-ab experiment (both arms, requests 2 to 8 on four prefixes each, about 36% of tokens from cache), priced the same way. Input only; output is capped at 48 tokens and not compared.</em></p>
<p>On list price, <code>gpt-oss-120b</code> is the cheap one: $0.10 per million input tokens against $0.30 for DeepSeek V4.1 Flash. With a stable prefix the order flips, because DeepSeek's cached tokens cost almost nothing and it hit far more often. If most of your tokens are a repeated prompt, compare models on cached cost and measured hit rate, not on the input price. Output is a separate sum: both models spent most of their 48 output tokens on reasoning in our runs, and DeepSeek's output costs $1.20 per million against $0.70.</p>
<p>The invoice agrees: the next day, the input, cache read input and output lines on our invoice preview had grown by exactly the token counts the API reported, for all three models (<code>results/bill-check.txt</code> in the repo).</p>
<h2>What breaks it</h2><p>The guide lists the patterns that break the prefix match: a timestamp or request ID in the system prompt, per-user data before the shared instructions, tool definitions that change order. We tried three of those layouts on the same runbook, 20 requests each, each from a fresh prefix:</p>
<pre><code class="hljs language-text" tabindex="0">stable           [revision][runbook .......................][question]
timestamp-first  [time][revision][runbook .................][question]
timestamp-last   [revision][runbook .................][time][question]
user-in-middle   [revision][runbook ...][name][runbook ....][question]
</code></pre><p><strong>Share of the prompt from cache after the first request, by layout</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Llama 4 Maverick, stable</td>
<td>99.7%</td>
<td>Llama 4 Maverick</td>
</tr>
<tr>
<td>Llama 4 Maverick, time first</td>
<td>0%</td>
<td>Llama 4 Maverick</td>
</tr>
<tr>
<td>Llama 4 Maverick, time last</td>
<td>99.5%</td>
<td>Llama 4 Maverick</td>
</tr>
<tr>
<td>Llama 4 Maverick, name in middle</td>
<td>27.4%</td>
<td>Llama 4 Maverick</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, stable</td>
<td>94%</td>
<td>DeepSeek V4.1 Flash</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, time first</td>
<td>0%</td>
<td>DeepSeek V4.1 Flash</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, time last</td>
<td>93.7%</td>
<td>DeepSeek V4.1 Flash</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, name in middle</td>
<td>26.5%</td>
<td>DeepSeek V4.1 Flash</td>
</tr>
<tr>
<td>gpt-oss-120b, stable</td>
<td>67.9%</td>
<td>gpt-oss-120b</td>
</tr>
<tr>
<td>gpt-oss-120b, time first</td>
<td>0.8%</td>
<td>gpt-oss-120b</td>
</tr>
<tr>
<td>gpt-oss-120b, time last</td>
<td>63.2%</td>
<td>gpt-oss-120b</td>
</tr>
<tr>
<td>gpt-oss-120b, name in middle</td>
<td>19.5%</td>
<td>gpt-oss-120b</td>
</tr>
</tbody></table>
<p><em>Cached tokens as a share of prompt tokens over requests 2 to 20 of each run. 8k-token runbook, 19 October 2026.</em></p>
<p>A timestamp on the first line changes the first tokens of every request, so nothing after it can match. All 57 requests after the first missed on all three models (the 0.8 percent on <code>gpt-oss-120b</code> is its 64 template tokens). The requests still succeeded, and the only sign in the response was the usage block. On Llama 4 Maverick that layout cost $1.93 per 1,000 requests instead of $0.39.</p>
<p>Moving the timestamp after the runbook brought the cached share back to the stable run's level on Llama 4 Maverick and DeepSeek V4.1 Flash (99.5 and 93.7 percent), and to 63.2 percent on <code>gpt-oss-120b</code>, inside its usual swing from run to run. The name a quarter of the way in shows how prefix matching degrades. When the cache was used, it served the part before the name, 27 to 28 percent of the prompt, and nothing after it. DeepSeek V4.1 Flash and <code>gpt-oss-120b</code> also missed outright on 1 and 6 of the 19 requests.</p>
<p>The fix is in the template, not the platform:</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-comment">// Breaks the cache: the first tokens change on every request.</span>
<span class="hljs-keyword">const</span> system = <span class="hljs-string">`Current time: <span class="hljs-subst">${<span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>().toISOString()}</span>\n\n<span class="hljs-subst">${runbook}</span>`</span>;

<span class="hljs-comment">// Keeps it: the runbook is a fixed prefix and the time comes after it.</span>
<span class="hljs-keyword">const</span> system = <span class="hljs-string">`<span class="hljs-subst">${runbook}</span>\n\nCurrent time: <span class="hljs-subst">${<span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>().toISOString()}</span>`</span>;

<span class="hljs-comment">// Better still: keep the system prompt byte-identical and put</span>
<span class="hljs-comment">// per-request data in the user message.</span>
<span class="hljs-keyword">const</span> messages = [
  { <span class="hljs-attr">role</span>: <span class="hljs-string">'system'</span>, <span class="hljs-attr">content</span>: runbook },
  { <span class="hljs-attr">role</span>: <span class="hljs-string">'user'</span>, <span class="hljs-attr">content</span>: <span class="hljs-string">`On-call engineer: <span class="hljs-subst">${name}</span>\nTime: <span class="hljs-subst">${now}</span>\n\n<span class="hljs-subst">${question}</span>`</span> },
];
</code></pre><p>"Byte-identical" includes things that are easy to miss: a JSON blob serialized with keys in a different order, a tool list built from a <code>Set</code>, trailing whitespace from a template engine. If you build the prefix in code, hash it and log the hash. Two hashes for one prompt version is a cache problem waiting to show up on the bill.</p>
<h2>Bigger prompts, bigger latency win</h2><p>We ran the same comparison at four more prompt sizes: 5 requests on a fresh prefix each (all misses), and 6 requests on one prefix per size, where requests 2 to 6 that hit give the hit column. Medians of the time to the first streamed token; each miss cell is 5 requests, and each hit cell is 5 unless marked:</p>
<table>
<thead>
<tr>
<th>Prompt size</th>
<th>Llama 4 Maverick, miss / hit</th>
<th>DeepSeek V4.1 Flash, miss / hit</th>
<th><code>gpt-oss-120b</code>, miss / hit</th>
</tr>
</thead>
<tbody><tr>
<td>~1k tokens</td>
<td>940 / 773 ms</td>
<td>707 / 524 ms (4 hits)</td>
<td>599 / 564 ms (3 hits)</td>
</tr>
<tr>
<td>~4k tokens</td>
<td>939 / 807 ms</td>
<td>642 / 546 ms</td>
<td>848 / 519 ms (4 hits)</td>
</tr>
<tr>
<td>~16k tokens</td>
<td>1,458 / 878 ms</td>
<td>1,032 / 626 ms</td>
<td>1,169 / 609 ms (2 hits)</td>
</tr>
<tr>
<td>~64k tokens</td>
<td>5,015 / 1,444 ms</td>
<td>3,068 / 911 ms</td>
<td>4,770 ms / no hit</td>
</tr>
</tbody></table>
<p><strong>Time to first streamed token, nominal 64k-token prompt</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Llama 4 Maverick, miss</td>
<td>5015ms</td>
<td>Miss</td>
</tr>
<tr>
<td>Llama 4 Maverick, hit</td>
<td>1444ms</td>
<td>Hit</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, miss</td>
<td>3068ms</td>
<td>Miss</td>
</tr>
<tr>
<td>DeepSeek V4.1 Flash, hit</td>
<td>911ms</td>
<td>Hit</td>
</tr>
<tr>
<td>gpt-oss-120b, miss</td>
<td>4770ms</td>
<td>Miss</td>
</tr>
</tbody></table>
<p><em>Median of 5 requests each. gpt-oss-120b did not hit on any of 5 requests at this size. Run on 19 October 2026.</em></p>
<p>At 1k to 4k tokens the medians differed by 35 to 330 milliseconds. At 64k they differed by 2.2 seconds on DeepSeek V4.1 Flash and 3.6 seconds on Llama 4 Maverick, seconds in which nothing streams back. If you put whole documents or long tool output in the prompt, this can matter more than the discount. The groups are small and were collected a few minutes apart rather than paired, so read the shape, not the exact milliseconds.</p>
<h2>How long a cached prefix lasts</h2><p>The guide says entries have no fixed expiry and are evicted when the space is needed. We wanted to know what that means in minutes. Each prefix was sent once or twice, left alone, and asked again. The minute gaps are nominal: we counted them from the start of priming, so the real idle time was up to 35 seconds shorter (the raw data has the exact times).</p>
<table>
<thead>
<tr>
<th>Asked again after</th>
<th>Llama 4 Maverick</th>
<th>DeepSeek V4.1 Flash</th>
<th><code>gpt-oss-120b</code></th>
</tr>
</thead>
<tbody><tr>
<td>5 to 45 seconds (sent twice)</td>
<td>8 of 8 hit</td>
<td>8 of 8 hit</td>
<td>4 of 8 hit</td>
</tr>
<tr>
<td>1 minute, sent once / twice</td>
<td>hit / hit</td>
<td>miss / hit</td>
<td>miss / miss</td>
</tr>
<tr>
<td>5 minutes, sent once / twice</td>
<td>hit / hit</td>
<td>miss / hit</td>
<td>miss / miss</td>
</tr>
<tr>
<td>15 minutes, sent once / twice</td>
<td>miss / miss</td>
<td>miss / hit</td>
<td>miss / miss</td>
</tr>
<tr>
<td>30 minutes, sent once / twice</td>
<td>miss / miss</td>
<td>miss / hit</td>
<td>miss / miss</td>
</tr>
<tr>
<td>60 minutes, sent once</td>
<td>miss</td>
<td>miss</td>
<td>miss</td>
</tr>
</tbody></table>
<p>Each cell is one request (two per gap for the seconds row), so this is a sketch of one afternoon, not a guarantee. A miss also does not prove the entry was evicted; the request may have gone to a replica that never had it. It still says three useful things:</p>
<ul>
<li><strong>Llama 4 Maverick</strong> hit at the nominal 5-minute probes and missed at 15 minutes, whether it had seen the prefix once or twice. It was also the model returning 529 overload errors, and the guide says eviction comes sooner under heavy load.</li>
<li><strong>DeepSeek V4.1 Flash</strong> missed every prefix it had seen only once, even after one minute, and hit every prefix it had seen twice, up to 30 minutes later. That is the same pattern as the second-request misses above, so it may have nothing to do with time: on this model, the cache paid off from the third request.</li>
<li><strong><code>gpt-oss-120b</code></strong> hit about half the time within 45 seconds and missed all 9 probes at a minute or more.</li>
</ul>
<p>For traffic that repeats every few seconds, all three caches work. For a prefix used once every 15 minutes or so, such as a per-tenant prompt on a quiet tenant or a nightly job, plan on paying full price.</p>
<h2>Bursts on a cold prefix</h2><p>What happens when a deploy or a queue drain sends ten requests at once with a prefix nobody has sent before? We sent ten at once on a cold prefix, then ten more:</p>
<ul>
<li>DeepSeek V4.1 Flash hit on 8 of the first 10 and <code>gpt-oss-120b</code> on 7, even though none of them had seen the prefix before the burst started.</li>
<li>Llama 4 Maverick refused 5 of the first 10 with 529 (9 refusals in all) and the harness retried them a few seconds later, so its number mixes concurrent and later requests. Of the 5 that went through on the first try, 4 hit.</li>
<li>The second wave of ten hit 10 of 10 on all three models.</li>
<li>One request alone and then ten at once on another new prefix: 10, 9 and 9 of the ten hit.</li>
</ul>
<p>So a cold burst does not pay full price ten times. The responses do not show how the server scheduled the requests, only that most of them reported cached tokens.</p>
<h2>What prompt_cache_key does</h2><p><code>prompt_cache_key</code> is an optional field on the request. The guide describes it as a way to keep cached context apart inside one account, for different tenants or workloads, and as a cache reset when you roll out a new prompt. Caches are already isolated between accounts.</p>
<p>We checked the separation. On four fresh prefixes per model, we sent key A twice, then key B, then key A again, then no key at all, then key B a second time:</p>
<p><strong>node 03-score.mjs (cache-key rows)</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># a-1, a-2: key A twice. b-1: key B. a-3: key A again. none: no key. b-2: key B again.</span>
$ STAMPS_OK=... node 03-score.mjs | grep cache-key
  cache-key        a-1 0/4, a-2 4/4, b-1 0/4, a-3 4/4, none 0/4, b-2 4/4
  cache-key        a-1 0/4, a-2 3/4, b-1 0/4, a-3 4/4, none 0/4, b-2 3/4
  cache-key        a-1 0/4, a-2 2/4, b-1 0/4, a-3 1/4, none 0/4, b-2 1/4
</code></pre><p>The rows are Llama 4 Maverick, DeepSeek V4.1 Flash and <code>gpt-oss-120b</code>. The first request with key B missed 12 of 12 times across the three models, on a prefix that had just hit under key A in 9 of those 12 trials. The same prefix with no key also missed 12 of 12, so in our runs "no key" behaved as a namespace of its own. Key A kept hitting after B was used on Llama 4 Maverick and DeepSeek V4.1 Flash (8 of 8); <code>gpt-oss-120b</code> hit 1 of 4, in line with its usual misses.</p>
<p>Does a key also improve hit rates, by keeping related requests together? We ran two new prefixes side by side, one with a key and one without, 8 requests each, four times per model. Llama 4 Maverick hit 28 of 28 later requests either way, DeepSeek V4.1 Flash 24 of 28 either way, and <code>gpt-oss-120b</code> 10 of 28 either way. The guide says the key does not pin requests to a replica, and it made no difference here.</p>
<p>Use <code>prompt_cache_key</code> for what it is for: separating tenants that share a prompt template, and forcing a clean start when you ship a new prompt version. Use a value that is stable per tenant or workload. A request ID as the key gives every request its own empty cache.</p>
<h2>Watch the cached share in production</h2><p>The cache never fails loudly. A template change that puts a timestamp first looks exactly like a working deploy, until the invoice arrives. The usage block on every response is enough to catch it:</p>
<pre><code class="hljs language-javascript" tabindex="0"><span class="hljs-comment">// After each streamed response, with stream_options: { include_usage: true }</span>
<span class="hljs-keyword">const</span> prompt = usage.<span class="hljs-property">prompt_tokens</span>;
<span class="hljs-keyword">const</span> cached = usage.<span class="hljs-property">prompt_tokens_details</span>?.<span class="hljs-property">cached_tokens</span> ?? usage.<span class="hljs-property">cache_read_input_tokens</span> ?? <span class="hljs-number">0</span>;

metrics.<span class="hljs-title function_">counter</span>(<span class="hljs-string">'llm_prompt_tokens_total'</span>, prompt, { feature, model });
metrics.<span class="hljs-title function_">counter</span>(<span class="hljs-string">'llm_cached_tokens_total'</span>, cached, { feature, model });
<span class="hljs-comment">// Alert on sum(cached) / sum(prompt) per feature and model, over 15 minutes,</span>
<span class="hljs-comment">// against that feature's normal rate. A drop right after a deploy points at the prompt first.</span>
</code></pre><p>The DigitalOcean control panel shows the hit rate per model too (Serverless Inference, Analyze tab). Per-feature numbers from your own logs are more useful, because one model usually serves several features and only one of them will break.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>One afternoon, one account.</strong> All 867 recorded requests ran between 12:48 and 13:55 UTC on 19 October 2026. Hit rates and eviction depend on load: the share of <code>gpt-oss-120b</code>'s prompt served from cache went from 68 to 36 percent between two runs about an hour apart.</li>
<li><strong>Small samples where it hurts.</strong> The idle table is one request per cell, and the latency table is 5 misses and at most 5 hits per cell. They show direction, not exact values.</li>
<li><strong>No view of replicas.</strong> The response does not say which replica served it, so the DeepSeek second-request pattern and the <code>gpt-oss-120b</code> misses are described, not explained.</li>
<li><strong>First streamed token, not first answer word.</strong> For the two reasoning models, the latency numbers measure the first reasoning token.</li>
<li><strong>Synthetic prompts, ungraded answers.</strong> The runbook is generated, and answers were capped at 48 tokens. This measures input caching, not answer quality.</li>
</ul>
<h2>What to do with this</h2><ol>
<li>Put everything static first and keep it byte-identical: system prompt, policies, tool definitions, reference documents. Put the time, request IDs, user names and retrieved passages after it, or in the user message.</li>
<li>Log <code>cached_tokens / prompt_tokens</code> per feature and model, and alert when it drops. The response does not warn you any other way.</li>
<li>Budget at list price and treat the cache as a discount you have to keep earning. On <code>gpt-oss-120b</code>, and on prefixes that repeat less than every few minutes, it may not be there.</li>
<li>When you compare models for a long-prompt workload, compare cached cost and measured hit rate. On our runbook, DeepSeek V4.1 Flash had the higher input price and the lower input bill.</li>
<li>Use <code>prompt_cache_key</code> per tenant or per prompt version, never per request.</li>
<li>If your prompts are tens of thousands of tokens, the latency win is worth more than the discount: on our 64k prompt, a hit reached the first streamed token about 3.5 times faster on Llama 4 Maverick and 3.4 times faster on DeepSeek V4.1 Flash.</li>
</ol>
<p>The cache is free, automatic and easy to break. Five minutes with the usage block tells you whether yours is working, and the <a href="https://github.com/The-DevOps-Daily/do-inference-prompt-cache" rel="noopener noreferrer">harness</a> runs the same tests on your own prompt if you replace the runbook. If you want the per-feature cost side of this, <a href="https://devops-daily.com/posts/ai-feature-cost-before-finance-does">Knowing What Your AI Feature Costs Before Finance Does</a> shows how to tag usage by feature on one key.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Neon Realtime, Measured: Live SQL Queries Replace the LISTEN/NOTIFY Layer We Built by Hand]]></title>
      <link>https://devops-daily.com/posts/neon-realtime-live-queries-measured</link>
      <description><![CDATA[Neon Realtime turns a SQL query into a live subscription. We rebuilt our hand-made LISTEN/NOTIFY layer on it as a live order board and measured it on a real Neon project: 48,600 writes at up to 100 per second, which queries are re-sent in full on every write, what a reconnect delivers, scale to zero, and 14 attempts to tamper with a sealed query.]]></description>
      <pubDate>Mon, 19 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/neon-realtime-live-queries-measured</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Neon]]></category><category><![CDATA[Postgres]]></category><category><![CDATA[Realtime]]></category><category><![CDATA[Live Queries]]></category><category><![CDATA[Drizzle]]></category><category><![CDATA[WebSockets]]></category><category><![CDATA[Benchmarks]]></category>
      <content:encoded><![CDATA[<p>In July we built realtime for a Neon app by hand: a Neon Function holding one server-sent events stream per browser, a Postgres <code>LISTEN</code> connection per function isolate, and a <code>pg_notify</code> call after every write. It kept one counter in sync. In August we did it again for a chat, with WebSockets in place of SSE and the same <code>LISTEN</code>/<code>NOTIFY</code> underneath.</p>
<p><a href="https://neon.com/docs/realtime/overview" rel="noopener noreferrer">Neon Realtime</a>, in public beta since October 15, replaces all of that with a SQL query you subscribe to. We rebuilt the idea as a live order board on Drizzle and measured it on a real Neon project in <code>aws-eu-central-1</code>, from a client about 32 ms away.</p>
<p><strong>A plain live query kept pace with the write itself.</strong> Across 16,200 writes at 1, 10 and 100 per second, the update reached the subscriber 35 to 43 ms (p50) after the write was issued, while the write's own commit took 34 to 40 ms to come back. Joins and aggregates were live too. All 48,600 writes in the matrix reached the subscriber, and every check we ran on the updates passed (which rows each one added, how many it removed, the size of every full result, the aggregate's total order count). We did not check every column value.</p>
<p><strong>But in this beta, most of the joins we tried, every aggregate and any <code>ORDER BY</code> were re-sent in full on every write.</strong> The three-table join we first wrote for the board pulled 27 KB per write on a 200-row result, 52 times what the plain query used, and the cost grows with the result: 134 KB per write at 1,000 rows. At 100 writes per second, Neon kept up with those full results in our two complete raw-protocol runs. The Realtime SDK on our client needed about 9.4 ms of CPU for each one, and in one of our two SDK runs it fell about two seconds behind within a minute.</p>
<p>An open subscription let the compute scale to zero under it, and kept working when the next write woke it. The board did too, once its API stopped querying Postgres on every renewal. None of 11 tampered, forged or cross-branch attempts disclosed a victim row, an expired capability was refused, and a copied valid one kept working until it expired, which is how Neon documents it.</p>
<p><a href="https://github.com/The-DevOps-Daily/neon-realtime-measured" rel="noopener noreferrer">The-DevOps-Daily/neon-realtime-measured on GitHub</a></p>
<blockquote>
<p><strong>Warning</strong></p>
<p>Realtime is in public beta, and Neon publishes no benchmarks for it yet: its own docs say performance work is still under way. Every number here is a beta number from one project, one region and one client machine, recorded on October 16, 2026, with <code>@neon/realtime</code> 0.1.0.</p>
</blockquote>
<h2>TLDR</h2><ul>
<li><strong>Setup is one deploy.</strong> <code>neon deploy</code> with <code>preview: { realtime }</code> in <code>neon.ts</code> turned it on in about 19 seconds, with no waitlist. The primary's <code>wal_level</code> became <code>logical</code>; nothing else changed there that we could see.</li>
<li><strong>A plain live query is about as fast as the write.</strong> p50 35 to 43 ms from write issued to update applied, within 4 ms of the write's own acknowledgement, from a client 32 ms away. p99 53 to 85 ms.</li>
<li><strong>All 48,600 writes arrived</strong>, and every check on the updates passed: rows added, rows removed, result sizes and aggregate totals.</li>
<li><strong>Most query forms we tried are re-sent whole on every write:</strong> a join to <code>customers</code>, <code>EXISTS</code>, <code>IN</code>, aggregates, <code>ORDER BY</code> and <code>LIMIT</code>. That is about 134 bytes per row each time, 27 KB per write at 200 rows. Single-table filters and a membership join pinned to one row stay incremental.</li>
<li><strong>At 100 full results a second, our client was the limit.</strong> Two complete raw-protocol runs kept up. The SDK on our client needs about 9.4 ms of CPU per 200-row result, and one of two SDK runs fell two seconds behind.</li>
<li><strong>Sessions end now and then, and a reconnect brings back the current result.</strong> The server ended a session 11 times in an hour, and the SDK recovered in about 1.2 s each time. After a 75-second network cut, its backoff took 33 to 48 s to reconnect, and the result then held the same rows as the database.</li>
<li><strong>Open subscriptions let the compute scale to zero</strong>, unless the renew path queries Postgres: a membership lookup on every seal kept ours awake for the whole 300 s we watched.</li>
<li><strong>Sealed queries held, and the access rule belongs in the query.</strong> No tampered, forged or cross-branch attempt disclosed a victim row. A removed member's result was empty 8 ms after the removal, against 56 s of leaked orders with a check at seal time only.</li>
<li><strong>Budget the bytes.</strong> Neon's docs say network transfer is billed as usual, even though the project's meter showed far less than our test received.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Neon project in a Realtime region: <code>aws-us-east-1</code>, <code>aws-us-east-2</code>, <code>aws-eu-central-1</code> or <code>aws-ap-southeast-1</code> during the beta. There is no waitlist.</li>
<li>The Neon CLI (<code>npm i -g neon</code>) and Node.js 22.18 or later (the repo's scripts run as TypeScript with Node's type stripping).</li>
<li>The <a href="https://github.com/The-DevOps-Daily/neon-realtime-measured" rel="noopener noreferrer">companion repository</a>, if you want to rerun anything. Every script writes its raw output to <code>results/</code>.</li>
<li>Our earlier posts for context: <a href="https://devops-daily.com/posts/neon-functions-realtime-without-websockets">Realtime Without a WebSocket Service</a> and <a href="https://devops-daily.com/posts/neon-realtime-chat-with-auth">Realtime Chat With Auth</a>.</li>
</ul>
<h2>What we are replacing</h2><p>The July layer had three moving parts, all ours:</p>
<ol>
<li>A Neon Function holding one server-sent events stream open per browser.</li>
<li>One <code>LISTEN counter_updates</code> connection per function isolate, on the direct connection string, because the transaction pooler cannot hold a session.</li>
<li>A <code>pg_notify</code> call after every write, carrying the new value.</li>
</ol>
<p>Its own limits section said where it stopped. <code>NOTIFY</code> is fire-and-forget, so a browser that is offline at that moment misses the event. Payloads are capped at 8,000 bytes. And the app decides what each event means for the screen: a notify says "something changed", not "here is the new result".</p>
<p>With Realtime, the subscription is the query the screen shows. There is no function, listener or notify call in the app:</p>
<p><strong>The live part of the app, before and after</strong></p>
<p><strong>July: LISTEN/NOTIFY + SSE</strong></p>
<pre><code class="hljs language-typescript" tabindex="0"><span class="hljs-comment">// Function: one LISTEN connection per isolate, on the unpooled URL</span>
<span class="hljs-keyword">const</span> listener = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Client</span>({ <span class="hljs-attr">connectionString</span>: env.<span class="hljs-property">postgres</span>.<span class="hljs-property">databaseUrlUnpooled</span> })
<span class="hljs-keyword">await</span> listener.<span class="hljs-title function_">connect</span>()
<span class="hljs-keyword">await</span> listener.<span class="hljs-title function_">query</span>(<span class="hljs-string">'LISTEN counter_updates'</span>)
listener.<span class="hljs-title function_">on</span>(<span class="hljs-string">'notification'</span>, <span class="hljs-function">(<span class="hljs-params">msg</span>) =&gt;</span> {
  <span class="hljs-keyword">for</span> (<span class="hljs-keyword">const</span> c <span class="hljs-keyword">of</span> clients) c.<span class="hljs-title function_">enqueue</span>(<span class="hljs-title function_">encode</span>(<span class="hljs-string">`data: <span class="hljs-subst">${msg.payload}</span>\n\n`</span>))
})

<span class="hljs-comment">// Every write path has to remember to notify</span>
<span class="hljs-keyword">await</span> pool.<span class="hljs-title function_">query</span>(<span class="hljs-string">'SELECT pg_notify($1, $2)'</span>, [<span class="hljs-string">'counter_updates'</span>, <span class="hljs-title class_">String</span>(row.<span class="hljs-property">value</span>)])

<span class="hljs-comment">// Browser</span>
<span class="hljs-keyword">const</span> events = <span class="hljs-keyword">new</span> <span class="hljs-title class_">EventSource</span>(<span class="hljs-string">'/events'</span>)
events.<span class="hljs-property">onmessage</span> = <span class="hljs-function">(<span class="hljs-params">e</span>) =&gt;</span> <span class="hljs-title function_">render</span>(<span class="hljs-title class_">Number</span>(e.<span class="hljs-property">data</span>))
</code></pre><p><strong>Now: a sealed live query</strong></p>
<pre><code class="hljs language-typescript" tabindex="0"><span class="hljs-comment">// API: seal the query the board shows, scoped by the user id from the verified token</span>
app.<span class="hljs-title function_">post</span>(<span class="hljs-string">'/api/board/live'</span>, <span class="hljs-title function_">async</span> (c) =&gt; {
  <span class="hljs-keyword">const</span> userId = <span class="hljs-keyword">await</span> <span class="hljs-title function_">requireUser</span>(c)
  <span class="hljs-keyword">const</span> { storeId } = <span class="hljs-keyword">await</span> c.<span class="hljs-property">req</span>.<span class="hljs-title function_">json</span>()
  <span class="hljs-keyword">return</span> c.<span class="hljs-title function_">json</span>(<span class="hljs-keyword">await</span> realtime.<span class="hljs-title function_">seal</span>({ <span class="hljs-attr">query</span>: <span class="hljs-title function_">openOrdersQuery</span>(db, userId, storeId) }))
})

<span class="hljs-comment">// Writes are ordinary writes. There is nothing to notify.</span>

<span class="hljs-comment">// Browser (React)</span>
<span class="hljs-keyword">const</span> { data, status } = <span class="hljs-title function_">useLiveQuery</span>(sealedQuery, { refreshQuery })
</code></pre><p>The browser never sees SQL. The API builds the query with the user id from a verified Neon Auth token, and <code>realtime.seal()</code> encrypts the SQL, the parameters and the database name into a capability that expires after 60 seconds. The client can subscribe to it but cannot read or change it, and it renews by asking the API again.</p>
<h2>Setup</h2><p>Realtime is enabled per branch. In <code>neon.ts</code> it sits next to Neon Auth, which the board uses for sign-in:</p>
<pre><code class="hljs language-ts" tabindex="0"><span class="hljs-keyword">import</span> { defineConfig } <span class="hljs-keyword">from</span> <span class="hljs-string">'@neon/config/v1'</span>;

<span class="hljs-keyword">export</span> <span class="hljs-keyword">default</span> <span class="hljs-title function_">defineConfig</span>({
  <span class="hljs-attr">auth</span>: <span class="hljs-literal">true</span>,
  <span class="hljs-attr">preview</span>: {
    <span class="hljs-comment">// Browsers may connect only from these origins. Servers and scripts</span>
    <span class="hljs-comment">// that send no Origin header are not restricted by this list.</span>
    <span class="hljs-attr">realtime</span>: { <span class="hljs-attr">allowedOrigins</span>: [<span class="hljs-string">'http://localhost:5173'</span>] },
  },
});
</code></pre><p>One deploy applied both and wrote the variables the app needs:</p>
<p><strong>one deploy enables Neon Auth and Realtime on the branch</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ npx neon deploy
→ Applying to branch main (br-frosty-queen-baejcesa)
Applied changes
  + Neon Auth
  + Realtime

Utilized services: Postgres, Neon Auth, Realtime
Pulled 7 Neon variables into .<span class="hljs-built_in">env</span>
  Postgres        DATABASE_URL, DATABASE_URL_UNPOOLED
  Branch          NEON_BRANCH
  Neon Auth       NEON_AUTH_BASE_URL, NEON_AUTH_JWKS_URL
  Realtime        NEON_REALTIME_URL, NEON_REALTIME_SECRET
</code></pre><p>We read the primary before and after:</p>
<ul>
<li><code>wal_level</code> went from <code>replica</code> to <code>logical</code>, because Realtime consumes logical replication.</li>
<li>In three checks (before, after, and while live queries were open), the primary showed no new replication slot, publication or WAL sender. The Neon API still listed a single read-write compute.</li>
</ul>
<p>Neon's FAQ says the incremental work "is incurred on a replica". We found no replica we could query, so we could not measure its lag on its own. What we measured is the time from a write to the update arriving, which includes it.</p>
<p>The CLI is the quickest way to see a live query. Here is a join, subscribed in one terminal while another made three transactions:</p>
<p><strong>a join as a live query, from the CLI</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># terminal 1: subscribe to a join</span>
$ npx neon realtime subscribe <span class="hljs-string">"SELECT o.id, o.status, o.total_cents, c.name AS customer FROM orders o JOIN customers c ON c.id = o.customer_id"</span>
Live query
Project   purple-term-17791629
Branch    main (br-frosty-queen-baejcesa)
Database  neondb
Mode      Materialized result
Query     SELECT o.id, o.status, o.total_cents, c.name AS customer FROM orders o JOIN customers c ON c.id = o.customer_id
[2026-10-16T14:46:41.545Z] Connecting…

[2026-10-16T14:46:43.904Z] Initial snapshot · 0 rows
Query returned no rows.
[2026-10-16T14:46:43.905Z] Live — waiting <span class="hljs-keyword">for</span> changes.
<span class="hljs-comment"># terminal 2: seed a store, then three transactions three seconds apart</span>
$ node scripts/smoke-write.ts
[2026-10-16T14:46:48.851Z] insert order { <span class="hljs-built_in">id</span>: 1, txid: <span class="hljs-string">'1052'</span> }
[2026-10-16T14:46:51.906Z] update order { <span class="hljs-built_in">id</span>: 1, txid: <span class="hljs-string">'1053'</span> }
[2026-10-16T14:46:54.947Z] insert + update <span class="hljs-keyword">in</span> one transaction { new_id: 2, txid: <span class="hljs-string">'1054'</span> }
<span class="hljs-comment"># terminal 1: each transaction arrives as a complete new result (Rows), not as a change</span>
[2026-10-16T14:46:48.844Z] Rows · 0 rows
Query returned no rows.

[2026-10-16T14:46:48.862Z] Rows · 1 row
ID  Status  Total Cents  Customer
1   new     1250         Ada

[2026-10-16T14:46:51.913Z] Rows · 1 row
ID  Status     Total Cents  Customer
1   preparing  1250         Ada

[2026-10-16T14:46:54.952Z] Rows · 2 rows
ID  Status  Total Cents  Customer
1   ready   1250         Ada
2   new     830          Grace
</code></pre><p>Each transaction came back as a complete new result (<code>Rows</code>), not as a change. In a second run with <code>--changes</code>, the same three kinds of transaction against a plain <code>SELECT ... FROM orders WHERE store_id = 1</code> arrived as small change sets, <code>Transaction 1056 · 1 upsert, 0 removals</code> and so on. That difference turned out to matter more than anything else in this post.</p>
<h2>How we measured</h2><p>Everything ran from one client machine against the project in <code>aws-eu-central-1</code>. <code>SELECT 1</code> over a direct connection took 31 to 39 ms (p50 per run, median 32 ms), and a WebSocket ping to the Live SQL API 34 to 45 ms (median 41 ms). That round trip is the floor under every latency below.</p>
<p><strong>The workload.</strong> Each run creates a fresh store with 200 orders. The writer is open-loop: it issues writes on a fixed schedule (1, 10 or 100 per second) whether or not earlier ones have finished, over a pool of 12 direct connections. When its timer runs late it catches up, and latency counts from the moment each write was actually issued, including any wait for a free connection. Each write is one transaction that inserts an order and deletes the store's oldest one, so the live result stays at 200 rows for the whole run, and the three shapes are compared at the same size. Each write returns its own transaction id from <code>pg_current_xact_id()</code>.</p>
<p><strong>Three query shapes</strong>, written with Drizzle and sealed by the SDK's server half:</p>
<ul>
<li>plain: <code>orders</code> filtered by <code>store_id</code></li>
<li>join: <code>orders</code> joined to <code>customers</code> for the customer's name and to <code>store_members</code> for access</li>
<li>aggregate: count and revenue per status for the store</li>
</ul>
<p><strong>Matching updates to writes.</strong> The subscriber runs the browser client (<code>createRealtimeClient</code>, with the <code>ws</code> package as its WebSocket) in a separate process on the same machine, and both processes stamp events from the same clock. An incremental update carries the ids of the transactions it covers, so it matches directly. A full result carries no ids, but the wire protocol opens each new result with the Postgres snapshot it was read at (<code>xmin</code>, <code>xmax</code>, <code>xip</code>), and the scorer applies Postgres's own visibility rule to work out which writes it includes. Latency runs from the moment the writer issued a statement to the moment the SDK handed over the first update that included it.</p>
<p><strong>Checking answers, not only timing.</strong> Every update is checked against the writes. An incremental batch must add exactly the rows its transactions inserted (by the sequence number each write stamps on its row) and remove one row for each write that deleted one. A full result must hold every stamped row the visible writes leave in the window, and exactly as many rows as the window should have. An aggregate's order counts must add up to the window. These checks do not cover which seed or deleted rows were involved, per-status counts, revenue or other column values. A self-test corrupts copies of recorded runs (a dropped update, a wrong row, a missing row, an aggregate off by one, a stall, a line from another run) and checks that the scorer reports each one.</p>
<p><strong>Separating our own client.</strong> In our first recording of the matrix, other jobs on the client machine pushed it into swap, and our recorder processes froze for up to 11 seconds at a time. A frozen recorder looks exactly like a slow server in the timestamps. So the recorders now log every gap of more than 60 ms in their own event loops, the scorer sets apart writes whose window overlaps one, and the first recording stays in the repo marked as superseded. Each configuration ran twice, in two rounds with the shape order rotated.</p>
<h2>The latency</h2><p><strong>Write issued to update applied, p50</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>1 write/s</th>
<th>10 writes/s</th>
<th>100 writes/s</th>
</tr>
</thead>
<tbody><tr>
<td>plain</td>
<td>41.9ms</td>
<td>42.8ms</td>
<td>34.7ms</td>
</tr>
<tr>
<td>join</td>
<td>56.2ms</td>
<td>53.6ms</td>
<td>60.4ms</td>
</tr>
<tr>
<td>aggregate</td>
<td>47.2ms</td>
<td>42.4ms</td>
<td>40.7ms</td>
</tr>
<tr>
<td>write acknowledged</td>
<td>42.5ms</td>
<td>38.4ms</td>
<td>35ms</td>
</tr>
</tbody></table>
<p><em>Neon Realtime beta, aws-eu-central-1, client about 32 ms away. Each point pools two runs: 600 writes at 1/s, 3,600 at 10/s, 12,000 at 100/s. The live result is held at 200 rows. Dashed: how long the write itself took to be acknowledged, all shapes.</em></p>
<p>All 18 runs, 48,600 writes:</p>
<table>
<thead>
<tr>
<th>Shape</th>
<th>Writes/s</th>
<th>Writes</th>
<th>p50</th>
<th>p90</th>
<th>p99</th>
<th>max</th>
<th>After the commit ack, p50</th>
</tr>
</thead>
<tbody><tr>
<td>plain</td>
<td>1</td>
<td>600</td>
<td>41.9</td>
<td>44.8</td>
<td>52.6</td>
<td>1,423</td>
<td>3.5</td>
</tr>
<tr>
<td>plain</td>
<td>10</td>
<td>3,600</td>
<td>42.8</td>
<td>46.9</td>
<td>58.6</td>
<td>938</td>
<td>2.7</td>
</tr>
<tr>
<td>plain</td>
<td>100</td>
<td>12,000</td>
<td>34.7</td>
<td>37.5</td>
<td>85.0</td>
<td>1,011</td>
<td>1.7</td>
</tr>
<tr>
<td>join</td>
<td>1</td>
<td>600</td>
<td>56.2</td>
<td>64.9</td>
<td>90.3</td>
<td>105</td>
<td>12.2</td>
</tr>
<tr>
<td>join</td>
<td>10</td>
<td>3,600</td>
<td>53.6</td>
<td>59.8</td>
<td>94.4</td>
<td>1,335</td>
<td>17.5</td>
</tr>
<tr>
<td>join</td>
<td>100</td>
<td>12,000</td>
<td>60.4</td>
<td>244</td>
<td>2,083</td>
<td>2,320</td>
<td>23.8</td>
</tr>
<tr>
<td>aggregate</td>
<td>1</td>
<td>600</td>
<td>47.2</td>
<td>50.7</td>
<td>63.0</td>
<td>176</td>
<td>3.9</td>
</tr>
<tr>
<td>aggregate</td>
<td>10</td>
<td>3,600</td>
<td>42.4</td>
<td>44.9</td>
<td>120</td>
<td>1,220</td>
<td>4.0</td>
</tr>
<tr>
<td>aggregate</td>
<td>100</td>
<td>12,000</td>
<td>40.7</td>
<td>47.6</td>
<td>648</td>
<td>1,374</td>
<td>7.2</td>
</tr>
</tbody></table>
<p>All times are in milliseconds, from the write being issued to the update that included it. The last column is the same, measured from the moment the writer got its commit acknowledged; the update usually arrived within a few milliseconds of the writer hearing back, and sometimes before.</p>
<p>Three things in that table need explaining.</p>
<p><strong>The plain query is about as fast as the network allows.</strong> Its p50 is within 4 ms of the write's own acknowledgement at every rate. Both the writes and the updates got faster as the rate went up, by about 7 ms between 1 and 100 writes per second. The likely reason is our client, not Neon: its Wi-Fi saves power when traffic is light (more on that at the end).</p>
<p><strong>The join costs 11 to 26 ms more than the plain query at p50,</strong> and the reason is not the join itself. It is that every write re-sends the whole result, which the next section is about. At 100 writes per second its tail is our client falling behind, also below.</p>
<p><strong>The maximums of about one second at most rates are the server ending the session.</strong> Eleven times in 59 minutes of subscriptions, a subscription received <code>connection_error: backend_session_lost</code> ("live query session ended; reconnect required"). Every time, the SDK reconnected and was live again 0.95 to 1.51 s later (median 1.2 s), with a complete fresh result. The scorer tied 233 writes slower than 300 ms to these episodes, 66 of them at 1 or 10 writes per second, and they account for every maximum above 900 ms at those rates. All eleven came at the same second past the minute (:42 or :43), as if a task on a 60-second cycle sometimes ends sessions. Our first, superseded recording had nine more. We cannot see the cause from the client.</p>
<p>At 100 writes per second, our own client stalled during part of the plain and aggregate runs (107 and 433 writes overlap a stall); without those writes, the p99 is 46 ms for the plain query and 92 ms for the aggregate. Treat those two as diagnostics, not corrected Neon numbers: a slow write has more chances to overlap a stall, and at high rates the SDK's own work causes some of the stalls.</p>
<h2>Most queries are re-sent whole on every write</h2><p>Neon's CLI docs say "some queries, such as ones with <code>ORDER BY</code>, can get a complete replacement for a change". In our tests, most query forms got one on every write. We subscribed to several forms of the same question at once, on a store with 300 orders, and made three single-row inserts:</p>
<table>
<thead>
<tr>
<th>Query form (300 orders in the store)</th>
<th>Rows in result</th>
<th>On each insert</th>
<th>Bytes per write</th>
</tr>
</thead>
<tbody><tr>
<td><code>WHERE store_id = $1</code></td>
<td>300</td>
<td>incremental change</td>
<td>479</td>
</tr>
<tr>
<td><code>... ORDER BY id DESC LIMIT 20</code></td>
<td>20</td>
<td>full result again</td>
<td>2,947</td>
</tr>
<tr>
<td><code>JOIN customers</code> for the name</td>
<td>300</td>
<td>full result again</td>
<td>37,346</td>
</tr>
<tr>
<td><code>JOIN store_members</code>, pinned to one user and store</td>
<td>300</td>
<td>incremental change</td>
<td>472</td>
</tr>
<tr>
<td><code>EXISTS (SELECT 1 FROM store_members ...)</code></td>
<td>300</td>
<td>full result again</td>
<td>33,245</td>
</tr>
<tr>
<td><code>store_id IN (SELECT ... FROM store_members ...)</code></td>
<td>300</td>
<td>full result again</td>
<td>33,245</td>
</tr>
<tr>
<td><code>count(*) ... GROUP BY status</code></td>
<td>3</td>
<td>full result again</td>
<td>977</td>
</tr>
<tr>
<td><code>count(*)</code></td>
<td>1</td>
<td>full result again</td>
<td>760</td>
</tr>
</tbody></table>
<p>Of these forms, only two stayed incremental: a single-table filter, and a join to <code>store_members</code> whose <code>WHERE</code> clause pins it to one row. Everything else, including <code>EXISTS</code> and <code>IN</code>, which are the obvious ways to write a membership check, was re-sent in full: 110 to 125 bytes per row on every write, whether the write touched one row or all of them.</p>
<p>A second set, against the board's own queries and three kinds of write:</p>
<table>
<thead>
<tr>
<th>Query form</th>
<th>Order insert</th>
<th>Order marked done</th>
<th>Customer insert</th>
</tr>
</thead>
<tbody><tr>
<td>Board: open orders (<code>JOIN store_members</code>, <code>status &lt;&gt; 'done'</code>)</td>
<td>change, 396 B</td>
<td>change, 354 B</td>
<td>nothing sent</td>
</tr>
<tr>
<td>Board: customers (<code>JOIN store_members</code>)</td>
<td>nothing sent</td>
<td>nothing sent</td>
<td>change, 389 B</td>
</tr>
<tr>
<td>Board: counts per status (<code>GROUP BY</code>)</td>
<td>full result, 885 B</td>
<td>full result, 986 B</td>
<td>nothing sent</td>
</tr>
<tr>
<td><code>JOIN customers</code>, both tables filtered to the store</td>
<td>full result, 37,180 B</td>
<td>full result, 37,175 B</td>
<td>full result, 37,175 B</td>
</tr>
<tr>
<td><code>ORDER BY id</code>, no <code>LIMIT</code></td>
<td>full result, 33,104 B</td>
<td>full result, 33,099 B</td>
<td>nothing sent</td>
</tr>
<tr>
<td><code>LIMIT 50</code>, no <code>ORDER BY</code></td>
<td>full result, 5,978 B</td>
<td>full result, 5,972 B</td>
<td>nothing sent</td>
</tr>
<tr>
<td><code>SELECT DISTINCT status</code></td>
<td>full result, 868 B</td>
<td>full result, 965 B</td>
<td>nothing sent</td>
</tr>
<tr>
<td><code>WHERE store_id = $1</code> (control)</td>
<td>change, 387 B</td>
<td>change, 388 B</td>
<td>nothing sent</td>
</tr>
</tbody></table>
<p>Two useful details are in there. A write to a table a query does not read sends that subscriber nothing at all. And sorting alone is enough to lose incremental updates: <code>ORDER BY id</code> with no <code>LIMIT</code> re-sent all 300 rows on every write. Sort in the client instead.</p>
<p>The cost of a full result grows with the result. The join, at 1 write per second for 60 seconds, at four result sizes, with a 5,000-row plain query as the control:</p>
<table>
<thead>
<tr>
<th>Live query</th>
<th>Rows in result</th>
<th>Bytes per write</th>
<th>p50</th>
<th>p90</th>
</tr>
</thead>
<tbody><tr>
<td>join</td>
<td>10</td>
<td>1,980</td>
<td>48.1 ms</td>
<td>52.1 ms</td>
</tr>
<tr>
<td>join</td>
<td>100</td>
<td>13,953</td>
<td>52.9 ms</td>
<td>62.8 ms</td>
</tr>
<tr>
<td>join</td>
<td>1,000</td>
<td>134,225</td>
<td>99.2 ms</td>
<td>111.9 ms</td>
</tr>
<tr>
<td>join</td>
<td>5,000</td>
<td>668,496</td>
<td>225.1 ms</td>
<td>273.4 ms</td>
</tr>
<tr>
<td>plain</td>
<td>5,000</td>
<td>572</td>
<td>42.6 ms</td>
<td>45.6 ms</td>
</tr>
</tbody></table>
<p>That is linear: about 134 bytes per row, re-sent on every write that changes anything in the result. Neon's docs say egress from Realtime is billed as usual (Free includes 5 GB per project per month; Launch and Scale include 500 GB, then $0.10 per GB). One browser tab watching that 1,000-row join while the table takes a steady 10 writes per second receives about 4.8 GB (decimal) an hour. Twenty tabs for an 8-hour day, 22 working days a month, is about 17 TB, or roughly $1,650 a month past the Launch allowance, if nothing else in the project uses that allowance. The plain query in the same setting comes to about 73 GB.</p>
<p>One thing we cannot square with that. When we read Neon's consumption API at 19:26 UTC, it showed 16.2 MB of public network transfer for the project for 15:00 to 18:00 UTC, three hours in which our bench recorders alone received at least 862 MB of Realtime messages. We did not find out why. Realtime is free during the beta, but the docs say the waiver does not cover network transfer, so plan on paying for the bytes above rather than going by the meter's current reading.</p>
<h2>At 100 writes per second, the client falls behind first</h2><p>A full result has to be decoded and applied by the client. For the 200-row join at 100 writes per second, that is 100 full results a second, about 2.7 MB/s.</p>
<p><strong>The 200-row join at 100 writes/s, p50 per 5 seconds</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>0s</th>
<th>5s</th>
<th>10s</th>
<th>15s</th>
<th>20s</th>
<th>25s</th>
<th>30s</th>
<th>35s</th>
<th>40s</th>
<th>45s</th>
<th>50s</th>
<th>55s</th>
</tr>
</thead>
<tbody><tr>
<td>SDK client, round A</td>
<td>51.6ms</td>
<td>50.4ms</td>
<td>51.7ms</td>
<td>59.2ms</td>
<td>60.5ms</td>
<td>60ms</td>
<td>61.1ms</td>
<td>60.8ms</td>
<td>62.3ms</td>
<td>70.7ms</td>
<td>745.8ms</td>
<td>1787.3ms</td>
</tr>
<tr>
<td>SDK client, round B</td>
<td>49.8ms</td>
<td>50.2ms</td>
<td>51.8ms</td>
<td>59.1ms</td>
<td>59.1ms</td>
<td>59.3ms</td>
<td>60ms</td>
<td>53.4ms</td>
<td>61.3ms</td>
<td>61.4ms</td>
<td>64.4ms</td>
<td>64.1ms</td>
</tr>
<tr>
<td>raw recorder, run D1</td>
<td>40.8ms</td>
<td>40.2ms</td>
<td>40.2ms</td>
<td>48.9ms</td>
<td>47.8ms</td>
<td>48.4ms</td>
<td>50.7ms</td>
<td>48.8ms</td>
<td>48.4ms</td>
<td>48.7ms</td>
<td>48.9ms</td>
<td>48.7ms</td>
</tr>
<tr>
<td>raw recorder, round B</td>
<td>45.6ms</td>
<td>45.3ms</td>
<td>45.8ms</td>
<td>53.4ms</td>
<td>53.4ms</td>
<td>53.2ms</td>
<td>52.2ms</td>
<td>53.7ms</td>
<td>48.7ms</td>
<td>52.5ms</td>
<td>52ms</td>
<td>53.3ms</td>
</tr>
</tbody></table>
<p><em>Every write re-sends the whole 200-row result (about 27 KB). The raw recorder speaks the wire protocol without decoding rows; the SDK client decodes and applies each full result. Same project, same writer.</em></p>
<p>To find out which side was falling behind, we wrote a second recorder that speaks the wire protocol directly and never decodes the rows. With it, at the same rate, the join held a p50 of 44 to 49 ms and a p99 of 73 to 114 ms over two complete runs of 6,000 writes, with no backlog. A third raw run started badly: 220 writes in its first five seconds were delayed by up to 957 ms, with no recorded client stall overlapping those delays, and a session loss then cut it off 16 seconds in, because the raw recorder does not reconnect. The two complete runs kept up; the third had update-path delays we cannot explain before the server ended its session.</p>
<p>The SDK client is the slow part, on our machine. In a separate benchmark at 20 updates a second, with a materialized subscription, it used 2.1 ms of CPU per update for the plain query, 2.0 ms for the aggregate and 9.4 ms for each 200-row join result. If that cost held at 100 a second, it would take about 94% of one core. In one of our two SDK runs at that rate it fell behind and never caught up (p50 1.8 s in the last five seconds); in the other it held on, with stalls of up to 470 ms. We did not measure other hardware. Either way, the work grows with result size times write rate, on every subscriber.</p>
<p>All four runs in the chart, raw and SDK alike, stepped up by 7 to 9 ms about 15 seconds in and stayed there. We did not find out why.</p>
<h2>What a reconnect delivers</h2><p>Our July layer had no answer for a dropped connection: whatever <code>NOTIFY</code> fired while a browser was away was gone. Realtime's answer is a fresh copy of the result. We watched two live queries (the plain one and the join) while a writer added an order every 500 ms, then cut the network twice: every socket destroyed, and new connections sent to an address that never answers. The second cut lasted 75 seconds, longer than a sealed query lives, and renewal failed during it, as it would for a browser that is offline:</p>
<p><strong>cutting the network for 5 s, then for 75 s</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># two live queries, plain and join, on one store; a writer adds an order every 500 ms</span>
$ node scripts/reconnect.ts
   0.57s  plain state live 
   0.92s  <span class="hljs-built_in">join</span>  state live 
   0.92s  setup both live: plain 100 rows, <span class="hljs-built_in">join</span> 100 rows 
   8.92s  net   short: <span class="hljs-built_in">cut</span> <span class="hljs-keyword">for</span> 5 s 
   8.94s  <span class="hljs-built_in">join</span>  state connecting 
   8.94s  plain state connecting 
  13.93s  net   short: restored; 10 orders were written during the <span class="hljs-built_in">cut</span> 
  16.72s  plain state live 
  16.77s  <span class="hljs-built_in">join</span>  state live 
  20.60s  plain short +2 s: 134 rows, database 134, missing 0, extra 0 {<span class="hljs-string">"status"</span>:<span class="hljs-string">"live"</span>}
  20.65s  <span class="hljs-built_in">join</span>  short +2 s: 134 rows, database 134, missing 0, extra 0 {<span class="hljs-string">"status"</span>:<span class="hljs-string">"live"</span>}
  28.65s  net   long: <span class="hljs-built_in">cut</span> <span class="hljs-keyword">for</span> 75 s 
  28.65s  <span class="hljs-built_in">join</span>  state connecting 
  28.65s  plain state connecting 
  49.85s  <span class="hljs-built_in">join</span>  renew failed: offline 
<span class="hljs-comment"># 107 more renewal failures follow, one a second for each subscription, until the network returns</span>
 103.65s  net   long: restored; 140 orders were written during the <span class="hljs-built_in">cut</span> 
 136.93s  plain renewed 
 137.02s  plain state live 
 151.87s  <span class="hljs-built_in">join</span>  renewed 
 151.97s  <span class="hljs-built_in">join</span>  state live 
 153.92s  plain renewed 
 153.94s  <span class="hljs-built_in">join</span>  renewed 
 156.06s  plain long +2 s: 382 rows, database 382, missing 0, extra 0 {<span class="hljs-string">"status"</span>:<span class="hljs-string">"live"</span>}
 156.14s  <span class="hljs-built_in">join</span>  long +2 s: 382 rows, database 382, missing 0, extra 0 {<span class="hljs-string">"status"</span>:<span class="hljs-string">"live"</span>}
 159.73s  plain final: 382 rows, database 382, missing 0, extra 0 {<span class="hljs-string">"status"</span>:<span class="hljs-string">"live"</span>}
 159.78s  <span class="hljs-built_in">join</span>  final: 382 rows, database 382, missing 0, extra 0 {<span class="hljs-string">"status"</span>:<span class="hljs-string">"live"</span>}
 159.79s  plain state closed 
 159.79s  <span class="hljs-built_in">join</span>  state closed 
</code></pre><p>After both cuts, the first thing to arrive was a complete result (130 rows, then 351 and 378), not a replay of the changes that were missed, and both live results held exactly the same rows as the database. That is what you want for a screen. It is not a durable event log: if your app needs to see every intermediate state, Realtime will not give it to you after a gap, and neither did <code>NOTIFY</code>.</p>
<p>The wait after the long cut is the part to plan for. The SDK reconnects with jittered exponential backoff (a 1-second base, capped at 60 seconds), and version 0.1.0 does not listen for the browser's <code>online</code> event (we checked the source). So after a long outage the network can be back for half a minute or more before the board is. Show the subscription's <code>status</code> in the UI rather than assuming it is live. The short cut cost 2.8 seconds.</p>
<h2>Sealed queries: what we could and could not forge</h2><p>A sealed query is an AES-256-GCM encrypted capability (a compact JWE) that holds the SQL, its parameters and the database name, and expires after 60 seconds. The browser holds it but cannot read it, and the Live SQL API checks it without calling your server. We tried to get rows we were not sealed for. The attacker holds a valid sealed query for their own store, as any signed-in user of the board would, and wants another store's orders. The script speaks the wire protocol directly, because an attacker is not bound to use the SDK. Each attempt has one expected outcome, and anything else (another error or a timeout) counts as a failure, as does a single row from the victim's store, except in the documented stolen-query case. The control, the stolen-query case and the renewal attack each include a write during the attempt, so incremental changes are checked for leaks as well as first results:</p>
<table>
<thead>
<tr>
<th>Attempt</th>
<th>Result</th>
</tr>
</thead>
<tbody><tr>
<td>Control: the attacker's own sealed query, then an order in their store</td>
<td>delivered: the attacker's 5 rows, then the new order as a change</td>
</tr>
<tr>
<td>One character changed in the ciphertext, the tag or the IV</td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>Header and IV from one capability, ciphertext and tag from another</td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>Ciphertext truncated</td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>The victim's query sealed with a guessed secret in the right format</td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>Header rewritten to <code>alg: none</code></td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>A <code>sql</code> field added to the subscribe message</td>
<td><code>connection_error: invalid_payload</code></td>
</tr>
<tr>
<td>Renewing a live subscription with a forged capability for the victim's query, then an order in the victim's store</td>
<td><code>renewal_failed</code>, and nothing from the victim's store</td>
</tr>
<tr>
<td>A query sealed with a child branch's secret, sent to the parent branch</td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>A query sealed with the parent's secret, sent to the child branch</td>
<td><code>authorization_failed</code></td>
</tr>
<tr>
<td>The attacker's own sealed query, 65 seconds after sealing</td>
<td><code>authorization_expired</code></td>
</tr>
<tr>
<td>The victim's real sealed query, copied, then an order in the victim's store</td>
<td><strong>delivered: all 30 victim rows, then the new order</strong></td>
</tr>
</tbody></table>
<p>All 14 outcomes were as expected, in a first run and again in this stricter one after review. No tampered, forged or cross-branch attempt disclosed a victim row or change, and the expired capability was refused. A child branch got its own Realtime URL and its own secret, so a preview branch's capabilities do not open production. To check that the harness could fail, a self-test swaps one attack's payload for the victim's real sealed query; the checks then report that attack as a leak.</p>
<p>The last row is the documented catch: <strong>a sealed query is a bearer token.</strong> Anyone who holds one can subscribe until it expires. Expiry closes that window within 60 seconds of sealing; it does not take back rows already received. Keep sealed queries out of URLs, logs and persistent browser storage, as Neon's docs say.</p>
<h2>Scoping per user without row-level security</h2><p>Realtime has no row-level security yet. Neon's guidance is to authorize in the request handler and put the authorization inside the query: take the user id from a verified token, and join through the membership table so the query itself only matches rows that user may see. We tested that against two weaker designs, through the board's API, with real Neon Auth users. These runs used a version of the API that also looked the user up in <code>store_members</code> before sealing, which is why renewals were refused after a removal; the board in the repo now skips that lookup (see the next two sections) and seals a query whose membership join simply returns nothing for a non-member:</p>
<table>
<thead>
<tr>
<th>Design</th>
<th>After the user lost access</th>
</tr>
</thead>
<tbody><tr>
<td>Membership join inside the live query (the board)</td>
<td>Their result was empty <strong>8 ms</strong> after the removal was acknowledged (one test, checked every 5 ms). 0 of 6 orders added afterwards reached them.</td>
</tr>
<tr>
<td>Membership checked when sealing, but the query only filters by <code>store_id</code></td>
<td><strong>12 of 18</strong> new orders still reached them. The last one was added 56 s after the removal. It stopped when the sealed query expired and the API refused to renew it.</td>
</tr>
<tr>
<td>A member's sealed query copied to a non-member</td>
<td><strong>12 of 18</strong> new orders reached the non-member. The last one was added 56 s after sealing. The API refused the non-member's renewal.</td>
</tr>
</tbody></table>
<p>So the pattern works, and the join is what makes it work. A check at seal time alone leaves a window of up to one sealed-query lifetime, and only because the renewal goes back through your API, which refuses it. Put the access rule in the query, with the user id sealed in from a verified token, and a removal reaches the live result with the next update, without waiting for the sealed query to expire.</p>
<p>One more consequence of the full-result rule: the obvious ways to write that access check, <code>EXISTS</code> and <code>IN (SELECT ...)</code>, are re-sent whole on every write. The join to <code>store_members</code>, pinned to one row by the <code>WHERE</code> clause, is not.</p>
<h2>The board</h2><p>The live order board in the repo is a Vite and React app with a Hono API, Drizzle for the schema and the queries, and Neon Auth for sign-in. Its first version used the three-table join from the measurements, which re-sends every order on every write. The version in the repo subscribes to three queries instead, each scoped by a join to <code>store_members</code> on the user id from the verified token:</p>
<pre><code class="hljs language-typescript" tabindex="0"><span class="hljs-comment">// Open orders: incremental. The customer name is not joined in here.</span>
<span class="hljs-keyword">export</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">openOrdersQuery</span>(<span class="hljs-params"><span class="hljs-attr">db</span>: <span class="hljs-title class_">Db</span>, <span class="hljs-attr">userId</span>: <span class="hljs-built_in">string</span>, <span class="hljs-attr">storeId</span>: <span class="hljs-built_in">number</span></span>) {
  <span class="hljs-keyword">return</span> db
    .<span class="hljs-title function_">select</span>({
      <span class="hljs-attr">id</span>: orders.<span class="hljs-property">id</span>,
      <span class="hljs-attr">customer_id</span>: orders.<span class="hljs-property">customer_id</span>,
      <span class="hljs-attr">status</span>: orders.<span class="hljs-property">status</span>,
      <span class="hljs-attr">total_cents</span>: orders.<span class="hljs-property">total_cents</span>,
      <span class="hljs-attr">created_at</span>: orders.<span class="hljs-property">created_at</span>,
    })
    .<span class="hljs-title function_">from</span>(orders)
    .<span class="hljs-title function_">innerJoin</span>(storeMembers, <span class="hljs-title function_">eq</span>(storeMembers.<span class="hljs-property">store_id</span>, orders.<span class="hljs-property">store_id</span>))
    .<span class="hljs-title function_">where</span>(
      <span class="hljs-title function_">and</span>(<span class="hljs-title function_">eq</span>(storeMembers.<span class="hljs-property">user_id</span>, userId), <span class="hljs-title function_">eq</span>(orders.<span class="hljs-property">store_id</span>, storeId), <span class="hljs-title function_">ne</span>(orders.<span class="hljs-property">status</span>, <span class="hljs-string">'done'</span>))
    );
}

<span class="hljs-comment">// Customers: incremental, and an order write sends this subscription nothing.</span>
<span class="hljs-keyword">export</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">storeCustomersQuery</span>(<span class="hljs-params"><span class="hljs-attr">db</span>: <span class="hljs-title class_">Db</span>, <span class="hljs-attr">userId</span>: <span class="hljs-built_in">string</span>, <span class="hljs-attr">storeId</span>: <span class="hljs-built_in">number</span></span>) {
  <span class="hljs-keyword">return</span> db
    .<span class="hljs-title function_">select</span>({ <span class="hljs-attr">id</span>: customers.<span class="hljs-property">id</span>, <span class="hljs-attr">name</span>: customers.<span class="hljs-property">name</span> })
    .<span class="hljs-title function_">from</span>(customers)
    .<span class="hljs-title function_">innerJoin</span>(storeMembers, <span class="hljs-title function_">eq</span>(storeMembers.<span class="hljs-property">store_id</span>, customers.<span class="hljs-property">store_id</span>))
    .<span class="hljs-title function_">where</span>(<span class="hljs-title function_">and</span>(<span class="hljs-title function_">eq</span>(storeMembers.<span class="hljs-property">user_id</span>, userId), <span class="hljs-title function_">eq</span>(customers.<span class="hljs-property">store_id</span>, storeId)));
}

<span class="hljs-comment">// The header: an aggregate, re-sent whole on every write, which is fine at four rows.</span>
<span class="hljs-keyword">export</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">statusCountsQuery</span>(<span class="hljs-params"><span class="hljs-attr">db</span>: <span class="hljs-title class_">Db</span>, <span class="hljs-attr">userId</span>: <span class="hljs-built_in">string</span>, <span class="hljs-attr">storeId</span>: <span class="hljs-built_in">number</span></span>) {
  <span class="hljs-comment">/* GROUP BY status */</span>
}
</code></pre><p>The browser joins the first two with a <code>Map</code> from customer id to name, and sorts on its own side, because <code>ORDER BY</code> in the live query would re-send everything. Status changes are optimistic: the card moves at once with React's <code>useOptimistic</code>, the API's <code>UPDATE ... RETURNING pg_current_xact_id()</code> returns the transaction id, and <code>awaitTxId()</code> on the live query holds the optimistic state until that transaction arrives.</p>
<p>The API's seal endpoint verifies the Neon Auth token and seals, and that is all: sealing is local crypto, so a renewal never touches Postgres. It does not look the user up in <code>store_members</code> first. That lookup would only turn a non-member's empty board into a 403, because the join inside each query is what keeps them out, and it would cost a database query on every renewal of every subscription. The next section shows what that does to scale to zero.</p>
<p>A scripted run through the same API and SDK calls the app makes (<code>scripts/board-check.ts</code>) passed every check. A new order appeared on the board 99 ms after the POST was sent, moving it to <code>preparing</code> showed up in 70 ms, and marking it <code>done</code> took it off the board in 54 ms, each measured to the moment <code>awaitTxId</code> resolved. A user from another store got an empty board from the same endpoint, and a 403 when adding an order to it.</p>
<h2>Scale to zero: open subscriptions let the compute sleep</h2><p>Neon's docs say holding a subscription open does not keep the database awake, but starting one can wake it. We checked both, with the suspend timeout shortened to 60 seconds through the Neon API and the compute's state polled every 5 seconds:</p>
<table>
<thead>
<tr>
<th>Step</th>
<th>What happened</th>
</tr>
</thead>
<tbody><tr>
<td>Control: nothing open</td>
<td>The compute suspended 65 s after we started watching.</td>
</tr>
<tr>
<td>A live query started while the compute was suspended</td>
<td>First result in 0.98 s. The compute was active again 0.3 s later.</td>
</tr>
<tr>
<td>That subscription held open, renewed every 50 s, no writes</td>
<td>The compute suspended 75 s later, and was still suspended 60 s after that. The subscription stayed <code>live</code> the whole time.</td>
</tr>
<tr>
<td>A write while suspended, subscription still open</td>
<td>Connecting to the waking compute took 853 ms and the insert 247 ms (1.14 s in all, closing the connection included). The open subscription received the change 1.12 s after the write was issued, with no reconnect.</td>
</tr>
<tr>
<td>A second live query started from a suspended compute</td>
<td>First result in 1.84 s, and it woke the compute.</td>
</tr>
</tbody></table>
<p>So an open subscription costs no compute while nothing writes, and it survives the compute going to sleep under it. The cost is on the first write or the first new subscriber after a quiet spell: about a second, the same cold start any query pays. We could not see the replica that Neon says does the incremental work, so we cannot say whether it suspends too; Neon does not charge for it during the beta.</p>
<p><strong>The board itself is another matter, because its renewals go through your API.</strong> We ran alice's three board queries with the browser's renewal timing (10 seconds before each 60-second expiry), sealed and renewed through the board's API, with the suspend timeout at 60 seconds:</p>
<table>
<thead>
<tr>
<th>Board API</th>
<th>Compute while the board sat open with no writes</th>
</tr>
</thead>
<tbody><tr>
<td>Looks the user up in <code>store_members</code> on every seal</td>
<td>Never suspended in 300 s. 18 renewals, each one a database query.</td>
</tr>
<tr>
<td>Seals without touching Postgres (the version in the repo)</td>
<td>Suspended 75 s after the board opened.</td>
</tr>
</tbody></table>
<p>The membership lookup looks harmless, and it cost scale to zero entirely: with it, each open tab made three database queries roughly every 50 seconds while its renewals succeeded, one every 17 seconds on average, and that kept the compute awake for the whole 300 seconds we watched. Keep the renew path free of database calls. Signing in, and refreshing the Neon Auth token (it lasts about 15 minutes), go through Neon Auth, which keeps its tables in the same database; we did not measure whether a token refresh wakes the compute.</p>
<h2>Writes on the primary, with and without 100 live queries</h2><p>Neon's FAQ says Realtime has "zero impact on the write-latency, write-throughput or write-compute of your primary database". We ran a writer at 20 writes per second for the whole test and switched 100 live queries on and off in alternating 60-second blocks: off, on, off, on, off, on. Each query belonged to a different user and every write touched all 100 of them: nominally 2,000 updates a second (20 writes times 100 queries) to the subscriber process. Writes during each switch and for 20 seconds after it were left out.</p>
<table>
<thead>
<tr>
<th></th>
<th>100 live queries off</th>
<th>100 live queries on</th>
</tr>
</thead>
<tbody><tr>
<td>Write execution time on the primary (<code>pg_stat_statements</code>, mean per block)</td>
<td>0.157 to 0.164 ms</td>
<td>0.150 to 0.166 ms</td>
</tr>
<tr>
<td>Write latency on the client, p50 per block</td>
<td>34.3 to 39.4 ms</td>
<td>32.3 to 33.8 ms</td>
</tr>
<tr>
<td>Write latency on the client, p99 (all blocks)</td>
<td>47.6 ms</td>
<td>42.5 ms</td>
</tr>
<tr>
<td>Writes</td>
<td>3,603</td>
<td>3,603</td>
</tr>
</tbody></table>
<p>On the server, the write's own execution time did not move. On the client, the pooled median write was about 2.5 ms faster with the live queries on, and each on block had a lower median than the off block before it, which is the opposite of a slowdown. We think that is our client and not Neon: it is on Wi-Fi with power saving on, and the extra incoming traffic keeps the radio awake. We did not test that explanation. A first version of this test let the CPU clock follow the load, tagged some switching time to the wrong block, and lacked an index that keeps the write's <code>DELETE</code> cheap (its execution time jumped from 0.3 to 3.2 ms partway through). It showed the same direction and is kept in the repo as superseded. Both versions ran with Realtime enabled throughout, so neither measures what turning Realtime on (<code>wal_level = logical</code>) costs.</p>
<p>So under this workload we saw no slowdown of writes on the primary with 100 subscriptions open, which matches Neon's claim. <code>pg_stat_statements</code> does not include commit time, though, and our client's own noise is larger than a millisecond, so "no slowdown we could measure" is as far as this goes.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>The replica's lag on its own.</strong> Neon says the incremental work runs on a replica. We could not see one in the project, the Neon API or the primary's replication views, so every number here is end to end.</li>
<li><strong>Why the server ends sessions.</strong> Twenty <code>backend_session_lost</code> errors across both recordings, most of them at the same second past the minute, as if on a 60-second cycle. The SDK handled every one, but we cannot see the cause.</li>
<li><strong>Why the four join runs at 100 writes per second in the chart got 7 to 9 ms slower about 15 seconds in,</strong> the raw recorder's included.</li>
<li><strong>What the SDK costs on other hardware.</strong> We measured it on one client, a Raspberry Pi on Wi-Fi with power saving on, and nowhere else. The Wi-Fi also shaves a few milliseconds off everything when traffic is heavy, which is the likely reason most numbers improved at higher write rates.</li>
<li><strong>More than 100 subscribers, or more than one client.</strong> Neon says thundering-herd protection is built in; we did not test a reconnect storm.</li>
<li><strong>The bill.</strong> Realtime is free during the beta and its network transfer is documented as billed, but the project's transfer meter showed far less than our Realtime traffic, so we cannot tell you what this test would have cost.</li>
<li><strong>Whether any of this survives the beta.</strong> Neon calls the protocol unstable and the performance work unfinished. The full-result behaviour in particular may change; rerun <code>scripts/explore-incremental.ts</code> before you design around it.</li>
</ul>
<h2>Wrapping up</h2><p>For a live order board, Neon Realtime replaces everything our July layer did and covers what it could not. There is no function holding connections, no <code>LISTEN</code> session, and no <code>NOTIFY</code> call to forget on a write path. A reconnect brings back the right result instead of losing events, and the board shows a query result rather than an event the app has to interpret. A plain live query arrived as fast as the write's own acknowledgement, and none of the 48,600 writes went missing.</p>
<p>The part to design for is the full re-send. Write the live queries as single-table filters, or join only to a membership table pinned to one row. Do the other joins and all sorting in the client, and keep live aggregates to a handful of groups. Check each query with the CLI's <code>--changes</code> flag: a <code>Rows</code> line on every write means the whole result is coming back each time. Put the access rule inside the query, and budget for the bytes a full result costs on every write, times every subscriber.</p>
<p>The <a href="https://github.com/The-DevOps-Daily/neon-realtime-measured" rel="noopener noreferrer">repo</a> has the board, every script, and the raw output of every run, including the ones we superseded.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[SOC 2 Evidence From GitHub and Terraform, and the Part That Still Needs a Person]]></title>
      <link>https://devops-daily.com/posts/soc2-evidence-from-github-and-terraform</link>
      <description><![CDATA[We pointed a small evidence collector at a repository whose main branch had a strict ruleset. The ruleset passed and three change-management controls failed. Here is which SOC 2 evidence you can pull from GitHub and Terraform, which still needs a person, and ten ways an evidence script can pass a control without evidence.]]></description>
      <pubDate>Mon, 19 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/soc2-evidence-from-github-and-terraform</guid>
      <category><![CDATA[Security]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Security]]></category><category><![CDATA[SOC 2]]></category><category><![CDATA[Compliance]]></category><category><![CDATA[GitHub]]></category><category><![CDATA[Terraform]]></category><category><![CDATA[DevSecOps]]></category><category><![CDATA[Audit]]></category>
      <content:encoded><![CDATA[<p>We pointed a small evidence collector at a GitHub repository whose main branch had a strict ruleset: every change through a pull request, one approval, and a required <code>test</code> check. The ruleset control passed. Three of the change-management controls under it failed: none of the seven merged pull requests had an independent approval, one was merged five seconds before its check finished, and one commit reached main without a pull request at all. All three were real, and all three would have looked fine in a screenshot of the settings page.</p>
<p>That gap is the whole job when a customer asks for SOC 2. An auditor does not only want to know what you configured. For a Type II report they want to see what happened under that configuration across the review period, usually by sampling. This post shows which of that evidence you can pull from the GitHub API and your Terraform code, which still needs a person and a signature, and the ten ways the first version of our collector could pass a control without evidence.</p>
<h2>TL;DR</h2><ul>
<li>Settings are not evidence of operation. Our main branch required approvals and a passing check, and the evidence still showed 0 of 7 pull requests independently approved and 1 merged before its check finished, because admins could bypass the rule.</li>
<li>"The checks are green" is the wrong question. Pull request #3's check passed 5 seconds after the merge. It is green today; it did not gate the merge. Compare the check's completion time with the merge time.</li>
<li>An evidence script must fail closed. Two reviews of our first version found ten ways it could pass a control on missing or partial data, from approvals of an older commit to a branch name that could change the API query. Each now reports <code>fail</code> or <code>incomplete</code>, with a test.</li>
<li>Terraform scanning is evidence of a configuration standard, but a pass is only as good as the rule list. Trivy flagged the public database flag, and its security-group rule passed a Postgres port open to the whole internet, because that rule only covers SSH and RDP.</li>
<li>Access reviews, policy sign-offs, training, risk assessment, vendor reviews and incident-response tests still need a person. Automation can gather the list; it cannot make the judgement.</li>
<li>One run took 23 seconds and 35 API calls for one repository and 90 days, and produced 15 timestamped evidence files with a SHA-256 manifest.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A GitHub repository you are admin on, and a token in <code>GITHUB_TOKEN</code> or a logged-in <code>gh</code> CLI</li>
<li>Python 3.10 or later (the collector uses only the standard library)</li>
<li><a href="https://trivy.dev" rel="noopener noreferrer">Trivy</a> for the Terraform scan</li>
<li>A rough idea of what SOC 2 is: an AICPA attestation report on your controls, mapped to the Trust Services Criteria</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/soc2-evidence-collector" rel="noopener noreferrer">The-DevOps-Daily/soc2-evidence-collector on GitHub</a></p>
<h2>What an auditor actually asks for</h2><p>SOC 2 reports on the <a href="https://www.aicpa-cima.com/resources/download/2017-trust-services-criteria-with-revised-points-of-focus-2022" rel="noopener noreferrer">Trust Services Criteria</a> (free download with an AICPA account), a set of criteria with "points of focus" that describe what a control usually covers. You choose the controls; the auditor tests them. A Type I report says the controls were designed and in place on one day. A Type II report says they operated over a period, often 3 to 12 months, and that is where the evidence work goes: the auditor picks a sample of changes, alerts and access changes from the period and asks you to show each one.</p>
<p>Three criteria cover most of what a code repository can prove. In the AICPA's words:</p>
<blockquote>
<p><strong>CC8.1</strong> The entity authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes to infrastructure, data, software, and procedures to meet its objectives.</p>
</blockquote>
<blockquote>
<p><strong>CC7.1</strong> To meet its objectives, the entity uses detection and monitoring procedures to identify (1) changes to configurations that result in the introduction of new vulnerabilities, and (2) susceptibilities to newly discovered vulnerabilities.</p>
</blockquote>
<blockquote>
<p><strong>CC6.1</strong> The entity implements logical access security software, infrastructure, and architectures over protected information assets to protect them from security events to meet the entity's objectives.</p>
</blockquote>
<p>For a change, "approves" and "tests" are the parts people get caught on. If your change-management control says that someone other than the author approves each change and that it is tested before it reaches production, the auditor will ask you to show both for each sampled change. For vulnerabilities, they want to see that you detect them and fix the serious ones within the time your own policy promises.</p>
<p>The usual way to produce this is screenshots: the branch protection page, the pull request page for each sampled change, the Dependabot tab, the security settings. A screenshot shows one moment, and nothing in it says where the values came from. An API response saved with the time it was collected is easier to check: the auditor can see the source of each value, and you can run the same query again. A hash manifest lets you show that a file did not change after collection, as long as the manifest is stored where nobody can edit it. And a script can collect the evidence for every change in the period instead of a sample.</p>
<h2>The collector</h2><p><code>collect.py</code> reads one repository through the GitHub API and evaluates eight controls:</p>
<table>
<thead>
<tr>
<th>Criterion</th>
<th>Control</th>
<th>Where the evidence comes from</th>
</tr>
</thead>
<tbody><tr>
<td>CC8.1</td>
<td>Changes to the default branch go through a pull request</td>
<td>branch rules and classic branch protection</td>
</tr>
<tr>
<td>CC8.1</td>
<td>Every merged change passed the required checks before the merge</td>
<td>the last run of each required check on the pull request's head commit that started before the merge</td>
</tr>
<tr>
<td>CC8.1</td>
<td>Every merged change was approved by someone other than its author</td>
<td>each reviewer's last decision before the merge, for the merged commit</td>
</tr>
<tr>
<td>CC8.1</td>
<td>Nothing reached the default branch without a pull request</td>
<td>the merged pull request behind each commit, and the branch's activity log</td>
</tr>
<tr>
<td>CC7.1</td>
<td>Dependency vulnerability alerts are on</td>
<td>the Dependabot setting</td>
</tr>
<tr>
<td>CC7.1</td>
<td>No critical or high alert is open past the SLA</td>
<td>open Dependabot alerts and their age</td>
</tr>
<tr>
<td>CC6.1</td>
<td>Secret scanning and push protection are on</td>
<td>the repository's security settings</td>
</tr>
<tr>
<td>CC6.3</td>
<td>Repository access follows least privilege</td>
<td>collaborator roles, judged by a person</td>
</tr>
</tbody></table>
<p>It saves the fields each control uses from each API response, with the collection time, in a timestamped folder with a SHA-256 manifest, and marks each control <code>pass</code>, <code>fail</code>, <code>needs a person</code> or <code>incomplete</code>:</p>
<ol>
<li><strong>GitHub API</strong> rules, PRs, checks, alerts</li>
<li><strong>collect.py</strong> 8 controls</li>
<li><strong>raw/*.json</strong> timestamped, hashed</li>
<li><strong>summary.json</strong> pass, fail, incomplete</li>
<li><strong>A person</strong> judgement, sign-off</li>
</ol>
<p>Three of the controls look back over the window: the merged pull requests with their reviews and checks, the commits on main, and the branch's activity log. The other five read the settings and the open alerts as they are on the day you run it. One run shows the history of changes. For the history of settings across a Type II period, run it on a schedule and keep every folder.</p>
<h3>The test subject</h3><p>We made the collector's own repository its test subject, with a history built on purpose. Its main branch has a ruleset that requires a pull request, one approval and the <code>test</code> check, with a bypass for organization admins. Then:</p>
<ul>
<li>The first commit went straight to main, before the ruleset existed.</li>
<li>Every pull request was merged by an admin through the bypass. Nobody else could approve: it is a one-person setup, which is exactly the situation many small teams are in when the first enterprise customer asks for SOC 2.</li>
<li>Pull request #3 was merged while its <code>test</code> check was still running.</li>
<li><code>examples/app/requirements.txt</code> pins PyYAML 5.3 and Jinja2 2.10, which have published advisories, so Dependabot has alerts to report.</li>
</ul>
<h3>The run</h3><p><strong>collect.py</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ python3 collect.py The-DevOps-Daily/soc2-evidence-collector --days 90
The-DevOps-Daily/soc2-evidence-collector, last 90 days: 15 evidence files, 35 API calls, 22.7 s
  CC8.1  pass            Changes to the default branch go through a pull request
         rule on main: pull_request, 1 approval(s) required
  CC8.1  fail            Every merged change passed the required checks before the merge
         required: <span class="hljs-built_in">test</span>; 6 of 7 merged PRs passed every required check before the merge (#3 not finished)
  CC8.1  fail            Every merged change was approved by someone other than its author
         0 of 7 merged PRs had an independent approval of the merged commit
  CC8.1  fail            Nothing reached the default branch without a pull request
         commits not linked to a merged PR: 1 of 8; branch updates that were not PR merges: 1 (branch_creation)
  CC7.1  pass            Dependency vulnerability alerts are on
         Dependabot alerts on
  CC7.1  pass            No critical or high alert is open past 30 days
         8 open alerts, 0 critical or high open longer than 30 days
  CC6.1  pass            Secret scanning and push protection are on
         secret scanning enabled, push protection enabled
  CC6.3  needs a person  Repository access follows least privilege
         roles: {<span class="hljs-string">'admin'</span>: 4}, outside collaborators: 0; the list is evidence, the judgement is not
evidence: evidence/The-DevOps-Daily__soc2-evidence-collector/20261019T131928Z
</code></pre><p>It found every gap we put in, and it exits non-zero, so it can gate a pipeline or a weekly job.</p>
<h2>What the failures mean</h2><h3>The rule was on, and the evidence still failed</h3><p>The first control passed: a pull request with one approval is required on main. The approval control under it failed for all seven merged pull requests. An admin bypass is not a bug in the ruleset. Many teams keep one for emergencies, and an auditor's sample may well include a change that used it. If your system description says all changes are approved by a second person and your evidence shows admin merges without approval, expect the auditor to report them as exceptions.</p>
<p>For a one-person or two-person team, the honest fix is not to pretend. Common approaches are a reviewer from outside the team for production repositories, or a documented control that someone reviews merged changes after the fact within a fixed time. Which of these your auditor accepts is a conversation to have before the review period starts, not after.</p>
<h3>Green now is not the same as tested before the merge</h3><p>Here is the evidence file for pull request #3 as the collector saved it, without the author's login and with the indentation widened for reading:</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"collected_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2026-10-19T13:19:28.577807+00:00"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"source"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"GitHub API, The-DevOps-Daily/soc2-evidence-collector"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"data"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"pull"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span>
      <span class="hljs-attr">"number"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">3</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"title"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Examples to scan, and the IaC scan script"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"base"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"main"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"merged_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2026-10-19T12:50:03Z"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"html_url"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://github.com/The-DevOps-Daily/soc2-evidence-collector/pull/3"</span>
    <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"head_sha"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"d4b17526cfcaf516833023b9ec402f45a70a10ec"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"reviews"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"check_runs"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span>
      <span class="hljs-punctuation">{</span>
        <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"test"</span><span class="hljs-punctuation">,</span>
        <span class="hljs-attr">"app_id"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">15368</span><span class="hljs-punctuation">,</span>
        <span class="hljs-attr">"head_sha"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"d4b17526cfcaf516833023b9ec402f45a70a10ec"</span><span class="hljs-punctuation">,</span>
        <span class="hljs-attr">"status"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"completed"</span><span class="hljs-punctuation">,</span>
        <span class="hljs-attr">"conclusion"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"success"</span><span class="hljs-punctuation">,</span>
        <span class="hljs-attr">"started_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2026-10-19T12:50:01Z"</span><span class="hljs-punctuation">,</span>
        <span class="hljs-attr">"completed_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2026-10-19T12:50:08Z"</span>
      <span class="hljs-punctuation">}</span>
    <span class="hljs-punctuation">]</span>
  <span class="hljs-punctuation">}</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>The check started at 12:50:01. The pull request was merged at 12:50:03. The check succeeded at 12:50:08. Look at the pull request page today and you see a green check, which is what most screenshots record. The collector compares the timestamps, and so should anything you hand to an auditor.</p>
<p>It asks GitHub for every run of the check, not only the latest, and judges each required check by its last run that started before the merge: that run is the check's state at the merge. If it had not finished by then, the check did not gate the merge, even if an earlier run had passed. A re-run started after the merge does not count. Only <code>success</code> passes: GitHub lets a <a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#require-status-checks-before-merging" rel="noopener noreferrer">skipped or neutral required check</a> through, but a skipped check ran no tests. A required check stops a normal merge until it passes, so a merge before completion usually means someone used a bypass. It can also mean the required checks changed after the merge, because the collector compares with today's rules. Either way, it is an exception to explain.</p>
<h3>The commit with no pull request</h3><p>The one commit without a pull request is the repository's first commit, pushed before the ruleset existed. The branch's activity log shows the same event from the other side: one update to main that was not a pull request merge, the creation of the branch. That is a normal, explainable exception, and the evidence lets you explain it with a date.</p>
<p>The two sources catch different things. The commit list only holds commits that are still on main, so a force push that rewrote history leaves nothing in it. The activity log records updates to the branch, including direct pushes and force pushes, so an admin pushing a hotfix straight to main shows up there even when the commit list looks clean. If the log does not reach back to the start of the window, the control reports <code>incomplete</code> instead of <code>pass</code>.</p>
<h3>The alerts that passed</h3><p>Dependabot raised 8 open alerts for the two pinned packages: 2 critical, 1 high and 5 medium. The SLA control passed, correctly, because the alerts were less than an hour old. Our default SLA is 30 days for critical and high; use the number your own vulnerability policy states, since that is what the auditor will hold you to. If those alerts are still open, the same run in five weeks would fail.</p>
<p>This control reads the alerts that are open today. A critical alert that stayed open for two months last spring and was fixed in the summer does not appear in it. To cover a period, keep each run's folder, or extend the collector to fixed alerts and compare their <code>fixed_at</code> with <code>created_at</code>.</p>
<h2>Ten ways our first version could pass without evidence</h2><p>Two reviews of the first version of the collector, one of the code and one of the post against the recorded output, found ten ways it could pass a control, or count something as evidence, when the data did not support it. For a tool whose output goes to an auditor, each one is a real defect: a control must never pass because data was missing.</p>
<table>
<thead>
<tr>
<th>What went wrong</th>
<th>What the collector does now</th>
</tr>
</thead>
<tbody><tr>
<td>An approval of an older commit counted</td>
<td>Only an approval whose <code>commit_id</code> is the merged head commit counts; the rest are reported as stale</td>
</tr>
<tr>
<td>A withdrawn approval counted</td>
<td>Only each reviewer's last decision before the merge counts, so a later change request or a dismissal replaces an approval</td>
</tr>
<tr>
<td>Any passing or skipped check counted as the required one</td>
<td>Each required check is matched by name, and by app when the rule names one, and only <code>success</code> passes</td>
</tr>
<tr>
<td>An earlier pass could hide a later failure, or a re-run still going at the merge</td>
<td>Each required check is judged by its last run that started before the merge, which must have succeeded before the merge</td>
</tr>
<tr>
<td>Unreadable alerts looked like zero alerts</td>
<td>A 403 or 404 makes the SLA control <code>incomplete</code>, with the HTTP status</td>
</tr>
<tr>
<td>Lists were cut off silently</td>
<td>A list cut at its limit makes its control <code>incomplete</code>, and an error on a later page stops the run</td>
</tr>
<tr>
<td>Any pull request link counted</td>
<td>The pull request must be merged into the default branch and be one of the merged pull requests evaluated</td>
</tr>
<tr>
<td>A force push left no trace</td>
<td>The branch's activity log is read alongside the commit list</td>
</tr>
<tr>
<td>The SLA was rounded down to whole days, and alerts without a severity were skipped</td>
<td>The age is compared as a time, and an unknown severity makes the control <code>incomplete</code></td>
</tr>
<tr>
<td>Branch names went into URLs unencoded, and answers were not checked</td>
<td>Every value is encoded, and a pull request, activity entry or check run for another branch or commit makes its list <code>incomplete</code></td>
</tr>
</tbody></table>
<p>The last one is easy to miss. Git allows <code>&amp;</code> and <code>#</code> in branch names, so a default branch with a crafted name could change the query string, and the collector would judge GitHub's answer to a different question.</p>
<p>None of these changed a result for our demo repository: it has no approvals at all, a readable list of 8 alerts and eight commits. They would change results for a real repository, where a stale approval, a re-run check or a missing permission is common. Each fix has a test in the repository.</p>
<p>Running the collector also found one more problem: the Dependabot alerts API rejects page numbers (<code>Pagination using the page parameter is not supported</code>) and pages with cursors in the <code>Link</code> header instead. The collector now follows the <code>Link</code> header for every list.</p>
<h2>Evidence from Terraform</h2><p>CC7.1's points of focus start with "Uses Defined Configuration Standards" and "Monitors Infrastructure and Software" for noncompliance with them. For infrastructure as code, a misconfiguration scan in CI is evidence of both: the rule set is your standard, and the scan result on each change is the monitoring.</p>
<p>The repository's <code>examples/terraform</code> holds a small AWS stack: a customer-data bucket with a KMS key, versioning and a public access block; an exports bucket added later without any of those; and a Postgres instance, encrypted and backed up, with a security group open to <code>0.0.0.0/0</code> on port 5432 and <code>publicly_accessible = true</code>. <code>scan-iac.sh</code> runs Trivy and keeps the report, a summary and a manifest:</p>
<p><strong>scan-iac.sh</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./scan-iac.sh examples/terraform
92 checks, 12 failed, 4.6 s
  HIGH     AWS-0086  aws_s3_bucket.exports        S3 Access block should block public ACL
  HIGH     AWS-0087  aws_s3_bucket.exports        S3 Access block should block public policy
  HIGH     AWS-0091  aws_s3_bucket.exports        S3 Access Block should Ignore Public ACL
  HIGH     AWS-0093  aws_s3_bucket.exports        S3 Access block should restrict public bucket to <span class="hljs-built_in">limit</span> access
  HIGH     AWS-0132  aws_s3_bucket.exports        S3 encryption should use Customer Managed Keys
  HIGH     AWS-0180  aws_db_instance.app          RDS Publicly Accessible
  MEDIUM   AWS-0090  aws_s3_bucket.exports        S3 Data should be versioned
  MEDIUM   AWS-0176  aws_db_instance.app          RDS IAM Database Authentication Disabled
  LOW      AWS-0089  aws_s3_bucket.customer_data  S3 Bucket Logging
  LOW      AWS-0089  aws_s3_bucket.exports        S3 Bucket Logging
  LOW      AWS-0094  aws_s3_bucket.exports        S3 buckets should each define an aws_s3_bucket_public_access_block
  LOW      AWS-0133  aws_db_instance.app          Enable Performance Insights to detect potential problems
evidence: evidence/iac/20261019T130914Z
</code></pre><p>The exports bucket and the public database were both caught. The open security group was not. Trivy's ingress rule AWS-0107, titled "Security groups should not allow unrestricted ingress to SSH or RDP from any IP address", passed: port 5432 is neither SSH nor RDP. The database was flagged only because <code>publicly_accessible</code> was also set. We removed that line, kept the security group and scanned again:</p>
<p><strong>scan-iac.sh, without publicly_accessible</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ <span class="hljs-built_in">cp</span> -r examples/terraform /tmp/sg-only &amp;&amp; sed -i <span class="hljs-string">'/publicly_accessible/d'</span> /tmp/sg-only/main.tf
$ ./scan-iac.sh /tmp/sg-only
92 checks, 11 failed, 4.2 s
  HIGH     AWS-0086  aws_s3_bucket.exports        S3 Access block should block public ACL
  HIGH     AWS-0087  aws_s3_bucket.exports        S3 Access block should block public policy
  HIGH     AWS-0091  aws_s3_bucket.exports        S3 Access Block should Ignore Public ACL
  HIGH     AWS-0093  aws_s3_bucket.exports        S3 Access block should restrict public bucket to <span class="hljs-built_in">limit</span> access
  HIGH     AWS-0132  aws_s3_bucket.exports        S3 encryption should use Customer Managed Keys
  MEDIUM   AWS-0090  aws_s3_bucket.exports        S3 Data should be versioned
  MEDIUM   AWS-0176  aws_db_instance.app          RDS IAM Database Authentication Disabled
  LOW      AWS-0089  aws_s3_bucket.customer_data  S3 Bucket Logging
  LOW      AWS-0089  aws_s3_bucket.exports        S3 Bucket Logging
  LOW      AWS-0094  aws_s3_bucket.exports        S3 buckets should each define an aws_s3_bucket_public_access_block
  LOW      AWS-0133  aws_db_instance.app          Enable Performance Insights to detect potential problems
evidence: evidence/iac/20261019T130925Z
</code></pre><p>The only database findings left are IAM authentication and Performance Insights. Nothing points at a database port open to the world.</p>
<p>That is the general limit of configuration scanning as evidence. A pass means "none of these rules failed", not "this is safe". Your configuration standard is the rule set you run plus the rules you add for what it misses. Trivy and similar tools accept custom policies, and a rule for "no database port open to <code>0.0.0.0/0</code>" is a few lines.</p>
<p>The scan reads code, not the running account. For the period itself, the auditor may ask for evidence that production matches the code: drift detection, or reading the live configuration through the cloud provider's API. That is the same pattern as the GitHub collector, with a different API.</p>
<h2>What still needs a person</h2><p>Some controls produce evidence a script can gather but not judge, and some produce no machine evidence at all. These are the ones that usually remain:</p>
<table>
<thead>
<tr>
<th>Criterion</th>
<th>What the auditor wants</th>
<th>Why a script cannot finish it</th>
</tr>
</thead>
<tbody><tr>
<td>CC6.2, CC6.3</td>
<td>Periodic access reviews: someone confirms each account and role is still needed</td>
<td>The collector lists the 4 admins; whether each should be an admin is a decision, with a name and a date on it</td>
</tr>
<tr>
<td>CC6.2</td>
<td>Access removed when someone leaves</td>
<td>Needs the HR leaving date to compare with when access ended</td>
</tr>
<tr>
<td>CC1.4, CC2.2</td>
<td>Security awareness training completed</td>
<td>The record lives in a training tool or a signed list</td>
</tr>
<tr>
<td>CC5.3</td>
<td>Policies written, approved and acknowledged</td>
<td>A policy document and each employee's acknowledgment</td>
</tr>
<tr>
<td>CC3.2</td>
<td>A documented risk assessment and its periodic review</td>
<td>A document and a dated review</td>
</tr>
<tr>
<td>CC9.2</td>
<td>Vendor and subprocessor reviews</td>
<td>An assessment of each vendor's security, often based on the vendor's own SOC 2 report</td>
</tr>
<tr>
<td>CC7.4, CC7.5</td>
<td>Incident response that works</td>
<td>A tabletop exercise or a real incident, with its write-up</td>
</tr>
<tr>
<td>A1.3</td>
<td>Recovery tested (if Availability is in scope)</td>
<td>A restore test with its result</td>
</tr>
</tbody></table>
<p>Automation helps with all of these by collecting the inputs and reminding people, but the evidence is a person's decision. In our collector, CC6.3 deliberately ends at <code>needs a person</code>: it gives the reviewer the list, and the reviewer's sign-off is the evidence.</p>
<h2>Build it or buy it</h2><p>The compliance automation platforms, such as Vanta, Drata, Secureframe, Sprinto and Thoropass, do the same thing at a much wider scale. They connect to source control, cloud accounts, identity providers and HR systems, map what they collect to controls, track the people-side tasks such as training and policy acknowledgments, and give the auditor a place to review the evidence. For a team going through its first audit, that coverage and the auditor relationships that come with it are the reason to pay for one.</p>
<p>What a small script like ours is good for is understanding what those tools check, and covering what is specific to you: a custom deployment pipeline, a check a platform does not have, or a control you want tested on every change. It can run alongside a platform as well as instead of one. Whichever you use, test it the way we tested ours: give it a repository with known problems and confirm it fails on each of them. A tool you have only seen report <code>pass</code> has not shown you that it can fail.</p>
<h2>Time</h2><p>For one repository and 90 days, the collector made 35 API calls and wrote 15 evidence files in 23 seconds. By hand, the closest you get is a screenshot of the ruleset, one of each merged pull request with its checks and reviews, the Dependabot tab, the security settings and the collaborator list: eleven screenshots here. They still miss the branch's history and the links from each commit to its pull request, which the direct-push control reads, and the pull request page does not put the check's start and completion times next to the merge time. We did not time the manual version, so we will not give a number for it. The difference grows with the sample: an auditor sampling 25 changes across 10 repositories is 25 pull request pages by hand, and one run per repository for the script.</p>
<h2>Summary</h2><ul>
<li>Collect evidence of what happened, not only of what is configured. Our ruleset passed while three change-management controls failed under it.</li>
<li>Compare timestamps. A check that is green today may have finished after the merge.</li>
<li>Make evidence scripts fail closed: stale or withdrawn approvals, the wrong check run, unreadable data, wrong links and truncated lists must never pass.</li>
<li>Treat a configuration scan as one rule set, not as proof of safety, and add rules for what it misses. Ours passed a database port open to the internet.</li>
<li>Leave the judgement to people, and make their sign-off the evidence. A script can gather the access list; it cannot decide who should be on it.</li>
<li>Agree with your auditor on how exceptions, such as admin bypasses, are handled before the review period starts.</li>
</ul>
<p>The collector, the Terraform example and the tests are in <a href="https://github.com/The-DevOps-Daily/soc2-evidence-collector" rel="noopener noreferrer">The-DevOps-Daily/soc2-evidence-collector</a>. Point it at a repository you own and see which controls fail.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DevOps Weekly Digest - Week 43, 2026]]></title>
      <link>https://devops-daily.com/news/2026-week-43</link>
      <description><![CDATA[⚡ Curated updates from Kubernetes, cloud native tooling, CI/CD, IaC, observability, and security - handpicked for DevOps professionals!]]></description>
      <pubDate>Mon, 19 Oct 2026 00:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/news/2026-week-43</guid>
      <category><![CDATA[DevOps News]]></category>
      <content:encoded><![CDATA[<blockquote>
<p>📌 <strong>Handpicked by DevOps Daily</strong> - Your weekly dose of curated DevOps news and updates!</p>
</blockquote>
<hr />
<h2>⚓ Kubernetes</h2><h3>📄 How AI workloads are reshaping Kubernetes autoscaling</h3><p>Kubernetes has quietly run much of the world’s containerized software on a division of labor between two control loops that rarely had to talk to each other. The scheduler places each pod on a node th</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/19/how-ai-workloads-are-reshaping-kubernetes-autoscaling/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Scale isn’t one number: three ways a Kubernetes control plane runs out of room</h3><p>Three clusters in trouble, and they look nothing alike. The first is a training fleet. It sat at a few The post Scale isn’t one number: three ways a Kubernetes control plane runs out of room appeared </p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/scaling-kubernetes-control-planes/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Automating secure container registry management for air-gapped AWS workloads</h3><p>Organizations in regulated industries run container workloads in air-gapped AWS environments, which means migrating hundreds of Helm charts and images into private Amazon ECR with consistent security </p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 AWS Containers Blog</strong></p>
<p><a href="https://aws.amazon.com/blogs/containers/automating-secure-container-registry-management-for-air-gapped-aws-workloads/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Announcing Wiz Runtime Sensor add-on for GKE: Turnkey threat detection for Kubernetes</h3><p>Everything is moving faster in the age of AI, and that includes security threats. What used to take attackers weeks to exploit can now happen in just a few minutes. Standard vulnerability scans that r</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/identity-security/wiz-sensor-for-gke-turnkey-threat-detection-for-kubernetes/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why Is My Kubernetes Job Still Pending? Tracing Kueue Admission with Headlamp</h3><p>You submit a Kubernetes Job and wait for its Pods to appear. A minute passes. Then another. The Job is still suspended, there are no application logs, and there may not even be a Pod to inspect. In a </p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Kubernetes Blog</strong></p>
<p><a href="https://kubernetes.io/blog/2026/10/14/tracing-kueue-admission-with-headlamp/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>☁️ Cloud Native</h2><h3>📄 The distroless security gap: Why bottom-up containers are the next evolution in OCI images</h3><p>Ever tried to shrink your container images to reduce their attack surface? If you have, you’ve likely discovered a frustrating The post The distroless security gap: Why bottom-up containers are the ne</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/distroless-container-security-gap/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Deploying, maintaining, and migrating production n8n on EKS</h3><p>n8n is easy to try and harder to run under production traffic. A single container with SQLite works great for a quick demo. Production is a different story entirely. You need workers, webhook processo</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/16/deploying-maintaining-and-migrating-production-n8n-on-eks/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 From Containers to Sandboxes: Developer Trust in the Agentic Era</h3><p>The 2026 Stack Overflow Developer Survey shows only 7% of developers trust AI output for important decisions. Here’s what that means for running agents safely.</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/stack-overflow-survey-2026-ai-agent-trust/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Putting Nepal on the CNCF Golden Kubestronaut map</h3><p>In this blog, I will share my Golden Kubestronaut journey, how I became the first and youngest Golden Kubestronaut from Nepal, what motivated me to pursue this path, and what it actually takes to earn</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/16/putting-nepal-on-the-cncf-golden-kubestronaut-map/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Humans of OpenTelemetry - KubeCon Japan 2026</h3><p>We’re back with our fifth edition of Humans of OpenTelemetry, and for the first time, from Japan! At KubeCon + CloudNativeCon Japan 2026 in Yokohama, I interviewed OpenTelemetry end users and practiti</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/humans-of-otel-japan/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Harbor + Docker Sandboxes: Run Safe, Reproducible Agent Evals</h3><p>Harbor can now run every eval task in Docker Sandboxes Docker Sandboxes are MicroVM environments to run agents like Claude Code, Codex, Cursor, and more safely With Sandboxes, define custom network po</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/harbor-docker-sandboxes-run-safe-reproducible-agent-evals/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Docker Hub security advisory: personal access tokens transferred during account-to-organization conversions</h3><p>CVE-2026-105445: Docker Hub account conversion could re-assign personal access tokens to a new owner. Read what happened, who was affected, and the fix.</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-hub-security-advisory-cve-2026-105445/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔄 CI/CD</h2><h3>📄 Three developer tasks worth handing to an agent</h3><p>You already have an AI assistant. It completes your code, explains code you didn’t write, and maybe even drafts a whole feature while you sip your coffee. An agent on GitLab picks up where your editor</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/three-developer-tasks-worth-handing-to-an-agent/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 5-minute demo: DevSecOps with Red Hat OpenShift to speed up application delivery</h3><p>If you're a platform engineer, you've likely lived this: your team has assembled a toolchain for CI/CD, monitoring, security scanning, GitOps, and networking. Each tool does its specific job well, but</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/5-minute-demo-devsecops-red-hat-openshift-speed-application-delivery" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Running a Green CI Pipeline Isn't the Hard Part Anymore</h3><p>AI-assisted coding is changing CI pipelines. Learn how Harness CI uses Cache, Build and Test Intelligence to reduce redundant work and speed up builds. | Blog</p>
<p><strong>📅 Oct 17, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/green-ci-pipeline-isnt-enough" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Fast, Fragile, and Burned Out: The Cost of AI Speed</h3><p>The faster teams deploy, the more releases fail, and burnout climbs right with it. What 700 engineers told Harness about AI-scale delivery. | Blog</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/burnout-is-rising-heres-what-you-can-fix" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AI Coding Assistants: Trust Is the Attack Surface</h3><p>Explore how AI coding agents can be exploited through trusted files, hooks, and MCP tools, and learn how to secure agent workflows against hidden instructions a | Blog</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/ai-coding-assistants-trust-is-the-attack-surface" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The next developer interface is ambient intelligence</h3><p>AI, agents, voice, and physical controls are shifting software development from operating tools to expressing intent—and helping developers stay focused on building. The post The next developer interf</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/news-insights/company-news/the-next-developer-interface-is-ambient-intelligence/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Staying Evergreen: How we automated secure access and repo hygiene in a shared org</h3><p>Put native GitHub features to work to review outside collaborator access and archive stale repositories. The post Staying Evergreen: How we automated secure access and repo hygiene in a shared org app</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/engineering/platform-security/staying-evergreen-how-we-automated-secure-access-and-repo-hygiene-in-a-shared-org/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitLab Secrets Manager: Govern secrets like your code</h3><p>Credentials often start in an engineer's local dev environments, then get copied into unscoped CI/CD variables to keep pipelines running. Stored there as a raw value rather than a reference to a scope</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/secrets-manager-ga/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Get every merge request in front of the right reviewer, faster</h3><p>The code is written. Now who reviews it? The volume of code has never been greater, but delivery hasn’t kept pace. At Meta, code changes on internal platforms grew 220% year over year, while changes t</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/recommend-reviewers-flow/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Connect your agents to your software lifecycle with the GitLab MCP server</h3><p>Agents are working across your team's software development lifecycle, but most still operate as separate tools, leaving the context, handoffs, and permissions for your team to stitch together. When yo</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/mcp-server-ga/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitHub availability report: September 2026</h3><p>In September, we experienced two incidents that resulted in degraded performance across GitHub services. The post GitHub availability report: September 2026 appeared first on The GitHub Blog.</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/news-insights/company-news/github-availability-report-september-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Stop Letting Deployment Approvals Slow Down Your Releases</h3><p>Streamline deployment approvals with Harness CD native approval steps and policies to reduce manual effort and accelerate software releases. | Blog</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/stop-letting-deployment-approvals-slow-down-your-releases" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🏗️ IaC</h2><h3>📄 Pulumi ESC now issues OIDC tokens for any service</h3><p>A Pulumi ESC environment can now mint a short-lived OpenID Connect (OIDC) token for any service that accepts one. Every pipeline and agent that touches your infrastructure eventually needs credentials</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/esc-oidc-tokens-for-any-service/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Scale your team's use of AWS IAM account access manager with Pulumi</h3><p>In September 2026, AWS launched AWS IAM account access manager (AAM), which lets administrators assign IAM roles to workforce users and groups in AWS IAM Identity Center. AAM combines IAM Identity Cen</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/scale-your-teams-use-of-aws-iam-account-access-manager-with-pulumi/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📊 Observability</h2><h3>📄 Announcing the 2026 OpenTelemetry Governance Committee Election Candidates</h3><p>The OpenTelemetry Election Committee is excited to announce the final list of candidates for the 2026 OpenTelemetry Governance Committee election! Eligible voters can help fill four Governance Committ</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/gc-candidates/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Get observability in your pocket with the Grafana mobile app</h3><p>When you’re on the go, your phone is often the only observability tool you have. And what you can do with it in the first few minutes matters. With the incredible innovation that has gone into mobile </p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/get-observability-in-your-pocket-with-the-grafana-mobile-app/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Carbon as a first-class signal: Bringing software carbon intensity to OpenTelemetry</h3><p>At Observability Summit Europe 2026 in Prague, I presented a keynote titled Carbon as a First-Class Signal: Designing a Standard for OpenTelemetry. The talk explored how Software Carbon Intensity (SCI</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/15/carbon-as-a-first-class-signal-bringing-software-carbon-intensity-to-opentelemetry/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Coverage-first or cost-first: two ways to automatically apply Adaptive Metrics' recommendations</h3><p>Adaptive Metrics , which is part of the Adaptive Telemetry suite, looks at how your metrics are actually used and recommends how much detail you should keep for each metric. This helps you cut costs w</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/coverage-first-or-cost-first-two-ways-to-automatically-apply-adaptive-metrics-recommendations/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Go from alert to post-incident review with Grafana Cloud IRM and AI</h3><p>You’ve got 10 minutes before your next meeting when your phone starts ringing. You’re being paged because the frontend is erroring with failed API requests. You open one tool to see what fired, then a</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/go-from-alert-to-post-incident-review-with-grafana-cloud-irm-and-ai/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing Claude Platform Tracing in New Relic AI Monitoring</h3><p>Claude Platform exports OpenTelemetry (OTel) traces natively from Claude Managed Agents, the Messages API and the Claude SDKs, in public beta.</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/news/anthropic-otel-ai-observability" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Sentry's Flutter SDK 10.0: Streaming Traces and More Accurate Crash-Free Rates</h3><p>Flutter SDK 10.0: streaming traces by default, accurate crash-free rates, dedicated app startup traces, and expanded Dio network error capture.</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/flutter-sdk-10/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 From Backup to Assurance: Monitoring Veeam with Zabbix</h3><p>Your data isn’t truly protected if you don’t know when that protection fails. Today’s organizations depend on backup infrastructure to safeguard critical workloads, yet a single backup failure, dwindl</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Zabbix Blog</strong></p>
<p><a href="https://blog.zabbix.com/from-backup-to-assurance-monitoring-veeam-with-zabbix/33694/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Versioned metrics: Shipping the first Release Candidate metrics in hostmetrics</h3><p>OpenTelemetry Collector Contrib v0.160.0 adds the ability to emit release candidate (RC) process semantic conventions metrics to the hostmetrics receiver’s process scraper, the last stage before a sta</p>
<p><strong>📅 Oct 13, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/hostmetrics-process-scraper-rc/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Improving developer productivity with New Relic AI</h3><p>Context switching is one of the biggest hidden costs in a developer's day. New Relic Ground Truth closes that gap by providing AI systems with trusted operational context from your production environm</p>
<p><strong>📅 Oct 13, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/news/improving-developer-productivity-with-new-relic-ai" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Models commoditize. Context compounds.</h3><p>As AI models commoditize, trusted operational context becomes the advantage. Learn why closing the AI Trust Gap starts with better context.</p>
<p><strong>📅 Oct 13, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/ai/models-commoditize-context-compounds" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔐 Security</h2><h3>📄 Threats Making WAVs - Incident Response to a Cryptomining Attack</h3><p>Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/threats-making-wavs-incident-reponse-cryptomining-attack" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Beyond the patch: Protecting your NetScaler gateway against future vulnerabilities</h3><p>CVE-2026-88771 lets attackers run commands on NetScaler Gateway before anyone signs in. Patching takes time most teams don't have, so the sign-in decision has to move in front of the vulnerable code.</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/protect-your-netscaler-gateway/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Red Hat Enterprise Linux is now a native offering on Oracle Cloud Infrastructure</h3><p>Organizations running mission-critical workloads face a fundamental challenge: how to take advantage of Oracle Cloud Infrastructure's performance and economics without sacrificing the stability, secur</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/red-hat-enterprise-linux-now-native-offering-oracle-cloud-infrastructure" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 A patch only reduces risk when it runs in production</h3><p>The patch was ready in days. It took months to ship. Why?I hear that question in nearly every customer conversation about vulnerability management. The honest answer is rarely “We didn’t know about it</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/patch-only-reduces-risk-when-it-runs-production" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Gitea 28.2.0 is released</h3><p>We are excited to announce the release of <strong>Gitea 28.2.0</strong>, the second maintenance release for the 28 series. It contains security fixes and a number of bug fixes for Actions, pull requests, authentic</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 Gitea Blog</strong></p>
<p><a href="https://blog.gitea.com/release-of-28.2.0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Model Context Protocol gateway for Red Hat OpenShift is now generally available</h3><p>The Model Context Protocol (MCP) gateway is now generally available (GA) as part of the Red Hat Connectivity Link 1.5 release, and natively integrated into the Red Hat AI portfolio. This follows a suc</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/model-context-protocol-gateway-red-hat-openshift-now-generally-available" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Different Starting Points, One Data Foundation: What’s New in TiDB Cloud</h3><p>Nobody approves a project called “add complexity.” It builds up one shard, one region, and one compliance requirement at a time, until a single database has turned into a fleet and the team running it</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 TiDB Blog</strong></p>
<p><a href="https://www.pingcap.com/blog/tidb-cloud-scaile-2026-product-announcements/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Canonical releases Ubuntu 26.10 Stonking Stingray</h3><p>The latest interim release of Ubuntu previews privacy conscious speech-to-text, enables post-quantum package signing, completes the rollout of Rust coreutils, and improves enterprise identity integrat</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/canonical-releases-ubuntu-26-10-stonking-stingray" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing Evo Agentic AppSec: Security That Keeps Up With AI Development</h3><p>AI-driven development is outpacing traditional security workflows. Snyk’s Agentic AppSec uses security agents to find, fix, and verify vulnerabilities continuously, while humans set the bar.</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/announcing-snyk-agentic-appsec/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?</h3><p>VulnBump tests AI agents on 307 real dependency upgrades. A fix counts only if the vulnerability is gone and the build and tests stay green.</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/vulnbump/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AI Agent Security Risks: What Can Go Wrong When Agents Act on Their Own</h3><p>Learn the top AI agent security risks, from prompt injection to excessive autonomy and credential exposure, and how to contain them.</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/ai-agent-security-risks/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Fixes available for CVE-2026-92392 and other curl vulnerabilities</h3><p>Introduction On 7 October 2026, Daniel Stenberg published a blog post describing a plan to release a new version of curl on 14 October 2026 in order to address 22 known security vulnerabilities. These</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/curl-vulnerabilities-fixes-available" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>💾 Databases</h2><h3>📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers</h3><p>Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/please-read-me-opportunistic-ransomware-devastating-mysql-servers" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Feast Feature and Vector Serving at ScyllaDB Speed</h3><p>A new and tighter integration between Feast and ScyllaDB online feature store, including vector search support and better performance</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 ScyllaDB Blog</strong></p>
<p><a href="https://www.scylladb.com/2026/10/19/feast-feature-and-vector-serving-at-scylladb-speed/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Your Client Connection Pool isn’t the Problem</h3><p>A customer faced 1,920 required connections against a cluster sized for 360 days before go-live. More hardware or smaller pools was not the solution. Discover how YSQL Connection Manager resolved the </p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Yugabyte Blog</strong></p>
<p><a href="https://www.yugabyte.com/blog/client-connection-pool/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pg_hint_plan v1.9.0 Released</h3><p>The pg_hint_plan development team is pleased to announce pg_hint_plan 1.9.0. This release of pg_hint_plan adds compatibility with PostgreSQL 19, and includes various improvements and bug fixes. For mo</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pg_hint_plan-v190-released-3408/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pg_utl_smtp v2.0 released</h3><p>Antananarivo, Madagascar - October 06, 2026 PostgreSQL UTL_SMTP compatibility extension pg_utl_smtp is a PostgreSQL extension to create, manage and use Oracle-style UTL_SMTP package. The use and behav</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pg_utl_smtp-v20-released-3395/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 FOSDEM PGDay 2027: two days of PostgreSQL, with calls for papers and sponsors now open!</h3><p>Join us for FOSDEM PGDay 2027 on January 28 and 29 at the Brussels Marriott Hotel Grand Place. This year, we're expanding to two days, bringing you more PostgreSQL talks and more time to connect with </p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/fosdem-pgday-2027-two-days-of-postgresql-with-calls-for-papers-and-sponsors-now-open-3412/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pgBackRest 2.60.0 Released</h3><p>October 18, 2026: The pgBackRest community is pleased to announce the release of pgBackRest 2.60.0, the latest version of the reliable, easy-to-use backup and restore solution that can seamlessly scal</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pgbackrest-2600-released-3416/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 TiDB Cloud BYOC Public Preview: Fully Managed Distributed SQL in Your AWS Account</h3><p>Teams running distributed SQL in production face a trade-off. A managed service removes the work of upgrades, scaling, and backups, and it moves the database into the vendor’s cloud account. Self-mana</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 TiDB Blog</strong></p>
<p><a href="https://www.pingcap.com/blog/tidb-cloud-byoc-public-preview/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 From 2,500 requests to 24: Rebuilding star history without the stargazers</h3><p>Our new repo star history API now provides trends without returning individual identities or requiring unbounded database work. These are the engineering tradeoffs we made. The post From 2,500 request</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/engineering/architecture-optimization/from-2500-requests-to-24-rebuilding-star-history-without-the-stargazers/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 YSQL Connection Manager is Now GA</h3><p>We are excited to announce the General Availability (GA) of YSQL Connection Manager in YugabyteDB 2026.1.1, YugabyteDB Anywhere, and YugabyteDB Aeon. Built directly into every YugabyteDB node, YSQL Co</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 Yugabyte Blog</strong></p>
<p><a href="https://www.yugabyte.com/blog/ysql-connection-manager-is-now-ga/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Top 5 operators for Red Hat OpenShift</h3><p>In the 1980s, the spreadsheet was considered such a valuable tool that would make buying a computer just to run it worthwhile. The term coined to describe such an application was a killer app. Is ther</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/top-5-operators-red-hat-openshift" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Eliminate the Database War Room with New Relic Database 360</h3><p>Unify apps, databases, and infrastructure with New Relic Database 360. Pinpoint query bottlenecks faster and eliminate war rooms.</p>
<p><strong>📅 Oct 13, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/infrastructure-monitoring/new-relic-database-360-ga" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🌐 Platforms</h2><h3>📄 The Oracle of Delphi Will Steal Your Credentials</h3><p>Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-oracle-of-delphi-steal-your-credentials" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger</h3><p>In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-nansh0u-campaign-hackers-arsenal-grows-stronger" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon CloudWatch log centralization now available in AWS GovCloud and additional commercial Regions</h3><p>Amazon CloudWatch log centralization is now available in AWS GovCloud (US) and 13 additional commercial Regions. CloudWatch log centralization copies log data from multiple AWS accounts and Regions in</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-cloudwatch-centralization-region-expansion" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AWS Lambda durable functions are now available in AWS GovCloud (US) Regions</h3><p>AWS Lambda durable functions are now available in the AWS GovCloud (US-West) and AWS GovCloud (US-East) Regions. Lambda durable functions enable developers to build reliable multi-step applications an</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/durablefunctions-govcloud/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Nemotron 3.5 Super VL model now available on Amazon SageMaker JumpStart</h3><p>NVIDIA's Nemotron 3.5 Super VL is now available on Amazon SageMaker JumpStart, expanding the portfolio of foundation models available to AWS customers. This vision-language model from the Nemotron 3.5</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/01/nvidia-nemotron-3.5-super-vl-on-sagemaker-jumpstart/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Day 0 support for Reflection’s Beam on Google Cloud: 5x faster perf with llm-d and NVIDIA Dynamo</h3><p>Today, Reflection released Beam, its first open-weight model built to give developers, researchers, and enterprises open control and customization for frontier software engineering and long-horizon ag</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/ai-infrastructure/day-0-support-with-5x-perf-for-reflection-beam-on-google-cloud/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What’s new with Google Cloud</h3><p>Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/inside-google-cloud/whats-new-google-cloud/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Architecting autonomy: A design pattern for high-compute, governed agents in biotech</h3><p>In production systems today, most AI agents operate as lightweight dispatchers. They parse a prompt, make an API call, retrieve a record, and hand back control. Their execution is fast, stateless and </p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/topics/healthcare-life-sciences/architecting-scalable-agentic-biotech-workloads/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Beam model by Reflection is now available on Amazon SageMaker JumpStart and AWS Trainium</h3><p>Beam, the first open weight model offered by US-based Reflection, is now available in Amazon SageMaker JumpStart, the model hub in Amazon SageMaker AI. According to Beam, they operate at the frontier </p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/01/reflection-beam-on-sagemaker-jumpstart-and-aws-trainium/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Junie Local Comes to the NVIDIA RTX Spark</h3><p>Junie Local is supported on the NVIDIA RTX Spark from day one, and Windows support moves from experimental to stable. NVIDIA’s RTX Spark platform pairs a Blackwell RTX GPU with a Grace CPU to run larg</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/junie/2026/10/junie-local-rtx-spark/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Continuous accountability for Verified bots</h3><p>Identity alone doesn't guarantee a bot will behave responsibly. We are introducing a system to continuously evaluate whether Verified bots honor site preferences, like robots.txt and HTTP 429s, ensuri</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/verified-bots-accountability/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 A fully programmable secure web gateway</h3><p>Cloudflare Gateway and Data Loss Prevention have inherited the flexibility and power of Cloudflare Workers through our full-stack integration with the Cloudflare Developer platform.</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/programmable-gateway-dlp/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📰 Misc</h2><h3>📄 Visual Studio Code 1.143 (Insiders)</h3><p>Learn what is new in Visual Studio Code Insiders, including the latest features and improvements. Read the full article</p>
<p><strong>📅 Oct 21, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_143" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 MicroCloud and LXD LTS releases now available</h3><p>Canonical announces the latest Long Term Supported (LTS) releases for MicroCloud and LXD</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/microcloud-and-lxd-lts-releases-2026" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 JetBrains .NET Day Online 2026: Watch the Recordings</h3><p>Every session from October 7 is now on YouTube. .NET Day Online 2026 took place on October 7. Thanks to everyone who watched the livestream and asked questions in the chat! If you missed a session, or</p>
<p><strong>📅 Oct 19, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/dotnet/2026/10/19/jetbrains-dotnet-day-online-2026-watch-the-recordings/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Can you bake Maven into a neural network?</h3><p>We tried, and it builds. We compiled the decision-making part of a Maven-style build tool into a fixed weight matrix and used it to build a real Java library. The output matches Apache Maven's byte fo</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/ai/2026/10/can-you-bake-maven-into-a-neural-network/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Gemini 4 Argon vs. Claude Opus 5.5: Half the price and about half as accurate</h3><p>When Google announced Gemini 4 Argon to a small group of testers on September 30, it came with a benchmark The post Gemini 4 Argon vs. Claude Opus 5.5: Half the price and about half as accurate appear</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/gemini-argon-claude-opus/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Liquid AI caught its model cheating on coding tests — then it recognized the sandbox</h3><p>Liquid AI’s new reasoning model was built to solve coding problems, but during training it kept combing Git history, GitHub, The post Liquid AI caught its model cheating on coding tests — then it reco</p>
<p><strong>📅 Oct 18, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/liquid-ai-reward-hacking/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How to lower Android™ development costs by removing hardware from the critical path</h3><p>As an Android developer, you’ll almost always have access to a device. However, when handling hardware, the device that you actually need often happens to be already in use, set up incorrectly, stored</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/lower-android-development-cost" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 5 Simple Tips to Supercharge Your Claude SDK Experience with MLflow</h3><p>The Claude SDKs now emit OpenTelemetry traces natively. Five tips to trace your agent with standard OpenTelemetry, see each run as one trace, find where the tokens go, score every run, and compare the</p>
<p><strong>📅 Oct 16, 2026</strong> • <strong>📰 MLflow Blog</strong></p>
<p><a href="https://mlflow.org/blog/claude-sdk-otel-tips/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Google Summer of Code 2026 With the Kotlin Foundation: Recap</h3><p>Google Summer of Code 2026 has wrapped up! This was the Kotlin Foundation’s fourth year participating in GSoC, bringing new contributors into the Kotlin open-source ecosystem through hands-on projects</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/kotlin/2026/10/google-summer-of-code-2026-with-kotlin/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What designers should know about semiotics</h3><p>What happens when you look at an apple? When you see it, you perceive the apple and assign meaning to it based on your language, experience, and personal preferences – such as memories of school lunch</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 Canonical Blog</strong></p>
<p><a href="https://canonical.com//blog/what-designers-should-know-about-semiotics" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Modernizing the VS Code UI</h3><p>See how developer feedback is shaping the modern VS Code interface, from layout density and editor tabs to themes and responsiveness. Read the full article</p>
<p><strong>📅 Oct 15, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/blogs/2026/10/15/modern-ui" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Visual Studio Code 1.142</h3><p>Learn what's new in Visual Studio Code 1.142 Read the full article</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_142" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Postgres 19 Slipped to November: Every Pulled Feature and Why]]></title>
      <link>https://devops-daily.com/posts/postgres-19-delay-pulled-features</link>
      <description><![CDATA[PostgreSQL 19 now ships on 19 November, and 15 items left its release notes after beta 1. Here is each pulled feature with the reason from its revert, what to use instead, a script that finds beta-era code that depends on them, and why "AI found too many bugs" is only part of the story.]]></description>
      <pubDate>Sat, 17 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/postgres-19-delay-pulled-features</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[PostgreSQL]]></category><category><![CDATA[Databases]]></category><category><![CDATA[DevOps]]></category><category><![CDATA[Upgrades]]></category><category><![CDATA[Code Review]]></category><category><![CDATA[AI]]></category>
      <content:encoded><![CDATA[<blockquote>
<p><strong>Note</strong></p>
<p><strong>Updated 15 November 2026:</strong> RC2 was stamped on 9 November and released on 12 November together with the minor releases, and general availability is still planned for 19 November. Since the first version of this post, one more release-note feature was reverted (the libpq <code>servicefile</code> option, on 18 October), and so was a beta behavior change that affects expression indexes built on JSON constructors (on 6 November). The counts are refreshed at the 14 November branch head.</p>
</blockquote>
<p>PostgreSQL 19 was planned for October 2026. The project's roadmap now says 19 November, and the second release candidate came out on 12 November. It will ship without SQL/PGQ graph queries, <code>UPDATE</code>/<code>DELETE ... FOR PORTION OF</code>, online checksum changes, <code>ALTER TABLE ... MERGE/SPLIT PARTITION</code>, and eight smaller features that were in the beta 1 release notes. The short version going around is that AI reviewers found more bugs than the developers could fix. That is part of it. Peter Eisentraut, who committed several of the reverted features, lists several possible causes and says it is probably a different combination for each feature. The mailing-list threads and the commit log fit his account, and which cause matters to you depends on what you planned to use.</p>
<p>This post goes through every feature that was pulled, with the reason given in its revert and what to use instead. It gives you a script that finds beta-era code using them, covers a limit in REPACK CONCURRENTLY (a feature that stayed) that its reference page does not mention, and counts what the commit log actually says about AI.</p>
<h2>TLDR</h2><ul>
<li><strong>15 release-note items, covering 12 features, are gone since beta 1.</strong> Eleven features were reverted after beta 1 was stamped on 1 June, the last of them on 18 October. The twelfth was reverted in May, but its release-note entry stayed until 13 October.</li>
<li><strong>The five big reverts landed between 27 August and 16 September.</strong> Their revert messages list 47 commits (SQL/PGQ), 30 (online checksums, some only partly reverted), 23 (FOR PORTION OF), 14 (MERGE/SPLIT PARTITION) and 13 (<code>pg_get_*_ddl()</code>).</li>
<li><strong>The causes, as Eisentraut describes them:</strong> some code was not good enough, some features had design problems that beta could not fix, LLM-assisted review found a long tail of edge-case bugs faster than they could be handled, security work from April to August took the senior reviewers, and "everything works with everything" in Postgres makes some features very hard to finish.</li>
<li><strong>The November date has its own reason:</strong> the release team is folding the fixes from the 12 November minor releases into 19 before general availability.</li>
<li><strong>AI in the commit log:</strong> 49 of the 1,232 commits after beta 1 mention AI assistance, an AI tool or an AI lab, 24 of them in the 9 November security batch. That counts only explicit credit, so it is a floor.</li>
<li><strong>For your upgrade:</strong> run the check script over migrations and ops code written against a beta, drop or redefine beta-era expression indexes, generated columns and partition keys that use <code>JSON_ARRAY()</code> or <code>JSON_OBJECT()</code> over arrays, composite types or ranges, plan REPACK CONCURRENTLY runs one at a time across the whole cluster, and expect the first scheduled minor release on 11 February 2027.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Code, migrations or runbooks written against a PostgreSQL 19 beta, if you want to check them</li>
<li><code>bash</code> and GNU <code>grep</code> for the check script</li>
<li>Python 3 and an authenticated <code>gh</code> CLI if you want to rebuild the counts from the source tree</li>
</ul>
<p>Everything below comes from the <code>REL_19_STABLE</code> branch at commit <code>2f15cec90d</code> (14 November 2026; the first version of this post used <code>221423812d</code> from 17 October), the release notes at the <code>REL_19_BETA1</code> tag, and the threads linked in each section. The scripts and the data snapshot are in the companion repo:</p>
<p><a href="https://github.com/The-DevOps-Daily/pg19-revert-ledger" rel="noopener noreferrer">The-DevOps-Daily/pg19-revert-ledger on GitHub</a></p>
<h2>The timeline</h2><p>Dates are commit and tag dates in the source tree. Public announcements follow a few days after a tag.</p>
<table>
<thead>
<tr>
<th>Date</th>
<th>Event</th>
</tr>
</thead>
<tbody><tr>
<td>April 2026</td>
<td>Feature freeze for 19</td>
</tr>
<tr>
<td>11 May</td>
<td>23 security commits for the May minor releases</td>
</tr>
<tr>
<td>24 May</td>
<td>Database-specific logical replication snapshots reverted</td>
</tr>
<tr>
<td>1 June</td>
<td>Beta 1 stamped</td>
</tr>
<tr>
<td>8 June to 17 July</td>
<td>Four smaller reverts: fast defaults for domains, <code>pg_dumpall</code> formats, a lower <code>ALTER DOMAIN</code> lock, <code>GROUP BY ALL</code></td>
</tr>
<tr>
<td>13 July</td>
<td>Beta 2 stamped</td>
</tr>
<tr>
<td>10 August</td>
<td>Beta 3 stamped, and 35 security commits for the August minor releases</td>
</tr>
<tr>
<td>25 August</td>
<td>Robert Haas posts an LLM ranking of the riskiest PostgreSQL 19 features</td>
</tr>
<tr>
<td>27 August to 16 September</td>
<td>MERGE/SPLIT PARTITION, SQL/PGQ, CREATE SCHEMA, <code>pg_get_*_ddl()</code>, FOR PORTION OF and online checksums reverted</td>
</tr>
<tr>
<td>21 September</td>
<td>Beta 4 stamped</td>
</tr>
<tr>
<td>12 October</td>
<td>RC1 stamped</td>
</tr>
<tr>
<td>18 October</td>
<td>libpq <code>servicefile</code> connection option reverted</td>
</tr>
<tr>
<td>6 November</td>
<td>More complete <code>to_json()</code>/<code>to_jsonb()</code> immutability checks reverted</td>
</tr>
<tr>
<td>9 November</td>
<td>RC2 stamped, and 84 security commits for the November minor releases</td>
</tr>
<tr>
<td>12 November</td>
<td>RC2 released together with the minor releases</td>
</tr>
<tr>
<td>19 November</td>
<td>General availability planned</td>
</tr>
</tbody></table>
<p>The cluster from late August to mid September is the part that made news. Five large features came out in three weeks, two of them in the week before the last beta.</p>
<h2>What was pulled, and why</h2><p>The list comes from comparing the release notes at beta 1 with the branch head. An item counts as removed when none of the commits it cites appear in any current item, so items that were only reworded or merged do not count. That gives 15 items. The three <code>pg_get_*_ddl()</code> functions are one feature, and the libpq <code>servicefile</code> option and its psql variable are another, so 12 features.</p>
<table>
<thead>
<tr>
<th>Feature</th>
<th>Reverted</th>
<th>Reason given</th>
<th>What to do instead</th>
</tr>
</thead>
<tbody><tr>
<td>SQL/PGQ property graphs (<code>CREATE PROPERTY GRAPH</code>, <code>GRAPH_TABLE</code>)</td>
<td>7 Sep</td>
<td>Open design questions the release management team listed on 2 September: graph metadata left orphaned by <code>DROP ... CASCADE</code> (which could make a graph impossible to dump and restore), how labels and properties are scoped, whether a graph needs a <code>pg_class</code> entry, and <code>pg_dump</code> dependency loops. Some fixes still needed catalog changes.</td>
<td>Recursive CTEs, or a graph extension such as Apache AGE (check its support for your version)</td>
</tr>
<tr>
<td><code>UPDATE</code>/<code>DELETE ... FOR PORTION OF</code></td>
<td>15 Sep</td>
<td>Lost updates under <code>READ COMMITTED</code> when concurrent temporal updates and deletes touched the same rows. The documented <code>SELECT ... FOR UPDATE</code> workaround did not cover <code>DELETE</code>.</td>
<td>Split the ranges in application code or a trigger. The <code>WITHOUT OVERLAPS</code> constraints from Postgres 18 still enforce no overlap.</td>
</tr>
<tr>
<td>Online data checksums (<code>pg_enable_data_checksums()</code>, <code>pg_disable_data_checksums()</code>)</td>
<td>16 Sep</td>
<td>The revert message says it "saw a number of postcommit fixes during the beta period" and adds: "Suspicions were raised about the risk of more issues surfacing after GA."</td>
<td><code>pg_checksums --enable</code> on a stopped cluster, as before. Clusters that <code>initdb</code> creates on 18 or later have checksums on by default.</td>
</tr>
<tr>
<td><code>ALTER TABLE ... MERGE PARTITIONS</code> / <code>SPLIT PARTITION</code></td>
<td>27 Aug</td>
<td>Reverted "due to multiple design issues which are too late to address in this release cycle". The thread lists moved rows decoded as plain inserts (which breaks logical replication), stored generated columns that silently get new values, partition-specific indexes and constraints that are silently lost, and open questions about publication membership.</td>
<td><code>DETACH PARTITION</code>, create the new partition, <code>INSERT ... SELECT</code>, <code>ATTACH PARTITION</code></td>
</tr>
<tr>
<td><code>pg_get_role_ddl()</code>, <code>pg_get_tablespace_ddl()</code>, <code>pg_get_database_ddl()</code></td>
<td>13 Sep</td>
<td>Design review: role output could not recreate password hashes the way <code>pg_dumpall</code> does, there was no dependency ordering, and it would be a second, separate implementation of translating catalog state into SQL beside <code>pg_dump</code>.</td>
<td><code>pg_dumpall --roles-only</code> or <code>--tablespaces-only</code>, and <code>pg_dump --create</code> for the <code>CREATE DATABASE</code> statement</td>
</tr>
<tr>
<td><code>pg_dumpall</code> custom, directory and tar formats, and <code>pg_restore --no-globals</code></td>
<td>18 Jun</td>
<td>A post-commit review recommended reverting and finishing it through a commitfest.</td>
<td><code>pg_dumpall --globals-only</code>, plus <code>pg_dump -Fc</code> or <code>-Fd</code> per database</td>
</tr>
<tr>
<td><code>GROUP BY ALL</code> (with no columns after it)</td>
<td>17 Jul</td>
<td>Wrong results when <code>ORDER BY</code> used non-default equality semantics. A clean fix "seems like too much code churn for late beta", and the revert says "We'll revert and try again in v20".</td>
<td>List the grouping columns</td>
</tr>
<tr>
<td>Fast default for columns on domains with constraints</td>
<td>8 Jun</td>
<td>Failed with an empty table and an invalid default. The revert says the fix will involve an addition to the table access method API, so it was not ready for 19.</td>
<td>Expect a table rewrite when you add such a column, as in 18</td>
</tr>
<tr>
<td>More object types inside <code>CREATE SCHEMA</code></td>
<td>11 Sep</td>
<td>It depended on a change to the order of <code>CREATE SCHEMA</code> subcommands, and the compatibility risk of that change was judged larger than the gain. Foreign keys in <code>CREATE SCHEMA</code> still run at the end, because the SQL standard requires it.</td>
<td>Create the objects in separate statements after <code>CREATE SCHEMA</code></td>
</tr>
<tr>
<td>Lower lock level for <code>ALTER DOMAIN ... VALIDATE CONSTRAINT</code></td>
<td>18 Jun</td>
<td>Unsafe: DML that had already started when a <code>NOT VALID</code> constraint was added could still write rows that violate it.</td>
<td>It takes <code>ShareLock</code> on tables that use the domain, as before. Run it in a quiet window.</td>
</tr>
<tr>
<td>libpq <code>servicefile</code> connection option and the psql <code>SERVICEFILE</code> variable</td>
<td>18 Oct</td>
<td>The revert says "PGSERVICEFILE is sufficient for the purpose of specifying a per-session service file", and that some details "will need more attention before being actually usable".</td>
<td>Set the <code>PGSERVICEFILE</code> environment variable, as in 18</td>
</tr>
<tr>
<td>Logical decoding plugins that skip shared catalogs (database-specific snapshots)</td>
<td>24 May</td>
<td>The revert says it "was determined to have fundamental flaws".</td>
<td>Nothing to replace, but it limits REPACK CONCURRENTLY (see below)</td>
</tr>
</tbody></table>
<p>Three smaller behavior changes that never made the release notes were reverted too: forcing <code>LC_COLLATE</code> to <code>C</code> in the postmaster (the commit gives the reason as "Out of an abundance of caution"), basing identifier case folding on the default locale, which behaved differently for the builtin provider with locale <code>C</code> and a single-byte encoding, and, on 6 November, more complete immutability checks in <code>to_json()</code> and <code>to_jsonb()</code>. The last one can break your upgrade if you used it on a beta; see the upgrade section below.</p>
<p>Two features that stayed were cut back. REPACK CONCURRENTLY lost the snapshot change it depended on, which is the limit covered later in this post. The new fast path for foreign key checks went back to SPI on 15 October for keys whose values need a cast function.</p>
<h2>Cause 1: design problems that beta could not fix</h2><p>Eisentraut starts his own list with the plain possibility that some of the code was not good enough. The threads behind the biggest reverts show something more specific: questions that had no agreed answer before the release.</p>
<p>FOR PORTION OF is the clearest example. On 10 September Andres Freund flagged its concurrency behavior in a code review, and the following day he showed that under <code>READ COMMITTED</code>, concurrent temporal updates and deletes on the same rows can lose all or part of the second operation. The leftover rows one statement inserts are not chained to the original row, and <code>EvalPlanQual</code> does not re-scan for them. With <code>DELETE ... FOR PORTION OF</code> in the mix, even the documented <code>SELECT ... FOR UPDATE</code> workaround did not prevent it. The author proposed raising a serialization failure so the client could retry. Andres replied:</p>
<blockquote>
<p>For better or worse, READ COMMITTED is extremely widely used, with one reason for that being not needing to retry, and making FPO not really work that way doesn't seem convincing to me.</p>
</blockquote>
<p>When the release management team asked what the committer, Peter Eisentraut, was proposing, he answered: "I was trying to nudge toward reverting without saying it." He reverted it the next day.</p>
<p>SQL/PGQ went the same way. The release management team posted the list of open design issues on 2 September, and Tom Lane had already written on 25 August:</p>
<blockquote>
<p>At this point I'd be willing to bet dinner that if we ship it in v19 there will be post-release bug discoveries that are unfixable until v20.</p>
</blockquote>
<p>His reason was specific: fixes still under discussion needed parse tree or catalog changes, which bump <code>catversion</code>. Minor releases do not change the catalog, so a bug that needs such a change after release can stay unfixed until 20.</p>
<p>MERGE/SPLIT PARTITION, the database-specific snapshots and the <code>pg_get_*_ddl()</code> functions were reverted for design reasons too. For the DDL functions, Noah Misch's review argued that PostgreSQL should not carry two independent implementations of the logic that translates catalog state into SQL. The fast default for domains needed an addition to the table access method API, and its revert says it was not ready for 19.</p>
<h2>Cause 2: a long tail found faster</h2><p>Eisentraut's post about the reverts is the most direct account from someone on the receiving end. On the long tail:</p>
<blockquote>
<p>But with LLM-assisted code review, these kinds of things can get found much faster, and then you’re staring at a list of like forty defect reports. And even if each of them requires only a three-line fix on average, there is a fixed overhead to dealing with each of these, and then you run out of time.</p>
</blockquote>
<p>He adds that most of the reverted code was written before LLM-assisted review was this useful, so "the effective quality bar has been raised after the time the feature was written and committed." He also says, as far as he can tell, the problem is not vibe coding: the code predates that.</p>
<p>A mailing-list post from 25 August shows how this fed the decisions. Robert Haas asked Claude to rank the PostgreSQL 19 features by the number and type of bugs already fixed after feature freeze, and posted the top six to ask whether any should be reverted. The counts are the model's tally of fixed bugs, not an audited total, and the ranking found no new bugs itself. Here are its six entries, sorted by fix count, next to what happened to each feature:</p>
<p><strong>Post-freeze fixes per PostgreSQL 19 feature, as tallied by an LLM, and what happened to it</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>REPACK / REPACK CONCURRENTLY</td>
<td>28 fixes</td>
<td>Kept in 19</td>
</tr>
<tr>
<td>Online data checksums</td>
<td>25 fixes</td>
<td>Reverted</td>
</tr>
<tr>
<td>UPDATE/DELETE FOR PORTION OF</td>
<td>17 fixes</td>
<td>Reverted</td>
</tr>
<tr>
<td>SQL/PGQ property graphs</td>
<td>17 fixes</td>
<td>Reverted</td>
</tr>
<tr>
<td>Foreign key check fast path</td>
<td>16 fixes</td>
<td>Kept in 19</td>
</tr>
<tr>
<td>postgres_fdw statistics import</td>
<td>7 fixes</td>
<td>Kept in 19</td>
</tr>
</tbody></table>
<p><em>Counts from the LLM ranking Robert Haas posted to pgsql-hackers on 25 August 2026 ("scary patch contest"). Values marked ~ in the post are shown as the round number. Sorted by count; the post ranked the foreign key fast path first. Outcome from the REL_19_STABLE commit log, 17 October 2026.</em></p>
<p>The feature with the most fixes, REPACK, is still in 19. The fix count did not decide anything on its own. In the threads, the reverted features had open design questions (FOR PORTION OF, SQL/PGQ) or a failure mode too dangerous to risk a long tail. For online checksums the ranking put it plainly: "A corruption-detection feature producing false positives is exactly the wrong failure mode."</p>
<h3>What the commit log says about AI</h3><p>The commit messages credit AI in a few clear forms: security reports from Anthropic's and OpenAI's research teams, "Reported-by: Claude Code (via Noah Misch)", "Found by Bharath using AI assisted review with Claude", and disclosures that a draft of a patch was written with an LLM. Counting every commit whose message mentions AI assistance, an AI tool or an AI lab:</p>
<p><strong>REL_19_STABLE commits that mention AI assistance, tools or labs, and security commits, per month</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Apr</th>
<th>May</th>
<th>Jun</th>
<th>Jul</th>
<th>Aug</th>
<th>Sep</th>
<th>Oct</th>
<th>Nov</th>
</tr>
</thead>
<tbody><tr>
<td>AI-credited</td>
<td>0 commits</td>
<td>1 commits</td>
<td>1 commits</td>
<td>2 commits</td>
<td>14 commits</td>
<td>2 commits</td>
<td>5 commits</td>
<td>25 commits</td>
</tr>
<tr>
<td>Security: CVE</td>
<td>0 commits</td>
<td>23 commits</td>
<td>0 commits</td>
<td>0 commits</td>
<td>35 commits</td>
<td>0 commits</td>
<td>0 commits</td>
<td>84 commits</td>
</tr>
</tbody></table>
<p><em>From the commit messages on REL_19_STABLE, 1 April to 14 November 2026 (1,796 commits; November is a partial month). AI-credited = the message names Claude, Codex, an LLM, AI-assisted review, Anthropic, OpenAI or similar. Security = a 'Security: CVE-' trailer. Explicit credit only, so the AI line is a floor.</em></p>
<p>After beta 1 that is 49 of 1,232 commits, or 4.0%. In the first version of this post, at 17 October, it was 23 of 937, or 2.5%. Of the 25 in November, 24 are in the 9 November security batch, credited mostly to Anthropic's OSS program, to "Claude and Ada Logics" and to OpenAI's security teams. It only counts credit in the commit message. Much of the review happens on the mailing list, and a reviewer who used an LLM to find a bug and then reported it in their own words is not counted unless the commit says so. So read it as a floor, not as the share of bugs AI found.</p>
<p>The threads add two details that the headline version leaves out. In the FOR PORTION OF thread, Andres Freund raised the concurrency problem in his own review first. His reproduction of the lost update was then an AI-generated isolation test, but he wrote that he "really needed to force both Opus 5 and Fable 5.1 to get to it, they both swore up and down that this isn't a real issue at first". And in the <code>pg_get_*_ddl()</code> review, Noah Misch ran an Opus 4.8 review as well and wrote that it "found some minor issues that aren't worth addressing before the above". The revert came from his own design objections. In both cases the LLM was a tool in a person's review, and people made the calls.</p>
<h2>Cause 3: security work took the reviewers</h2><p>The third cause is in Eisentraut's post too, and the commit log shows the timing. He calls it the "Vulnpocalypse": from April, the start of feature freeze, to August, the most recent security release when he wrote, many senior developers were busy fixing security issues. The branch has 142 commits with a <code>Security: CVE</code> trailer since April, covering 94 CVEs, all on three days: 23 on 11 May, 35 on 10 August and 84 on 9 November, the last batch for the 12 November minor releases. Several of the AI-credited commits in August, and most of those in November, are those security fixes.</p>
<p>The review and testing that normally happens during beta was reduced while that work ran, and it picked up again in August. He offers that as the possible reason so many issues surfaced, and so many reverts happened, so late. Commit counts cannot measure how busy reviewers were, but the May and August security dates and the late-August cluster of reverts fit it. He thinks that with about two more months, much of the long tail could have been fixed, but the community chose to keep to the schedule instead of delaying for more fixes.</p>
<h2>Cause 4: everything works with everything</h2><p>The last cause is structural. Eisentraut's example is a new SQL feature that has to work with domains, domains over composite types, a domain field with a <code>NOT NULL</code> constraint, a dropped and re-added column, a partition detached and reattached with a different column order, inside a view called from a security-definer function from a trigger. Nobody can test those combinations by hand. He says LLM-assisted fuzzing finds problems in them quickly and probably has to become part of feature development, and that internal interfaces should change so that each new combination does not open so many interactions.</p>
<p>He also points to something the project does not have: a way to commit a feature as experimental and mature it in the main tree. Some of the reverted work suffered from misunderstandings about exactly that.</p>
<h2>Why the date is 19 November</h2><p>The reverts explain what is missing. They do not explain the date. The RC1 announcement does: the minor releases on 12 November "will include a large number of fixes", and "the release team decided to include these fixes in PostgreSQL 19 before its general availability". RC2 shipped with those minor releases on 12 November, and its announcement says it "also includes the fixes in this update release that apply to PostgreSQL 19". That update release fixed 52 security vulnerabilities and over 140 bugs. General availability follows a week later.</p>
<p>So the extra weeks are for fixes going into 19, not for features coming back. Some of the reverted features are aimed at 20. The <code>GROUP BY ALL</code> revert says "We'll revert and try again in v20", and Eisentraut writes "I'm confident that many of the reverted features will come back before too long", with a few "80%-ready features" already queued for 20. Nothing promises any particular one.</p>
<h2>What this means for your upgrade</h2><h3>Check code written against the betas</h3><p>If your team wrote migrations, reports or runbooks against a 19 beta, they may use syntax that no longer exists. The companion repo has a check script for eight patterns: SQL/PGQ, <code>FOR PORTION OF</code>, <code>MERGE</code>/<code>SPLIT PARTITION</code>, bare <code>GROUP BY ALL</code>, the online checksum functions, <code>pg_get_*_ddl()</code>, <code>pg_dumpall</code> non-text formats and <code>pg_restore --no-globals</code>. It matches per line and on purpose loosely, so read each hit. <code>GROUP BY ALL region, status</code> has been valid since Postgres 14 (<code>ALL</code> is the default set quantifier) and is not flagged, but a valid <code>GROUP BY ALL</code> whose columns start on the next line is. A clean result means no pattern matched, not that the upgrade is safe. Here it is on the repo's example folders:</p>
<p><strong>check-pulled-syntax.sh</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ scripts/check-pulled-syntax.sh examples/beta-era
FOR PORTION OF             examples/beta-era/migrations/20260901_orders.sql:11:UPDATE prices FOR PORTION OF valid_at FROM <span class="hljs-string">'2026-10-01'</span> TO <span class="hljs-string">'2026-11-01'</span>
MERGE/SPLIT                examples/beta-era/migrations/20260901_orders.sql:2:ALTER TABLE orders MERGE PARTITIONS (orders_2026_07, orders_2026_08) INTO orders_2026_q3;
GROUP BY ALL               examples/beta-era/migrations/20260901_orders.sql:8:SELECT region, status, count(*) FROM orders GROUP BY ALL
online checksums           examples/beta-era/ops/backup.sh:5:psql -c <span class="hljs-string">"SELECT pg_enable_data_checksums()"</span>
pg_get_*_ddl()             examples/beta-era/ops/backup.sh:4:psql -Atc <span class="hljs-string">"SELECT pg_get_role_ddl('app')"</span> &gt; /backups/app-role.sql
pg_dumpall formats         examples/beta-era/ops/backup.sh:3:pg_dumpall --format=directory -f /backups/all-$(<span class="hljs-built_in">date</span> +%F)
pg_restore globals         examples/beta-era/ops/backup.sh:6:pg_restore --no-globals -d postgres /backups/all-2026-09-01
$ <span class="hljs-built_in">echo</span> <span class="hljs-string">"exit $?"</span>
<span class="hljs-built_in">exit</span> 1
$ scripts/check-pulled-syntax.sh examples/postgres-18
no pulled PostgreSQL 19 pattern matched under examples/postgres-18
$ <span class="hljs-built_in">echo</span> <span class="hljs-string">"exit $?"</span>
<span class="hljs-built_in">exit</span> 0
</code></pre><p>One reverted change is not syntax, so the script cannot find it. On the betas and RC1, <code>JSON_ARRAY()</code> and <code>JSON_OBJECT()</code> over arrays, composite types, ranges and multiranges counted as immutable, so you could use them in expression indexes, generated columns and partition keys. The revert on 6 November (<code>4b80cba35a</code>) leaves objects created that way in place, but its message warns that "a dump and restore or a pg_upgrade will now fail to recreate them. They need to be dropped or redefined before upgrading." If a cluster ran a beta or RC1 and you plan to carry it forward, look for those objects first.</p>
<p>The script exits 1 on a hit and 2 if the scan itself fails, so you can run it in CI against the repository that holds your migrations:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># Fails the job if a pulled-syntax pattern matches. Pinned to a commit, not main;</span>
<span class="hljs-comment"># or copy the script into your own repo.</span>
curl -fsSL -o check-pulled-syntax.sh \
  https://raw.githubusercontent.com/The-DevOps-Daily/pg19-revert-ledger/2588e0660651b4b6fd132a85699d12c35d16be62/scripts/check-pulled-syntax.sh
bash check-pulled-syntax.sh db/migrations
</code></pre><h3>Plan REPACK CONCURRENTLY one table at a time</h3><p><code>REPACK (CONCURRENTLY)</code> is one of the headline features of 19, and it stayed. But the change it relied on to build logical decoding snapshots per database was the one reverted in May. The revert message says what that means: it "restricts REPACK (CONCURRENTLY) so that only one process can run it concurrently on different tables and even on different databases". The original REPACK commit lists the same limit as a loose end, and Robert Haas's August post describes 19's REPACK CONCURRENTLY as "restricted to one process cluster-wide".</p>
<p>The <code>REPACK</code> reference page lists <code>max_repack_replication_slots</code> (default 5) as a limit, but not this one, and raising the slot setting does not lift it. If your plan was to run REPACK CONCURRENTLY across several bloated tables in parallel during one maintenance window, plan a queue instead. In the source, the decoding worker waits for every transaction that already has a transaction ID before it starts, so a second run most likely waits for the first. We did not run two at once, so treat that as a reading of the code, not a measurement.</p>
<h3>Plan the upgrade window</h3><ul>
<li><strong>Test on RC2 now.</strong> It came out on 12 November with the minor-release fixes, and the announcement says it "will be mostly identical to the initial release of PostgreSQL 19". A few more fixes have landed on the branch since RC2 and will be in 19.0.</li>
<li><strong>Rehearse the upgrade on a copy of your data.</strong> The pulled features are easy to find with the script; the changes that did ship are where an upgrade surprises you. If you run on Neon, a branch gives you a copy-on-write copy of production in seconds, so you can clean up beta-era migrations and test the queries the script flags against real data without touching the primary. A major version upgrade on Neon is a move to a new project on the new version, and Neon's <a href="https://neon.com/docs/postgresql/postgres-upgrade" rel="noopener noreferrer">major version upgrade guide</a> covers the Import Data Assistant (for databases under 10 GB), dump and restore, and logical replication. Check which major versions Neon offers before you pick a date.</li>
<li><strong>General availability is planned for 19 November.</strong> The next scheduled minor release after that is 11 February 2027, per the project's roadmap. If your policy is to wait for the first minor release, that is when 19.1 is due, unless an out-of-cycle release comes first.</li>
<li><strong>Do not plan on a pulled feature returning in a 19.x minor release.</strong> Minor releases carry fixes, not new features, so the earliest any of them can return is 20.</li>
</ul>
<h2>Summary</h2><ul>
<li>15 release-note items, covering 12 features, left the PostgreSQL 19 release notes after beta 1. Eleven of the features were reverted after beta 1, one in May. The big ones are SQL/PGQ, FOR PORTION OF, online data checksums and MERGE/SPLIT PARTITION.</li>
<li>The reasons differ by feature. In the threads, design questions with no agreed answer drove the big reverts. By Eisentraut's account, a long tail of edge-case bugs found faster with LLM-assisted review filled the queue, four months of security work took reviewers away during beta, and the need for every feature to work with every other one makes some of them very hard to finish.</li>
<li>The commit log credits AI in 49 of 1,232 commits after beta 1, 24 of them in the 9 November security batch. That is a floor, and in the threads people used the tools and people made the decisions.</li>
<li>The 19 November date is about folding the 12 November minor-release fixes into 19, not about features coming back.</li>
<li>Before you upgrade, check beta-era code with the script, redefine beta-era indexes, generated columns and partition keys built on <code>JSON_ARRAY()</code> or <code>JSON_OBJECT()</code> over container types, and plan REPACK CONCURRENTLY as one run at a time across the cluster.</li>
</ul>
<p>Sources: Peter Eisentraut, <a href="https://peter.eisentraut.org/blog/2026/10/09/the-reverts-will-continue-until-morale-improves" rel="noopener noreferrer">The reverts will continue until morale improves</a>; the <a href="https://www.postgresql.org/about/news/postgresql-19-rc-1-released-3409/" rel="noopener noreferrer">PostgreSQL 19 RC 1 announcement</a> and the <a href="https://www.postgresql.org/about/news/postgresql-187-1712-1616-1520-1425-and-19-rc-2-released-3442/" rel="noopener noreferrer">RC 2 announcement</a>; Robert Haas, <a href="https://www.postgresql.org/message-id/CA%2BTgmob9NY6m0YNFTQ4nFH2d0iC9SQRruDYxfndGKKzh8OC80w%40mail.gmail.com" rel="noopener noreferrer">scary patch contest</a> and <a href="https://www.postgresql.org/message-id/646368.1787697184%40sss.pgh.pa.us" rel="noopener noreferrer">Tom Lane's reply</a>; the <a href="https://www.postgresql.org/message-id/CAAKRu_bEtjWYWhYxSo0o_t3DaZYRQd5PkC0abA0g9Mp4%2BovH0w%40mail.gmail.com" rel="noopener noreferrer">SQL/PGQ design issues</a>; the <a href="https://www.postgresql.org/message-id/flat/62bf70cc-dc33-4188-8070-a8c177bb61de%40eisentraut.org" rel="noopener noreferrer">FOR PORTION OF thread</a>; <a href="https://www.postgresql.org/message-id/20260827015242.54.noahmisch%40microsoft.com" rel="noopener noreferrer">pg_get_*_ddl() needs a redesign</a>; <a href="https://www.postgresql.org/message-id/CAN4CZFNCU%3Dt09M%3D%2Br2t9hHLJuujdM4oQ8hCK_Sx-GpfiwMAicw%40mail.gmail.com" rel="noopener noreferrer">MERGE/SPLIT PARTITIONS issues</a>; and <a href="https://www.theregister.com/databases/2026/10/16/postgres-19-delayed-after-ai-reviewers-find-more-bugs-than-humans-can-fix/5303471" rel="noopener noreferrer">The Register's report</a>.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Let's Encrypt Goes to 64-Day Certificates: When Does Your ACME Client Actually Renew?]]></title>
      <link>https://devops-daily.com/posts/lets-encrypt-64-day-certificates-renewal-check</link>
      <description><![CDATA[Let's Encrypt moves to 64-day certificates on 10 February 2027, and its staging environment already issues them. We issued real 64-day staging certificates with certbot, acme.sh and lego and recorded when each one renews. The ARI clients renewed or scheduled their renewal around day 42. acme.sh 3.0.7 with --days 80 scheduled its renewal for day 79, 15 days after the certificate expired, and upgrading to 3.1.6 fixed that at the next check.]]></description>
      <pubDate>Thu, 15 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/lets-encrypt-64-day-certificates-renewal-check</guid>
      <category><![CDATA[Networking]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Networking]]></category><category><![CDATA[TLS]]></category><category><![CDATA[Let's Encrypt]]></category><category><![CDATA[certbot]]></category><category><![CDATA[acme.sh]]></category><category><![CDATA[Linux]]></category>
      <content:encoded><![CDATA[<p>Let's Encrypt will issue <strong>64-day certificates by default from 10 February 2027</strong>, down from 90, on the way to 45 days in 2028. Its staging environment switched on 14 October 2026, and we had a watcher waiting for it: the first 64-day certificate arrived at 21:35 UTC. We issued 64-day staging certificates with five ACME client installs and recorded when each one renews, and ran the same test at 45 and 90 days for comparison. The clients that use ARI renewed or scheduled a 64-day certificate for <strong>day 42 or 43</strong>, at the same two-thirds share of the lifetime as at 45 and 90 days. The one that breaks is old acme.sh: version 3.0.7 scheduled a 64-day certificate for day 59 with its defaults, five days before it expires, and for <strong>day 79 with <code>--days 80</code>, fifteen days after it expired</strong>. Upgrading to 3.1.6 moved that schedule to day 42 at the next check.</p>
<h2>TL;DR</h2><ul>
<li>On 64-day staging certificates, Let's Encrypt's ARI endpoint suggested renewing between <strong>day 42.04 and 43.32</strong>, 65.7 to 67.7 percent of the lifetime. We measured the same share at 45 and 90 days. That is about 21 days before expiry.</li>
<li>certbot 5.8 renewed inside that window (it picks a random time in it on every run), acme.sh 3.1.6 scheduled day 42.16, and lego 5.5.2 allowed renewal from day 42.67. These configurations are ready for shorter lifetimes.</li>
<li>certbot 2.9, the version apt installed on our Ubuntu 24.04 machine, does not use ARI and became due 30 days before expiry: day 34 of 64. Early, but safe.</li>
<li>acme.sh 3.0.7 schedules the next renewal at issue + <code>--days</code> - 1 day, with no check against the expiry date. Its default of 60 gave day 59.04 of 64, and <code>--days 80</code> gave day 79.04. Every acme.sh tag we scanned from 3.0.7 to 3.1.2 has the same default of 60.</li>
<li>Search your cron jobs, wrapper scripts and monitoring for hardcoded 60, 80 and 83, as Let's Encrypt itself suggests, and still check your renewal timer, challenge path and deploy hook.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A server that renews certificates with certbot, acme.sh or lego</li>
<li>Shell access to check the client version and its stored renewal settings</li>
<li>To reproduce the test: a throwaway VM with port 80 reachable, and <code>faketime</code></li>
</ul>
<p>Everything we ran, and every output file, is in the repository:</p>
<p><a href="https://github.com/The-DevOps-Daily/le-64-day-renewal-check" rel="noopener noreferrer">The-DevOps-Daily/le-64-day-renewal-check on GitHub</a></p>
<h2>What changes, and when</h2><p>From Let's Encrypt's <a href="https://letsencrypt.org/2026/10/07/64-day-certs" rel="noopener noreferrer">announcement</a> of 7 October 2026:</p>
<ul>
<li><strong>14 October 2026</strong>: staging switches to 64-day certificates. Our watcher's request at 21:25 UTC still returned a 90-day certificate, and the one at 21:35 UTC returned a 64-day certificate.</li>
<li><strong>10 February 2027</strong>: all subscribers move to 64-day certificates by default. The last 90-day certificates expire around 11 May 2027.</li>
<li>The <strong>authorization reuse period</strong> drops from 30 days to 10 days, and to seven hours in 2028.</li>
<li><strong>2028</strong>: 45-day certificates by default.</li>
</ul>
<p>The advice in the announcement is short: if your renewals are automated and your client supports ARI, you should be all set; if not, renew at about two thirds of the lifetime, and look for hardcoded values such as 83, 80 or 60 in cron jobs, wrapper scripts and runbooks. We wanted to see which clients actually follow that, and what happens to the ones that do not.</p>
<h2>How we measured it</h2><p>On a throwaway Ubuntu 24.04 droplet we installed five client versions side by side: certbot 2.9.0 from apt, certbot 5.8.0 from snap, acme.sh 3.0.7 and 3.1.6 from the upstream repository, and lego 5.5.2. They requested real certificates from the Let's Encrypt staging environment, using hostnames under <code>sslip.io</code> that resolved to the droplet, at three lifetimes. Where a client could not request a lifetime, we used copied state or its source:</p>
<ul>
<li><strong>64-day certificates</strong> from the default <code>classic</code> profile, after staging switched. A watcher asked for a classic certificate every 10 minutes from 12:36 UTC and started the suite as soon as the lifetime dropped.</li>
<li><strong>90-day certificates</strong> from the same <code>classic</code> profile, earlier that evening.</li>
<li><strong>45-day certificates</strong> from the <code>tlsserver</code> profile, the lifetime planned for 2028. acme.sh 3.0.7 and certbot 2.9 cannot request a profile, so certbot 2.9 was tested on a copy of certbot 5.8's 45-day certificate and acme.sh 3.0.7 was read from its source.</li>
</ul>
<p>Then, for each certificate:</p>
<ul>
<li>acme.sh stores its planned renewal time in its config, so we read it.</li>
<li>lego prints, at issuance, when a renewal will be allowed; we ran it with ARI on and with <code>--ari-disable</code>.</li>
<li>certbot only decides when <code>certbot renew</code> runs, so we ran it under <code>faketime</code> at chosen hours after issuance, each time on a fresh copy of the state, and recorded whether it renewed.</li>
<li>A small script asked the ARI endpoint (<a href="https://www.rfc-editor.org/rfc/rfc9773" rel="noopener noreferrer">RFC 9773</a>) for the suggested window of every certificate.</li>
</ul>
<p>Days are counted from each certificate's <code>notBefore</code> time, which Let's Encrypt sets about an hour before issuance. This is the certbot 2.9 probe on the 64-day certificate, as recorded:</p>
<p><strong>certbot 2.9.0 under faketime, 64-day staging certificate</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ probe-certbot.sh certbot <span class="hljs-variable">$R</span>/cb29 c29n.46-101-102-127.sslip.io 800 812 815 818 830
<span class="hljs-comment"># certbot 2.9.0 c29n.46-101-102-127.sslip.io notBefore=Oct 14 20:37:00 2026 GMT notAfter=Dec 17 20:36:59 2026 GMT</span>
hour 800 (day 33.33, 2026-11-17 04:37:00 UTC): not due
hour 812 (day 33.83, 2026-11-17 16:37:00 UTC): not due
hour 815 (day 33.96, 2026-11-17 19:37:00 UTC): not due
hour 818 (day 34.08, 2026-11-17 22:37:00 UTC): RENEWED
hour 830 (day 34.58, 2026-11-18 10:37:00 UTC): RENEWED
</code></pre><p>Not due at day 33.96, renewed at day 34.08: the threshold is between the two, consistent with 30 days before the 64-day expiry.</p>
<h2>Results</h2><table>
<thead>
<tr>
<th>Client</th>
<th>64-day</th>
<th>45-day</th>
<th>90-day</th>
</tr>
</thead>
<tbody><tr>
<td>ARI suggested window</td>
<td>day 42.04 to 43.32</td>
<td>day 29.56 to 30.46</td>
<td>day 59.11 to 60.91</td>
</tr>
<tr>
<td>certbot 5.8.0 (snap), ARI</td>
<td>first renewal seen at day 42.50</td>
<td>first renewal seen at day 30.29</td>
<td>first renewal seen at day 59.92</td>
</tr>
<tr>
<td>acme.sh 3.1.6, ARI</td>
<td>scheduled day 42.16</td>
<td>scheduled day 29.6 to 30.3</td>
<td>scheduled day 60.7 to 60.9</td>
</tr>
<tr>
<td>lego 5.5.2</td>
<td>allowed from day 42.67</td>
<td>allowed from day 30.0</td>
<td>allowed from day 60.0</td>
</tr>
<tr>
<td>certbot 2.9.0 (apt), no ARI</td>
<td>due between day 33.96 and 34.08</td>
<td>due between day 14.96 and 15.08</td>
<td>due between day 59.79 and 60.21</td>
</tr>
<tr>
<td>acme.sh 3.0.7, default <code>--days 60</code></td>
<td>scheduled day 59.04, 5 days before expiry</td>
<td>day 59, after expiry (source)</td>
<td>scheduled day 59.0</td>
</tr>
<tr>
<td>acme.sh 3.0.7, <code>--days 80</code></td>
<td>scheduled day 79.04, <strong>15 days after expiry</strong></td>
<td>day 79, after expiry (source)</td>
<td>scheduled day 79.0</td>
</tr>
</tbody></table>
<p><strong>Day each client renews, or plans to renew, a 64-day certificate</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>certbot 2.9 (apt)</td>
<td>34 days</td>
<td>safe</td>
</tr>
<tr>
<td>certbot 5.8 (ARI)</td>
<td>42.5 days</td>
<td>safe</td>
</tr>
<tr>
<td>acme.sh 3.1.6 (ARI)</td>
<td>42.16 days</td>
<td>safe</td>
</tr>
<tr>
<td>lego 5.5.2</td>
<td>42.67 days</td>
<td>safe</td>
</tr>
<tr>
<td>acme.sh 3.0.7, defaults</td>
<td>59.04 days</td>
<td>tight</td>
</tr>
<tr>
<td>acme.sh 3.0.7, --days 80</td>
<td>79.04 days</td>
<td>expired</td>
</tr>
<tr>
<td>same, after upgrading to 3.1.6</td>
<td>42.23 days</td>
<td>safe</td>
</tr>
</tbody></table>
<p><em>The certificate expires on day 64. Measured on Let's Encrypt staging on 2026-10-14, one certificate per row. certbot 2.9 became due between day 33.96 and 34.08. certbot 5.8 is its first renewal in our probes; it picks a random time inside the ARI window (day 42.04 to 43.32) on every run. acme.sh shows its stored schedule and lego the day it reported at issuance. The last bar is the --days 80 certificate after upgrading acme.sh to 3.1.6. Source: results/measured-64.txt in the repo.</em></p>
<h3>ARI clients: the window does the work</h3><p>The ARI window opened at 65.7 percent of the lifetime and closed at 67.7 percent at all three lengths: day 42.04 to 43.32 of 64, day 29.56 to 30.46 of 45, and day 59.11 to 60.91 of 90. On a 64-day certificate, that is 21 to 22 days before expiry. A client that follows the window needs no change for 45 days in 2028 either.</p>
<p>While we waited, we projected the 64-day column from the 45-day and 90-day runs and committed the projection at 21:32 UTC (commit <code>f909cfc</code> in the repository). Our next request, at 21:35 UTC, returned the first 64-day certificate. Every 64-day result in the table matches that projection (<code>results/projection-64.txt</code> and <code>results/measured-64.txt</code>).</p>
<p>certbot 5.8 does not renew at one fixed moment. Each time it runs, it asks for the window, picks a random time inside it, and renews if that time has passed. From its source (<code>acme/client.py</code>):</p>
<pre><code class="hljs language-python" tabindex="0">delta_seconds = (end - start).total_seconds()
random_seconds = random.uniform(<span class="hljs-number">0</span>, delta_seconds)
random_time = start + datetime.timedelta(seconds=random_seconds)
</code></pre><p>So a run a third of the way into the window renews about a third of the time. These are our 64-day probes, each from a fresh copy of the same state:</p>
<pre><code class="hljs language-text" tabindex="0">hour 1010 (day 42.08, 2026-11-25 22:37:12 UTC): not due
hour 1014 (day 42.25, 2026-11-26 02:37:12 UTC): not due
hour 1020 (day 42.50, 2026-11-26 08:37:12 UTC): RENEWED
hour 1030 (day 42.92, 2026-11-26 18:37:12 UTC): not due
</code></pre><p>The run at day 42.50 had about a 36 percent chance and renewed. The run at day 42.92 had about 68 percent and did not. This spreads renewals out, as RFC 9773 intends, so a fleet does not renew all at once. Once the window has closed, every run renews: on the 45-day and 90-day certificates, every probe after the window renewed. If the ARI request fails, certbot 5.8 catches the error and gets no ARI time; it then renews at two thirds of the lifetime (day 42.67 of 64), or by <code>renew_before_expiry</code> if that is set in the renewal config. We read this in its source (<code>results/certbot-5.8-ari-source.txt</code>) but did not test a failed ARI request.</p>
<p>acme.sh 3.1.6 scheduled its next renewal for day 42.16, with its default and with <code>--days 80</code>. It also stayed inside the window with <code>--days 60</code> at 45 days and with <code>--days 80</code> at 90 days. lego 5.5.2, checked at issuance, gave the same answer with ARI on and off: renewal allowed from day 42.67, exactly two thirds of the lifetime.</p>
<p>These are the clients' decisions, not your whole setup. A renewal still needs a timer or cron job that runs often enough, a working challenge path and a deploy hook that reloads the server.</p>
<h3>certbot 2.9: early, and fine</h3><p>certbot added ARI support in <a href="https://community.letsencrypt.org/t/certbot-4-1-0-release/238369" rel="noopener noreferrer">version 4.1.0</a> (June 2025). The certbot that apt installed on our Ubuntu 24.04 machine was 2.9.0, and its probes became due 30 days before expiry at every lifetime: between day 33.96 and 34.08 of 64, between day 14.96 and 15.08 of 45, and between day 59.79 and 60.21 of 90. It renews more often than it needs to, and it leaves about 30 days to notice and fix a failed renewal, if its timer runs normally.</p>
<h3>acme.sh 3.0.7: the one that breaks</h3><p>acme.sh 3.0.7 computes the next renewal when it issues a certificate, and never compares it with the expiry date. From its source:</p>
<pre><code class="hljs language-bash" tabindex="0">Le_NextRenewTime=$(_math <span class="hljs-string">"<span class="hljs-variable">$Le_CertCreateTime</span>"</span> + <span class="hljs-string">"<span class="hljs-variable">$Le_RenewalDays</span>"</span> \* 24 \* 60 \* 60)
Le_NextRenewTime=$(_math <span class="hljs-string">"<span class="hljs-variable">$Le_NextRenewTime</span>"</span> - 86400)
</code></pre><p><code>Le_RenewalDays</code> is the <code>--days</code> value, 60 by default. The 64-day staging run shows the result:</p>
<pre><code class="hljs language-text" tabindex="0">acme307 a307n notBefore=Oct 14 20:37:26 2026 GMT notAfter=Dec 17 20:37:25 2026 GMT next='2026-12-12T21:36:01Z' days=
acme307 a307n80 notBefore=Oct 14 20:37:43 2026 GMT notAfter=Dec 17 20:37:42 2026 GMT next='2027-01-01T21:36:18Z' days='80'
</code></pre><p>Both certificates expire on 17 December. With the default, the next renewal is 12 December, day 59.04, so one failed renewal (a DNS hiccup, a firewall change, a rate limit) leaves very little time before expiry. With <code>--days 80</code>, the next renewal is 1 January 2027, day 79.04, fifteen days after the certificate expired, and acme.sh 3.0.7 does not try before then. The 90-day run gave the same days, 59 and 79, which were safe at that lifetime. At 45 days in 2028, the default fails too.</p>
<p>We scanned each acme.sh tag from 3.0.7 to 3.1.6 for the scheduling code (the scan is in the repo):</p>
<table>
<thead>
<tr>
<th>acme.sh tag</th>
<th>Tag date</th>
<th>Default <code>--days</code></th>
<th>Cap one day before expiry</th>
<th>ARI code</th>
</tr>
</thead>
<tbody><tr>
<td>3.0.7 to 3.1.2</td>
<td>Oct 2023 to Nov 2025</td>
<td>60</td>
<td>no</td>
<td>none</td>
</tr>
<tr>
<td>3.1.3</td>
<td>Apr 2026</td>
<td>30</td>
<td>no</td>
<td>first ARI function</td>
</tr>
<tr>
<td>3.1.4</td>
<td>Jul 2026</td>
<td>30</td>
<td>yes</td>
<td>ARI used when scheduling</td>
</tr>
<tr>
<td>3.1.5, 3.1.6</td>
<td>Sep 2026</td>
<td>30</td>
<td>yes</td>
<td>ARI also checked on every renewal run</td>
</tr>
</tbody></table>
<p>We only ran 3.0.7 and 3.1.6. The rows for the versions in between come from reading the source, so check your own version's behaviour if it is one of those.</p>
<h3>What upgrading does to an existing certificate</h3><p>acme.sh stores <code>Le_RenewalDays</code> and the next renewal time in each certificate's config, so an upgrade does not by itself change a schedule written by an old version. We tested what happens next: we copied the state of the 64-day certificate that 3.0.7 had scheduled for day 79 and ran acme.sh 3.1.6 <code>--renew</code> on it, without <code>--force</code>:</p>
<pre><code class="hljs language-text" tabindex="0">== before: acme.sh 3.0.7 state
Le_RenewalDays='80'
Le_NextRenewTimeStr='2027-01-01T21:36:18Z'
notAfter=Dec 17 20:37:42 2026 GMT
== acme.sh 3.1.6 --renew on the same state (no --force), 2026-10-14T23:22:34Z
[Wed Oct 14 23:22:34 UTC 2026] Renew: a307n80.46-101-102-127.sslip.io
[Wed Oct 14 23:22:34 UTC 2026] Renewing: 'a307n80.46-101-102-127.sslip.io'
[Wed Oct 14 23:22:34 UTC 2026] Renewing using Le_API=https://acme-staging-v02.api.letsencrypt.org/directory
[Wed Oct 14 23:22:35 UTC 2026] Current renewal time: 2027-01-01T21:36:18Z
[Wed Oct 14 23:22:35 UTC 2026] ARI suggestedWindow: 2026-11-25T21:36:11Z to 2026-11-27T04:18:13Z
[Wed Oct 14 23:22:35 UTC 2026] Updating renewal time picked from ARI window: 2026-11-26T02:10:18Z
[Wed Oct 14 23:22:35 UTC 2026] Skipping. Next renewal time is: 2026-11-26T02:10:18Z
== after
Le_RenewalDays='80'
Le_NextRenewTimeStr='2026-11-26T02:10:18Z'
</code></pre><p>The stored <code>--days 80</code> stays, but 3.1.6 asked the ARI endpoint on its first check and moved the renewal from 1 January, after expiry, to 26 November, day 42.23. The same test on the 90-day certificate gave the same result. Its source says it does this on every renewal check unless <code>NO_ARI=1</code> is set. On a CA without ARI, or with <code>NO_ARI=1</code>, the stored schedule is what counts.</p>
<h2>What to check on your servers</h2><p>Find out which client and version renews each certificate, and what it has stored:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># certbot: version, and any renewal setting that overrides the default</span>
certbot --version
grep -H renew_before_expiry /etc/letsencrypt/renewal/*.conf

<span class="hljs-comment"># acme.sh: version, the stored --days, and the stored next renewal time</span>
~/.acme.sh/acme.sh --version
grep -H -E <span class="hljs-string">'^Le_RenewalDays=|^Le_NextRenewTimeStr='</span> ~/.acme.sh/*/*.conf

<span class="hljs-comment"># lego: version (ARI is on unless you pass --ari-disable)</span>
lego --version
</code></pre><p>Then:</p>
<ol>
<li><strong>Upgrade acme.sh</strong> to 3.1.6 or later wherever it runs (<code>acme.sh --upgrade</code>), and run <code>acme.sh --cron</code> once to see the new schedule. On anything that has to stay on an older version, compare each stored <code>Le_NextRenewTimeStr</code> with the certificate's expiry, and lower <code>--days</code> so the next renewal comes well before it.</li>
<li><strong>Look for 60, 80 and 83</strong> in cron jobs, systemd timers, wrapper scripts and runbooks, as Let's Encrypt suggests. A script that renews "when the certificate is 60 days old" leaves four days of margin at 64 days and none at 45.</li>
<li><strong>Check your expiry alerts.</strong> The ARI clients chose renewal times with about 21 days left on our 64-day certificates, so an alert at 30 days left fires on every certificate while renewal is working fine, and people learn to ignore it. Alert on failed renewals, and on expiry with a threshold below the normal renewal point, for example 14 days left.</li>
<li><strong>Check what else issues certificates.</strong> The shorter authorization reuse period (10 days instead of 30) does not change a renewal every 42 days, since both periods have passed by then. It matters when you issue again 10 to 30 days after the last validation, for example a manual reissue or a renewal forced early: that order now needs a fresh challenge.</li>
<li><strong>Rehearse on staging before February.</strong> Staging's <code>classic</code> profile already issues 64-day certificates, and its <code>tlsserver</code> profile issues 45-day ones. Issue a staging certificate with separate state from production and run your normal renewal and deploy path against it.</li>
</ol>
<h2>What this test does not show</h2><ul>
<li><strong>One certificate per configuration.</strong> We issued one 64-day certificate for each client setup, on one evening. certbot 5.8 picks a random time inside the window, so our probes show that it renews inside the window, not how its renewals spread across a fleet.</li>
<li><strong>ARI windows can move.</strong> Let's Encrypt can change a certificate's window, for example to ask for early renewal before a mass revocation. Clients that check ARI on each run, as certbot 5.8 and acme.sh 3.1.6 do, pick up the change; we did not test that. We measured the normal window. The acme.sh 3.0.7 and certbot 2.9 rows would not change, because neither looks at ARI.</li>
<li><strong>acme.sh 3.0.7 at 45 days comes from its source.</strong> It cannot request the 45-day profile, so we could not issue one with it. The 64-day and 90-day runs confirm the formula.</li>
<li><strong>lego was checked at issuance only.</strong> We recorded when it said a renewal would be allowed, not a renewal at that time.</li>
<li><strong>Five client installs.</strong> We did not test Caddy, Traefik, cert-manager, win-acme or hosting panels that embed their own ACME code. Check those against the same two questions: does it use ARI, and if not, how does it decide?</li>
<li><strong>Staging, not production.</strong> Production should behave the same way from February, but we only issued staging certificates.</li>
</ul>
<p>The short version: in our tests, the clients with ARI enabled renewed on Let's Encrypt's window, and that window was the same share of the lifetime at 45, 64 and 90 days. If it does not, find out what number it renews on, because that number was chosen for 90 days.</p>
<p>For an expiry check that tests what your servers actually present, with a threshold set relative to the certificate's lifetime, read <a href="https://devops-daily.com/guides/cryptography-essentials/03-tls-ssl#h2-monitor-certificate-expiry">Monitor Certificate Expiry</a> in our Cryptography Essentials guide.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage?]]></title>
      <link>https://devops-daily.com/posts/cheap-model-alert-triage-digitalocean-inference</link>
      <description><![CDATA[We gave six models on DigitalOcean Serverless Inference one written on-call policy and 200 alerts, three times each. A model at $0.16 per 1,000 alerts made every call the same way as one at $1.64. The real differences were latency, missed pages and a bug in our own test data that the models caught.]]></description>
      <pubDate>Wed, 14 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/cheap-model-alert-triage-digitalocean-inference</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[AI]]></category><category><![CDATA[LLM]]></category><category><![CDATA[Alerting]]></category><category><![CDATA[On-call]]></category><category><![CDATA[DigitalOcean]]></category>
      <content:encoded><![CDATA[<p>We gave six models on DigitalOcean Serverless Inference the same written on-call policy and the same 200 alerts, three times each, and asked one question per alert: page, ticket or ignore. DeepSeek V4 Flash made the policy's call on all 600 for an estimated <strong>$0.16 per 1,000 alerts</strong>. Qwen3.8-Max, the most expensive model we tested, also got all 600, for <strong>$1.64 per 1,000</strong>. Routing on the alert's own <code>severity</code> label, with no model at all, got 102 of 200, but we built that label to mislead.</p>
<p>So on this test the cheap model was good enough. That is not the whole answer. DeepSeek V4 Flash took 4.4 seconds at the median and once took a full minute. Two other cheap models each let through alerts the policy says must page. And 200 clean results still leave room for a real error rate of 1.5 percent. The most useful thing we learned came from a bug in our own test data: most of the models found it, and our second reviewer did not.</p>
<h2>TL;DR</h2><ul>
<li>With a clear written policy, DeepSeek V4 Flash and Qwen3.8-Max made the policy's call on all 600 alert runs. <code>gpt-oss-120b</code> got 599 (one empty reply). Qwen3.8-Max costs about ten times as much per alert as DeepSeek V4 Flash, and it answered about three times faster.</li>
<li>Llama 4 Maverick failed to page on 2 different page-level alerts (3 times in all) and <code>gpt-oss-20b</code> on 3 (once each). Ministral 3 14B missed 4.7 page-level alerts per run out of 50, and 17 of its 600 replies contained two different decisions.</li>
<li>DeepSeek V4 Flash spends most of its output on reasoning: p95 latency of 10.7 seconds, 33 calls over 10 seconds and one at 60.6 seconds.</li>
<li>At 10,000 alerts a month, the token cost is about $1.57 for the cheap model and $16.40 for the expensive one. Choose on missed pages and latency, not on price.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A DigitalOcean account with a Serverless Inference model access key</li>
<li>Node.js 20 or later (the scripts have no dependencies)</li>
<li>About $1.50 of inference if you rerun all six models</li>
</ul>
<p>The code, the alerts, every recorded decision and reply, and the scoring are in the repository:</p>
<p><a href="https://github.com/The-DevOps-Daily/alert-triage-bench" rel="noopener noreferrer">The-DevOps-Daily/alert-triage-bench on GitHub</a></p>
<h2>The task: one policy, three answers</h2><p>The simplest way to route alerts is the <code>severity</code> label. Whoever wrote the alert rule decided that <code>KubePodCrashLooping</code> is a warning and <code>EndpointDown</code> is critical, and the route pages on critical. That works until the label and the situation disagree: a critical alert from a sandbox, a warning on a checkout outage, a probe that still fires for a service you turned off last month.</p>
<p>The question here is narrower than "can a model run on-call". Give a model the alert and a written policy, and ask it to make the one call a human makes first. The policy is short on purpose, the kind a team could actually keep up to date. In summary:</p>
<pre><code class="hljs language-text" tabindex="0">Page    production, and customer impact now or within 1 hour:
        a tier 1 service down, &gt;2% errors for 5+ min or p99 &gt;2 s for 10+ min,
        a tier 1 SLO burning at 14x or faster,
        a database, cache or queue full within 1 hour,
        a public cert expiring in under 24 h, the primary DB unreachable,
        confirmed unauthorized access
Ticket  a real problem with no customer impact within the hour:
        tier 2 problems, tier 1 problems below the page thresholds,
        capacity gone in more than 1 hour but less than 7 days,
        certs expiring in 1 to 14 days,
        one pod or node failing while the service is fine,
        failed backups and jobs, staging problems that block a release
Ignore  non-production (unless it blocks a release),
        alerts that resolved within 5 minutes with no customer impact,
        heartbeats, deploy and autoscaling notices,
        maintenance windows and decommissioned or test targets
        (these two win even over a page)
        The severity label is a hint only.
</code></pre><p>The full text, including the list of tier 1 services, is in <code>POLICY.md</code>. Every model got the whole policy as its system prompt and one alert as the user message, about 720 input tokens per call, and was told to reply with one JSON object, <code>{"decision": ..., "reason": ...}</code>, and nothing else.</p>
<p>A typical hard case looks like this. The service is tier 1 and the restart count is alarming, but one pod out of eight is not an outage:</p>
<pre><code class="hljs language-text" tabindex="0">status: firing
labels: alertname=KubePodCrashLooping env=production service=search severity=warning team=discovery
summary: Pod search-ad1 is crash looping
description: 1 of 8 search pods has restarted 21 times in the last hour (OOMKilled). The other 7 replicas are healthy and the service error ratio is 0.16%.
startsAt: 2026-10-15T03:41:00Z
maintenance window: none
</code></pre><p>The answer is ticket. So is a staging outage that blocks a release, even though staging is otherwise ignored. A <code>severity=critical</code> alert showing 42 percent errors on <code>search</code> in staging, with no release waiting, is ignore.</p>
<h2>How the 200 alerts were built and checked</h2><p>The alerts are generated, not collected. Real alert payloads would not come with a known correct answer, and hand-labelling 200 of them would test our labelling as much as the models. Instead, a seeded script uses 31 scenario templates. Each template has the decision the policy gives it, and draws its numbers (service, error rate, minutes firing, hours to expiry) from ranges that keep that decision. The text the models see is rendered from the same fields. Writing templates by hand can put mistakes in the text or the label, and one of ours did, as you will see.</p>
<p><strong>alert-triage-bench</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/01-generate-alerts.mjs
alerts: 200
  page   50 (hard 14)
  ticket 84 (hard 40)
  ignore 66 (hard 54)
  severity label points the wrong way: 98
$ node scripts/03b-baseline-severity.mjs
severity-label baseline: 102 of 200 correct
</code></pre><p>Two numbers in that output need explaining. "Hard" marks alerts where a shortcut gives the wrong answer: the severity label points the wrong way, or the alert name sounds worse than the policy says it is. And 98 of the 200 severity labels are wrong <strong>on purpose</strong>. That makes the no-model baseline deliberately weak: it scores 51 percent and lets 14 of the 50 page-level alerts through. It measures whether a model reads the alert instead of the label. It says nothing about how a model compares with a well-written rules engine, which we did not test.</p>
<p>To check that the text carries what the labels assume, we gave one AI second reader, Codex, only <code>POLICY.md</code> and the 200 rendered alerts, never the labels, and asked it to label them. It agreed on all 200. Hold on to that number, because it turned out to be wrong in an interesting way.</p>
<h2>Results</h2><p>Six models, three runs each, 3,600 calls. All six are open-weight models. Our account's key returned <code>403 this model is not available for your subscription tier</code> for the GPT-5 and Claude models we tried, so they are not in this test (the check is saved in the repo). DigitalOcean's Inference Router is in public preview, and its preview terms do not allow publishing test results without DigitalOcean's written consent, so this post covers the direct models only.</p>
<table>
<thead>
<tr>
<th>Configuration</th>
<th>Correct (of 600)</th>
<th>Page-level alerts missed in any run (of 50)</th>
<th>False pages per run</th>
<th>$ per 1,000 alerts</th>
<th>p50</th>
<th>p95</th>
</tr>
</thead>
<tbody><tr>
<td>DeepSeek V4 Flash</td>
<td>600</td>
<td>0</td>
<td>0</td>
<td>0.157</td>
<td>4.4 s</td>
<td>10.7 s</td>
</tr>
<tr>
<td>Qwen3.8-Max</td>
<td>600</td>
<td>0</td>
<td>0</td>
<td>1.640</td>
<td>1.3 s</td>
<td>2.6 s</td>
</tr>
<tr>
<td><code>gpt-oss-120b</code></td>
<td>599</td>
<td>0</td>
<td>0</td>
<td>0.194</td>
<td>3.1 s</td>
<td>5.1 s</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>597</td>
<td>2</td>
<td>0</td>
<td>0.189</td>
<td>1.9 s</td>
<td>3.3 s</td>
</tr>
<tr>
<td><code>gpt-oss-20b</code></td>
<td>595</td>
<td>3</td>
<td>0</td>
<td>0.144</td>
<td>2.4 s</td>
<td>5.3 s</td>
</tr>
<tr>
<td>Ministral 3 14B</td>
<td>532</td>
<td>8</td>
<td>6.0</td>
<td>0.149</td>
<td>0.8 s</td>
<td>1.8 s</td>
</tr>
<tr>
<td>Severity label, no model</td>
<td>102 of 200</td>
<td>14</td>
<td>39</td>
<td>0</td>
<td></td>
<td></td>
</tr>
</tbody></table>
<p>Costs are estimates: recorded token usage times DigitalOcean's list prices on 2026-10-14.</p>
<p><strong>Wrong answers out of 600 (200 alerts, 3 runs)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>DeepSeek V4 Flash</td>
<td>0</td>
<td>direct</td>
</tr>
<tr>
<td>Qwen3.8-Max</td>
<td>0</td>
<td>direct</td>
</tr>
<tr>
<td>gpt-oss-120b</td>
<td>1</td>
<td>direct</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>3</td>
<td>direct</td>
</tr>
<tr>
<td>gpt-oss-20b</td>
<td>5</td>
<td>direct</td>
</tr>
</tbody></table>
<p><em>Lower is better. An empty reply counts as wrong. Ministral 3 14B got 68 of 600 wrong and is left out so the rest stay readable. Source: results/summary.json in the repo.</em></p>
<p><strong>Estimated cost per 1,000 alerts</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>gpt-oss-20b</td>
<td>0.144$</td>
<td>direct</td>
</tr>
<tr>
<td>Ministral 3 14B</td>
<td>0.149$</td>
<td>direct</td>
</tr>
<tr>
<td>DeepSeek V4 Flash</td>
<td>0.157$</td>
<td>direct</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>0.189$</td>
<td>direct</td>
</tr>
<tr>
<td>gpt-oss-120b</td>
<td>0.194$</td>
<td>direct</td>
</tr>
<tr>
<td>Qwen3.8-Max</td>
<td>1.64$</td>
<td>direct</td>
</tr>
</tbody></table>
<p><em>Recorded token usage times DigitalOcean list prices checked 2026-10-14.</em></p>
<p>Two models scored 600 of 600, three more got between one and five wrong, and one is clearly not good enough. Ministral 3 14B is the fastest and among the cheapest, but it got 42 of the 200 alerts wrong in at least one run, missed 4.7 page-level alerts per run and gave the same answer in all three runs for only 80 percent of the alerts. It also ignored the format: 593 of its 600 replies had text around the JSON, and 17 contained two different decisions, typically an answer followed by a "correction".</p>
<p>That raises a scoring question, so here is the rule: the scorer takes the first decision in the reply, because an automated router would act on the first answer it can parse, and an empty reply counts as wrong. The repo records the first 400 characters of each reply from this run. The runner now keeps the full text for future runs.</p>
<p>The misses from the near-perfect models are worth reading, because they are the errors that hurt. Llama 4 Maverick called a Postgres database that would run out of connections in about 34 minutes a ticket, twice. Once it answered ticket for a public certificate expiring in 16 hours, then wrote "Wait, the correct response should be based on the given policy" and started a second answer, which we score as the ticket it gave first. <code>gpt-oss-20b</code> let three different Redis memory alerts through, all in its second run. Each of those replies would have suppressed a page that the policy requires.</p>
<h2>Latency is where the cheap model pays</h2><p>DeepSeek V4 Flash matched Qwen3.8-Max's accuracy at about a tenth of the token cost. Latency goes the other way. It is a reasoning model, and 81 percent of its output tokens were reasoning, a median of 143 output tokens per call against Qwen3.8-Max's 33 tokens of plain JSON.</p>
<p><strong>Latency per alert: p50 bar, p95 marker</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Tick</th>
</tr>
</thead>
<tbody><tr>
<td>Ministral 3 14B</td>
<td>0.8s</td>
<td>1.8s</td>
</tr>
<tr>
<td>Qwen3.8-Max</td>
<td>1.3s</td>
<td>2.6s</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>1.9s</td>
<td>3.3s</td>
</tr>
<tr>
<td>gpt-oss-20b</td>
<td>2.4s</td>
<td>5.3s</td>
</tr>
<tr>
<td>gpt-oss-120b</td>
<td>3.1s</td>
<td>5.1s</td>
</tr>
<tr>
<td>DeepSeek V4 Flash</td>
<td>4.4s</td>
<td>10.7s</td>
</tr>
</tbody></table>
<p><em>Wall-clock time of the final attempt of each call, 600 calls per model; retries are not included (only Ministral 3 14B needed any, on 11 calls). The models ran one after another with 4 to 6 calls in flight, so service conditions may have differed between them.</em></p>
<p>DeepSeek V4 Flash had 33 calls over 10 seconds and one at 60.6 seconds. For a ticket, four seconds does not matter. For a page, a minute does. If you put a model in front of your pager, give each call a timeout and decide in advance what a timeout means. The safe default is to page: a model that is slow or down should never be the reason an outage goes unheard.</p>
<h2>The bug the models found</h2><p>In the first run, nobody scored 600. Qwen3.8-Max and <code>gpt-oss-120b</code> "got wrong" the same two alerts every time. This was one of them:</p>
<pre><code class="hljs language-text" tabindex="0">status: firing
labels: alertname=PostgresPrimaryUnreachable env=production service=postgres-orders severity=critical team=platform
summary: postgres-orders primary not responding
description: The primary of postgres-orders has not answered health checks for 7 minutes.
startsAt: 2026-10-23T05:35:00Z
maintenance window: active until 05:00 UTC, change CHG-4200 (major version upgrade of postgres-orders)
</code></pre><p>Our label said ignore, because the alert carries a maintenance window. But the window, read as ending at 05:00 that morning, ended before the alert started at 05:35. Our generator picked the start time and the window separately and never checked that one was inside the other. Here is what three models said about this alert in run 1, copied from the run logs:</p>
<pre><code class="hljs language-text" tabindex="0">qwen3.8-max:
{"decision": "page", "reason": "Production primary database unreachable for 7 minutes and the maintenance window ended at 05:00 UTC, so the page criteria apply."}

openai-gpt-oss-120b:
{"decision":"page","reason":"Production primary database is unreachable and no active maintenance window"}

llama-4-maverick:
{"decision": "ignore", "reason": "service is in an active maintenance window"}
</code></pre><p>Across the two broken alerts and three runs, Qwen3.8-Max and <code>gpt-oss-120b</code> paged all 6 times. Llama 4 Maverick and Ministral 3 14B trusted the word "active" all 6 times. DeepSeek V4 Flash paged three times, ignored twice and once used up its 2,000-token allowance on reasoning without giving a decision. <code>gpt-oss-20b</code> paged four times and ignored twice. The second reader that "agreed on all 200" had trusted the word too. After the fix, it labelled the two corrected alerts ignore, which matches their labels.</p>
<p>Three lessons from that:</p>
<ol>
<li><strong>Agreement between two readers is not proof the labels are right.</strong> Both can take the same shortcut. Here, the disagreements between models found what our check missed, so read them before you trust a score.</li>
<li><strong>A score is only as good as its labels.</strong> With the bug in place, Llama 4 Maverick ranked first (99.5 percent) and Qwen3.8-Max third (99.0 percent), marked down for reading the timestamps.</li>
<li><strong>Real alerts can contain this kind of contradiction.</strong> A silence that expired, a maintenance flag that nobody cleared. The models that read the times treated stale context the way an engineer should.</li>
</ol>
<p>We fixed the inputs, not the labels: the generator now keeps maintenance alerts inside their window, only those two alerts changed (their start times moved to 03:00 and 04:35), and we reran them on every configuration. This correction came after we had seen the results, so treat the final numbers as exploratory rather than a clean holdout. The original two alerts and every configuration's answers to them are in <code>results/v1-maintenance-bug/</code> in the repo. The window text still has an end time but no date, which a stronger dataset would fix.</p>
<h2>What 200 alerts cannot tell you</h2><p>Perfect scores on 200 alerts do not mean a perfect model. Alerts are the unit here (three runs of one alert are not three independent samples), so we count the alerts a model got wrong in at least one run and compute a one-sided 95 percent upper confidence limit on that share. These limits assume the alerts are a random sample, and they are not: they come from 31 templates. Read them as a sense of scale, not as a bound on production error rates:</p>
<table>
<thead>
<tr>
<th>Configuration</th>
<th>Alerts wrong in any run (of 200)</th>
<th>Upper 95% limit</th>
<th>Page-level alerts missed in any run (of 50)</th>
<th>Upper 95% limit</th>
</tr>
</thead>
<tbody><tr>
<td>DeepSeek V4 Flash, Qwen3.8-Max</td>
<td>0</td>
<td>1.5%</td>
<td>0</td>
<td>5.8%</td>
</tr>
<tr>
<td><code>gpt-oss-120b</code></td>
<td>1</td>
<td>2.3%</td>
<td>0</td>
<td>5.8%</td>
</tr>
<tr>
<td>Llama 4 Maverick</td>
<td>2</td>
<td>3.1%</td>
<td>2</td>
<td>12.1%</td>
</tr>
<tr>
<td><code>gpt-oss-20b</code></td>
<td>5</td>
<td>5.2%</td>
<td>3</td>
<td>14.8%</td>
</tr>
<tr>
<td>Ministral 3 14B</td>
<td>42</td>
<td>26.3%</td>
<td>8</td>
<td>27.0%</td>
</tr>
</tbody></table>
<p>So "missed no pages" means "missed none of these 50", which is still consistent with missing about one in seventeen. Compared with Qwen3.8-Max, the other models except Ministral 3 14B were between 0 and 0.83 percentage points less accurate. A paired bootstrap over alerts puts <code>gpt-oss-20b</code> at -0.83 (95% interval -1.67 to -0.17), while the intervals for <code>gpt-oss-120b</code> and Llama 4 Maverick reach zero. When two models made identical calls, the bootstrap has nothing to resample, and that is not evidence that they are equivalent. We cannot rank DeepSeek V4 Flash and Qwen3.8-Max against each other from this data.</p>
<p>There is also what this test does not cover:</p>
<ul>
<li><strong>The alerts carry their own context.</strong> Ours say which service is affected, whether a release is waiting and whether a maintenance window is active. Real alerts often do not, and getting that context into the prompt may be the hard part. We did not test alerts without it.</li>
<li><strong>One policy, one invented company.</strong> A messier policy, or one with judgement calls ("page if it looks bad"), may separate the models more. We did not test that either.</li>
<li><strong>Only the models our key could call.</strong> We did not test the GPT-5 or Claude models.</li>
</ul>
<h2>Cost in perspective</h2><p>Each call used about 720 input tokens and a median of 20 to 200 output tokens, depending on the model. At 10,000 alerts a month, which is a busy on-call rotation, the estimated token cost is $1.57 for DeepSeek V4 Flash and $16.40 for Qwen3.8-Max. The whole test, 3,600 scored calls plus 36 reruns, came to an estimated $1.50.</p>
<p>That changes the question. Neither bill matters next to a single missed page, so price should be the last thing you choose on. We tested each model on all three decisions, not a cascade where cheap rules or a cheap model sort alerts first and only some go to a second model. On this test, DeepSeek V4 Flash and Qwen3.8-Max made the same calls. Qwen3.8-Max answered in 1.3 seconds at the median and never took more than 6.6. DeepSeek V4 Flash cost a tenth as much and sometimes took ten seconds or more. If a few extra seconds before a page are acceptable to you, the cheap model is the better buy. Either way, test it on your own alerts first.</p>
<h2>Run it on your own alerts</h2><p>The generator is the part to replace. Export a few hundred real alerts from Alertmanager, write the decision a senior on-call engineer would make next to each one in the same JSON shape as <code>data/alerts.jsonl</code>, and run the same scripts on a clean results directory. The runner skips any call it already has on disk, so leftover results from our run would be reused:</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/alert-triage-bench
<span class="hljs-built_in">cd</span> alert-triage-bench
<span class="hljs-built_in">cp</span> .env.example .<span class="hljs-built_in">env</span>   <span class="hljs-comment"># add your model access key</span>
<span class="hljs-built_in">set</span> -a; . ./.env; <span class="hljs-built_in">set</span> +a

<span class="hljs-built_in">rm</span> -rf results/raw &amp;&amp; <span class="hljs-built_in">mkdir</span> results/raw                <span class="hljs-comment"># start clean</span>
node scripts/03-run-models.mjs --runs 3 --budget 5     <span class="hljs-comment"># stops near $5</span>
node scripts/04-score.mjs                              <span class="hljs-comment"># writes results/summary.md</span>
</code></pre><p>The budget is checked after each call, so the calls already in flight can take it a little past the limit.</p>
<p>What we would do with the results:</p>
<ul>
<li><strong>Write the policy down first.</strong> Every model in this test, even the weak one, beat a deliberately misleading severity label by a wide margin, and all of them were reading the same 31 non-blank lines of policy.</li>
<li><strong>Score missed pages separately from accuracy.</strong> A model can be 99 percent accurate and still miss the one alert that mattered.</li>
<li><strong>Run each alert more than once.</strong> One run hides models that change their mind.</li>
<li><strong>Read the cases where the models disagree with your labels.</strong> In our test, some of them exposed contradictions in our own inputs.</li>
<li><strong>Time out to a page, not to silence.</strong> The model is a filter in front of your pager, never a reason for it to stay quiet.</li>
</ul>
<p>On this test, with this policy, the cheap model was good enough. The price difference was never the thing to worry about. The latency, the missed pages and the labels were.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DORA Metrics From Git History: The Definition Picks Your Tier]]></title>
      <link>https://devops-daily.com/posts/dora-metrics-from-git-history</link>
      <description><![CDATA[We computed deployment frequency, lead time, change failure rate and recovery time from one year of Kubernetes, uv, Go and Prometheus history, once per common definition. The definition alone moved every project across two or three DORA performance levels, and the merge button decides whether Git still knows when the work started.]]></description>
      <pubDate>Wed, 14 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/dora-metrics-from-git-history</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DORA Metrics]]></category><category><![CDATA[Engineering Metrics]]></category><category><![CDATA[Git]]></category><category><![CDATA[Kubernetes]]></category><category><![CDATA[Go]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>Most DORA dashboards start from the same raw material: your Git history, your tags and your pipeline events. We wanted to know how much of the final number comes from that data and how much comes from the definitions the tool picks for you. So we computed the four DORA metrics for one year of four open-source projects, under several definitions a script or tool could use, and placed the deployment frequency and lead time results in the performance levels of the 2024 DORA report.</p>
<p>The definition alone moved every project across at least two levels. From the same Kubernetes history, deployment frequency is <strong>Medium</strong> (42 final releases, 0.81 a week), <strong>High</strong> (63 tags with the alpha, beta and rc tags, 1.21 a week) or <strong>Elite</strong> (3,286 landings on master, 63 a week). Lead time is <strong>High</strong> (5.2 days) if a landing on master counts as the deploy, <strong>Medium</strong> (25.9 days) to the first tag, and <strong>Low</strong> (76.6 days) to the first final release. For uv, which squash-merges, the lead time from first commit to landing is 0.0 hours, because its squash commits keep no record of when the work started.</p>
<h2>TL;DR</h2><ul>
<li><strong>A deploy is a definition.</strong> Every project lands in <strong>Elite</strong> for deployment frequency if each landing on the default branch counts, and in <strong>Medium</strong> or <strong>High</strong> if releases count.</li>
<li><strong>Lead time depends on where the clock starts and stops.</strong> Kubernetes ranges from 5.2 days to 81.9 days on the same history, depending on the start timestamp, the event that stops the clock, and whether backports count.</li>
<li><strong>The merge button can erase history.</strong> In 3,090 of uv's 3,132 landings, the author date is within a minute of the commit date. Prometheus uses both buttons: changes landed with a merge commit show 48.2 hours from first commit to landing, single-commit landings (squashed or pushed) show 0.0 hours.</li>
<li><strong>Change failure rate is the least stable of the four.</strong> uv is at 86.1%, 19.8% or 5.0% depending on whether a failure is a quick patch release, any revert, or a revert of a change that shipped.</li>
<li><strong>What to do:</strong> write the definitions down, keep them fixed, and compare a team with its own past, not with a level chart or another tool.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li><code>git</code> and <code>python3</code> (standard library only), plus <code>bc</code> for the merge-style demo</li>
<li>A repository whose tags mark releases with version numbers</li>
<li>The companion repo <a href="https://github.com/The-DevOps-Daily/dora-from-git" rel="noopener noreferrer">dora-from-git</a>, which holds every script and every recorded result in this post</li>
</ul>
<h2>How we measured</h2><p>We took one year, 1 October 2025 to 1 October 2026, and four projects that merge in different ways:</p>
<table>
<thead>
<tr>
<th>Repo</th>
<th>How changes land</th>
<th>What marks a release</th>
</tr>
</thead>
<tbody><tr>
<td>kubernetes/kubernetes</td>
<td>merge commits</td>
<td><code>vX.Y.Z</code> tags, plus alpha, beta and rc tags</td>
</tr>
<tr>
<td>astral-sh/uv</td>
<td>squash</td>
<td><code>X.Y.Z</code> tags</td>
</tr>
<tr>
<td>golang/go (go.googlesource.com)</td>
<td>rebase through Gerrit, almost linear history</td>
<td><code>goX.Y.Z</code> tags, plus beta and rc tags</td>
</tr>
<tr>
<td>prometheus/prometheus</td>
<td>mixed: 626 merge commits and 615 squashed or pushed commits on main</td>
<td><code>vX.Y.Z</code> tags, plus rc tags</td>
</tr>
</tbody></table>
<p>The scripts use plain Git and nothing else. Go is hosted on Gerrit, not GitHub, and the same code reads it. Treeless clones keep every commit and tag but no file contents, which is enough for metadata: the four repos take 132 MB, and a full run took 137 seconds on a Raspberry Pi 4 (<code>results/benchmark.txt</code>).</p>
<p>The pipeline is four stages, each writing a file the next one reads:</p>
<ol>
<li><code>00-clone.sh</code> clones each repo and records its head commit.</li>
<li><code>01-extract.py</code> records, for every tag in the window, the commits it shipped for the first time, every first-parent commit on the default branch with the commits it brought in, and every revert with the commit or Gerrit change its message names.</li>
<li><code>02-metrics.py</code> computes each metric under each definition.</li>
<li><code>03-tiers.py</code> places the results in the 2024 levels.</li>
</ol>
<p>Before we trusted the extract, <code>check-method.py</code> compared it with plain Git on seeded random samples. The commits each merge brought in matched <code>git rev-list --no-merges &lt;first-parent&gt;..&lt;merge&gt;</code> on 58 of 58 merges, and the first final release that shipped a commit matched <code>git tag --contains</code> on 60 of 60 commits.</p>
<p>The levels come from the <a href="https://dora.dev/research/2024/dora-report/" rel="noopener noreferrer">2024 Accelerate State of DevOps report</a> (version 2024.3, page 13):</p>
<table>
<thead>
<tr>
<th>Level</th>
<th>Change lead time</th>
<th>Deployment frequency</th>
<th>Change fail rate</th>
<th>Failed deployment recovery time</th>
</tr>
</thead>
<tbody><tr>
<td>Elite</td>
<td>Less than one day</td>
<td>On demand (multiple deploys per day)</td>
<td>5%</td>
<td>Less than one hour</td>
</tr>
<tr>
<td>High</td>
<td>Between one day and one week</td>
<td>Between once per day and once per week</td>
<td>20%</td>
<td>Less than one day</td>
</tr>
<tr>
<td>Medium</td>
<td>Between one week and one month</td>
<td>Between once per week and once per month</td>
<td>10%</td>
<td>Less than one day</td>
</tr>
<tr>
<td>Low</td>
<td>Between one month and six months</td>
<td>Between once per month and once every six months</td>
<td>40%</td>
<td>Between one week and one month</td>
</tr>
</tbody></table>
<p>The report describes these as clusters of survey answers, not as thresholds. We use the ranges the way a dashboard does, as bands. We place only deployment frequency and lead time, because change fail rate is a single value per level and is not in order: Medium (10%) sits below High (20%).</p>
<blockquote>
<p><strong>Note</strong></p>
<p>These projects ship releases, not production deploys, so a tag stands in for a deploy. That makes them a fair test of the definitions, and an unfair basis for ranking the projects. Nothing here says one project is better run than another.</p>
</blockquote>
<h2>What counts as a deploy</h2><p>Four definitions: every tag, every final release, every day with a final release, and every landing on the default branch (a merge, a squash or a direct commit). A tool that reads only the Git host sees no deploys, so the last one is the event it can count.</p>
<table>
<thead>
<tr>
<th>Repo</th>
<th>Every tag</th>
<th>Final releases</th>
<th>Release days</th>
<th>Landings on default branch</th>
</tr>
</thead>
<tbody><tr>
<td>Kubernetes</td>
<td>1.21/week, High</td>
<td>0.81/week, Medium</td>
<td>0.29/week, Medium</td>
<td>63.0/week, Elite</td>
</tr>
<tr>
<td>uv</td>
<td>1.94/week, High</td>
<td>1.94/week, High</td>
<td>1.78/week, High</td>
<td>60.1/week, Elite</td>
</tr>
<tr>
<td>Go</td>
<td>0.69/week, Medium</td>
<td>0.58/week, Medium</td>
<td>0.29/week, Medium</td>
<td>65.6/week, Elite</td>
</tr>
<tr>
<td>Prometheus</td>
<td>0.75/week, Medium</td>
<td>0.50/week, Medium</td>
<td>0.46/week, Medium</td>
<td>23.8/week, Elite</td>
</tr>
</tbody></table>
<p>Kubernetes shows how much the choice matters. It cut 42 final releases in the year, but only on 15 days, because it patches several release lines on the same day. Count tags, releases or release days and you get three different numbers from the same calendar. Count landings and every project is Elite, including Go, whose master branch is not a release.</p>
<p>Your company has the same choice, with different names: every merge to main, every pipeline run, every rollout to production, or every release train. Pick one and write it down.</p>
<h2>Where the lead time clock starts and stops</h2><p>Lead time has two ends, and Git offers several timestamps for each.</p>
<ol>
<li><strong>First commit</strong> author date</li>
<li><strong>Last rebase or amend</strong> committer date</li>
<li><strong>Lands on main</strong> merge or submit</li>
<li><strong>First tag</strong> alpha, beta or rc</li>
<li><strong>Final release</strong> the deploy</li>
</ol>
<p>Here is Kubernetes, from the same history, seven ways. The rows do not cover identical sets of changes: a landing-based row counts what landed in the window, a release-based row counts what releases in the window shipped, and each stop event picks up a slightly different set.</p>
<table>
<thead>
<tr>
<th>Clock starts</th>
<th>Clock stops</th>
<th align="right">Changes</th>
<th align="right">Median</th>
<th>Level</th>
</tr>
</thead>
<tbody><tr>
<td>first commit of the change</td>
<td>landing on master, if that is the deploy</td>
<td align="right">3,286</td>
<td align="right">5.2 d</td>
<td>High</td>
</tr>
<tr>
<td>commit author date</td>
<td>first tag, alpha to rc included</td>
<td align="right">6,812</td>
<td align="right">25.9 d</td>
<td>Medium</td>
</tr>
<tr>
<td>landing on master (commit date for backports)</td>
<td>first final release</td>
<td align="right">3,734</td>
<td align="right">60.6 d</td>
<td>Low</td>
</tr>
<tr>
<td>commit committer date</td>
<td>first final release</td>
<td align="right">6,091</td>
<td align="right">63.1 d</td>
<td>Low</td>
</tr>
<tr>
<td>commit author date</td>
<td>first final release</td>
<td align="right">6,091</td>
<td align="right">76.6 d</td>
<td>Low</td>
</tr>
<tr>
<td>first commit of the change</td>
<td>first final release</td>
<td align="right">3,734</td>
<td align="right">77.0 d</td>
<td>Low</td>
</tr>
<tr>
<td>commit author date, backports left out</td>
<td>first final release</td>
<td align="right">5,486</td>
<td align="right">81.9 d</td>
<td>Low</td>
</tr>
</tbody></table>
<p><strong>Kubernetes change lead time, same history, seven definitions</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Landing is the deploy</td>
<td>5.2 days</td>
<td>High</td>
</tr>
<tr>
<td>To first tag (alpha to rc)</td>
<td>25.9 days</td>
<td>Medium</td>
</tr>
<tr>
<td>Landing to final release</td>
<td>60.6 days</td>
<td>Low</td>
</tr>
<tr>
<td>Committer date to final</td>
<td>63.1 days</td>
<td>Low</td>
</tr>
<tr>
<td>Author date to final</td>
<td>76.6 days</td>
<td>Low</td>
</tr>
<tr>
<td>First commit of change to final</td>
<td>77 days</td>
<td>Low</td>
</tr>
<tr>
<td>Author date, no backports</td>
<td>81.9 days</td>
<td>Low</td>
</tr>
</tbody></table>
<p><em>Median days. 1 Oct 2025 to 1 Oct 2026. Source: results/summary.md in The-DevOps-Daily/dora-from-git.</em></p>
<p>Four choices do most of the moving:</p>
<ul>
<li><strong>Which release stops the clock.</strong> Kubernetes cuts alpha tags from master, so a commit reaches its first tag 25.9 days after it is written, and its first final release 76.6 days after. If your release candidates never serve users, they are not deploys.</li>
<li><strong>Author date or committer date.</strong> The committer date changes every time a commit is rebased or amended, the author date does not. On Kubernetes, switching to the committer date moves the median from 76.6 to 63.1 days.</li>
<li><strong>Backports.</strong> For 605 of the 6,091 commits that Kubernetes final releases shipped, we found no landing on master between October 2024 and October 2026. They were made on release branches: mostly cherry-picked fixes, plus the release commits themselves. <code>git cherry-pick</code> keeps the original author date, so a cherry-pick counts the time since the fix was first written. Leaving these commits out moves the median from 76.6 to 81.9 days.</li>
<li><strong>Per commit or per change.</strong> We count a change as one landing on the default branch, so a pull request merged with a merge commit counts once, while a rebased pull request with ten commits counts ten times, and a commit with no landing on master counts on its own, from its commit date. On Kubernetes, per commit and per change barely differ (76.6 against 77.0 days). The unit matters more where pull requests are large.</li>
</ul>
<p>Go is Low in every release-based definition, from 69.1 days (to the first tag, beta and rc included) to 138.0 days (to the final release, backports left out). It ships a feature release every six months, and in our window go1.26.0 was tagged on 10 February 2026 and go1.27.0 on 19 August 2026. Count a landing on master as the deploy and the same project is High, at 4.1 days.</p>
<p>The other repos show smaller spreads. Prometheus is Low (31.9 to 40.5 days) in every release-based definition. uv releases about twice a week, so it is High (28.6 to 30.1 hours) in all of them.</p>
<h2>The merge button decides what Git remembers</h2><p>This is the part we did not expect to be so clean. With the default behaviour of each button, the way a change lands rewrites its timestamps, and the lead time follows. <code>demo-date-rewrite.sh</code> writes two commits on Monday and Wednesday, lands them on Friday three ways, and reads the result back:</p>
<p><strong>one change, three merge buttons</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./scripts/demo-date-rewrite.sh
merge commit:
  feature: part 2 | author 2026-09-09T15:00:00+00:00 | committer 2026-09-09T15:00:00+00:00
  feature: part 1 | author 2026-09-07T10:00:00+00:00 | committer 2026-09-07T10:00:00+00:00
  lead <span class="hljs-keyword">time</span> from author <span class="hljs-built_in">date</span> 102.0 h, from committer <span class="hljs-built_in">date</span> 102.0 h

squash:
  feature (squashed) | author 2026-09-11T16:00:00+00:00 | committer 2026-09-11T16:00:00+00:00
  lead <span class="hljs-keyword">time</span> from author <span class="hljs-built_in">date</span>   0.0 h, from committer <span class="hljs-built_in">date</span>   0.0 h

rebase:
  feature: part 2 | author 2026-09-09T15:00:00+00:00 | committer 2026-09-11T16:00:00+00:00
  feature: part 1 | author 2026-09-07T10:00:00+00:00 | committer 2026-09-11T16:00:00+00:00
  lead <span class="hljs-keyword">time</span> from author <span class="hljs-built_in">date</span> 102.0 h, from committer <span class="hljs-built_in">date</span>   0.0 h
</code></pre><p>A merge commit keeps the original commits and their dates. A squash writes one new commit dated at the landing. A rebase keeps the author dates but resets the committer dates. Git lets you override both dates, so these are defaults, not laws, and a commit date is not proof of when a commit was pushed.</p>
<p>The real repos behave the same way:</p>
<ul>
<li><strong>uv</strong> lands every change as a single commit. In 3,090 of its 3,132 landings, the author date is within 60 seconds of the commit date, so "first commit to landing" is 0.0 hours at the median and at the 75th percentile. Git cannot tell how long a uv pull request was open.</li>
<li><strong>Prometheus</strong> uses both buttons. The 626 changes it landed with a merge commit took 48.2 hours from first commit to merge at the median. The 615 squashed or pushed ones show 0.0 hours, and in 612 of them the author date is within a minute of the commit date. Same project, same maintainers, same year.</li>
<li><strong>Go</strong> submits through Gerrit, which rebases but keeps the author date. None of its 3,414 single-commit landings has an author date within 60 seconds of the commit date, and the median from author date to submit is 99.0 hours (4.1 days).</li>
</ul>
<p>So with these defaults, a team that squash-merges looks Elite on "first commit to landing" lead time however long its reviews take, and a team that switches from merge commits to squash improves that number overnight without changing anything else.</p>
<h2>Three rules for a failed deploy</h2><p>Git has no incidents, so a Git-only tool has to infer failures. We tested three rules a Git-only tool can apply:</p>
<table>
<thead>
<tr>
<th>Repo</th>
<th>Next patch within 7 days</th>
<th>Release ships any revert</th>
<th>A later release reverts a change it shipped</th>
</tr>
</thead>
<tbody><tr>
<td>Kubernetes</td>
<td>0 of 42 (0.0%)</td>
<td>4 of 42 (9.5%)</td>
<td>1 of 42 (2.4%)</td>
</tr>
<tr>
<td>uv</td>
<td>87 of 101 (86.1%)</td>
<td>20 of 101 (19.8%)</td>
<td>5 of 101 (5.0%)</td>
</tr>
<tr>
<td>Go</td>
<td>4 of 30 (13.3%)</td>
<td>2 of 30 (6.7%)</td>
<td>1 of 30 (3.3%)</td>
</tr>
<tr>
<td>Prometheus</td>
<td>5 of 26 (19.2%)</td>
<td>7 of 26 (26.9%)</td>
<td>1 of 26 (3.8%)</td>
</tr>
</tbody></table>
<p>Each rule fails in its own way:</p>
<ul>
<li><strong>The hotfix rule reads a version scheme as failures.</strong> uv increments the patch number for nearly every release and shipped 101 in the year, so the next patch almost always follows within a week: 86.1%. Kubernetes publishes patch releases monthly, so the same rule never fires: 0.0%. Neither number says anything about failures.</li>
<li><strong>"Ships a revert" counts reverts that users never saw.</strong> Many reverts undo a change before it reaches a release. In Kubernetes, 34 reverts that shipped in a final release undid a change that had not shipped in an earlier release in the window: either it was never released, or it shipped before the window.</li>
<li><strong>"Reverts a shipped change" needs the revert to name its target in a form a script can read.</strong> Our parser looks for "reverts commit " and "reverts CL ". It could not resolve 21 Kubernetes reverts and 19 uv reverts: their messages name no commit that way, sometimes none at all, so this rule cannot match them. Go writes "This reverts CL 847405" instead of a commit hash, and the script resolves those through the <code>Reviewed-on</code> trailer that Gerrit adds. Your conventions decide what this rule can see.</li>
</ul>
<p>The strict rule finds one failure each in Kubernetes, Go and Prometheus. With one event, the rate is noise: one more revert would double it.</p>
<p>Recovery time inherits all of this. Under the hotfix rule it is the time to the next patch: 3.0 days for uv, 5.2 for Prometheus, 6.0 for Go. It can never be longer than 7 days, because the rule only counts a failure when the fix came within 7 days. The number measures the rule, not the recovery.</p>
<h2>What a dashboard adds, and what it does not</h2><p>The script shows what Git alone supports. A full engineering-metrics tool (vendors in this space include LinearB, Swarmia, Jellyfish, DX, Sleuth and Faros AI) can read other systems, and those systems hold data Git does not:</p>
<ul>
<li><strong>When the work started.</strong> The forge API knows when a pull request was opened and when its first commit was pushed, which is exactly what a squash merge removes from Git.</li>
<li><strong>Real deploy events.</strong> A CD pipeline knows when a version reached production, per environment. A tag is a stand-in.</li>
<li><strong>Real failures.</strong> Linking deploys to incidents, rollbacks and pages measures failure directly instead of inferring it from reverts or patch numbers.</li>
<li><strong>Ownership and trends.</strong> Mapping repos and paths to services and teams, and recomputing the same way every week.</li>
</ul>
<p>What a tool does not remove is the choice of definition. Before you compare a number across teams or tools, ask which event counts as a deploy, which timestamp starts lead time, whether backports and release candidates count, and what counts as a failure. If the answers differ, the numbers do not compare.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>These projects do not deploy.</strong> Tags stand in for deploys. The results test the definitions, not the projects.</li>
<li><strong>The levels are survey clusters.</strong> The report presents them as groups of answers, and the band edges are our reading.</li>
<li><strong>The failure numbers are small.</strong> The strict revert rule found one failure in three of the four projects, so its rates move a lot with one release.</li>
<li><strong>Unshipped work is left out.</strong> Lead time counts only commits that reached a release before the window closed, so the medians describe completed work. Go's work since the go1.27 freeze is not in them.</li>
<li><strong>Tag dates differ.</strong> uv and Go use lightweight tags, dated by the tagged commit. Kubernetes and Prometheus use annotated tags, dated by the tagger. A release can be published hours after its tag.</li>
</ul>
<h2>Run it on your repositories</h2><pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/dora-from-git.git
<span class="hljs-built_in">cd</span> dora-from-git
./run.sh
<span class="hljs-built_in">cat</span> results/summary.md
</code></pre><p>To measure your own repositories, edit <code>repos.json</code>. Each entry needs a clone URL, the default branch, and a regular expression for your release tags, with groups for major, minor and patch and, if you tag release candidates, a fourth group for the prerelease suffix:</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"payments-api"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"url"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"git@gitlab.example.com:platform/payments-api.git"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"branch"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"main"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"merge_style"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"squash"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"tag_pattern"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"^v(\\d+)\\.(\\d+)\\.(\\d+)(-rc\\.\\d+)?$"</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>If your deploys are not tagged, have your pipeline tag each production deploy with a version that the pattern matches, for example <code>v1.42.0</code>, and the same scripts measure deploys instead of releases. The hotfix rule reads the patch number, so it only means something if a patch bump marks a fix.</p>
<p><a href="https://github.com/The-DevOps-Daily/dora-from-git" rel="noopener noreferrer">The-DevOps-Daily/dora-from-git on GitHub</a></p>
<h2>Summary</h2><ul>
<li>The same Git history put each of the four projects in two or three different DORA levels, depending only on the definition.</li>
<li>What counts as a deploy moved deployment frequency by up to two levels, and every project is Elite if a landing on the default branch counts.</li>
<li>Where the clock starts and stops moved Kubernetes lead time from 5.2 to 81.9 days on the same history.</li>
<li>With default settings, squash merges and rebases rewrite the timestamps lead time depends on, so the merge button can change the metric without changing the work.</li>
<li>Change failure rate inferred from Git measured version schemes and commit message habits more than failures.</li>
<li>Write the definitions down, keep them fixed, record the unit (commit or change), and compare a team with its own past.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Datadog Alternatives in 2026: 12 Tools, Their Real Prices, and What 300 GB of Logs Costs on Each]]></title>
      <link>https://devops-daily.com/posts/datadog-alternatives-2026-real-prices</link>
      <description><![CDATA[We read the pricing page of Datadog and 12 alternatives on the same day and priced one workload on each: 300 GB of logs a month. Datadog with standard indexing of every event is $780 to $1,155 a month; the alternatives we could price run from about $19 to $180. Here is each tool, who it suits, and the catch on its pricing page.]]></description>
      <pubDate>Mon, 12 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/datadog-alternatives-2026-real-prices</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[Observability]]></category><category><![CDATA[Datadog]]></category><category><![CDATA[OpenTelemetry]]></category><category><![CDATA[Logging]]></category><category><![CDATA[Monitoring]]></category><category><![CDATA[FinOps]]></category>
      <content:encoded><![CDATA[<p>Many teams start looking for a Datadog alternative the same way: a bill arrives, someone opens the usage page, and logs are a big line. Datadog charges separately to ingest logs (per GB) and to index them (per event). A busy service that writes small lines pays far more for indexing than its byte count suggests.</p>
<p>Every "Datadog alternatives" list says the alternatives are cheaper. Few say by how much, because the vendors price in different units: per GB, per event, per host, per user, per compute credit. So we did the arithmetic. On October 12, 2026 we read the pricing page of Datadog and 12 alternatives, and priced one workload on each at list price: <strong>300 GB of logs a month, kept for 30 days</strong>. Then we wrote down what each tool is good at and the catch we found on its pricing page.</p>
<h2>TLDR</h2><ul>
<li><strong>The workload:</strong> 300 GB of logs a month, 30-day retention, about 1 KB per log event (so 300 million events), list prices, no negotiated discounts.</li>
<li><strong>Datadog:</strong> $780 a month with annual billing if you index every event with 30-day retention, $1,155 on demand. Index 10% of events and it drops to $105, but the other 90% are then not searchable unless you archive them or use Flex Logs.</li>
<li><strong>Cheapest on paper:</strong> Uptrace ($19 to $25, its page gives two rates), Axiom ($25 if your logs compress at least 3:1 and you stay inside the included query compute), Better Stack ($45 in its EU region).</li>
<li><strong>Generous free tiers:</strong> Axiom Personal (500 GB of data loading), New Relic (100 GB of ingest), Grafana Cloud (50 GB each of logs, traces and profiles).</li>
<li><strong>Best fit by need:</strong> Grafana Cloud if you already run Prometheus, Honeycomb for debugging traces, SigNoz or OpenObserve if you may self-host later, ClickStack if you want ClickHouse and long retention.</li>
<li><strong>The catches:</strong> a 30% annual discount folded into a "pay as you go" price, ingest that stops when you pass the free tier, a daily quota that blocks data until midnight UTC, and region-based prices that differ 3.5x.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A rough idea of your monthly log volume in GB. Your current vendor's usage page has it.</li>
<li>Your average log event size. If you do not know it, divide bytes ingested by events ingested on your usage page.</li>
<li>Ten minutes with the OpenTelemetry Collector docs if you plan to try two tools side by side (more on that at the end).</li>
</ul>
<h2>How we priced it</h2><p>Prices change often, so everything below is a snapshot of each vendor's own pricing page on October 12, 2026. We did not use third-party price lists.</p>
<p>The workload is logs only, because logs are the line that sends most people looking, and because every vendor prices logs in a way we can convert:</p>
<ul>
<li><strong>Volume:</strong> 300 GB ingested per month (about 10 GB a day).</li>
<li><strong>Retention:</strong> 30 days. Where the listed price includes less, we say so.</li>
<li><strong>Event size:</strong> 1 KB on average, so 1 GB is 1 million events and the month is 300 million events. This matters for the vendors that bill per event. If your events are smaller, per-event pricing gets worse fast. Our <a href="https://devops-daily.com/posts/your-logs-cost-more-than-the-service">logs cost post</a> measured 209 bytes per nginx line on a real server, which is about 5 million events per GB.</li>
<li><strong>Not included:</strong> hosts, APM, metrics, support plans, cloud egress from wherever your workloads run, and query charges where they are separate (OpenObserve and Axiom bill queries; most others include them). We add a required user or plan fee where the vendor needs one to send logs at all, and call out the rest per vendor.</li>
</ul>
<p><strong>Monthly list price for 300 GB of logs</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Datadog, standard indexing, on demand</td>
<td>1155$</td>
<td>Datadog</td>
</tr>
<tr>
<td>Datadog, standard indexing, annual</td>
<td>780$</td>
<td>Datadog</td>
</tr>
<tr>
<td>Dash0</td>
<td>180$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>OpenObserve, annual commitment</td>
<td>150$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>Grafana Cloud</td>
<td>131.5$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>SigNoz (15-day)</td>
<td>90$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>New Relic, retention unconfirmed</td>
<td>90$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>Better Stack (EU)</td>
<td>45$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>Uptrace (28-day)</td>
<td>25$</td>
<td>Alternatives</td>
</tr>
<tr>
<td>Axiom</td>
<td>25$</td>
<td>Alternatives</td>
</tr>
</tbody></table>
<p><em>Vendor pricing pages, October 12, 2026. 1 KB per event (300M events), 30-day retention unless the label says otherwise. Logs only; query charges extra on OpenObserve and Axiom. Uptrace shows the higher of the two rates on its page. Axiom assumes at least 3:1 compression. ClickStack, Coralogix, Elastic, Honeycomb and self-hosting are left out because their pages do not give a single comparable figure; see each section.</em></p>
<p>A cheaper log line does not make a cheaper platform. Most of these tools also charge for metrics, traces or seats, and some charge for queries. Use the number to see the gap, then price your own mix with each vendor's calculator.</p>
<h2>The baseline: Datadog</h2><p><strong>Pricing model:</strong> per host for infrastructure and APM, per GB to ingest logs, per million events to index them. Infrastructure Pro is $15 per host per month billed annually ($18 on demand). APM is $31 per host ($36 on demand).</p>
<p><strong>300 GB of logs:</strong> ingest is $0.10 per GB, so $30. Indexing with 30-day retention is $2.50 per million events billed annually, $3 billed monthly and $3.75 on demand. At 300 million events:</p>
<pre><code class="hljs language-text" tabindex="0">annual:     $30 ingest + 300 x $2.50 = $780 per month
on demand:  $30 ingest + 300 x $3.75 = $1,155 per month
index 10%:  $30 ingest +  30 x $2.50 = $105 per month
</code></pre><p><strong>Fair to Datadog:</strong> you rarely need to index every log. Exclusion filters, shorter retention (7-day indexing is $1.27 per million) and Flex Logs (storage at $0.05 per million events per month, with compute for queries priced separately) all cut the bill. The 10% figure above is only fair if the other 90% can stay unsearchable or go to an archive. Datadog also lists more than 1,000 integrations.</p>
<p><strong>The catch:</strong> the billable host count is the "high-water mark of the lower 99 percent" of hourly usage. The top 1% of hours (about 7 hours in a 730-hour month) are dropped, so an autoscaling spike that lasts longer than that sets your host bill for the month. On the log side, a spike raises the ingest charge, and the indexing charge too if those events pass your index filters.</p>
<h2>1. Grafana Cloud</h2><p><strong>Best for:</strong> teams that already run Prometheus, Loki or Grafana dashboards.</p>
<p><strong>Pricing model:</strong> Pro is "from $19/month + usage". Logs, traces and profiles start at $0.05 per GB to process plus $0.40 per GB to write, and the first 30 days of retention are included. Metrics start at $6.50 per 1,000 active series.</p>
<p><strong>Free tier:</strong> 10,000 active series, 50 GB each of logs, traces and profiles, 3 active users, 14-day retention.</p>
<p><strong>300 GB of logs:</strong> $19 + (300 - 50) x $0.45 = <strong>$131.50</strong>.</p>
<p><strong>Strengths:</strong> the open source stack underneath (Grafana, Loki, Tempo and Mimir, all AGPL-3.0), native PromQL, and a free tier that covers logs, traces and profiles. You can move to self-hosting later without changing query languages.</p>
<p><strong>The catch:</strong> Loki indexes labels, not log content. Its own docs warn that too many label combinations give "a large index and small chunks", which "can actually reduce performance". Plan your labels before you send 300 GB. The pricing page also carries a banner saying Grafana is changing its pricing, so check it again before you commit.</p>
<h2>2. SigNoz</h2><p><strong>Best for:</strong> teams that want one per-GB rate for logs and traces, and the option to self-host.</p>
<p><strong>Pricing model:</strong> the Teams plan is $49 a month, and that includes $49 of usage. After that, logs and traces are $0.30 per GB ingested and metrics are $0.10 per million samples. No per-host or per-user fees.</p>
<p><strong>Free tier:</strong> a cloud trial. Self-hosting is free; the code is MIT licensed apart from the <code>ee/</code> and <code>cmd/enterprise/</code> directories.</p>
<p><strong>300 GB of logs:</strong> 300 x $0.30 = $90, which absorbs the $49 minimum, so <strong>$90</strong>, at the listed 15-day retention. The page offers 30, 90 and 180 days and a year, but the text we captured does not give their prices.</p>
<p><strong>Strengths:</strong> built on OpenTelemetry and ClickHouse, with OTLP over gRPC and HTTP. Logs, traces and metrics share one UI and one query builder.</p>
<p><strong>The catch:</strong> SSO and SAML are cloud only, RBAC is in beta, and audit logs and multi-tenancy are listed as "coming soon". If compliance features matter, check where they stand.</p>
<h2>3. Better Stack</h2><p><strong>Best for:</strong> small teams that want uptime checks, on-call, status pages and logs on one bill.</p>
<p><strong>Pricing model:</strong> per GB ingested plus per GB-month retained, priced by region. In Germany (the default) ingest is $0.10 per GB and retention $0.05 per GB per month. In the US it is $0.15 and $0.08. In Singapore $0.35 and $0.18. On-call responders are $29 a month per seat billed annually.</p>
<p><strong>Free tier:</strong> 10 monitors, 1 status page, 3 GB of logs and 3 GB of traces (kept for 3 days), 30 GB of metrics.</p>
<p><strong>300 GB of logs:</strong> 300 x $0.10 + 300 x $0.05 = <strong>$45</strong> in Germany, $69 in the US. The plan includes 3 GB a month (kept for 3 days), which changes this by less than a dollar.</p>
<p><strong>Strengths:</strong> one of the lowest per-GB list prices here, a full UI, and OTLP over HTTP. You can keep the data in your own S3 bucket for $250 a month.</p>
<p><strong>The catch:</strong> region matters more than you would guess: Singapore ingest costs 3.5x the German rate. The pricing page also says it is "currently not HIPAA compliant".</p>
<h2>4. ClickStack (HyperDX on ClickHouse Cloud)</h2><p><strong>Best for:</strong> teams that want long retention on cheap storage, or already use ClickHouse.</p>
<p><strong>Pricing model:</strong> you pay for ClickHouse Cloud compute and storage, not per GB of logs. Storage is $25.30 per TB per month. Compute is metered per minute at $0.2181 per unit-hour on Basic (a unit is 8 GiB of RAM and 2 vCPUs, AWS us-east-1), and the Basic tier offers 8 to 12 GiB of memory. Compute scales to zero when idle.</p>
<p><strong>Free tier:</strong> $300 of credits for 30 days. HyperDX is MIT and ClickHouse is Apache-2.0, so you can self-host both.</p>
<p><strong>300 GB of logs:</strong> it depends on how long compute runs. Logs arrive all day, so assume one 8 GiB unit stays awake: $0.2181 x 730 hours = $159.21. Storage at ClickHouse's own 10x compression estimate is 30 GB, about $0.76. That gives <strong>about $160</strong>, before backups and query load. We left it off the chart because the real figure depends on sizing you choose.</p>
<p><strong>Strengths:</strong> storage is so cheap that a year of retention changes the bill very little. SQL on your logs, and native OTLP.</p>
<p><strong>The catch:</strong> you are sizing a database, not buying a per-GB service. Continuous ingestion keeps compute awake, so at small volume the compute line dominates the bill.</p>
<h2>5. OpenObserve</h2><p><strong>Best for:</strong> teams that want a cheap self-hosted option with a cloud fallback.</p>
<p><strong>Pricing model:</strong> $0.50 per GB ingested plus $0.01 per GB queried, with 30 days of log retention included.</p>
<p><strong>Free tier:</strong> a 14-day cloud trial. The self-hosted Enterprise edition is free up to 50 GB a day, and the open source version is AGPL-3.0.</p>
<p><strong>300 GB of logs:</strong> 300 x $0.50 = <strong>$150</strong> at the annual-commitment rate (see the catch), plus queries (scanning 1 TB in a month adds $10).</p>
<p><strong>Strengths:</strong> the free self-hosted tier is generous: 50 GB a day is 1.5 TB a month, five times our workload.</p>
<p><strong>The catch:</strong> the page calls the $0.50 rate "pay-as-you-go, billed monthly", but a footnote says it "includes 30% discount for annual commitment". The page does not show the price without the commitment.</p>
<h2>6. Axiom</h2><p><strong>Best for:</strong> high log volume with modest query load.</p>
<p><strong>Pricing model:</strong> the Axiom Cloud plan is $25 a month, and that includes 1,000 GB of data loading, 100 GB-hours of query compute and 100 GB of storage. Past that, data loading starts at $0.12 per GB, queries at $0.20 per GB-hour, and storage at $0.03 per GB per month, charged on compressed data.</p>
<p><strong>Free tier:</strong> the Personal plan gives 500 GB of data loading, 10 GB-hours of queries, 25 GB of storage and 30-day retention, with no card.</p>
<p><strong>300 GB of logs:</strong> the loading fits in the included 1,000 GB. Storage is charged on compressed data, so 300 GB fits in the included 100 GB at any compression ratio better than 3:1. That leaves <strong>$25</strong>, plus query compute if you go past 100 GB-hours. If your logs compress worse than 3:1, storage adds up to $6 (at no compression, 200 GB over the allowance at $0.03).</p>
<p><strong>Strengths:</strong> one of the cheapest results for our workload, and the largest free allowance for data loading on this list.</p>
<p><strong>The catch:</strong> queries are metered, and query compute from AI agents counts the same as from people. A team that queries all day, or wires an assistant into its logs, should model query cost first.</p>
<h2>7. Honeycomb</h2><p><strong>Best for:</strong> debugging distributed systems through traces with many unique attribute values.</p>
<p><strong>Pricing model:</strong> per event, with unlimited seats. Each span, span event and link counts as an event. Pro starts at $150 a month for 50 million events and goes up to 750 million.</p>
<p><strong>Free tier:</strong> 20 million events and 100 million metric data points a month.</p>
<p><strong>300 GB of logs:</strong> our 300 million events fit inside Pro's range, but the page does not list the price for that tier, so we left it off the chart.</p>
<p><strong>Strengths:</strong> built for asking new questions of trace data. High-cardinality fields (user ID, build ID, feature flag) cost nothing extra, because pricing is per event, not per field.</p>
<p><strong>The catch:</strong> 60 days of retention are included, and longer retention is an Enterprise add-on. For our workload you need a quote, and per-event pricing means smaller events cost more per GB.</p>
<h2>8. New Relic</h2><p><strong>Best for:</strong> small teams that fit in the free tier, and teams that want no host-based pricing.</p>
<p><strong>Pricing model:</strong> $0.40 per GB beyond 100 free GB a month, plus users. Core users are $49 a month. A full platform user on Standard is $10 for the first and $99 each after that (up to 5).</p>
<p><strong>Free tier:</strong> 100 GB of ingest a month, 1 full user, unlimited basic users.</p>
<p><strong>300 GB of logs:</strong> (300 - 100) x $0.40 = $80, plus $10 for the first full platform user on Standard, so <strong>$90</strong>. The pricing page we read does not state log retention on paid plans (the free tier keeps data for at least 8 days), so confirm it covers 30 days.</p>
<p><strong>Strengths:</strong> 100 GB of free ingest a month, OTLP support and no per-host charges.</p>
<p><strong>The catch:</strong> on the free tier, going over 100 GB means "data ingest will cease and you will lose access to the platform" until you upgrade to a paid edition or the next month starts. New Relic says it warns you first, but it is still a bad surprise during an incident, which is exactly when log volume spikes. For a wider look at the two vendors, see <a href="https://devops-daily.com/comparisons/new-relic-vs-datadog">New Relic vs Datadog</a>.</p>
<h2>9. Dash0</h2><p><strong>Best for:</strong> teams already standardized on OpenTelemetry.</p>
<p><strong>Pricing model:</strong> per million signals, with separate ingest and storage prices. Spans and log records cost $0.060 per million to ingest and $0.540 per million to store for 30 days.</p>
<p><strong>Free tier:</strong> a 14-day trial.</p>
<p><strong>300 GB of logs:</strong> 300 x ($0.060 + $0.540) = <strong>$180</strong>.</p>
<p><strong>Strengths:</strong> built around OpenTelemetry from the start, and storage is billed only on what you keep.</p>
<p><strong>The catch:</strong> data you filter out still pays the ingest price, and per-event pricing punishes small events. At the 209-byte nginx line from our logs post, the same 300 GB would be about 1.4 billion events.</p>
<h2>10. Coralogix</h2><p><strong>Best for:</strong> teams that want to keep logs in their own bucket and still query them.</p>
<p><strong>Pricing model:</strong> units at $1.50 each, charged per GB by pipeline against a daily quota. The headline prices are $0.42 per GB for logs, $0.16 for traces and $0.06 for metrics.</p>
<p><strong>Free tier:</strong> a 14-day trial with 8 units.</p>
<p><strong>300 GB of logs:</strong> <strong>$126 to $345</strong>, depending on the pipeline. The headline $0.42 per GB gives $126. The FAQ on the same page prices Monitoring at $0.50 per GB ($150) and Frequent Search at $1.15 per GB ($345). The page does not say which pipeline the $0.42 applies to, which is why it is not on the chart.</p>
<p><strong>Strengths:</strong> you can query the archive in your own bucket, and the priority pipelines let you pay less for logs you only keep for compliance.</p>
<p><strong>The catch:</strong> when you hit the daily quota (2x with pay-as-you-go turned on), data is blocked until 00:00 UTC, and unused units expire.</p>
<h2>11. Uptrace</h2><p><strong>Best for:</strong> cost-sensitive teams in Europe.</p>
<p><strong>Pricing model:</strong> per GB for spans and logs, per million data points for metrics, 28-day retention. The page says "50 GB free/mo · then $0.075/GB" in one place and rates that "start at $0.10/GB" in another.</p>
<p><strong>Free tier:</strong> 50 GB and 5,000 time series a month. The self-hosted Community Edition is AGPL-3.0.</p>
<p><strong>300 GB of logs:</strong> (300 - 50) x $0.075 = $18.75, or $25 at $0.10. Call it <strong>$19 to $25</strong>, and check the page before you budget.</p>
<p><strong>Strengths:</strong> among the lowest per-GB rates on this list, with unlimited users and hosts.</p>
<p><strong>The catch:</strong> the pricing page contradicts itself on the starting rate, and the cloud runs in one region, Hetzner in Germany. That suits EU data residency, but workloads in the US or Asia send every log across an ocean.</p>
<h2>12. Elastic Observability (Serverless)</h2><p><strong>Best for:</strong> teams that need full-text search across log content.</p>
<p><strong>Pricing model:</strong> per GB ingested plus per GB retained. Logs Essentials is "as low as $0.07/GB" to ingest and "as low as" $0.017 to retain. The Complete tier lists logs and traces "as low as $0.09/GB". Limited support is included; higher support levels add 5% to 15%.</p>
<p><strong>Free tier:</strong> a 14-day trial.</p>
<p><strong>300 GB of logs:</strong> not computed. The page shows only the lowest volume tier ("as low as"), not the rate at 300 GB.</p>
<p><strong>Strengths:</strong> it indexes the text of every log line, so full-text searches over long time ranges stay fast. Loki can also search log content, but it scans the chunks instead of using an index. Self-managed and hosted options exist. The <a href="https://devops-daily.com/comparisons/elk-stack-vs-loki-grafana">ELK Stack vs Loki + Grafana comparison</a> goes deeper into that trade-off.</p>
<p><strong>The catch:</strong> the page shows only "as low as" rates, so you need the calculator for a real number.</p>
<h2>The self-hosted option: Prometheus, Grafana, Loki and Tempo</h2><p><strong>Best for:</strong> teams with spare infrastructure and the people to run it.</p>
<p><strong>Pricing model:</strong> no license fee. Prometheus is Apache-2.0. Grafana, Loki, Tempo and Mimir are AGPL-3.0.</p>
<p><strong>300 GB of logs:</strong> whatever your servers, object storage and on-call time cost. For 300 GB a month, Loki on object storage is cheap to store. The real cost is the people.</p>
<p><strong>Strengths:</strong> full control, no per-GB pricing, and every piece speaks OpenTelemetry: Prometheus has an OTLP receiver (off by default, <code>--web.enable-otlp-receiver</code>), Loki accepts OTLP at <code>/otlp</code>, and Tempo takes OTLP natively.</p>
<p><strong>The catch:</strong> four systems to deploy, upgrade, scale and back up. If nobody owns them, they rot. For the metrics part of this decision, see <a href="https://devops-daily.com/comparisons/prometheus-vs-datadog">Prometheus vs Datadog</a>.</p>
<h2>How to switch without a big-bang migration</h2><p>Do not move everything at once. The cheap, safe way is to send the same data to two places and compare:</p>
<ol>
<li><strong>Put an OpenTelemetry Collector in front of your apps</strong> if you do not have one. Our post on <a href="https://devops-daily.com/posts/opentelemetry-graduated-what-to-retire-this-quarter">what to retire now that OpenTelemetry has graduated</a> covers the move.</li>
<li><strong>Add a second exporter</strong> that sends logs to the tool you want to test. The Collector can fan out to several backends from one pipeline:</li>
</ol>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">exporters:</span>
  <span class="hljs-attr">datadog:</span>
    <span class="hljs-attr">api:</span>
      <span class="hljs-attr">key:</span> <span class="hljs-string">${env:DD_API_KEY}</span>
  <span class="hljs-attr">otlphttp/candidate:</span>
    <span class="hljs-attr">endpoint:</span> <span class="hljs-string">https://otlp.example-vendor.com</span>   <span class="hljs-comment"># the candidate's OTLP endpoint</span>
    <span class="hljs-attr">headers:</span>
      <span class="hljs-attr">authorization:</span> <span class="hljs-string">Bearer</span> <span class="hljs-string">${env:CANDIDATE_TOKEN}</span>

<span class="hljs-attr">service:</span>
  <span class="hljs-attr">pipelines:</span>
    <span class="hljs-attr">logs:</span>
      <span class="hljs-attr">receivers:</span> [<span class="hljs-string">otlp</span>]
      <span class="hljs-attr">processors:</span> [<span class="hljs-string">batch</span>]
      <span class="hljs-attr">exporters:</span> [<span class="hljs-string">datadog</span>, <span class="hljs-string">otlphttp/candidate</span>]   <span class="hljs-comment"># send to both while you compare</span>
</code></pre><ol>
<li><strong>Run both for a full billing cycle.</strong> One week is not enough, because month-end batch jobs and incident spikes change the volume.</li>
<li><strong>Compare three things:</strong> the bill, whether your top 10 saved searches and alerts still work, and how long a real incident investigation takes in each tool.</li>
<li><strong>Move logs first.</strong> Logs are often the largest and most portable line. Keep APM where it is until the logs move is done.</li>
</ol>
<h2>Summary</h2><ul>
<li>At list price, 300 GB of logs a month costs $780 on Datadog with annual billing if you index every event, and about $19 to $180 on the alternatives we could price. The biggest saving on Datadog itself is to index less.</li>
<li>The cheapest log line is not the cheapest platform. Axiom, Uptrace and Better Stack win on logs alone. Grafana Cloud and New Relic win on free tiers. Honeycomb wins on trace debugging. ClickStack wins on long retention.</li>
<li>Read the footnotes. We found a 30% annual discount in a "pay as you go" price, ingest that stops at the free-tier limit, a daily quota that blocks data until midnight UTC, regional prices 3.5x apart, and a page that gives two different rates.</li>
<li>Event size changes everything for per-event vendors. Measure yours before you trust any comparison, including this one.</li>
<li>Prices change. Every number here is from the vendors' own pages on October 12, 2026.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DevOps Interviews Now Test How You Decide. How to Practice Situational Questions]]></title>
      <link>https://devops-daily.com/posts/devops-interviews-situational-questions</link>
      <description><![CDATA[Fewer interviewers ask what a Kubernetes Service is. More of them describe a broken system and ask what you do first. Here is what they grade, five worked scenarios with weak and strong answers, and a way to practice making the call.]]></description>
      <pubDate>Mon, 12 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/devops-interviews-situational-questions</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[Interviews]]></category><category><![CDATA[Career]]></category><category><![CDATA[SRE]]></category><category><![CDATA[Incident Response]]></category><category><![CDATA[On-call]]></category>
      <content:encoded><![CDATA[<p>"What is a Kubernetes Service?" is a question you can answer from memory. "Error rates on checkout went from 0.2% to 2.5% ten minutes after your Friday deploy, and the deploy included a database migration. What do you do?" is not. Many DevOps and SRE interviews now include questions like the second one. There is no single correct answer, and the interviewer is not waiting for a specific command. They want to hear how you decide: what you check first, which risk you are willing to take, and what you tell your team while you do it.</p>
<p>You cannot memorize your way through that, but you can practice it. This post covers what interviewers grade in a situational question, a five-step way to structure your answer, five worked scenarios with a weak and a stronger answer for each, and a practice routine that trains the decision instead of the definition.</p>
<h2>TLDR</h2><ul>
<li>Situational questions grade your <strong>reasoning</strong>, not your recall. The same final action can score well or badly depending on how you got there.</li>
<li>Structure every answer the same way: <strong>scope the problem, name your first check, pick the reversible move, say what you tell people, and say what you change afterwards.</strong></li>
<li>Always say the <strong>risk you accept</strong> out loud. "I roll back, and we lose the new feature for the evening" is a stronger answer than "I roll back."</li>
<li>Practice by <strong>changing one fact and answering again.</strong> The decision changes when the migration is destructive, when it is 3 a.m., or when the VP says no. That boundary is what interviewers probe.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Some hands-on time with production systems. The scenarios assume you know roughly what a deploy pipeline, a Postgres primary, a Terraform plan and an AWS access key are.</li>
<li>A timer and a way to record yourself. Speaking an answer out loud is a different skill from thinking it.</li>
</ul>
<h2>What interviewers are actually grading</h2><p>A situational question is a small simulation of the job. The interviewer has a rough rubric in their head, even if nobody wrote it down. Here is a version of it, with what a weak and a strong signal sound like:</p>
<table>
<thead>
<tr>
<th>What they listen for</th>
<th>Weak signal</th>
<th>Strong signal</th>
</tr>
</thead>
<tbody><tr>
<td>First check</td>
<td>"I'd check the logs"</td>
<td>Names one signal and what it rules in or out</td>
</tr>
<tr>
<td>Scope</td>
<td>Jumps straight to a fix</td>
<td>Asks who and what is affected, and since when</td>
</tr>
<tr>
<td>Reversibility</td>
<td>Treats every action the same</td>
<td>Prefers the step that can be undone, and names the one that cannot</td>
</tr>
<tr>
<td>Risk</td>
<td>"There is no risk"</td>
<td>States the cost of the chosen move</td>
</tr>
<tr>
<td>Communication</td>
<td>Fixes it alone and quietly</td>
<td>Says who hears what, and when the next update is</td>
</tr>
<tr>
<td>Follow-up</td>
<td>Stops at "fixed"</td>
<td>Names the change that stops this class of failure</td>
</tr>
</tbody></table>
<p>Notice that none of the rows is "knows the right tool". Tools come up, but as evidence for a decision, not as the answer. A candidate who says "I'd run <code>kubectl rollout undo</code>" and stops has given a command. A candidate who says "the errors started with the rollout and only on the new pods, so I undo the rollout first, because it takes a minute and can be redone" has given a decision.</p>
<h2>A structure that works for any scenario</h2><p>You do not need a framework with a clever acronym. You need the same five moves every time, so the interviewer can follow you and you do not forget the communication part under pressure.</p>
<p><em>Goal: Say the risk you accept out loud</em></p>
<ol>
<li><strong>Scope</strong> who, what, since when</li>
<li><strong>First check</strong> one signal, and why</li>
<li><strong>Decide</strong> reversible move first</li>
<li><strong>Communicate</strong> who hears what, next update</li>
<li><strong>Follow up</strong> fix the class, not the case</li>
</ol>
<p><em>new evidence changes the decision, then back to step 1.</em></p>
<ol>
<li><strong>Scope.</strong> Restate the problem with the facts you have, and ask for the ones you need. "How many users? Is it one region? Did anything change in the last hour?" Asking is not stalling. It is the first thing a good on-call engineer does.</li>
<li><strong>First check.</strong> Name one specific thing you look at first and what each result would mean. Good first checks split the problem in half: "if the spike starts with the rollout, the deploy is the main suspect; if it started before, I look elsewhere first."</li>
<li><strong>Decide.</strong> Pick the move that reduces harm fastest and can be undone. Rollbacks, feature flags, scaling up and failing over are usually reversible. Deleting data, dropping replication slots and force-pushing are not. If you choose an irreversible step, say why the alternative is worse.</li>
<li><strong>Communicate.</strong> Say who you tell, what you say, and when the next update is. Write the actual message if you can. Interviewers notice when candidates forget this, because in real incidents the silence is often what makes it worse.</li>
<li><strong>Follow up.</strong> Say what you change so this does not happen again: a test, an alert, a guardrail in the pipeline, a runbook. This is usually where senior answers separate from junior ones.</li>
</ol>
<blockquote>
<p><strong>Tip</strong></p>
<p>When new information arrives, say so and go back through the loop: "If the migration turns out to drop a column, the rollback is no longer safe, so I would switch to fixing forward." Interviewers often add a twist on purpose to see whether you update your decision or defend your first one.</p>
</blockquote>
<h2>Five scenarios, worked through</h2><p>Each scenario below has a short weak answer and a stronger one. The stronger answers are not the only right answers. They are examples of reasoning that would score well, and the point of each is the decision and its trade-off, not the specific commands.</p>
<h3>1. The Friday deploy</h3><blockquote>
<p>You deployed checkout v2.31 at 16:40. By 16:50 the 5xx rate on checkout is 2.5%, up from 0.2%. The release also ran a database migration. What do you do?</p>
</blockquote>
<p><strong>Weak answer:</strong> "I'd look at the logs, find the bug and fix it." Or: "Roll back immediately."</p>
<p><strong>A stronger answer:</strong></p>
<ol>
<li><strong>Scope:</strong> Is it all checkout requests or one endpoint? Did the error rate jump at the moment the new pods went live? Are payments failing, or only something cosmetic?</li>
<li><strong>First check:</strong> Compare the error graph with the rollout timeline and split errors by version. If the old pods are clean and the new ones fail on the same traffic, the release is the main suspect.</li>
<li><strong>Decide:</strong> The key question is the migration. If it only added a nullable column or a new table, and the old code works with the new schema and with any rows the new code has already written, I roll back the application and leave the migration in place. If the migration dropped or renamed a column the old code needs, a rollback makes things worse, so I fix forward, or turn the feature off behind its flag if the flag really avoids the broken path.</li>
<li><strong>Risk I accept:</strong> Rolling back loses the new feature until Monday and may annoy the team that wanted it out. That is cheaper than a weekend of failed checkouts.</li>
<li><strong>Communicate:</strong> In the incident channel: "Checkout errors at 2.5% since the 16:40 deploy of v2.31. Rolling back the app now; the migration stays because it is additive. Next update 17:15." Then tell support, because they will get the customer emails first.</li>
<li><strong>Follow up:</strong> Why did the canary or the staging run not catch it? Should migrations follow an expand-and-contract sequence, so the previous release always works with the current schema?</li>
</ol>
<h3>2. The database disk is almost full</h3><blockquote>
<p>The primary Postgres instance is at 95% disk and growing by about 1% every ten minutes. What do you do?</p>
</blockquote>
<p><strong>Weak answer:</strong> "I'd delete old files to free up space."</p>
<p><strong>A stronger answer:</strong></p>
<ol>
<li><strong>Scope:</strong> At this rate there are about 50 minutes left. What happens at 100%? Writes fail, and Postgres may shut down. That makes this urgent but not a reason to act blindly.</li>
<li><strong>First check:</strong> What is growing? The usual suspects are WAL files held back by a replication slot whose replica is down, a long-running query writing temp files, a runaway table or log, or a bulk import. <code>pg_replication_slots</code> and the size of the WAL directory answer the first question quickly.</li>
<li><strong>Decide:</strong> Buy time with a safe move first. On many managed databases and cloud disks you can grow storage online; on a self-managed disk you also have to extend the filesystem. Check that the resize will finish before the disk fills. Growing is one-way (you usually cannot shrink it back), but it does not risk the data. I do not delete files I do not understand: deleting WAL by hand can corrupt the database.</li>
<li><strong>The trade-off they are fishing for:</strong> If an inactive replication slot is holding the WAL, dropping it lets Postgres remove that WAL at the next checkpoints. But whatever used the slot (a replica, or a logical replication or CDC consumer) may no longer be able to catch up and may need a rebuild or a resync. That is a real decision: I find out who owns the slot, drop it only if the disk cannot grow in time, and say so out loud.</li>
<li><strong>Communicate:</strong> Tell the database owner and the teams whose writes would fail, with the time left and what I am doing. Update when space is secured.</li>
<li><strong>Follow up:</strong> An alert on disk growth rate, not only on percentage. An alert on replication slot lag. A runbook for this exact case.</li>
</ol>
<h3>3. The Terraform plan that replaces production</h3><blockquote>
<p>The release pipeline is blocked on a Terraform plan. It shows <code>-/+ destroy and then create replacement</code> for the production RDS instance. The release manager asks you to approve it so the release can go out.</p>
</blockquote>
<p><strong>Weak answer:</strong> "The plan is what the code says, so I'd approve it."</p>
<p><strong>A stronger answer:</strong></p>
<ol>
<li><strong>Scope:</strong> A <code>-/+</code> replacement destroys the existing database before it creates the new one. Unless a restore and a cutover are planned, that means downtime and probably data loss. That is not a routine change, whatever the pipeline says.</li>
<li><strong>First check:</strong> Read which attribute the plan marks with <code># forces replacement</code>. Some attributes cannot change in place, so changing them forces a new instance. I also check the rest of the plan: if someone renamed the resource or moved it into a module, Terraform shows a destroy at the old address and a create at the new one instead.</li>
<li><strong>Decide:</strong> Do not approve. If it is a rename or a refactor, the fix is a <code>moved</code> block (or <code>terraform state mv</code>) so Terraform knows it is the same resource. If the attribute really cannot change in place, that becomes a planned migration with a snapshot, a restore and a maintenance window, not a pipeline click.</li>
<li><strong>Risk I accept:</strong> The release is late, and the release manager is unhappy. I can live with that. I cannot undo a destroyed production database.</li>
<li><strong>Communicate:</strong> Tell the release manager directly: "The plan would replace the production database because of X. I am blocking it; the fix is a <code>moved</code> block, and I expect to have a clean plan in an hour."</li>
<li><strong>Follow up:</strong> Add <code>lifecycle { prevent_destroy = true }</code> to stateful resources. It only protects a resource while its configuration exists, so also make the pipeline fail on any plan that deletes or replaces a resource tagged as stateful.</li>
</ol>
<h3>4. A leaked cloud key</h3><blockquote>
<p>A developer pushed an AWS access key to a public GitHub repository 20 minutes ago. What do you do?</p>
</blockquote>
<p><strong>Weak answer:</strong> "Remove the commit from the history and force-push."</p>
<p><strong>A stronger answer:</strong></p>
<ol>
<li><strong>Scope:</strong> Public repositories are scanned for keys within minutes, so assume it has already been copied, and treat it as a security incident from the start. Removing the commit does not help with that.</li>
<li><strong>First check:</strong> Which identity does the key belong to, and what can it do? I start a CloudTrail search for its use across accounts and regions, knowing that events arrive with a delay and that a clean result does not prove the key was not used.</li>
<li><strong>Decide:</strong> Deactivate the key now, even though something that depends on it will break. This is the case where the disruptive move is the right one, because the alternative is an attacker with unknown access for an unknown time. Then I check whether anyone used it to create other credentials, roles or temporary sessions, because deactivating the key does not cancel temporary credentials already issued with it.</li>
<li><strong>Risk I accept:</strong> The service using that key goes down until it has a new credential. I say that before I do it, not after.</li>
<li><strong>Communicate:</strong> Tell security and the owner of the affected service right away, and follow the security incident process.</li>
<li><strong>Follow up:</strong> Secret scanning on push, short-lived credentials (for example OIDC federation from CI instead of stored keys), and a check of what else that identity could reach.</li>
</ol>
<h3>5. Pressure to skip a check</h3><blockquote>
<p>A release is promised to a customer today. The security scan in the pipeline fails on a dependency, and the release manager asks you to disable the scan so the release can ship.</p>
</blockquote>
<p><strong>Weak answer:</strong> "I'd disable it, ship, and fix it next sprint." Or: "Absolutely not, security comes first."</p>
<p><strong>A stronger answer:</strong></p>
<ol>
<li><strong>Scope:</strong> What exactly is failing? A known exploitable vulnerability in code we actually call is very different from a low-severity finding in a test-only dependency, or a flaky scanner.</li>
<li><strong>First check:</strong> Read the finding: severity, whether the vulnerable path is reachable, and whether a patched version exists.</li>
<li><strong>Decide:</strong> Do not silently turn the scan off. If the finding is real and reachable, try a patched version first; if there is none, ship without the affected feature, but only if that really removes the vulnerable code from the build. If it is low risk or a false positive, use a documented, time-limited exception for that one finding, with an owner and an expiry date, instead of disabling the whole check.</li>
<li><strong>Risk I accept:</strong> With an exception, we knowingly carry one finding for a few days. That is acceptable only if the risk is assessed, someone with the authority to accept security risk approves it, and it has an owner and an expiry date.</li>
<li><strong>Communicate:</strong> Put the decision and who approved it in writing, in the ticket or the release notes. Interviewers listen for this, because unrecorded exceptions are how temporary bypasses become permanent.</li>
<li><strong>Follow up:</strong> Review why a release promise was made without time for the pipeline to pass, and make exceptions expire automatically.</li>
</ol>
<h2>How to practice making the call</h2><p>Reading good answers helps a little. Saying your own answers out loud, under a time limit, helps a lot more. A routine that works:</p>
<ol>
<li><strong>Build a deck of scenarios.</strong> Write 20 short prompts from your own incidents, from public postmortems, and from the questions above. Keep each one to three or four sentences, like a real interviewer would.</li>
<li><strong>Answer out loud in three minutes.</strong> Use a timer and record yourself. Three minutes forces you to pick a first check instead of listing ten.</li>
<li><strong>Score yourself with the rubric table.</strong> Did you scope? Did you name one first check and what it rules out? Did you say the risk you accept? Did you write the update message? Most people find they skip communication and follow-up.</li>
<li><strong>Change one fact and answer again.</strong> This is the most useful step. Make the migration destructive. Make it 3 a.m. and you are alone. Make the VP say "do not roll back". If your answer does not change when the facts change, you are reciting, not deciding.</li>
<li><strong>Practice with someone who pushes back.</strong> A friend or colleague who interrupts with "the logs show nothing, now what?" is closer to a real interview than any list of questions.</li>
<li><strong>Write the message, literally.</strong> For each scenario, write the two-line update you would post in the incident channel. It takes thirty seconds and it is the part most candidates forget.</li>
</ol>
<p>You can also practice parts of this in tools that put you under time pressure. The <a href="https://devops-daily.com/games/incident-commander-simulator">incident commander simulator</a> makes you triage and decide while the impact grows. The <a href="https://devops-daily.com/interview-questions/practice">interview questions practice mode</a> has scenario-style questions on topics like <a href="https://devops-daily.com/interview-questions/topic/sre#incident-postmortem">incident postmortems</a> and <a href="https://devops-daily.com/interview-questions/topic/cicd#progressive-delivery-rollback-strategy">progressive delivery rollbacks</a>, and the <a href="https://devops-daily.com/flashcards/on-call-rotation-escalation-policy">on-call rotations flashcards</a> cover the vocabulary. If you want feedback on full written answers, the mock interviews on <a href="https://learning.devops-daily.com" rel="noopener noreferrer">DevOps Daily Pro</a> ask a follow-up question about your answer, the way a real interviewer would.</p>
<blockquote>
<p><strong>Note</strong></p>
<p>If you are the one running interviews, the same structure works in reverse. Give a short scenario, let the candidate scope it, then change one fact halfway through. The way they update their decision tells you more than any definition question.</p>
</blockquote>
<h2>Summary</h2><p>Situational questions are not a trick. They are a way to watch you work without a real outage. The interviewer wants to see you scope the problem, pick a first check that splits it in half, choose the move you can undo, say the cost of that choice, tell the right people, and fix the class of failure afterwards. You get better at that the same way you get better at incidents: by making the call many times, with the facts changing each time. Build a small deck, answer out loud against a timer, change one fact and answer again. Practice until you use the structure without thinking about it, so in the interview you can think about the problem instead of about how to sound senior.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[How YouTube, Slack and GitHub Shard MySQL: We Built the Vitess Pattern on Five Servers and Measured What It Costs]]></title>
      <link>https://devops-daily.com/posts/how-youtube-slack-github-shard-mysql-vitess</link>
      <description><![CDATA[YouTube built Vitess, and Slack and GitHub use it for their largest MySQL workloads. We built the same primitives over five real MySQL servers and measured them: a scatter read took 4 shard queries and got a quarter of the throughput, every simulated half-finished cross-shard transfer lost money, an online split lost no rows, and a hot channel stayed hot at 128 shards.]]></description>
      <pubDate>Mon, 12 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/how-youtube-slack-github-shard-mysql-vitess</guid>
      <category><![CDATA[DevOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[DevOps]]></category><category><![CDATA[MySQL]]></category><category><![CDATA[Vitess]]></category><category><![CDATA[Sharding]]></category><category><![CDATA[Databases]]></category><category><![CDATA[Scalability]]></category><category><![CDATA[PlanetScale]]></category>
      <content:encoded><![CDATA[<p>Every large MySQL shop tells the same story with different numbers. <a href="https://vitess.io/docs/overview/history/" rel="noopener noreferrer">Vitess</a> was created at YouTube in 2010, after replicas and application-level sharding stopped being enough, to put a routing layer in front of many MySQL servers. Slack moved <a href="https://slack.engineering/scaling-datastores-at-slack-with-vitess/" rel="noopener noreferrer">99% of its MySQL traffic</a> onto Vitess, peaking at 2.3 million queries per second. GitHub split its main database by domain, and by 2023 reported <a href="https://github.blog/engineering/infrastructure/upgrading-github-com-to-mysql-8-0/" rel="noopener noreferrer">horizontally sharded Vitess clusters</a> for the large domains that outgrew one primary.</p>
<p>The engineering blogs explain the architecture. What they rarely show is what each piece costs. So we built the core primitives ourselves, a hash vindex, a router, a lookup vindex and an online reshard, over five real MySQL 8.4 servers, loaded 200,000 chat messages and measured. The router is about 170 lines of Python, so you can read every decision.</p>
<h2>TLDR</h2><ul>
<li><strong>The sharding key decides the price of a query.</strong> Fetching one message by id with its channel known took 1 shard query, and our setup served 1,194 of those reads per second. Without the channel, the router had to ask all 4 shards: 307 reads per second, about a quarter.</li>
<li><strong>A lookup vindex buys part of it back:</strong> 2 shard queries per read and 710 reads per second, 2.3x the scatter version and 59% of the targeted one, at the cost of an extra write on every insert.</li>
<li><strong>Cross-shard transactions without atomic commit can leave partial results.</strong> We simulated a failure between the two commits in 52 of 2,000 transfers between accounts on different shards, and all 52 lost their 10 credits. The same failure on single-shard transfers lost nothing.</li>
<li><strong>An online split lost nothing.</strong> We split a shard in two, all three tables, while a writer kept inserting and a reader kept querying: 0 write errors, every one of 2,158 lookups found its message, 2,158 scatter reads returned no duplicates, and at the end every expected message id existed exactly once, on the right shard. Insert latency roughly doubled while the copy ran.</li>
<li><strong>More shards do not fix a hot key.</strong> The busiest channel held 12.3% of all messages. At 128 shards, where the ideal is 0.78% per shard, the busiest shard still held 14.6%.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Familiarity with MySQL and with what replication and a primary are.</li>
<li>To run the demo: Linux with <code>sudo</code>, <code>curl</code> and about 3 GB of disk. It builds an Ubuntu chroot with MySQL inside, so nothing is installed on your host. The setup downloads the <code>arm64</code> image (we used a Raspberry Pi 4); change <code>IMG</code> in <code>scripts/00-setup.sh</code> on x86.</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/mysql-shard-router" rel="noopener noreferrer">The-DevOps-Daily/mysql-shard-router on GitHub</a></p>
<h2>Why these companies shard MySQL at all</h2><p>Read replicas fix read load. Once writes or the dataset outgrow the biggest primary you can buy, after you have tuned what you can, horizontal sharding is what is left. The Vitess history page puts YouTube's problem in that order: replicas came first, then "write traffic became too high for the primary database to handle, requiring YouTube to shard data." Connections were the other pressure. The Vitess docs put each MySQL connection at between 256 KB and almost 3 MB of memory, and Vitess's answer was to multiplex many light client connections onto a small pool of real ones.</p>
<p>Resharding was the painful part. Sugu Sougoumarane, one of Vitess's creators, <a href="https://www.percona.com/blog/2016/03/10/percona-live-featured-talk-with-sugu-sougoumarane-vitess-the-complete-story/" rel="noopener noreferrer">told Percona</a> about YouTube's first one: "It took an entire night of master downtime."</p>
<p>Slack's reason was a bad sharding key. Its original design put each workspace on one shard with all of its data. As big customers grew, their shards "reached the largest available hardware", while the fleet was "hitting some major hotspots, while also massively underutilizing the majority" of the databases. Slack re-sharded messages by channel id, migrating from the fall of 2016 to the end of 2020 with double writes and a parallel read-diffing system to compare old and new.</p>
<p>GitHub did the cheaper thing first. In 2021 it <a href="https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/" rel="noopener noreferrer">described</a> splitting its main <code>mysql1</code> cluster by <strong>schema domain</strong>, groups of tables that are queried together, with linters that stopped a query from joining tables across domains. It moved its 130 busiest tables at once. In 2019 <code>mysql1</code> answered 950,000 queries per second; by 2021 the split clusters answered 1.2 million and the average load per host had halved. By 2023 GitHub ran 1,200+ MySQL hosts, 300+ TB and 5.5 million queries per second, with horizontally sharded Vitess clusters for the large domains that outgrew a single primary.</p>
<h2>The model: keyspace IDs, vindexes and ranges</h2><p>Vitess does not hash a row straight to a server. It maps the sharding key to a <strong>keyspace ID</strong> (with a 64-bit hash vindex, as in our demo, that is 8 bytes), and each shard owns a range of that ID space. The function that does the mapping is a <strong>vindex</strong>. The primary vindex is to sharding what the primary key is to a table: every sharded table needs one.</p>
<p>Shard names are the ranges, in hex: <code>-40</code> owns IDs from <code>0x00</code> up to but not including <code>0x40</code>, and <code>c0-</code> owns <code>0xc0</code> to the end. Four equal shards are <code>-40</code>, <code>40-80</code>, <code>80-c0</code> and <code>c0-</code>. Splitting <code>c0-</code> gives <code>c0-e0</code> and <code>e0-</code>, and no row outside that range has to move. That is the property that makes resharding possible without rehashing everything.</p>
<ol>
<li><strong>channel_id 1</strong> sharding key</li>
<li><strong>hash vindex</strong> 8-byte keyspace ID</li>
<li><strong>0xf6fc42…</strong> falls in c0-</li>
<li><strong>shard c0-</strong> MySQL :3314</li>
</ol>
<h2>What we built</h2><p>Five MySQL 8.4 servers, one per shard plus one spare for the reshard, all on one Raspberry Pi 4 inside an Ubuntu chroot. A Python router plays the part of VTGate: it hashes the key, picks the shard, and for queries without a key, sends the query to every shard in parallel and merges the results.</p>
<ol>
<li><strong>client threads</strong></li>
<li><strong>router</strong> vindex + shard map</li>
</ol>
<ul>
<li><strong>Raspberry Pi 4</strong> Ubuntu 26.04 chroot<ul>
<li><strong>-40</strong> mysqld :3311</li>
<li><strong>40-80</strong> mysqld :3312</li>
<li><strong>80-c0</strong> mysqld :3313</li>
<li><strong>c0-</strong> mysqld :3314</li>
<li><strong>spare</strong> mysqld :3315</li>
</ul>
</li>
</ul>
<p>The data is a chat app, like Slack's messages keyspace: 200,000 messages in 2,000 channels, sharded by <code>channel_id</code>. Channel activity follows a Zipf curve, as real chat, ticket and order data does: the busiest channel has 12.3% of all messages, and the top 10 have 35.8%. The core of the router is short:</p>
<pre><code class="hljs language-python" tabindex="0"><span class="hljs-keyword">def</span> <span class="hljs-title function_">keyspace_id</span>(<span class="hljs-params">key: <span class="hljs-built_in">int</span></span>) -&gt; <span class="hljs-built_in">bytes</span>:
    <span class="hljs-comment"># Stand-in for Vitess's hash/xxhash vindex: any stable, well-mixed 64-bit hash works.</span>
    <span class="hljs-keyword">return</span> hashlib.blake2b(<span class="hljs-built_in">str</span>(key).encode(), digest_size=<span class="hljs-number">8</span>).digest()

<span class="hljs-keyword">def</span> <span class="hljs-title function_">port_for</span>(<span class="hljs-params">self, key</span>):
    ksid = keyspace_id(key)
    <span class="hljs-keyword">for</span> name, port <span class="hljs-keyword">in</span> shards.items():      <span class="hljs-comment"># {"-40": 3311, "40-80": 3312, ...}</span>
        <span class="hljs-keyword">if</span> in_range(ksid, name):
            <span class="hljs-keyword">return</span> port
</code></pre><p>All five servers share one machine and one SD card, so the absolute numbers are higher and noisier than on real hosts, and every shard competes for the same four CPU cores. Read the ratios, not the milliseconds.</p>
<h2>1. The sharding key decides the price of every query</h2><p>We measured two access patterns. <strong>A</strong> fetches one message by id, from the same 2,000 random ids in all three variants: with the channel known (targeted), through a lookup vindex, or by scattering to every shard. <strong>B</strong> fetches the latest 50 messages, by channel (the sharding key, with channels picked by the same Zipf weights as the data) or by user (not the sharding key, so scatter and merge). For latency, each type ran 1,000 times, interleaved in random order after a warm-up. For throughput, each type ran for 15 seconds on 8 client threads:</p>
<table>
<thead>
<tr>
<th>Query</th>
<th>Shard queries</th>
<th>p50</th>
<th>p99</th>
<th>Queries/s</th>
</tr>
</thead>
<tbody><tr>
<td>A. one message by id, channel known</td>
<td>1</td>
<td>1.27 ms</td>
<td>2.27 ms</td>
<td>1,194</td>
</tr>
<tr>
<td>A. one message by id, lookup vindex</td>
<td>2</td>
<td>2.12 ms</td>
<td>3.67 ms</td>
<td>710</td>
</tr>
<tr>
<td>A. one message by id, scatter</td>
<td>4</td>
<td>3.06 ms</td>
<td>5.75 ms</td>
<td>307</td>
</tr>
<tr>
<td>B. latest 50 by channel</td>
<td>1</td>
<td>4.90 ms</td>
<td>7.39 ms</td>
<td>253</td>
</tr>
<tr>
<td>B. latest 50 by user, scatter + merge</td>
<td>4</td>
<td>5.87 ms</td>
<td>9.37 ms</td>
<td>157</td>
</tr>
</tbody></table>
<p><strong>One message by id: queries per second on 4 shards</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>channel known (1 shard)</td>
<td>1194</td>
<td>targeted</td>
</tr>
<tr>
<td>lookup vindex (2 shard queries)</td>
<td>710</td>
<td>lookup</td>
</tr>
<tr>
<td>scatter (4 shards)</td>
<td>307</td>
<td>scatter</td>
</tr>
</tbody></table>
<p><em>8 client threads for 15 s per query type, the same 2,000 message ids in all three cases, four MySQL 8.4 shards and the router on one Raspberry Pi 4. Total shard queries per second stayed between 1,194 and 1,419 in all three cases; the difference is how many of them each read needs.</em></p>
<p>The latency gap is modest because the router asks the shards in parallel. The throughput gap is not. Whatever the query type, the whole setup handled between 1,194 and 1,419 shard queries per second. A scatter read spends four of them, so the same setup serves about a quarter as many reads. Add shards and a scatter read gets more expensive, not less, which is the opposite of why you sharded.</p>
<p>Be clear about what the ceiling is here: the router, its 8 client threads, its 8-worker scatter pool and all four MySQL servers share one 4-core Pi, so we measured the capacity of that whole system, not of MySQL alone. On separate hosts the arithmetic is the same in kind: every scatter read still takes a query slot on every shard host.</p>
<p>Ordered scatter queries can also over-fetch. "Latest 50 by user" asks each shard for up to 50 rows, so for a busy user it can fetch 200 and discard 150 in the merge. (Our users average 40 messages each, so most of these reads returned fewer.) Vitess has to do the same for <code>ORDER BY ... LIMIT</code> across shards.</p>
<p>This is how a scatter query took Slack down. In its <a href="https://slack.engineering/slacks-incident-on-2-22-22/" rel="noopener noreferrer">February 2022 incident</a>, a client boot query read a keyspace "sharded by user" without the sharding key, and Vitess ran it as a scatter query, which "sends the query to every shard". When a cache problem pushed more of those reads to the database, every shard took the load at once.</p>
<h2>2. A lookup vindex buys part of it back</h2><p>Sometimes you need to find a row by something other than the sharding key: a message by its id, an order by its number. Vitess's answer is a <strong>lookup vindex</strong>, a table that maps the other key to the keyspace ID. In our router it is <code>message_lookup</code>, sharded by message id, holding the channel:</p>
<pre><code class="hljs language-python" tabindex="0"><span class="hljs-keyword">def</span> <span class="hljs-title function_">by_id_lookup</span>(<span class="hljs-params">self, mid</span>):
    row = <span class="hljs-variable language_">self</span>.run(<span class="hljs-variable language_">self</span>.port_for(mid), <span class="hljs-string">"SELECT channel_id FROM message_lookup WHERE id=%s"</span>, (mid,))
    <span class="hljs-keyword">return</span> <span class="hljs-variable language_">self</span>.run(<span class="hljs-variable language_">self</span>.port_for(row[<span class="hljs-number">0</span>][<span class="hljs-number">0</span>]), <span class="hljs-string">f"SELECT <span class="hljs-subst">{COLS}</span> FROM messages WHERE id=%s"</span>, (mid,))
</code></pre><p>Two shard queries instead of four took throughput from 307 to 710 reads per second: 2.3x the scatter version, but still only 59% of the targeted read, because the lookup is a second round trip. The cost moves to writes: the <a href="https://vitess.io/docs/reference/features/vindexes/" rel="noopener noreferrer">Vitess docs</a> say a lookup vindex needs "an extra write to the lookup table for insert and delete operations". In our demo that write usually lands on a different shard than the message, so keeping the two consistent is its own problem; Vitess documents consistent lookup vindexes for that. Add lookup vindexes for the few access paths that are hot, not for every column.</p>
<h2>3. Cross-shard transactions without two-phase commit lose data</h2><p>Once data lives on several servers, a transaction that touches two shards is two transactions. Vitess's default transaction mode, <code>multi</code>, commits them one after the other with, in the <a href="https://vitess.io/docs/reference/features/distributed-transaction/" rel="noopener noreferrer">docs' words</a>, "no atomicity". We tested what that means with credit transfers between accounts. With a 2% probability per transfer, we simulate a failure between the two steps: the script commits the debit and rolls back the credit, as if the process had died after the first commit. For single-shard transfers, the same failure means rolling back the one transaction before it commits. This is a simulation of the failure, not a crash-recovery test.</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"same-shard"</span><span class="hljs-punctuation">:</span>  <span class="hljs-punctuation">{</span> <span class="hljs-attr">"transfers"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">2000</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"injected_failures"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">32</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"credits_before"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1000000</span><span class="hljs-punctuation">,</span>
                   <span class="hljs-attr">"credits_after"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1000000</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"credits_lost"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">0</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"p50_ms"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">3.26</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"p99_ms"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">484.49</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"cross-shard"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"transfers"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">2000</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"injected_failures"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">52</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"credits_before"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1000000</span><span class="hljs-punctuation">,</span>
                   <span class="hljs-attr">"credits_after"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">999480</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"credits_lost"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">520</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"p50_ms"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">5.91</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"p99_ms"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">467.95</span> <span class="hljs-punctuation">}</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>Every simulated cross-shard failure lost money: 52 failures (2.6% of transfers in this run), 520 credits gone, because the debit had committed and the credit never did. On one shard, the 32 failures rolled back cleanly. The cross-shard version also took longer at the median, 5.9 ms against 3.3 ms, because it waits for two commits.</p>
<p>Vitess has an atomic two-phase commit mode (<code>TWOPC</code>). When the <a href="https://vitess.io/blog/2024-10-29-announcing-vitess-21/" rel="noopener noreferrer">Vitess 21 announcement</a> reintroduced it in 2024 it was "currently in an experimental state"; check the current docs for its status. The docs also warn that it "does not provide isolation", so other readers can see a half-finished transfer, and that it adds commit latency. Teams that have written about sharding with Vitess designed around cross-shard transactions. Shopify's Shop app <a href="https://shopify.engineering/horizontally-scaling-the-rails-backend-of-shop-app-with-vitess" rel="noopener noreferrer">removed all cross-shard transactions</a> as part of sharding by user id. Cash App <a href="https://developer.squareup.com/blog/sharding-cash/" rel="noopener noreferrer">kept each customer's "entity group"</a> together on one shard through every split. The rule that falls out: pick a sharding key that keeps the rows a transaction touches on the same shard, and use an idempotent, retried workflow or two-phase commit, with its costs, for the rest.</p>
<h2>4. Resharding online, without the night of downtime</h2><p>This is the operation that cost YouTube a night. Vitess now does it online with VReplication: copy the rows in chunks from a consistent snapshot, stream the changes that happen meanwhile from the binlog, compare both sides, then switch traffic. We did an insert-only version in miniature, splitting <code>c0-</code> into <code>c0-e0</code> (staying on port 3314) and <code>e0-</code> (moving to the spare on 3315). All three tables moved: messages (keyed by channel), the lookup table (keyed by message id) and accounts. A writer thread inserted messages and their lookup rows the whole time, one at a time with a 5 ms pause between them. A reader thread looked up random seeded messages through the lookup vindex and checked that each was found, and ran scatter reads by user and checked them for duplicate rows:</p>
<ol>
<li><strong>Mirror:</strong> inserts for the moving range <code>e0-</code> go to both the old and the new shard. This stands in for VReplication's change stream, and it is the same double-write approach Slack used for its migration. Unlike VReplication it handles only inserts: no ordering of updates and deletes, and no replay after a crash.</li>
<li><strong>Copy:</strong> existing <code>e0-</code> rows move in chunks of 2,000. These are plain reads, not a consistent snapshot, which is safe here only because rows are never updated.</li>
<li><strong>Verify:</strong> for messages, the row count and an XOR of per-row CRC32 values (over id, channel, user and body) up to a watermark id must match on both sides; for the lookup and accounts rows, the count and a hash of the sorted rows. This is the job <a href="https://vitess.io/docs/reference/vreplication/movetables/" rel="noopener noreferrer">VDiff</a> does, more thoroughly.</li>
<li><strong>Switch:</strong> one atomic change to the router's in-memory shard map routes <code>e0-</code> to the new server. Scatter reads keep only rows inside each shard's range, so rows still sitting on the old shard are not returned twice. The script then waits 3 seconds before cleanup instead of draining in-flight requests, so the safety of the hand-over depends on timing; Vitess fences and buffers traffic during <code>SwitchTraffic</code> instead.</li>
<li><strong>Clean up:</strong> delete the moved rows from the old shard, then check the end state.</li>
</ol>
<p>From <code>results/reshard.json</code>:</p>
<pre><code class="hljs language-text" tabindex="0">copy            44,937 messages, 24,796 lookup rows, 129 accounts in 23.4 s
verify          44,963 messages up to the watermark: count and CRC32 XOR match;
                lookup and accounts rows identical
map update      0.026 ms (the in-process shard-map swap, not a client cutover)
cleanup         moved rows deleted from the old shard in 23.7 s
writer          994 inserts during the run, 0 errors
reader          2,158 lookups all found; 2,158 scatter reads, no duplicates
end state       every expected message id exactly once (200,994), 200,994 lookup
                rows, 1,000 accounts, 0 rows on the wrong shard or with a wrong ksid
</code></pre><p>Nothing was lost and no write failed. Because the writer waited for each insert before sending the next, a slower write simply meant fewer writes; this does not show how a system with a fixed incoming rate would queue. The cost showed up as latency: the writer's median insert (a message plus its lookup row, across all shards) went from 8.3 ms before the move to 16.5 ms during the copy and 21.3 ms during the cleanup. The copy and the deletes most likely competed with it for the shared SD card. Vitess ships a throttler that VReplication uses to limit exactly this kind of impact.</p>
<p>For comparison, the simple way is to stop writes, copy, switch and start again. We did not measure that. Our online copy took 23.4 seconds for about 70,000 rows while sharing the machine with the writer and reader, and copy time grows with the size of the shard.</p>
<h2>5. More shards do not fix a hot key</h2><p>Hashing spreads keys evenly, not load. Every message in a channel has the same sharding key, so a busy channel is a busy shard, however many shards you have. We took the real per-channel row counts and worked out the busiest shard's share for 2 to 128 equal shards:</p>
<p><strong>Share of all messages on the busiest shard</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>2</th>
<th>4</th>
<th>8</th>
<th>16</th>
<th>32</th>
<th>64</th>
<th>128</th>
</tr>
</thead>
<tbody><tr>
<td>ideal (1/n)</td>
<td>50%</td>
<td>25%</td>
<td>12.5%</td>
<td>6.25%</td>
<td>3.12%</td>
<td>1.56%</td>
<td>0.78%</td>
</tr>
<tr>
<td>busiest shard</td>
<td>55.27%</td>
<td>31.03%</td>
<td>22.42%</td>
<td>17.52%</td>
<td>15.67%</td>
<td>15.19%</td>
<td>14.58%</td>
</tr>
<tr>
<td>hot channel bucketed</td>
<td>50.1%</td>
<td>29.61%</td>
<td>16.87%</td>
<td>11.64%</td>
<td>9.88%</td>
<td>9.21%</td>
<td>8.34%</td>
</tr>
</tbody></table>
<p><em>200,000 messages, 2,000 channels with Zipf activity; the busiest channel holds 12.3%. Equal keyspace ranges. 'Hot channel bucketed' splits that one channel's messages over 16 sharding keys.</em></p>
<p>At 128 shards, where an even split would put 0.78% on each, the busiest shard still held 14.6%, because the busiest channel alone is 12.3%. We also modeled splitting that one channel's messages evenly over 16 sharding keys (for example, channel plus <code>message_id % 16</code>). That brought the busiest shard down to 8.3%, and the next-busiest channel, 6.1% of messages, became the new floor. Bucketing has its own cost: a read of the whole channel now has to ask up to 16 places and merge, which is the scatter problem from section 1 on a smaller scale.</p>
<p>Slack wrote up a related failure in 2023. In <a href="https://slack.engineering/the-query-strikes-again/" rel="noopener noreferrer">The Query Strikes Again</a>, about an incident in October 2022, one shard "contained 6% of the user's subscription data", and write load on it made the tablets run out of memory until "the kernel OOM-killed the MySQL process." That was a concentration of load on one shard during a bulk operation, not necessarily a single hot key, but the shape is the same: one shard takes far more than its share. The Vitess docs' advice for hot shards is to <a href="https://vitess.io/docs/reference/features/sharding/" rel="noopener noreferrer">split shards</a>, and that works for a hot range of many keys. For a single hot key, the fix is in the data model: bucket the key, or pick a different one.</p>
<h2>When to shard, and how to pick the key</h2><ul>
<li><strong>Do the cheap things first.</strong> Read replicas for read load. Functional partitioning by domain, as GitHub did, buys a lot of headroom before you need horizontal sharding at all, and it is easier to undo.</li>
<li><strong>Shard when one primary cannot take the writes or hold the data.</strong> That is the point YouTube, Slack and GitHub each reached.</li>
<li><strong>Pick the key from your most common queries.</strong> PlanetScale's guidance is to optimize for "the most common queries with the goal of avoiding cross-shard and cross-keyspace queries". Our numbers show why: a query with the key used 1 shard query, one without it used all of them.</li>
<li><strong>Keep transactions on one shard.</strong> Choose a key that groups the rows a transaction touches, like Cash App's entity groups, and do the rest with idempotent workflows.</li>
<li><strong>Measure key skew before you choose.</strong> Count rows and traffic per key value. If one value is a large share, consider bucketing it, and weigh that against the extra fan-out on reads of that key.</li>
</ul>
<h2>Vitess, PlanetScale and TiDB</h2><ul>
<li><strong>Vitess</strong> is the open source layer all of this comes from, a CNCF graduated project since November 2019. You run VTGate, VTTablet and the MySQL servers yourself, often on Kubernetes. HubSpot <a href="https://product.hubspot.com/blog/improving-reliability" rel="noopener noreferrer">reported</a> over 750 Vitess shards per data center, each a 3-instance MySQL cluster, all inside Kubernetes.</li>
<li><strong>PlanetScale</strong> runs Vitess as a managed service and says it builds its product "on top of Vitess, which provides a transparent sharding solution for MySQL". You get the same primitives, keyspaces, vindexes and online resharding workflows, without operating the layer.</li>
<li><strong>TiDB</strong> takes a different route. It is a distributed SQL database with a MySQL-compatible protocol, and its storage layer, TiKV, splits data into Regions automatically: <a href="https://docs.pingcap.com/tidb/stable/tikv-overview/" rel="noopener noreferrer">when a Region grows past 384 MiB</a> by default, it splits. You do not choose a sharding key or run reshards yourself. We did not test how it handles a single hot key.</li>
</ul>
<h2>What we could not conclude</h2><ul>
<li><strong>Absolute performance.</strong> Five servers on one Raspberry Pi with an SD card is not a production fleet. Commits wait on the card, which put write p99 in the hundreds of milliseconds throughout, before, during and after the reshard.</li>
<li><strong>Vitess's own overhead.</strong> Our router is Python, not VTGate. Real VTGate adds query planning and connection pooling, and we did not measure how its scatter queries compare with ours.</li>
<li><strong>Large reshards.</strong> We moved about 45,000 messages, or 70,000 rows across three tables. At hundreds of millions, copy throughput, replica lag and throttling dominate, and we did not test them.</li>
<li><strong>Real two-phase commit.</strong> We tested the default mode's failure, not Vitess's atomic mode.</li>
</ul>
<h2>Summary</h2><ul>
<li>Sharding is a routing problem first. A query the router can send to one shard costs one shard; a query it cannot narrow down costs all of them. On our four shards, that was 1,194 against 307 reads per second for the same message.</li>
<li>A lookup vindex more than doubles the throughput of the scatter read for one access path, at the price of an extra write and a second round trip.</li>
<li>Without two-phase commit, a cross-shard transaction that dies between commits leaves partial data every time. Design keys so transactions stay on one shard.</li>
<li>Online resharding (mirror, copy, verify, switch) moved about 70,000 rows across three tables with zero write errors and zero lost rows, while lookups kept finding their rows. The cost was higher insert latency while it ran.</li>
<li>Hashing does not fix a hot key. Measure skew before you pick the key, and bucket hot values.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Ubuntu 26.10 Moves cp, mv and rm to Rust. We Ran 162 Shell Commands Through GNU and uutils to See What Changes]]></title>
      <link>https://devops-daily.com/posts/ubuntu-26-10-rust-coreutils-what-changes</link>
      <description><![CDATA[Ubuntu 26.10 ships on October 15 with Rust coreutils for every tool. On the same system, 150 of our 162 shell test cases gave byte-identical results with GNU and uutils. The other 12 come down to four changes that can break a script (GNU detection, cut -c, timeout -s KILL, base64 -d) and five error messages.]]></description>
      <pubDate>Mon, 12 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/ubuntu-26-10-rust-coreutils-what-changes</guid>
      <category><![CDATA[Linux]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Linux]]></category><category><![CDATA[Ubuntu]]></category><category><![CDATA[Bash]]></category><category><![CDATA[Rust]]></category><category><![CDATA[Coreutils]]></category><category><![CDATA[Shell Scripting]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>Ubuntu 26.10 comes out on Thursday, October 15. Its <a href="https://documentation.ubuntu.com/release-notes/26.10/" rel="noopener noreferrer">release notes</a> say that "the default core utilities now run entirely on the Rust-based uutils implementation", and that the last GNU holdouts, <code>cp</code>, <code>mv</code> and <code>rm</code>, "have now been migrated". Every <code>ls</code>, <code>sort</code>, <code>date</code>, <code>cut</code> and <code>cp</code> in your shell scripts now runs Rust code.</p>
<p>Most of that move happened earlier than people think. In 26.04 LTS most tools already come from uutils, so the question for most teams is narrower: what is different in practice, and what will your scripts notice? We ran 162 shell test cases in chroots of the Ubuntu 24.04, 25.10, 26.04 LTS and 26.10 beta base images, and in 26.10 we ran each one twice: once with uutils and once with the GNU binaries Ubuntu still ships. Same files, same OS, only the implementation changed.</p>
<h2>TLDR</h2><ul>
<li>On the same 26.10 system, <strong>150 of 162 test cases gave byte-identical captured stdout, stderr and exit codes</strong> with GNU and uutils.</li>
<li><strong>The other 12 come down to four changes that can alter what a script does, plus five error messages:</strong><ul>
<li><code>ls --version</code> now says "uutils coreutils", so scripts that look for "GNU" to pick GNU or BSD flags take the wrong branch.</li>
<li><code>cut -c</code> counts characters, not bytes, on UTF-8 text. This is new in 26.10; 26.04 still counts bytes.</li>
<li><code>timeout -s KILL</code> exits with 124 instead of 137.</li>
<li><code>base64 -d</code> printed nothing for our invalid input, where GNU printed the bytes it decoded before failing.</li>
</ul>
</li>
<li><strong>The rest are message text:</strong> different wording and quote marks in errors. They matter only if a script greps stderr.</li>
<li><strong>If you are on 26.04 LTS,</strong> most of this already applies to you. 26.10 changes 6 of our cases: the version strings, <code>cut -c</code>, and three error messages.</li>
<li><strong>25.10 shipped real bugs that are gone now:</strong> <code>comm</code> dropped a line in up to 19 of 20 runs, and <code>date -r</code> ignored the file.</li>
<li><strong>Speed, on our Raspberry Pi 4:</strong> plain sorts and counting got faster (<code>sort</code> about 16 to 19%), hashing is the same, and multi-key sorts and walking many small files got slower (<code>sort -k</code> 35%, copying a 5,000-file tree 61%, <code>ls -lR</code> twice as slow).</li>
<li><strong>Two other 26.10 changes servers will notice:</strong> dbus-broker replaces dbus-daemon, and the default OpenSSH client no longer understands GSSAPI options.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A Linux machine with <code>sudo</code>, <code>curl</code>, Python 3 and about 2 GB of free disk if you want to run the tests yourself. No Docker is needed: the scripts use <code>chroot</code> on Ubuntu's base images. They download the <code>arm64</code> images (we ran them on a Raspberry Pi 4); change the file names in <code>scripts/01-fetch.sh</code> for <code>amd64</code>.</li>
<li>The repo:</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/ubuntu-2610-coreutils-check" rel="noopener noreferrer">The-DevOps-Daily/ubuntu-2610-coreutils-check on GitHub</a></p>
<h2>What changed, release by release</h2><p>We checked each base image with <code>dpkg-query</code> and by following the <code>/usr/bin</code> links (<code>results/inventory.txt</code> in the repo):</p>
<table>
<thead>
<tr>
<th>Release</th>
<th>coreutils implementation</th>
<th>Still GNU</th>
</tr>
</thead>
<tbody><tr>
<td>24.04.5 LTS</td>
<td>GNU 9.4</td>
<td>everything</td>
</tr>
<tr>
<td>25.10</td>
<td>uutils 0.2.2</td>
<td><code>cp</code>, <code>mv</code>, <code>rm</code>, <code>chmod</code>, <code>chown</code>, <code>df</code>, <code>true</code></td>
</tr>
<tr>
<td>26.04.1 LTS</td>
<td>uutils 0.8.0</td>
<td><code>cp</code>, <code>mv</code>, <code>rm</code>, <code>df</code>, <code>true</code></td>
</tr>
<tr>
<td>26.10 beta</td>
<td>uutils 0.12.0</td>
<td>nothing</td>
</tr>
</tbody></table>
<p>The GNU code has not gone away. Every release since 25.10 ships the <code>gnu-coreutils</code> package, with each tool installed under a <code>gnu</code> prefix: <code>gnuls</code>, <code>gnucp</code>, <code>gnusort</code>. In 26.10 that is 104 binaries, and it is what made a clean comparison possible.</p>
<p><strong>ubuntu 26.10 beta</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># ls --version | head -1</span>
<span class="hljs-built_in">ls</span> (uutils coreutils) 0.12.0
<span class="hljs-comment"># gnuls --version | head -1</span>
<span class="hljs-built_in">ls</span> (GNU coreutils) 9.10
<span class="hljs-comment"># readlink -f /usr/bin/cp</span>
/usr/lib/cargo/bin/coreutils/cp
</code></pre><h2>How we tested</h2><p>Each test is one line in <code>cases.tsv</code>, run with <code>bash -c</code> in a fresh copy of the same input files, with <code>LC_ALL=C.UTF-8</code> and <code>TZ=UTC</code>. The "releases" are chroots of Ubuntu's arm64 base images on one Raspberry Pi 4 host (kernel 6.12), with the host's <code>/dev</code> mounted read-only and a fresh <code>/proc</code>, running as root. That is enough for coreutils, which is userspace, but it says nothing about kernel, service or permission behavior. We wrote the cases from what shell scripts actually do: <code>ls -l</code> with time styles, <code>sort -h</code>, <code>-V</code> and <code>-k</code>, <code>date -d</code> parsing, <code>stat -c</code>, <code>cut</code>, <code>tr</code>, <code>sha256sum -c</code>, <code>timeout</code>, <code>mktemp</code>, <code>install -D</code>, <code>dd</code>, error messages for missing files and bad options, and so on.</p>
<p>We ran all 162 cases in five setups:</p>
<ul>
<li><code>gnu-24.04</code>: Ubuntu 24.04.5, GNU 9.4.</li>
<li><code>mixed-25.10</code>: Ubuntu 25.10.</li>
<li><code>mixed-26.04</code>: Ubuntu 26.04.1 LTS.</li>
<li><code>uutils-26.10</code>: Ubuntu 26.10 beta, as shipped.</li>
<li><code>gnu-26.10</code>: the same 26.10 rootfs with a directory of links to the <code>gnu*</code> binaries first in <code>PATH</code>.</li>
</ul>
<p>A case "differs" when its captured stdout, stderr or exit code is not byte-identical. Some cases filter output through <code>awk</code>, <code>head</code> or <code>sed</code>, or print a tool's exit code with <code>echo "rc=$?"</code> (so the captured exit code is the shell's), and file side effects are only checked where a case looks at them. Bash builtins (<code>echo</code>, <code>printf</code>, <code>test</code>) would hide the coreutils binary, so those cases call the binary through <code>env</code>.</p>
<pre><code class="hljs language-text" tabindex="0">gnu-26.10  vs uutils-26.10   12 of 162 differ   (same OS, only the implementation)
gnu-24.04  vs uutils-26.10   13 of 162 differ   (an LTS user jumping to 26.10)
gnu-24.04  vs mixed-26.04    12 of 162 differ   (LTS to LTS)
mixed-26.04 vs uutils-26.10   6 of 162 differ   (what 26.10 adds)
gnu-24.04  vs gnu-26.10       4 of 162 differ   (GNU 9.4 to 9.10 on its own; one is only a PID in a bash message)
</code></pre><h2>The four changes that can change what a script does</h2><h3>1. Scripts that detect GNU now take the BSD branch</h3><p>Portable scripts often check whether they run with GNU or BSD tools, because the flags differ (<code>date -d</code> vs <code>date -v</code>, <code>stat -c</code> vs <code>stat -f</code>). The most common check reads the version string:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-keyword">if</span> <span class="hljs-built_in">date</span> --version 2&gt;/dev/null | grep -q GNU; <span class="hljs-keyword">then</span>
  tomorrow=$(<span class="hljs-built_in">date</span> -d tomorrow +%F)      <span class="hljs-comment"># GNU flags</span>
<span class="hljs-keyword">else</span>
  tomorrow=$(<span class="hljs-built_in">date</span> -v+1d +%F)            <span class="hljs-comment"># BSD and macOS flags</span>
<span class="hljs-keyword">fi</span>
</code></pre><p>On 26.10, <code>date --version</code> prints <code>date (uutils coreutils) 0.12.0</code> (on 26.04, <code>0.8.0</code>). The <code>grep</code> fails, the script takes the BSD branch, and <code>date -v+1d</code> fails with an unknown-option error. Our <code>detect-gnu-date</code> and <code>detect-gnu-stat</code> cases printed <code>gnu-path</code> on 24.04 and <code>bsd-path</code> on 25.10, 26.04 and 26.10.</p>
<p>uutils implements the GNU flags, so the fix is to test for the feature, not the vendor:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-keyword">if</span> <span class="hljs-built_in">date</span> -d tomorrow +%F &gt;/dev/null 2&gt;&amp;1; <span class="hljs-keyword">then</span>
  tomorrow=$(<span class="hljs-built_in">date</span> -d tomorrow +%F)      <span class="hljs-comment"># GNU-style date: GNU or uutils</span>
<span class="hljs-keyword">else</span>
  tomorrow=$(<span class="hljs-built_in">date</span> -v+1d +%F)            <span class="hljs-comment"># BSD and macOS</span>
<span class="hljs-keyword">fi</span>
</code></pre><p>Search your repos for <code>--version | grep</code> and <code>--version 2&gt;&amp;1 | grep -q GNU</code>. This one is easy to miss, because nothing errors where the check runs. The script fails later, in a branch that was written for a Mac.</p>
<h3>2. <code>cut -c</code> counts characters now (new in 26.10)</h3><p><strong>cut -c1-4 on UTF-8 text</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># input: 'héllo wörld' and 'naïve café'</span>
<span class="hljs-comment"># GNU 9.4, GNU 9.10, and uutils 0.8.0 on 26.04</span>
$ <span class="hljs-built_in">cut</span> -c1-4 utf8.txt
hél
naï
<span class="hljs-comment"># uutils 0.12.0 on 26.10</span>
$ <span class="hljs-built_in">cut</span> -c1-4 utf8.txt
héll
naïv
</code></pre><p>GNU <code>cut</code> treats <code>-c</code> as bytes, so <code>é</code> (two bytes in UTF-8) takes two of the four positions. <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/cut.html" rel="noopener noreferrer">POSIX</a> says <code>-c</code> selects characters, and uutils 0.12.0 now does that, so the new behavior is the standard one. Either way, the output changed under the same command, and 26.04 LTS still behaves like GNU. If a script slices fixed-width fields out of text that can contain non-ASCII names, paths or log messages, the columns shift. Use <code>cut -b</code> when you mean bytes.</p>
<h3>3. <code>timeout -s KILL</code> exits with 124, not 137</h3><pre><code class="hljs language-text" tabindex="0">$ timeout -s KILL 1 sleep 5; echo "rc=$?"

GNU 9.4 / 9.10:   bash: line 1: ... Killed   timeout -s KILL 1 sleep 5
                  rc=137
uutils (25.10+):  rc=124
</code></pre><p>GNU documents this in <code>timeout --help</code>: it exits with 137 "if COMMAND (or timeout itself) is sent the KILL (9) signal (128+9)". uutils returns 124. A wrapper that treats 124 as "timed out" and anything else as "the command failed" now reports a timeout where it used to report a failure, or the other way around. Check for both. The default signal (TERM) returned 124 in every setup, and <code>--preserve-status</code> behaved the same everywhere.</p>
<h3>4. <code>base64 -d</code> on bad input prints nothing</h3><pre><code class="hljs language-text" tabindex="0">$ echo 'not*base64' | base64 -d; echo "rc=$?"

GNU:     &lt;two garbage bytes&gt;rc=1      stderr: base64: invalid input
uutils:  rc=1                         stderr: base64: error: invalid input
</code></pre><p>Both fail with exit code 1. GNU writes the bytes it managed to decode before it hit the bad character, and for this input uutils wrote nothing. We tested one short malformed input, so do not assume either tool decodes all-or-nothing on long streams. This only matters if something downstream reads partial output, but that is exactly the kind of thing that turns up in a script that pipes <code>base64 -d</code> into a file without checking the exit code.</p>
<h2>Error messages: different text, same exit codes</h2><p>The other differences are wording. Exit codes matched in every case below, so they only affect scripts that parse stderr or tests that compare it:</p>
<table>
<thead>
<tr>
<th>Case</th>
<th>GNU</th>
<th>uutils 0.12.0</th>
</tr>
</thead>
<tbody><tr>
<td><code>ls --bogus</code></td>
<td><code>ls: unrecognized option '--bogus'</code></td>
<td><code>error: unexpected argument '--bogus' found</code> plus a usage block</td>
</tr>
<tr>
<td><code>date -d 'not a date'</code></td>
<td><code>invalid date ‘not a date’</code> (curly quotes)</td>
<td><code>invalid date 'not a date'</code> (straight quotes)</td>
</tr>
<tr>
<td><code>env printf '%d' abc</code> (the binary, not the bash builtin)</td>
<td><code>‘abc’: expected a numeric value</code></td>
<td><code>'abc': expected a numeric value</code></td>
</tr>
<tr>
<td><code>ln -s</code> onto an existing file</td>
<td><code>...: File exists</code></td>
<td><code>...: Already exists</code></td>
</tr>
<tr>
<td><code>cp -n</code></td>
<td>warns that <code>-n</code> is non-portable</td>
<td>no warning</td>
</tr>
</tbody></table>
<p>The quote marks are worth knowing about: GNU uses Unicode curly quotes in a UTF-8 locale, so a <code>grep "invalid date 'not"</code> in a test suite matched nothing on GNU and matches now.</p>
<h2>A change that is not about Rust: <code>mv -n</code></h2><p>One difference shows up between GNU versions, before uutils is involved. On 24.04 (GNU 9.4), <code>mv -n</code> onto an existing file prints <code>not replacing</code> and exits 1. GNU 9.5 and later (see the <a href="https://lists.gnu.org/archive/html/coreutils-announce/2024-03/msg00000.html" rel="noopener noreferrer">9.5 release announcement</a>), and uutils, skip silently and exit 0. If you move from 24.04 to 26.04 or 26.10 and a script used <code>mv -n ... || handle_conflict</code>, the handler no longer runs.</p>
<h2>25.10 shipped bugs that 26.04 fixed</h2><p>The 25.10 results are a reminder of why interim releases exist. With uutils 0.2.2, 25.10 differed from 24.04 in 17 cases, and some were plain bugs:</p>
<ul>
<li><strong><code>comm</code> lost a line.</strong> <code>comm &lt;(sort dup.txt | uniq) &lt;(printf 'apple\ndate\n')</code> dropped <code>cherry</code> from the output. It is timing-dependent: in our batches of repeats it dropped the line 7 and 19 times out of 20 on 25.10, and never on 26.04 or 26.10 (<code>results/comm-repeat.txt</code> has the last batch).</li>
<li><strong><code>date -r FILE</code> ignored the file</strong> and printed the current time instead of the file's modification time.</li>
<li><strong>Error messages</strong> for <code>cat</code> and <code>sha256sum</code> on a missing file ended with Rust's <code>(os error 2)</code>.</li>
</ul>
<p>All three are fixed in 26.04 and 26.10, except that <code>stat</code> on a missing file still printed <code>(os error 2)</code> in 26.04; 26.10 fixed that too. 25.10 also gave a different order from GNU for <code>shuf --random-source</code> with the same source bytes. That is a compatibility difference rather than a bug, since nothing promises two implementations map random bytes the same way, and 26.04 matches GNU again. If you skipped 25.10, you never saw most of this.</p>
<h2>Speed</h2><p>We timed 15 workloads with GNU 9.10 and uutils 0.12.0 in the same 26.10 rootfs on a Raspberry Pi 4 (4 cores, 4 GB): 9 runs each, alternating the two implementations, inputs already in the page cache, the first run dropped as a warm-up. The inputs were a million random integers (10 MB), a million lines of text (19 MB), a 100 MiB binary file and a tree of 5,000 small files. Each pair ran GNU first, then uutils. The harness records time only: it does not check exit codes or compare outputs, so treat these as timings of the same commands, not a validated benchmark suite.</p>
<p><strong>uutils 0.12.0 run time as a share of GNU 9.10 (lower is faster)</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>wc -l</td>
<td>55%</td>
<td>faster</td>
</tr>
<tr>
<td>wc -w</td>
<td>77%</td>
<td>faster</td>
</tr>
<tr>
<td>sort | uniq -c</td>
<td>80%</td>
<td>faster</td>
</tr>
<tr>
<td>sort -n</td>
<td>81%</td>
<td>faster</td>
</tr>
<tr>
<td>sort -u</td>
<td>81%</td>
<td>faster</td>
</tr>
<tr>
<td>sort</td>
<td>84%</td>
<td>faster</td>
</tr>
<tr>
<td>base64</td>
<td>89%</td>
<td>about the same</td>
</tr>
<tr>
<td>sha256sum</td>
<td>102%</td>
<td>about the same</td>
</tr>
<tr>
<td>md5sum</td>
<td>102%</td>
<td>about the same</td>
</tr>
<tr>
<td>cut -d' ' -f2</td>
<td>107%</td>
<td>about the same</td>
</tr>
<tr>
<td>tr a-z A-Z</td>
<td>107%</td>
<td>about the same</td>
</tr>
<tr>
<td>sort -k3,3n -k1,1</td>
<td>135%</td>
<td>slower</td>
</tr>
<tr>
<td>cp -r + rm -rf (5,000 files)</td>
<td>161%</td>
<td>slower</td>
</tr>
<tr>
<td>du -s (5,000 files)</td>
<td>180%</td>
<td>slower</td>
</tr>
<tr>
<td>ls -lR (5,000 files)</td>
<td>198%</td>
<td>slower</td>
</tr>
</tbody></table>
<p><em>Median of 8 timed runs per workload, Ubuntu 26.10 beta rootfs, Raspberry Pi 4. 100% = same speed as GNU. Raw runs in results/bench.tsv.</em></p>
<table>
<thead>
<tr>
<th>Workload</th>
<th>GNU 9.10</th>
<th>uutils 0.12.0</th>
</tr>
</thead>
<tbody><tr>
<td><code>sort -n</code> on 1M integers</td>
<td>1.26 s</td>
<td>1.02 s</td>
</tr>
<tr>
<td><code>sort</code> on 1M lines</td>
<td>1.04 s</td>
<td>0.87 s</td>
</tr>
<tr>
<td><code>sort -k3,3n -k1,1</code> on 1M lines</td>
<td>1.83 s</td>
<td>2.47 s</td>
</tr>
<tr>
<td><code>wc -l</code> on 19 MB</td>
<td>0.039 s</td>
<td>0.021 s</td>
</tr>
<tr>
<td><code>sha256sum</code> on 100 MiB</td>
<td>0.79 s</td>
<td>0.81 s</td>
</tr>
<tr>
<td><code>cp -r</code> then <code>rm -rf</code> of 5,000 small files</td>
<td>0.50 s</td>
<td>0.81 s</td>
</tr>
<tr>
<td><code>ls -lR</code> of 5,000 small files</td>
<td>0.056 s</td>
<td>0.111 s</td>
</tr>
<tr>
<td><code>du -s</code> of 5,000 small files</td>
<td>0.020 s</td>
<td>0.036 s</td>
</tr>
</tbody></table>
<p>On this machine and these inputs the spread between runs was small (<code>results/bench-summary.tsv</code> has the ranges), and the pattern is consistent: plain sorts and counting are faster with uutils, hashing is the same, and multi-key sorts and anything that walks many small files are slower. Copying and deleting the tree took 61% longer, <code>ls -lR</code> twice as long. Results on x86 servers and other disks can differ. For most scripts the absolute numbers are small. A backup or deploy script that copies a large tree of small files is where you would notice.</p>
<p>A note on method: our first benchmark run wrote every command's output to a regular file, because the chroot had no <code>/dev</code> and <code>&gt;/dev/null</code> created one. That made output-heavy commands look slow on both sides. We threw those numbers away and reran with <code>/dev</code> mounted; the scripts in the repo do this.</p>
<h2>Two more 26.10 changes that servers notice</h2><h3>dbus-broker replaces dbus-daemon</h3><p><a href="https://discourse.ubuntu.com/t/ubuntu-26-10-is-switching-to-dbus-broker/84060/" rel="noopener noreferrer">Canonical announced in June</a> that 26.10 replaces dbus-daemon, the default since 2004, with dbus-broker for both the system bus and user session buses, on fresh installs and upgrades. The announcement calls it "a drop-in replacement. Same protocol, same config files, same bus policies." The archive shows the switch:</p>
<pre><code class="hljs language-text" tabindex="0">                          26.04 LTS                 26.10 beta
provides the system bus   dbus                      dbus-broker
dbus-broker               universe                  main
dbus, dbus-daemon         main                      universe
</code></pre><p>For most servers nothing visible changes. The thing to check is a container image or CI job that installs <code>dbus</code> by name to get a session bus with <code>dbus-run-session</code> (headless browser tests often do). In 26.10, <code>apt-get install dbus</code> still pulls in <code>dbus-daemon</code>, but both are in universe now, so an image with universe disabled cannot install them. The announcement also documents how to switch back.</p>
<h3>The default OpenSSH client no longer understands GSSAPI</h3><p>The release notes say OpenSSH has been split into <code>openssh</code> and <code>openssh-gssapi</code>, and that fresh installs get the version without GSSAPI and Kerberos. We installed the default client in each release and gave it the GSSAPI options that many existing configs carry:</p>
<pre><code class="hljs language-text" tabindex="0">26.04 (OpenSSH 10.2p1):  no GSSAPI warning, exit 0
26.10 (OpenSSH 10.5p1):  /root/.ssh/config line 2: Unsupported option "gssapiauthentication"
                         /root/.ssh/config line 3: Unsupported option "gssapidelegatecredentials"
                         exit 0
</code></pre><p>We tested this with <code>ssh -G</code>, which evaluates the configuration without connecting. It warns and carries on, so a config with these lines still loads, and every <code>ssh</code> run that reads them will print the warning on stderr. Kerberos authentication is gone with the GSSAPI build, as the release notes say; we did not test a real Kerberos login. The default <code>/etc/ssh/ssh_config</code> in 26.04 had <code>GSSAPIAuthentication yes</code> enabled; in 26.10 that line is gone. If you use Kerberos for SSH, install <code>openssh-client-gssapi</code> (and <code>openssh-server-gssapi</code> on servers). If you do not, remove the GSSAPI lines from configs you copy onto new machines, or the warnings will end up in every log of every Ansible run.</p>
<h2>How to check your own scripts before you upgrade</h2><ol>
<li><strong>Run them against 26.10 without upgrading anything.</strong> The repo downloads Ubuntu's base image and runs commands in a <code>chroot</code>. Put your script in <code>rootfs/26.10/work</code> and run it with <code>sudo chroot rootfs/26.10 /bin/bash -c 'cd /work &amp;&amp; ./your-script.sh'</code>. Mount <code>/dev</code> and <code>/proc</code> first, as <code>scripts/03-run-cases.sh</code> does.</li>
<li><strong>Compare with GNU on the same box.</strong> Put the <code>gnu*</code> binaries first in <code>PATH</code> with a directory of links (see <code>scripts/02-extract.sh</code>), run the script again, and diff the output.</li>
<li><strong>Grep for the patterns above:</strong> <code>--version | grep</code>, <code>cut -c</code> on text that may hold non-ASCII, <code>timeout -s KILL</code>, and error-message matching in tests.</li>
<li><strong>Switch back if you need to.</strong> Ubuntu's <a href="https://documentation.ubuntu.com/release-notes/26.04/summary-for-lts-users/" rel="noopener noreferrer">26.04 notes</a> give the commands. They also apply to 26.10:</li>
</ol>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># to GNU coreutils</span>
<span class="hljs-built_in">sudo</span> apt install coreutils-from-gnu coreutils-from-uutils- --allow-remove-essential
<span class="hljs-comment"># back to Rust coreutils</span>
<span class="hljs-built_in">sudo</span> apt install coreutils-from-uutils coreutils-from-gnu- --allow-remove-essential
</code></pre><p>For a single script, calling <code>gnudate</code>, <code>gnucut</code> or <code>gnutimeout</code> directly is enough.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>Locales other than C.UTF-8.</strong> The base images ship no other locales, and <code>sort</code> and <code>ls</code> ordering depend on the locale. Test <code>en_US.UTF-8</code> on a full install if your scripts depend on sort order.</li>
<li><strong>Long-running and terminal behavior.</strong> We did not test <code>tail -f</code>, <code>stty</code>, <code>dd</code> progress output or anything that needs a TTY.</li>
<li><strong>Your edge cases.</strong> 150 of 162 identical test cases is a good sign, not a guarantee. The cases are in a text file in the repo; add yours and send them back.</li>
</ul>
<h2>Summary</h2><ul>
<li>26.10 finishes a move that 26.04 LTS mostly made: <code>cp</code>, <code>mv</code> and <code>rm</code> are Rust now, and uutils jumps from 0.8.0 to 0.12.0.</li>
<li>On the same system, 150 of 162 shell test cases gave identical captured results with GNU and uutils.</li>
<li>Fix these before you upgrade: GNU detection by version string, <code>cut -c</code> on non-ASCII text, scripts that expect 137 from <code>timeout -s KILL</code>, and anything that parses error messages.</li>
<li>Moving from 24.04, also check <code>mv -n</code>: its exit code changed in GNU itself.</li>
<li>Check container images that install <code>dbus</code> by name (it is in universe now), and remove GSSAPI lines from SSH configs unless you install <code>openssh-client-gssapi</code>.</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DevOps Weekly Digest - Week 42, 2026]]></title>
      <link>https://devops-daily.com/news/2026-week-42</link>
      <description><![CDATA[⚡ Curated updates from Kubernetes, cloud native tooling, CI/CD, IaC, observability, and security - handpicked for DevOps professionals!]]></description>
      <pubDate>Mon, 12 Oct 2026 00:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/news/2026-week-42</guid>
      <category><![CDATA[DevOps News]]></category>
      <content:encoded><![CDATA[<blockquote>
<p>📌 <strong>Handpicked by DevOps Daily</strong> - Your weekly dose of curated DevOps news and updates!</p>
</blockquote>
<hr />
<h2>⚓ Kubernetes</h2><h3>📄 Introducing DigitalOcean Insights: Built-in Observability for Your Agents, Applications, and Infrastructure</h3><p>Coding agents have changed how fast teams ship, and more of the software they ship now includes agents. A production agent spans model calls, routing decisions, tools, and the infrastructure underneat</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/introducing-insights" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Who owns NIS2 and DORA on a Kubernetes platform team</h3><p>There is a meeting that repeats across a lot of European engineering organizations right now. Someone from legal or risk brings a NIS2 control, a DORA article, or a “can we evidence this” request into</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/09/who-owns-nis2-and-dora-on-a-kubernetes-platform-team/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Actionable disconnected cluster analysis, available now with pen drive powered by Red Hat Lightspeed</h3><p>Red Hat OpenShift doesn't stop at the cloud boundary. Telecommunications providers, government agencies, and financial institutions run OpenShift in environments that are intentionally air-gapped — an</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/actionable-disconnected-cluster-analysis-available-now-pen-drive-powered-red-hat-lightspeed" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Shift to cgroup v2 in Kubernetes: What You Need to Know</h3><p>In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications </p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 Kubernetes Blog</strong></p>
<p><a href="https://kubernetes.io/blog/2026/10/06/kubernetes-cgroups-v2-shift/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Scaling Kubernetes Workloads with Node Swap</h3><p>Memory is often the first hard limit a Kubernetes cluster hits. Nodes run out of RAM long before they run out of CPU, and the new wave of agentic AI workloads makes this worse. These workloads demand </p>
<p><strong>📅 Oct 5, 2026</strong> • <strong>📰 Kubernetes Blog</strong></p>
<p><a href="https://kubernetes.io/blog/2026/10/05/scaling-kubernetes-workloads-with-node-swap/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>☁️ Cloud Native</h2><h3>📄 Docker Hub security advisory: personal access tokens transferred during account-to-organization conversions</h3><p>CVE-2026-105445: Docker Hub account conversion could re-assign personal access tokens to a new owner. Read what happened, who was affected, and the fix.</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Docker Blog</strong></p>
<p><a href="https://www.docker.com/blog/docker-hub-security-advisory-cve-2026-105445/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 From zero to agent builder: Running kagent on your laptop</h3><p>If you have seen a demo of a no-code agent builder, you know the moment. Someone types one sentence, waits a minute, and there it is: a new agent, running, with tools. It is impressive. For...</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/12/from-zero-to-agent-builder-running-kagent-on-your-laptop/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Join Platform Engineering Day at KubeCon + CloudNativeCon North America 2026</h3><p>As organizations accelerate their adoption of AI and cloud native technologies, platform teams are being asked to solve a growing set of challenges. How can developers access new capabilities such as </p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/09/join-platform-engineering-day-at-kubecon-cloudnativecon-north-america-2026/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Making the Golden Kubestronaut a little more golden</h3><p>When I became a Golden Kubestronaut, I wanted to celebrate the achievement in a way that felt lasting. Like most technical achievements, certifications usually live digitally. You receive the badge, u</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CNCF Blog</strong></p>
<p><a href="https://www.cncf.io/blog/2026/10/09/making-the-golden-kubestronaut-a-little-more-golden/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Harness CI Pipeline Tutorial: Build and Push to Docker</h3><p>Step-by-step: connect GitHub, let Harness AI generate a CI pipeline for your Java project, add Docker push. Full pipeline ran in 3 minutes. | Blog</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/harness-ci-pipeline-tutorial-build-push-docker" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Docker Registry: Public, Private, and Pull-Through caching</h3><p>Harness Artifacts provides a secure, private, and cloud-native Docker registry where developers can store, proxy, and manage Docker images with enterprise-grade | Blog</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/what-is-a-docker-registry-public-private-and-pull-through-caching" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 OpenTelemetry Community Awards 2026</h3><p>We are delighted to announce the 3rd annual OpenTelemetry Community Awards this year! This project and its newly minted status as a graduated CNCF project would not be possible without many, many awes</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/community-awards/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔄 CI/CD</h2><h3>📄 GitLab Critical Patch Release: 19.4.2, 19.3.4, 19.2.8</h3><p><strong>📅 Oct 12, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-4-2-released/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Hack the World: Why hackathons are still the best place to learn to build</h3><p>The barriers to building software have collapsed. Today, anyone can build. Hackathons are a great place to start learning. The post Hack the World: Why hackathons are still the best place to learn to </p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/developer-skills/career-growth/hack-the-world-why-hackathons-are-still-the-best-place-to-learn-to-build/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Gitea Runner 5.0.0 is released</h3><p>We are happy to announce the release of <strong>Gitea Runner 5.0.0</strong>. This is a quiet major release. A handful of small breaking changes required the version bump, but most setups upgrade without changes. T</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Gitea Blog</strong></p>
<p><a href="https://blog.gitea.com/release-of-runner-5.0.0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How one bug bounty researcher chooses the features they investigate</h3><p>As we kick off Cybersecurity Awareness Month, the GitHub Bug Bounty team spotlights @vaib25vicky, exploring their methodology, techniques, and experiences hacking on GitHub. The post How one bug bount</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/security/how-one-bug-bounty-researcher-chooses-the-features-they-investigate/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How to Deploy Your First App with Harness CD in 15 Minutes</h3><p>A first deployment walkthrough for Harness CD using a real sample app. | Blog</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/deploy-your-first-app-with-harness-cd" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Track organization-wide security risk in one dashboard</h3><p>If you run application security across more than one top-level group, getting a single organization-wide view of your risk has meant manually pulling together data. That is operational work rebuilt in</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/security-risk-in-one-dashboard/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Secret protection must scale with software</h3><p>Developers aren’t becoming more careless; they’re being outpaced. The tools that let developers create more software should also take on more of the work of protecting it. The post Secret protection m</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/ai-and-ml/github-copilot/secret-protection-must-scale-with-software/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Building Git infrastructure for agent-scale development</h3><p>We're rebuilding GitHub's Git infrastructure while GitHub keeps running, creating a foundation for agent-scale software development. The post Building Git infrastructure for agent-scale development ap</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 GitHub Blog</strong></p>
<p><a href="https://github.blog/engineering/architecture-optimization/building-git-infrastructure-for-agent-scale-development/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 GitLab Transcend: Speed you can trust, all the way to production</h3><p>At Transcend, we made over a dozen announcements across all four layers of our architecture for agentic software engineering: agent orchestration, data and context, DevOps workflows, and governance an</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/transcend-india-announcements/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Dependency Firewall: Block risky packages before the build</h3><p>Attackers disguise malicious packages as ones you trust. In June 2026, GitLab researchers found five malicious PyPI packages, four of them typosquats of Flask, Requests, and NumPy, that run at install</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 GitLab Blog</strong></p>
<p><a href="https://about.gitlab.com/blog/transcend-dependency-firewall/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🏗️ IaC</h2><h3>📄 Amazon Bedrock now supports reasoning summaries for OpenAI models</h3><p>Amazon Bedrock now supports the reasoning.summary parameter for OpenAI models through the Responses API. This capability lets you request a human-readable summary of a model’s reasoning alongside its </p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-bedrock-reasoning-summaries-openai/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Concurrent change requests in Pulumi ESC</h3><p>With ESC Approvals, you can propose, review, and apply environment changes with change requests. Until now, an environment could only have one open change request at a time. If a teammate’s change was</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/esc-concurrent-change-requests/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Pulumi Google Cloud Provider Version 10.0.0</h3><p>TL;DR: we are releasing a new major version of the Google Cloud provider. To migrate, ask Neo or your coding agent to run the Pulumi provider-upgrade skill (how to use it) or follow the v10 migration </p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 Pulumi Blog</strong></p>
<p><a href="https://www.pulumi.com/blog/gcp-v10-release/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📊 Observability</h2><h3>📄 Introducing DigitalOcean Signals and Simulations: see how your agents and models really behave in production, then test the fix before it ships</h3><p>You ship an agent. It mostly works. Then the feedback loop becomes a support ticket: a user got the wrong answer, a tool call looped four times, a refund went out that shouldn’t have. Standard observa</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/introducing-signals-and-simulations" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Prometheus and OpenTelemetry interoperability in 2026: Survey results</h3><p>We ran a survey asking users of OpenTelemetry and Prometheus how they collect, process, and store metrics. The goal was to understand, with real usage data rather than assumptions, how far the ecosyst</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/otel-prometheus-interoperability/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 OTel-Native by Design - Building Products That Export to Any Observability Stack</h3><p>With contributions from Dan Gomez Blanco (New Relic). If you’re building self-hosted software or a SaaS product, your users will eventually ask to send logs, traces, and metrics to their own observabi</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/otel-native-by-design/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How New Relic engineers reliability at scale</h3><p>Learn how New Relicengineers reliability at scale with telemetry, resilient architecture, and automation to detect, contain and respond to failure.</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/observability/engineer-reliability-at-scale" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Sentry JavaScript SDK v11 adds tracing for all JS platforms</h3><p>Sentry's v11 JavaScript SDKs trace dependencies with a code transformer instead of OpenTelemetry instrumentation, adding Cloudflare Workers, Bun, and Deno</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/tracing-for-all-js-platforms/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Take Grafana Labs' 5th annual Observability Survey</h3><p>Over the past year, "agentic" has gone from an emerging buzzword to an integral part of worklife for many engineers—and that's just scratching the surface on all that's changed. With so much upheaval,</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/take-grafana-labs-5th-annual-observability-survey/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Manage your OpenTelemetry Collectors with Fleet Management in Grafana Cloud</h3><p>If you’ve built your telemetry pipelines around OpenTelemetry Collectors, you’ve already invested in a collector distribution, YAML configuration, and a deployment model that fits your infrastructure.</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/manage-your-opentelemetry-collectors-with-fleet-management-in-grafana-cloud/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Reducing Android scope-sync overhead in Sentry Flutter</h3><p>How Sentry Flutter uses a worker isolate for Android SDK operations and reduces duplicate normalization and JSON encoding overhead.</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Sentry Blog</strong></p>
<p><a href="https://blog.sentry.io/flutter-android-scope-sync/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Manage synthetic checks at scale: Introducing folders in Grafana Cloud Synthetic Monitoring</h3><p>As your use of Grafana Cloud Synthetic Monitoring grows, so does the number of checks you need to manage across services, environments, and teams. Eventually, a single flat list of checks becomes diff</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 Grafana Blog</strong></p>
<p><a href="https://grafana.com/blog/manage-synthetic-checks-at-scale-introducing-folders-in-grafana-cloud-synthetic-monitoring/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing AI Evaluation: Close the Loop Across the Full AI Lifecycle</h3><p>Discover New Relic AI Evaluation: Bridge pre-production prompt testing with live transaction telemetry to evaluate quality, reduce costs, and scale GenAI.</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/news/introducing-ai-evaluation" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Velocity Dilemma in Modern Software Delivery</h3><p>LaunchDarkly and New Relic close the loop between release and observability, automating rollbacks before incidents reach your users.</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/observability/launchdarkly-new-relic-close-the-loop" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing AI Observability for OpenTelemetry</h3><p>Unify OpenTelemetry AI with New Relic. Track token costs, discover shadow AI, and block prompt injections in real time without data duplication.</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 New Relic Blog</strong></p>
<p><a href="https://newrelic.com/blog/news/introducing-ai-observability-for-opentelemetry" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🔐 Security</h2><h3>📄 Threats Making WAVs - Incident Response to a Cryptomining Attack</h3><p>Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/threats-making-wavs-incident-reponse-cryptomining-attack" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Agentic data science with the Data Agent Kit</h3><p>Imagine your compliance team notices that payments are taking longer than expected to clear. They suspect fraud, and you’re the data scientist helping figure out which transactions to investigate firs</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/data-analytics/bringing-you-the-data-agent-kit/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Developer-first innovation: Reducing technical debt with Red Hat Enterprise Linux</h3><p>Managing software teams means balancing 2 conflicting priorities: developers want freedom to pick their tools, while leadership needs security and stability. When every team runs a different operating</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/developer-first-innovation-reducing-technical-debt-red-hat-enterprise-linux" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 AWS Security Hub now exports findings to S3 in CSV or JSON format</h3><p>Today, AWS Security Hub announces support for exporting findings to Amazon S3 in CSV or JSON (OCSF) format. Security teams that need findings outside the console for use cases such as compliance repor</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/security-hub-exports-s3-csv-json/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 2026-10-07: PostgreSQL JDBC 42.7.14 Security update for multiple CVE's</h3><p>The PostgreSQL JDBC team has released a security release for 2 CVE's CVE-2026-107314 and the GitHub Security Advisory GHSA-rhp9-mr79-r74h and CVE-2026-107315 and the GitHub Security Advisory GHSA-f64h</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/2026-10-07-postgresql-jdbc-42714-security-update-for-multiple-cves-3399/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why "secure by design" is the new standard for open source</h3><p>Open source software faces significant regulatory shifts, making "secure by design" development practices increasingly important. The Cyber Resilience Act (CRA) takes effect on December 11, 2027, and </p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/why-secure-design-new-standard-open-source" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Harness acquires Augment Code to advance the Autonomous SDLC</h3><p>Harness Cosmos Software Factory Agent automates engineering from idea to code, connecting code context with delivery, security, testing, and production workflow | Blog</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Harness Blog</strong></p>
<p><a href="https://www.harness.io/blog/harness-acquires-augment-code-cosmos" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Frontier models found the vulnerabilities. Only the attacker found the chains.</h3><p>Static analysis found the flaws, but only live attack testing proved how they could be chained into breaches. A comparison of Evo COS, Claude Security, and Claude Code Security.</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/frontier-models-vulnerabilities-attacker-chains/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Red Hat OpenShift Platform Plus bundle available on hyperscaler marketplaces</h3><p>Note: This post was updated on October 7th with the latest information.Organizations using Red Hat OpenShift cloud services are increasingly seeking ways to extend their platform’s capabilities with e</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 OpenShift Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/red-hat-openshift-platform-plus-rosa-aws-marketplace" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Autonomous Attacks Are Already Here. The Defense Has to Match Their Speed.</h3><p>Autonomous attackers are shrinking the window for defense. Learn how continuous discovery, remediation, validation, and prevention can help security teams keep pace.</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 Snyk Blog</strong></p>
<p><a href="https://snyk.io/blog/autonomous-attacks-already-here-defense-match-their-speed/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Gitea 28.1.0 is released</h3><p>We are excited to announce the release of <strong>Gitea 28.1.0</strong>, the first maintenance release for the 28 series. It contains security fixes and fixes for regressions reported after 28.0.0, including pushe</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 Gitea Blog</strong></p>
<p><a href="https://blog.gitea.com/release-of-28.1.0/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 What’s new in security for Ubuntu 26.10?</h3><p>Ubuntu 26.10 “Stonking Stingray” arrives at a time when AI tools are accelerating vulnerability discovery. Patching faster is essential, but the resilience of the underlying operating system must also</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/ubuntu-26-10-security" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>💾 Databases</h2><h3>📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers</h3><p>Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/please-read-me-opportunistic-ransomware-devastating-mysql-servers" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing TimesFM 3.0 with multivariate time-series forecasting in BigQuery</h3><p>Overview Since introducing zero-shot time-series forecasting in BigQuery with AI.FORECAST — initially powered by TimesFM 2.0 and later upgraded to TimesFM 2.5 — you have been able to generate forecast</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/data-analytics/bigquery-timesfm-3-0-with-multivariate-time-series-forecasting/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Filter, reshape, and redact logs before export with Transformers</h3><p>Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export. Powered by Apache DataFusion and Apache Arrow, Transformers apply those changes di</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/transformers-on-logpush/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Inside the Database Internals Talks at P99 CONF 2026</h3><p>Since database performance is so near and dear to ScyllaDB, we reached out to our friends across the community to ensure that a nice range of distributed data systems, approaches, and challenges would</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 ScyllaDB Blog</strong></p>
<p><a href="https://www.scylladb.com/2026/10/12/database-internals-talks-at-p99-conf/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Chaos engineering in practice: Lessons from enterprise customers</h3><p>Working with enterprise customers on chaos engineering has reinforced 1 lesson: Resilience is a business requirement. Failed transactions interrupt revenue, infrastructure outages affect multiple work</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Red Hat Blog</strong></p>
<p><a href="https://www.redhat.com/en/blog/chaos-engineering-practice-lessons-enterprise-customers" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How we built the Redis Docs MCP for agents</h3><p>Ask a coding agent how to configure eviction in Redis 8 and you'll usually get an answer. Less often than we'd like, that answer comes from authoritative docs. The agent either recalls something from </p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Redis Blog</strong></p>
<p><a href="https://redis.io/blog/how-we-built-the-redis-docs-mcp-for-agents/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Redis OSS Is Now Available on AWS Marketplace</h3><p>Redis Open Source Software (OSS) is now available through an official AWS Marketplace listing owned by Redis. The listing includes a Redis-published Amazon Machine Image (AMI), allowing teams to launc</p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 Redis Blog</strong></p>
<p><a href="https://redis.io/blog/redis-oss-is-now-available-on-aws-marketplace/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pgx-bm25 1.0: BM25 ranked full-text search as a native PostgreSQL index</h3><p>PGX Inc. has released version 1.0 of pgx-bm25, an extension that adds Okapi BM25 ranked full-text search to PostgreSQL as a native index access method. It supports PostgreSQL 17 and 18, and installs a</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pgx-bm25-10-bm25-ranked-full-text-search-as-a-native-postgresql-index-3396/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 pg_plan_filter 1.0.0 released</h3><p>pg_plan_filter version 1.0.0 has been released. This loadable module test sstatements against specific configured criteria before execution, raising an error if the criteria are violated. This allows </p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 PostgreSQL News</strong></p>
<p><a href="https://www.postgresql.org/about/news/pg_plan_filter-100-released-3352/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The OpenTelemetry Java agent 3.0 is almost here — preview it today</h3><p>The 2.32.0 release of the OpenTelemetry Java agent is now out and serves as the release candidate for 3.0, which is targeted for October 2026. Some behavior may still change before 3.0, but you can pr</p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 OpenTelemetry Blog</strong></p>
<p><a href="https://opentelemetry.io/blog/2026/java-agent-3.0-preview/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Why Open Source Still Matters When Agents Write the Code</h3><p>Maintainers of well-known open source projects are dealing with a new kind of inbox. Pull requests still arrive, often in higher volume than ever, and many are clean, well-formatted, and plausible on </p>
<p><strong>📅 Oct 6, 2026</strong> • <strong>📰 TiDB Blog</strong></p>
<p><a href="https://www.pingcap.com/blog/open-source-agents-write-code/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>🌐 Platforms</h2><h3>📄 The Oracle of Delphi Will Steal Your Credentials</h3><p>Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-oracle-of-delphi-steal-your-credentials" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger</h3><p>In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Linode Blog</strong></p>
<p><a href="https://www.akamai.com/blog/security/the-nansh0u-campaign-hackers-arsenal-grows-stronger" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Trusted Data Apps for the Agentic Data Cloud, powered by Looker</h3><p>As a business intelligence user, this probably sounds familiar: Most dashboards only provide you with a standardized canvas and a fixed library of charts, and the “analysis” consists of a CSV export t</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/business-intelligence/introducing-looker-data-apps/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 10 Years of the SRE book: Announcing the second edition of "Site Reliability Engineering”</h3><p>A decade ago, Google released Site Reliability Engineering: How Google Runs Production Systems with O'Reilly Media. What began as a collection of our practices for managing large-scale infrastructure </p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 Google Cloud Blog</strong></p>
<p><a href="https://cloud.google.com/blog/products/devops-sre/sre-book-second-edition/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing DigitalOcean Agent Data Services: The Data Foundation for Agents</h3><p>Agents are only as capable as the context they can reach. An agent with reliable access to the right information makes decisions grounded in fact; one without it guesses. And the work agents produce i</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/introducing-agent-data-services" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 DigitalOcean MicroVMs: Fast, isolated compute for your AI agent infrastructure</h3><p>Coding-agent platforms, sandbox products, and code-execution services all need the same thing: isolated machines that start up fast, retain their state between bursts of work, and don’t consume comput</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 DigitalOcean Blog</strong></p>
<p><a href="https://www.digitalocean.com/blog/microvms-public-preview" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon EC2 R8gd instances are now available in additional regions</h3><p>Amazon Elastic Compute Cloud (Amazon EC2) R8gd instances are available in AWS European Sovereign Cloud (Germany) region. These instances feature up to 11.4 TB of local NVMe-based SSD block-level stora</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-ec2-r8gd-thf/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Amazon EC2 R8g instances now available in additional regions</h3><p>Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R8g instances are available in the AWS European Sovereign Cloud (Germany) region. These instances are powered by AWS Graviton4 processors and </p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 CloudFormation Updates</strong></p>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-ec2-r8g-instances-thf/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash</h3><p>We are expanding the Clef decision model family with Clef-omni, natively processing audio, video, images, and text in a single pipeline. We’ve also lowered Clef-flash pricing and boosted Clef inferenc</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/clef-faster-cheaper-multimodal/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects</h3><p>On-demand CPU and memory profiling for Cloudflare Workers and Durable Objects is now available. Now you can generate interactive flamegraphs directly in production to quickly track down memory leaks a</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/workers-on-demand-profiling/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Deno is joining Cloudflare</h3><p>The Deno team is joining Cloudflare to radically simplify self-hosting Workers and Durable Objects, so developers can use the same primitives in more places.</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Cloudflare Blog</strong></p>
<p><a href="https://blog.cloudflare.com/deno-joins-cloudflare/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 FinOps Was Built for Cloud. Here’s How It Adapts for AI</h3><p>AI adoption is moving fast, and the teams responsible for managing its costs are trying to keep up. Between new models, agentic applications and changing pricing structures, even figuring out what the</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 Kubecost Blog</strong></p>
<p><a href="https://www.apptio.com/blog/finops-was-built-for-cloud-heres-how-it-adapts-for-ai/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<hr />
<h2>📰 Misc</h2><h3>📄 Visual Studio Code 1.142 (Insiders)</h3><p>Learn what's new in Visual Studio Code 1.142 (Insiders) Read the full article</p>
<p><strong>📅 Oct 14, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_142" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Andrej Karpathy says only 5,000 people have seen what AI can really do</h3><p>OpenAI founding member Andrej Karpathy, who joined Anthropic in May, thinks most of the world has little idea of what The post Andrej Karpathy says only 5,000 people have seen what AI can really do ap</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/karpathy-levie-ai-capability-gap/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Build Your Game Challenge: Join in and Compete for Prizes</h3><p>This October, we’re launching a new challenge for the JetBrains Academy community. To take part, complete our Build a Game With an AI Coding Agent course, customize the game with your own ideas, and s</p>
<p><strong>📅 Oct 12, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/education/2026/10/12/build-your-game-challenge-join-in-and-compete-for-prizes/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Claude Code found my app’s shared contract in two MCP calls. Then the records ran out.</h3><p>Every Claude Code user has watched a new session start from zero. You open it on a system you built The post Claude Code found my app’s shared contract in two MCP calls. Then the records ran out. appe</p>
<p><strong>📅 Oct 11, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/claude-code-mcp-records/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Claude found 29,000 possible bugs in open source. Only 516 have been fixed.</h3><p>Anthropic’s new OSS Scanner, launched last week as part of its broader Cyber Mission, exposes a problem inside the company’s The post Claude found 29,000 possible bugs in open source. Only 516 have be</p>
<p><strong>📅 Oct 11, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/anthropic-oss-scanner-vulnerabilities/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 How to turn AI production feedback into better agents</h3><p>Your agent is live. The service is healthy. But are its answers getting better? By connecting production traces, curated data, The post How to turn AI production feedback into better agents appeared f</p>
<p><strong>📅 Oct 10, 2026</strong> • <strong>📰 The New Stack</strong></p>
<p><a href="https://thenewstack.io/ai-agent-evaluation-loop/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 UI freeze reports are now available in JetBrains Marketplace</h3><p>A user reports that the IDE stopped responding, and your plugin is the prime suspect. You try the same steps, but everything works. Without more details, such as a thread dump captured during the free</p>
<p><strong>📅 Oct 9, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/platform/2026/10/ui-freeze-reports-in-marketplace/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Mellum2.1 Gets to Work: A Fast Open Model for Coding Agents</h3><p>Trained with reinforcement learning in real environments, Mellum2.1 is built for coding agents and fast sub-agents that run on your own hardware. Today, we’re releasing Mellum2.1, the next version of </p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/ai/2026/10/mellum2-1-gets-to-work-a-fast-open-model-for-coding-agents/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Code Quality Never Changes – Or Is AI Challenging This Notion?</h3><p>This post was adapted from Kai Schmithuesen’s talk at JetBrains Game Development Day 2026. See the full video presentation and Q&amp;A below, or keep reading! First we are going to define what we mean by </p>
<p><strong>📅 Oct 8, 2026</strong> • <strong>📰 JetBrains Blog</strong></p>
<p><a href="https://blog.jetbrains.com/qodana/2026/10/code-quality-never-changes-or-is-ai-challenging-this-notion/" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Visual Studio Code 1.141</h3><p>Learn what's new in Visual Studio Code 1.141 Read the full article</p>
<p><strong>📅 Oct 7, 2026</strong> • <strong>📰 VS Code Blog</strong></p>
<p><a href="https://code.visualstudio.com/updates/v1_141" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
<h3>📄 Devpack for Rust: Zero to Rust in seconds</h3><p>One of the best things about Rust is how easy it is to get started. It takes just a second to snap install rustup; rustup default stable, and from there you have a whole toolchain ready to compile Rus</p>
<p><strong>📅 Oct 5, 2026</strong> • <strong>📰 Ubuntu Blog</strong></p>
<p><a href="https://ubuntu.com//blog/devpack-for-rust-support" rel="noopener noreferrer"><strong>🔗 Read more</strong></a></p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[DigitalOcean Agent Droplets: When the $50 Plan Pays Off]]></title>
      <link>https://devops-daily.com/posts/digitalocean-agent-droplets-when-the-plan-pays-off</link>
      <description><![CDATA[Agent Droplets is a prepaid discount on DigitalOcean Managed Agents, not a server. We priced our real agent runs from a real invoice to find where the $50 and $200 plans start to save money, and found a rounding rule that made our runs cost more than the token math.]]></description>
      <pubDate>Sun, 11 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/digitalocean-agent-droplets-when-the-plan-pays-off</guid>
      <category><![CDATA[FinOps]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[FinOps]]></category><category><![CDATA[DigitalOcean]]></category><category><![CDATA[AI Agents]]></category><category><![CDATA[Pricing]]></category><category><![CDATA[Cloud]]></category>
      <content:encoded><![CDATA[<p>The $50 Agent Droplets plan is cheaper than paying as you go only when you would spend more than $50 a month on DigitalOcean Managed Agents anyway, and then it saves you at most $8.82. For the coding agent we built last month, $50 of usage is about 780 runs. We know what a run costs because we read the September invoice line by line. The invoice also showed something the token math hides: on our invoice, every session's token lines were rounded to the cent, with a one-cent minimum, so our five recorded runs were billed $0.32 for tokens worth $0.274.</p>
<p><a href="https://www.digitalocean.com/blog/introducing-agent-droplets" rel="noopener noreferrer">Agent Droplets</a> launched on 1 October 2026 as a public preview. The name suggests a server you rent. It is a monthly plan that prepays and discounts the usage you already pay for in <a href="https://docs.digitalocean.com/products/managed-agents/" rel="noopener noreferrer">Managed Agents</a>. This post shows the plan math, the real cost of our runs, the break-even for two kinds of agent work, and a calculator you can run with your own numbers.</p>
<h2>TLDR</h2><ul>
<li><strong>Pro</strong> costs $50 a month and covers $58.82 of usage at list price (15% off). <strong>Team</strong> costs $200 and covers $250 (20% off). After that, usage continues at list price.</li>
<li>Pro beats pay as you go above <strong>$50 a month</strong> of discounted usage and saves at most <strong>$8.82</strong>. Team beats Pro above <strong>$208.82</strong> and saves at most <strong>$50</strong>.</li>
<li>The discount covers compute, memory, storage, tools, egress and DigitalOcean-hosted models such as GLM-5.3 and Kimi K3. <strong>Claude and GPT are not discounted.</strong></li>
<li>Our five recorded issue-to-PR runs were billed <strong>$0.064 per run</strong>, $0.107 per merged pull request. At that rate, Pro is cheaper from about <strong>782 runs a month</strong>.</li>
<li>One 39-minute interactive session on GLM-5.3 was billed <strong>$0.78</strong>, almost all of it tokens. At that rate, Pro is cheaper from <strong>65 sessions a month</strong>.</li>
<li>On our invoice, every token line of every session matched <strong>max(1 cent, amount rounded to the cent)</strong>. That added 17% to our five runs and 25% to all 13 sessions on the invoice. Compute for sessions of about a minute was billed at <strong>$0.00</strong>.</li>
<li>DigitalOcean's docs do not say whether unused plan balance rolls over. Check that before you subscribe.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A DigitalOcean account. Nothing here needs a plan; we did not subscribe to one.</li>
<li>Node 20 or later, to run the calculator from the <a href="https://github.com/The-DevOps-Daily/agent-droplets-break-even" rel="noopener noreferrer">companion repo</a>.</li>
<li>Optional: an API token with billing read access, if you want to price your own invoice with the same scripts.</li>
</ul>
<h2>What you actually buy</h2><p>Managed Agents runs coding agents such as Claude Code, Codex and OpenCode in microVMs, and bills for the parts separately: compute, memory, storage, tool calls, egress and model tokens. Agent Droplets does not change any of that. It adds a monthly fee that funds a balance, and your usage draws from that balance at a discount.</p>
<table>
<thead>
<tr>
<th>Plan</th>
<th>Monthly fee</th>
<th>Discount</th>
<th>Usage covered at list price</th>
</tr>
</thead>
<tbody><tr>
<td>Free trial</td>
<td>$5 credit</td>
<td>none</td>
<td>$5</td>
</tr>
<tr>
<td>Pro</td>
<td>$50</td>
<td>15%</td>
<td>$58.82</td>
</tr>
<tr>
<td>Team</td>
<td>$200</td>
<td>20%</td>
<td>$250.00</td>
</tr>
</tbody></table>
<p>The figures come from DigitalOcean's <a href="https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/details/pricing/" rel="noopener noreferrer">Harness Runtime pricing page</a> and the <a href="https://www.digitalocean.com/pricing/agent-droplets" rel="noopener noreferrer">Agent Droplets pricing page</a>, both read on 10 October 2026.</p>
<p>Not everything is discounted. The docs list what is:</p>
<ul>
<li>Harness Runtime compute (CPU) and memory</li>
<li>Session storage, snapshots and custom templates</li>
<li>Inference on DigitalOcean-hosted models, such as Kimi K3 and GLM-5.3</li>
<li>Action Gateway tools: code interpreter, browser automation, web search and web fetch</li>
<li>Public internet egress</li>
</ul>
<p>Third-party models such as Anthropic Claude and OpenAI GPT are billed at standard rates and do not draw down the plan balance at a discount. So if your agents run on Claude, the plan saves you nothing on the biggest part of the bill.</p>
<p>When the balance runs out, nothing stops. In DigitalOcean's words, "you just move from the discounted rate to list price for the rest of the month."</p>
<blockquote>
<p><strong>Warning</strong></p>
<p>We could not find what happens to unused balance at the end of the month. Neither pricing page says whether it rolls over. If it expires, a quiet month costs the full fee for less than the full amount of usage. Ask DigitalOcean before you subscribe to Team.</p>
</blockquote>
<h2>The math</h2><p>Write your monthly <strong>discount-eligible</strong> usage at list price as U: compute, memory, storage, tools, egress and DigitalOcean-hosted models. Claude, GPT and other charges the plans do not discount are added to every line unchanged, so leave them out of U. Then:</p>
<pre><code class="hljs language-text" tabindex="0">Pay as you go:  U
Pro:            50  + max(0, U - 58.82)
Team:           200 + max(0, U - 250)
</code></pre><p>Pro costs a flat $50 until your usage passes $58.82, then grows with usage again. So Pro is cheaper than pay as you go as soon as U is above $50, and the gap never grows past $58.82 - $50 = $8.82. Team overtakes Pro where Pro's cost reaches $200, at U = $208.82, and its saving against pay as you go is capped at $250 - $200 = $50.</p>
<p><strong>Monthly cost by plan</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>$0</th>
<th>$25</th>
<th>$50</th>
<th>$75</th>
<th>$100</th>
<th>$150</th>
<th>$200</th>
<th>$250</th>
<th>$300</th>
<th>$400</th>
</tr>
</thead>
<tbody><tr>
<td>Pay as you go</td>
<td>0$</td>
<td>25$</td>
<td>50$</td>
<td>75$</td>
<td>100$</td>
<td>150$</td>
<td>200$</td>
<td>250$</td>
<td>300$</td>
<td>400$</td>
</tr>
<tr>
<td>Pro ($50)</td>
<td>50$</td>
<td>50$</td>
<td>50$</td>
<td>66.18$</td>
<td>91.18$</td>
<td>141.18$</td>
<td>191.18$</td>
<td>241.18$</td>
<td>291.18$</td>
<td>391.18$</td>
</tr>
<tr>
<td>Team ($200)</td>
<td>200$</td>
<td>200$</td>
<td>200$</td>
<td>200$</td>
<td>200$</td>
<td>200$</td>
<td>200$</td>
<td>200$</td>
<td>250$</td>
<td>350$</td>
</tr>
</tbody></table>
<p><em>Monthly cost against monthly Managed Agents usage at list price. Only usage that the plans discount counts here; Claude and GPT tokens do not. Plan terms read 2026-10-10.</em></p>
<p>The shape is the whole story. Below $50, Pro costs more than it saves. Between $50 and $208.82, Pro is the cheapest, by at most $8.82. Above that, Team is cheapest, by at most $50 against pay as you go. A plan is worth it only once you know that your usage sits reliably above those lines.</p>
<p>The useful question is what $50 of usage looks like in agent runs. For that, we needed the real cost of a real run.</p>
<h2>What our agent runs actually cost</h2><p>Last month we built an <a href="https://devops-daily.com/posts/issue-to-pull-request-digitalocean-managed-agents">issue-to-pull-request agent on Managed Agents</a>: OpenCode on DeepSeek V4 Pro in a 2 vCPU / 4 GB microVM, one session per GitHub issue. That post estimated the model cost of each run from the token counts that the agent reported. The September invoice has now arrived, and it tells a different story.</p>
<p>The invoice API returns one line per session for compute, and one line per session for each kind of token: input, output and cache read. Stage 1 of the <a href="https://github.com/The-DevOps-Daily/agent-droplets-break-even" rel="noopener noreferrer">companion repo</a> pulls those lines, and stage 2 prices them.</p>
<p><a href="https://github.com/The-DevOps-Daily/agent-droplets-break-even" rel="noopener noreferrer">The-DevOps-Daily/agent-droplets-break-even on GitHub</a></p>
<p>These are the five recorded runs, matched to their invoice sessions by the session name in each run's log:</p>
<table>
<thead>
<tr>
<th>Run</th>
<th>Outcome</th>
<th>Input / output / cache-read tokens</th>
<th>Exact list price</th>
<th>Billed</th>
</tr>
</thead>
<tbody><tr>
<td>Issue #1</td>
<td>Merged</td>
<td>8,989 / 1,028 / 61,696</td>
<td>$0.0407</td>
<td>$0.05</td>
</tr>
<tr>
<td>Issue #2, first try</td>
<td>Closed in review</td>
<td>15,136 / 1,852 / 105,472</td>
<td>$0.0695</td>
<td>$0.08</td>
</tr>
<tr>
<td>Issue #2, second try</td>
<td>Push refused</td>
<td>15,184 / 2,272 / 82,176</td>
<td>$0.0629</td>
<td>$0.07</td>
</tr>
<tr>
<td>Issue #2, third try</td>
<td>Merged</td>
<td>14,984 / 1,609 / 57,344</td>
<td>$0.0516</td>
<td>$0.06</td>
</tr>
<tr>
<td>Issue #3</td>
<td>Merged</td>
<td>9,514 / 2,113 / 72,192</td>
<td>$0.0490</td>
<td>$0.06</td>
</tr>
<tr>
<td><strong>Total</strong></td>
<td>3 merged</td>
<td></td>
<td><strong>$0.274</strong></td>
<td><strong>$0.32</strong></td>
</tr>
</tbody></table>
<p>Two things stand out.</p>
<p><strong>Cache-read tokens are most of the volume.</strong> An agent sends its growing conversation to the model on every step, and the provider bills the repeated part as cache reads, at one fifth of the input price for this model. Our earlier post counted only input and output tokens, so it put the model cost at about two cents per run. With cache reads and the invoice's rounding, the real figure was five to eight cents. We have added a correction to that post.</p>
<p><strong>Each line was rounded to the cent, with a one-cent minimum.</strong> We checked a rule against all 31 token lines on our September invoice: billed = max($0.01, exact amount rounded to the cent). All 31 match. That is a pattern on one pay-as-you-go invoice, not a published billing policy, but it is consistent. On a long session this hardly matters. On short agent runs it does. For our five runs it added 17%, and across all 13 sessions on the invoice, including short probes, it added 25%: $0.62 billed for $0.497 of tokens. A probe that produced four output tokens was billed a full cent for them.</p>
<p><strong>Exact token cost vs billed, per run</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>Issue #1</td>
<td>0.0407$</td>
<td>Exact</td>
</tr>
<tr>
<td>Issue #1</td>
<td>0.05$</td>
<td>Billed</td>
</tr>
<tr>
<td>Issue #2 (1)</td>
<td>0.0695$</td>
<td>Exact</td>
</tr>
<tr>
<td>Issue #2 (1)</td>
<td>0.08$</td>
<td>Billed</td>
</tr>
<tr>
<td>Issue #2 (2)</td>
<td>0.0629$</td>
<td>Exact</td>
</tr>
<tr>
<td>Issue #2 (2)</td>
<td>0.07$</td>
<td>Billed</td>
</tr>
<tr>
<td>Issue #2 (3)</td>
<td>0.0516$</td>
<td>Exact</td>
</tr>
<tr>
<td>Issue #2 (3)</td>
<td>0.06$</td>
<td>Billed</td>
</tr>
<tr>
<td>Issue #3</td>
<td>0.049$</td>
<td>Exact</td>
</tr>
<tr>
<td>Issue #3</td>
<td>0.06$</td>
<td>Billed</td>
</tr>
</tbody></table>
<p><em>DeepSeek V4 Pro at $1.74 / $3.48 / $0.348 per million input / output / cache-read tokens. Billed amounts are the invoice lines before account credits, September 2026.</em></p>
<p><strong>Compute was billed at $0.00.</strong> Compute is billed per second for the CPU a session actually uses, plus its peak memory. Even at full use, 2 vCPU and 4 GB list at $0.126 an hour, and the five sessions existed for 38 to 79 seconds each, so each one's compute was less than three tenths of a cent. The invoice shows $0.00 for every compute line. Unlike the token lines, compute lines below a cent were not raised to a cent.</p>
<p>So one recorded run cost <strong>$0.064</strong> on the bill, tokens and compute together. Two of the five runs did not end in a merged pull request, so a merged pull request cost <strong>$0.107</strong>.</p>
<h2>Turning $50 into runs</h2><p>Stage 3 finds the first monthly volume at which each plan is cheaper. These are estimates: they assume a plan draws down its balance at the same charges our pay-as-you-go invoice showed, which we could not check without subscribing.</p>
<p><strong>break-even</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/03-break-even.mjs data/priced-2026-09.json
Pro is cheaper than pay as you go above <span class="hljs-variable">$50</span>.00 a month of discount-eligible usage at list price; it saves at most <span class="hljs-variable">$8</span>.82.
Team is cheaper than Pro above <span class="hljs-variable">$208</span>.82; it saves at most <span class="hljs-variable">$50</span>.

workload                       cost per unit  Pro cheaper from      Team cheaper than Pro from
issue-to-PR run (2026-09)      <span class="hljs-variable">$0</span>.0640        782 runs/month        3263 runs/month
merged pull request (2026-09)  <span class="hljs-variable">$0</span>.1067        469 merged PRs/month  1958 merged PRs/month
</code></pre><p>782 runs a month is about 26 runs a day, every day. A team that points an agent at every new issue in a busy repository might get there. A team that uses it a few times a day will not, and pays $50 for less than $50 of usage.</p>
<h2>One long session: $0.78 for 39 minutes</h2><p>Short runs are the worst case for the rounding rule. To see the other end, stage 4 of the repo ran one longer session the way a person works with a coding agent: six prompts, one after another, in the same 2 vCPU / 4 GB session, on GLM-5.3, a DigitalOcean-hosted model that the plans discount. The prompts asked the agent to read a small Python package, add ISO 8601 duration parsing, add a command-line entry point, rewrite the error messages, add type hints and documentation, and then review its own changes. The session ran for 39 minutes, changed five files with 346 added lines, and finished with 45 tests passing. Nothing was pushed.</p>
<p>It appeared on the invoice preview the next morning:</p>
<p><strong>long session</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node scripts/01-invoice-lines.mjs preview
data/invoice-2026-10.json: 4 Managed Agents lines { <span class="hljs-string">'Harness Runtime Serverless Inference'</span>: 0.74, <span class="hljs-string">'OHS Compute'</span>: 0.04 }
$ node scripts/02-price-sessions.mjs data/invoice-2026-10.json
all sessions: { exactModelCost: 0.7417, billedModelCost: 0.74 } invoice says: { <span class="hljs-string">'Harness Runtime Serverless Inference'</span>: 0.74, <span class="hljs-string">'OHS Compute'</span>: 0.04 }
rounding rule: {
  rule: <span class="hljs-string">'max(0.01, round(exact, 2)) per token line'</span>,
  lines: 3,
  matches: 3
}
</code></pre><p><strong>What the 39-minute session was billed</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Output tokens (102,865)</td>
<td>0.45$</td>
</tr>
<tr>
<td>Cache-read tokens (971,520)</td>
<td>0.25$</td>
</tr>
<tr>
<td>Input tokens (26,087)</td>
<td>0.04$</td>
</tr>
<tr>
<td>Compute (2,318 s)</td>
<td>0.04$</td>
</tr>
</tbody></table>
<p><em>Invoice preview lines for one GLM-5.3 session in a 2 vCPU / 4 GB microVM, October 2026. List prices on the invoice: $1.40 input, $4.40 output and $0.26 cache read per million tokens.</em></p>
<p>Three things changed from the short runs:</p>
<ul>
<li><strong>The rounding rule stops mattering.</strong> The three token lines match the same rule, but on $0.74 of tokens it moved the bill by less than a cent.</li>
<li><strong>Cache reads are a third of the bill.</strong> Each prompt sends the whole conversation so far. The cache makes that cheap per token, but the session read 971,520 cached tokens, 37 times its 26,087 fresh input tokens, and paid $0.25 for them. A long session pays for its own history.</li>
<li><strong>Compute is still small.</strong> $0.04 for 2,318 seconds. At full use, 2 vCPU and 4 GB would list at about $0.08 for that time, so the session was billed for roughly half, which fits billing for the CPU actually used. Tokens were 95% of the bill.</li>
</ul>
<p>At $0.78 a session, the break-even comes much sooner:</p>
<p><strong>calc</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node calc.mjs --runs 65 --cost 0.78
Discount-eligible usage at list price: <span class="hljs-variable">$50</span>.70 a month
  Pay as you go  <span class="hljs-variable">$50</span>.70
  Pro            <span class="hljs-variable">$50</span>.00
  Team           <span class="hljs-variable">$200</span>.00
Cheapest: Pro.
$ node calc.mjs --runs 268 --cost 0.78
Discount-eligible usage at list price: <span class="hljs-variable">$209</span>.04 a month
  Pay as you go  <span class="hljs-variable">$209</span>.04
  Pro            <span class="hljs-variable">$200</span>.22
  Team           <span class="hljs-variable">$200</span>.00
Cheapest: Team.
</code></pre><p>Pro is cheaper from <strong>65 sessions like this a month</strong>, about three every working day for one person. Team is cheaper than Pro from <strong>268</strong>, which is about 13 people each running one such session every working day. This is one session on one model, so treat $0.78 as an example, not a rate. A longer conversation reads more cache, and the same token counts on Kimi K3 would list at about $1.91, 2.6 times as much.</p>
<h2>Where the plan makes sense</h2><ul>
<li><strong>Agents that run for hours.</strong> Long interactive sessions use far more tokens than our short runs, and a busy agent keeps the CPU working. Our one recorded 39-minute session cost 12 times as much as a short run, and sessions like it reach the $50 and $208.82 lines much faster.</li>
<li><strong>Agents on DigitalOcean-hosted models.</strong> GLM-5.3 at $1.40 / $4.40 per million input / output tokens and Kimi K3 at $3 / $15 are discounted. Claude Sonnet 5.5 at $2 / $10 is not, even though it is billed through the same account.</li>
<li><strong>Many people sharing one account.</strong> Every plan includes unlimited seats and agents, so the whole team's usage counts toward one balance. With short runs like ours, ten people would need about 11 runs each per day to pass $208.82; with long sessions, far fewer.</li>
</ul>
<h2>Where it does not</h2><ul>
<li><strong>Occasional or bursty use.</strong> Below $50 a month, pay as you go is cheaper, and the free trial's $5 credit covers a lot of short runs.</li>
<li><strong>Claude Code or Codex on their own vendors' models.</strong> The model tokens are most of the bill, and the plan does not touch them.</li>
<li><strong>Before you have a month of real usage.</strong> Run pay as you go for a month, then price the invoice. The <a href="https://github.com/The-DevOps-Daily/agent-droplets-break-even" rel="noopener noreferrer">scripts</a> do that for you.</li>
</ul>
<h2>Price your own usage</h2><p>Clone the repo and pass your own numbers:</p>
<pre><code class="hljs language-bash" tabindex="0">git <span class="hljs-built_in">clone</span> https://github.com/The-DevOps-Daily/agent-droplets-break-even
<span class="hljs-built_in">cd</span> agent-droplets-break-even
node calc.mjs --runs 1500 --cost 0.064
</code></pre><p><strong>calc</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ node calc.mjs --runs 1500 --cost 0.064
Discount-eligible usage at list price: <span class="hljs-variable">$96</span>.00 a month
  Pay as you go  <span class="hljs-variable">$96</span>.00
  Pro            <span class="hljs-variable">$87</span>.18
  Team           <span class="hljs-variable">$200</span>.00
Cheapest: Pro.
</code></pre><p>To price your own invoice instead, run stage 1 with a billing-read token and your invoice id (or <code>preview</code> for the current month), then stage 2 on the file it writes. Stage 2 also checks the rounding rule against every token line, so you can see whether your bill follows it too.</p>
<h2>What we could not conclude</h2><ul>
<li><strong>How a plan invoice looks.</strong> We stayed on pay as you go. We do not know whether the 15% discount is applied before or after the one-cent rounding, which matters for short runs.</li>
<li><strong>Rollover.</strong> As above, the docs do not say.</li>
<li><strong>A typical long session.</strong> We priced one. Its cost depends on the model, the length of the conversation and how much the agent reads back, so your sessions can cost a lot more or less than $0.78.</li>
<li><strong>Speed.</strong> Managed Agents is in preview, and DigitalOcean's preview terms treat benchmark results as confidential, so this post prices usage only.</li>
<li><strong>Prices after the preview.</strong> Plan terms and list prices can change before general availability. The repo keeps every price in one file with its source and the date we read it.</li>
</ul>
<h2>Summary</h2><p>Agent Droplets is a commitment discount, not a cheaper server. It pays off only once your discounted usage reliably passes $50 a month, and even then it saves at most $8.82 on Pro and $50 on Team. Our short issue-to-PR runs cost six cents each on the bill, so a team would need about 780 of them a month before Pro is worth it. Sessions like our recorded long one reach it much sooner: that 39-minute session on GLM-5.3 cost $0.78, so about 65 of them a month pass the Pro line. Whichever you run, read your invoice before you subscribe: it showed us cache-read tokens and a rounding rule that the token counts alone did not.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[OpenSSH 10.6 Refuses DOMAIN\user and Weakens ssh -C. We Measured Both]]></title>
      <link>https://devops-daily.com/posts/openssh-10-6-domain-usernames-and-compression</link>
      <description><![CDATA[OpenSSH 10.6 came out on October 6. It refuses usernames with a backslash or a dollar sign before it connects, which breaks ssh, scp, rsync, git and Ansible logins written as CORP\alice. It also cuts ssh -C to Huffman coding only, so a 32 MiB log now compresses to 21 MiB instead of 3.7 MiB. We built 10.5 and 10.6 side by side and tested 13 ways of passing a username, 4 kinds of data and 8 SSH clients.]]></description>
      <pubDate>Sat, 10 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/openssh-10-6-domain-usernames-and-compression</guid>
      <category><![CDATA[Linux]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Linux]]></category><category><![CDATA[OpenSSH]]></category><category><![CDATA[SSH]]></category><category><![CDATA[Ansible]]></category><category><![CDATA[Windows]]></category><category><![CDATA[Security]]></category><category><![CDATA[Networking]]></category>
      <content:encoded><![CDATA[<p>OpenSSH 10.6 was released on October 6, and Homebrew, Arch Linux and Debian unstable already ship it. Its <a href="https://www.openssh.com/txt/release-10.6" rel="noopener noreferrer">release notes</a> list three changes that alter what you see day to day. <code>ssh</code> now refuses usernames that contain <code>\</code> or <code>$</code> when they come from the command line. <code>ssh -C</code> compression no longer uses LZ77, the part of deflate that does most of the work. And <code>sshd</code> now logs a warning for every connection that does not use a post-quantum key exchange.</p>
<p>The first change hits anyone who logs in to Windows or AD-joined machines as <code>CORP\alice</code>. Microsoft's own <a href="https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse" rel="noopener noreferrer">OpenSSH guide</a> shows exactly that form: <code>ssh domain\username@servername</code>. The second change hits anyone who turned on <code>-C</code> for a slow link, and Ansible users on its default OpenSSH connection, because Ansible passes <code>-C</code> by default. We built OpenSSH 10.5p1 and 10.6p1 from source on the same machine and measured what each change does.</p>
<h2>TLDR</h2><ul>
<li>With a 10.6 client, <code>CORP\alice</code> and <code>svc-backup$</code> were refused before any connection on 11 of the 13 ways we passed them: <code>user@host</code>, <code>-l</code>, <code>-o User=</code>, <code>ssh://</code> URIs (also percent-encoded), scp, sftp, rsync, git, Ansible's <code>ansible_user</code> and a <code>User ${VAR}</code> line in ssh_config. With 10.5, all 26 attempts reached the server.</li>
<li>Two ways still work: a literal <code>User CORP\alice</code> line in ssh_config, and the <code>alice@corp.example.com@host</code> form.</li>
<li><code>ssh -C</code> on 10.6 compressed a 32 MiB access log to 21.09 MiB. 10.5 compressed the same file to 3.71 MiB. That is 5.7 times more compressed data to send. JSON logs and source code came out 3.1 to 3.4 times bigger.</li>
<li>The side that sends the data decides. If you upgrade only your laptop, downloads from an old server still compress well, and uploads do not.</li>
<li><code>zstd -1</code> in a pipe compressed the access log 7.6x in 0.44 CPU seconds. zlib level 6, which <code>ssh -C</code> used before 10.6, needed 1.50 CPU seconds for 8.7x.</li>
<li>A default 10.6 sshd logged a warning for paramiko 5.0.0, libssh2 1.10.0 and any client limited to classic key exchange. asyncssh 2.24.1, Go's x/crypto/ssh v0.58.0 and OpenSSH 9.2p1, 10.5p1 and 10.6p1 on default settings caused no warning. The setting is global only, so you cannot turn it off for one subnet.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>A shell with OpenSSH. Run <code>ssh -V</code> to see your client version.</li>
<li>Optional: the demo repo, which builds both versions and runs every test in this post. It needs a C toolchain, <code>libssl-dev</code> and <code>zlib1g-dev</code>, and builds both versions from source.</li>
</ul>
<p><a href="https://github.com/The-DevOps-Daily/openssh-10-6-check" rel="noopener noreferrer">The-DevOps-Daily/openssh-10-6-check on GitHub</a></p>
<h2>How we tested</h2><p>Everything ran on one Debian 12 machine (aarch64, 4 cores, recorded in <code>results/00-environment.txt</code>). <code>01-build.sh</code> downloads <code>openssh-10.5p1</code> and <code>openssh-10.6p1</code>, checks the SHA256 sums printed in each release note, and builds both into separate prefixes. Each <code>sshd</code> runs as an ordinary user on <code>127.0.0.1</code>. The username test uses one 10.6 sshd and points both client versions at it. The server log is the evidence: if <code>sshd</code> logs <code>Invalid user CORP\\alice</code>, the name got through. If the client prints <code>remote username contains invalid characters</code> and the log has no new line, the client refused it before connecting.</p>
<p>All usernames other than the local account are fake, so each login that reaches the server ends in <code>Permission denied</code>. That is the result we want here. It proves the client sent the name.</p>
<h2>Change 1: usernames with a backslash or a dollar sign</h2><p>The new check is in <code>ssh_valid_ruser()</code> in <code>readconf.c</code>. The list of forbidden characters gained two entries, <code>$</code> and <code>\</code>. The release notes say why: a username from an untrusted source can end up in a shell through <code>%r</code> in <code>ProxyCommand</code> or <code>Match exec</code>. The check runs on every username that came from the command line, and on any <code>User</code> value from ssh_config that changed when OpenSSH expanded it.</p>
<p>Here is the same command on both versions, recorded by <code>09-transcript.sh</code>:</p>
<p><strong>OpenSSH 10.5p1 and 10.6p1, same command</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># every ssh below also gets -F work/base_config: port 18022, the test key, BatchMode</span>
$ ssh <span class="hljs-string">'CORP\alice@127.0.0.1'</span> <span class="hljs-literal">true</span>   <span class="hljs-comment"># 10.5p1</span>
CORP\\alice@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).
$ ssh <span class="hljs-string">'CORP\alice@127.0.0.1'</span> <span class="hljs-literal">true</span>   <span class="hljs-comment"># 10.6p1</span>
remote username contains invalid characters
$ ssh -o <span class="hljs-string">'User=CORP\alice'</span> 127.0.0.1 <span class="hljs-literal">true</span>   <span class="hljs-comment"># 10.6p1</span>
remote username contains invalid characters
$ <span class="hljs-built_in">head</span> -3 ssh_config   <span class="hljs-comment"># the Host block the next two commands use</span>
Host winbox
  HostName 127.0.0.1
  User CORP\alice
$ ssh winbox <span class="hljs-literal">true</span>   <span class="hljs-comment"># 10.6p1, User line in ssh_config</span>
CORP\\alice@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).
$ WIN_USER=<span class="hljs-string">'CORP\alice'</span> ssh winbox <span class="hljs-literal">true</span>   <span class="hljs-comment"># 10.6p1, User ${WIN_USER} in ssh_config</span>
remote username contains invalid characters
$ ssh <span class="hljs-string">'alice@corp.example.com@127.0.0.1'</span> <span class="hljs-literal">true</span>   <span class="hljs-comment"># 10.6p1, UPN form</span>
alice@corp.example.com@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).
</code></pre><p>The doubled backslash in <code>CORP\\alice</code> is how OpenSSH prints the name. The server received <code>CORP\alice</code>.</p>
<p><code>03-usernames.sh</code> runs four usernames through thirteen entry paths with each client version. This is the 10.6p1 column. With 10.5p1, every <code>CORP\alice</code> and <code>svc-backup$</code> cell reached the server.</p>
<table>
<thead>
<tr>
<th>How the username is passed</th>
<th><code>CORP\alice</code></th>
<th><code>svc-backup$</code></th>
<th><code>alice@corp.example.com</code></th>
</tr>
</thead>
<tbody><tr>
<td><code>ssh user@host</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>ssh -l user host</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>ssh -o User=user host</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>ssh ssh://user@host:port</code></td>
<td>refused</td>
<td>refused</td>
<td>usage error on both versions</td>
</tr>
<tr>
<td><code>ssh://</code> with the name %-encoded</td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td>ssh_config <code>User &lt;name&gt;</code>, literal</td>
<td>reaches server</td>
<td>reaches server</td>
<td>reaches server</td>
</tr>
<tr>
<td>ssh_config <code>User ${SSH_TEST_USER}</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>scp user@host:file .</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>sftp user@host</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>rsync -e ssh user@host:file .</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td><code>git ls-remote user@host:repo.git</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td>Ansible with <code>ansible_user</code></td>
<td>refused</td>
<td>refused</td>
<td>reaches server</td>
</tr>
<tr>
<td>Ansible, <code>User</code> line in ssh_config</td>
<td>reaches server</td>
<td>reaches server</td>
<td>reaches server</td>
</tr>
</tbody></table>
<p>scp, sftp, rsync and git all start <code>ssh</code> with the username as an argument, so they get the same refusal. The message does not change. git prints it and exits with 128. Ansible wraps it:</p>
<pre><code class="hljs language-text" tabindex="0">t | UNREACHABLE!: Task failed: Failed to connect to the host via ssh: remote username contains invalid characters
</code></pre><p>The Ansible row matters because the usual inventory for Windows over SSH sets <code>ansible_user: 'CORP\alice'</code>. ansible-core 2.21.5 passes that as <code>-o User=</code>, which counts as the command line.</p>
<h3>Who this breaks</h3><ul>
<li>Admins on macOS (Homebrew), Arch or Debian unstable who log in to Windows Server with <code>DOMAIN\user</code>, as in Microsoft's guide.</li>
<li>Linux hosts joined to Active Directory through SSSD, where people log in with the short domain form.</li>
<li>Ansible inventories that manage Windows over SSH with a down-level logon name in <code>ansible_user</code>.</li>
<li>Scripts that build <code>ssh "$user@$host"</code> from a variable that can contain <code>\</code> or <code>$</code>.</li>
</ul>
<p>Windows ships its own OpenSSH build on its own schedule. We did not test the Windows client.</p>
<h3>What to do</h3><ol>
<li><p>Use the <code>user@domain</code> form. The <a href="https://github.com/PowerShell/Win32-OpenSSH/wiki/ssh.exe-examples" rel="noopener noreferrer">Win32-OpenSSH examples</a> list <code>ssh user@domain@host</code> next to <code>ssh domain\user@host</code>. OpenSSH splits at the last <code>@</code>, so the name reaches the server intact. Inside an <code>ssh://</code> URI, write it percent-encoded as <code>alice%40corp.example.com</code>. We only checked that the name reaches the server. Whether your Windows server or SSSD setup accepts that form is up to its configuration, and we did not test it.</p>
</li>
<li><p>Put the name in a literal <code>User</code> line in ssh_config. The check skips literal values:</p>
<pre><code class="hljs language-text" tabindex="0">Host win-*.corp.example.com
  User CORP\alice
</code></pre></li>
<li><p>For Ansible, remove <code>ansible_user</code> from those hosts and let ssh_config supply the name. Pass the file with <code>ansible_ssh_common_args: '-F /path/to/ssh_config'</code>, or put the block in <code>~/.ssh/config</code>. Our test shows this path reaches the server on 10.6.</p>
</li>
<li><p>Do not template the username into ssh_config as <code>User ${VAR}</code>. 10.6 checks the expanded value and refuses it.</p>
</li>
</ol>
<h2>Change 2: ssh -C still compresses, but much less</h2><p>Before 10.6, <code>packet.c</code> started the compressor with <code>deflateInit(stream, 6)</code>, which is zlib level 6, the same as <code>gzip -6</code>. In 10.6 the call is <code>deflateInit2(..., Z_BEST_SPEED, Z_DEFLATED, 15, 8, Z_HUFFMAN_ONLY)</code>. Huffman-only deflate still gives short codes to common bytes. It does not replace repeated strings with references to earlier data, and that is where text gets most of its compression.</p>
<p>The reason is a 2026 paper, <a href="https://arxiv.org/abs/2609.07709" rel="noopener noreferrer">"Crossing the Streams"</a>. All channels in one SSH session share one compression dictionary. If an attacker can send data through one channel, such as a forwarded port, the size of the compressed output can leak secrets from another channel: when the attacker's guess matches the secret, deflate replaces it with a short back-reference and the packet gets smaller. Huffman-only coding has no back-references, which removes that signal. The output size still depends a little on which bytes are sent, so the <code>ssh_config</code> manual in 10.6 still says not to enable compression on connections that mix trusted and untrusted traffic.</p>
<p><code>05-compression.sh</code> sends four files through every client and server pair, up and down, three times each. <code>ssh -v</code> prints the raw and compressed byte counts at exit. These are the compressor's own counters, so SSH framing, MAC tags and TCP add a little on top. The counts changed by at most one byte between runs.</p>
<p><strong>Compressed SSH payload with ssh -C, by the version of the side that sends</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>access.log, 10.5</td>
<td>3.71 MiB</td>
<td>10.5p1 sends</td>
</tr>
<tr>
<td>access.log, 10.6</td>
<td>21.09 MiB</td>
<td>10.6p1 sends</td>
</tr>
<tr>
<td>events.ndjson, 10.5</td>
<td>6 MiB</td>
<td>10.5p1 sends</td>
</tr>
<tr>
<td>events.ndjson, 10.6</td>
<td>20.09 MiB</td>
<td>10.6p1 sends</td>
</tr>
<tr>
<td>openssh-src.tar, 10.5</td>
<td>2.25 MiB</td>
<td>10.5p1 sends</td>
</tr>
<tr>
<td>openssh-src.tar, 10.6</td>
<td>6.93 MiB</td>
<td>10.6p1 sends</td>
</tr>
</tbody></table>
<p><em>OpenSSH 10.5p1 and 10.6p1, <a href="mailto:zlib@openssh.com">zlib@openssh.com</a>, median of 3 runs. access.log and events.ndjson are 32 MiB each, the OpenSSH source tar is 11.2 MiB. A 16 MiB random file came out at 16.01 MiB with both versions.</em></p>
<table>
<thead>
<tr>
<th>File</th>
<th align="right">Raw</th>
<th align="right">10.5p1 sends</th>
<th align="right">10.6p1 sends</th>
<th align="right">More bytes</th>
</tr>
</thead>
<tbody><tr>
<td>access.log (nginx format)</td>
<td align="right">32.0 MiB</td>
<td align="right">3.71 MiB (8.62x)</td>
<td align="right">21.09 MiB (1.52x)</td>
<td align="right">5.7x</td>
</tr>
<tr>
<td>events.ndjson (JSON logs)</td>
<td align="right">32.0 MiB</td>
<td align="right">6.00 MiB (5.34x)</td>
<td align="right">20.09 MiB (1.59x)</td>
<td align="right">3.4x</td>
</tr>
<tr>
<td>openssh-src.tar (real source)</td>
<td align="right">11.2 MiB</td>
<td align="right">2.25 MiB (4.99x)</td>
<td align="right">6.93 MiB (1.62x)</td>
<td align="right">3.1x</td>
</tr>
<tr>
<td>random.bin</td>
<td align="right">16.0 MiB</td>
<td align="right">16.01 MiB (1.00x)</td>
<td align="right">16.01 MiB (1.00x)</td>
<td align="right">none</td>
</tr>
</tbody></table>
<p>The access log, the JSON events and the random file are generated from fixed seeds, with IP addresses from the documentation ranges. The source tar is the real <code>openssh-10.6p1</code> release, so it is the one file that we did not generate. As a cross-check, <code>07-app-compression.sh</code> runs plain zlib with <code>Z_HUFFMAN_ONLY</code> on the same files. It gave 1.52x, 1.59x and 1.62x, the same as the 10.6 ssh results.</p>
<h3>The sender decides</h3><p>Each side of an SSH connection compresses what it sends. In our matrix, the upload ratio followed the client version and the download ratio followed the server version, in every case:</p>
<ul>
<li>A 10.6 client downloading from a 10.5 server got 8.62x on the access log.</li>
<li>A 10.5 client downloading from a 10.6 server got 1.52x.</li>
</ul>
<p>So the change reaches you in stages. Upgrade your laptop and your uploads get bigger. Your downloads change when the servers upgrade.</p>
<h3>What it costs in time</h3><p>Without a bandwidth limit, on loopback, the times in <code>05-compression.txt</code> were mixed. The access log came back faster from a 10.6 server (1.84 s median, against 2.19 to 2.54 s from 10.5). The JSON file was slower in one pair (3.35 s against 2.47 s). The runs were not interleaved and the machine was shared, so we do not read CPU cost from them. On a slow link, the bytes decide.</p>
<p><code>08-shaped-link.sh</code> adds a 20 Mbit/s <code>tc</code> rule on the loopback device, for one port only, and downloads the access log five times with a 10.6 client:</p>
<table>
<thead>
<tr>
<th>Method</th>
<th>Server</th>
<th align="right">Median</th>
<th align="right">Range</th>
</tr>
</thead>
<tbody><tr>
<td><code>ssh -C</code></td>
<td>10.5p1</td>
<td align="right">4.55 s</td>
<td align="right">3.39 to 7.50 s</td>
</tr>
<tr>
<td><code>ssh -C</code></td>
<td>10.6p1</td>
<td align="right">10.46 s</td>
<td align="right">10.08 to 13.91 s</td>
</tr>
<tr>
<td>no compression</td>
<td>10.6p1</td>
<td align="right">15.89 s</td>
<td align="right">13.57 to 17.08 s</td>
</tr>
<tr>
<td><code>zstd -1</code> in a pipe, no <code>-C</code></td>
<td>10.6p1</td>
<td align="right">4.94 s</td>
<td align="right">1.92 to 6.94 s</td>
</tr>
</tbody></table>
<p>The slowest 10.5 run was faster than the fastest 10.6 run. The ranges are wide because the test machine was doing other work. The byte counts above are exact, so plan with them: at 20 Mbit/s, 3.71 MiB needs at least 1.6 s and 21.09 MiB needs at least 8.8 s.</p>
<h3>Who this affects</h3><ul>
<li>Anyone with <code>Compression yes</code> in ssh_config, or <code>ssh -C</code> and <code>scp -C</code> in scripts, over VPNs, satellite or mobile links.</li>
<li>Ansible on its default OpenSSH connection. The default <code>ssh_args</code> in ansible-core 2.21.5 are <code>-C -o ControlMaster=auto -o ControlPersist=60s</code>, and we saw them in its <code>-vvv</code> output. Most of that traffic is small modules, but <code>copy</code> and <code>fetch</code> move whole files through the same connection.</li>
<li>Probably less so <code>rsync -z</code> and git. Both compress their bulk data before SSH sees it, so <code>-C</code> should have had little left to do. We did not measure them.</li>
</ul>
<h3>What to do</h3><p>Compress at the application level, as the release notes recommend. On our files:</p>
<table>
<thead>
<tr>
<th>Method</th>
<th align="right">access.log ratio</th>
<th align="right">CPU seconds</th>
</tr>
</thead>
<tbody><tr>
<td>zlib level 6 (<code>ssh -C</code> before 10.6)</td>
<td align="right">8.73x</td>
<td align="right">1.50</td>
</tr>
<tr>
<td>zlib Huffman only (<code>ssh -C</code> in 10.6)</td>
<td align="right">1.52x</td>
<td align="right">1.25</td>
</tr>
<tr>
<td><code>gzip -6</code></td>
<td align="right">8.73x</td>
<td align="right">1.82</td>
</tr>
<tr>
<td><code>zstd -1</code></td>
<td align="right">7.63x</td>
<td align="right">0.44</td>
</tr>
<tr>
<td><code>zstd -3</code></td>
<td align="right">7.57x</td>
<td align="right">0.67</td>
</tr>
</tbody></table>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># A file, compressed with zstd on the way and decompressed on arrival.</span>
<span class="hljs-comment"># -o Compression=no stops a Compression yes in ssh_config from compressing twice.</span>
zstd -1 -c access.log | ssh -o Compression=no host <span class="hljs-string">'zstd -dc &gt; access.log'</span>

<span class="hljs-comment"># A directory</span>
tar -cf - ./logs | zstd -1 | ssh -o Compression=no host <span class="hljs-string">'zstd -dc | tar -xf - -C /srv/archive'</span>

<span class="hljs-comment"># rsync with its own compression (rsync 3.2 and newer can use zstd)</span>
rsync -az --compress-choice=zstd ./logs/ host:/srv/archive/
</code></pre><p>Compressing one stream on its own, as above, keeps data from other channels out of its dictionary, so the attack from the paper does not apply to it. For data that is already compressed, such as images, archives and most backups, <code>-C</code> usually gains little, as the random file above shows, and you can leave it off.</p>
<h2>Change 3: sshd now warns about classic key exchange</h2><p><code>WarnWeakCrypto</code> came to the client in 10.1. In 10.6 it is also a server option, on by default. Each connection that does not use a post-quantum key exchange adds one line to the log at INFO, the default level:</p>
<pre><code class="hljs language-text" tabindex="0">WARNING: connection from 127.0.0.1 port N is not using a post-quantum key exchange algorithm: "curve25519-sha256" [preauth]
</code></pre><p><code>06-weakcrypto.sh</code> connects eight clients to a 10.5 and a 10.6 sshd, both with default settings, and reads the negotiated key exchange from the log:</p>
<table>
<thead>
<tr>
<th>Client</th>
<th>Key exchange</th>
<th>10.6 sshd warns</th>
</tr>
</thead>
<tbody><tr>
<td>OpenSSH 10.6p1</td>
<td>mlkem768x25519-sha256</td>
<td>no</td>
</tr>
<tr>
<td>OpenSSH 10.5p1</td>
<td>mlkem768x25519-sha256</td>
<td>no</td>
</tr>
<tr>
<td>OpenSSH 9.2p1 (Debian 12)</td>
<td>sntrup761x25519-sha512</td>
<td>no</td>
</tr>
<tr>
<td>asyncssh 2.24.1</td>
<td>mlkem768x25519-sha256</td>
<td>no</td>
</tr>
<tr>
<td>Go golang.org/x/crypto/ssh v0.58.0</td>
<td>mlkem768x25519-sha256</td>
<td>no</td>
</tr>
<tr>
<td>paramiko 5.0.0</td>
<td><a href="mailto:curve25519-sha256@libssh.org">curve25519-sha256@libssh.org</a></td>
<td>yes</td>
</tr>
<tr>
<td>libssh2 1.10.0 (Debian 12 curl, sftp)</td>
<td>curve25519-sha256</td>
<td>yes</td>
</tr>
<tr>
<td>OpenSSH 10.6p1 with <code>KexAlgorithms=curve25519-sha256</code></td>
<td>curve25519-sha256</td>
<td>yes</td>
</tr>
</tbody></table>
<p>The 10.5 sshd logged no warning for any of them. We did not test Fabric or Netmiko. Both are built on paramiko, so expect one warning per connection from them too, unless your version negotiates a post-quantum key exchange. If you alert on <code>WARNING</code> in auth logs, expect new alerts from automation hosts.</p>
<p>You can turn it off with <code>WarnWeakCrypto no-pq-kex</code> (only this warning) or <code>WarnWeakCrypto no</code> (all of them). The option is global only. In a <code>Match Address</code> block, <code>sshd -t</code> fails:</p>
<pre><code class="hljs language-text" tabindex="0">sshd-match-test.conf line 10: Directive 'WarnWeakCrypto' is not allowed within a Match block
</code></pre><p>So you cannot keep the warning for people and hide it for one automation subnet. The better fix is on the client side: update the library, or move the job to a client that supports <code>mlkem768x25519-sha256</code>.</p>
<h2>What we could not conclude</h2><ul>
<li>We tested one machine and one OpenSSL build (3.0.22). The username check and the compression change are in OpenSSH's own code, so other platforms should behave the same, but we only measured this one.</li>
<li>We did not test the OpenSSH client that ships with Windows, or any vendor build that patches these changes out.</li>
<li>The timings vary because the machine was shared. The byte counts and the refusals do not depend on load.</li>
<li>Three of the four files are generated. The real source tar compressed less than either generated log (4.99x against 5.34x and 8.62x), and your own files will compress differently again.</li>
<li>We checked that <code>alice@corp.example.com</code> reaches the server. We did not test whether a Windows server or an SSSD host accepts it for login.</li>
<li>We tested libssh2 1.10.0 only, the version in Debian 12. Newer libssh2 releases may negotiate something else.</li>
</ul>
<h2>Summary</h2><p>OpenSSH 10.6 makes three changes that people will see within days of upgrading. Logins written as <code>CORP\alice</code> fail with <code>remote username contains invalid characters</code>, through ssh, scp, sftp, rsync, git and Ansible, unless the name is a literal <code>User</code> line in ssh_config or you switch to <code>alice@corp.example.com@host</code>. <code>ssh -C</code> now compresses text about 1.5x instead of 5x to 9x, and the side that sends the data decides, so move the compression to zstd or rsync. And a 10.6 sshd logs one warning per connection for paramiko and older libssh2 clients, with no way to scope that per subnet. Run <code>ssh -V</code> on your laptop today. If it says 10.6, check your inventories and scripts for <code>\</code> in usernames before the next on-call shift finds them for you.</p>
<p>For the post-quantum key exchange behind the third change, in TLS and in SSH, read <a href="https://devops-daily.com/guides/cryptography-essentials/03-tls-ssl#h2-the-same-change-in-ssh">TLS in Practice: Handshakes, Post-Quantum Key Exchange and Server Setup</a> in our Cryptography Essentials guide.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Python 3.15 Made UTF-8 the Default. We Ran 13 Cases in Seven Linux Locales to See What Changes]]></title>
      <link>https://devops-daily.com/posts/python-3-15-utf-8-default-linux-locales</link>
      <description><![CDATA[Python 3.15 turns on UTF-8 mode everywhere. With container locale settings, none of our 13 encoding cases changed. On a UTF-8 server, only stdin and stdout changed. On a legacy-locale host, almost everything did, and the PYTHONUTF8=0 opt-out dropped LC_ALL=C processes to ASCII.]]></description>
      <pubDate>Sat, 10 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/python-3-15-utf-8-default-linux-locales</guid>
      <category><![CDATA[Python]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Python]]></category><category><![CDATA[Python 3.15]]></category><category><![CDATA[UTF-8]]></category><category><![CDATA[Unicode]]></category><category><![CDATA[Linux]]></category><category><![CDATA[Containers]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>Python 3.15.0 came out on October 9. The headline change for anyone who runs Python on servers is <a href="https://peps.python.org/pep-0686/" rel="noopener noreferrer">PEP 686</a>: UTF-8 mode is now on by default. The <a href="https://docs.python.org/3.15/whatsnew/3.15.html" rel="noopener noreferrer">What's New page</a> says that <code>open('flying-circus.txt')</code> with no <code>encoding</code> now reads UTF-8, "independent of the system's environment". The PEP says the change "mostly affects Windows users", because most Unix systems already use a UTF-8 locale.</p>
<p>"Mostly" is not "only". We wanted to know exactly what changes on Linux, so we ran the same 13 cases on Python 3.14.8 and 3.15.0, each with its default settings and with <code>PYTHONUTF8=0</code>, in seven locales, from a bare container to a Latin-1 server. A file read with no <code>encoding</code> is the least of it. The changes we found are in stdin and stdout, on hosts with an old locale, and in what the opt-out switch really gives you.</p>
<h2>TLDR</h2><ul>
<li><strong>Container locale settings: no change.</strong> With no locale, <code>LANG=C</code> or <code>LC_ALL=C</code>, 3.14 and 3.15 gave the same result in all 13 cases. Python 3.7 and later already turn UTF-8 mode on in the C locale. The official <code>python</code> images set no <code>LANG</code> from 3.13, and <code>LANG=C.UTF-8</code> before that. Both settings gave zero differences.</li>
<li><strong>UTF-8 servers (<code>en_US.UTF-8</code>): only stdin and stdout change.</strong> <code>open()</code>, <code>subprocess</code> and environment variables gave the same results. Bytes that are not valid UTF-8 on stdin crashed 3.14 and pass through on 3.15 as lone surrogates. When we turned them into JSON, jq and Node replaced them with U+FFFD, so the original byte was lost.</li>
<li><strong>Legacy locales (<code>LANG=en_US</code>, which is ISO-8859-1 in glibc, and <code>ja_JP.eucJP</code>): 11 and 10 of 13 cases change.</strong> UTF-8 data starts to work. ISO-8859-1 data stops working. In the ISO-8859-1 locale, 3.14 never raised an error on UTF-8 input. It gave mojibake.</li>
<li><strong>The opt-out is not "what 3.14 did".</strong> <code>PYTHONUTF8=0</code> gave the same result on 3.14 and 3.15 in every case. What changed is the default: 3.14 turned UTF-8 mode on by itself in the C locale. So with <code>LC_ALL=C</code>, 3.15 with <code>PYTHONUTF8=0</code> fell back to ASCII. 7 cases differed from 3.14's defaults, and 4 that worked raised errors.</li>
<li><strong>You can find the affected calls today.</strong> <code>-X warn_default_encoding</code> reported the same 4 calls on 3.14 and 3.15. On a sample script, our static checker and the runtime warning reported the same 6 lines.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Python 3.14 or 3.15 on Linux, and a shell</li>
<li>Optional: the <a href="https://github.com/The-DevOps-Daily/python-315-utf8-default" rel="noopener noreferrer">demo repo</a>, which needs <a href="https://docs.astral.sh/uv/" rel="noopener noreferrer">uv</a> and glibc's <code>localedef</code></li>
</ul>
<h2>What UTF-8 mode changes, and where it was already on</h2><p>UTF-8 mode is not new. <a href="https://peps.python.org/pep-0540/" rel="noopener noreferrer">PEP 540</a> added it in Python 3.7. When it is on, Python ignores the locale encoding for four things, which the <a href="https://docs.python.org/3.15/library/os.html#utf8-mode" rel="noopener noreferrer">3.15 docs</a> list:</p>
<ol>
<li><code>sys.getfilesystemencoding()</code> returns <code>utf-8</code>, so file names, <code>os.environ</code> and <code>sys.argv</code> decode as UTF-8.</li>
<li><code>locale.getpreferredencoding()</code> returns <code>utf-8</code>, so <code>open()</code>, <code>Path.read_text()</code> and <code>subprocess</code> with <code>text=True</code> use UTF-8 when you give no <code>encoding</code>.</li>
<li>stdin and stdout use UTF-8 with the <code>surrogateescape</code> error handler.</li>
<li><code>os.device_encoding()</code> returns <code>utf-8</code> for a terminal on Unix.</li>
</ol>
<p>Before 3.15, the 3.14 docs say, it was on "if the LC_CTYPE locale is <code>C</code> or <code>POSIX</code> at Python startup". <a href="https://peps.python.org/pep-0538/" rel="noopener noreferrer">PEP 538</a>, also in 3.7, coerces a C locale to <code>C.UTF-8</code> and gives the coercion target locales <code>surrogateescape</code> on stdin and stdout too. That covers most containers. So on Linux, 3.15 changes behaviour in two places: real UTF-8 locales such as <code>en_US.UTF-8</code>, and locales that are not UTF-8 at all.</p>
<h2>The test</h2><p>The <a href="https://github.com/The-DevOps-Daily/python-315-utf8-default" rel="noopener noreferrer">repo</a> has 13 small cases. Each one does one thing that depends on the default encoding: read an ISO-8859-1 file, read a UTF-8 file, write <code>café 25€</code>, read a child process's output with <code>text=True</code>, read an ISO-8859-1 value from <code>os.environ</code> and <code>sys.argv</code>, read stdin, print a file name that is not valid UTF-8, and so on. Two of the 13 are controls that should not depend on the locale: a read with <code>encoding="locale"</code>, and the <code>EncodingWarning</code> check. A separate check, <code>handoff</code>, has one version write a file and the other read it on the same host.</p>
<p>The runner starts every case from an empty environment, with only <code>PATH</code>, <code>LOCPATH</code>, <code>PYTHONDONTWRITEBYTECODE</code> and the locale variables. It runs each case with four interpreter settings (3.14.8 and 3.15.0, each with its default and with <code>PYTHONUTF8=0</code>) in seven locales:</p>
<table>
<thead>
<tr>
<th>Locale</th>
<th>Stands for</th>
</tr>
</thead>
<tbody><tr>
<td>no locale</td>
<td>an image that sets no locale, such as the official <code>python:3.13</code> and later images</td>
</tr>
<tr>
<td><code>LANG=C</code></td>
<td>an image or script that sets the C locale</td>
</tr>
<tr>
<td><code>LC_ALL=C</code></td>
<td>the same, set the way build scripts set it for stable sort order</td>
</tr>
<tr>
<td><code>LANG=C.UTF-8</code></td>
<td>the official <code>python:3.12</code> and older images</td>
</tr>
<tr>
<td><code>LANG=en_US.UTF-8</code></td>
<td>a typical server or laptop</td>
</tr>
<tr>
<td><code>LANG=en_US</code></td>
<td>ISO-8859-1 in glibc's list of supported locales: an older server set up for a legacy app</td>
</tr>
<tr>
<td><code>LANG=ja_JP.eucJP</code></td>
<td>a legacy CJK locale</td>
</tr>
</tbody></table>
<p><code>localedef</code> builds the locales into the repo, so the result does not depend on what the host has installed. Before it trusts a row, the runner checks that the locale reports the encoding it should. If it does not, the runner stops, so a missing locale cannot pass as a result. A case "differs" between two settings when anything recorded differs: the returned value, the error message, the exit code or the bytes written. We wrote the claim in <a href="https://github.com/The-DevOps-Daily/python-315-utf8-default/blob/main/CLAIM.md" rel="noopener noreferrer"><code>CLAIM.md</code></a> before the first run, with the results that would prove it wrong, and <code>check_claim.py</code> checks it against the recorded results.</p>
<p><strong>python-315-utf8-default</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./scripts/setup.sh
locales: C.UTF-8 en_US en_US.UTF-8 ja_JP.eucJP
$ python3 scripts/run_matrix.py
406 results from 420 interpreter runs written to results/matrix.json
</code></pre><p>Then <code>python3 scripts/check_claim.py</code> checks the claim. This is what it wrote to <code>results/claim-check.txt</code>:</p>
<pre><code class="hljs language-text" tabindex="0">PASS  no locale: 3.14 and 3.15 agree in every case
PASS  LANG=C: 3.14 and 3.15 agree in every case
PASS  LC_ALL=C: 3.14 and 3.15 agree in every case
PASS  LANG=C.UTF-8: open(), subprocess and environment cases agree
FAIL  LANG=C.UTF-8: invalid UTF-8 on stdin/stdout differs between 3.14 and 3.15
      differs: nothing
PASS  LANG=en_US.UTF-8: open(), subprocess and environment cases agree
      stdin_latin1, stdin_to_json, stdout_filename
PASS  LANG=en_US.UTF-8: invalid UTF-8 on stdin/stdout differs between 3.14 and 3.15
      differs: stdin_latin1, stdin_to_json, stdout_filename
PASS  LANG=en_US: every case that uses the default encoding differs (11 of 13)
      agree: read_latin1_locale, encoding_warning
FAIL  LANG=ja_JP.eucJP: every case that uses the default encoding differs (10 of 13)
      agree: read_latin1_locale, environ_argv, encoding_warning
FAIL  3.15 + PYTHONUTF8=0 gives the 3.14 default result in every case
      LC_ALL=C: read_latin1; LC_ALL=C: read_utf8; LC_ALL=C: write_text; LC_ALL=C: subprocess_latin1; LC_ALL=C: subprocess_utf8; LC_ALL=C: stdin_utf8; LC_ALL=C: stdout_utf8
PASS  PYTHONUTF8=0 gives the same result on 3.14 and 3.15 in every case
PASS  -X warn_default_encoding reports the same 4 calls on 3.14 and 3.15
</code></pre><p>The three <code>FAIL</code> lines are where our claim was wrong. We expected <code>C.UTF-8</code> to behave like <code>en_US.UTF-8</code>. We expected every default-encoding case to change in <code>ja_JP.eucJP</code>. And we expected <code>PYTHONUTF8=0</code> to bring back 3.14 behaviour everywhere. All three are covered below.</p>
<p><strong>Cases where 3.15 gives a different result from 3.14</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>no locale</td>
<td>0 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LANG=C</td>
<td>0 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LC_ALL=C</td>
<td>0 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LANG=C.UTF-8</td>
<td>0 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LANG=en_US.UTF-8</td>
<td>3 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LANG=en_US (ISO-8859-1)</td>
<td>11 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LANG=ja_JP.eucJP</td>
<td>10 of 13</td>
<td>3.15 default</td>
</tr>
<tr>
<td>LC_ALL=C, PYTHONUTF8=0</td>
<td>7 of 13</td>
<td>3.15 with PYTHONUTF8=0</td>
</tr>
</tbody></table>
<p><em>13 cases per locale, Python 3.14.8 vs 3.15.0, Linux aarch64, glibc 2.36. A case differs when the value, error, exit code or bytes written differ. The last row compares 3.14 defaults with 3.15 run with PYTHONUTF8=0. Source: results/matrix.json in the demo repo.</em></p>
<h2>Container locale settings: nothing changed</h2><p>In the first three locales, 3.14 and 3.15 agreed in every case, including the ones that fail. An ISO-8859-1 file read with no <code>encoding</code> raised <code>UnicodeDecodeError</code> on both. A UTF-8 file read correctly on both. That is UTF-8 mode working on 3.14 already, because the locale was C.</p>
<p>We expected <code>LANG=C.UTF-8</code> to show the stdin and stdout change, since 3.14 has UTF-8 mode off there. It did not. 3.14 already uses <code>surrogateescape</code> on stdin and stdout in <code>C.UTF-8</code>, because that is one of the locales PEP 538 coerces C to, and the PEP gives those locales the same error handler. So all four container-style locales agreed on all 13 cases.</p>
<p>That matters for images. The <a href="https://github.com/docker-library/python" rel="noopener noreferrer">official python Dockerfiles</a> dropped <code>ENV LANG C.UTF-8</code> for 3.13 and later in <a href="https://github.com/docker-library/python/commit/0d539116871a326fd6b3d35b0c528b8fbdffe573" rel="noopener noreferrer">December 2023</a>, and the 3.12 and older images still set it. Both settings are rows with zero differences. We did not run the images themselves. We ran their locale settings with uv's Python builds on one glibc host, so treat this as the result for those settings, not a test of every image. Alpine uses musl, a different C library, and we did not test it.</p>
<h2>UTF-8 servers: stdin and stdout are what change</h2><p>In <code>en_US.UTF-8</code>, the <code>open()</code>, <code>subprocess</code> and environment cases agreed. The locale encoding was UTF-8, so the default was UTF-8 before and after. Three cases changed, and all three are about bytes that are not valid UTF-8 on stdin or stdout:</p>
<table>
<thead>
<tr>
<th>Case, <code>LANG=en_US.UTF-8</code></th>
<th>3.14</th>
<th>3.15</th>
</tr>
</thead>
<tbody><tr>
<td><code>sys.stdin.read()</code> on <code>caf\xe9</code></td>
<td><code>UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3</code></td>
<td><code>'caf\udce9\n'</code></td>
</tr>
<tr>
<td>Print a file named <code>report-\xe9.csv</code></td>
<td>exit 1, <code>UnicodeEncodeError: ... surrogates not allowed</code></td>
<td>exit 0, writes the original bytes</td>
</tr>
<tr>
<td>stdin to <code>json.dumps</code> to a file</td>
<td><code>UnicodeDecodeError</code> at the read</td>
<td>writes <code>{"line": "caf\udce9\n"}</code></td>
</tr>
</tbody></table>
<p>3.14 used the <code>strict</code> error handler on stdin and stdout in a normal UTF-8 locale. 3.15 uses <code>surrogateescape</code>, which maps each bad byte to a lone surrogate (<code>\udce9</code> for <code>0xe9</code>). You get the original byte back only where something encodes the text with <code>surrogateescape</code> again. stdout does, which is why printing the odd file name now works. A strict UTF-8 encode, such as writing to a file opened with <code>encoding="utf-8"</code>, raises. And some consumers replace the surrogate. So the error moves from the input to a later step, or there is no error and the data changes.</p>
<p>The JSON case shows the second outcome. <code>json.dumps</code> with its defaults writes the surrogate as the escape <code>\udce9</code>, which is valid JSON syntax but not a valid character. <a href="https://www.rfc-editor.org/rfc/rfc8259#section-8.2" rel="noopener noreferrer">RFC 8259 section 8.2</a> says the behaviour of software that receives such values "is unpredictable". We piped the line into two consumers:</p>
<pre><code class="hljs language-text" tabindex="0">input bytes on stdin:   63 61 66 e9 0a (the script strips the newline)
3.15 wrote:             {"line": "caf\udce9"}
jq-1.6 -r .line: 63 61 66 ef bf bd 0a
node v24.14.0 JSON.parse, written as UTF-8: 63 61 66 ef bf bd 0a
ef bf bd is U+FFFD, the replacement character: the original byte e9 is gone.
</code></pre><p>On 3.14 this pipeline crashed at the read, on a UTF-8 server. On 3.15 it runs, and both consumers we tried replaced the byte with U+FFFD. In the container locales, 3.14 already wrote the same line, so the risk is not new. What is new is that a log shipper or ETL script that crashed on bad input on your UTF-8 servers now writes slightly wrong data instead.</p>
<p>If you want the 3.14 crash back for one script, either of the first two lines below works on 3.15. The third keeps the raw bytes instead:</p>
<pre><code class="hljs language-text" tabindex="0">default:                      'caf\udce9\n'
stdin.reconfigure(strict):    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
PYTHONIOENCODING=utf-8:strict UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
sys.stdin.buffer (bytes):     b'caf\xe9\n'
</code></pre><p><code>sys.stdin.reconfigure(errors="strict")</code> at the top of the script is the most explicit way to fail at the input. If the input can be any bytes, read <code>sys.stdin.buffer</code> and decode it yourself with the encoding you expect.</p>
<h2>Legacy locales: one set of failures for another</h2><p>On a host with <code>LANG=en_US</code>, glibc uses ISO-8859-1. 11 of the 13 cases changed, which is every case except the two controls:</p>
<ul>
<li><strong>Five got better.</strong> Reading a UTF-8 file, writing <code>café 25€</code>, reading UTF-8 from a child process, reading UTF-8 on stdin, and printing <code>café 25€</code> all work on 3.15. On 3.14, the two that write <code>€</code> raised errors, because ISO-8859-1 has no <code>€</code>. The other three gave mojibake with no error: <code>'caf\xc3\xa9 25\xe2\x82\xac'</code> instead of <code>café 25€</code>. ISO-8859-1 maps every byte to a character, so decoding never fails. It only gives wrong text.</li>
<li><strong>Five got worse.</strong> Reading an ISO-8859-1 file now raises <code>UnicodeDecodeError</code>. So does reading ISO-8859-1 output from a child process. An ISO-8859-1 value in <code>os.environ</code> or <code>sys.argv</code> now decodes as <code>'caf\udce9'</code> and fails later, when you encode it as UTF-8. Stdin turns into surrogates, and the JSON line changes from <code>caf\u00e9</code> to <code>caf\udce9</code>.</li>
<li><strong>One printed the same bytes</strong> by a different route: <code>os.listdir()</code> returned a different string, and stdout turned it back into the same bytes.</li>
</ul>
<p>In <code>ja_JP.eucJP</code>, 10 of 13 cases changed. The environment case did not, because its ISO-8859-1 byte is invalid in EUC-JP too, so both versions decode it to the same surrogate. Here 3.14's EUC-JP decoder raised on UTF-8 input, where ISO-8859-1 had given mojibake. Our fixtures hold UTF-8 and ISO-8859-1 text only, so this row says nothing about valid EUC-JP data. If you have Japanese legacy data, test it with your own files.</p>
<p>The <code>handoff</code> check shows the risk during a rolling upgrade. On the <code>LANG=en_US</code> host, a 3.15 job wrote <code>café 25€</code> as UTF-8, and a 3.14 job read it back as <code>'caf\xc3\xa9 25\xe2\x82\xac\n'</code>, with no error. The other direction failed loudly: 3.14 could not encode <code>€</code> in ISO-8859-1, and because <code>open("w")</code> creates or truncates the file before the write, it left an empty file.</p>
<p>If your legacy data really is ISO-8859-1, say so in the code. <code>encoding="latin-1"</code> reads it correctly on every version. <code>encoding="locale"</code>, added in Python 3.10, uses the locale encoding even in UTF-8 mode. It read the ISO-8859-1 file correctly on 3.14 and 3.15 in the <code>en_US</code> locale.</p>
<h2>The opt-out is not "what 3.14 did"</h2><p>The release notes give one way back: "To retain the previous behaviour, Python's UTF-8 mode may be disabled with the <code>PYTHONUTF8=0</code> environment variable or the <code>-X utf8=0</code> command-line option." The switch itself did not change. With <code>PYTHONUTF8=0</code>, 3.14 and 3.15 gave the same result in all 13 cases in all seven locales. What changed is the default, and the default in 3.14 was not "off". It was "on in the C locale, off elsewhere". <code>PYTHONUTF8=0</code> is off everywhere.</p>
<p>In six of the seven locales, that made no difference: 3.15 with <code>PYTHONUTF8=0</code> matched 3.14's defaults. With <code>LANG=C</code>, PEP 538 still coerces the locale to <code>C.UTF-8</code>, so the encoding stays UTF-8. With <code>LC_ALL=C</code>, coercion does not happen (PEP 538 skips it when <code>LC_ALL</code> is set), so the locale encoding is ASCII, and that is what 3.15 used:</p>
<pre><code class="hljs language-text" tabindex="0">## LC_ALL=C with PYTHONUTF8=0: reading a UTF-8 file
no encoding:                  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
encoding="utf-8":             'caf\xe9 25\u20ac\n'
</code></pre><p>7 of the 13 cases differed from 3.14's defaults. Four that worked on 3.14 raised: reading a UTF-8 file, writing <code>café 25€</code>, reading UTF-8 from a child process, and printing <code>café 25€</code>. One more turned UTF-8 on stdin into surrogates with no error. The other two failed on both, with a different error message.</p>
<p>So do not put <code>PYTHONUTF8=0</code> in a base image or a shared profile as a "keep the old behaviour" switch. Any job that also runs with <code>LC_ALL=C</code> gets ASCII, which 3.14 with its defaults never gave it. If you need the opt-out, set it only for the processes that run in a legacy locale and read legacy data, and plan to remove it.</p>
<h2>What to do before you upgrade</h2><p>Most of this goes away when the code names its encoding. Three steps, in order:</p>
<ol>
<li><p><strong>Find the calls.</strong> The repo has a static checker, <code>check_default_encoding.py</code>. It reports <code>open()</code> without <code>encoding</code> (or with <code>encoding=None</code>), <code>Path.read_text()</code> and <code>write_text()</code>, <code>.open()</code> on a <code>Path</code> it can see, <code>subprocess</code> in text mode with no <code>encoding</code>, <code>os.popen()</code>, logging file handlers, <code>tempfile</code> and <code>gzip.open()</code> in text mode, <code>io.TextIOWrapper()</code> and <code>locale.getpreferredencoding()</code>. On a sample release-notes script, it found 6 calls:</p>
<pre><code class="hljs language-text" tabindex="0">examples/release_notes.py:12:1: logging.basicConfig(filename=...) without encoding
examples/release_notes.py:16:11: subprocess.run() in text mode without encoding
examples/release_notes.py:23:24: .read_text() without encoding
examples/release_notes.py:24:16: open() without encoding
examples/release_notes.py:27:5: .write_text() without encoding
examples/release_notes.py:28:10: open() without encoding
6 call(s) use the default encoding
</code></pre><p>When the same script ran on 3.15 with <code>-X warn_default_encoding</code>, Python reported <code>EncodingWarning</code> on the same 6 lines. A static check misses calls through variables, wrappers and <code>**kwargs</code> it cannot read, so do step 2 as well.</p>
</li>
<li><p><strong>Run your tests with the warning as an error.</strong> This works on 3.10 and later, so you can do it on 3.14 now:</p>
<pre><code class="hljs language-bash" tabindex="0">python -X warn_default_encoding -W error::EncodingWarning -m pytest
</code></pre><p>In our matrix, the warning fired for the same 4 calls on 3.14 and 3.15. Fix each <code>open()</code>-style call with <code>encoding="utf-8"</code>, or <code>encoding="locale"</code> where the file really follows the locale. <code>locale.getpreferredencoding()</code> takes no encoding: replace it with <code>locale.getencoding()</code> if you need the locale encoding, or with <code>"utf-8"</code> if that is what the code meant.</p>
</li>
<li><p><strong>Decide what stdin and stdout should do with bad bytes.</strong> If a script reads input it does not control, such as logs, file names or another tool's output, choose on purpose: <code>sys.stdin.reconfigure(errors="strict")</code> to fail at the input, or <code>sys.stdin.buffer</code> to keep the bytes. Do not let the default choose for you.</p>
</li>
</ol>
<p>Then check which locale your hosts really run in. <code>locale</code> and <code>python3 -c "import locale; print(locale.getencoding())"</code> on each host type tell you. If every host is <code>C</code>, <code>C.UTF-8</code> or no locale, none of our 13 cases changed for those settings.</p>
<p><a href="https://github.com/The-DevOps-Daily/python-315-utf8-default" rel="noopener noreferrer">The-DevOps-Daily/python-315-utf8-default on GitHub</a></p>
<h2>What we could not conclude</h2><ul>
<li><strong>Windows was not tested.</strong> The PEP says Windows is where most of the change is, because the locale encoding there is the ANSI code page, for example cp1252. If you run Python on Windows hosts or build agents, run the checker and the warning there first.</li>
<li><strong>macOS and musl (Alpine) were not tested.</strong> We ran uv's Python builds on Linux aarch64 with glibc 2.36. We have no result for either platform.</li>
<li><strong>The EUC-JP row used UTF-8 and ISO-8859-1 fixtures</strong>, not valid EUC-JP text.</li>
<li><strong>13 cases are not your code.</strong> Third-party libraries that call <code>locale.getpreferredencoding()</code> themselves now get <code>utf-8</code> in UTF-8 mode. The checker only sees the code you point it at.</li>
<li><strong>We did not count how many hosts use legacy locales.</strong> We know the behaviour on such a host, not how common those hosts are.</li>
</ul>
<h2>Summary</h2><p>PEP 686 is a smaller change for Linux containers than its headline suggests, because Python has used UTF-8 mode in the C locale since 3.7. With the locale settings that containers use, it changed none of our 13 cases. It changes how stdin and stdout treat invalid bytes on UTF-8 servers, so a crash at the input becomes a lone surrogate further down the pipeline. It changes almost everything on hosts with a legacy locale. And its opt-out, <code>PYTHONUTF8=0</code>, means "off", so <code>LC_ALL=C</code> processes get ASCII, which 3.14 with its defaults never gave them. Name the encoding in your code, run your tests once with <code>-X warn_default_encoding</code>, and decide what your scripts do with bytes that are not UTF-8.</p>
]]></content:encoded>
    </item>
    <item>
      <title><![CDATA[Docker Desktop Switched New Containers to the local Log Driver. We Measured What That Changes]]></title>
      <link>https://devops-daily.com/posts/docker-desktop-local-log-driver</link>
      <description><![CDATA[Docker Desktop 4.94 made local the default log driver for new Linux containers, while Docker Engine on your servers still uses json-file with no rotation. We wrote the same 500,000 lines through both. local kept every line in 24 MB where json-file used 84 MB. A file-tailing collector read none of them.]]></description>
      <pubDate>Fri, 09 Oct 2026 09:00:00 GMT</pubDate>
      <guid isPermaLink="true">https://devops-daily.com/posts/docker-desktop-local-log-driver</guid>
      <category><![CDATA[Docker]]></category>
      <dc:creator><![CDATA[DevOps Daily Team]]></dc:creator>
      <category><![CDATA[Docker]]></category><category><![CDATA[Logging]]></category><category><![CDATA[Docker Desktop]]></category><category><![CDATA[Fluent Bit]]></category><category><![CDATA[Observability]]></category><category><![CDATA[DevOps]]></category>
      <content:encoded><![CDATA[<p>Docker Desktop 4.94.0 came out on October 5. One line in its <a href="https://docs.docker.com/desktop/release-notes/" rel="noopener noreferrer">release notes</a> changes where your container logs go: "Changed the default logging driver for new Linux containers to <code>local</code> to enable automatic log rotation and reduce disk usage." Docker Engine, which runs on your servers, did not change. It still writes <code>json-file</code> logs, and by default it never rotates them.</p>
<p>So your laptop and your servers now store logs in two different formats. Most of the time you will not notice, because <code>docker logs</code> reads both. You will notice if something reads the log files directly, such as a collector that tails <code>/var/lib/docker/containers/*/*.log</code>. We wrote the same log lines through both drivers on Docker Engine 29.9.0 and recorded what each one stores, what <code>docker logs</code> gives back, and what a file-tailing collector reads.</p>
<h2>TLDR</h2><ul>
<li>Docker Desktop 4.94 uses <code>local</code> for new Linux containers. Docker Engine still defaults to <code>json-file</code> with <code>max-size</code> unlimited, and Docker's docs say that is only for backward compatibility.</li>
<li>500,000 log lines (47.9 MB of text) took 83.8 MB with <code>json-file</code> and no options. <code>local</code> kept all 500,000 lines in 24.1 MB. <code>json-file</code> capped at 10 MB x 3 used 23.8 MB, but kept only 142,146 lines.</li>
<li><code>docker logs</code> returned all 500,000 lines from the uncapped <code>json-file</code> container and from the <code>local</code> one.</li>
<li>Fluent Bit's <code>tail</code> input, set up the usual way for a Docker host, read 500,000 records from the <code>json-file</code> container and 0 from the <code>local</code> one. Pointed at the <code>local</code> file directly, it read binary framing that the docker parser could not parse.</li>
<li>A changed default only reaches containers created after the change. Existing containers keep <code>json-file</code> until you recreate them.</li>
</ul>
<h2>Prerequisites</h2><ul>
<li>Docker Engine or Docker Desktop, and <code>docker info</code> access</li>
<li><code>sudo</code> on a Linux Docker host, if you want to look at the log files yourself</li>
<li>Optional: the <a href="https://github.com/The-DevOps-Daily/docker-log-driver-check" rel="noopener noreferrer">demo repo</a>, which runs every test in this post</li>
</ul>
<h2>What changed, and what did not</h2><p>Docker has always written container output to files on the host, in a format chosen by the log driver. The default has been <code>json-file</code> since the early days: one JSON object per line, with the text, the stream and a timestamp. Its <a href="https://docs.docker.com/engine/logging/drivers/json-file/" rel="noopener noreferrer">documentation</a> gives <code>max-size</code> a default of <code>-1</code>, which means unlimited, so one noisy container can fill the disk. The <a href="https://docs.docker.com/engine/logging/configure/" rel="noopener noreferrer">logging overview</a> says Docker keeps <code>json-file</code> without rotation as the default "to remain backwards compatible with older versions of Docker", and recommends <code>local</code> for other cases because "it performs log-rotation by default, and uses a more efficient file format."</p>
<p>The <a href="https://docs.docker.com/engine/logging/drivers/local/" rel="noopener noreferrer"><code>local</code> driver</a> keeps 5 files of 20 MB each per container, so 100 MB of log text, and compresses the files it rotates out. Docker Desktop 4.94 now uses it for new Linux containers. Docker Engine 29.9.0, the newest Engine release when we tested, still reports <code>json-file</code>:</p>
<p><strong>Docker Engine 29.9.0 on Linux</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ docker version --format <span class="hljs-string">'docker client {{.Client.Version}}, engine {{.Server.Version}}'</span>
docker client 29.9.0, engine 29.9.0
$ docker info --format <span class="hljs-string">'{{.LoggingDriver}}'</span>
json-file
</code></pre><p>We could not run Docker Desktop on our test machine. The Desktop behaviour in this post comes from its release notes. Every measurement comes from Docker Engine 29.9.0 with the same two drivers. The output above is from a daemon with no logging settings (<code>runs/.../00-default.txt</code>).</p>
<h2>The test</h2><p>The <a href="https://github.com/The-DevOps-Daily/docker-log-driver-check" rel="noopener noreferrer">demo repo</a> starts three containers that each print the same 500,000 lines, about 96 bytes each, shaped like access logs:</p>
<ul>
<li><code>logs-json-default</code>: <code>json-file</code> with no options, which is what Docker Engine does out of the box</li>
<li><code>logs-json-capped</code>: <code>json-file</code> with <code>max-size=10m</code> and <code>max-file=3</code>, a common hand-made fix</li>
<li><code>logs-local</code>: <code>local</code> with its defaults, which is what Docker Desktop 4.94 now does</li>
</ul>
<p>Then it measures the files each driver wrote, counts what <code>docker logs</code> returns, and runs a real collector against the files. We ran it on a Raspberry Pi 4 (arm64) with Docker Engine 29.9.0 and Fluent Bit 5.1.3. The outputs below come from <code>runs/2026-10-09-engine-29.9.0/</code> in the repo.</p>
<p><strong>Same 500,000 lines, three setups (excerpt)</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ LINES=500000 ./scripts/01-write-logs.sh
logs-json-default: driver=json-file options={}
logs-json-capped: driver=json-file options={<span class="hljs-string">"max-file"</span>:<span class="hljs-string">"3"</span>,<span class="hljs-string">"max-size"</span>:<span class="hljs-string">"10m"</span>}
logs-local: driver=<span class="hljs-built_in">local</span> options={}
$ ./scripts/02-measure.sh
container          driver       file_bytes  files   lines_kept  bytes_of_text
logs-json-default  json-file      83833258      1       500000       47888900
logs-json-capped   json-file      23833155      3       142146       13614435
logs-local         <span class="hljs-built_in">local</span>          24134700      3       500000       47888900
<span class="hljs-comment"># the file listings and byte dumps that follow are in 02-measure.txt</span>
</code></pre><p><strong>Size of the log files for the same 500,000 lines</strong></p>
<table>
<thead>
<tr>
<th></th>
<th>Value</th>
<th>Series</th>
</tr>
</thead>
<tbody><tr>
<td>json-file, no options</td>
<td>83.8MB</td>
<td>json-file</td>
</tr>
<tr>
<td>json-file, 10m x 3</td>
<td>23.8MB</td>
<td>json-file</td>
</tr>
<tr>
<td>local, defaults</td>
<td>24.1MB</td>
<td>local</td>
</tr>
</tbody></table>
<p><em>Docker Engine 29.9.0, 500,000 lines of about 96 bytes (47.9 MB of text). json-file with no options kept all 500,000 lines, json-file capped at 10 MB x 3 kept 142,146, local kept all 500,000. These lines repeat a lot; compression depends on your log content.</em></p>
<p>Three things stand out:</p>
<ol>
<li><strong><code>json-file</code> costs more than the text.</strong> 47.9 MB of text took 83.8 MB of files, 1.75 times the text, because every line is wrapped in <code>{"log":...,"stream":...,"time":...}</code>. That is about 72 extra bytes per line here. With no <code>max-size</code>, the file only grows.</li>
<li><strong>Our size cap on <code>json-file</code> threw data away.</strong> The capped container used 23.8 MB and <code>docker logs</code> returned only 142,146 of the 500,000 lines. The rest rotated out.</li>
<li><strong><code>local</code> kept everything in about the same space.</strong> It used 24.1 MB and returned all 500,000 lines. The current file was 19.4 MB, uncompressed, framing included. Each of the two rotated files was about 2.3 MB after gzip, and the driver rotates a file at 20 MB before compression.</li>
</ol>
<p>The two capped setups do not have the same budget. <code>json-file</code> at 10 MB x 3 allows 30 MB. <code>local</code> allows 5 files of 20 MB each before compression, framing included, and it also drops its oldest lines when it reaches that limit. A container that writes more than about 100 MB of log records loses lines with either driver. These lines repeat a lot, so they compressed well; how well your logs compress depends on their content.</p>
<h2>The file format is what changes</h2><p><code>docker logs</code> returned all 500,000 lines from both uncapped containers, because it reads through the Docker daemon. The files on disk are a different story. Here are the first bytes of each file, from the same run:</p>
<p><strong>First bytes of each driver's file, from 02-measure.txt (first 3 of 6 od lines)</strong></p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># json-file: &lt;container-id&gt;/&lt;container-id&gt;-json.log</span>
0000000   {   <span class="hljs-string">"   l   o   g   "</span>   :   <span class="hljs-string">"   2   0   2   6   -   1   0   -
0000020   0   9   T   1   0   :   0   0   :   0   0   .   0   0   0   0
0000040   0   1   Z       l   e   v   e   l   =   i   n   f   o       m
# local: &lt;container-id&gt;/local-logs/container.log
0000000  \0  \0  \0   s  \n 006   s   t   d   o   u   t 020 205 330 232
0000020 342 244 221 265 356 030 032   _   2   0   2   6   -   1   0   -
0000040   0   9   T   1   0   :   0   0   :   0   0   .   3   4   1   6</span>
</code></pre><p><code>json-file</code> writes one JSON object per line, in a file named <code>&lt;id&gt;-json.log</code> in the container's directory. <code>local</code> writes length-prefixed binary records in a <code>local-logs/</code> subdirectory, named <code>container.log</code>, and the rotated files end in <code>.gz</code>. Docker's <a href="https://docs.docker.com/engine/logging/drivers/local/" rel="noopener noreferrer">docs</a> say these files "are designed to be exclusively accessed by the Docker daemon", and that reading them with external tools "should be avoided".</p>
<h2>What a file-tailing collector sees</h2><p>Many Docker hosts ship logs with an agent that tails the files. We ran Fluent Bit 5.1.3 with its <code>tail</code> input and the <code>docker</code> parser on each container's <code>*.log</code> files, the usual setup for a Docker host:</p>
<p><strong>Fluent Bit tail input, docker parser (excerpt)</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./scripts/03-collector.sh
Fluent Bit: Fluent Bit v5.1.3

== the usual Docker host pattern: &lt;container <span class="hljs-built_in">dir</span>&gt;/*.<span class="hljs-built_in">log</span>, docker parser
logs-json-default  records <span class="hljs-built_in">read</span>: 500000
logs-json-capped   records <span class="hljs-built_in">read</span>: 22860
logs-local         records <span class="hljs-built_in">read</span>: 0

== aimed at the <span class="hljs-built_in">local</span> driver<span class="hljs-string">'s own file: local-logs/container.log
records read: 158382
records the docker parser could parse: 0
# two sample records follow; the first 20 are in 03-local-raw-head.txt</span>
</code></pre><p>The <code>json-file</code> container gave all 500,000 records. For the <code>local</code> container the counter reported 0, because <code>*.log</code> in the container directory does not match <code>local-logs/container.log</code>. When we pointed Fluent Bit at that file directly, it read 158,382 records and the parser understood none of them. The tail input splits on newline bytes, which this binary format also uses inside its framing, so the records are fragments: the first one holds only a length byte, and the others hold one message each with binary framing around it, in the form <code>^Fstdout^P...</code> (shown with <code>cat -v</code>). It also left out the two rotated <code>.gz</code> files.</p>
<p>The capped container's 22,860 is a side effect of the test. Fluent Bit started after the container had finished, so it found only the current <code>-json.log</code> file. A collector that runs all the time can read lines before they rotate, but this run did not test that.</p>
<p>What this means for other collectors depends on how they read:</p>
<ul>
<li><strong>Collectors that read through the Docker daemon</strong> should keep working, because the daemon reads the <code>local</code> format for them, as it does for <code>docker logs</code>. Grafana Alloy's <a href="https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.docker/" rel="noopener noreferrer"><code>loki.source.docker</code></a> "reads log entries from Docker containers" through the daemon address you give it. Vector's <a href="https://vector.dev/docs/reference/configuration/sources/docker_logs/" rel="noopener noreferrer"><code>docker_logs</code></a> source connects to <code>docker_host</code>. We did not test either of them with <code>local</code>.</li>
<li><strong>Collectors that parse the files as Docker JSON</strong> cannot decode <code>local</code>. Fluent Bit's <code>tail</code> input with the docker parser is one, as measured above. Filebeat's <a href="https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-container" rel="noopener noreferrer">container input</a> reads files and parses the <code>docker</code> and <code>cri</code> formats, and the <code>local</code> format is neither. That input is deprecated, and Filebeat's docs say to use the <code>filestream</code> input with its <code>container</code> parser instead.</li>
</ul>
<p>We tested Fluent Bit only. For the others, the docs above are our source, so test your own agent before you depend on this.</p>
<h2>Existing containers keep their driver</h2><p>The log driver is set when a container is created. Docker's <a href="https://docs.docker.com/engine/logging/configure/" rel="noopener noreferrer">docs</a> say that changing the default "only affects containers that are created after the configuration is changed". We checked it: we created a container, restarted the daemon with <code>local</code> as the default, then created a second one and restarted the first.</p>
<p><strong>Changing the daemon default</strong></p>
<pre><code class="hljs language-bash" tabindex="0">$ ./scripts/04-default-change.sh before
== before: daemon default is json-file
logs-before: json-file
<span class="hljs-comment"># restart dockerd with --log-driver local</span>
$ ./scripts/04-default-change.sh after
== after: daemon default is <span class="hljs-built_in">local</span>
logs-before (created before, started again now): json-file
logs-after  (created after): <span class="hljs-built_in">local</span>
</code></pre><p>We tested this on Engine, not on a Desktop upgrade, but the rule is the same. On a laptop that you upgrade to 4.94, the containers you already have keep the driver they were created with. A container that you remove and run again, or that Compose recreates, gets the new default, unless its configuration names a driver. For a while, one machine can have both formats side by side.</p>
<h2>What to do</h2><p>Check which driver each container actually has, not only the default:</p>
<pre><code class="hljs language-bash" tabindex="0"><span class="hljs-comment"># The daemon default</span>
docker info --format <span class="hljs-string">'{{.LoggingDriver}}'</span>

<span class="hljs-comment"># The driver of every container, running or not</span>
docker ps -aq | xargs docker inspect --format <span class="hljs-string">'{{.Name}} {{.HostConfig.LogConfig.Type}}'</span>
</code></pre><p><strong>On servers running Docker Engine</strong>, the risk is the old one: <code>json-file</code> with no limit. Pick a default in <code>/etc/docker/daemon.json</code>. Use <code>local</code> if your collectors read through the Docker API:</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"log-driver"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"local"</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>Or keep <code>json-file</code> for a file-tailing collector and give it a limit:</p>
<pre><code class="hljs language-json" tabindex="0"><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"log-driver"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"json-file"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"log-opts"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"max-size"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"10m"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"max-file"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"3"</span>
  <span class="hljs-punctuation">}</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>Restart the daemon, then recreate the containers so that they pick up the setting. As the measurements show, a size cap on <code>json-file</code> means the oldest lines are gone from disk. That is fine if a collector has already shipped them, and a loss if it has not.</p>
<p><strong>On Docker Desktop</strong>, you probably need to do nothing. <code>docker logs</code> works with both drivers, and tools that use the same API should too. If you run a file-tailing collector on your laptop, set <code>"log-driver": "json-file"</code> in Docker Desktop's daemon settings, or move the collector to a source that reads through the Docker API. We could not see where 4.94 sets its new default, so run the <code>docker info</code> check above after the upgrade.</p>
<p><strong>In Compose files</strong>, set the driver for each service if a service depends on it, so the laptop and the server do the same thing:</p>
<pre><code class="hljs language-yaml" tabindex="0"><span class="hljs-attr">services:</span>
  <span class="hljs-attr">api:</span>
    <span class="hljs-attr">image:</span> <span class="hljs-string">example/api:1.4</span>
    <span class="hljs-attr">logging:</span>
      <span class="hljs-attr">driver:</span> <span class="hljs-string">json-file</span>
      <span class="hljs-attr">options:</span>
        <span class="hljs-attr">max-size:</span> <span class="hljs-string">'10m'</span>
        <span class="hljs-attr">max-file:</span> <span class="hljs-string">'3'</span>
</code></pre><p><a href="https://github.com/The-DevOps-Daily/docker-log-driver-check" rel="noopener noreferrer">The-DevOps-Daily/docker-log-driver-check on GitHub</a></p>
<h3>Clear a log that already fills the disk</h3><p>A limit only applies to containers created after you set it. A container that already has a huge <code>json-file</code> log keeps it until you clear it. Find the file and truncate it in place:</p>
<pre><code class="hljs language-bash" tabindex="0">LOG=$(docker inspect -f <span class="hljs-string">'{{.LogPath}}'</span> my-app)
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">du</span> -h <span class="hljs-string">"<span class="hljs-variable">$LOG</span>"</span>
<span class="hljs-built_in">sudo</span> <span class="hljs-built_in">truncate</span> -s 0 <span class="hljs-string">"<span class="hljs-variable">$LOG</span>"</span>
</code></pre><p>Truncate, do not delete. We tested both on Docker Engine 28.0.4 with a container that writes 100 lines a second:</p>
<ul>
<li><strong>After <code>truncate</code></strong>, Docker kept writing at the start of the empty file, and plain <code>docker logs</code> and <code>docker logs --since 2s</code> showed the new lines. <code>docker logs --tail 2</code> printed nothing until we restarted the container.</li>
<li><strong>After <code>rm</code></strong>, <code>docker logs</code> failed with <code>Error grabbing logs: open ...-json.log: no such file or directory</code>, and the daemon still held the deleted file open, so its disk space stays in use while the container runs.</li>
</ul>
<p>A <code>truncate</code> throws away every line in the file, so ship or copy what you need first. The lasting fix is a recreate with a limit: with <code>--log-opt max-size=1m --log-opt max-file=2</code>, our test container kept one full 1 MB file (<code>-json.log.1</code>) and the current one, and <code>docker logs --tail</code> kept working. The <code>local</code> driver's files are not plain text, so do not truncate them; recreate the container instead. On Docker Desktop, the log files are inside its Linux VM, and recreating the container is the simple way to start from an empty log.</p>
<h2>Summary</h2><p>Docker Desktop 4.94 moved new Linux containers to the <code>local</code> log driver. That is the driver Docker's own docs recommend, and in our test it kept all 500,000 lines in 24 MB of files where unbounded <code>json-file</code> used 84 MB. Docker Engine on servers still defaults to <code>json-file</code> with no rotation, so set a default on purpose. If something reads the files under <code>/var/lib/docker/containers</code>, it needs <code>json-file</code>. If it reads through the Docker API, <code>local</code> is the better default for both laptops and servers. In both cases, check <code>docker inspect</code>, because a new default does nothing for containers that already exist.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>