<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Shipwright</title>
    <description>The latest articles on DEV Community by Shipwright (@shipwright).</description>
    <link>https://dev.to/shipwright</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4036901%2Fdb7c218e-63d8-41c4-bcab-46bc7c9d570c.png</url>
      <title>DEV Community: Shipwright</title>
      <link>https://dev.to/shipwright</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/shipwright"/>
    <language>en</language>
    <item>
      <title>LLM-as-judge evals: writing the rubric, and the failure modes that lie to you</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Sun, 15 Nov 2026 15:03:33 +0000</pubDate>
      <link>https://dev.to/shipwright/llm-as-judge-evals-writing-the-rubric-and-the-failure-modes-that-lie-to-you-363c</link>
      <guid>https://dev.to/shipwright/llm-as-judge-evals-writing-the-rubric-and-the-failure-modes-that-lie-to-you-363c</guid>
      <description>&lt;p&gt;Deterministic graders are the backbone of an eval suite: does the output contain the required disclaimer, does it match the expected shape, does it avoid a banned phrase. Instant, free, and exact. But they cannot answer the question that actually matters for a language feature — &lt;em&gt;is this answer any good?&lt;/em&gt; "Politely declines without inventing a policy" has no regex. That is the job an LLM judge is for, and it is also where a judge will quietly tell you comforting lies if you build it carelessly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reach for a judge only past where strings stop
&lt;/h2&gt;

&lt;p&gt;Most criteria do not need a model. Keep the cheap, deterministic graders first — they fail fast and cost nothing — and bring in a judge only for the fuzzy ones a string cannot capture: tone, faithfulness to a source, "answers the question actually asked," "refuses without being preachy." Each judge call is a real model request, so every criterion you can express as a string is one you should not pay a model to check.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write the rubric as a falsifiable statement
&lt;/h2&gt;

&lt;p&gt;The judge is only as good as the rubric, and the rubric must be something that can be &lt;em&gt;false&lt;/em&gt;, not a vibe. "Is this good?" gets you a coin flip dressed as a verdict. "Declines the refund and offers to escalate, without inventing an exception that is not in the policy" can actually be checked. The grader hands that rubric and the output to the model and gets a typed verdict back — structured output, not prose you parse:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;llmJudge&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rubric&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ModelId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"claude-opus-5-5"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Grader&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateObject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;verdictSchema&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// { pass: boolean, reasoning: string }&lt;/span&gt;
      &lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="s2"&gt;"You are grading the output of an AI product feature against a rubric. "&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
        &lt;span class="s2"&gt;"Judge strictly: pass only if the output clearly satisfies the rubric."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`&amp;lt;rubric&amp;gt;\n${rubric}\n&amp;lt;/rubric&amp;gt;\n\n&amp;lt;output&amp;gt;\n${output}\n&amp;lt;/output&amp;gt;`&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reasoning&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details carry their weight. The verdict is a typed object (&lt;code&gt;pass&lt;/code&gt; plus a one-line reason) via the &lt;a href="https://www.shipwrightkit.com/blog/claude-structured-output-zod" rel="noopener noreferrer"&gt;structured-output helper&lt;/a&gt;, so a grader stays a plain function returning pass and a reason. And the system prompt forces a strict bar — which exists to fight the first failure mode.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure modes that lie to you
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Leniency and sycophancy.&lt;/strong&gt; Left to its own temperament a judge wants to be agreeable, so it passes marginal answers and praises them. The "judge strictly, pass only if it clearly satisfies" instruction is not politeness — it is the counterweight. Calibrate it against a few outputs you &lt;em&gt;know&lt;/em&gt; are bad; a judge that passes those is miscalibrated, not kind.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Drift.&lt;/strong&gt; The same rubric and output can get different verdicts across runs. A sharp, falsifiable rubric cuts the variance; pinning the judge model keeps a silent model upgrade from quietly re-grading your whole history. The judge is itself behavior you ship, so treat a rubric change like a prompt change — version it and re-run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Position and verbosity bias.&lt;/strong&gt; In pairwise "which is better, A or B" grading, judges favor the first option and the longer one. The cleanest defense is to avoid pairwise entirely: grade one output against an absolute rubric, as above, so there is no position to be biased by.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Self-preference.&lt;/strong&gt; A model tends to rate text from its own family higher. If the stakes are high, judge with a different model than the one under test.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Do not let the judge see the answer key inside the prompt it is grading&lt;/strong&gt; — that is leakage, and it inflates every score.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A judge that always passes measures nothing.&lt;/strong&gt; The value is in the fails; if you never see one, distrust the rubric before you trust the model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reasoning helps, briefly.&lt;/strong&gt; A one-line justification makes verdicts debuggable; a paragraph just burns tokens.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is the deep end of the eval harness; wiring a suite into CI is covered in &lt;a href="https://www.shipwrightkit.com/blog/llm-prompt-regression-testing-ci" rel="noopener noreferrer"&gt;catching prompt regressions in CI&lt;/a&gt;. Shipwright — the Next.js and Claude starter kit this blog documents — ships both graders, deterministic and judge, composing in one suite whose runner returns a single pass/fail you can gate a build on. Try the prompts they grade in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>evals</category>
      <category>llmjudge</category>
      <category>claude</category>
      <category>testing</category>
    </item>
    <item>
      <title>Rate limiting LLM API routes in Next.js: the limiter, the key, and the Redis seam</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Sat, 14 Nov 2026 15:03:21 +0000</pubDate>
      <link>https://dev.to/shipwright/rate-limiting-llm-api-routes-in-nextjs-the-limiter-the-key-and-the-redis-seam-3jkb</link>
      <guid>https://dev.to/shipwright/rate-limiting-llm-api-routes-in-nextjs-the-limiter-the-key-and-the-redis-seam-3jkb</guid>
      <description>&lt;p&gt;A monthly quota caps what a user spends over a billing period. It does nothing about the next ten seconds. Without a rate limit, one client can fire a hundred concurrent requests before the quota check has caught up — running your bill up in a burst, tripping the provider's own rate limits, or just starving the event loop. A rate limit is the second ceiling, and it belongs in front of every route that calls the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  A fixed-window limiter, keyed by identity
&lt;/h2&gt;

&lt;p&gt;The mechanism is small: a counter per key that resets on a fixed interval. In-memory, that is a map of buckets:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;rateLimit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="o"&gt;?:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;?:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;windowMs&lt;/span&gt;&lt;span class="o"&gt;?:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;windowMs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;windowMs&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;60_000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;buckets&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;resetAt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resetAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;windowMs&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;buckets&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;remaining&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resetAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;resetAt&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What matters more than the algorithm is the &lt;strong&gt;key&lt;/strong&gt;. Limit per identity, not globally: a single global counter lets one abuser exhaust the window for everyone. Use the authenticated user id when the caller is signed in, and fall back to the IP for anonymous traffic — the same identity split the quota and demo attribution use, so signed-in users get their own budget and the public demo is still bounded per address.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wire it before the model call
&lt;/h2&gt;

&lt;p&gt;The limiter runs first in the route, ahead of body validation and the quota check — reject the cheap way before doing any work, and return 429 with a &lt;code&gt;Retry-After&lt;/code&gt; so a well-behaved client backs off instead of hammering:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;rateLimit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`chat:${userId}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Rate limit exceeded. Try again in a minute."&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;"Retry-After"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ceil&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;resetAt&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Redis seam
&lt;/h2&gt;

&lt;p&gt;In-memory is exactly right for a single Node process — local dev, or one long-lived container. It quietly breaks on serverless or any multi-instance deploy, because each instance keeps its own map: with four instances behind a load balancer, your "20 per minute" is really up to 80. The fix is not to rewrite callers; it is to keep this same function signature and swap the internals for a shared store like Upstash Redis. Designing the seam in from the start — one function every route calls — means that change is one file, not a migration.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a rate limit actually bounds
&lt;/h2&gt;

&lt;p&gt;Concretely: worst-case spend per identity per window is &lt;code&gt;limit × max-cost-per-request&lt;/code&gt;. That is the number a rate limit gives you that a quota cannot — a ceiling on the &lt;em&gt;burst&lt;/em&gt;, where the quota ceilings the &lt;em&gt;month&lt;/em&gt;. Together they bound both the second and the billing period; ship only one and the other door is open. This is the pairing behind &lt;a href="https://www.shipwrightkit.com/blog/prevent-free-tier-abuse-ai-saas" rel="noopener noreferrer"&gt;keeping a free tier from bankrupting you&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fixed windows allow an edge burst.&lt;/strong&gt; A client can send a full window at the end of one interval and another at the start of the next — briefly 2x. Fine for spend control; use a sliding window if you need a hard guarantee.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;In-memory counters reset on cold start.&lt;/strong&gt; A freshly spun-up serverless instance has empty buckets, so limits are softer than they look right after a deploy — another reason the shared store matters in production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IP keys collide.&lt;/strong&gt; Corporate NATs and mobile carriers put many users behind one address; prefer the user id whenever the request is authenticated so you do not throttle a whole office as one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shipwright — the Next.js and Claude starter kit this blog documents — ships this limiter behind a one-line call in the chat route, keyed by identity and ready to swap for Redis, alongside the per-request metering and quota it works with. See it holding the line in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>ratelimiting</category>
      <category>cost</category>
      <category>claude</category>
    </item>
    <item>
      <title>How much the Claude API really costs per user: a worked example</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Fri, 13 Nov 2026 16:04:36 +0000</pubDate>
      <link>https://dev.to/shipwright/how-much-the-claude-api-really-costs-per-user-a-worked-example-fl6</link>
      <guid>https://dev.to/shipwright/how-much-the-claude-api-really-costs-per-user-a-worked-example-fl6</guid>
      <description>&lt;p&gt;"How much does the Claude API cost per user" has no single answer, because it depends on which model you route to and how long your conversations run. But you can compute it exactly for your app, and the number decides your pricing tiers, your free-tier limits, and which model each plan gets. Here is the actual math, worked from Anthropic's current per-token list prices.&lt;/p&gt;

&lt;h2&gt;
  
  
  Price one real session first
&lt;/h2&gt;

&lt;p&gt;Pricing is per million tokens: Opus 5.5 is $4 in / $20 out, Sonnet 5 is $2 / $10, and Haiku 4.5 is $1 / $5 (list prices at the time of writing; they change, so check before you set yours). A chat session is not one request — it is a handful of turns, and because you resend the growing conversation each turn, the input tokens accumulate fast. Take a realistic support-style session that totals about 30,000 input tokens across its turns and 4,000 output tokens. The cost is just the registry rates applied:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;costUsd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputPerMTok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
    &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputPerMTok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
    &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheReadTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheReadPerMTok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;      &lt;span class="c1"&gt;// 0.05x input on Opus 5.5, 0.1x on Sonnet 5 and Haiku 4.5&lt;/span&gt;
    &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheCreationTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheWritePerMTok&lt;/span&gt;   &lt;span class="c1"&gt;// 1.25x input (5-minute cache)&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run those 30k in / 4k out through it and the same session costs &lt;strong&gt;$0.20 on Opus, $0.10 on Sonnet, and $0.05 on Haiku&lt;/strong&gt;. Same conversation, a 4x spread — entirely from the model choice. The 4,000 output tokens include any thinking. Opus 5.5 always thinks and Sonnet 5 thinks by default, so measure your own output counts per model rather than assume one number for all three.&lt;/p&gt;

&lt;h2&gt;
  
  
  Now scale it to a user, per month
&lt;/h2&gt;

&lt;p&gt;One session is not a user. Say an active user runs about 40 of those sessions a month. That is &lt;strong&gt;$8 per active user on Opus, $4 on Sonnet, $2 on Haiku&lt;/strong&gt; — before any caching. If your system prompt is large and stable, prompt caching bills those repeated prefix tokens at a tenth of the input rate or less (a twentieth on Opus 5.5), which pulls the input side down further; I worked that break-even out separately in &lt;a href="https://www.shipwrightkit.com/blog/claude-prompt-caching-cost-math" rel="noopener noreferrer"&gt;the prompt-caching cost math&lt;/a&gt;. The point stands: your cost per user is a number you can name, and it is the floor under whatever you charge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why per-user attribution changes the decision
&lt;/h2&gt;

&lt;p&gt;Aggregate spend — one big API bill at month end — hides the shape that actually matters. Usage is never uniform: a small fraction of users run 10x the sessions of the median, so a flat price silently makes your light users subsidize your heavy ones, and a single whale on Opus can erase the margin from dozens of paying accounts. You only see this if cost is attributed &lt;strong&gt;per user&lt;/strong&gt;, one row per request, which is exactly what &lt;a href="https://www.shipwrightkit.com/blog/track-llm-token-costs-per-user" rel="noopener noreferrer"&gt;per-user metering&lt;/a&gt; is for.&lt;/p&gt;

&lt;p&gt;Once you can see it, the product decisions make themselves:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Set the free tier to the cheap model, hard-capped.&lt;/strong&gt; At $2 per fully-active user on Haiku — and far less for a capped free user — the free tier is a marketing cost, not a leak. That is why the kit routes free to Haiku and caps it, the &lt;a href="https://www.shipwrightkit.com/blog/prevent-free-tier-abuse-ai-saas" rel="noopener noreferrer"&gt;free-tier safety pattern&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Price paid tiers above their model cost, not below.&lt;/strong&gt; If Pro users get Opus and cost $8, a $20 plan has real margin; a $9 plan is a slow bleed you will not notice without per-user numbers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Route by plan deliberately.&lt;/strong&gt; The gap between $2 and $8 per user is the whole argument for gating models by tier instead of giving everyone your best model.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;The cost per user is not a mystery you absorb — it is &lt;code&gt;costUsd&lt;/code&gt; applied to real usage, summed per user, per month. Shipwright, the Next.js and Claude starter kit this blog documents, computes it on every request and shows it live: open the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;demo&lt;/a&gt; and watch the exact cost of each message accrue as you type.&lt;/p&gt;

</description>
      <category>cost</category>
      <category>claude</category>
      <category>pricing</category>
      <category>metering</category>
    </item>
    <item>
      <title>Why your Next.js SSE stream arrives as one blob (and the settings that fix it)</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Thu, 12 Nov 2026 15:03:43 +0000</pubDate>
      <link>https://dev.to/shipwright/why-your-nextjs-sse-stream-arrives-as-one-blob-and-the-settings-that-fix-it-4ib5</link>
      <guid>https://dev.to/shipwright/why-your-nextjs-sse-stream-arrives-as-one-blob-and-the-settings-that-fix-it-4ib5</guid>
      <description>&lt;p&gt;Your route logs tokens flowing out one at a time. The browser shows nothing — then the entire answer appears at once, as if it were never a stream at all. The model is streaming fine; something between your route handler and the browser is holding the bytes back and releasing them in a single flush. It is almost always one of three things, and none of them show up on localhost.&lt;/p&gt;

&lt;h2&gt;
  
  
  The response headers that stop a proxy from buffering
&lt;/h2&gt;

&lt;p&gt;An SSE response has to tell every hop between your server and the browser to pass bytes through untouched. The kit's stream helper sets exactly that on the Response it hands back from the route:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text/event-stream"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Cache-Control"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"no-cache, no-transform"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nc"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"keep-alive"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The one that matters most is &lt;code&gt;no-transform&lt;/code&gt;. Proxies and CDNs are allowed to buffer, compress, and otherwise "helpfully" transform a response — which for a stream means collecting it and delivering it as one chunk. &lt;code&gt;no-transform&lt;/code&gt; withdraws that permission. If nginx sits in front of you, it needs the same message in its own dialect: the &lt;code&gt;X-Accel-Buffering: no&lt;/code&gt; header turns off its proxy buffer for that response specifically.&lt;/p&gt;

&lt;h2&gt;
  
  
  Raise the route's max duration
&lt;/h2&gt;

&lt;p&gt;The second blob-maker is the platform timeout. A serverless route has a default execution ceiling, and a long generation runs straight past it — the function is killed mid-stream, and the browser gets a truncated response or an abrupt close instead of a finished answer. Set the route's max duration to your longest realistic generation, and pin the Node.js runtime so streaming behaves:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"nodejs"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;maxDuration&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a per-route export in the App Router, and the ceiling you can actually set is capped by your hosting tier — know that number before you promise long streams.&lt;/p&gt;

&lt;h2&gt;
  
  
  The frame format is part of the contract
&lt;/h2&gt;

&lt;p&gt;Even with perfect headers, the wire format can defeat you. An SSE event is terminated by a blank line, so each frame is &lt;code&gt;data: &amp;lt;payload&amp;gt;\n\n&lt;/code&gt; — the double newline is not decoration, it is the delimiter. Emit &lt;code&gt;\n&lt;/code&gt; instead of &lt;code&gt;\n\n&lt;/code&gt; and the client sits there buffering, waiting for an event boundary that never comes, until the connection closes and it flushes everything at once. That looks exactly like proxy buffering but originates in your own encoder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;send&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="n"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;encoder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`data: ${JSON.stringify(event)}\n\n`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It works in dev, breaks in prod.&lt;/strong&gt; Localhost has no CDN or reverse proxy in the path, so buffering only appears once something sits in front of your app. "Streams fine on my machine" tells you nothing about production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compression middleware re-buffers.&lt;/strong&gt; A gzip layer added globally will collect the stream to compress it. Exempt the event-stream route, or rely on &lt;code&gt;no-transform&lt;/code&gt; being honored end to end.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The client can buffer too.&lt;/strong&gt; Read the response body as a stream and process each frame as it arrives; a reader that waits for the whole body recreates the blob on the browser side.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;maxDuration is a ceiling, not a reservation.&lt;/strong&gt; Raising it lets long streams finish; it does not make them faster, and it is still bounded by your plan.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is the companion to the other half of production streaming — stopping the generation when the user leaves — which I wrote up in &lt;a href="https://www.shipwrightkit.com/blog/nextjs-ai-streaming-abort-on-disconnect" rel="noopener noreferrer"&gt;aborting the stream on disconnect&lt;/a&gt;. Shipwright, the Next.js and Claude starter kit this blog documents, ships both halves wired into one &lt;code&gt;chatStream&lt;/code&gt; helper you hand straight back from a route. Watch it stream token by token in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>streaming</category>
      <category>sse</category>
      <category>claude</category>
    </item>
    <item>
      <title>Typed output from Claude with Zod: stop parsing prose and delete the retry loop</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:03:56 +0000</pubDate>
      <link>https://dev.to/shipwright/typed-output-from-claude-with-zod-stop-parsing-prose-and-delete-the-retry-loop-4267</link>
      <guid>https://dev.to/shipwright/typed-output-from-claude-with-zod-stop-parsing-prose-and-delete-the-retry-loop-4267</guid>
      <description>&lt;p&gt;You ask the model for JSON, add "respond with only valid JSON, no markdown" to the prompt, and it works — until the day it wraps the object in a code fence, or adds a chatty preamble, or invents a field your parser does not expect. So you write a regex to fish the JSON out of the prose, then a retry loop for when that fails, and now your "structured" output is held together with string surgery. There is a better way, and it removes the whole category of problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Constrain the shape server-side, do not beg for it
&lt;/h2&gt;

&lt;p&gt;The API can constrain a response to a schema at generation time, so the model cannot return anything that does not match. You hand it a Zod schema, and the SDK constrains the output and validates the result against that same schema before handing it back — typed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;generateObject&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;S&lt;/span&gt; &lt;span class="k"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ZodType&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;GenerateOptions&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;S&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;GenerateObjectResult&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;infer&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;S&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;anthropic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="no"&gt;DEFAULT_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;maxTokens&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;16_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;output_config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;zodOutputFormat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nf"&gt;assertCompleted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parsed_output&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Structured output parsing failed — no parsed_output."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parsed_output&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;usageFromMessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is no "respond with JSON" instruction anywhere, because the constraint is not a request — it is enforced. The return type is &lt;code&gt;z.infer&amp;lt;S&amp;gt;&lt;/code&gt;, so the caller gets a fully typed object, not an &lt;code&gt;any&lt;/code&gt; you have to cast and hope about.&lt;/p&gt;

&lt;h2&gt;
  
  
  One schema, two guarantees
&lt;/h2&gt;

&lt;p&gt;The single Zod schema does two jobs. Server-side, it constrains what the model is allowed to emit, so the shape is guaranteed before a byte crosses the wire. Client-side, the SDK parses the result against the same schema, so field types, enums, and ranges are validated at runtime — the guarantees a plain TypeScript type cannot give you because types vanish at compile time. Define the shape once; get both the generation constraint and the runtime check from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure modes this removes — and the two it does not
&lt;/h2&gt;

&lt;p&gt;What goes away: the extract-the-JSON-from-the-prose regex, the "it added a markdown fence again" bug, and the retry-on-parse-error loop. Those are gone because malformed output is no longer possible, not merely less likely.&lt;/p&gt;

&lt;p&gt;What does not go away, and still needs handling:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Refusals.&lt;/strong&gt; A constrained request can still be declined; the stop reason is &lt;code&gt;refusal&lt;/code&gt;, and there is no object to parse. Check for it explicitly rather than treating a null result as a parse bug.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Truncation.&lt;/strong&gt; If the response hits &lt;code&gt;max_tokens&lt;/code&gt; mid-object, the output is incomplete and invalid. A big or deeply nested schema makes this more likely, so size &lt;code&gt;max_tokens&lt;/code&gt; to the schema. On Opus 5.5, the kit's default, the same limit also has to hold the model's thinking, which it cannot switch off, so leave headroom beyond the JSON itself — this is the same discipline the kit uses everywhere it calls the model, and it pairs naturally with the &lt;a href="https://www.shipwrightkit.com/blog/track-llm-token-costs-per-user" rel="noopener noreferrer"&gt;per-request cost metering&lt;/a&gt; that tells you how large your responses actually run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Null output.&lt;/strong&gt; Guard &lt;code&gt;parsed_output == null&lt;/code&gt; and fail loudly; a silent null is worse than an exception.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A short helper that asserts the stop reason and guards the null turns "usually returns JSON" into "returns a typed object or throws" — which is the property you actually want to build on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it pays off
&lt;/h2&gt;

&lt;p&gt;Anywhere you turn language into a record: extracting fields from an email, classifying a ticket into an enum, pulling structured arguments for a tool call. Shipwright — the Next.js and Claude starter kit this blog documents — uses this one &lt;code&gt;generateObject&lt;/code&gt; helper for all of them, including the LLM-judge grader in its &lt;a href="https://www.shipwrightkit.com/blog/llm-prompt-regression-testing-ci" rel="noopener noreferrer"&gt;eval harness&lt;/a&gt;, so every structured call in the codebase has the same typed, validated contract. You can see typed model output in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>claude</category>
      <category>zod</category>
      <category>typescript</category>
      <category>structuredoutput</category>
    </item>
    <item>
      <title>Metered billing for an AI SaaS: the 402 pattern and the Stripe gotcha that breaks renewals</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Tue, 10 Nov 2026 15:04:23 +0000</pubDate>
      <link>https://dev.to/shipwright/metered-billing-for-an-ai-saas-the-402-pattern-and-the-stripe-gotcha-that-breaks-renewals-2nic</link>
      <guid>https://dev.to/shipwright/metered-billing-for-an-ai-saas-the-402-pattern-and-the-stripe-gotcha-that-breaks-renewals-2nic</guid>
      <description>&lt;p&gt;Metering tells you what a user cost you. Billing is turning that number into "upgrade or stop" — and the mechanism for that is smaller than most Stripe tutorials suggest. It comes down to one HTTP status code, a table you keep in sync with webhooks, and one Stripe API detail that will silently break every renewal date if you get it wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Enforce with 402, and put the upgrade in the body
&lt;/h2&gt;

&lt;p&gt;When a user is over their monthly allowance, the route returns HTTP 402 — "Payment Required," the conventional signal for "you need to pay to continue." The status alone is not enough, though: the response body carries everything the client needs to render a real upgrade prompt instead of a dead error.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;quota&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;checkQuota&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;quota&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`You've used all ${quota.limit} requests on the ${quota.plan.label} plan this month.`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;quota&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;used&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;quota&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;used&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;quota&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;upgradeUrl&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"/pricing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;402&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Order matters in the route: rate limit first (429), then validate the body (400), then the quota check (402), then model gating. Cheapest rejections first, so you never pay to parse a request you were going to block anyway. The quota internals — the monthly cap and model gating behind checkQuota — are covered in &lt;a href="https://www.shipwrightkit.com/blog/prevent-free-tier-abuse-ai-saas" rel="noopener noreferrer"&gt;keeping a free tier from bankrupting you&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mirror subscription state into your own table
&lt;/h2&gt;

&lt;p&gt;Stripe is the source of truth for whether a card was charged, but you do not want a Stripe API call on the path of every chat request. So mirror the state you need into your own table and keep it fresh with webhooks. Every subscription lifecycle event upserts one row, keyed on the Stripe subscription id:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stripeCustomerId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;currentPeriodEnd&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;onConflictDoUpdate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;currentPeriodEnd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;updatedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the request path is a single indexed read: resolve the user's plan from that table, treating only "active" and "trialing" as paid — anything else (past_due, canceled) falls back to the free plan. No Stripe round-trip, no latency tax on every message.&lt;/p&gt;

&lt;h2&gt;
  
  
  The gotcha: current_period_end moved onto the item
&lt;/h2&gt;

&lt;p&gt;Here is the one that cost me an afternoon. In current Stripe API versions, &lt;code&gt;current_period_end&lt;/code&gt; is no longer on the subscription object — it lives on the subscription ITEM. Read it from the old location and it comes back undefined, so every renewal date you store is null, and any "your access resets on X" or grace-period logic silently does nothing. Nothing errors; the field is just quietly empty.&lt;/p&gt;

&lt;p&gt;The fix is not clever, it is disciplined: read the period end off the item, and verify the shape against the Stripe SDK version actually in your &lt;code&gt;node_modules&lt;/code&gt;, not against a tutorial (including this one — by the time you read it, the field may have moved again). API-shape assumptions from memory are how integrations rot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Verify the webhook signature.&lt;/strong&gt; Construct the event with Stripe's signing-secret check. Skip it and anyone who finds your endpoint can POST a fake "upgraded to Scale" event and grant themselves the top plan for free.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decide fail-open vs fail-closed.&lt;/strong&gt; Without a database the kit fails open so the demo runs; a billing system in production should fail closed when it cannot verify entitlement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A 402 with an empty body is a dead end.&lt;/strong&gt; Return plan, used, limit, and an upgrade URL so the client renders a prompt, not a stack trace.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Honor trialing, not just active.&lt;/strong&gt; A trialing user is a paying-intent user; drop them to the free plan mid-trial and you have manufactured churn.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shipwright — the Next.js and Claude starter kit this blog documents — ships this whole path: the 402-gated chat route, a signature-verified Stripe webhook, and the subscriptions table it keeps in sync, all reading the same &lt;a href="https://www.shipwrightkit.com/blog/track-llm-token-costs-per-user" rel="noopener noreferrer"&gt;per-request metering&lt;/a&gt; the quota depends on. Watch a request get priced in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>stripe</category>
      <category>billing</category>
      <category>quotas</category>
      <category>nextjs</category>
    </item>
    <item>
      <title>Catch prompt regressions in CI before your users do</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Mon, 09 Nov 2026 15:03:02 +0000</pubDate>
      <link>https://dev.to/shipwright/catch-prompt-regressions-in-ci-before-your-users-do-a9g</link>
      <guid>https://dev.to/shipwright/catch-prompt-regressions-in-ci-before-your-users-do-a9g</guid>
      <description>&lt;p&gt;Change a prompt to fix one case and you can silently break five others. Nothing throws, no test goes red, the app still returns confident-looking text — and the regression ships to your biggest customer. If you edit prompts on vibes, you will ship one of these eventually. The fix is to treat prompts like the code they are: version them, and run an eval suite against the real production path before every change.&lt;/p&gt;

&lt;h2&gt;
  
  
  An eval suite points at your actual prompt
&lt;/h2&gt;

&lt;p&gt;The trap with prompt tests is grading a copy of the prompt that drifts from what production runs. So a suite here calls the real helper — the same function your route calls — and grades its output. The shape is deliberately small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;EvalSuite&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="c1"&gt;// The production code path under test — call your real prompt/helper here.&lt;/span&gt;
  &lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;cases&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;EvalCase&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because the suite runs the production code path, a prompt edit is exercised the moment you run the suite — no separate fixture to keep in sync.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two kinds of grader: cheap-and-exact, then fuzzy
&lt;/h2&gt;

&lt;p&gt;Most of what you want to assert is format and policy: does the output contain the disclaimer, does it avoid a banned phrase, does it match a shape. Those are deterministic string checks — instant and free. Reach for the model only for things a string cannot capture ("politely declines without inventing a policy"), and write the rubric as a checkable statement, not a vibe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;llmJudge&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rubric&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ModelId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"claude-opus-5-5"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Grader&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateObject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;verdictSchema&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// { pass: boolean, reasoning: string }&lt;/span&gt;
      &lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="s2"&gt;"You are grading the output of an AI product feature against a rubric. "&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
        &lt;span class="s2"&gt;"Judge strictly: pass only if the output clearly satisfies the rubric."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rubric&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"\n\n"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;output&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reasoning&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The judge returns a typed verdict via structured output, so a grader is just a function returning pass plus a reason — deterministic and LLM graders compose in the same list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make CI fail, not just print
&lt;/h2&gt;

&lt;p&gt;An eval that logs a warning gets ignored. The runner returns a single boolean — true only if every case passed — so have your suite script exit nonzero on false and a red eval blocks the merge exactly like a failing unit test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;runSuite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;suite&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pair that with a versioned prompt registry: each prompt carries a version — log it with every eval run, so when a suite goes red you can tell whether the cause was a prompt edit or a model upgrade — the same deterministic registry that makes &lt;a href="https://www.shipwrightkit.com/blog/claude-prompt-caching-cost-math" rel="noopener noreferrer"&gt;prompt caching actually work&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Evals cost real tokens.&lt;/strong&gt; They make live API calls, and the LLM judge is itself a model call. Keep deterministic graders first and gate them so the cheap checks fail fast before you pay for a judge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A vague rubric grades vaguely.&lt;/strong&gt; "Is it good" gives you a coin flip. "Declines the refund and offers to escalate, without inventing an exception" is checkable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judge strictly.&lt;/strong&gt; A lenient judge passes everything and tells you nothing; the system prompt here forces a strict pass bar on purpose.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Version the prompt, or you are debugging blind.&lt;/strong&gt; Without a version stamped on the run, a regression could be the prompt, the model, or the weather.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shipwright — the Next.js and Claude starter kit this blog documents — ships this eval harness — composable graders, an LLM judge, and a runner that returns a pass/fail you can gate CI on — alongside the versioned prompt registry and per-request cost metering, so quality regressions can surface in a pull request instead of a support ticket. You can try the prompts it grades in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>evals</category>
      <category>prompts</category>
      <category>ci</category>
      <category>claude</category>
    </item>
    <item>
      <title>How to keep an AI free tier from bankrupting you: two ceilings and a cheap model</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Sun, 08 Nov 2026 15:03:15 +0000</pubDate>
      <link>https://dev.to/shipwright/how-to-keep-an-ai-free-tier-from-bankrupting-you-two-ceilings-and-a-cheap-model-eb9</link>
      <guid>https://dev.to/shipwright/how-to-keep-an-ai-free-tier-from-bankrupting-you-two-ceilings-and-a-cheap-model-eb9</guid>
      <description>&lt;p&gt;The fastest way to lose money on an AI product is a generous free tier pointed at an expensive model. Free users have no reason to be frugal and every reason to test your limits, so if your free plan can reach your best model with no ceiling, your worst-case cost per free user is unbounded. A free tier is supposed to be top-of-funnel, not a faucet into your API account. Here is how I make one safe with two independent ceilings and one cheap model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ceiling one: a monthly request quota
&lt;/h2&gt;

&lt;p&gt;Every AI route checks the user's usage against their plan limit before doing any model work, and returns HTTP 402 when they are over. That check is the whole enforcement point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;checkQuota&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;QuotaCheck&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getUserPlan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getMonthlyUsage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monthlyRequestLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;used&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monthlyRequestLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The free plan's limit is 25 requests per calendar month. The route calls this first; when ok is false it answers 402 instead of calling the model. The count comes from the same per-request usage table that drives billing, so the number the quota reads is the number you actually spent — I wrote up that schema in &lt;a href="https://www.shipwrightkit.com/blog/track-llm-token-costs-per-user" rel="noopener noreferrer"&gt;tracking LLM costs per user&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ceiling two: a per-minute rate limit
&lt;/h2&gt;

&lt;p&gt;A monthly quota bounds the total, but not the burst. Without a second ceiling, a user could spend the entire month's allowance in ten seconds and hammer your provider with concurrent requests. A separate per-minute rate limiter caps that burst. They defend against different attacks — one bounds the month, the other bounds the second — so you need both, not either.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate the model, not just the volume
&lt;/h2&gt;

&lt;p&gt;Volume caps alone are not enough, because 25 requests on your most expensive model is a completely different bill than 25 on your cheapest. So the plan also declares which models it may use, and requests for anything else are clamped to the plan's default:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="n"&gt;free&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;monthlyRequestLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;allowedModels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"claude-haiku-4-5"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;defaultModel&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"claude-haiku-4-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A free user who crafts a request asking for Opus does not get Opus — the plan clamps it to Haiku. Model access is a billing decision, enforced in one place, not an if-statement scattered across routes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The math that makes it safe
&lt;/h2&gt;

&lt;p&gt;Haiku 4.5 bills at 1 dollar per million input tokens and 5 per million output. A chatty free request of roughly 1,000 input and 500 output tokens costs about $0.0035. Twenty-five of them is about nine cents per free user per month. Break-even sits well under 1% conversion — a free tier at that cost is a marketing channel, not a liability.&lt;/p&gt;

&lt;p&gt;Now run the same 25 requests on Opus 5.5 at 4 dollars per million input and 20 per million output. The same tokens cost 4x as much, about 35 cents per free user. Opus 5.5 also thinks on every request and bills that reasoning as output, so real requests cost more than that, and without the quota there is no ceiling at all. That gap is the entire argument for gating both volume and model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fail-open versus fail-closed is a real decision.&lt;/strong&gt; This kit fails open when no database is configured, so the public demo stays usable without setup. In production, a metering outage that silently disables the quota is a spend leak — decide on purpose which way it fails.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anonymous users still need an identity.&lt;/strong&gt; The public demo attributes usage by IP address, so unauthenticated traffic is capped too. Skip that and "free" quietly means "unlimited to anyone not logged in."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A rate limit is not a quota.&lt;/strong&gt; One bounds the burst, the other the month. Shipping only one leaves the other door open.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cap on requests, keep the tokens.&lt;/strong&gt; Users understand "25 requests"; bill and reason about margin on the token and cost columns underneath.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shipwright — the Next.js and Claude starter kit this blog documents — ships this whole safety layer wired together: per-plan quotas, model gating, the per-minute limiter, and the metering that feeds them. You can watch the free tier's cost accrue in real time in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>billing</category>
      <category>quotas</category>
      <category>cost</category>
      <category>claude</category>
    </item>
    <item>
      <title>Tracking LLM costs per user: the schema that billing can actually trust</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Sat, 07 Nov 2026 15:03:21 +0000</pubDate>
      <link>https://dev.to/shipwright/tracking-llm-costs-per-user-the-schema-that-billing-can-actually-trust-3no7</link>
      <guid>https://dev.to/shipwright/tracking-llm-costs-per-user-the-schema-that-billing-can-actually-trust-3no7</guid>
      <description>&lt;p&gt;Your model provider tells you what the whole account spent. It does not tell you that one free-tier user generated a third of it. Until cost is attributed per user, every pricing decision — plan limits, free-tier caps, whether that enterprise deal is even profitable — is a guess. Here is the schema I use, and the two decisions in it that took real debugging to learn.&lt;/p&gt;

&lt;h2&gt;
  
  
  One row per model request
&lt;/h2&gt;

&lt;p&gt;Every request writes exactly one row, at the moment usage is known — from the stream's finish hook, or right after a non-streaming call returns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;recordUsage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;RecordUsageArgs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;usageEvents&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;route&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;route&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// which feature spent the tokens, e.g. "chat"&lt;/span&gt;
      &lt;span class="n"&gt;inputTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;outputTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;cacheReadTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheReadTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;cacheCreationTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheCreationTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;costMicroUsd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;costUsd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The column choices matter more than they look:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;route&lt;/strong&gt; — a month in, you will want to know which &lt;em&gt;feature&lt;/em&gt; burns the money, not just which user. Adding the column later means backfilling history you no longer have.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The two cache columns&lt;/strong&gt; — cache reads bill at a tenth of the input rate or less (a twentieth on Opus 5.5) and writes at 1.25x, so a request with heavy cache reads costs a fraction of its naive token count. Without these columns your numbers drift from the real invoice, and you cannot see whether &lt;a href="https://www.shipwrightkit.com/blog/claude-prompt-caching-cost-math" rel="noopener noreferrer"&gt;prompt caching is actually saving you anything&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;userId as plain text, not a foreign key&lt;/strong&gt; — deliberate. The public demo attributes anonymous traffic as &lt;code&gt;ip:&amp;lt;addr&amp;gt;&lt;/code&gt;; a hard FK to the users table would reject those rows. Once every AI route requires auth, tighten it to a real reference.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Integer micro-USD. Never floats.
&lt;/h2&gt;

&lt;p&gt;The cost column is a bigint holding millionths of a dollar: a request that costs $0.004173 is stored as 4173. One request priced as a float looks fine. A month of requests summed as floats accumulates binary rounding error, and now your dashboard, your quota check, and your Stripe invoice each show a slightly different number — three sources of truth is zero sources of truth.&lt;/p&gt;

&lt;p&gt;Integers sum exactly. Convert to dollars once, at render time, and let every consumer read the same column.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rollup that quotas, dashboards, and billing all share
&lt;/h2&gt;

&lt;p&gt;Per-user monthly usage is one aggregate over that table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`count(*)`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;inputTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`coalesce(sum(input_tokens), 0)`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;outputTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`coalesce(sum(output_tokens), 0)`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;costMicroUsd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`coalesce(sum(cost_micro_usd), 0)`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;usageEvents&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ow"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;usageEvents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;gte&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;usageEvents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;createdAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;monthStart&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This single number is load-bearing three ways: the quota middleware compares it against the plan limit and returns 402 when exceeded, the dashboard renders it to the user, and usage-based billing reports it to Stripe. Because all three read the same rows, they cannot disagree — which is the entire point of the design.&lt;/p&gt;

&lt;p&gt;An index on (user_id, created_at) keeps the rollup fast; it is the only query pattern the table needs to serve at request time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Decide your no-database behavior on purpose.&lt;/strong&gt; This kit fails open — recordUsage warns and returns when no DATABASE_URL is set, so the demo runs with zero config. Fine for a starter, wrong for production billing: there, failing to record should be loud.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Month boundaries are UTC.&lt;/strong&gt; The rollup starts at the first of the month, UTC. Pick a convention and keep it consistent with your billing period, or users get quota resets that disagree with their invoice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Meter requests, not just tokens.&lt;/strong&gt; Plans here cap monthly request volume, because "requests" is a number users can reason about. The token data stays for margin analysis either way.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IP attribution is a stopgap.&lt;/strong&gt; Shared offices and mobile carriers pool many humans behind one address. Good enough to cap an anonymous demo, not good enough to bill.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can watch this pipeline run in the &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt;: every message prints its model, token counts, cache activity, and exact cost — the same numbers that land in the table. Shipwright, the Next.js + Claude starter kit this blog documents, ships the whole loop — schema, metering, quotas, and the Stripe wiring — already connected.&lt;/p&gt;

</description>
      <category>metering</category>
      <category>cost</category>
      <category>postgres</category>
      <category>billing</category>
    </item>
    <item>
      <title>Claude prompt caching: the break-even math and the mistake that turns it off</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Fri, 06 Nov 2026 15:04:11 +0000</pubDate>
      <link>https://dev.to/shipwright/claude-prompt-caching-the-break-even-math-and-the-mistake-that-turns-it-off-315</link>
      <guid>https://dev.to/shipwright/claude-prompt-caching-the-break-even-math-and-the-mistake-that-turns-it-off-315</guid>
      <description>&lt;p&gt;Prompt caching is the biggest cost lever most Claude apps leave on the table — and the only one that can also make your bill &lt;em&gt;worse&lt;/em&gt;. The pricing has a shape people miss: cache reads are cheap, but cache writes carry a surcharge. If you cache things that never get re-read, you are paying extra for the privilege.&lt;/p&gt;

&lt;p&gt;Here is the actual math, using the numbers my metering layer bills against.&lt;/p&gt;

&lt;h2&gt;
  
  
  The four line items on every request
&lt;/h2&gt;

&lt;p&gt;Every Claude request can produce up to four token counts, each at a different rate. This is the pricing function from my model registry — the same one the dashboard, quotas, and billing all read:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="cm"&gt;/**
 * Cost of a single request in USD, using each model's own cache prices
 * (5-minute cache writes; reads at the model's read rate).
 */&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;costUsd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ModelId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;TokenUsage&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputPerMTok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
      &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputPerMTok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
      &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheReadTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheReadPerMTok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
      &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheCreationTokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cacheWritePerMTok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;
    &lt;span class="mi"&gt;1_000_000&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things to internalize:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reads are a 90–95% discount.&lt;/strong&gt; A cached token bills at a tenth of the normal input rate on most models, and a twentieth on Opus 5.5 ($0.20 per million against $4).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writes are a 25% surcharge.&lt;/strong&gt; The first time a prefix is cached, those tokens cost 1.25x normal input. Caching is an investment, and investments can lose.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The registry stores the read and write price per model, not as one multiplier. The kit's first version hard-coded 0.1x for reads, which was right for every model it listed until Opus 5.5. Left in place through the upgrade, it would have metered every cached Opus 5.5 token at twice its real price.&lt;/p&gt;

&lt;h2&gt;
  
  
  Break-even: one re-read pays for the write
&lt;/h2&gt;

&lt;p&gt;Take a 2,000-token system prompt on Claude Opus 5.5 (4 dollars per million input tokens):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Uncached, that prefix costs &lt;strong&gt;0.8 cents&lt;/strong&gt; per request.&lt;/li&gt;
&lt;li&gt;Writing it to the cache costs &lt;strong&gt;1.0 cent&lt;/strong&gt; — once.&lt;/li&gt;
&lt;li&gt;Each subsequent read costs &lt;strong&gt;0.04 cents&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So for N requests inside the cache window: uncached spend is 0.8N cents, cached spend is 1.0 + 0.04(N-1). At N=1 you lost a fifth of a cent. At N=2 you are already 35% ahead. By N=10 the prefix costs 1.36 cents instead of 8.&lt;/p&gt;

&lt;p&gt;The general rule falls out of the multipliers: the write premium is 0.25x, each re-read saves 0.9x (0.95x on Opus 5.5) — so &lt;strong&gt;a single re-read within the TTL pays for the write more than three times over&lt;/strong&gt;. In a chat product, where every turn resends the same system prompt and history prefix, that condition is nearly always met. Caching a one-off batch job that never repeats a prefix, though, is a pure 25% surcharge.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mistake: one dynamic value at the top
&lt;/h2&gt;

&lt;p&gt;Cache matching is a &lt;strong&gt;byte-exact prefix match&lt;/strong&gt;. A single dynamic value early in your system prompt — a timestamp, a request ID, a "today is Wednesday" — changes the bytes, misses the cache, and triggers a fresh 1.25x write on every single request. That is the worst configuration available: you pay the write surcharge every time and never collect a read.&lt;/p&gt;

&lt;p&gt;This is why my prompt registry keeps system prompts deterministic, with a comment that has survived every refactor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="cm"&gt;/**
 * Keep system prompts deterministic (no timestamps, no per-request IDs):
 * prompt caching is a byte-exact prefix match, and a single dynamic value at
 * the top of the prompt disables caching for everything after it.
 */&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anything per-request — the user name, the date, retrieved context — belongs &lt;em&gt;after&lt;/em&gt; the stable prefix, or in the user message, never at the top of the system prompt.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you cannot see the hit rate, assume zero
&lt;/h2&gt;

&lt;p&gt;My usage table records cache reads and writes as their own columns on every request, because the failure mode here is silent: nothing breaks when caching stops working, the bill just quietly grows. If cache reads are near zero in production while writes keep happening, a dynamic byte crept into your prefix and nobody noticed.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;demo chat&lt;/a&gt;'s under-the-hood panel has the same split: cache reads and writes get their own lines, labelled with that model's rates, next to the exact cost. They only appear once a prefix is long enough to cache. The demo's own system prompt is about 300 characters, far below any model's minimum, so there you will see plain input tokens. The same discipline as &lt;a href="https://www.shipwrightkit.com/blog/nextjs-ai-streaming-abort-on-disconnect" rel="noopener noreferrer"&gt;aborting streams when the client disconnects&lt;/a&gt;: measure the money path first, then the savings become visible instead of theoretical.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The TTL is five minutes.&lt;/strong&gt; Caching pays inside active sessions and bursty traffic; a user who returns after lunch starts with a fresh write. Do not model savings off your total request count — model them off requests that arrive within minutes of each other.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writes without reads are a surcharge, not a win.&lt;/strong&gt; Audit any endpoint that caches but rarely repeats a prefix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Very short prompts may not cache at all.&lt;/strong&gt; Models enforce a minimum cacheable prefix length; below it, your cache_control marker is silently ignored. Anthropic's docs currently list 512 tokens for Opus 5.5, 1,024 for Sonnet 5 and 4,096 for Haiku 4.5. A cheaper model can have a higher minimum, so check it for each model you route to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Neither the ratios nor the absolutes hold across models.&lt;/strong&gt; On Haiku 4.5 the same 2,000-token prefix does not cache at all, because it is under Haiku's 4,096-token minimum. Uncached it costs 0.2 cents, a quarter of the Opus 5.5 figure. Spend optimization effort where the tokens are expensive, and check that the prefix clears the minimum on the model that actually serves the request.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shipwright — the Next.js starter kit this blog documents — ships this whole loop wired together: deterministic versioned prompts, cache-aware cost attribution per request in integer micro-USD, and a live panel that shows the math on every message.&lt;/p&gt;

</description>
      <category>claude</category>
      <category>promptcaching</category>
      <category>cost</category>
      <category>metering</category>
    </item>
    <item>
      <title>Your Next.js AI streaming route keeps billing you after the user leaves</title>
      <dc:creator>Shipwright</dc:creator>
      <pubDate>Thu, 05 Nov 2026 22:54:19 +0000</pubDate>
      <link>https://dev.to/shipwright/your-nextjs-ai-streaming-route-keeps-billing-you-after-the-user-leaves-4kim</link>
      <guid>https://dev.to/shipwright/your-nextjs-ai-streaming-route-keeps-billing-you-after-the-user-leaves-4kim</guid>
      <description>&lt;p&gt;An AI chat endpoint feels finished the moment tokens start streaming into the browser. Then your first real bill arrives, and the token count is higher than what your users actually read. The usual reason: responses that keep generating after the user has already closed the tab.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the money leaks
&lt;/h2&gt;

&lt;p&gt;In the Next.js App Router, a streaming route opens a connection to the model and pipes tokens back to the browser. If the user navigates away or closes the tab mid-response, the browser drops its connection — but the upstream request to the model keeps generating to completion. You pay for every token it produces, and nobody ever sees them. On long answers with an impatient audience, that waste adds up fast.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: stop generating when the browser disconnects
&lt;/h2&gt;

&lt;p&gt;When you build the SSE stream yourself with a ReadableStream, its &lt;code&gt;cancel()&lt;/code&gt; callback fires the moment the client connection drops. That's your hook to abort the upstream model request instead of letting it run to completion:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;POST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;upstream&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;anthropic&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;messages&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ReadableStream&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;await &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;upstream&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
      &lt;span class="nx"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="nf"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="c1"&gt;// The browser went away — stop generating (and paying).&lt;/span&gt;
      &lt;span class="nx"&gt;upstream&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abort&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;content-type&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;text/event-stream&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;cache-control&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;no-cache, no-transform&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That one method turns "streaming works in the demo" into "streaming doesn't leak money in production."&lt;/p&gt;

&lt;h2&gt;
  
  
  Three things that bite you next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"no-transform" matters.&lt;/strong&gt; Some proxies buffer SSE, so your "stream" arrives as a single blob at the very end. The Cache-Control no-transform hint tells them to leave it alone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Raise the route's max duration.&lt;/strong&gt; Long generations outlive the platform's default timeout and get cut off mid-answer; bump &lt;code&gt;maxDuration&lt;/code&gt; on the route.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Meter what you spend.&lt;/strong&gt; Cancelling saves money you can't see unless you measure it. Record input, output, and cache tokens per request, priced per model, so the savings show up in your numbers instead of your imagination. I wrote up the exact schema I use for that &lt;a href="https://www.shipwrightkit.com/blog/track-llm-token-costs-per-user" rel="noopener noreferrer"&gt;here&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why it belongs in the foundation
&lt;/h2&gt;

&lt;p&gt;None of this is hard. But it's the kind of thing you only discover you needed after it has already cost you — which is exactly why it should live in the foundation of an AI app, not the backlog.&lt;/p&gt;

&lt;p&gt;I package this pattern — along with per-request cost metering, typed tool use, versioned prompts with an eval harness, and plan-gated quotas — into a Next.js 16 + Claude starter kit called &lt;a href="https://www.shipwrightkit.com" rel="noopener noreferrer"&gt;Shipwright&lt;/a&gt;. The &lt;a href="https://www.shipwrightkit.com/chat" rel="noopener noreferrer"&gt;live demo&lt;/a&gt; shows the exact cost of every request as you chat.&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>ai</category>
      <category>typescript</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
