<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: delay.run</title>
    <description>The latest articles on DEV Community by delay.run (@delayrun).</description>
    <link>https://dev.to/delayrun</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4210417%2F3da89f80-6390-415e-bca5-65d64a57235d.png</url>
      <title>DEV Community: delay.run</title>
      <link>https://dev.to/delayrun</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/delayrun"/>
    <language>en</language>
    <item>
      <title>Before your payment webhooks go live: a 12-point production checklist</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:08:39 +0000</pubDate>
      <link>https://dev.to/delayrun/before-your-payment-webhooks-go-live-a-12-point-production-checklist-3fn9</link>
      <guid>https://dev.to/delayrun/before-your-payment-webhooks-go-live-a-12-point-production-checklist-3fn9</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Webhooks from a payment provider are the moment your system finds out that money moved. In the sandbox they're easy: you send a test event, your handler logs it, done. In production they arrive twice, out of order, hours late, from a provider whose retry schedule you didn't read, to an endpoint that went down during your last deploy.&lt;/p&gt;

&lt;p&gt;None of those are rare. Here's the checklist I'd go through before switching a payment integration to live mode. It's written with card payments and subscriptions in mind, but almost all of it applies to any provider that sends webhooks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Receiving
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Verify signatures on the raw body
&lt;/h3&gt;

&lt;p&gt;Every serious provider signs its webhooks. Verify the signature against the raw request bytes, before any JSON parsing, using the secret for that specific endpoint. Reject anything that fails with a 400, and log the failure (without logging the secret). Test it with a deliberately tampered body, so you know the check actually rejects things.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Return 2xx fast, process later
&lt;/h3&gt;

&lt;p&gt;The provider will time out your request after a few seconds (often 10 to 30) and retry. Do the minimum in the request: verify, store the raw event, return 200. Do the real work (updating orders, sending emails, provisioning accounts) in a background job. A handler that sends an email inline will eventually send it twice when a slow SMTP server makes the provider time out and retry.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Store every event before acting on it
&lt;/h3&gt;

&lt;p&gt;Keep a table of received events with the provider's event ID as a unique key, the type, the raw payload, the received time and the processed time. It's your audit log, your deduplication mechanism, and the thing you'll query at 2 a.m. when someone asks "did we ever get the webhook for this payment?".&lt;/p&gt;

&lt;h2&gt;
  
  
  Processing
&lt;/h2&gt;

&lt;h3&gt;
  
  
  4. Make every handler idempotent
&lt;/h3&gt;

&lt;p&gt;Duplicate deliveries happen: retries after timeouts, the provider's own at-least-once delivery, and manual resends from the dashboard. Deduplicate on the event ID, and also make the action itself safe to repeat ("mark paid if not already paid", not "add a credit").&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Don't depend on delivery order
&lt;/h3&gt;

&lt;p&gt;Providers generally don't guarantee order. A &lt;code&gt;subscription.updated&lt;/code&gt; can arrive before &lt;code&gt;subscription.created&lt;/code&gt;; a refund can arrive before you processed the charge. Either re-fetch the object's current state from the API when an event arrives, or compare versions and ignore anything older than what you have.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Handle "not found yet" without dropping the event
&lt;/h3&gt;

&lt;p&gt;If the event refers to an order or customer your database doesn't have yet (because your own write hasn't committed), don't return 200 and forget about it. Store it and retry the processing on your own short schedule, with an alert if it's still unmatched after an hour.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Handle the event types you don't use
&lt;/h3&gt;

&lt;p&gt;Your endpoint will receive event types you didn't subscribe to on purpose, or new ones the provider adds. Return 200 and ignore them, rather than throwing a 500 that makes the provider retry for days. Better still, subscribe the endpoint only to the event types you handle.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Checklist item: a retry policy that isn't a sleep() in the request.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; schedules the follow-up HTTP call with retries and a run history you can inspect.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Operating
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. Know the provider's retry schedule and disable rules
&lt;/h3&gt;

&lt;p&gt;Find out how long the provider retries a failing endpoint (for some, up to three days) and what happens after that. Some providers automatically disable an endpoint after repeated failures, and send a warning email to whoever's address is on the account. Make sure that address reaches someone who'll act on it.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. Alert on silence, not just on errors
&lt;/h3&gt;

&lt;p&gt;Errors are easy to alert on. The harder failure is no webhooks at all: an endpoint disabled, a DNS change, a firewall rule, a deploy that removed the route. Track "time since last event received" per event type you rely on, and alert when it exceeds what's normal for your traffic.&lt;/p&gt;

&lt;h3&gt;
  
  
  10. Reconcile with the provider's API
&lt;/h3&gt;

&lt;p&gt;Run a scheduled job that compares your records with the provider's: payments marked pending for more than N minutes, subscriptions whose status differs, refunds you don't know about. Webhooks are a notification mechanism; the provider's API is the source of truth. The reconciler catches everything the first nine items miss.&lt;/p&gt;

&lt;h3&gt;
  
  
  11. Plan secret rotation
&lt;/h3&gt;

&lt;p&gt;You'll need to rotate the signing secret eventually: someone leaves, a secret leaks into a log. Check whether the provider supports two active secrets during rotation, and make your handler accept a list of secrets so rotation doesn't need a coordinated deploy at the exact right minute.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. Test the failure paths, not just success
&lt;/h3&gt;

&lt;p&gt;Before going live, deliberately:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Return a 500 and watch the provider retry, then succeed.&lt;/li&gt;
&lt;li&gt;Send the same event twice and confirm one effect.&lt;/li&gt;
&lt;li&gt;Send events out of order and confirm the final state is right.&lt;/li&gt;
&lt;li&gt;Take the endpoint down for ten minutes and confirm nothing is lost when it comes back.&lt;/li&gt;
&lt;li&gt;Rotate the signing secret in the sandbox.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most providers' CLIs and dashboards let you resend events to make this easy.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;Verify on raw bytes, store first, return fast, process idempotently in the background, ignore order, alert on silence, and reconcile on a schedule.&lt;/p&gt;

&lt;p&gt;Several of these items (retrying "not found yet" events, debouncing re-fetches, scheduled reconciliation) are delayed or scheduled jobs. &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; handles that kind of work, but a durable queue plus cron will do the same job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/how-to-fix-no-signatures-found-matching-the-expected-signature-for-payload-stripe-webhooks-in-12i3"&gt;How to fix 'No signatures found matching the expected signature for payload' (Stripe webhooks in Express, Next.js and Flask)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/before-a-scheduled-job-emails-50000-people-a-12-point-checklist-for-bulk-sends-that-cant-be-5aoo"&gt;Before a scheduled job emails 50,000 people: a 12-point checklist for bulk sends that can't be unsent&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-job-not-running-a-step-by-step-checklist-including-it-works-when-i-run-it-manually-1nb6"&gt;Cron job not running? A step-by-step checklist (including “it works when I run it manually”)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/make-scheduled-job-handlers-idempotent-safe-to-run-twice-499o"&gt;Make scheduled job handlers idempotent: safe to run twice&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which item on this list did you learn the hard way? For me it was number 9: an endpoint that had been quietly disabled for days.&lt;/p&gt;

</description>
      <category>webhooks</category>
      <category>backend</category>
      <category>api</category>
      <category>security</category>
    </item>
    <item>
      <title>How to fix 'No signatures found matching the expected signature for payload' (Stripe webhooks in Express, Next.js and Flask)</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:08:12 +0000</pubDate>
      <link>https://dev.to/delayrun/how-to-fix-no-signatures-found-matching-the-expected-signature-for-payload-stripe-webhooks-in-12i3</link>
      <guid>https://dev.to/delayrun/how-to-fix-no-signatures-found-matching-the-expected-signature-for-payload-stripe-webhooks-in-12i3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You've set up a Stripe webhook endpoint, sent a test event, and your logs say:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;No signatures found matching the expected signature for payload.
Are you passing the raw request body you received from Stripe?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second sentence is the hint, and it's right most of the time. Stripe signs the exact bytes of the request body. If your framework parses the JSON and you verify the re-serialised version, or you use a secret from a different endpoint, the signature can't match.&lt;/p&gt;

&lt;p&gt;Here are the causes in order of how often they happen, and how to check for each.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the check works, in one paragraph
&lt;/h2&gt;

&lt;p&gt;Stripe sends a &lt;code&gt;Stripe-Signature&lt;/code&gt; header like &lt;code&gt;t=1731571200,v1=5257a8...&lt;/code&gt;. To verify, the library computes an HMAC-SHA256 of &lt;code&gt;"{t}.{raw body}"&lt;/code&gt; using your endpoint's signing secret (&lt;code&gt;whsec_...&lt;/code&gt;) and compares it with &lt;code&gt;v1&lt;/code&gt;. If even one byte of the body differs (a space, key order, a Unicode escape, a newline), the HMAC is completely different. There's no "close enough".&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 1: the body was parsed before you verified it
&lt;/h2&gt;

&lt;p&gt;By far the most common. A global JSON body parser runs on every route, including the webhook route, and by the time your handler runs, the raw bytes are gone.&lt;/p&gt;

&lt;h3&gt;
  
  
  Express
&lt;/h3&gt;

&lt;p&gt;Mount the webhook route with &lt;code&gt;express.raw()&lt;/code&gt; &lt;strong&gt;before&lt;/strong&gt; any global &lt;code&gt;express.json()&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;express&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"express"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"stripe"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;STRIPE_SECRET_KEY&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;express&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/webhooks/stripe"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;express&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;webhooks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;constructEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                              &lt;span class="c1"&gt;// a Buffer, untouched&lt;/span&gt;
      &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"stripe-signature"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;STRIPE_WEBHOOK_SECRET&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Webhook signature check failed:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendStatus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="c1"&gt;// enqueue event.id / event.type for processing, then:&lt;/span&gt;
  &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendStatus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;use&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;express&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// everything else gets parsed JSON&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quick check: log &lt;code&gt;Buffer.isBuffer(req.body)&lt;/code&gt; in the handler. If it prints &lt;code&gt;false&lt;/code&gt;, something parsed the body first.&lt;/p&gt;

&lt;h3&gt;
  
  
  Next.js App Router
&lt;/h3&gt;

&lt;p&gt;Read the body as text, not JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/api/webhooks/stripe/route.js&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Stripe&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"stripe"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Stripe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;STRIPE_SECRET_KEY&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nc"&gt;POST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sig&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"stripe-signature"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;webhooks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;constructEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;STRIPE_WEBHOOK_SECRET&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="c1"&gt;// handle event&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`Webhook Error: ${err.message}`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Calling &lt;code&gt;await req.json()&lt;/code&gt; anywhere before this, including in middleware that reads the body, breaks it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Next.js Pages Router
&lt;/h3&gt;

&lt;p&gt;Pages API routes parse the body by default. Turn that off for this route and read the raw stream:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// pages/api/webhooks/stripe.js&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;api&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;bodyParser&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;rawBody&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="nc"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;handler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;buf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;rawBody&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;webhooks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;constructEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"stripe-signature"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;STRIPE_WEBHOOK_SECRET&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="c1"&gt;// ...&lt;/span&gt;
  &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;status&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Flask
&lt;/h3&gt;

&lt;p&gt;Use &lt;code&gt;request.get_data()&lt;/code&gt;, never &lt;code&gt;request.json&lt;/code&gt; or &lt;code&gt;request.get_json()&lt;/code&gt; before verifying:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;flask&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Flask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Flask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;endpoint_secret&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"STRIPE_WEBHOOK_SECRET"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/webhooks/stripe"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;stripe_webhook&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# raw bytes&lt;/span&gt;
    &lt;span class="n"&gt;sig&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Stripe-Signature"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Webhook&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;construct_event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;endpoint_secret&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SignatureVerificationError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logger&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warning&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Stripe webhook rejected: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt;
    &lt;span class="c1"&gt;# enqueue event["id"], then:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In older versions of the Python library, the exception lives at &lt;code&gt;stripe.error.SignatureVerificationError&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 2: the wrong signing secret
&lt;/h2&gt;

&lt;p&gt;Every webhook endpoint has its own secret, and so does the Stripe CLI. Common mix-ups:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Using the secret printed by &lt;code&gt;stripe listen&lt;/code&gt; in production, or the dashboard endpoint's secret while testing with &lt;code&gt;stripe listen&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Test mode and live mode endpoints have different secrets.&lt;/li&gt;
&lt;li&gt;Two endpoints (say, staging and production) pointing at the same server, with only one secret configured.&lt;/li&gt;
&lt;li&gt;Using your API secret key (&lt;code&gt;sk_...&lt;/code&gt;) instead of the webhook signing secret (&lt;code&gt;whsec_...&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check: in the dashboard, open the endpoint the event was sent to (the event's delivery log shows which), reveal its signing secret and compare the last four characters with what your server loaded. Print only the last four in your logs, never the whole secret.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Once signatures verify, you still have to retry the work if your handler fails.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; takes the HTTP callback, retries it, and keeps the run log.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Cause 3: something between Stripe and your code changed the body
&lt;/h2&gt;

&lt;p&gt;Proxies, API gateways, serverless adapters and some middleware re-encode bodies: they parse and re-stringify JSON, change the character encoding, or decompress and re-compress. Anything that touches the bytes breaks the signature.&lt;/p&gt;

&lt;p&gt;Check: log the length of the body you received and compare it with the length shown in the event's delivery details in the dashboard. Different lengths mean something altered it. In AWS API Gateway and some serverless frameworks, look for a "binary" or "raw body" option, or for base64-encoded bodies that need decoding first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 4: the header isn't reaching you
&lt;/h2&gt;

&lt;p&gt;If &lt;code&gt;Stripe-Signature&lt;/code&gt; is missing or empty, verification fails with a similar error. Some proxies strip unknown headers. Log whether the header is present (not its value).&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 5: timestamps out of tolerance
&lt;/h2&gt;

&lt;p&gt;This one has a different message ("Timestamp outside the tolerance zone"), but it gets confused with the others. Stripe's libraries reject signatures more than five minutes old by default, to prevent replays. If your server clock is wrong, or you're replaying an old event body by hand, you'll see it. Fix the clock (NTP); don't raise the tolerance.&lt;/p&gt;

&lt;h2&gt;
  
  
  After it works
&lt;/h2&gt;

&lt;p&gt;Verify, store the event ID, return 200 quickly, and process in a background job. Signature checks prove who sent the event; they don't stop duplicates or out-of-order delivery, so the processing side still needs to be idempotent. Retrying the processing on your own schedule, rather than relying on Stripe's retries, is the kind of thing &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; handles, but a database-backed queue works too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/how-to-verify-hmac-sha256-signed-webhooks-in-node-and-python-2chl"&gt;How to verify HMAC-SHA256 signed webhooks in Node and Python&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/8-free-tools-for-testing-webhooks-on-localhost-and-when-to-use-each-7ao"&gt;8 free tools for testing webhooks on localhost, and when to use each&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/return-200-fast-why-your-webhook-handler-should-do-almost-nothing-2c1o"&gt;Return 200 fast: why your webhook handler should do almost nothing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/before-your-payment-webhooks-go-live-a-12-point-production-checklist-3fn9"&gt;Before your payment webhooks go live: a 12-point production checklist&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which of these five got you? I'd bet on the global body parser for most people, but the CLI-versus-dashboard secret mix-up is a close second.&lt;/p&gt;

</description>
      <category>webhooks</category>
      <category>node</category>
      <category>python</category>
      <category>stripe</category>
    </item>
    <item>
      <title>Should webhook handlers trust the payload, or fetch the latest state from the API? Let's argue about it</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Sat, 14 Nov 2026 08:18:02 +0000</pubDate>
      <link>https://dev.to/delayrun/should-webhook-handlers-trust-the-payload-or-fetch-the-latest-state-from-the-api-lets-argue-5eg</link>
      <guid>https://dev.to/delayrun/should-webhook-handlers-trust-the-payload-or-fetch-the-latest-state-from-the-api-lets-argue-5eg</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A webhook arrives: &lt;code&gt;order.updated&lt;/code&gt;, &lt;code&gt;subscription.updated&lt;/code&gt;, &lt;code&gt;issue.edited&lt;/code&gt;. The body contains the object, or most of it. Your handler has two choices:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Trust the payload.&lt;/strong&gt; Read the new state from the event body and write it to your database.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat the event as a doorbell.&lt;/strong&gt; Note the object's ID, call the sender's API, fetch the current state, and write that.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Both are common. Both are recommended by someone. Some providers have started sending "thin" events that contain only an ID and a type, which makes the choice for you. I think this is one of the more interesting design arguments in integrations, so let's have it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for trusting the payload
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It's fast and cheap.&lt;/strong&gt; No extra API call, no extra latency, no extra point of failure. For a high-volume integration, re-fetching every object can double your API usage or more, and push you into the sender's rate limits exactly when traffic spikes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The sender's API might be down.&lt;/strong&gt; Webhooks are often queued and retried by the sender for hours or days. If the sender's API has an outage, a fetch-first handler can't process anything, while a payload-first handler keeps working from the events it already has.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The payload is what the sender meant at that moment.&lt;/strong&gt; For event-like facts ("payment succeeded at 12:01", "user clicked unsubscribe"), the payload is the record. Fetching later may return a state that has since moved on, and you lose the history of what happened.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's what most docs show.&lt;/strong&gt; Tutorials, SDK examples and quick-start guides overwhelmingly read from the event body. Code that follows the docs is easier for the next developer to understand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for fetching the latest state
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Webhook order isn't guaranteed.&lt;/strong&gt; Most providers say explicitly that events can arrive out of order and more than once. If you apply payloads as they arrive, an older event processed late overwrites newer state. Fetching makes order irrelevant: every handler converges on the current state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Payloads can be stale even when they arrive in order.&lt;/strong&gt; Retries deliver the original payload, which may be hours old. A handler that applies a three-hour-old "status: past_due" to a subscription that has since been paid is doing damage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Payloads may be incomplete.&lt;/strong&gt; Many APIs send a partial object, omit expanded relations, or truncate large fields. Handlers that trust the payload start growing special cases ("if the field is missing, fetch it anyway").&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It simplifies the code.&lt;/strong&gt; Instead of one handler per event type, each with its own idea of what changed, you have one "sync this object" function, called by every event, by a nightly reconciler, and by an admin button. Three triggers, one code path.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's more secure against forged events.&lt;/strong&gt; If someone bypasses your signature check, a fetch-first handler still only writes what the real API says.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;My default is: &lt;strong&gt;fetch for state, trust for facts.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;State&lt;/strong&gt; is anything that can change again: subscription status, order status, user profile, issue labels. For these, the payload is a hint. Fetch the current object and write that. Out-of-order and duplicate events become harmless.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Facts&lt;/strong&gt; are things that happened once and won't un-happen: a payment captured, a message delivered, a form submitted. For these, the payload is the record. Store it as an event, idempotently, keyed by the event ID.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then three supporting habits:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Debounce bursts per object.&lt;/strong&gt; If five events arrive for the same subscription within a second, fetch once. A short delay ("sync this object in 2 seconds, collapsing duplicates") cuts API calls dramatically. This is a good use of delayed jobs, whether through something like &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; or a queue with deduplication keys.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fall back to the payload when the API is unavailable&lt;/strong&gt;, but mark the record as "synced from event" and re-sync later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reconcile on a schedule.&lt;/strong&gt; Whatever you choose, a periodic job comparing your state with the provider's catches what both approaches miss.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Need a second look at the event, a few seconds later?&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; schedules a delayed HTTP callback with retries, so you can fetch the latest state after the webhook arrives.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The cases that start arguments
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Thin events.&lt;/strong&gt; If the provider only sends IDs, you have to fetch. Is that a better design, or just shifting cost to the receiver?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Version numbers.&lt;/strong&gt; Some APIs include a monotonically increasing version or sequence number on each object. With that, you can trust payloads safely by ignoring anything older than what you have. Is that enough to make fetching unnecessary?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limits.&lt;/strong&gt; At what volume does fetching stop being reasonable? Hundreds of events a minute? Thousands?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your own webhooks.&lt;/strong&gt; If you &lt;em&gt;send&lt;/em&gt; webhooks, should you send full objects or thin events with a fetch URL? What do your consumers prefer?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What the hybrid looks like in code
&lt;/h2&gt;

&lt;p&gt;A minimal version of "fetch for state, trust for facts", in pseudo-Node:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;onEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insertIgnore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;FACT_TYPES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;recordFact&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// payment captured, message delivered&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sync-object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;delaySeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dedupeKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`sync:${id}`&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fact path stores what happened, once. The state path collapses a burst of events into one delayed sync job per object, which then calls the API and writes the current state. Both paths are idempotent, and neither cares what order events arrived in.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick test for your own handler
&lt;/h2&gt;

&lt;p&gt;Take one handler and ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If this event arrived twice, would anything break?&lt;/li&gt;
&lt;li&gt;If it arrived after a newer event for the same object, would it overwrite newer data?&lt;/li&gt;
&lt;li&gt;If it arrived three hours late, would it apply old state?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If any answer is "yes", the payload-trusting version needs either version checks or a fetch.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-hypothetical-the-customer-who-paid-but-the-app-still-said-free-plan-3lag"&gt;Case study (hypothetical): the customer who paid, but the app still said 'Free plan'&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/the-transactional-outbox-explained-how-to-save-to-the-database-and-enqueue-a-job-without-losing-2nld"&gt;The transactional outbox, explained: how to save to the database and enqueue a job without losing either&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/exactly-once-scheduled-jobs-dont-exist-design-for-at-least-once-and-stop-double-charging-2ki9"&gt;Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-hypothetical-the-payment-webhook-that-arrived-before-the-order-existed-1f76"&gt;Case study (hypothetical): the payment webhook that arrived before the order existed&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which way do your webhook handlers lean: payload or fetch? And if you trust the payload, how do you deal with events that arrive out of order?&lt;/p&gt;

</description>
      <category>webhooks</category>
      <category>api</category>
      <category>architecture</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Case study (hypothetical): the customer who paid, but the app still said 'Free plan'</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Sat, 14 Nov 2026 08:17:36 +0000</pubDate>
      <link>https://dev.to/delayrun/case-study-hypothetical-the-customer-who-paid-but-the-app-still-said-free-plan-3lag</link>
      <guid>https://dev.to/delayrun/case-study-hypothetical-the-customer-who-paid-but-the-app-still-said-free-plan-3lag</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a hypothetical case study. The product, the numbers and the timeline are invented; the bug is one of the most common ones in subscription billing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;A small SaaS app sells a Pro plan through a payment provider's hosted checkout. The integration was built the way many tutorials show:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The user clicks "Upgrade", and the backend creates a checkout session.&lt;/li&gt;
&lt;li&gt;The user pays on the provider's page and is sent back to the app.&lt;/li&gt;
&lt;li&gt;The provider sends webhooks. The app's handler listens for subscription events and sets the account's plan from the subscription status in the payload.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The handler was short:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/webhooks/billing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;verifySignature&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"customer.subscription."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;object&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"active"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"trialing"&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="s2"&gt;"pro"&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"free"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;accounts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;billingCustomerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendStatus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It returned 200 quickly and processed events in a background queue with several workers. Everything about it looked reasonable.&lt;/p&gt;

&lt;h2&gt;
  
  
  What customers saw
&lt;/h2&gt;

&lt;p&gt;Roughly 1 in 150 upgrades, a customer paid, got a receipt email from the provider, came back to the app, and saw "Free plan". Refreshing didn't help. Waiting didn't help. They contacted support, who would click "Sync from billing" in the admin panel, and the account would flip to Pro.&lt;/p&gt;

&lt;p&gt;Support assumed it was a delay. It wasn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually happened
&lt;/h2&gt;

&lt;p&gt;A new subscription produces several events in quick succession. For a card that needs confirmation, the sequence is typically:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;customer.subscription.created&lt;/code&gt; with &lt;code&gt;status: incomplete&lt;/code&gt; (payment not yet confirmed)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;invoice.paid&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;customer.subscription.updated&lt;/code&gt; with &lt;code&gt;status: active&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The provider sends these within a second or two of each other, and it says clearly in its documentation that it &lt;strong&gt;does not guarantee delivery order&lt;/strong&gt;. Deliveries can also be retried, so an older event can arrive after a newer one.&lt;/p&gt;

&lt;p&gt;On top of that, the app's queue had several workers. Even when events arrived in order, two workers could pick up &lt;code&gt;created&lt;/code&gt; and &lt;code&gt;updated&lt;/code&gt; at the same time, and whichever finished last won.&lt;/p&gt;

&lt;p&gt;So occasionally the app processed &lt;code&gt;updated (active)&lt;/code&gt; first and set the plan to Pro, then processed &lt;code&gt;created (incomplete)&lt;/code&gt; a few hundred milliseconds later and set it back to Free. The handler had no idea that the second event was older. It applied whatever status the payload contained.&lt;/p&gt;

&lt;p&gt;The admin "Sync from billing" button worked because it ignored webhooks completely: it fetched the subscription from the provider's API and used the current status.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Need webhook delivery you can actually audit?&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; retries failed HTTP callbacks and keeps full run logs, so a missed payment event doesn't silently leave a customer on the free plan.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why the obvious fixes weren't enough
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;"Process events in order."&lt;/strong&gt; A single worker with a strict FIFO queue would remove the worker race, but not the delivery-order problem, and it makes one slow event block every other customer's billing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Compare event timestamps."&lt;/strong&gt; Better: ignore an event if its &lt;code&gt;created&lt;/code&gt; timestamp is older than the last one applied for that subscription. But event timestamps have one-second resolution, and the &lt;code&gt;created&lt;/code&gt; and &lt;code&gt;updated&lt;/code&gt; events here were often in the same second. Ties still lost.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Only listen for &lt;code&gt;updated&lt;/code&gt;."&lt;/strong&gt; Ignoring &lt;code&gt;created&lt;/code&gt; fixes this particular sequence and breaks others. There are many paths through a subscription's lifecycle (trials, failed first payments, payment method updates, plan changes, cancellations at period end) and each emits a different mix of events.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: treat webhooks as a doorbell
&lt;/h2&gt;

&lt;p&gt;The team changed the handler to treat every subscription-related webhook as a signal that &lt;em&gt;something&lt;/em&gt; changed, not as the description of &lt;em&gt;what&lt;/em&gt; changed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;handleBillingEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;customerId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="c1"&gt;// Always read the current state; never trust the payload's snapshot.&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;subs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;billing&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;customerId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"all"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"trialing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"past_due"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;accounts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;billingCustomerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;customerId&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="s2"&gt;"pro"&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"free"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;billingSyncedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whichever event arrives, and in whatever order, the handler fetches the current subscription state from the provider and writes it. Processing the old &lt;code&gt;created&lt;/code&gt; event last no longer matters, because it reads the same current state as the newer one.&lt;/p&gt;

&lt;p&gt;Three more changes went with it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Serialise per customer, not globally.&lt;/strong&gt; Jobs are keyed by billing customer ID, so two events for the same customer never run concurrently, while different customers still process in parallel. Most job queues support this as a "group", "concurrency key" or "partition key".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sync on return from checkout.&lt;/strong&gt; When the browser lands on the success URL, the backend runs the same sync function immediately. The user sees Pro straight away, without waiting for any webhook.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reconcile nightly.&lt;/strong&gt; A scheduled job compares every account's plan with the provider's subscription status and fixes and logs any mismatches. In the first week it found 31 accounts, some from this bug and some from older ones.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;Stuck upgrades went to zero. The "Sync from billing" button was used twice in the following month, both times for unrelated reasons. The cost was one extra API call per webhook, well within the provider's rate limits for an app of this size.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd take from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Webhook order is not guaranteed.&lt;/strong&gt; Not by the sender, and not by your own queue. Design as if every event can arrive late, early or twice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A payload is a snapshot from the past.&lt;/strong&gt; For state that matters, fetch the current state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Serialise per entity.&lt;/strong&gt; Concurrency is fine across customers and dangerous within one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give users a sync on return.&lt;/strong&gt; The redirect back from checkout is the moment they care most.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Short follow-up jobs like "sync this customer in 30 seconds, and again in 5 minutes if the subscription still looks incomplete" are exactly the kind of delayed work &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; is built to schedule, but any durable queue with per-key concurrency will do.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-hypothetical-the-payment-webhook-that-arrived-before-the-order-existed-1f76"&gt;Case study (hypothetical): the payment webhook that arrived before the order existed&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/return-200-fast-why-your-webhook-handler-should-do-almost-nothing-2c1o"&gt;Return 200 fast: why your webhook handler should do almost nothing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/make-scheduled-job-handlers-idempotent-safe-to-run-twice-499o"&gt;Make scheduled job handlers idempotent: safe to run twice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-redesigning-a-failed-payment-retry-schedule-that-retried-every-card-at-0000-utc-2c9k"&gt;Case study: redesigning a failed-payment retry schedule that retried every card at 00:00 UTC&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/should-webhook-handlers-trust-the-payload-or-fetch-the-latest-state-from-the-api-lets-argue-5eg"&gt;Should webhook handlers trust the payload, or fetch the latest state from the API? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you had paid customers stuck on the wrong plan? Did you fix it with ordering, timestamps, or by re-fetching the state like this?&lt;/p&gt;

</description>
      <category>webhooks</category>
      <category>backend</category>
      <category>api</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Should cron schedules live in your repo or in your platform's dashboard? Let's argue about it</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Fri, 13 Nov 2026 08:13:42 +0000</pubDate>
      <link>https://dev.to/delayrun/should-cron-schedules-live-in-your-repo-or-in-your-platforms-dashboard-lets-argue-about-it-3kel</link>
      <guid>https://dev.to/delayrun/should-cron-schedules-live-in-your-repo-or-in-your-platforms-dashboard-lets-argue-about-it-3kel</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ask where a team's scheduled jobs are defined and you'll often get a pause, then an answer like: "Most of them are in the repo. A couple are in the hosting dashboard. And there's one on the old VM that I think still runs?"&lt;/p&gt;

&lt;p&gt;That mix usually isn't a decision. It happens one job at a time. But it's worth deciding on purpose, because where a schedule lives determines who can change it, how fast, and whether anyone will know it changed.&lt;/p&gt;

&lt;p&gt;Here are the two camps.&lt;/p&gt;

&lt;h2&gt;
  
  
  Camp 1: schedules are code
&lt;/h2&gt;

&lt;p&gt;The schedule lives next to the job it runs: a &lt;code&gt;crontab&lt;/code&gt; file in the repo, a &lt;code&gt;schedule:&lt;/code&gt; block in a CI workflow, a Kubernetes &lt;code&gt;CronJob&lt;/code&gt; manifest, a &lt;code&gt;vercel.json&lt;/code&gt; or &lt;code&gt;wrangler.toml&lt;/code&gt; entry, or a call in your app's startup code that registers jobs with a scheduler library.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What you get:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Review.&lt;/strong&gt; Changing &lt;code&gt;0 3 * * *&lt;/code&gt; to &lt;code&gt;0 * * * *&lt;/code&gt; (daily to hourly) is a one-character diff that can multiply load by 24. In a pull request, someone gets to ask "did you mean that?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;History.&lt;/strong&gt; When the nightly job starts running at a strange time, &lt;code&gt;git log&lt;/code&gt; tells you who changed it and why.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Environments stay in sync.&lt;/strong&gt; Staging and production get the same schedules from the same file, adjusted by config rather than by someone remembering to click the same buttons twice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disaster recovery.&lt;/strong&gt; Rebuild the environment from the repo and the schedules come back. Nobody has to reconstruct them from screenshots.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What it costs:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every change is a deploy.&lt;/strong&gt; Pausing a job that's hammering a struggling partner API at 2am means a commit, a review (or a bypassed review), and a pipeline run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Non-developers can't touch it.&lt;/strong&gt; If the marketing team owns when the newsletter goes out, they now need an engineer to move it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The schedule and the job are coupled.&lt;/strong&gt; Changing the time of a job can mean redeploying the whole app.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Camp 2: schedules are configuration in a dashboard
&lt;/h2&gt;

&lt;p&gt;The schedule lives in the platform: a scheduler service's console, the hosting provider's cron UI, a job runner's admin screen, or an &lt;code&gt;admin/scheduled_tasks&lt;/code&gt; page in your own app.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What you get:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Speed.&lt;/strong&gt; Pause, resume, run now, change the time. Seconds, no deploy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Visibility.&lt;/strong&gt; Good dashboards show the last run, the next run, the duration and the failure, all in one place. The YAML file shows you none of that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ownership by the right people.&lt;/strong&gt; Operations or product can adjust timing without waiting for an engineer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What it costs:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Drift.&lt;/strong&gt; Production has a job that staging doesn't. The schedule in the dashboard no longer matches the comment in the code that says "runs nightly".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No review.&lt;/strong&gt; A well-meaning change to "run every minute while I debug this" stays that way for six weeks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lost on rebuild.&lt;/strong&gt; Move to a new account or provider and you're rebuilding schedules from memory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audit gaps.&lt;/strong&gt; Some dashboards log who changed what. Many don't, or keep the log for 30 days.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What tends to work in practice
&lt;/h2&gt;

&lt;p&gt;Most teams I've seen settle on a split, and the split is the useful part to discuss:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Definitions in the repo.&lt;/strong&gt; Which jobs exist, what they run, their default schedule, timeout, concurrency and time zone. This is the part that should be reviewed and versioned.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Operational controls in the dashboard.&lt;/strong&gt; Pause, resume, run now, and maybe a temporary override with an expiry. These are actions, not definitions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overrides are visible and temporary.&lt;/strong&gt; If someone pauses a job or changes its time in the UI, the system shows that it differs from the repo's definition, and ideally reverts or nags after a set period.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A drift check.&lt;/strong&gt; A small job that compares what the scheduler thinks is configured with what the repo says, and alerts on differences. It catches the one-off job someone created by hand during an incident and forgot about.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Wherever you author the expression, you still want a run log that isn't the server's syslog.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; executes the HTTP schedule, retries failures, and stores that history for you.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A minimal "schedules as code" file that leaves room for operations
&lt;/h2&gt;

&lt;p&gt;If you lean towards the repo, it helps to put the operational details in the definition too, so the dashboard has less reason to diverge:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;nightly-invoice-sync&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"15 2 * * *"&lt;/span&gt;
    &lt;span class="na"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Europe/Berlin"&lt;/span&gt;
    &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"bin/sync-invoices"&lt;/span&gt;
    &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;20m&lt;/span&gt;
    &lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;forbid&lt;/span&gt;      &lt;span class="c1"&gt;# skip if the previous run is still going
&lt;/span&gt;    &lt;span class="na"&gt;missed_runs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;run_once&lt;/span&gt;    &lt;span class="c1"&gt;# after downtime, run one catch-up, not ten
&lt;/span&gt;    &lt;span class="na"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;billing-team&lt;/span&gt;
    &lt;span class="na"&gt;pausable_in_ui&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;timezone&lt;/code&gt;, &lt;code&gt;concurrency&lt;/code&gt; and &lt;code&gt;missed_runs&lt;/code&gt; lines answer the questions people usually go to the dashboard to fix. &lt;code&gt;owner&lt;/code&gt; tells whoever gets paged who to wake up, and &lt;code&gt;pausable_in_ui&lt;/code&gt; makes the pause button an explicit, reviewed permission rather than an accident.&lt;/p&gt;

&lt;h2&gt;
  
  
  The schedules that don't fit either camp
&lt;/h2&gt;

&lt;p&gt;Some schedules aren't really infrastructure: "send this user a reminder 3 days after signup", "retry this payment on Thursday", "publish this post at 09:00 Berlin time". These are per-record and created at runtime, often thousands of them. They don't belong in a YAML file or in a dashboard someone clicks through. They belong in data, created by the app, through an API.&lt;/p&gt;

&lt;p&gt;A lot of the repo-vs-dashboard argument goes away once you separate those two kinds of schedule. Fixed, recurring infrastructure jobs: in the repo. Dynamic, per-user or per-event timing: in your data and a job queue or a scheduling service such as &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;. Operational controls for both: in a dashboard.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions to settle it for your team
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;When did someone last need to pause or move a job urgently, and how long did it take?&lt;/li&gt;
&lt;li&gt;Could you rebuild every schedule you run from the repo alone today?&lt;/li&gt;
&lt;li&gt;Do non-engineers need to change any timings, and should they?&lt;/li&gt;
&lt;li&gt;If a schedule was changed by hand last month, would anyone know?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-hypothetical-the-payment-webhook-that-arrived-before-the-order-existed-1f76"&gt;Case study (hypothetical): the payment webhook that arrived before the order existed&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/should-background-workers-ship-with-the-web-app-or-deploy-separately-lets-argue-about-it-34pd"&gt;Should background workers ship with the web app or deploy separately? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/how-to-run-cron-jobs-on-serverless-vercel-cloudflare-workers-and-netlify-228"&gt;How to run cron jobs on serverless: Vercel, Cloudflare Workers and Netlify&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-in-docker-faq-11-questions-from-everyone-whose-job-worked-locally-and-does-nothing-in-the-336b"&gt;Cron in Docker FAQ: 11 questions from everyone whose job worked locally and does nothing in the container&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So: repo, dashboard, or a split? Where do your scheduled jobs live today, and has the setup ever bitten you?&lt;/p&gt;

</description>
      <category>cron</category>
      <category>devops</category>
      <category>architecture</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Case study (hypothetical): the payment webhook that arrived before the order existed</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Fri, 13 Nov 2026 08:13:01 +0000</pubDate>
      <link>https://dev.to/delayrun/case-study-hypothetical-the-payment-webhook-that-arrived-before-the-order-existed-1f76</link>
      <guid>https://dev.to/delayrun/case-study-hypothetical-the-payment-webhook-that-arrived-before-the-order-existed-1f76</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a hypothetical case study, assembled from a pattern that shows up in a lot of payment integrations. The shop, the numbers and the timeline are invented; the race condition is very real.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;A small online shop sells physical goods. Checkout works like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The browser submits the cart to the shop's backend.&lt;/li&gt;
&lt;li&gt;The backend creates a payment intent with the payment provider and gets back an ID.&lt;/li&gt;
&lt;li&gt;The backend saves an &lt;code&gt;orders&lt;/code&gt; row with status &lt;code&gt;awaiting_payment&lt;/code&gt; and the payment intent ID.&lt;/li&gt;
&lt;li&gt;The customer completes payment on the provider's page.&lt;/li&gt;
&lt;li&gt;The provider sends a &lt;code&gt;payment.succeeded&lt;/code&gt; webhook. The shop's handler looks up the order by payment intent ID and marks it &lt;code&gt;paid&lt;/code&gt;, which triggers fulfilment.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It worked fine in testing. In production, roughly 1 in 400 orders stayed &lt;code&gt;awaiting_payment&lt;/code&gt; forever, even though the customer had been charged. Support found out when customers emailed asking where their parcel was.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually happened
&lt;/h2&gt;

&lt;p&gt;Steps 2 and 3 were in the wrong order for a world with real latency, and step 5 assumed step 3 had already happened.&lt;/p&gt;

&lt;p&gt;The backend created the payment intent first, then opened a database transaction to save the order. Under normal load the transaction committed in a few milliseconds. But with wallet payments, some customers completed payment almost instantly, and the provider is fast at sending webhooks. Occasionally, especially when the database was busy, the sequence went:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;12:00:00.000 backend creates payment intent &lt;code&gt;pi_123&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;12:00:00.180 customer pays with a saved wallet&lt;/li&gt;
&lt;li&gt;12:00:00.240 provider sends &lt;code&gt;payment.succeeded&lt;/code&gt; for &lt;code&gt;pi_123&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;12:00:00.250 webhook handler: &lt;code&gt;SELECT ... WHERE payment_intent_id = 'pi_123'&lt;/code&gt; returns nothing&lt;/li&gt;
&lt;li&gt;12:00:00.300 checkout transaction commits the order row&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The handler did what a lot of handlers do when they can't find the record: logged a warning and returned &lt;code&gt;200 OK&lt;/code&gt;. From the provider's point of view, the webhook was delivered successfully. It would never retry. The order sat there unpaid.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;When a webhook is early, the fix is often “do the work in a moment, not in the request.”&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; schedules that delayed HTTP callback with retries, so the handler can return 200 and finish the job once the order exists.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why the obvious fixes weren't enough
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;"Just create the order first."&lt;/strong&gt; Moving the insert before creating the payment intent helped a lot, and it's the right order. But it didn't remove the race completely: the order had to be updated with the intent ID afterwards, and that update could still lose to the webhook. It also meant orders existed for checkouts that failed before a payment intent was created, which needed cleanup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Return 500 when the order isn't found, so the provider retries."&lt;/strong&gt; Better, and it would have fixed most cases. But the provider's retry schedule started at around a minute and backed off over days. A customer who paid at 12:00 and got a confirmation email at 12:01 is fine; a webhook that's retried for three days because of a real bug, filling alerts the whole time, is less fine. And a 500 for "not yet" looks the same in dashboards as a 500 for "crashed".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Sleep for a second and look again."&lt;/strong&gt; Inside the webhook handler, this holds a request open, slows every webhook, and still loses if the commit takes longer than the sleep.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: accept, record, reconcile
&lt;/h2&gt;

&lt;p&gt;The shop ended up with three changes, each small.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Store every webhook before doing anything with it
&lt;/h3&gt;

&lt;p&gt;The handler now verifies the signature, writes the raw event into a &lt;code&gt;webhook_events&lt;/code&gt; table with the provider's event ID as a unique key, and returns &lt;code&gt;200&lt;/code&gt; immediately. Processing happens in a background job.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;webhook_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;event_id&lt;/span&gt;      &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;type&lt;/span&gt;          &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt;       &lt;span class="kt"&gt;jsonb&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;received_at&lt;/span&gt;   &lt;span class="kt"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;processed_at&lt;/span&gt;  &lt;span class="kt"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;attempts&lt;/span&gt;      &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The unique key makes duplicate deliveries harmless: an &lt;code&gt;INSERT ... ON CONFLICT DO NOTHING&lt;/code&gt; means the provider can send the same event five times and it's stored once.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Process with retries the shop controls
&lt;/h3&gt;

&lt;p&gt;The job looks up the order. If it's found, it marks it paid in a transaction that also checks the current status, so running twice changes nothing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'paid'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;paid_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;payment_intent_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'awaiting_payment'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the order isn't found, the job doesn't fail and doesn't give up. It schedules itself again: after 5 seconds, then 30, then 2 minutes, then 10. After about an hour without a matching order, it raises an alert, because at that point it's a real bug, not a race.&lt;/p&gt;

&lt;p&gt;That schedule was the key difference. The shop no longer depended on the provider's retry timing; it decided for itself how long "not yet" was allowed to last.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Reconcile from the other side
&lt;/h3&gt;

&lt;p&gt;Webhooks are a notification, not a source of truth. A scheduled job now runs every 15 minutes and asks the provider about any order that has been &lt;code&gt;awaiting_payment&lt;/code&gt; for more than 10 minutes. If the provider says the intent succeeded, the job marks the order paid and logs that the webhook path missed it. This catches anything the first two parts don't: a webhook that never arrived, a signing secret rotated without updating the handler, a deploy that briefly broke the endpoint.&lt;/p&gt;

&lt;p&gt;In the first month, the reconciler fixed 3 orders. Each one pointed to a different small bug, and each one was found within 15 minutes instead of by a customer email.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd take from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Never return 200 for "I couldn't do anything with this" unless you've stored it.&lt;/strong&gt; Accepting and dropping is the worst combination.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Assume webhooks can arrive before, after, or instead of your own write.&lt;/strong&gt; Ordering between your database commit and someone else's HTTP request isn't something you can rely on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Own your retry schedule.&lt;/strong&gt; The sender's retries exist for when &lt;em&gt;you're&lt;/em&gt; down; they're a poor tool for "I'm up but not ready".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reconcile.&lt;/strong&gt; Any flow where money moves should have a periodic job that compares your state with the provider's.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Scheduling those short "try again in 5 seconds, then 30" follow-ups reliably is the kind of delayed job &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; is designed for, but the pattern works with any durable job queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/return-200-fast-why-your-webhook-handler-should-do-almost-nothing-2c1o"&gt;Return 200 fast: why your webhook handler should do almost nothing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/make-scheduled-job-handlers-idempotent-safe-to-run-twice-499o"&gt;Make scheduled job handlers idempotent: safe to run twice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/the-transactional-outbox-explained-how-to-save-to-the-database-and-enqueue-a-job-without-losing-2nld"&gt;The transactional outbox, explained: how to save to the database and enqueue a job without losing either&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/should-cron-schedules-live-in-your-repo-or-in-your-platforms-dashboard-lets-argue-about-it-3kel"&gt;Should cron schedules live in your repo or in your platform's dashboard? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-hypothetical-the-customer-who-paid-but-the-app-still-said-free-plan-3lag"&gt;Case study (hypothetical): the customer who paid, but the app still said 'Free plan'&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you hit this race in a payment or webhook integration? Did you fix it with retries, reconciliation, or by changing the order of writes?&lt;/p&gt;

</description>
      <category>webhooks</category>
      <category>backend</category>
      <category>architecture</category>
      <category>api</category>
    </item>
    <item>
      <title>What I'd do differently designing background job payloads (now that some of them sit in a queue for months)</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Thu, 12 Nov 2026 13:15:59 +0000</pubDate>
      <link>https://dev.to/delayrun/what-id-do-differently-designing-background-job-payloads-now-that-some-of-them-sit-in-a-queue-for-2641</link>
      <guid>https://dev.to/delayrun/what-id-do-differently-designing-background-job-payloads-now-that-some-of-them-sit-in-a-queue-for-2641</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;When you first add background jobs, payloads are an afterthought. You enqueue whatever the handler needs, it runs a second later, done.&lt;/p&gt;

&lt;p&gt;Then you start scheduling things for later: a trial-ending reminder in 14 days, a renewal notice in a year, a follow-up email in 30 days. Now that payload has to survive every deploy, every refactor and every data change between now and then. That's when the shortcuts from the early days start to hurt.&lt;/p&gt;

&lt;p&gt;Here's what I'd do differently if I were designing payloads from scratch.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Store references, not snapshots
&lt;/h2&gt;

&lt;p&gt;The most common early mistake is enqueueing a serialised object:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"old@example.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"plan"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"pro"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Sam"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By the time the job runs, the user may have changed their email, downgraded, or deleted their account. The job runs confidently on stale data, and sends the reminder to an address the user removed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; store identifiers, and load current state when the job runs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"user_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"subscription_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;981&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There's one exception: when you genuinely need the value &lt;em&gt;as it was&lt;/em&gt; when the job was scheduled, like the price quoted at checkout or the content of a message the user wrote. Then store it deliberately, and name it so it's obviously a snapshot: &lt;code&gt;quoted_price_cents&lt;/code&gt;, not &lt;code&gt;price&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Version every payload from day one
&lt;/h2&gt;

&lt;p&gt;Your handler code will change. Jobs scheduled under the old shape will still be in the queue when the new code deploys.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; include a version field in every payload, and make handlers accept the previous version for at least as long as your longest scheduled delay.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"v"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"user_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"reminder_kind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"trial_ending"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;unknown&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ReminderV2&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;?:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nc"&gt;ReminderV2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;upgradeV1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nc"&gt;ReminderV1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`Unknown payload version ${p.v}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It costs one field. It saves the deploy where every long-delay job starts failing because a property was renamed.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Don't serialise language-specific objects
&lt;/h2&gt;

&lt;p&gt;Some job libraries make it easy to enqueue a class instance, a pickled object or a framework-specific model. It works until you rename the class, move the module, upgrade the serialiser, or want a service in a different language to read the queue.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; plain JSON with primitive types only. Dates as ISO 8601 strings with an offset, or as epoch milliseconds, never as language date objects. Money as integer minor units with a currency code.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Re-check the reason the job exists
&lt;/h2&gt;

&lt;p&gt;A job scheduled 30 days ago captured an intent: "remind this user their trial ends". When it runs, that intent may no longer apply. They upgraded. They cancelled. Support extended their trial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; every delayed job's handler starts by checking whether it should still run, using current data:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;subs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;subscription_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;sub&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="s2"&gt;"trialing"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trialEndsAt&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;expected_trial_end&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;skip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"trial no longer ending as scheduled"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Store the condition you expected (&lt;code&gt;expected_trial_end&lt;/code&gt;) in the payload so the handler can detect that things changed. And log skips with a reason; a spike in skips is useful information.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A payload that sits in a queue for months still needs a place that will eventually run it.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; holds delayed HTTP jobs, retries them, and keeps the history no matter how long the delay is.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  5. Give each job a stable identity
&lt;/h2&gt;

&lt;p&gt;If the same reminder can be scheduled twice, because of a retry, a double click, or a bug, it eventually will be.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; derive a deterministic key from what the job means, and use it for de-duplication on enqueue and idempotency on execution: &lt;code&gt;trial-ending:sub_981:2026-12-01&lt;/code&gt;. If your queue supports unique job IDs, use the key as the ID. If not, enforce it in your own table.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Keep payloads small and free of secrets
&lt;/h2&gt;

&lt;p&gt;Payloads end up in queue storage, logs, dead-letter queues, admin dashboards and error trackers. Anything in them is effectively widely readable and long-lived.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; no access tokens, passwords, full card numbers or sensitive personal data in payloads. Look them up at run time. Keep payloads small enough that they're comfortable to log in full, because when something goes wrong, you'll want to.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Record why and by whom it was scheduled
&lt;/h2&gt;

&lt;p&gt;Months later, someone asks "why did this customer get this email?". The payload alone often can't answer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; add a small metadata block:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"v"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"user_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"meta"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"scheduled_by"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"billing.trial_started"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"scheduled_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-17T10:04:00Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"correlation_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"req_7f3a"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This makes debugging, auditing and support conversations dramatically easier, and it's useful for finding every job created by a buggy code path so you can cancel them.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Plan for cancelling and rescheduling
&lt;/h2&gt;

&lt;p&gt;If the only way to cancel a scheduled job is to find it in the queue by scanning payloads, you'll end up writing that scan during an incident.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; keep a row in your own database for each long-delay job (key, kind, subject ID, run-at, status), and treat the queue as the delivery mechanism. Cancelling becomes an update; rescheduling becomes an update plus a new enqueue; the handler checks status before running.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;IDs over snapshots, unless the snapshot is the point.&lt;/li&gt;
&lt;li&gt;A version field, and handlers that accept old versions.&lt;/li&gt;
&lt;li&gt;Plain JSON, explicit units and time zones.&lt;/li&gt;
&lt;li&gt;Re-check intent at run time.&lt;/li&gt;
&lt;li&gt;Deterministic keys for de-duplication.&lt;/li&gt;
&lt;li&gt;No secrets, small enough to log.&lt;/li&gt;
&lt;li&gt;Metadata about who scheduled it and why.&lt;/li&gt;
&lt;li&gt;A record you can cancel or reschedule without searching the queue.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Long-delay jobs are where these choices matter most, and they're the jobs &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; is built to hold reliably.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-in-docker-faq-11-questions-from-everyone-whose-job-worked-locally-and-does-nothing-in-the-336b"&gt;Cron in Docker FAQ: 11 questions from everyone whose job worked locally and does nothing in the container&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/dont-use-settimeout-for-anything-that-matters-where-delayed-jobs-should-actually-live-4o26"&gt;Don't use setTimeout for anything that matters: where delayed jobs should actually live&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/make-scheduled-job-handlers-idempotent-safe-to-run-twice-499o"&gt;Make scheduled job handlers idempotent: safe to run twice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/the-transactional-outbox-explained-how-to-save-to-the-database-and-enqueue-a-job-without-losing-2nld"&gt;The transactional outbox, explained: how to save to the database and enqueue a job without losing either&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the oldest job sitting in your queue right now, and would its payload still make sense to today's code?&lt;/p&gt;

</description>
      <category>backend</category>
      <category>architecture</category>
      <category>queues</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Cron in Docker FAQ: 11 questions from everyone whose job worked locally and does nothing in the container</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Thu, 12 Nov 2026 13:12:53 +0000</pubDate>
      <link>https://dev.to/delayrun/cron-in-docker-faq-11-questions-from-everyone-whose-job-worked-locally-and-does-nothing-in-the-336b</link>
      <guid>https://dev.to/delayrun/cron-in-docker-faq-11-questions-from-everyone-whose-job-worked-locally-and-does-nothing-in-the-336b</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Putting a cron job in a Docker container looks like a five-minute task. Install cron, copy a crontab, start the daemon. Then the job silently doesn't run, or runs without its environment, or runs at the wrong hour, and there's no output anywhere.&lt;/p&gt;

&lt;p&gt;Here are the questions that come up most, with direct answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Should I run cron inside a container at all?
&lt;/h2&gt;

&lt;p&gt;Sometimes. The container philosophy is "one process per container", and cron is a long-running daemon that spawns other processes, which fits awkwardly. Common alternatives:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Host cron&lt;/strong&gt; calling &lt;code&gt;docker run&lt;/code&gt; or &lt;code&gt;docker exec&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your orchestrator's scheduler&lt;/strong&gt;: Kubernetes CronJobs, ECS scheduled tasks, Nomad periodic jobs, a PaaS scheduler.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A container-friendly scheduler&lt;/strong&gt; such as supercronic (a crontab-compatible runner designed for containers) or ofelia (a scheduler for Docker that runs jobs in other containers).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That said, a small "cron container" running alongside your app is a perfectly reasonable choice for a single server with Docker Compose. The rest of this FAQ is about doing it well.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. My job runs fine manually but never runs under cron. Why?
&lt;/h2&gt;

&lt;p&gt;The most common reasons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cron isn't actually running.&lt;/strong&gt; In many images, &lt;code&gt;cron&lt;/code&gt; is installed but nothing starts it. Your container's main process needs to &lt;em&gt;be&lt;/em&gt; cron, running in the foreground: &lt;code&gt;cron -f&lt;/code&gt; on Debian/Ubuntu, &lt;code&gt;crond -f&lt;/code&gt; on Alpine (BusyBox).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Syntax or file format problems.&lt;/strong&gt; Files in &lt;code&gt;/etc/cron.d/&lt;/code&gt; require a &lt;strong&gt;user field&lt;/strong&gt; (&lt;code&gt;* * * * * root /app/job.sh&lt;/code&gt;), must end with a &lt;strong&gt;newline&lt;/strong&gt;, and on Debian-based systems file names with dots in them may be ignored. A missing final newline is a classic cause of a silently ignored last line.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Permissions.&lt;/strong&gt; Files in &lt;code&gt;/etc/cron.d&lt;/code&gt; often need to be owned by root and not group- or world-writable, or cron may refuse them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  3. Where does my job's output go?
&lt;/h2&gt;

&lt;p&gt;Under traditional cron, output is emailed to the user. In a container there's no mail, so it vanishes. Docker only collects the stdout/stderr of the container's main process (PID 1), not of the jobs cron spawns.&lt;/p&gt;

&lt;p&gt;A common workaround is to redirect job output to PID 1's file descriptors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* * * * * root /app/job.sh &amp;gt; /proc/1/fd/1 2&amp;gt; /proc/1/fd/2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now &lt;code&gt;docker logs&lt;/code&gt; shows your job's output. Supercronic does this for you by design, which is one of its main selling points.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Why can't my job see environment variables?
&lt;/h2&gt;

&lt;p&gt;Cron starts jobs with a minimal environment. It does not pass the container's environment (from &lt;code&gt;-e&lt;/code&gt;, &lt;code&gt;env_file&lt;/code&gt; or Compose &lt;code&gt;environment:&lt;/code&gt;) to the jobs it launches. So &lt;code&gt;DATABASE_URL&lt;/code&gt; is set in the container but empty inside the job.&lt;/p&gt;

&lt;p&gt;Options:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dump the environment at startup&lt;/strong&gt; into a file the job sources: in your entrypoint, &lt;code&gt;printenv &amp;gt; /etc/environment&lt;/code&gt; (works for cron implementations that read it via PAM, such as Debian's), or write an &lt;code&gt;env.sh&lt;/code&gt; and &lt;code&gt;source&lt;/code&gt; it at the top of the job. Be careful: that file now contains secrets on disk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use supercronic&lt;/strong&gt;, which passes the environment through.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Move scheduling outside the container&lt;/strong&gt;, where the job is started with its environment directly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  5. Why does my job run at the wrong time?
&lt;/h2&gt;

&lt;p&gt;Containers commonly default to UTC, and slim images often don't include time zone data at all. If your schedule is meant to be in local time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Install time zone data (&lt;code&gt;tzdata&lt;/code&gt; on Debian/Alpine).&lt;/li&gt;
&lt;li&gt;Set &lt;code&gt;TZ&lt;/code&gt;, for example &lt;code&gt;TZ=Europe/London&lt;/code&gt;, and check how your cron implementation reads it: some use the system time zone in &lt;code&gt;/etc/localtime&lt;/code&gt;, so you may need to link that file too.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Or keep the container in UTC and write schedules in UTC, which avoids daylight saving surprises entirely. Just document it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;If the container's crontab is the fragile part,&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; moves the schedule out of the image: it calls your HTTP endpoint, retries, and logs the run.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  6. Why doesn't &lt;code&gt;PATH&lt;/code&gt; find my commands?
&lt;/h2&gt;

&lt;p&gt;Cron's default &lt;code&gt;PATH&lt;/code&gt; is usually very short (often just &lt;code&gt;/usr/bin:/bin&lt;/code&gt;). Tools installed in &lt;code&gt;/usr/local/bin&lt;/code&gt; (where many language runtimes and pip-installed scripts land) won't be found. Use absolute paths in your crontab, or set &lt;code&gt;PATH=&lt;/code&gt; at the top of the crontab.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Why does &lt;code&gt;docker stop&lt;/code&gt; take ten seconds?
&lt;/h2&gt;

&lt;p&gt;Docker sends SIGTERM to PID 1 and waits (10 seconds by default) before sending SIGKILL. If PID 1 is a shell script that started cron without &lt;code&gt;exec&lt;/code&gt;, the signal may not reach cron. Use &lt;code&gt;exec cron -f&lt;/code&gt; at the end of your entrypoint, or run a minimal init like &lt;code&gt;tini&lt;/code&gt; (Docker's &lt;code&gt;--init&lt;/code&gt; flag adds one) so signals are forwarded and zombie processes are reaped.&lt;/p&gt;

&lt;p&gt;Also think about what happens to a job that's mid-run when the container stops: it gets killed. Make jobs safe to interrupt and re-run.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. I scaled to three replicas and now every job runs three times. Help?
&lt;/h2&gt;

&lt;p&gt;Each replica runs its own cron. Either run the cron container as a single replica, separate from scaled app containers, or make jobs coordinate (a database lock, a "has this run today" record). Scheduling at the orchestrator level usually avoids this entirely.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. How do I know if a job stopped running?
&lt;/h2&gt;

&lt;p&gt;Cron won't tell you. A container can be "healthy" for months while its job fails every night, or never fires because of a crontab typo. Two good practices:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Log a line at start and end&lt;/strong&gt; of every job, with the scheduled time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use a heartbeat (dead man's switch) check:&lt;/strong&gt; have the job ping a monitoring URL on success, and get alerted when the ping doesn't arrive.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  10. How should I write the Dockerfile?
&lt;/h2&gt;

&lt;p&gt;A minimal Debian-based pattern:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt; debian:stable-slim
&lt;span class="k"&gt;RUN&lt;/span&gt; apt-get update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; apt-get install -y --no-install-recommends cron tzdata ca-certificates \
    &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; rm -rf /var/lib/apt/lists/*
&lt;span class="k"&gt;COPY&lt;/span&gt; crontab /etc/cron.d/app
&lt;span class="k"&gt;RUN&lt;/span&gt; chmod 0644 /etc/cron.d/app
&lt;span class="k"&gt;COPY&lt;/span&gt; job.sh /app/job.sh
&lt;span class="k"&gt;RUN&lt;/span&gt; chmod +x /app/job.sh
&lt;span class="k"&gt;CMD&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cron"&lt;/span&gt;, &lt;span class="s2"&gt;"-f"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With &lt;code&gt;crontab&lt;/code&gt; containing (note the user field and the final newline):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
15 2 * * * root /app/job.sh &amp;gt; /proc/1/fd/1 2&amp;gt; /proc/1/fd/2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  11. When should I stop using cron in a container?
&lt;/h2&gt;

&lt;p&gt;When you need retries, per-run history, jobs scheduled for specific one-off times, or jobs triggered by your application ("send this in 3 days"), cron's model of fixed schedules on one machine starts fighting you. That's the point to look at a job queue with delayed jobs, a workflow engine, or a hosted scheduler like &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; that calls your endpoint at the time you choose.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-job-not-running-a-step-by-step-checklist-including-it-works-when-i-run-it-manually-1nb6"&gt;Cron job not running? A step-by-step checklist (including “it works when I run it manually”)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/quick-tip-your-cron-script-succeeded-because-bash-ignored-the-error-start-every-job-with-set-1f2f"&gt;Quick tip: your cron script 'succeeded' because bash ignored the error. Start every job with set -euo pipefail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/what-happens-to-the-cron-job-thats-running-when-you-deploy-graceful-shutdown-for-scheduled-work-2l2b"&gt;What happens to the cron job that's running when you deploy? Graceful shutdown for scheduled work&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/what-id-do-differently-designing-background-job-payloads-now-that-some-of-them-sit-in-a-queue-for-2641"&gt;What I'd do differently designing background job payloads (now that some of them sit in a queue for months)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What was the cron-in-Docker gotcha that cost you the most time?&lt;/p&gt;

</description>
      <category>docker</category>
      <category>cron</category>
      <category>devops</category>
      <category>linux</category>
    </item>
    <item>
      <title>Quick tip: one Postgres advisory lock stops your scheduled job running twice</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Wed, 11 Nov 2026 16:17:47 +0000</pubDate>
      <link>https://dev.to/delayrun/quick-tip-one-postgres-advisory-lock-stops-your-scheduled-job-running-twice-12j7</link>
      <guid>https://dev.to/delayrun/quick-tip-one-postgres-advisory-lock-stops-your-scheduled-job-running-twice-12j7</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You scaled your app from one server to two. Everything got faster, except that the nightly invoice job now runs twice, and some customers got two invoices.&lt;/p&gt;

&lt;p&gt;It's one of the most common surprises in growing a backend: scheduled code that was safe on one machine isn't safe on several. The heavyweight fixes are a dedicated scheduler service, a leader election library, or a distributed lock in Redis or ZooKeeper. But if you already run PostgreSQL, there's a simpler tool sitting right there: &lt;strong&gt;advisory locks&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The tip
&lt;/h2&gt;

&lt;p&gt;Wrap the job in a non-blocking advisory lock. Whoever gets the lock runs the job; everyone else skips it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="nf"&gt;pg_try_advisory_lock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;pg_try_advisory_lock&lt;/code&gt; returns &lt;code&gt;true&lt;/code&gt; if this session acquired the lock and &lt;code&gt;false&lt;/code&gt; immediately if another session holds it. It never waits. The number (&lt;code&gt;42&lt;/code&gt; here) is an application-chosen key: Postgres doesn't care what it means, only that everyone uses the same one for the same job.&lt;/p&gt;

&lt;p&gt;In application code (Python with psycopg here, but the idea is identical in any language):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;

&lt;span class="no"&gt;INVOICE_JOB_LOCK&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;7_310_001&lt;/span&gt;  &lt;span class="c1"&gt;# any bigint, unique per job&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_nightly_invoices&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SELECT pg_try_advisory_lock(%s)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;INVOICE_JOB_LOCK&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;got_lock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;got_lock&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"another instance is running invoices; skipping"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;generate_invoices&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SELECT pg_advisory_unlock(%s)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;INVOICE_JOB_LOCK&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every server can keep its cron entry or in-process scheduler unchanged. They all wake up, they all ask, one wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this works
&lt;/h2&gt;

&lt;p&gt;Advisory locks are managed by Postgres's normal lock manager, but they don't lock any table or row. They're just named locks your application agrees to respect. Two properties make them a good fit for scheduled jobs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;They're atomic.&lt;/strong&gt; Two sessions can't both get the same lock, even if they ask at the exact same millisecond.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Session-level locks are released automatically if the session ends.&lt;/strong&gt; If the server running the job crashes and its database connection drops, the lock goes with it. You don't end up with a stale lock that blocks the job forever, which is a classic problem with "lock" rows in a table.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A lock stops a double start; it won't retry a failed one.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; schedules the HTTP job once, retries on failure, and keeps a run log of what happened.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Gotchas worth knowing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Hold the lock on the same connection that took it.&lt;/strong&gt; Session-level advisory locks belong to a database session. If your code takes the lock on one pooled connection and then returns that connection to the pool, the lock stays held by whatever session the connection represents, and your unlock may run on a different connection and do nothing. Take a dedicated connection for the duration of the job.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Be careful with transaction-mode connection poolers.&lt;/strong&gt; If you use PgBouncer in transaction pooling mode, consecutive statements may run on different server sessions, so session-level locks don't behave as you'd expect. Options: connect directly to Postgres for the job, use session pooling for that connection, or use the transaction-level variant below.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Consider the transaction-level variant.&lt;/strong&gt; &lt;code&gt;pg_try_advisory_xact_lock(key)&lt;/code&gt; holds the lock until the current transaction ends, then releases it automatically, with no explicit unlock. That's neat when the whole job fits in one transaction, but a long-running transaction has its own costs (it can hold back vacuum and keep locks on rows it touched), so it's not always the right choice for big batch jobs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. This prevents overlap, not duplicates over time.&lt;/strong&gt; The lock guarantees two instances don't run &lt;em&gt;at the same time&lt;/em&gt;. It doesn't stop the job running twice in a row: if server A finishes at 02:00:03 and server B's scheduler fires late at 02:00:05, B will get the lock and run again. For jobs that must run exactly once per period, combine the lock with a record of what's been done, for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;job_runs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;period&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'invoices'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'2026-11-03'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;CONFLICT&lt;/span&gt; &lt;span class="k"&gt;DO&lt;/span&gt; &lt;span class="k"&gt;NOTHING&lt;/span&gt;
&lt;span class="k"&gt;RETURNING&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If no row comes back, this period is already done; skip it. With a unique constraint on &lt;code&gt;(job, period)&lt;/code&gt;, this is safe even without the lock, and with it, you also avoid wasted parallel work.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Choose keys deliberately.&lt;/strong&gt; Keys are a single 64-bit integer or two 32-bit integers. Keep a small registry in code (constants with clear names) so two different jobs don't accidentally share a key. Some teams derive keys by hashing the job name, which works, as long as everyone uses the same hash.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Make the skip visible.&lt;/strong&gt; Log when an instance skips because another holds the lock. If &lt;em&gt;every&lt;/em&gt; instance logs "skipping" and the job never runs, something is holding the lock that shouldn't be, and you can find it in &lt;code&gt;pg_locks&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;granted&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;objid&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_locks&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;locktype&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'advisory'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  When to reach for something else
&lt;/h2&gt;

&lt;p&gt;Advisory locks are a great fit when you have a handful of app servers, one Postgres primary, and jobs that run on a schedule. They're less suitable when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your jobs run across several databases or regions with no single shared primary.&lt;/li&gt;
&lt;li&gt;You need queues, retries, delays and visibility into each run, not just mutual exclusion.&lt;/li&gt;
&lt;li&gt;The job needs to keep running through a database failover. A lock held on the old primary doesn't carry over.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At that point you're building a job system, and it may be time for one, whether that's a library backed by your database or a hosted scheduler like &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; that runs each job once at the time you asked.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;pg_try_advisory_lock(key)&lt;/code&gt;: skip if someone else is running.&lt;/li&gt;
&lt;li&gt;Use a dedicated connection, and watch out for transaction-mode poolers.&lt;/li&gt;
&lt;li&gt;Add a &lt;code&gt;job_runs&lt;/code&gt; table with a unique constraint if "once per period" matters.&lt;/li&gt;
&lt;li&gt;Log skips so you notice when nothing runs at all.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Four lines of SQL, no new infrastructure, and no more double invoices.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-hypothetical-the-subscription-renewals-that-drifted-from-the-31st-to-the-28th-and-4loj"&gt;Case study (hypothetical): the subscription renewals that drifted from the 31st to the 28th, and stayed there&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/exactly-once-scheduled-jobs-dont-exist-design-for-at-least-once-and-stop-double-charging-2ki9"&gt;Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/make-scheduled-job-handlers-idempotent-safe-to-run-twice-499o"&gt;Make scheduled job handlers idempotent: safe to run twice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/what-happens-to-the-cron-job-thats-running-when-you-deploy-graceful-shutdown-for-scheduled-work-2l2b"&gt;What happens to the cron job that's running when you deploy? Graceful shutdown for scheduled work&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How do you stop scheduled jobs from running twice in your stack: database locks, a leader, or a separate scheduler?&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>cron</category>
      <category>backend</category>
      <category>devops</category>
    </item>
    <item>
      <title>Case study (hypothetical): the subscription renewals that drifted from the 31st to the 28th, and stayed there</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Wed, 11 Nov 2026 16:16:36 +0000</pubDate>
      <link>https://dev.to/delayrun/case-study-hypothetical-the-subscription-renewals-that-drifted-from-the-31st-to-the-28th-and-4loj</link>
      <guid>https://dev.to/delayrun/case-study-hypothetical-the-subscription-renewals-that-drifted-from-the-31st-to-the-28th-and-4loj</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a hypothetical case study. The company, numbers and timeline are invented to illustrate a real class of bug. If you've built anything that repeats monthly, you may recognise it anyway.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;A small SaaS company sells a monthly plan. When a customer subscribes, the system stores a &lt;code&gt;next_renewal_at&lt;/code&gt; timestamp. Every hour, a job looks for subscriptions whose &lt;code&gt;next_renewal_at&lt;/code&gt; is in the past, charges them, and sets the next renewal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nextRenewalAt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;addMonths&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nextRenewalAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;addMonths&lt;/code&gt; comes from a popular date library. It does the sensible thing when the target month is shorter: it clamps to the last day of that month. January 31 plus one month is February 28 (or 29).&lt;/p&gt;

&lt;p&gt;Tests pass. Renewals go out. For about a year, nobody thinks about it again.&lt;/p&gt;

&lt;h2&gt;
  
  
  The symptom
&lt;/h2&gt;

&lt;p&gt;The first sign isn't a bug report. It's a support ticket with a mild tone: "I signed up on the 31st, why am I now always charged on the 28th?"&lt;/p&gt;

&lt;p&gt;Then a few more, mostly from customers who'd signed up on the 29th, 30th or 31st. One says their company's expense process expects the charge at the end of the month and now it lands before their budget resets. Another says the charge is "random". It isn't random, but from the outside it looks it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually happened
&lt;/h2&gt;

&lt;p&gt;Walk through one subscription that started on January 31:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Renewal&lt;/th&gt;
&lt;th&gt;Computed from&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Jan 31 + 1 month&lt;/td&gt;
&lt;td&gt;Feb 28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Feb 28 + 1 month&lt;/td&gt;
&lt;td&gt;Mar 28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Mar 28 + 1 month&lt;/td&gt;
&lt;td&gt;Apr 28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Apr 28 + 1 month&lt;/td&gt;
&lt;td&gt;May 28&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The clamp in February was correct. The problem is what came after: each renewal was computed from the &lt;em&gt;previous renewal&lt;/em&gt;, not from the original signup date. Once a date has been clamped to the 28th, the information that the customer "really" renews on the 31st is gone. Every later month inherits the clamp.&lt;/p&gt;

&lt;p&gt;A customer who started on March 31 drifts at the first short month, April, to the 30th, then stays on the 30th, then drops to the 28th the following February and stays there for good.&lt;/p&gt;

&lt;p&gt;In other words, the system had a one-way ratchet. Billing days could move earlier, never later.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A calendar quirk is easier to catch when every renewal attempt is logged.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; runs the HTTP job on the schedule you set and records each attempt, so a drift shows up in the history.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why nobody caught it
&lt;/h2&gt;

&lt;p&gt;A few reasons, all ordinary:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tests checked one step.&lt;/strong&gt; The test suite verified that Jan 31 + 1 month = Feb 28. It never checked the twelfth renewal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Most customers were unaffected.&lt;/strong&gt; Anyone who signed up between the 1st and the 28th never drifts. Only customers on the last three days of the month are exposed, and only after a short month.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The data looked fine.&lt;/strong&gt; Every individual &lt;code&gt;next_renewal_at&lt;/code&gt; was a valid date a month after the last one. No row was "wrong" on its own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Support tickets didn't mention dates precisely.&lt;/strong&gt; "Always charged early" doesn't sound like a date arithmetic bug until someone lines up the history.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The fix: store the anchor, compute from it
&lt;/h2&gt;

&lt;p&gt;The fix is a change of model, not of library. Instead of storing only "when is the next renewal", the system stores &lt;strong&gt;the billing anchor&lt;/strong&gt;: the original day of month (and time, and time zone) the subscription is meant to renew on. Every renewal is computed from the anchor and a counter, not from the previous renewal.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// anchor: { year: 2026, month: 0, day: 31 }  (Jan 31)&lt;/span&gt;
&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;renewalDate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;month&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;month&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;lastDay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UTC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCDate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UTC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;day&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lastDay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the sequence for a January 31 anchor is Feb 28, Mar 31, Apr 30, May 31. The clamp happens only in the months that need it, and the customer returns to the 31st whenever the month allows.&lt;/p&gt;

&lt;p&gt;(In a real system you'd also handle the anchor's time of day and time zone; this snippet uses UTC days to keep the idea visible.)&lt;/p&gt;

&lt;h2&gt;
  
  
  The migration nobody enjoys
&lt;/h2&gt;

&lt;p&gt;Fixing future renewals was easy. Repairing existing subscriptions was the interesting part, because the original anchor had been overwritten for anyone who'd drifted.&lt;/p&gt;

&lt;p&gt;In this hypothetical, the team:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Recovered anchors from history.&lt;/strong&gt; The first successful charge or the subscription's creation date gave the original day of month for almost everyone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decided not to "jump" anyone suddenly.&lt;/strong&gt; Moving a customer from the 28th back to the 31st means one billing period is three days longer. The team chose to apply the corrected anchor at the &lt;em&gt;next&lt;/em&gt; renewal and to let the slightly longer period stand without extra charge, rather than charging a prorated amount.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Emailed affected customers&lt;/strong&gt; with a short, plain note: what had happened, what would change, and that nobody would pay more.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Added a property test&lt;/strong&gt; that generates random anchors and checks 36 consecutive renewals against an independent calculation.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Lessons that generalise beyond billing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Recurring schedules need an anchor.&lt;/strong&gt; Whether it's billing, reports, reminders or payroll, "every month on the 31st" is a rule. "The next one is on the 28th" is just one result of that rule. Store the rule, and derive the results.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Never compute the nth occurrence from the (n-1)th&lt;/strong&gt; when any step can lose information. Clamping, rounding, time zone shifts and daylight saving changes all lose information.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Test sequences, not steps.&lt;/strong&gt; Date bugs often hide in the composition of individually correct operations. Generate a year or three of occurrences and compare them with expectations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Decide the business rule explicitly.&lt;/strong&gt; "Last day of the month if the day doesn't exist" is one reasonable rule. "First day of the following month" is another. Some businesses avoid the problem by only allowing anchors from the 1st to the 28th. Any of these is fine, as long as it's chosen on purpose and written down.&lt;/p&gt;

&lt;p&gt;Getting recurring "later" right, so that the twelfth occurrence is as correct as the first, is the kind of problem we think about all day at &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/remind-me-next-tuesday-at-3-6-libraries-for-parsing-natural-language-dates-and-the-traps-in-each-4744"&gt;'Remind me next Tuesday at 3': 6 libraries for parsing natural-language dates, and the traps in each&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-job-timezone-problems-how-to-run-cron-in-a-specific-time-zone-and-survive-dst-23f"&gt;Cron job timezone problems: how to run cron in a specific time zone (and survive DST)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/case-study-redesigning-a-failed-payment-retry-schedule-that-retried-every-card-at-0000-utc-2c9k"&gt;Case study: redesigning a failed-payment retry schedule that retried every card at 00:00 UTC&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/quick-tip-one-postgres-advisory-lock-stops-your-scheduled-job-running-twice-12j7"&gt;Quick tip: one Postgres advisory lock stops your scheduled job running twice&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How does your code compute the next occurrence of something monthly: from the last one, or from an anchor?&lt;/p&gt;

</description>
      <category>datetime</category>
      <category>javascript</category>
      <category>billing</category>
      <category>architecture</category>
    </item>
    <item>
      <title>'Remind me next Tuesday at 3': 6 libraries for parsing natural-language dates, and the traps in each</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Tue, 10 Nov 2026 09:48:30 +0000</pubDate>
      <link>https://dev.to/delayrun/remind-me-next-tuesday-at-3-6-libraries-for-parsing-natural-language-dates-and-the-traps-in-each-4744</link>
      <guid>https://dev.to/delayrun/remind-me-next-tuesday-at-3-6-libraries-for-parsing-natural-language-dates-and-the-traps-in-each-4744</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If your product schedules anything, sooner or later someone asks for a text box where people can just type "tomorrow at 9am" or "in 2 hours" or "next Tuesday at 3". It's a great experience when it works. It's a confusing one when it doesn't, because a wrong guess looks exactly like a right one until the reminder fires on the wrong day.&lt;/p&gt;

&lt;p&gt;Here are six libraries that turn human-written dates into real timestamps, grouped by language, with the traps worth testing for each. Libraries change, so check each project's current documentation and maintenance status before you adopt one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The traps they all share
&lt;/h2&gt;

&lt;p&gt;Before the list, the problems every parser has to deal with. Test any library against these:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Next Friday."&lt;/strong&gt; On a Wednesday, is that in two days or nine? People genuinely disagree.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Day/month order.&lt;/strong&gt; "03/04" is March 4 in some countries and April 3 in others.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time zones.&lt;/strong&gt; "9am" in whose zone? The server's, the user's browser, or the user's saved setting?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Times in the past.&lt;/strong&gt; "At 9am", typed at 10am. Today, which has passed, or tomorrow?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ambiguous words.&lt;/strong&gt; "This weekend", "end of the month", "tonight".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Daylight saving.&lt;/strong&gt; "In 24 hours" and "tomorrow at the same time" differ on clock-change days.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The library is only half the solution. The other half is showing the user what you understood.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Chrono (JavaScript / TypeScript)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What it is:&lt;/strong&gt; A popular natural-language date parser for JavaScript, usable in Node and the browser.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Good for:&lt;/strong&gt; Extracting dates from free text: "Let's meet next Tuesday at 3pm" gives you the date and the position of the matched text. It supports a &lt;strong&gt;reference date&lt;/strong&gt;, so "tomorrow" is calculated relative to the moment you choose, and has options for preferring future dates.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch out for:&lt;/strong&gt; Language support varies by locale, with English the most complete. Make sure you pass the user's time zone and reference time explicitly rather than relying on the server's.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;chrono&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"chrono-node"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;// Pass the reference moment explicitly; check the docs for how your&lt;/span&gt;
&lt;span class="c1"&gt;// version accepts the user's time zone in the reference object.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;instant&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;chrono&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parseDate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"next Tuesday at 3pm"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;forwardDate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  2. dateparser (Python)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What it is:&lt;/strong&gt; A Python library that parses dates in many languages and formats, including relative expressions like "2 hours ago" or "in 3 days".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Good for:&lt;/strong&gt; Multilingual input and messy formats, for example from scraped pages or user messages. It has settings for preferring future or past dates, choosing date order and setting the time zone.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch out for:&lt;/strong&gt; It's flexible, which means it will confidently parse things you might want to reject. Configure &lt;code&gt;PREFER_DATES_FROM&lt;/code&gt;, &lt;code&gt;DATE_ORDER&lt;/code&gt; and the time zone settings deliberately, and test with your users' real phrases.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dateparser&lt;/span&gt;
&lt;span class="n"&gt;dateparser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"next tuesday 3pm"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;settings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"PREFER_DATES_FROM"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"future"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"TIMEZONE"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Europe/Lisbon"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"RETURN_AS_TIMEZONE_AWARE"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  3. parsedatetime (Python)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What it is:&lt;/strong&gt; An older, smaller Python library focused on English natural-language dates and times.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Good for:&lt;/strong&gt; Simple English phrases where you want something lightweight.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch out for:&lt;/strong&gt; It returns a time structure plus a status flag telling you whether it parsed a date, a time, both or neither. Check that flag; otherwise "banana" quietly becomes "now".&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Natural-language parsing is the easy part; firing the reminder is the part that has to survive a restart.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; holds that delayed HTTP callback, retries it, and logs the run.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  4. Duckling (Haskell, run as a service)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What it is:&lt;/strong&gt; A parser originally from Facebook that extracts structured data, including dates, times, durations and intervals, from text in many languages. It's typically run as an HTTP service.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Good for:&lt;/strong&gt; Chat interfaces and voice assistants, where you need dates, durations ("for 2 hours") and intervals ("from 3 to 5") in many languages, from any programming language.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch out for:&lt;/strong&gt; It's a separate service to deploy and maintain, and the repository's activity has varied over time. Check its current state before depending on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Chronic (Ruby)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What it is:&lt;/strong&gt; A long-standing natural-language date parser for Ruby.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Good for:&lt;/strong&gt; Ruby apps that need English phrases like "tomorrow", "3 months from now" or "last day of the month".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch out for:&lt;/strong&gt; Time zone handling depends on how you configure it with your app's time settings. Test with your app's real time zone configuration, not just the defaults.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Natty (Java)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What it is:&lt;/strong&gt; A natural-language date parser for Java.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Good for:&lt;/strong&gt; JVM applications that need English relative dates and date ranges.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch out for:&lt;/strong&gt; The project hasn't seen much recent activity, so treat it as stable but not actively evolving, and check compatibility with your Java version.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern that matters more than the library
&lt;/h2&gt;

&lt;p&gt;Whichever parser you choose, the most important step is the same: &lt;strong&gt;show the user what you understood before saving it.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;You typed: "next tuesday at 3"
We'll remind you: Tuesday 10 November, 15:00 (Lisbon time)   [Change]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That one line catches nearly every ambiguity on the list above. The user sees the actual date and time zone, and corrects it if it's wrong, before anything is scheduled.&lt;/p&gt;

&lt;p&gt;A few more habits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Store the parsed result in UTC, plus the user's time zone,&lt;/strong&gt; not the original text.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep the original text too,&lt;/strong&gt; for debugging and support.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reject or question low-confidence results&lt;/strong&gt; instead of guessing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write tests from real user input.&lt;/strong&gt; Collect the phrases people actually type; they're stranger than you'd expect.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Turning "next Tuesday at 3" into a job that runs at exactly the right moment is the kind of problem we think about at &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/can-nodejs-hold-a-million-settimeouts-measure-memory-accuracy-and-the-restart-problem-yourself-3jn9"&gt;Can Node.js hold a million setTimeouts? Measure memory, accuracy and the restart problem yourself&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-job-timezone-problems-how-to-run-cron-in-a-specific-time-zone-and-survive-dst-23f"&gt;Cron job timezone problems: how to run cron in a specific time zone (and survive DST)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/how-to-schedule-an-http-request-call-a-url-or-webhook-on-a-schedule-4j86"&gt;How to schedule an HTTP request (call a URL or webhook on a schedule)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/cron-expression-for-every-monday-at-9am-and-other-weekly-schedules-2anf"&gt;Cron expression for every Monday at 9am (and other weekly schedules)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the most ambiguous date phrase a user has typed into your product, and what did your code decide it meant?&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>python</category>
      <category>datetime</category>
      <category>tools</category>
    </item>
    <item>
      <title>Can Node.js hold a million setTimeouts? Measure memory, accuracy and the restart problem yourself</title>
      <dc:creator>delay.run</dc:creator>
      <pubDate>Tue, 10 Nov 2026 09:45:50 +0000</pubDate>
      <link>https://dev.to/delayrun/can-nodejs-hold-a-million-settimeouts-measure-memory-accuracy-and-the-restart-problem-yourself-3jn9</link>
      <guid>https://dev.to/delayrun/can-nodejs-hold-a-million-settimeouts-measure-memory-accuracy-and-the-restart-problem-yourself-3jn9</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;At some point every Node.js developer writes this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nb"&gt;setTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;sendReminder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;delayMs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It works. It's one line. And sooner or later someone asks: "how many of these can we have in flight at once?" The honest answer is "it depends on your machine," so rather than quoting numbers, this post gives you a small benchmark to produce your own, and explains how to read the results.&lt;/p&gt;

&lt;p&gt;I won't share figures from my laptop. They'd be meaningless for your server, Node version and workload. Run it yourself, it takes a few minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we're measuring
&lt;/h2&gt;

&lt;p&gt;Three things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Memory per pending timer.&lt;/strong&gt; How much heap does each scheduled callback cost?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lateness.&lt;/strong&gt; How far after the intended time does each callback actually fire, and how does that change as the number of timers grows?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Event-loop impact.&lt;/strong&gt; Does having lots of pending timers slow down other work?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And one thing we can't measure but must discuss: what happens to all of those timers when the process restarts.&lt;/p&gt;

&lt;h2&gt;
  
  
  The benchmark script
&lt;/h2&gt;

&lt;p&gt;Save this as &lt;code&gt;timers-bench.js&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// node --expose-gc timers-bench.js 100000&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;N&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="mi"&gt;100000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;SPREAD_MS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;30_000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// timers fire over the next 30 seconds&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;monitorEventLoopDelay&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"node:perf_hooks"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;heapMB&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gc&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;gc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;memoryUsage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;heapUsed&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;before&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;heapMB&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;lateness&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Float64Array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;N&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;fired&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;N&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;SPREAD_MS&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;due&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`user-${i}`&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// a realistic small closure&lt;/span&gt;
  &lt;span class="nb"&gt;setTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;lateness&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;fired&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;due&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"never"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fired&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="no"&gt;N&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;after&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;heapMB&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`timers: ${N}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`heap before: ${before.toFixed(1)} MB, after scheduling: ${after.toFixed(1)} MB`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`approx bytes per timer: ${(((after - before) * 1024 * 1024) / N).toFixed(0)}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;loop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;monitorEventLoopDelay&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;resolution&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;pct&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;arr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;arr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;disable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`lateness p50: ${pct(lateness, 50).toFixed(1)} ms`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`lateness p99: ${pct(lateness, 99).toFixed(1)} ms`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`lateness max: ${lateness.reduce((m, x) =&amp;gt; Math.max(m, x), 0).toFixed(1)} ms`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`event loop delay p99: ${(loop.percentile(99) / 1e6).toFixed(1)} ms`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it at a few sizes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; n &lt;span class="k"&gt;in&lt;/span&gt; 10000 100000 1000000&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; node --expose-gc timers-bench.js &lt;span class="nv"&gt;$n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--expose-gc&lt;/code&gt; lets the script force garbage collection before measuring, so the heap numbers are less noisy. They're still approximate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making it realistic
&lt;/h2&gt;

&lt;p&gt;The bare benchmark measures timers in a quiet process. Your real process is doing other things. Two variations worth trying:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Add background load.&lt;/strong&gt; Simulate request handling with some CPU work on an interval:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nb"&gt;setInterval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// 5ms of busy work&lt;/span&gt;
  &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now lateness reflects how timers behave when the event loop is busy, which is the situation you'll actually be in.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Make the payloads realistic.&lt;/strong&gt; If your real callbacks close over a user object, a template and a database connection reference, the memory cost per timer goes up. Put in a payload shaped like your real one.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Even if the process can hold the timers, a restart drops them all.&lt;/strong&gt; &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt; stores the delayed HTTP jobs outside the process, retries them, and survives a restart.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  How to read the results
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Memory.&lt;/strong&gt; Multiply bytes-per-timer by the number of pending jobs you expect at your busiest. Then remember that this is per process. If you run four instances, each may hold its own copy of timers it created.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lateness p50 vs p99.&lt;/strong&gt; Median lateness for timers is usually small. The tail is what matters. If your p99 jumps sharply when you add background load, timers are competing with your real work, and anything time-sensitive will feel it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Event-loop delay.&lt;/strong&gt; If the event-loop delay rises as N grows, the timers themselves are costing you responsiveness. Firing many callbacks close together, at the same scheduled moment for example, is a common cause.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Long delays.&lt;/strong&gt; Node's &lt;code&gt;setTimeout&lt;/code&gt; has a maximum delay of about 24.8 days (2³¹−1 milliseconds). Longer values don't wait longer: Node warns and fires the callback after roughly 1 ms instead. Try it with a big delay and see. If you're scheduling anything weeks out, timers are the wrong tool regardless of what the benchmark says.&lt;/p&gt;

&lt;h2&gt;
  
  
  The result no benchmark shows
&lt;/h2&gt;

&lt;p&gt;Here's the important part. Suppose the benchmark tells you a million timers fit comfortably in memory and fire on time. Then you deploy.&lt;/p&gt;

&lt;p&gt;During a deploy, the process restarts. &lt;strong&gt;Every pending timer is gone.&lt;/strong&gt; There's no error and no log line. Reminders that were due in the next hour simply never happen. The same applies to crashes, out-of-memory kills, autoscaling removing an instance, and a host rebooting for patches.&lt;/p&gt;

&lt;p&gt;That's why the question "how many timers can Node hold?" is usually the wrong one. The more useful question is "what happens to pending work when this process dies?" For in-process timers the answer is always "it's lost", whatever the count.&lt;/p&gt;

&lt;p&gt;The usual fix is to store the job, with its due time, somewhere durable: a database table, a queue with delayed delivery, or a scheduling service. A worker picks up due jobs and runs them. In-process timers can still be useful for short-lived, losable work: debouncing, retrying a request a few seconds later, or waking a worker up early. They're a poor home for anything a customer is waiting on.&lt;/p&gt;

&lt;h2&gt;
  
  
  A fair comparison to run next
&lt;/h2&gt;

&lt;p&gt;If you want to compare, try the same experiment with jobs stored in a table and a worker polling for due jobs every second. You'll typically see:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Lateness&lt;/strong&gt; roughly bounded by the polling interval, plus query time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory&lt;/strong&gt; that no longer grows with pending jobs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Restarts&lt;/strong&gt; that lose nothing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whether that trade-off is worth it depends on whether a lost job matters. For reminders, invoices and anything with a promise attached, it usually does.&lt;/p&gt;

&lt;p&gt;Durable "run this later" is the problem we work on at &lt;a href="https://delay.run" rel="noopener noreferrer"&gt;delay.run&lt;/a&gt;, but the experiment above is worth running whatever you end up using.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/dont-use-settimeout-for-anything-that-matters-where-delayed-jobs-should-actually-live-4o26"&gt;Don't use setTimeout for anything that matters: where delayed jobs should actually live&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/how-late-is-every-minute-an-experiment-you-can-run-on-cron-systemd-timers-and-sleep-loops-3b1a"&gt;How late is 'every minute'? An experiment you can run on cron, systemd timers and sleep loops&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/fixed-rate-fixed-delay-or-cron-three-ways-to-say-every-5-minutes-and-they-behave-very-5340"&gt;Fixed rate, fixed delay or cron? Three ways to say 'every 5 minutes', and they behave very differently&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/delayrun/remind-me-next-tuesday-at-3-6-libraries-for-parsing-natural-language-dates-and-the-traps-in-each-4744"&gt;'Remind me next Tuesday at 3': 6 libraries for parsing natural-language dates, and the traps in each&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How many pending in-process timers does your busiest service hold right now, and do you know what happens to them on deploy?&lt;/p&gt;

</description>
      <category>node</category>
      <category>javascript</category>
      <category>performance</category>
      <category>backend</category>
    </item>
  </channel>
</rss>
