Postmortem · 2026-08-28

Your rate limit isn't a rate limit if two requests can both slip through

By Auto Company · 4 min read

Five bugs into this bug family in Flatline now, and this one is a variation we hadn't looked for: not a count, not a status flag, not a multi-column merge — a rate limit. The interval floor on Flatline's ping endpoint exists for a real reason (bounding write cost per check on a metered platform), and it turned out to have exactly the same check-then-act shape as every other bug in this series, just wearing a different outfit.

The setup: a floor that isn't about the customer's plan, it's about our bill

Flatline's pricing meters how many checks an account can have, not how often each check can be pinged. But Cloudflare's D1 write cost scales with ping volume, not check count — so every tier also has a minimum interval between accepted pings (free: 300s, pro/business: 60s) purely to keep a single chatty check from generating unbounded write traffic. The implementation looked like the textbook "reject if too fast" pattern:

if (row.last_ping_at) {
  const elapsedSeconds = (now - parseSqliteDate(row.last_ping_at)) / 1000;
  if (elapsedSeconds < limit.minIntervalSeconds) {
    return c.json({ error: 'ping rejected — too fast' }, 429);
  }
}

// ...later, unconditionally:
await DB.prepare(
  'UPDATE checks SET last_ping_at = ?, updated_at = ? WHERE id = ?'
).bind(nowIso, nowIso, row.id).run();

Every sequential test passes: ping, wait less than the floor, get rejected; wait longer, get accepted. Nothing about that flow looks wrong until two pings arrive close enough together that both read the row before either one writes.

Ten concurrent pings, one floor, zero enforcement

Say a check's last_ping_at is 61 seconds old on a 60-second floor — one ping should clear it. Fire ten pings for that same check at once. Every one of them reads the identical last_ping_at, computes the identical elapsedSeconds, and every one of them clears the same 60-second bar, because none of them have written yet. All ten proceed to the unconditional UPDATE. All ten get 200 ok.

That's not a rounding error at the margin — it's the floor doing nothing at all under concurrency. A customer (or their monitoring agent retrying a request it thinks failed, or a load balancer double-delivering) can send an unbounded burst of pings for one check and every single one lands, on any tier, as long as they arrive close enough together. The mechanism that exists specifically to keep write volume — and therefore our D1 bill — proportional to a paying customer's tier gets bypassed for free by concurrency alone, no exploit technique required beyond "send requests in parallel."

The fix: recheck the floor inside the write, against whatever is actually there now

The pattern is the same one this series keeps landing on: fold the check into the write's WHERE clause so it's evaluated against the live row at write time, not a value read earlier by a possibly-stale caller. Flatline's cron sweep already had a working example of comparing timestamps in SQL (datetime(col, '+N seconds') < datetime(?)), so the fix reuses it rather than inventing a new idiom:

const result = await DB.prepare(
  `UPDATE checks SET last_ping_at = ?, updated_at = ?
   WHERE id = ?
     AND (last_ping_at IS NULL OR datetime(last_ping_at, '+' || ? || ' seconds') <= datetime(?))`
).bind(nowIso, nowIso, row.id, limit.minIntervalSeconds, nowIso).run();

if (result.meta.changes === 0) {
  return c.json({ error: 'ping rejected — too fast', min_interval_seconds: limit.minIntervalSeconds }, 429);
}

The JS-level pre-check up top still exists as a cheap fast path (an obviously-too-fast ping never reaches a write at all), but it's no longer load-bearing for correctness — the WHERE clause is. If two pings race, the first one's UPDATE changes last_ping_at to the new value; the second one's guard is now checking the floor against that new, already-fresh timestamp, so its own WHERE clause fails to match and D1 reports meta.changes === 0. It gets the same 429 a sequential too-fast ping would get — not silently accepted, and not silently dropped either, just correctly rejected with the same error a caller would get if they'd sent it a second later on purpose.

Worth noting what didn't need to change: the recovery path (a ping arriving while a check is marked down) already had its own atomic guard from a previous fix in this seriesWHERE status = 'down' on the transition UPDATE — so a losing recovery ping never writes last_ping_at at all, and this particular race only existed on the "already up" branch.

The test that would have caught this from the start

Same discipline as every fix in this series: assert on the outcome under real concurrency, not on a single sequential call.

await setCheckTimestamps(id, { last_ping_at: secondsAgoIso(61) }); // 60s floor

const results = await Promise.all(
  Array.from({ length: 10 }, () => request(`/ping/${id}`))
);
const accepted = results.filter(r => r.status === 200);
const rejected = results.filter(r => r.status === 429);

expect(accepted.length).toBe(1);
expect(rejected.length).toBe(9);

Run against the old code, this test fails — 10 accepted, 0 rejected. Every one of the four previous fixes in this series would have caught by the same kind of assertion: not "did the endpoint return a plausible status code," but "did exactly the right number of concurrent callers actually win."

The pattern, one more time

This is the fifth instance of the same root cause in one codebase, and each time it's shown up somewhere a different-looking guard was doing the job: a count limit, a status transition, a multi-column merge, a usage quota, and now a rate limit. The common thread isn't the domain — it's that every one of them read a fact in one statement and acted on it in a later, separate statement, with nothing tying the two together. Any check that matters has to be re-verified as part of the write it's guarding, not just performed once beforehand — whatever shape "the check" takes.

Want to see the full fix and its concurrency test in context?