Your recovery webhook can fire twice for one incident

A heartbeat monitor's whole job is: notice a check went silent, and notice when it comes back. The "comes back" half looks trivial to implement — the check pings, you see its status was down, you flip it to up, you fire the recovery webhook. Except the code that does this in one obvious pass has the exact same shape as a bug we'd already found and fixed twice before, just wearing a different outfit.

We found this in Flatline's ping-ingestion path — the route that flips a check from down back to up and pages the customer's on-call system to say the incident is over. The realistic trigger isn't exotic timing or an attacker: it's a monitored service doing exactly what monitored services do — retrying an HTTP call it thinks timed out. Send the same recovery ping twice within a few hundred milliseconds, which is routine client behavior, and the customer got paged twice for one incident.

Why "read status, then write" isn't atomic — even for a single row

The first two instances of this bug class we found were about aggregate quotas: SELECT COUNT(*) then a separate INSERT, which let concurrent requests blow past a per-tier limit (see the tier-limits postmortem). It's tempting to think a single-row status check is immune — there's no counting involved, just "is this row currently down?" But the shape of the bug isn't about counting, it's about the gap between a read and a write:

const row = await DB.prepare(
  'SELECT * FROM checks WHERE id = ?'
).bind(id).first();

if (row.status === 'down') {
  // recovery: flip to up, write a check_events row, fire the webhook
  await applyStatusChange(DB, row, 'up', 'down', nowIso);
}

Two recovery pings arriving close together both run the SELECT before either has committed its write. Both read status: 'down'. Both pass the if. Both proceed to write a check_events row and fire the webhook — for what the system will show as two separate transitions, even though only one thing actually happened. The number of rows involved (one) doesn't matter; what matters is that "check the current state" and "act on it" are two round-trips with daylight between them, and nothing stops a second caller from reading the state before the first caller's write lands.

The fix: make the write itself the check

Same principle as the quota fix: fold the read into the write's WHERE clause, so the database — not application code holding a stale snapshot — decides who wins.

const updateStmt = toStatus === 'up'
  ? DB.prepare(
      `UPDATE checks SET status = 'up', last_ping_at = ?, last_state_change_at = ?, updated_at = ?
       WHERE id = ? AND status = ?`
    ).bind(nowIso, nowIso, nowIso, check.id, fromStatus)
  : DB.prepare(
      `UPDATE checks SET status = 'down', last_state_change_at = ?, updated_at = ?
       WHERE id = ? AND status = ?`
    ).bind(nowIso, nowIso, check.id, fromStatus);

const result = await updateStmt.run();
if (result.meta.changes === 0) {
  // Lost the race — someone else already made this transition. Don't
  // write a check_events row or fire the webhook for it.
  return false;
}

// only the winner reaches here: write check_events, fire the webhook

The WHERE id = ? AND status = ? guard means the UPDATE only matches a row if it's still in the state this caller believes it's in. If a concurrent caller already flipped it, the guard fails, meta.changes comes back 0, and this caller quietly stops — no event row, no webhook, no double page. Exactly one of the two concurrent callers gets to see changes: 1 and proceed. The event insert and webhook delivery, which used to be batched unconditionally with the update, now only run after confirming the update actually won — which costs one extra round-trip on the rare recovery path, in exchange for never double-firing on it.

The test that actually catches it

Same lesson as before: a test that pings once and checks the response proves nothing about a race. The test has to create real concurrency and assert on the aggregate outcome:

await DB.prepare("UPDATE checks SET status = 'down' WHERE id = ?").bind(id).run();

// Fire 10 concurrent recovery pings for the same check.
const results = await Promise.all(
  Array.from({ length: 10 }, () => request(`/ping/${id}`))
);
expect(results.every(res => res.status === 200)).toBe(true);

const eventCount = await DB.prepare(
  'SELECT COUNT(*) as cnt FROM check_events WHERE check_id = ?'
).bind(id).first();
expect(eventCount.cnt).toBe(1);   // not 10

const deliveryCount = await DB.prepare(
  'SELECT COUNT(*) as cnt FROM webhook_deliveries WHERE check_id = ?'
).bind(id).first();
expect(deliveryCount.cnt).toBe(1);   // not 10

All 10 requests still return 200 ok — from the caller's point of view the check is up either way, and that's correct. The thing the test actually pins down is that only one of those ten calls gets credit for the transition: one check_events row, one webhook delivery. Run it against the unconditional-batch version and it fails immediately with 10 of each. Run it against the guarded version and it's stable every time.

Three instances of one bug, and the actual rule

This is the third time we've found this exact bug shape in the same codebase: a per-tier check count before an insert, an IP-based rate-limit count before an insert, and now a status read before a status-dependent write. None of them involve exotic timing — every one of them is triggered by ordinary, expected client behavior (concurrent signups, a burst of API calls, a service retrying a call it thinks failed). The common rule isn't "count rows atomically" — it's broader: any time a write's correctness depends on a fact your code read in an earlier, separate statement, encode that fact as a condition on the write itself. For quotas, that's INSERT ... SELECT ... WHERE count < limit. For a state machine transition, that's UPDATE ... WHERE current_state = expected_state. Different SQL, same principle: let the database's atomicity guarantee — one statement, all or nothing — do the work your application code can't do across two round-trips.