Event Sourcing for Webhook Systems

What Is Event Sourcing?

Instead of storing "current state" (last delivery status), we store every event that happened:

• Webhook created
• Delivery attempt 1 failed (503)
• Delivery attempt 2 failed (timeout)
• Delivery attempt 3 succeeded (200)

This gives us a complete audit trail and makes debugging failures trivial.

Benefits for Webhooks

Schema Design

CREATE TABLE webhook_events (
    event_id     TEXT PRIMARY KEY,  -- UUID, immutable
    webhook_id   TEXT NOT NULL,     -- Groups events for one delivery
    event_type   TEXT NOT NULL,     -- "order.created", etc.
    status       TEXT NOT NULL,     -- pending/attempting/delivered/failed
    target_url   TEXT NOT NULL,     -- Where it was sent
    attempt_num  INTEGER,           -- Which retry attempt
    status_code  INTEGER,           -- HTTP response code
    error_msg    TEXT,              -- Error details
    created_at   REAL NOT NULL      -- Unix timestamp
);

Key principle: APPEND ONLY

Never update rows. Never delete rows. Only insert new events. To find the current state, query the latest event for a given webhook_id.

See src/event_store.py for the SQLite implementation.

Part of API Developer Pro. Support: support@datanest.dev

Retry Strategies and Idempotency for Webhooks

Why Retries Matter

Webhooks fail. Networks are unreliable, servers restart, and databases go down. A webhook system without retry logic loses events silently. Industry benchmarks show that even well-maintained endpoints fail 1-5% of the time.

Exponential Backoff with Full Jitter

The gold standard for retry timing. Used by AWS, Stripe, and Google.

Attempt 1: immediate
Attempt 2: random(0, 1s)      = ~0.5s average
Attempt 3: random(0, 2s)      = ~1s average
Attempt 4: random(0, 4s)      = ~2s average
Attempt 5: random(0, 8s)      = ~4s average
Attempt 6: random(0, 16s)     = ~8s average
...capped at max_delay (e.g., 5 minutes)

Why jitter? Preventing thundering herd.

Without jitter, if 1000 webhooks fail at the same time, all 1000 retry at the same time. With jitter, retries are spread randomly across the backoff window.

Idempotency Keys

An idempotency key ensures that processing a webhook twice has the same effect as processing it once. This is critical because retries mean the receiver might get the same webhook multiple times.

Sender side:

{
  "webhook_id": "wh-abc-123",
  "idempotency_key": "wh-abc-123-1700000000",
  "event_type": "payment.completed",
  "data": { ... }
}

Receiver side:

def handle_webhook(payload):
    key = payload["idempotency_key"]
    if key in processed_keys:
        return {"status": "already_processed"}  # 200 OK
    
    process_event(payload)
    processed_keys.add(key)
    return {"status": "processed"}  # 200 OK

Dead Letter Queues

After exhausting all retries, webhooks go to a "dead letter" state. Options:

1. Store for manual review -- admin dashboard shows failed deliveries

2. Alert the operator -- email/Slack notification for dead letters

3. Retry later -- scheduled job retries dead letters once per hour

See src/delivery.py for the complete implementation.

Part of API Developer Pro. Support: support@datanest.dev