Designing Webhook Integrations That Survive Production

Webhook integrations look simple in a diagram: one system sends an HTTP request and another system handles it. Production adds the difficult parts—duplicate delivery, missing signatures, timeouts, reordered events, expired secrets, and downstream services that are temporarily unavailable.

The reliable design is not a clever controller. It is a small ingestion boundary followed by a durable processing pipeline.

Start with a narrow ingestion boundary

Your public endpoint has three jobs:

1. Read the raw request body.
2. Verify that the provider sent it.
3. Persist the event before returning a successful response.

Everything else belongs outside the request lifecycle. Updating orders, sending email, generating invoices, or calling another API increases the time before acknowledgement. That makes provider retries more likely and turns a small downstream incident into an integration incident.

public function __invoke(Request $request): Response
{
    $payload = $request->getContent();

    $this->signatures->verify(
        payload: $payload,
        signature: $request->header('Webhook-Signature')
    );

    $event = WebhookEvent::firstOrCreate(
        ['provider_id' => $request->header('Webhook-Id')],
        ['payload' => $payload, 'status' => 'received']
    );

    ProcessWebhook::dispatch($event->id);

    return response()->noContent();
}

The endpoint returns 204 after durable storage and queue dispatch. The worker owns the business logic.

Verify the exact bytes you received

Many webhook providers calculate their signature from the original byte sequence. Parsing JSON and serializing it again can change whitespace, key order, or escaping. Verify the raw body first, then decode it.

Use a constant-time comparison function for signatures. Check the provider timestamp when the protocol supplies one, and reject requests outside a short tolerance window to reduce replay risk.

Secrets need an operational lifecycle too. Support overlapping current and previous secrets during rotation, record which key verified an event, and never write the secret or full authorization headers to logs.

Make processing idempotent

Most providers use at-least-once delivery. A successful event can therefore arrive more than once. Treat duplication as normal behavior.

The provider event ID should have a unique database constraint. Business operations need their own guard as well. For example, creating a subscription should be keyed by the remote subscription ID, not by the queue job ID.

A queue retry is not exceptional. It is a normal execution path that must produce the same final state.

Use database transactions around related writes. If a worker calls an external system, store enough state to decide whether the call should happen again after a crash.

Separate receipt, processing, and outcome

A useful event record keeps the integration observable:

Field	Why it matters
Provider event ID	Deduplication and support lookup
Event type	Routing and metrics
Raw payload	Replay and incident analysis
Received time	Delivery latency
Processing status	Operational visibility
Attempt count	Retry health
Last error	Fast diagnosis

Keep the raw payload according to a documented retention period. Remove personal or sensitive data you do not need, and restrict access to the event store.

Design retries deliberately

Retry transient failures such as timeouts, connection errors, and provider rate limits. Do not endlessly retry invalid payloads or impossible state transitions.

Exponential backoff with jitter prevents many failed jobs from hitting a recovering dependency at the same moment. After the final attempt, move the event to a dead-letter state and alert on a meaningful threshold—not every individual failure.

Your operations screen should make replay explicit. A replay action must use the same idempotent processor, record who initiated it, and preserve the previous attempts.

Test the failure paths

Happy-path tests prove very little for an integration. Add tests for:

• An invalid signature.
• A valid duplicate event.
• Events arriving out of order.
• A worker failing after the first database write.
• A downstream timeout followed by success.
• Secret rotation with both keys active.
• Manual replay of a completed event.

A staging endpoint connected to the provider’s sandbox is useful, but deterministic fixtures belong in the repository. They let you reproduce edge cases without waiting for an external dashboard.

The production checklist

Before enabling a webhook in production, confirm that the endpoint verifies raw bytes, responds quickly, persists before acknowledging, enforces unique event IDs, processes idempotently, applies bounded retries, exposes useful status, and supports controlled replay.

That architecture costs a little more than placing logic in a controller. It costs far less than reconstructing missed payments or corrupted state from incomplete logs.