Beyond the Checkout Page: The Engineering Behind Payment Integrations

When you start building a new service or product, setting up a payment integration is mostly on the critical path. Whether you are running an e-commerce storefront or building a marketplace that connects service providers with customer demand, choosing the right payment infrastructure matters.

In our current stack, we use Chapa, and have designed our system with clean gateway interfaces so that plugging in another provider, such as Stripe, requires minimal code changes.

While data models vary across providers, most expose the same core transactional concepts. Once you understand these engineering principles, adapting to another gateway becomes much easier.

Redirecting a customer to a checkout page is the easy part. The real engineering begins once payment events start flowing back into your system—and once you decide what “balance” means inside your product.

The Mirage of the 3-Step Integration

When I first approached payment integrations, I fell into a common trap. I assumed it was a solved problem that boiled down to three simple steps:

1. Initiate a checkout session and redirect the user.
2. Listen for a webhook or callback notification.
3. Update the payment status to completed or failed.

While the happy path looks exactly like that, production environments are messy. Network connections drop, client browsers crash mid-redirect, and third-party webhooks can hit your application multiple times due to automatic retry.

To prevent webhook requests from timing out, we split the path into synchronous ingestion and asynchronous processing, with money-moving work in background workers—not inside the gateway’s HTTP timeout window.

System Topology Overview

  graph TD
    Gateway[Chapa / Stripe] -->|1. POST Webhook| API[HTTP Handler]
    API -->|2. SETNX idempotency claim| Cache[(Redis Cache)]
    API -->|3. XADD| Stream[(Redis Stream)]
    API -->|4. Fast Ack: 200 OK| Gateway

    subgraph Async Execution Layer
        Worker[Background Stream Worker] -->|5. XREADGROUP / XCLAIM| Stream
        Worker -->|6. Re-verify with provider| Verify[Provider API]
        Worker -->|7. Persist settlement + ledger| DB[(Core SQL Database)]
        Worker -->|8. Exhausted failures| DLQ[(Redis Stream DLQ)]
    end

The shape that matters: ack fast, verify later, write money once, recover crashes.

Settlement vs Ledger

A second trap, after “webhooks are easy,” is treating the gateway transaction row as the user’s balance.

Most money products need two layers:

Gateway charges, PaymentIntents, and bank claims are settlement. Balance transactions, wallet entries, or double-entry journal lines are the ledger. Collapse them and you lose in-transit states that never became money—and you mix provider noise into the product balance users care about.

Idempotency then belongs at every money-moving boundary: claim the webhook once, unique-constrain the settlement ref, and unique-constrain each ledger entry so retries cannot double-credit or double-debit.

Users should see the ledger as their transaction history. Settlement is for operators, audit, and “still pending” notices—not a second parallel history.

The Core Engineering Pillars

1. Offload Work Off the Webhook Path

If you run validation, ledger writes, and downstream side effects inside the webhook handler, a slow database costs you the gateway’s ack window. Miss it, and you get retries you did not want—or timeouts that look like lost payments.

Keep the handler thin: authenticate (HMAC or equivalent), claim idempotency, enqueue a reference, return 200 OK. Workers do verification and persistence.

We use Redis Streams as the append-only transaction log. The HTTP handler validates the request, XADDs the webhook payload (or tx_ref) onto the stream, and immediately returns 200 OK. A pool of background workers consumes the stream asynchronously.

Redis tip: Unacked messages sit in the consumer group’s Pending Entries List (PEL). They do not auto-redeliver. After a worker crash, a reclaim routine must XCLAIM (or XAUTOCLAIM) stale PEL entries and re-process them.

2. Idempotency Is a Stack, Not a Flag

Duplicates are normal: gateway retries, at-least-once queues, operator replays. A single “processed” boolean is not enough.

A common stack:

1. Ingest claim — atomic lock keyed by provider event or tx_ref. We use Redis SETNX on chapa:webhook:<tx_ref> with a 72-hour TTL (size it to your provider’s retry window).
2. Settlement uniqueness — database constraint on the external reference.
3. Ledger uniqueness — idempotency key per business effect (topup:…, charge:…).

The ingest TTL alone is not the money guard. When it expires, a late retry can re-enqueue; settlement and ledger uniqueness must still refuse a second credit. At-least-once delivery plus idempotent handlers is how most systems approximate “exactly once” for money.

3. Trust the Notification; Believe the Provider API

HMAC (or mTLS) proves the webhook came from someone who knows your secret. It does not prove the payload fields are still true. Treat the webhook as a wake-up: re-query the provider with the reference, then write settlement and ledger from verified state.

4. Backoff, Classification, and Dead Letters

Provider APIs fail. Our workers use a 5-stage exponential backoff with a 3-second base delay plus jitter. After five failures, the message moves to a Dead Letter Queue stream for manual review so poison cannot block the primary pipe.

Classify before you ack:

• Transient — do not XACK; leave the entry in the PEL, sleep with backoff, then retry or let reclaim pick it up after a crash.
• Permanent — write DLQ, then ack (retrying forever helps no one).
• Already applied — ack; the unique constraint already did its job.

Write the DLQ before acking the primary queue. The reverse order can drop money on a broker blip.

At higher volume, DLQ depth needs metrics and a replay path—not only a place to shove failures.

5. Reconciliation Closes the Gaps Webhooks Miss

Webhooks are best-effort from your perspective. A recurring reconciler (ours runs on a cron, e.g. every 30 minutes) scans pending or expected refs, checks the provider, and repairs the ledger when volume or ticket cost justifies it. Assert on ledger effects, not only on settlement status rows.

Patterns That Show Up as You Scale

Easy to skip on day one, expensive to bolt on later. You do not need all of these immediately—leave room in the design.

Thresholds and policy gates. Min top-up, max debit, daily velocity, and payout sweeps (“disburse when balance ≥ X”) turn a ledger into a product. Without them you get fraud, fat-finger charges, and payout spam.

Append-only money. Add new ledger entries (including compensating credits/debits). Do not mutate posted amounts—that kills auditability.

Serialize balance-critical writes. A debit that checks SUM(entries) needs a lock (or equivalent) so two concurrent spends cannot both pass an insufficient-funds check.

Dual-write discipline. Settlement insert and ledger credit should commit together in one DB transaction. Partial success is how “paid but not credited” tickets start.

Fees vs spendable credit. Store provider fees on settlement; keep the user’s spendable amount explicit so pricing stays reconcilable.

Payout seams for marketplaces. Provider balances, platform fees, and threshold sweeps show up eventually. Single-wallet prepaid is a fine start—plan the seam before ad-hoc payout side effects.

Event Lifecycle

  sequenceDiagram
    autonumber
    actor Chapa as Chapa or Stripe
    participant API as HTTP Handler
    participant Redis as Redis Cache and Streams
    participant Worker as Stream Worker
    participant Provider as Gateway API
    participant DB as Core SQL DB

    note over Chapa, API: Phase 1 - Ingestion
    Chapa->>API: POST webhook with signature
    API->>API: Verify HMAC signature
    API->>Redis: SETNX idempotency key EX 72h
    alt Duplicate event
        Redis-->>API: key exists
        API-->>Chapa: 200 OK exit early
    else First time
        Redis-->>API: claim acquired
        API->>Redis: XADD webhooks stream
        API-->>Chapa: 200 OK fast ack
    end

    note over Worker, DB: Phase 2 - Asynchronous processing
    Worker->>Redis: XREADGROUP new or XCLAIM stale PEL
    Redis-->>Worker: stream message with tx_ref
    Worker->>Provider: GET verify tx_ref

    alt Verified success
        Worker->>DB: Settlement and ledger credit atomic
        Worker->>Redis: XACK
    else Transient failure
        Worker->>Worker: Backoff, leave unacked in PEL
        Note right of Worker: Reclaim via XCLAIM after crash
    else Exhausted or permanent
        Worker->>Redis: XADD DLQ then XACK
    end

Final Thoughts

The principles hold across gateways and scales: keep ingestion off the money path, stack idempotency, re-verify before granting value, separate settlement from the spendable ledger, and add thresholds, recon, and payout seams before volume forces them.

How have you architected your payment integrations? Share your thoughts via email or on X.