Reference architecture · distributed systems
Real-time loyalty ledger
Two million customers, points that had to appear instantly, a three-day batch cycle, and a 99.9% SLA. No LLM anywhere near it — which is rather the point.
The engagement
A Nordic retail bank ran its credit-card loyalty scheme on a three-day batch cycle. A customer spent money on Monday and saw the points on Thursday, which is roughly as motivating as it sounds. The mandate was real-time, across two million customers, under a 99.9% SLA.
We built it serverless-first — Lambda, DynamoDB, the 2022 house style. It worked and it hit the SLA.
But it could not answer this question: what was this customer's points balance last Tuesday at 3pm, and why?
State was mutated in place. The current balance was the truth, and the path to it was gone. When a bad deploy double-credited a cohort, the fix was a bespoke correction script, run under pressure, on production, by a nervous engineer. I know because I signed it off.
The idea I'd build it around now
Points are a ledger, not a number.
Every transaction becomes an immutable event on an append-only log. The balance is not stored — it is derived by folding the events. Projections (the customer's balance, the tier status, the redemption history) are read models built by replaying the log.
That one decision dissolves the problems that made 2022 hard:
- "What was the balance last Tuesday?" — fold the events up to Tuesday.
- "Why is it 4,300 points?" — here are the 47 events that produced it.
- Bad deploy double-credited everyone? — fix the projection logic and replay. You don't patch state; you rebuild it. There is no correction script and no nervous engineer.
- Audit — a financial system where the balance is a derived quantity with a full causal history is dramatically easier to defend to an auditor than one where it's a mutable integer.
The parts that are actually hard
This is the project where the depth questions live, and none of them are about AI.
Exactly-once doesn't exist
The payments system will deliver the same transaction twice. Networks do that. What you get is effectively-once: every event carries an idempotency key derived from the transaction, ingest deduplicates on it, and the fold is therefore safe under redelivery.
Anyone who says "exactly-once" without qualification has not run a system like this. Being able to explain why it doesn't exist, and what you do instead, is worth more in an interview than any framework on your CV.
Ordering and the hot partition
Events for one customer must be ordered. Events across customers need not be. So you partition by customer ID — and then discover that 0.1% of customers generate 30% of the traffic, and one partition is on fire while the others idle.
Mitigations exist and all of them cost something. Having a considered answer here is a strong signal.
Eventual consistency meets a customer
The customer taps their card and immediately opens the app. The projection hasn't caught up. They see the old balance and they are, reasonably, annoyed.
Options: read-your-writes via a session token pinned to a log offset; optimistic UI; or simply being honest in the interface. This is the point where distributed-systems theory collides with a human being who wants their points, and the right answer is a product decision as much as an engineering one.
Replay at scale
Rebuilding a projection over two million customers and years of events is not a for loop. Parallel replay, checkpointing, and running the new projection alongside the old one until it catches up, then cutting over.
Schema evolution
Events are immutable and live forever. The event you wrote in 2022 will be read by code in 2027. Versioned schemas, upcasting on read, and a hard rule: never change the meaning of an existing field.
Results
| Metric | Target | Measured |
|---|---|---|
| Ingest throughput | 10k events/sec | ___ |
| End-to-end p99 (swipe → balance visible) | < 2s | ___ |
| Projection replay — 2M customers, 5 yrs | — | ___ |
| Duplicate delivery → double credit | 0 | ___ |
| Hot partition — p99 under 30% skew | — | ___ |
The row that matters is duplicate delivery → double credit. Fire the same transaction in ten times and the balance must move exactly once. That test is the system's reason for existing, and it runs in CI.
Decisions I'd defend in an architecture review
- Event-sourced, not state-mutating. The balance is derived. ADR-001.
- Effectively-once via idempotency keys. Exactly-once is not on offer. ADR-002.
- Partition by customer; accept and mitigate skew. ADR-003.
- Projections are disposable. Anything rebuildable from the log is not precious. ADR-004.
- Events are immutable and versioned. Upcast on read; never redefine a field. ADR-005.
Why this one, and why no AI in it
Two reasons.
Every architect now claims "event-driven architecture" on their CV. Mine has said it for a decade. This makes it true in my hands rather than in my slides — throughput, replay, idempotency, skew, measured on hardware I own.
And it is deliberately not an AI project. A portfolio consisting entirely of LLM systems invites the obvious question about whether the person can build anything that isn't a wrapper. This is the answer.