LedgerFlow — Payment Ledger System

Financial systems are unforgiving. A double-charged payment, a missing ledger entry, an unreconciled transaction — these aren't bugs, they're business-ending events. Building a toy payment system is easy. Building one that survives network failures, duplicate requests, and process crashes is a different beast entirely.

I built LedgerFlow to explore what it actually takes to process money reliably. Not with a payment gateway SDK abstracting away the complexity, but from scratch — with Spring Boot, PostgreSQL, and the hard lessons of distributed systems.

The result is a Java 21 application with two tightly-guarded modules: Payment handles transaction initiation while Ledger owns the accounting records. They communicate through asynchronous events, and every edge case — duplicates, crashes, missed events — has a deliberate response.

Two Modules, One Contract

The architecture is deliberately split into two Maven modules living in the same Spring Boot application:

Payment Module owns the transaction lifecycle. A Payment entity starts as PENDING, transitions to COMPLETED, and carries an idempotencyKey — a client-supplied string with a unique constraint at the database level. If the same key arrives twice, the second request is a no-op.

Ledger Module owns the accounting records. Every completed payment produces a LedgerEntry recording the debit or credit, linked to an Account that tracks running balances. The ledger doesn't care how the payment was initiated — it only cares that the event was valid.

The contract between them is the PaymentCompletedEvent:

@Data
@NoArgsConstructor
@AllArgsConstructor
public class PaymentCompletedEvent {
  private String eventId;
  private Long paymentId;
  private Long userId;
  private BigDecimal amount;
  private String currency;
  private EntryType type;
}

No shared database tables. No inter-module service calls. Just a well-defined event payload.

Events Are the Glue (and the Sharp Edge)

When PaymentService.processPayment() completes a transaction, it publishes a PaymentCompletedEvent through Spring's ApplicationEventPublisher. The ledger module listens via @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT):

@Retryable(
    retryFor = {Exception.class},
    maxAttempts = 3,
    backoff = @Backoff(delay = 1000, multiplier = 2))
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void handlePaymentCompleted(PaymentCompletedEvent event) {
    ledgerEventProcessor.processPaymentCompleted(event);
}

Why AFTER_COMMIT? Because if the payment transaction commits but the event handler crashes, the payment is still saved — the event is fire-and-forget. If the handler succeeds but the ledger write fails, the @Retryable annotation retries up to 3 times with exponential backoff.

This is the reliable-event-delivery pattern in practice. The payment module doesn't wait for the ledger. The ledger module doesn't block the payment. They're decoupled in time and space, connected only by the event contract.

The Idempotency Stack

Every payment carries an idempotencyKey — a UUID or client-generated hash that must be unique across the payments table. If a network timeout causes the client to retry, the second request hits the unique constraint and returns the existing payment without processing a second time:

return paymentRepository
    .findByIdempotencyKey(idempotencyKey)
    .orElseGet(() -> {
        // create and process payment
    });

But idempotency at the payment level isn't enough. The event handler itself must be idempotent too. The LedgerEventProcessor uses a processed_events table keyed by eventId:

try {
    processedEventRepository.saveAndFlush(
        ProcessedEvent.builder()
            .eventId(event.getEventId())
            .processedAt(LocalDateTime.now())
            .build());
} catch (DataIntegrityViolationException ex) {
    if (!isDuplicateKeyViolation(ex)) throw ex;
    log.info("Skipping already processed event {}", event.getEventId());
    return;
}

If the event has already been processed, the duplicate key violation is caught and the handler silently skips. This means at-most-once processing for ledger entries even if the event is delivered multiple times.

Dead Letter Queue — When Retries Aren't Enough

No retry policy survives every failure. After 3 retries with 1s/2s/4s backoff, the @Recover method kicks in and writes the failed event to the failed_events table:

@Recover
public void recover(Exception e, PaymentCompletedEvent event) {
    failedEventRepository.save(
        FailedEvent.builder()
            .eventId(event.getEventId())
            .eventType(event.getClass().getSimpleName())
            .payload(objectMapper.writeValueAsString(event))
            .errorMessage(e.getMessage())
            .createdAt(LocalDateTime.now())
            .build());
}

This is the dead letter queue. The payment itself is still valid — the money isn't lost. But the ledger entry is missing, creating a gap that needs manual or automated resolution. Which brings me to the final piece.

Scheduled Reconciliation — The Safety Net

No matter how careful the design, events can be lost. A crash between the AFTER_COMMIT trigger firing and the event handler starting. A race condition. A transient network partition.

The ReconciliationService runs every 5 minutes and finds COMPLETED payments that don't have corresponding ledger entries:

@Scheduled(fixedRate = 300000)
public void reconcilePayments() {
    List<Payment> orphanedPayments = findOrphanedPayments();
    for (Payment payment : orphanedPayments) {
        eventPublisher.publishEvent(new PaymentCompletedEvent(
            "RECON-" + payment.getId() + "-" + System.currentTimeMillis(),
            payment.getId(), payment.getUserId(),
            payment.getAmount(), payment.getCurrency(),
            EntryType.CREDIT));
    }
}

The query uses a NOT EXISTS subquery to find orphaned payments. Each one gets a new event ID prefixed with RECON- to distinguish it from the original event. The event handler's idempotency check naturally handles the case where the original event eventually arrives late.

This is the distributed systems safety net: detect drift, then re-converge.

Why Spring Boot 4 and Java 21

This project runs on Spring Boot 4.0.5 (the latest 4.x beta) with Java 21. The choice is deliberate — virtual threads (Project Loom) fundamentally change how you think about blocking I/O in a financial system. Each @Transactional method can block on the database without pinning a carrier thread, and the event publisher+handler chain benefits from structured concurrency patterns.

Spring Boot 4 also brings improved @Retryable integration, better @TransactionalEventListener semantics, and a streamlined Hibernate 6.x/Jakarta Persistence stack.

The build pipeline uses Spotless with Google Java Format for automated code formatting — enforced in CI via GitHub Actions. Every commit must pass formatting checks, producing a consistent codebase without formatting debates.

What I Learned

Building LedgerFlow from scratch reinforced one truth: reliability is not a feature, it's an architectural property.

Idempotency must be baked into every layer. Payment request idempotency prevents duplicate charges. Processed-event idempotency prevents duplicate ledger entries. Reconciliation idempotency prevents duplicate recovery actions. Each layer independently guarantees at-most-once processing.

Events are harder than they look. Spring's ApplicationEventPublisher makes publishing an event trivial, but making event delivery reliable requires explicit patterns — AFTER_COMMIT phase, retry with backoff, dead letter queues, and scheduled reconciliation. The simple path creates silent data loss.

A reconciliation job is not optional. I initially thought the event chain was reliable enough. I was wrong. The scheduled reconciliation caught several edge cases during testing — a transaction that committed but whose AFTER_COMMIT handler never fired, a constraint violation that wasn't properly caught, a serialization failure in the dead letter queue. Without reconciliation, every one of those would have been a silent accounting gap.

LedgerFlow is a learning project, not production financial software. But the patterns it implements — idempotency, event-driven processing, retry with backoff, dead letter queues, scheduled reconciliation — are the same patterns that production payment systems use every day. The code is open source on GitHub, and the README includes full instructions for running it locally with PostgreSQL.