Corren WCS — Warehouse Control System

In most backend jobs, the worst thing your bug can do is show a 500 page. In warehouse control, your bug can drive a 200-kilo autonomous robot into a steel shelf at full speed. That reality reshapes every technical decision — what you retry, what you refuse to retry, what you never let happen twice.

Corren WCS is the Warehouse Control System I've been building at Golden Owl Consulting since 2025. It orchestrates 100+ physical devices — autonomous pickers, conveyors, rail switches, smart storage cells — through 18 Java microservices, processes 1,000+ orders per day, and has to keep 50+ robots working the same aisles without ever touching each other. Every architectural choice in this system is a compromise between speed, safety, and the fact that hardware failures don't apologize.

Software on One Side, Steel on the Other

The hardest thing to internalize about industrial control is that the network is the physical world. When the picker service publishes a command, milliseconds later a real motor spins. If the message is duplicated, a robot moves twice. If it's dropped, the robot waits forever in the middle of an aisle, blocking every other robot behind it.

I settled on a strict bulkhead: high-level orchestration lives in Java microservices talking to each other over RabbitMQ; physical execution lives entirely inside ch.corren.opcua-driver, which is the only service allowed to speak OPC UA to the PLC layer. Nothing else touches the wire.

Under the hood, that driver is built on Eclipse Milo 0.6. A single "start picking" command translates into nine coordinated OPC UA node writes — shelf X/Y/Z, conveyor X/Y/Z, acceleration X/Y, and finally the execute = true boolean — chained with CompletableFuture so the whole sequence completes or fails atomically:

public CompletableFuture<Void> startPicking(PickingCommandEO cmd) {
    return writeNode(SHELF_X, cmd.getShelfX())
        .thenCompose(v -> writeNode(SHELF_Y, cmd.getShelfY()))
        .thenCompose(v -> writeNode(SHELF_Z, cmd.getShelfZ()))
        .thenCompose(v -> writeNode(CONVEYOR_X, cmd.getConveyorX()))
        .thenCompose(v -> writeNode(CONVEYOR_Y, cmd.getConveyorY()))
        .thenCompose(v -> writeNode(CONVEYOR_Z, cmd.getConveyorZ()))
        .thenCompose(v -> writeNode(ACCEL_X, cmd.getAccelerationX()))
        .thenCompose(v -> writeNode(ACCEL_Y, cmd.getAccelerationY()))
        .thenCompose(v -> writeNode(EXECUTE, true));
}

Feedback comes back through OPC UA subscriptions with 100ms sampling. When the PLC flips a status node, a MonitoredItem fires, we translate it into a CommandFeedbackMO, and publish it onto the AMQP bus. The rest of the system finds out about physical reality the same way it finds out about anything else — as an event.

Inside ch.corren.opcua-driver — The Only Service That Speaks to Metal

The OPC UA driver is the most interesting service in Corren because it's the only one whose bugs have physical consequences. It's also the smallest by line count and the highest by cognitive load per line. Everything I care about in this service — session lifecycle, ordering guarantees, failure classification — exists because OPC UA is a strict, stateful protocol talking to a strict, stateful PLC, and the middle-ware has to bridge that world into the loose, eventually-consistent world of everything else.

Why OPC UA in the First Place

The obvious alternative would be MQTT or a plain TCP protocol. Both are simpler. Both are wrong.

OPC UA gives us three things nothing else in the industrial space gives together: a strongly typed address space (the PLC exposes a tree of nodes with declared types, not opaque topics), built-in subscriptions with server-side sampling (no client-side polling, no thundering herd), and structured status codes (a write doesn't just fail — it fails with Bad_TypeMismatch, Bad_NodeIdInvalid, or Bad_SessionIdInvalid, and the driver responds to each differently). Losing that structure would mean rebuilding it in application code, badly.

The trade-off is that the Java tooling is thin. Eclipse Milo is the only credible OPC UA client for the JVM, and calling it "credible" is generous — the API is faithful to the spec, which is another way of saying the API is faithful to a 1,000-page ISO document. Every abstraction I built in the driver is a layer between the spec and the rest of Corren.

One Session per Endpoint, and It Never Goes Away

The driver holds exactly one persistent OpcUaClient session per PLC endpoint. Sessions are expensive — the handshake includes a SecureChannel negotiation with mutual X.509 certificate validation, activation, and namespace resolution — and they are stateful. Every MonitoredItem we've ever created lives inside that session. If the session dies, every subscription dies with it.

So the driver treats the session as a persistent resource with its own lifecycle, managed by a SessionSupervisor that runs on a dedicated thread:

public class SessionSupervisor {
    private volatile OpcUaClient client;
    private final AtomicReference<SessionState> state =
        new AtomicReference<>(SessionState.DISCONNECTED);
    private final RetryPolicy retry = RetryPolicy.exponential(
        Duration.ofSeconds(1), Duration.ofMinutes(1));

    public CompletableFuture<Void> ensureConnected() {
        if (state.get() == SessionState.CONNECTED) {
            return CompletableFuture.completedFuture(null);
        }
        return retry.attempt(this::connect)
            .thenRun(() -> subscriptions.restoreAll(client));
    }
}

Reconnection is the interesting part. When the session drops — PLC restarts, network blip, negotiated key expiry — the supervisor doesn't just reconnect. It reconnects and replays every previously registered subscription against the new session, because the PLC on the other side has forgotten about them. Getting that replay wrong is how you lose command feedback for the duration of an incident and only find out from angry warehouse operators.

The session is also intentionally not shared between command writes and subscription reads at the API level, even though they use the same underlying OpcUaClient. Writes go through a dedicated WriteExecutor that serializes them per-endpoint; subscriptions go through a SubscriptionRegistry that manages the MonitoredItem lifecycle. Same connection, different concerns, different failure handling.

Making Nine Writes Atomic Without Transactions

OPC UA doesn't have transactions. If a picking command needs nine node writes — shelf X/Y/Z, conveyor X/Y/Z, acceleration X/Y, and finally execute = true — and the fifth write fails, the PLC is left with a half-written command sitting in its input registers. The next execute = true on that endpoint would send the robot to whatever coordinates happened to be there.

The rule that fixes this is trivial and non-negotiable: execute = true is always the last write, and it only fires if every previous write returned StatusCode.GOOD. If any coordinate write fails, the driver aborts the command, clears the input registers with an explicit reset, and reports the failure upstream. The PLC never sees an execute signal on incomplete data.

public CompletableFuture<CommandOutcome> execute(PickingCommandEO cmd) {
    return writeCoordinates(cmd)
        .thenCompose(status -> {
            if (!status.allGood()) {
                return resetInputRegisters(cmd.getEndpoint())
                    .thenApply(v -> CommandOutcome.aborted(status));
            }
            return writeNode(EXECUTE, true)
                .thenApply(v -> CommandOutcome.dispatched(cmd.getCommandId()));
        })
        .exceptionally(ex -> CommandOutcome.aborted(ex));
}

Aborted commands go back on the queue with a fresh transactionId. The idempotency check upstream catches any duplicate; the state machine downstream decides whether to retry, escalate, or hold.

MonitoredItems, 100ms Sampling, and the Silent-Subscription Problem

The subscription side is where most of the interesting bugs lived. Each command creates a MonitoredItem on a specific status node with a 100ms sampling interval — the PLC checks that node every 100ms and pushes a notification if the value changed. On a normal command lifecycle, we see three notifications: EXECUTING, then COMPLETED (or ERROR), then the node clears.

Two things can go wrong here that took me a while to catch.

The first is silent subscriptions. A subscription can remain "alive" from the client's perspective — the session is up, no exception has fired — while the server has actually stopped sending notifications, usually because the underlying PublishRequest queue drained without being refilled. Milo will happily let you sit there forever waiting for a callback that will never come. The fix is a heartbeat: every subscription has a KeepAliveCount and a MaxNotificationsPerPublish, and the driver watches for missed keep-alives. Three missed keep-alives, tear down the subscription and re-create it.

The second is stale-good notifications. A PLC recovering from a physical fault sometimes republishes the last-known-good status on the monitored node, even though the command it refers to was actually aborted at the hardware level. From the driver's perspective, this looks like a normal COMPLETED. From the warehouse's perspective, the robot never actually finished the pick. The driver can't detect this on its own — the OPC UA layer has already reported success — so it publishes the feedback anyway, and lets Watchman catch the inconsistency by cross-referencing with the actual physical sensor readings on the picker.

Three-State Command Lifecycle

Working through the failure modes forced me to separate three distinct states that early versions of the driver collapsed into one:

1. Dispatched — the driver has written all nine nodes and observed execute = true acknowledged by the PLC. This does not mean the robot moved.
2. Acknowledged — the PLC has pushed the first EXECUTING notification through the subscription. This means the robot has started moving.
3. Completed — the PLC has pushed a terminal notification (COMPLETED or ERROR) and the physical sensors on the picker agree.

Every state transition has its own timeout, its own retry policy, and its own escalation path. Dispatched → Acknowledged that hangs for more than 500ms means the PLC accepted the write but never scheduled the job — usually a firmware bug, always requires operator attention. Acknowledged → Completed that hangs for more than the command's expected duration triggers a Watchman inspection. Collapsing any two of these states hides the class of failure and delays the response.

The three-state lifecycle is the piece of the driver I would ship to any other industrial control team as-is. It's the direct product of watching the same failure happen three times before separating "the software thinks it worked" from "the hardware moved" from "the hardware finished."

Eighteen Services, One Rule: No Synchronous Chains

The service topology looks like a lot from the outside — scm (system of record), orderplanning, orderpreplanning, orderexecution, pickermanager, inventory, setup, shuttle, smartcellop, watchman, equipmentmanager, and more — but the rule between them is simple: no service is allowed to synchronously call another to complete its work. Feign clients exist for read-only lookups; the actual work is event-driven.

That constraint stops the cascading-failure trap that kills so many warehouse systems. If inventory goes down for a database rebuild, orderplanning does not block. It publishes its OrderPlanningDTO, the message sits in the queue, and when inventory comes back it drains what it missed. No retries, no timeouts, no client-side circuit breakers babysitting the failure — the queue is the circuit breaker.

Every service owns its own PostgreSQL schema. Nothing is shared, and the only way state crosses a service boundary is as an event with a globally unique transactionId attached. That ID is what makes idempotency possible everywhere else.

The Order Lifecycle — Ten States, One Source of Truth

An order in Corren doesn't have a status field — it has a state machine.

CREATED → ALLOCATED → IN_PROGRESS → COMPLETE
              ↓             ↓
            FAILURE      ON_HOLD → IN_PROGRESS

There are actually ten states across the order and its jobs (CREATED, ALLOCATED, IN_PROGRESS, ON_HOLD, SUCCESS, DONE, ON_FAILURE, BLOCKED, COMPLETE, FAILURE), and the transitions are code, not convention. A ShippingOrderStarter runs on a 30-second schedule; it pulls orders in CREATED state whose nextAllocationDate has passed, and hands them to the SplitCreator.

Splitting is where warehouse reality bites. A single order might demand 40 units of an article, but those 40 units live across four different smart cells. SplitCreator fans one ShippingOrderPosition into four OrderPositionSplit rows, each pinned to a specific location, each with its own picker command downstream. The order doesn't reach COMPLETE until every split reports DONE or ON_FAILURE.

The ten-state model looks like overkill until the first partial failure. When a robot picks 30 of 40 units and then jams on the last shelf, you need ON_HOLD (recoverable, operator resumes), ON_FAILURE (unrecoverable, human intervention), and BLOCKED (waiting on upstream dependency) as distinct states — collapsing them loses the operational information the shop floor needs to react.

Working Zones — How 50 Robots Share an Aisle

The interesting problem isn't routing one robot. It's making sure fifty of them never meet in the same segment of aisle.

I built WorkingZoneService in pickermanager as a soft-real-time zone allocator. Every picker has an assigned zone at any moment, defined by a set of aisle-rail coordinates. Before a picker is dispatched to a new command, OverlapDetectionAlgorithm walks its target zone against every currently-assigned zone in the fleet:

public boolean hasOverlap(WorkingZone candidate, List<WorkingZone> active) {
    return active.stream()
        .filter(z -> !z.getPickerId().equals(candidate.getPickerId()))
        .anyMatch(z -> z.getRailSegments()
            .stream()
            .anyMatch(candidate.getRailSegments()::contains));
}

If there's overlap, the command doesn't dispatch. The order stays in ALLOCATED, the picker stays parked, and the scheduler retries on the next tick. It's a cheap, boring algorithm — but the safety property is that dispatch is a pull from the scheduler, not a push from the order flow, and the scheduler only pulls what it can prove is safe. There is no code path in the system that dispatches a command without passing overlap detection first.

A* on a Warehouse Grid

Once dispatch is safe, the next question is: what's the shortest path from where the robot is to where it needs to be? Warehouses look like grids from above — aisles, cross-aisles, rail switches, transfer cells — which makes them a natural fit for A* with grid costs weighted by hardware constraints (rail switches are expensive, straight runs are cheap, backwards moves are penalized).

The AutomatedWarehouseCalculationEngine generates forward and backward instruction sets so the same route can be executed in reverse for putaway. A picking path is calculated as a sequence of segments — Cargo Switch → Temp Cell → Cargo Switch — and each segment translates into one command in the OPC UA driver. When the driver reports COMPLETED on the last command, the position transitions to SUCCESS.

The optimization payoff is measurable: cycle time per pick dropped noticeably once A* replaced the earlier greedy nearest-neighbor router. But the more important effect was consistency — every pick takes roughly the same time regardless of which robot runs it, which is what makes daily throughput predictable enough to promise 1,000+ orders.

Everything Is Idempotent, Because Everything Is Duplicated

The single design choice that saved the most operational pain: every command handler is idempotent by transaction ID.

Every PickerCommandHandler, every RailSwitchCommandHandler, every AMQP consumer, checks its local database for the incoming transactionId before doing any work. If the ID has been processed, the handler acknowledges the message and exits — no side effects, no state change, no complaint. This turns "at-least-once" delivery into effectively "exactly-once" processing without any distributed consensus.

That one property lets me be aggressive elsewhere. Dead-letter queues can safely replay. Manual re-queues during incident recovery are safe. Kafka-style at-least-once guarantees don't scare me. The database enforces the invariant that hardware only executes once.

Watchman — The Service That Says No

org.openwms.projects.corren.watchman is the safety monitor and, by design, the only service in the system with the authority to stop things. It subscribes to every command feedback event, cross-references the current fleet state, and if it sees any signature of a fault — a picker reporting ERROR twice within a window, a rail switch that took too long to respond, a zone assignment that shouldn't exist — it publishes an emergency-stop event that the OPC UA driver treats with strict priority.

Watchman doesn't try to reason about what went wrong. Its only job is to catch anomalies fast and hand off to humans. The service is small — a few hundred lines — but it's the most important service in the system, because every other service in Corren is optimizing for doing more, and something has to be optimizing for not making it worse.

What Corren Taught Me

Distributed systems textbooks warn about "the fallacies of distributed computing" — the network isn't reliable, latency isn't zero, topology changes, and so on. Warehouse control forced me to internalize a stronger version of that same lesson: physical execution isn't reversible. There is no git reset on a robot that has already moved. The software has to make sure the movement was safe before it happens, or the system is broken.

Three decisions I'd make on any comparable system again:

1. Idempotency as a system-wide invariant, not a per-handler concern. Making every consumer idempotent from day one meant every other guarantee — retries, replays, DLQ recovery — became free. Attempting to bolt this on later almost always fails.

2. A single bulkhead between the digital and the physical. Only one service (the OPC UA driver) talks to hardware. Everything else operates on events describing intent. When something goes wrong on the physical side, you know exactly where to look, and when something needs to change in the industrial protocol, exactly one service ships.

3. State machines over status fields. The ten-state order lifecycle looked heavy until the first partial failure. Collapsing states loses the operational vocabulary the shop floor uses to talk about what's happening. It's cheaper to design them explicitly upfront than to reconstruct them from log archaeology later.

Corren is the project that convinced me most enterprise architecture patterns aren't overengineering — they're what happens when software has to survive contact with the physical world. Event-driven design, database-per-service, idempotency, state machines, careful bulkheads — none of these are academic. They're the reason the robots don't crash into each other at 3am when nobody is watching.