The moment more than one jaque process shares a log, something has to decide which process owns which object and how a command finds the owner. This page is that mechanism, stated as mechanism rather than as scenario: membership buckets, rendezvous hashing over their live members, a heartbeat TTL that declares a member dead, an epoch that fences a partitioned engine, and exactly one forwarding hop for a command that lands on the wrong engine. Topologies is the set of shapes this enables. None of it is observable with a single engine, and none of it needs a coordinator process: the coordination store is the same log server the event log lives on.

1. -queue

Required for -target engine and worker. memory only makes sense when engine and worker share one binary in development; nats://host:port is the option for running them on separate machines.

2. -coordination and membership

-coordination (default memory, in-process) is where engines announce themselves. nats://host:port names a shared store, which the same log server the log uses can serve, or a separate one. -heartbeat-ttl (default 10s) is how long a missed heartbeat is tolerated before the rest of the cluster considers an engine dead; each engine refreshes at a third of that interval. With one engine (replicas: 1, the default) none of this is observable; it matters once more than one process shares the coordination store.

Each role that scales independently gets its own membership bucket: engines under membership/, -target sink under sink/membership/, -target notifier under notifier/membership/. The three are siblings, not nested. Ownership within each bucket is computed by rendezvous hashing (HRW) over that bucket's live member list. Every member computes the same answer from the same list, so the assignment is stable across members and no coordinator has to broadcast a plan.

3. Heartbeat TTL, in practice

With replicas: 1 nothing changes from a single-process deployment.

With replicas: 2 each object is owned by exactly one of the two. If one dies, the other recomputes ownership after -heartbeat-ttl and picks up its objects; that recomputation is the failover.

With replicas: 3+ the same hashing spreads objects across all of them. Every engine is active; none is a standby.

4. Epoch fencing

An epoch fence protects correctness during a partition. Every membership change bumps an epoch, every event carries the epoch it was written under, and an up-to-date engine discards any event with a stale epoch, counting it in jaque_engine_stale_epoch_total, rather than let a partitioned engine corrupt state another engine has already claimed. The worst case for an isolated engine is one extra check run; it can never write a state transition that contradicts the current owner's.

5. Command forwarding, one hop

A command that lands on an engine that does not own the target object is forwarded once, to whichever engine currently does, over jaque.cmd.<engine-id> on the same log server that -coordination uses, and applied there. jaque_engine_commands_forwarded_total{outcome} counts the attempt as sent or error. If the object still has no owner after that one hop, because membership changed again in the meantime, the command is dropped and counted in jaque_engine_not_owned_total. There is never a second hop; a command that cannot find its owner in one bounce is lost rather than circulated. With -coordination memory nothing is forwarded, because there is only one engine.

6. UI submission via ingress

A -target ui process has no engine loop of its own, so a write it receives is submitted to the engine fleet through jaque.work.cmd.ingress. Exactly one engine claims each submitted command, resolves the real owner and forwards it by object the same way as in section 5. This requires -coordination; without it, -target ui rejects writes with 503.

7. Security considerations

The coordination store is a write surface. Any process that can publish to jaque.cmd.<engine-id> or jaque.work.cmd.ingress can have a command applied by an engine, with no check beyond what the log server enforces; -api-token guards the API listener, not these subjects. Treat the log server's network as part of the perimeter described in Security.