YantrikDB Cluster Setup — YRP Native Replication, Witness HA, Failover

YantrikDB Server clusters on YRP — the YantrikDB Replication Protocol (RFC 028, raft_mode = "yrp"). YRP is a Raft-shaped safety core — leader election, log replication, snapshots, all model-checked in a deterministic simulator — plus purpose-built recovery for a memory database:

• Quorum-durable writes — a write is acknowledged only once a quorum of data nodes has durably accepted it, and applied it to engine state (semantic index included) on every replica.
• Exactly-once keyed writes — idempotency_key is checked at the leader’s ingress, carried inside the replicated log entry, and answered from a durable outcome record. A retry during any replication state (failover, partition, rejoin) dedupes to the original result — never a double write.
• Linearizable reads — a read barrier commits a no-op through the replicated log, so a read served after it reflects every write committed before it, and a deposed-but-unaware leader cannot serve stale data.
• Quarantine, not wedge — a node that boots with torn or alien replication state starts non-voting and serves diagnostics, then rejoins via a quorum-authorized snapshot. It never split-brains and never crash-loops (the failure mode that took an earlier deployment down for 10 days).
• Engine backfill — a straggler that falls behind the log-compaction horizon heals its full engine state (memories, embeddings, entities) from the leader, not just protocol metadata.
• Witness quorum — a tiny vote-only member gives safe HA with just 2 data nodes.
• Chaos-gated releases — kill-leader-under-load, partition-and-heal, torn-state rejoin, and beyond-GC recovery run on every commit and block any release that regresses cluster safety.

Live on a 4-member Proxmox homelab cluster; a benchmarked leader kill under keyed write load re-elects and resumes writes in ~3 seconds.

Topology

Minimum safe HA is 2 data voters + 1 witness; 3 data voters + 1 witness tolerates the loss of a data node while still accepting writes.

┌──────────────────┐   YRP over HTTP   ┌──────────────────┐
│  data node 1     │ ◄───────────────▶ │  data node 2     │
│  (voter)         │   /v1/yrp/msg     │  (voter)         │
│  full storage    │                   │  full storage    │
└────────┬─────────┘                   └────────┬─────────┘
         │                                      │
         │        ┌──────────────────┐          │
         └───────▶│  witness         │◄─────────┘
                  │  (vote-only)     │
                  └──────────────────┘

The witness votes in elections but never stores data and never counts toward a write’s durability — so a witness-assisted election can only ever elect a data node that already holds every committed write. It breaks ties so 2 data nodes run safe HA without a 3rd full node, the same pattern as Azure SQL (witness), MongoDB (arbiter), and Redis Sentinel.

Configuration

Generate configs with cluster init

yantrikdb cluster init writes a complete YRP config. Run it once per node with the same member list — change only --node-id. Members are node_id@http_url (YRP node ids are stable identities); YRP wire traffic rides the HTTP plane (POST /v1/yrp/msg, authenticated by the shared cluster_secret).

On node 1:

yantrikdb cluster init \
  --node-id 1 \
  --cluster-id 28 \
  --data-dir /var/lib/yantrikdb \
  --peer 1@http://10.0.0.1:7438 \
  --peer 2@http://10.0.0.2:7438 \
  --peer 3@http://10.0.0.3:7438 \
  --witness 4@http://10.0.0.4:7438 \
  --output /etc/yantrikdb.toml

It prints a fresh cluster_secret — save it; you’ll pass it to every other node as --secret and use it as the master Bearer token from clients.

On nodes 2, 3, and the witness (4), re-run with the same flags, changing only --node-id and adding --secret <the one from node 1>. A node whose --node-id matches a --witness entry is configured as a witness automatically.

The generated /etc/yantrikdb.toml looks like:

[server]
wire_port = 7437
http_port = 7438
data_dir = "/var/lib/yantrikdb"

[cluster]
node_id = 1
role = "voter"                 # "witness" on the witness member
raft_mode = "yrp"
cluster_secret = "ydb_cluster_...same-on-every-node..."

[yrp]
cluster_id = 28                # immutable cluster identity; identical on every member
tick_ms = 50                   # driver tick; all timing is counted in ticks
election_ticks_min = 10        # randomized election timeout range
election_ticks_max = 20
heartbeat_ticks = 2            # leader heartbeat cadence

[[yrp.peers]]
node_id = 1
addr = "http://10.0.0.1:7438"

# ... peers 2 and 3 ...

[[yrp.peers]]
node_id = 4
addr = "http://10.0.0.4:7438"
witness = true                 # votes only; never stores data (at most one per cluster)

The witness runs the same yantrikdb binary as the voters — in YRP mode a witness is a full peer with witness = true, not a separate daemon.

Start each node

yantrikdb serve --config /etc/yantrikdb.toml

After a few seconds an election runs and one voter becomes leader. Check any node:

curl -s http://10.0.0.1:7438/v1/health | jq .cluster

{
  "node_id": 1,
  "role_label": "leader",
  "term": 1,
  "leader": 1,
  "accepts_writes": true,
  "last_applied_index": 42,
  "replication_lag_log_entries": 0,
  "raft_mode": "yrp"
}

Reads and writes

Point writes at the leader. A follower refuses a write with 503 and the leader’s address, so clients redirect:

{ "error": "read-only: not the leader (current leader: node 3)",
  "leader_node_id": 3, "leader_addr": "http://10.0.0.3:7438", "raft_mode": "yrp" }

Keyed writes carry the same contract on the cluster as single-node: a fresh idempotency_key returns {rid}; the same key + same text returns the original {rid} with zero new writes; the same key + different text returns {stored: false, idempotency_conflict: true, rid}. Reads (/v1/recall, /v1/memory/{rid}) served from any node reflect committed writes; a node still catching up after a rejoin reports engine_backfilling on /v1/health and is not counted read- or leadership-eligible until it is caught up.

Failover

Kill the leader (systemctl stop yantrikdb). Within a few seconds:

1. A surviving voter misses heartbeats and campaigns.
2. The witness (and any other data node) grant votes.
3. The new leader resumes accepting writes; a keyed write that was in flight resolves exactly-once on retry.

The old leader, on restart, sees the higher term, adopts canonical history (truncating any tentative tail), and rejoins as a follower.

Failure	Behavior
Leader data node dies	Surviving voter + witness elect a new leader; writes resume in ~3s
Follower data node dies (3-voter)	Leader keeps writing (still has a data quorum)
Witness dies	Voters keep serving; no new election can complete until it returns
Torn / alien state on boot	Node quarantines (non-voting), serves diagnostics, rejoins via authorized snapshot
Straggler past the GC horizon	Rejoins via snapshot + engine backfill; recall reflects the recovered range
Network partition isolates the leader	Isolated leader cannot commit (no quorum), goes read-only; majority elects

Cluster authentication

The cluster_secret authenticates peer-to-peer traffic (it’s the bearer YRP nodes present to each other on /v1/yrp/msg). It is not a client data-plane token.

Client tokens are created per node with yantrikdb token create — in YRP mode the control plane (databases + tokens) is local to each node and not replicated, so create a token on each node you’ll direct clients at:

yantrikdb token --data-dir /var/lib/yantrikdb create --db default --label app

Point writes at the current leader with a token valid on that node; a follower answers 503 with the leader’s address so clients redirect. (Replicated control-plane / master-token auth is a tracked follow-up; per-node tokens are the model today.)

---

Upgrading from the openraft cluster path? The openraft engine was removed in v0.12.0 — YRP replaces it. A config with raft_mode = "openraft" is refused at startup with a migration hint. Switch to raft_mode = "yrp" with a [yrp] section as above, or raft_mode = "disabled" for single-node. Standalone deployments are unaffected.