Talk about your practice

High availability

Three nodes, fencing before promotion, and no split brain — with one model of the truth across every one of them.

What happens when a node stops

A node stops mid-answer and the answer completes. A fence fails and the cluster goes read-only rather than promote. Two nodes lose sight of each other and neither can claim to be in charge. And a migration stalls a cluster because it did not say what it was going to do. The sequences where it refuses are the ones worth watching.

And the better outcome is the one where it never had to. He drains a node that is going to fail, while nothing is being interrupted.

Most MSPs run one box, and that is a supported answer

ONE NODE N1 Survives nothing. Agents queue. Nothing is lost. THREE NODES · ONE SITE N1 N2 N3 Survives any one node. Not the rack or the switch. THREE NODES · TWO SITES SITE A SITE B N1 N2 N3 Survives a site. Quorum decides which survives. Two nodes is not on this list. Two nodes cannot establish quorum, and we do not call it high availability.

This platform was designed for a single machine in an MSP's own rack, and most practices will run exactly that. High availability is a configuration you choose, never a dependency you inherit — a single node runs no consensus, holds no election and pays no latency for a capability it is not using.

And there is a second engine beside the platform that survives what stops it, maintains the box overnight, and can do four things and no more. How the two engines work, and what each may never do.

And the single-node answer is better than it sounds. If the node stops, monitoring and automation stop until it is back. Your agents keep running and keep queueing; nothing is lost. Your clients' systems are unaffected because we are not in their path. Compare that to a cloud platform's outage, which takes your whole practice offline and which you can do nothing about.

Three shapes, and what each one does not survive

1 Node stops or goes silent 2 Quorum agrees not one node's opinion 3 Old primary is fenced made unable to write 4 Replica promotes only after the fence 5 Work resumes agents reconnect If step 3 fails, step 4 does not happen. The cluster goes read-only and tells you. A node that stopped answering may still be writing — and an outage is recoverable where a split ledger is not.

Two nodes is not on that list, and we will not call it high availability. Two nodes cannot establish quorum — when they lose sight of each other, each has exactly as much claim to be in charge as the other. We support the shape as a primary and a warm spare requiring a person to promote, and we label it as exactly that.

Selling a pair as HA is the most common lie in this market, and it is discovered during the first real failure. Every shape we support is listed with what it does not survive, because a shape sold without its failure mode is sold dishonestly.

What actually happens when a node stops

What everyone else ends up with. clients A-F clients G-M clients N-Z Three models of the truth. No question can cross a boundary without a nightly sync and some hope. What this is. every client every client every client One model. Every node answers every question about the whole estate. A node is throughput. And every comparison reads one node, in one transaction. A reconciliation that read the claim from one node and the evidence from another would be comparing two different moments — and inventing differences that were never there.

The step people skip is the third one. A node that has stopped answering may still be alive and still be writing — so before anything is promoted, the old primary is made incapable of writing. Not asked to stand down. Fenced.

And if the fence fails, the promotion does not happen. The cluster goes read-only and tells you. An outage is recoverable; two nodes both accepting writes for ninety seconds produces a divergence in an append-only ledger that nothing can undo.

Failover is driven, not asserted. The gate kills a node and watches the work continue — a query mid-answer, a ticket mid-write, a collection sweep mid-run. A cluster that has never lost a node under test is a cluster nobody has tested.

Adding nodes never adds a boundary

GROWING WITHOUT REBUILDING one node 1 GPU Where most practices start Your whole book, one machine GPU GPU Add capacity More requests at once, same answers Add survival Every step is additive. None of them is a migration. The same deployment, the same data, the same licence — and no client is ever moved onto a different box, because no client lives on one.

This is the commitment everything else in the product rests on. Reconciliation is possible here because there is one shared model of what is true, and it is structurally impossible for an assembly of acquired products because there are four. A cluster that partitioned your clients across nodes would have arrived at the same problem by a different road.

So no client, site or tenant ever lives on a node. Every node can answer every question about your whole estate, and adding one adds throughput. We prove it by asking each node the same question about the same client and comparing the answers character for character.

The Tuesday afternoon nobody plans for

The failure people prepare for is a node stopping. The one that actually bites is somebody applying an update. A migration takes an exclusive lock on the primary, every replica stalls behind it, replication lag climbs past tolerance, the replicas leave the read rotation, and the failover machinery sees a cluster in distress during a change nobody told it about.

So every migration declares its shape before it runs. Whether it takes an exclusive lock, how long it is expected to hold it, and what the cluster should do while it does. A migration that has not declared its shape does not run on a cluster.

Rolling upgrade, one node at a time, with the cluster serving throughout. A node joins live and leaves live, and the gate drives both while work is in flight rather than on an idle cluster.

Capacity, answered from measurement

IT TELLS YOU BEFORE YOU NEED IT 0.4s 3.0s YOUR TOLERANCE 2.1s today ~18 days 3 WEEKS AGO TODAY PROJECTED Move the morning sweep off the peak Costs nothing. Offered first. Or add a fourth node If the cheaper answer does not fit your practice.

The questions an MSP actually asks are capacity questions, and every incumbent answers them with a spreadsheet somebody filled in during a launch. How many clients can this carry. What happens at twice the size. When do I need another node.

Work is placed on the node with headroom, measured now rather than assumed from a static weight. Headroom is four facts, never one score — GPU memory resident and free, inference queue depth and its wait, CPU and IO pressure, and whether this node holds the scheduler lease.

A composite score hides which of the four is the constraint, and the constraint is the only part that tells you what to buy. So the four are reported separately and the answer names which one you would hit first.

What the AI does here, and what it never touches

Genie reads the cluster and tells you what he sees: which node is under pressure, which is drifting, when a tolerance will be crossed and what the two options are. He predicts the crossing before it happens rather than alerting on the day it broke.

He does not promote a node, fence a primary, or decide an election. Failover is a deterministic mechanism with a quorum and a fence, and an inference call has no place in it. An AI that could promote a node is an AI that could split a brain.

The same rule as everywhere else in this platform: he does the work and a person keeps the decisions that cannot be undone.

What we chose, and what it costs

Every choice here has a price and we would rather you knew it before you bought than discovered it during an outage.

The choiceWhat it buysWhat it costs
Fence before promoteNo split brain, everA failed fence means no failover — read-only until a person acts
Quorum of threeA node can be lost without a humanThree machines, not two
One model on every nodeAny node answers anything; adding one adds throughputEvery node carries the whole model rather than a shard of it
Migrations declare their shapeAn update cannot look like a failureA migration that has not declared cannot run on a cluster

None of this is a default. A single node runs none of it, pays for none of it, and is what most practices will run for years.

Try it on your own estate

Thirty days, read-only, on your own hardware. No card, no call, and nothing to uninstall if you walk away.