Talk about your practice
High availability
Three nodes, fencing before promotion, and no split brain — with one model of the truth across every one of them.
What happens when a node stops
A node stops mid-answer and the answer completes. A fence fails and the cluster goes read-only rather than promote. Two nodes lose sight of each other and neither can claim to be in charge. And a migration stalls a cluster because it did not say what it was going to do. The sequences where it refuses are the ones worth watching.
And the better outcome is the one where it never had to. He drains a node that is going to fail, while nothing is being interrupted.
Most MSPs run one box, and that is a supported answer
This platform was designed for a single machine in an MSP's own rack, and most practices will run exactly that. High availability is a configuration you choose, never a dependency you inherit — a single node runs no consensus, holds no election and pays no latency for a capability it is not using.
And there is a second engine beside the platform that survives what stops it, maintains the box overnight, and can do four things and no more. How the two engines work, and what each may never do.
And the single-node answer is better than it sounds. If the node stops, monitoring and automation stop until it is back. Your agents keep running and keep queueing; nothing is lost. Your clients' systems are unaffected because we are not in their path. Compare that to a cloud platform's outage, which takes your whole practice offline and which you can do nothing about.
Three shapes, and what each one does not survive
Two nodes is not on that list, and we will not call it high availability. Two nodes cannot establish quorum — when they lose sight of each other, each has exactly as much claim to be in charge as the other. We support the shape as a primary and a warm spare requiring a person to promote, and we label it as exactly that.
Selling a pair as HA is the most common lie in this market, and it is discovered during the first real failure. Every shape we support is listed with what it does not survive, because a shape sold without its failure mode is sold dishonestly.
What actually happens when a node stops
The step people skip is the third one. A node that has stopped answering may still be alive and still be writing — so before anything is promoted, the old primary is made incapable of writing. Not asked to stand down. Fenced.
And if the fence fails, the promotion does not happen. The cluster goes read-only and tells you. An outage is recoverable; two nodes both accepting writes for ninety seconds produces a divergence in an append-only ledger that nothing can undo.
Failover is driven, not asserted. The gate kills a node and watches the work continue — a query mid-answer, a ticket mid-write, a collection sweep mid-run. A cluster that has never lost a node under test is a cluster nobody has tested.
Adding nodes never adds a boundary
This is the commitment everything else in the product rests on. Reconciliation is possible here because there is one shared model of what is true, and it is structurally impossible for an assembly of acquired products because there are four. A cluster that partitioned your clients across nodes would have arrived at the same problem by a different road.
So no client, site or tenant ever lives on a node. Every node can answer every question about your whole estate, and adding one adds throughput. We prove it by asking each node the same question about the same client and comparing the answers character for character.
The Tuesday afternoon nobody plans for
The failure people prepare for is a node stopping. The one that actually bites is somebody applying an update. A migration takes an exclusive lock on the primary, every replica stalls behind it, replication lag climbs past tolerance, the replicas leave the read rotation, and the failover machinery sees a cluster in distress during a change nobody told it about.
So every migration declares its shape before it runs. Whether it takes an exclusive lock, how long it is expected to hold it, and what the cluster should do while it does. A migration that has not declared its shape does not run on a cluster.
Rolling upgrade, one node at a time, with the cluster serving throughout. A node joins live and leaves live, and the gate drives both while work is in flight rather than on an idle cluster.
Capacity, answered from measurement
The questions an MSP actually asks are capacity questions, and every incumbent answers them with a spreadsheet somebody filled in during a launch. How many clients can this carry. What happens at twice the size. When do I need another node.
Work is placed on the node with headroom, measured now rather than assumed from a static weight. Headroom is four facts, never one score — GPU memory resident and free, inference queue depth and its wait, CPU and IO pressure, and whether this node holds the scheduler lease.
A composite score hides which of the four is the constraint, and the constraint is the only part that tells you what to buy. So the four are reported separately and the answer names which one you would hit first.
What the AI does here, and what it never touches
Genie reads the cluster and tells you what he sees: which node is under pressure, which is drifting, when a tolerance will be crossed and what the two options are. He predicts the crossing before it happens rather than alerting on the day it broke.
He does not promote a node, fence a primary, or decide an election. Failover is a deterministic mechanism with a quorum and a fence, and an inference call has no place in it. An AI that could promote a node is an AI that could split a brain.
The same rule as everywhere else in this platform: he does the work and a person keeps the decisions that cannot be undone.
What we chose, and what it costs
Every choice here has a price and we would rather you knew it before you bought than discovered it during an outage.
| The choice | What it buys | What it costs |
|---|---|---|
| Fence before promote | No split brain, ever | A failed fence means no failover — read-only until a person acts |
| Quorum of three | A node can be lost without a human | Three machines, not two |
| One model on every node | Any node answers anything; adding one adds throughput | Every node carries the whole model rather than a shard of it |
| Migrations declare their shape | An update cannot look like a failure | A migration that has not declared cannot run on a cluster |
None of this is a default. A single node runs none of it, pays for none of it, and is what most practices will run for years.
Try it on your own estate
Thirty days, read-only, on your own hardware. No card, no call, and nothing to uninstall if you walk away.