Try it free

Documentation  /  Operations

When inference is slow

What to check when Genie takes longer than expected, and what the platform does rather than guessing.

CHECK, IN THIS ORDERLOOKS LIKE 01 Is the GPU being used at all CPU inference uniformly slow, every request 02 Is another process on the card slow in bursts, correlated with something else 03 Is the model swapping first request slow, subsequent ones fine 04 Is the context long slow in proportion to the question 05 Is the queue deep wait rising while generation stays flat time to first token and generation rate move for different reasons · measure them separately

First, what it is not

Slow inference does not affect monitoring, alerting, collection, patching or the service desk. Those are deterministic paths and they do not wait on the model.

Check in this order

01

GPU contention

Another process holding VRAM is the most common cause. Check what else is resident — a browser, a video call, or a second model.

02

VRAM headroom

A model that cannot fit its context window spills, and spilling is orders of magnitude slower than resident inference.

03

Context length

A long conversation or a large retrieved document costs proportionally. The platform reports token counts per request.

04

Concurrent requests

Several analysts asking at once queue against one model. The queue depth is on the health surface.

05

Thermal throttling

A laptop GPU under sustained load throttles. Temperature and clock are reported alongside utilisation.

What the platform does about it

Under load, guarded operations refuse rather than skip. A guardrail that stops running when the system is busy is the worst possible failure, so the cost is bounded and exceeding the bound is a refusal, not a bypass.

Sizing

Inference speed scales with VRAM and memory bandwidth more than with core count. See Requirements and sizing for the one configuration measured end to end.

← Users & access  ·  Migrating in →

Not running it yet? Start a thirty-day evaluation — read-only, on your own estate, no card.

Two numbers move for two different reasons

Generation speed — how fast the answer appears once it starts — tracks memory bandwidth and the model's size.

Time to first token — how long you wait before anything happens — is prefill. It scales with how much context you sent and how many people asked at once, not with bandwidth. On this hardware it is the one an operator actually feels.

Measured on a 12 GB card, 14B model, 8,192-token context:

0.4 s to first token on a short question · 1.7 s on a 4,500-token one · 3.8 s at eight concurrent requests · 17.5 s for the first question of the morning if the model was evicted overnight.

If Genie feels slow to start, look at context length and concurrency. If he feels slow to type, look at VRAM.

And the one that dwarfs the others: asking for a context window the card cannot hold. Measured, the same card, same model: 43 tok/s at 8,192 tokens against 16.9 tok/s at 32,768 — a factor of 2.5, because the weights spill to host memory and every token crosses the PCIe bus. This is what happens the first time somebody pastes a large document into a conversation on a card with no headroom.

One configuration has been measured end to end and the full run is published on Requirements and sizing. Everything else there is arithmetic from published specifications, labelled as projected.

Not running it yet? Start a thirty-day evaluation — read-only, on your own estate, no card.

Thirty days · read-only · no card

Run it beside what you already have, against your real clients. It tells you what your tools are reporting that is not true.

Start an evaluation