Skip to content

The engines

An engine is the program that actually runs a local model. TheYgent manages three of them — llama.cpp, MLX, and vLLM — plus a fourth binding, openai-compatible, for endpoints it merely reaches. This page explains what each engine is for, where it runs, how the managed ones start and share memory, and which modalities (chat, vision, embeddings, speech, images) each covers.

If you only need to use models, you don't have to think about engines much — a catalog install picks the right one for you. Read on when you're choosing hardware, registering weights by hand, or wondering why a first call is slow. For the vocabulary of bindings, sources, and modalities, see Models and engines.

The four bindings

Every registered model has a binding that says what runs it. There are exactly four values, and their meaning never changes:

Binding What it runs Managed?
llamacpp A llama.cpp server on GGUF weights. Runs on any platform where the binary is installed. Yes — spawned and supervised.
mlx The MLX family of servers, on Apple Silicon. Yes.
vllm A vLLM server on an NVIDIA/CUDA GPU host. Yes (experimental).
openai-compatible Any OpenAI-compatible server or hosted API at a base URL you supply. No — reached, never managed.

The first three are managed: TheYgent starts the server on first use, health-checks it, and stops it when idle. openai-compatible is reached — nothing is spawned or stopped; see Remote models.

llama.cpp

llama.cpp runs models in the GGUF format and is the broadest, most portable local engine — it runs anywhere its llama-server binary is installed (macOS, Linux, with or without a GPU). It's the default choice when you want a model to run on the same machine regardless of platform. GGUF models come in quantizations (labels like Q4_K_M), and the catalog lists each quant as a separate variant with its own fit badge.

Under the llamacpp binding, TheYgent covers several modalities by spawning the right server for the registered modality:

  • Chat and embeddings — the llama-server binary (embeddings use mean pooling).
  • Vision — the same llama-server with a multimodal projector; TheYgent finds the projector file next to the weights or falls back to auto-detection.
  • Speech-to-text — served by whisper.cpp's whisper-server, registered under binding: llamacpp with modality: audio.transcription. With ffmpeg installed it also transcodes browser microphone recordings (webm/opus); without it, only WAV works. Whisper needs its own native ggml-*.bin weights — a chat-style GGUF won't work, and an ambiguous weights path raises a clear error telling you which file to point at.
  • Image generation — a bundled wrapper around stable-diffusion.cpp's sd-cli (see the note under Installing the engine binaries).

llama.cpp launch settings

A llama.cpp chat, vision, or embeddings model starts with settings sized to fit a laptop rather than the model's maximum:

Setting (params key) llama-server flag Default
ctxSize -c Chat and vision: the model's trained context, capped at 32,768 tokens. Embeddings: the trained context, capped at 8,192 (2,048 when the file doesn't say).
parallel -np 1 — one request at a time, so the whole context serves it.
batchSize -b Embeddings: the context size. Otherwise llama-server's own default.
ubatchSize -ub Embeddings: the context size, so any input that fits the context fits one batch. Otherwise llama-server's own default.

Set any of them in the model's registration params to override, for example "params": {"ctxSize": 65536} for a longer context, or {"parallel": 4, "ctxSize": 65536} to serve four requests at once (the context is shared by the slots). They configure the engine when it starts — they are never sent with a request — and a value that isn't a positive integer, or one set on a non-llama.cpp model, is refused when you register it. The context a running model actually has shows as max context in its capabilities.

Out of GPU memory

If a large model fails with Compute error or runs out of GPU memory, lower its ctxSize: the context's cache is allocated up front, on top of the weights.

MLX

MLX is Apple's array framework, and TheYgent's MLX engines run only on Apple Silicon (M-series Macs). On that hardware they're fast and memory-efficient because the CPU and GPU share one pool of unified memory. When you're on an Apple Silicon Mac, MLX is usually the best local option for chat and vision; the same model repo often ships both a GGUF (for llama.cpp) and an MLX variant, and the catalog shows whichever your ready engines can run.

Under the mlx binding, the modalities map to distinct server programs:

  • Chat — mlx_lm.server.
  • Vision — mlx_vlm.server (a separate program).
  • Text-to-speech — mlx_audio.server.
  • Image generation — a wrapper around mflux-generate (see the binaries note below).

MLX does not serve embeddings or speech-to-text — use llama.cpp (embeddings) and whisper.cpp (transcription) for those, on the same machine.

MLX launch settings

mlx_lm.server keeps the processed state of recent prompts and reuses the one whose start matches a new request, so an agent's next step processes only what's new since its last call. On a 35B model that's the difference between about 0.9 seconds and 2–3 seconds before the first token of each step.

Setting (params key) Default Effect
promptCacheSize 10 How many prompt states the engine keeps for reuse. Raise it when one model serves several agents or concurrent runs; each state takes memory (about 0.1 GB for a 1,500-token prompt on a 35B model).

Like the other launch settings it applies when the engine starts, is never sent with a request, and is refused at registration (422 invalid_binding) unless it's a positive integer on an MLX chat model.

MoE models bigger than memory (expert streaming)

A mixture-of-experts (MoE) model splits most of its weights into many small "experts" and uses only a few of them for each token — Qwen3.6-35B-A3B, for example, picks 8 of its 256 experts per layer. With expert streaming, TheYgent keeps the rest of the model in memory and reads each layer's chosen experts from disk as they're needed, holding the recently used ones in a memory cache. A model whose full weights don't fit in your Mac's memory can still run; the answers are exactly the ones the fully loaded model gives, only slower.

It's off unless you turn it on for a model, because a model that fits in memory is always faster fully loaded. Three ways to turn it on:

  • When installing: the catalog marks mixture-of-experts models MoE, and for an MLX variant that won't fit, the install dialog offers Stream experts from disk, already ticked. See Installing local models.
  • Later: in Registries, open the model and tick Stream experts from disk in its settings; Expert cache (GB) sets the cache size. Untick it to load the model fully again.
  • Through the API, with these registration params:
Setting (params key) Default Effect
streamExperts false true serves the model with its experts streamed from disk.
expertCacheGb Half your machine's memory, less the part of the model that stays loaded How much memory the expert cache may use. Requires streamExperts: true.

For example, registering a model through the inference plane's API:

curl -X PUT http://127.0.0.1:8081/admin/models/big-moe \
  -H 'Content-Type: application/json' \
  -d '{"binding": "mlx", "source": "hf", "model": "mlx-community/Qwen3.6-35B-A3B-4bit",
       "params": {"streamExperts": true, "expertCacheGb": 8}}'

Like the llama.cpp launch settings, these configure the engine when it starts and are never sent with a request. A value that isn't true/false or a positive number, expertCacheGb without streamExperts, or either setting on anything but an MLX chat model is refused when you register it (422 invalid_binding).

What to expect, measured with Qwen3.6-35B-A3B-4bit (20 GB) on a 36 GB Apple Silicon Mac, generating 128 tokens:

How it runs Memory used Speed
Fully loaded (streaming off) 19.7 GB 53 tokens/s
Streaming, every expert in the cache 20 GB 34 tokens/s
Streaming, 4 GB expert cache, experts read from the SSD 5.7 GB 9 tokens/s
Streaming, smallest cache, every expert read from the SSD 1.8 GB 4 tokens/s

Speed depends mostly on how many of the experts a reply needs are already in the cache, then on your SSD.

Limits of expert streaming

  • Only for models that don't fit. A model that fits in memory is always faster fully loaded.
  • MoE models in MLX format only, on Apple Silicon, for chat. A model the feature can't stream — one without experts, one whose experts are a kind it doesn't read, or a checkpoint that wasn't converted for MLX — is refused when its engine starts, and the model's engine log says why.
  • Long prompts are slow from a cold disk. Processing a long prompt touches nearly every expert, so the first reply to a long prompt can take several seconds per 2,000 tokens before the cache warms up.
  • Parallel requests use more distinct experts at once, so the cache helps less.
  • LoRA adapters aren't supported with streaming.

vLLM

vLLM targets NVIDIA/CUDA GPU hosts and is built for high-throughput serving. In this build it is experimental and unverified: the vllm binding exists and targets chat, but it has not been confirmed on the maintainers' hardware, and it is deliberately left out of the automatic engine install.

Treat vLLM as experimental

Only the chat modality is wired for vLLM, and none of it is verified in this codebase. If you have a CUDA GPU, you can try it — install vLLM yourself and register a model under the vllm binding — but validate it on your own hardware before relying on it. llama.cpp and MLX are the proven local paths.

Modality support matrix

Which engine covers which modality:

Modality llama.cpp MLX vLLM
Chat (chat) ✅ ✅ ⚠️ experimental
Vision (vision) ✅ ✅ —
Embeddings (embeddings) ✅ — —
Speech-to-text (audio.transcription) ✅ (whisper.cpp) — —
Text-to-speech (audio.speech) — ✅ —
Image generation (images.generation) ✅ (sd-cli) ✅ (mflux) —

A registered model whose exact (engine, modality) combination has no server is refused with a clear error rather than being served wrong. openai-compatible endpoints aren't in this table — they serve whatever the upstream serves, and you declare the modality when you register them.

Image generation needs extra setup

The image-generation engines (stable-diffusion.cpp's sd-cli and mflux) are the newest and least-exercised paths, and their binaries are not installed by the standard engine install. You must build/install them yourself and point TheYgent at them (THEYGENT_SDCPP_BIN, THEYGENT_MFLUX_BIN) before an image-generation model will run. See Image generation in chat.

How managed engines run

You never start or stop a managed engine by hand. TheYgent handles the whole lifecycle, and understanding it explains a few things you'll notice.

Lazy start. An engine spawns on the first call to one of its logical ids, on a free local port, and TheYgent waits for it to become healthy before forwarding your request. This is why the very first call to a model is slower than the ones after it — the model is loading. Large models take longer; startup waits allow a few minutes for llama.cpp and MLX, and up to ten for vLLM. Once loaded, the engine stays warm for subsequent calls.

A resident ceiling. Only a limited number of engines stay loaded at once — two by default (THEYGENT_MAX_RESIDENT). When a new engine is needed and the ceiling is full, TheYgent evicts one to make room, choosing the lowest-priority, least-recently-used engine. An engine with a request in flight is never evicted; if nothing can be freed, the new call is refused with a no_capacity error rather than thrashing.

Abandoned calls are cancelled. If the caller hangs up before an answer arrives — its client timed out, or the connection closed — TheYgent closes the request to the engine right away instead of holding it open for an answer nobody will read. That holds for every call that returns one answer rather than a stream: chat answers, embeddings, transcriptions, speech, and generated images. Engines that watch for a closed connection stop working on the request at once, so the engine is free for the next call — llama.cpp stops generating a chat answer, and the bundled image generator stops an abandoned render and skips one still waiting its turn, which matters most because a render can take minutes. Either way, a model you asked to evict no longer waits for the abandoned answer before it unloads. A request to a hosted endpoint is closed the same way.

Idle reaping. An engine left idle longer than its idle timeout (900 seconds by default) is torn down automatically by a background sweep, freeing its memory. Mark a model Keep warm to exempt it from reaping.

Manual control. From a model's row in Registries, Warm preloads an engine (so the next call skips the cold start) and Evict frees it now. Both are safe no-ops for reachable endpoints and models that aren't loaded.

Missing binaries fail cleanly. If a model's engine binary isn't installed on the machine, the call returns engine_unavailable — a clear, up-front error — never a mysterious crash on first inference.

A failed engine is replaced. If an engine reports that it can no longer compute (llama.cpp's Compute error, usually after it ran out of GPU memory) or its process dies, TheYgent stops using it: the call fails with engine_failed (503, safe to retry), the engine is shut down, and the next call starts a fresh one. The model's row in Registries shows a red failed badge, with the reason on hover, until the replacement starts.

A dropped request is named. If an engine closes the connection before it finishes an answer, the call fails with engine_dropped_request, and the engine keeps serving other requests. This covers a reply cut off partway through a stream, which would otherwise read as a finished, shorter answer. The inference plane doesn't retry a request to a local engine; the caller's own retry policy decides. See troubleshooting for the known MLX tool-call cause.

Engine logs. Each managed engine writes its own output — model loading, warnings, errors — to a log file under the inference plane's state directory (~/.theygent/inference/logs/ by default), one file per model and modality, kept across restarts and rotated at 8 MB. Open it from the Logs button on the model's row, or with GET /admin/models/{id}/logs.

stateDiagram-v2
  [*] --> Cold
  Cold --> Loading: first call to the id
  Loading --> Resident: healthy
  Resident --> Cold: idle timeout / evicted for room
  Resident --> Resident: more calls (stays warm)

Installing the engine binaries

The engines are separate programs. On macOS, one target installs the managed set:

make engines

make engines installs the full set: llama.cpp, whisper.cpp, and ffmpeg via Homebrew, plus the MLX chat, vision, and text-to-speech servers via uv tool install. This gives you local chat, vision, embeddings, speech-to-text, and text-to-speech.

make engines prints guidance instead of installing. Install llama-server (llama.cpp), whisper-server (whisper.cpp), and ffmpeg with your package manager. The MLX servers are Apple-Silicon-only and won't run here.

Install vLLM yourself with pip install vllm. It is deliberately not part of make engines. Remember it is experimental in this build.

TheYgent finds each binary by checking an explicit environment override first, then your PATH, then (for Python-module servers) python -m …. To point at a binary in a non-standard location, set its override — THEYGENT_LLAMACPP_BIN, THEYGENT_MLX_BIN, THEYGENT_MLX_VLM_BIN, THEYGENT_MLX_AUDIO_BIN, THEYGENT_WHISPERCPP_BIN, THEYGENT_VLLM_BIN, THEYGENT_SDCPP_BIN, or THEYGENT_MFLUX_BIN. The full list is in the environment reference.

TheYgent launches and supervises these binaries but never builds or downloads them — that part is on you (or on make engines).

Checking what's ready

Which engines and modalities your machine can actually run is reported by the inference plane's readiness endpoint. Every (engine, modality) slot shows separately — a chat slot as the bare engine name (llamacpp, mlx), a non-chat slot as engine:modality (mlx:vision, llamacpp:audio.transcription) — and a not-ready slot carries a reason with an install hint:

curl http://localhost:8081/readyz

The service stays "ready" as long as any engine is available, so a Mac without CUDA still serves llama.cpp and MLX happily. The catalog's engine filters mirror this — it only offers models for slots that are ready.