Skip to content

Troubleshooting

Common problems, what causes them, and how to fix them. Most issues fall into one of a few buckets: a service didn't start, a model won't load, or a run surfaced an error code. When you hit an error code you don't recognize, the API reference lists where each one comes from.

First stop: the health checks

Before digging in, check both planes are up and ready:

curl http://localhost:8081/readyz   # inference plane
curl http://localhost:8080/readyz   # control plane

The inference plane's /readyz tells you exactly which (engine, modality) slots are missing and why; the control plane's tells you if Postgres or the inference plane is unreachable. See Running TheYgent.

Services and startup

A service didn't start — "already on :PORT — skipping"

Cause. Something is already listening on that port (a stale instance, or another app on :8080, :8081, or :5174). make up refuses to clobber it and skips that service loudly.

Fix. Stop the other process, then make restart. To stop TheYgent's own services, make down. Confirm what's up with make status, and read the per-service logs under .run/ (.run/control-plane.log, .run/inference-plane.log, .run/interface.log) with make logs.

The control plane won't come up / /readyz says the database is down

Cause. DATABASE_URL is unset, wrong, or points at a Postgres that isn't running. The control plane requires Postgres and fails readiness without it.

Fix. Set DATABASE_URL in .env to a reachable async DSN — postgresql+asyncpg://user:pass@host:5432/dbname — and make sure the database and user exist. You supply your own Postgres; TheYgent does not provision one. See Installation and DATABASE_URL.

Database and migrations

"DATABASE_URL is not set (copy .env.example to .env)"

Cause. make migrate (or make up) ran without DATABASE_URL in the environment.

Fix. cp .env.example .env, set DATABASE_URL, and re-run. Migrations use the same variable — the migration tool raises DATABASE_URL must be set for migrations when it's missing.

Migrations fail or the schema looks out of date

Cause. The control-plane schema hasn't been applied to your database, or a pull added new migrations.

Fix. Apply them:

make migrate
# or, directly:
uv run --package theygent-control-plane alembic upgrade head

Durable-runtime tables live in a separate dbos schema on the same Postgres and are migrated automatically when durable mode starts — there is no separate step for them.

Models and engines

model_not_found (404)

Cause. Either the model field is a logical id you never registered, or you put an engine name (like llamacpp or mlx) where a logical id belongs. The model field is always a logical id, never an engine name.

Fix. Register the model on the Registries page (or PUT /admin/models/{id}), and use that logical id. Check what's registered with GET /admin/models or GET /v1/models.

engine_unavailable (503)

Cause. The model is registered, but the engine binary it needs isn't installed on this machine (for example an mlx model on a host without the MLX servers).

Fix. Install the engine. On macOS, make engines installs the local engine servers; otherwise install llama-server, whisper-server, and ffmpeg from your package manager. GET /readyz on the inference plane names the missing slot with an install hint. See Engines.

/readyz shows a slot as not-ready

Cause. That (engine, modality) combination has no runnable binary — for example llamacpp:audio.transcription needs whisper-server, and image generation needs sd-cli or mflux-generate, which make engines does not install.

Fix. Install the named binary, or point the matching binary-override variable (e.g. THEYGENT_WHISPERCPP_BIN, THEYGENT_SDCPP_BIN) at your build, then restart the inference plane. The service stays generally "ready" as long as any one engine works, so a single missing slot doesn't take everything down.

incomplete_weights (422 on install or registration, 503 on use)

Cause. The weights are only part of a model. Text-to-image repositories often publish the denoiser by itself, without the VAE and text encoder it needs to render — the file is the right size and the right format, but nothing can generate an image from it alone. The same code also catches a language model registered under a images.generation modality.

Fix. Install a complete text-to-image checkpoint: one file that bundles the denoiser, the VAE, and the text encoder. The message names which parts are absent. See Incomplete image models.

no_capacity (503)

Cause. Every resident engine slot is full and busy — the inference plane keeps at most THEYGENT_MAX_RESIDENT engines loaded (default 2), and none could be evicted because they're all handling requests.

Fix. Wait for an in-flight request to finish, evict a model you're not using (the Evict action on Registries, or POST /admin/models/{id}:evict), or raise THEYGENT_MAX_RESIDENT if you have the RAM.

engine_failed (503)

Cause. The model's engine stopped being able to compute — llama.cpp answers Compute error once its GPU backend has failed, most often because the model ran out of GPU memory — or the engine process died. TheYgent has shut that engine down; the next request starts a new one.

Fix. Retry. If it fails again, the model doesn't fit with its current settings: lower its context with the ctxSize launch setting, pick a smaller quantization, or close other GPU-heavy apps. The message names the engine log, which has the engine's own error (also under the model's Logs button).

engine_dropped_request (502, or an error at the end of a stream)

Cause. The engine closed the connection before it finished answering, so that request was lost. The engine itself keeps running and serves the next request. On MLX, a known cause is mlx-lm's tool-call parser: when a model fills a list or object parameter of a tool with plain text instead of JSON (for example ramen restaurant where the tool expects ["ramen restaurant"]), mlx-lm 0.31.3 fails to parse the call and drops the request. A streamed reply that stops this way ends with this error, not with a finished but empty answer.

For a remote model, the server behind it closed the connection, and the message says so. A remote model's streamed reply is checked for this only when its registration has Report a reply cut off mid-stream (requireFinishReason) turned on. If that option is on and every streamed reply fails this way right after the full answer, the server doesn't send a finish reason, so turn it off.

Fix. Open the model's Logs: the engine's traceback names the cause. For the MLX tool-call case, give the tool a single string parameter in place of the list, or describe the expected JSON in the parameter's description. A model writes a single value reliably. A retry can succeed if the model writes the argument differently, but at temperature 0 it usually repeats the same call. The inference plane doesn't retry a request to a local engine, because one that failed this way usually fails the same way again. For a remote model, the server's own log, or a hosted provider's status page, has the reason.

modality_not_supported (404)

Cause. The engine answered the endpoint with a 404 — usually you called a modality the model can't do (embeddings against a chat-only model), or, for a remote openai-compatible model, the baseUrl is missing its /v1 suffix.

Fix. Use a model registered for the right modality, or fix the base URL. Hosted APIs usually need a trailing /v1. See Remote models.

A model download shows a "too large" fit badge, or loads slowly / fails

Cause. The variant's estimated memory need exceeds a safe fraction of your RAM. The fit badge warns before you install.

Fix. Pick a smaller quantization variant (a lower-precision quant), or a smaller model. The catalog marks one variant recommended — the largest that comfortably fits. If the card is marked MoE and you're on an Apple Silicon Mac, the MLX variant can run anyway with Stream experts from disk (offered in the install dialog). See Installing local models.

A model with expert streaming won't start, or is slow

Cause. Expert streaming works only for mixture-of-experts chat models in MLX format, and is refused when its engine starts for anything else — a model without experts, one whose experts are a kind it doesn't read, or a checkpoint that wasn't converted for MLX. The reason is in the model's engine log. When it does run, its speed depends on how many of the experts a reply needs are already in its cache, then on your SSD.

Fix. Open the model's Logs to read the refusal; for a model that can't stream, untick Stream experts from disk in its settings. If a streamed model is slow, give it a bigger Expert cache (GB), and expect the first reply to a long prompt to be slow from a cold disk. If the model fits in memory, it's faster with streaming off. See expert streaming.

logical_id_exists (409) when installing from the catalog

Cause. You chose a logical id that's already registered.

Fix. Pick a different logical id in the install dialog, or delete the existing registration first.

Text-to-speech returns a 502 "try rephrasing"

Cause. The speech engine aborted mid-synthesis or produced empty audio — some input texts trip the engine's text processing.

Fix. Rephrase or shorten the input text and try again. This is a clean, non-fatal error, not a crash.

Speech-to-text won't load the weights

Cause. The transcription engine expects whisper.cpp's native ggml-*.bin weights; a chat-style GGUF conversion won't work.

Fix. Point the model at the exact native weights file. See Engines.

Runs

A run is stuck at status waiting

Cause. The run reached a human node and is paused for input. This is expected — waiting runs are excluded from the crash sweep and can wait indefinitely.

Fix. Deliver the input: on the run detail page use the Resume panel, or POST /runs/{id}/resume with {input: …}. See Human-in-the-loop.

durable_required (400)

Cause. You tried to run an agent that contains a human, subgraph, loop, or map node — or you asked for a durable run — but the control plane isn't in durable mode.

Fix. Set THEYGENT_DURABLE=1 in .env and make restart, then run it with Run durably (or POST /agents/{id}/durable-runs). See Durable runs.

A run failed with "interrupted: control-plane restarted while run was in-flight"

Cause. The control plane restarted while a non-durable run was still executing. On startup, any run left created or streaming is swept to failed because its progress is unknown.

Fix. Re-run it. To make unattended runs survive restarts and resume from the last completed step, run them in durable mode — durable and waiting runs are excluded from the sweep.

template_error, router_error, or transform_error (422)

Cause. An input-reference problem. template_error = a $in token that didn't resolve (an unknown port or a missing field). router_error = a router's select resolved to no handle or an undeclared one — there is no fallback guessing. transform_error = a transform node's expression is invalid or references something absent.

Fix. Check the token against the node's actual ports. Remember the first segment after $in. is always a port name ($in.in.field drills into the default in port). See Referencing inputs.

The run completed but the output is empty (with an amber note)

Cause. The graph reached no output node on the taken branch, an upstream node errored, or a reasoning model spent its whole token budget thinking (finish_reason=length). The run stays completed and the note explains which.

Fix. If it's the token budget, raise maxTokens on the model binding (or Max tokens in chat). Otherwise check that your branches route to an output node. See Runs & sessions.

"a second output node executed — the run output would be ambiguous"

Cause. Two output nodes were live in the same run. TheYgent refuses to guess which one is the answer.

Fix. Route exclusive branches (with a router or guardrail) so at most one output node is reachable per run. Having multiple output nodes is fine as long as only one executes.

A tool node shows red in the waterfall but the run is green

This is normal, not a bug. A tool or MCP error is delivered as structured output on the node's err port and the run continues (completed). The node's bar shows red while the run stays green.

Fix (if you want it to stop the run). Wire the err port somewhere that handles the failure, rather than leaving it dead. See Tools.

Agents and publishing

version_conflict (409) when publishing

Cause. You tried to publish different content under a version string that already exists. Published versions are immutable.

Fix. Bump the version field in the editor toolbar and publish again. (Re-publishing identical content under the same version is a harmless no-op.)

agent_exists (409)

Cause. You tried to create a brand-new agent with an id that's already taken.

Fix. Publish it as a new version of that agent instead (POST /agents/{id}/versions). The editor does this automatically when you publish a graph whose id already exists. See Drafts & publishing.

Triggers, webhooks, and invocation

Webhook returns invalid_signature (401)

Cause. The X-TheYgent-Signature header didn't match. The signature must be the HMAC-SHA256 of the exact raw request body, keyed by the trigger's config.secret.

Fix. Sign the raw bytes you send (any reformatting of the JSON changes the signature). Both sha256=<hex> and a bare hex digest are accepted. Confirm you're using the right secret. See Triggers & webhooks.

invoke returns unauthorized (401)

Cause. POST /agents/{id}/invoke requires the bearer token THEYGENT_INVOKE_TOKEN, and it's either unset (the endpoint is closed by default) or your Authorization header doesn't match.

Fix. Set THEYGENT_INVOKE_TOKEN in .env, restart, and send Authorization: Bearer <that token>.

trigger_disabled (409) or trigger_not_found (404) on a webhook

Cause. The trigger is disabled, or the id isn't a webhook trigger.

Fix. Enable it (PATCH /triggers/{id} with {enabled: true}) or check the id. A schedule that stops firing after downtime is expected — schedules don't backfill missed windows.

Secrets and credentials

Connection secrets disappear after a restart

Cause. THEYGENT_SECRET_KEY is unset, so the secret store uses an ephemeral in-memory key (you'll have seen a loud warning at startup). Encrypted secrets don't survive a restart with an ephemeral key.

Fix. Generate a real Fernet key and set THEYGENT_SECRET_KEY in .env. To rotate later, supply a comma-separated list — the first key encrypts, all decrypt.

credential_error (502) from a remote model

Cause. A model's credentialRef (secret://NAME) couldn't be resolved — the named credential isn't in the local store and isn't a process environment variable.

Fix. Add the credential under Settings → Local credentials (or PUT /admin/credentials/{name}). Credential values are write-only and resolve on your machine only. See Remote models.

A tool with a connection binds an err at run time

Cause. The connection was deleted, disabled, or its secret can't be decrypted (for example after rotating away the key that encrypted it). Connections resolve at step time, not at save time, so this surfaces as a clean err — never a crash.

Fix. Re-enable or re-create the connection, or set the correct THEYGENT_SECRET_KEY.

Import and export

invalid_include (400) on export

Cause. POST /export was called with an empty include list or a value it doesn't know. The valid sections are agents, runs, traces, sessions, mcp, and rag.

Fix. Send a non-empty subset of those values. Note traces implies runs — the server adds them. See Import & export.

invalid_bundle (400) on import

Cause. The body isn't a bundle that plane understands. On the control plane (POST /import) it means the body isn't a JSON object carrying format_version; on the inference plane (POST /admin/import) it means the body is malformed or carries fields the plane doesn't recognize — unknown keys are rejected loudly rather than silently dropped.

Fix. Import the unmodified file an export produced — control-plane.json to :8080/import, inference/models.json to :8081/admin/import — or let the Settings Import / Export tab unpack the zip and route the halves for you.

unsupported_bundle_version (400)

Cause. The bundle's format_version is newer than this install understands — it was exported by a newer TheYgent.

Fix. Upgrade the importing install, then re-run the import.

The import report lists version_conflict warnings

Cause. The target already has that agent version with different content. Published versions are immutable, so the import refuses to overwrite it — that version is skipped while the rest of the bundle lands.

Fix. Usually nothing — the target's version wins by design. If you want the bundled graph too, publish it under a new version on the target.

An imported model warns weights_unavailable

Cause. The registration points at a file path from the exporting machine and carries no catalog-install provenance, so there is nothing to re-download. It is registered but will fail at first use.

Fix. Put the weights at that path on this machine, or delete the registration and reinstall the model under the same logical id. Catalog-installed models don't hit this — they re-download automatically.

The import report lists download_in_progress or params_path_dropped

Cause. download_in_progress: the model's weights are already downloading (usually from a previous run of the same import), so this run skipped it rather than start a duplicate download. params_path_dropped: the exported registration carried a machine-absolute auxiliary path (such as a vision projector file) from the old machine; the import dropped it so the model can start here — the engine finds the file next to the re-downloaded weights instead.

Fix. Nothing for either — wait for the in-flight download to finish (notification center), and the dropped path re-resolves automatically. If a vision model can't find its projector after import, re-set it in the registration's parameters.

The import report lists invalid_section

Cause. A top-level section of the bundle (runs, sessions, …) isn't the JSON list the format defines — the file was hand-edited or truncated. The section was skipped; everything else imported.

Fix. Re-export and import the unmodified file.

An imported webhook trigger doesn't fire

Cause. Webhook signing secrets never travel in a bundle, so webhook triggers import disabled and secretless — an armed webhook with no secret would reject every caller.

Fix. Set a fresh secret with PATCH /triggers/{id} (body {"config": {"secret": "..."}}), then enable the trigger. The same rule explains imported connections and MCP servers failing with auth errors: re-enter their secrets and env values — only key names were exported.

AI gateway

Claude Code shows an error from the TheYgent AI gateway

The gateway names the reason in its message (and in the x-theygent-gateway-error header): a missing or revoked gateway key, no Anthropic credential where you are billed to your own, no organization key on the gateway, or Anthropic unreachable. Each code and its fix: errors the gateway returns.

The AI Gateway page shows no gateway, or no traffic

A gateway appears on the Overview only when it is attached to the control plane — THEYGENT_CONTROL_PLANE_URL and a service key (Running the gateway). An attached gateway records nothing until an admin saves a configuration in Observer mode on the Settings tab; and Claude Code reaches it only with ANTHROPIC_BASE_URL pointing at it (Set up Claude Code).

The Settings tab says it can't reach the gateway

The organization's key is read and written on the gateway itself, from your browser. Set the gateway's address on Setup & keys, and if the interface is served from another origin than http://localhost:5174, add that origin to THEYGENT_AI_GATEWAY_ADMIN_ORIGINS.

Requests I expected on a local model went to Anthropic

Open the request on the Sessions tab: its reasons say why. No local model fits: no listed model takes tool calls, images or a prompt that size — list one that does, with its real context. Outgrew local model: the conversation grew past every listed model that could take it — list a larger one, and make sure developers use the current setup from Setup & keys, which tells Claude Code to compact in time. Target refused: the local model could not take it — an error, no answer in time, an unusable answer, a document or server tool it can't be given, or no inference plane at THEYGENT_INFERENCE_PLANE_URL; the conversation then stays with Anthropic until its next cold point. Would route → a local model: the gateway is in observer mode, which only records. No local reason at all: the conversation is outside the share, or it started before the share was raised and moves at its next cold point. Some requests never go local, such as Claude Code's quota check, and compaction goes local only with its conversation. See Local models.

Claude Code shows a higher cost than the Report

In router mode Claude Code prices every token at the model it asked for, even when a cheaper model or a local one answered. The Report prices what actually ran, and shows Claude Code's figure as without routing. See Claude Code's own cost display.

The Report warns that tool search is off

Claude Code turns MCP tool search off for any custom base URL and then sends every tool's definition with every request. Set ENABLE_TOOL_SEARCH=true (the desktop app: tool search on). See why each setting.

Browser and interface

The interface can't reach a plane / CORS errors in the console

Cause. Your browser origin isn't in the allowed CORS list, or a plane is on a non-default port the interface wasn't built to call.

Fix. Add your origin to THEYGENT_CORS_ORIGINS (comma-separated), and if you moved a plane's port, rebuild the interface with the matching VITE_CONTROL_PLANE_URL / VITE_INFERENCE_URL. See Environment variables.

The model catalog is empty — "No local engine is ready"

Cause. The catalog only lists models you can actually run, and no local engine is installed yet.

Fix. Install at least one engine (make engines on macOS, or llama.cpp on any platform), then refresh. See Installing local models.