GGUF Runtime

Runs GGUF-format models on your own machine through llama.cpp. Install it if you want local models instead of remote ones. It has no panel of its own — once installed, GGUF models simply become runnable.

What you can do

• Run a GGUF model locally for chat, without sending anything to a remote service
• Start a model straight from the Models panel, with the local server launched for you
• Point the runtime at your own llama-server build and your own .gguf files
• Bring a local model up or down from an assistant or a flow, instead of clicking through the Models panel

Requirements

• Desktop (Electron) app — the local server can't run in the browser version.
• A llama-server binary from llama.cpp.

Settings

Setting	Description
llm.llamaServer.path	Path to your llama-server (llama.cpp) executable.
llm.gguf.path	Path to a .gguf model file, used when a model has no downloaded file of its own.

Starting and stopping it from an agent

The server lifecycle is also available as three capabilities, so an assistant or a flow can bring a local model up without going through the Models panel.

Capability	What it does
llms.gguf.server.start	Starts llama-server and points local model calls at it. Starting a process is high risk, so this one asks for approval every time.
llms.gguf.server.stop	Stops the server and releases the local address, so later calls report that nothing is running instead of failing against a dead one.
llms.gguf.server.status	Reports whether the server is up, and where. Reads only.

The host runs one llama-server at a time, so stopping is host wide. The address is released only when it still belongs to this runtime, which means stopping GGUF never disconnects a running ONNX server.

Getting started

1. Point llm.llamaServer.path at a llama-server executable built from llama.cpp.
2. Start a GGUF model from the Models panel. The runtime launches the local server and serves the model for chat.