ONNX Runtime

Runs ONNX-format models on your own machine through an onnxruntime server you provide. It has no panel of its own; once it is installed, an ONNX model you add as an AI core in LLM Capabilities can be started and used.

What you can do

• Chat with an ONNX model by starting it and making it the active core.
• Embed text with an ONNX embedding model.
• Rerank passages against a question with an ONNX reranking model.
• Transcribe speech, when your server offers that route.
• Describe an image with a vision model, when your server offers that route.
• Stop the server when you are done, so later requests say nothing is running instead of failing against a dead address.

Getting started

1. Set the llm.onnx.serverPath setting to the script or program that starts your OpenAI-compatible onnxruntime server, and llm.onnx.serverArgs to its arguments. The arguments can use {model}, {port} and {host}.
2. Open LLM Capabilities, choose Add Local Model, then Choose a weights file… and pick your .onnx file. The panel says which engine will run it. Press Add as an AI core.
3. In AI Cores, choose Start on the new core. The runtime launches your server with the model and waits until it answers.

Requirements

• The desktop app. The local server cannot run in the browser version.
• LLM Capabilities, where ONNX models are added and started, and Ollama Library, whose panel the Add Local Model entry opens.
• An OpenAI-compatible onnxruntime server program or script. This extension starts it, and does not include one.

Good to know

• By default the server is started on 127.0.0.1, port 8090. The llm.onnx.host and llm.onnx.port settings change that.
• Starting twice returns the server that is already running rather than putting a second one on the same port.
• Speech-to-text sends the audio as base64 inside JSON, so it only works with a server that accepts audio that way.
• Starting the server from an assistant or a flow goes through Studio's approval rules, because it starts a process on your machine.