Serving the local frontier.

halogen™ is local inference that behaves like a frontier API. Qwen3.8-Flash-Next and a set of small NPU models, served from one AMD Strix Halo machine over the APIs your agents already speak.

Release 0.18.17

mkdir -p ~/halogen-models

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.18.17

Set the BIOS carve-out (UMA frame buffer) to its minimum, not Auto: this engine wants host RAM, not dedicated VRAM. The weights, about 111 GiB, are fetched on first start. The server answers on port 8731, OpenAI-compatible.

Read the Quickstart

Measured

An agent turn, start to last byte4.2 sContext of 30,000 to 96,000 tokens. Median of 115 turns, about 200 tokens of answer each.

Prefill, 8,192-token prompt1,821 tok/sThe image's own sweep, synthetic prompt, prompt cache off.

Decode, repetitive outputup to 94 tok/s73 writing code, 54 prose. Decode depends on how predictable the text is.

All three: published 0.18.0 image, default configuration (v2 checkpoint, prompt cache on, greedy, thinking off), AMD Ryzen AI Max+ 395, 128 GB, about 85 W of sustained package power. The agent turns are three sessions of a coding agent fixing a bug in a Python repository, on the client's clock. Every table, with its conditions

One port

Built for agents

The APIs your harness already speaks, on one port.

From an agent harness
Using it as a classifier

OpenAI API: /v1
Chat Completions and Completions, streaming, tool calls, structured output.
API reference
Codex and Responses: /v1/responses
The OpenAI Responses API. Point the Codex CLI at the server and it works, tool calls included.
Codex guide
Claude Code and Messages: /v1/messages
The Anthropic Messages API, built from what Claude Code actually sends.
Claude Code guide
Sessions: keep, fork, join: A harness that runs several branches from one conversation can keep its state under a name and start new requests from it.
Sessions

Ryzen AI NPU

On the NPU, same port

Small models run beside the Flash model. A request picks one by its model. Leave HALOGEN_NPU_MODELS out and nothing changes.

The full roster
Small models on the NPU

Decidedecider-0.8b d3-lite d3-nano
Embedqwen3-embedding-0.6b embeddinggemma-2
Rerankqwen3-reranker-0.6b
Moderateqwen3guard-gen-0.6b qwen3guard-stream-0.6b
Textqwen3.5-2b
Imagesflux2-klein-4b
Speech to textqwen3-asr-0.6b qwen3-asr-1.7b
Text to speechqwen3-tts-0.6b-customvoice qwen3-tts-1.7b-customvoice

From the changelog

Patch notes

Newest first. The changelog gives release numbers, not dates.

The whole changelog

0.18.17
With a keep in use, a full server no longer hands one conversation another's turn. Claude Code's MCP tool search works, tool arguments that hold tool-call markup arrive whole, a short request starts during a long prompt while /metrics is scraped, and /v1/models says whether the model takes images. The Flash checkpoints are unchanged.
0.18.16
The automatic reply-end keep is off by default again, a conversation the server moves aside to make room keeps its whole history, and generation with drafting is back to full speed. The Flash checkpoints are unchanged.
0.18.15
Long answers no longer stop inside a code block, and a request whose client has gone no longer holds up the requests behind it. The Flash checkpoints are unchanged.

System requirements

Machine: An AMD Strix Halo machine: Ryzen AI Max+ 395, 128 GB. One GPU, gfx1151 only; the build rejects other architectures. Built to own its machine: the weights stay resident and the KV pool is reserved up front, which on a 128 GB machine leaves roughly twelve gigabytes free.
Operating system: Native Linux only. WSL2 is not a supported host.
Kernel: AMD's two gfx1151 fixes to the KFD driver, with CONFIG_HSA_AMD_SVM built in. AMD's floor is Linux 6.18.4, Ubuntu 24.04 HWE 6.17.0-19.19~24.04.2, or Ubuntu 24.04 OEM 6.14.0-1018, each or later.
BIOS: The UMA frame buffer (dedicated graphics memory) at its smallest explicit value, not Auto.
Containers: Podman with --group-add keep-groups. On Docker, use --group-add video --group-add render instead.
Disk: The first start fetches the weights, about 111 GiB.
NPU models (optional): The NPU driver, XRT with its NPU plugin, the NPU firmware, and the IOMMU on. The GPU's fabric clock held at its top speed, or the server refuses to start and prints the commands for your machine.

Sources: host settings, a machine of its own, NPU host details, Quickstart, limits.