llama-swap

Use it when

• You run several local models with llama.cpp, vllm, or tabbyAPI but cannot keep them all loaded, and want one endpoint that starts and swaps the right server per request.
• You want a single OpenAI and Anthropic compatible base URL for your local stack, with ttl-based auto-unload, profiles, a web UI, and log streaming instead of manually restarting inference servers.

What it solves

Not the fit when

• Not an inference engine; it proxies to upstream servers like llama-server, vllm, or stable-diffusion.cpp.
• MacPorts and WinGet packages are community-maintained, not official.
• running inference itself without an upstream server
• cloud provider api key management
• prompt or output validation
• fine tuning workflows

Install

brew tap mostlygeek/llama-swap && brew install llama-swap

Invoke

Write a config.yaml mapping model IDs to server commands (e.g. cmd: llama-server --port ${PORT} --model /path/to/model.gguf), run 'llama-swap --config config.yaml --listen localhost:8080', then send OpenAI or Anthropic API requests; the proxy starts or swaps the right upstream based on the model field.

Alternatives

No reviewed alternatives recorded yet.