anaconda ai launch

Usage

anaconda ai launch <MODEL> [OPTIONS] 

Description

Launches an OpenAI-compatible inference server for a quantized model. If the model has not been downloaded, it is downloaded automatically before the server starts.

By default, the server runs in the foreground and is stopped and removed when you press Ctrl+C. Use --detach to leave the server running in the background after the command exits.

Arguments

Options

Server options

Additional server options can be appended as --key=value pairs or --key boolean flags. These options are passed directly to the backend server.

Server options are backend-specific. The options below apply to the ai-navigator and anaconda-desktop backends and map to llama-server parameters. The ai-catalyst backend may support different options.

Common server options for ai-navigator and anaconda-desktop include:

Examples

Launch a server in the foreground

anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M

Launch in the background (detached)

anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M --detach

The command returns to the terminal immediately while the server runs in the background.

Launch with custom server options

anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M --ctx_size=512 --jinja

Launch and open the server in a browser

anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M --show