Servers

The .servers accessor provides methods for creating, listing, starting, and stopping servers.

Some methods like .match() and the parameter classes below are backend-specific and may not be available on all backends.

Creating servers

The .create() method creates a new server configuration. By default, it downloads the model file (if not already downloaded) and selects a random, unused port for the server.

If a server with the specified configuration is already running, the existing configuration is returned, and no new server is created.

Create servers using one of the following approaches:

Using model string reference

from anaconda_ai import AnacondaAIClient

client = AnacondaAIClient()
server = client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M')

Using QuantizedFile object

from anaconda_ai import AnacondaAIClient

client = AnacondaAIClient()
quant = client.models.get('OpenHermes-2.5-Mistral-7B').get_quantization('Q4_K_M')
server = client.servers.create(quant)

Using server reference

from anaconda_ai import AnacondaAIClient

client = AnacondaAIClient()
# Get existing server ID from client.servers.list()
server = client.servers.create('server/my-server-name')  # ai-catalyst
# or
server = client.servers.create('server/abc-123-uuid')    # ai-navigator/anaconda-desktop

Server reference behavior by backend:

ai-catalyst supports named servers. Use server/<server-name> to reference existing servers by name.

ai-navigator and anaconda-desktop:

• Use server/<uuid> to reference existing servers by UUID (obtained from client.servers.list())
• Have automatic server matching. When calling client.servers.create('<model>/<quant>', extra_options=...), these backends automatically search for running servers with the same model and options. If a match is found, the existing server is returned instead of creating a new one.

Server configuration parameters

The .create() method accepts an extra_options parameter to customize server behavior:

def create(
    model: str,
    download_if_needed: bool = True,
    extra_options: Optional[Dict[str, Any]] = None,
    show_progress: bool = True,
    console: Optional[Console] = None
) -> Server:

Server configuration options are backend-specific. The parameters below apply to the ai-navigator and anaconda-desktop backends.

Common server options include:

For example, to create a server with custom configuration:

server = client.servers.create(
    'OpenHermes-2.5-Mistral-7B/Q4_K_M',
    extra_options={
        'host': '127.0.0.1',
        'port': 9999,
        'ctx_size': 4096,
        'n_gpu_layers': 20,
        'temp': 0.7,
        'top_p': 0.9
    },
    download_if_needed=False
)

Managing servers

New servers are not automatically started when their configuration is created. You can start or stop a server using the following methods:

Using the server object

    server.start()
    server.stop()

Using the .servers accessor

    # Start or stop the server object returned from .create
    client.servers.start(server)
    client.servers.stop(server)

Using server as a context manager

This approach automatically starts and stops the server:

    with client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M') as server:
        openai_client = server.openai_client()
        # Make requests to the server

Server attributes