> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LiteLLM

This provides a CustomLLM provider for use with `litellm`. Since litellm does not currently support [entrypoints](https://github.com/BerriAI/litellm/issues/7733) to register the provider, you must import the module before use.

```py theme={null}
import litellm
import anaconda_ai.integrations.litellm

response = litellm.completion(
    'anaconda/openhermes-2.5-mistral-7b/q4_k_m',
    messages=[{'role': 'user', 'content': 'what is pi?'}]
)
```

To use an already running server, use `anaconda/server/<server-name>` as the model:

```python theme={null}
response = litellm.completion(
    'anaconda/server/my-server',
    messages=[{'role': 'user', 'content': 'what is pi?'}]
)
```

This integration supports `litellm.completion()` for both standard and streamed completions (`stream=True`). Most OpenAI-compatible inference parameters are available and behave as expected, with the exception of the `n` parameter (for multiple completions), which is not currently supported.

You can also configure server behavior using the `optional_params` argument. Pass server configuration options under the `"server"` key:

```py theme={null}
response = litellm.completion(
    'anaconda/openhermes-2.5-mistral-7b/q4_k_m',
    messages=[{'role': 'user', 'content': 'what is pi?'}],
    optional_params={
        "server": {
            "main_gpu": 1,
            "port": 9999,
            "ctx_size": 4096,
            "n_gpu_layers": 20,
            "temp": 0.7
        }
    }
)
```

For more information on using the `litellm` package, see the [official documentation](https://docs.litellm.ai/docs//).