> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Model servers
export const Comments = ({children}) => {
return ;
};
export const TroubleshootSolution = ({children}) => <>
Solution
{children}
;
export const TroubleshootCause = ({children}) =>
Cause
{children}
;
export const TroubleshootTitle = ({children}) => <>
{children}
;
export const Troubleshoot = ({children}) => ;
The built-in API server enables you to run open-source large language models (LLMs) locally on your own machine. It facilitates direct interaction with your models through API calls, providing a powerful tool for testing your application's AI workflows without the need for external cloud services. By hosting your own API server locally, you have full control over the model's behavior and maintain the privacy and security of your data.
Prefer working on the command line? Use the [Anaconda AI CLI](/docs/cli-reference/anaconda-ai/getting-started) to manage your servers.
## Understanding the API server
The API server is the core component of Anaconda Desktop that enables you to interact with your locally downloaded LLMs through API calls. Access the API server by selecting **Model Servers** from the left-hand navigation.
Hover over a tooltip to view information about the API Server's configurable parameters.
### Server address
Your local server address, often referred to as `localhost` or `127.0.0.1`, is the default address for the API server. It is a loopback address, meaning any network communication sent to this address remains within the same machine, keeping your data secure and private. This is common for developing applications.
`localhost` and `127.0.0.1` are semantically identical.
If you are working in an office network and would like to make your server accessible to other devices within the same network, you can set the server address to your machine's local IP address. These are typically private networks, meaning they're not routable on the public internet.
Setting the server address to `0.0.0.0` configures it to accept connections on all network interfaces. This can expose your server to the public internet if your network is configured to allow external connections and route traffic from external sources to your server.
### Server port
The server port tells the built-in API server where to listen for incoming traffic (that is, where to listen and pick up API requests).
For more information, see [Cloudflare's article on ports](https://www.cloudflare.com/learning/network-layer/what-is-a-computer-port/).
Your base URL for communicating with the API server combines the server address and port number from the **Model Servers** page. For example, `http://localhost:8080/`.
### API key
Establish an API key that must be passed as an `Authorization: Bearer` token in the header of requests made to your server. You can choose any unique string you want as the API key, but the value you enter on the **Model Servers** page must match what you use in the `Authorization: Bearer` header.
To test an API key:
1. Select **Model Servers** from the left-hand navigation.
2. Enter your API key in the **API Key** field.
3. [Load a model into the API server](#loading-a-model).
You must use a `text-generation` type model.
4. Open [Anaconda Prompt](/docs/reference/glossary#anaconda-prompt) (Terminal on macOS/Linux) and run the following command:
```sh Anaconda Prompt theme={null}
curl --request POST ^
--url http://localhost:8080/completion ^
--header 'Authorization: Bearer ' ^
--header "Content-Type: application/json" ^
--data '{
"prompt": "Hello, how are you?"
}'
```
```sh Anaconda PowerShell Prompt theme={null}
curl --request POST `
--url http://localhost:8080/completion `
--header 'Authorization: Bearer ' `
--header "Content-Type: application/json" `
--data '{
"prompt": "Hello, how are you?"
}'
```
```sh Terminal theme={null}
curl --request POST \
--url http://localhost:8080/completion \
--header 'Authorization: Bearer ' \
--header "Content-Type: application/json" \
--data '{
"prompt": "Hello, how are you?"
}'
```
Replace \ with the value you entered on the **Model Servers** page.
This command will reach out to the local API server's `/completion` endpoint to interact with the model you have loaded. If you receive a response, your API key is working. If you receive a 401 error, double check your command and try again.
### Loading a model
Use the **Model** dropdown to select the model you want to load into the API server and the **File** dropdown to choose the quantization level to interact with.
### Model won't load (exit code 3221225781)
The model you are trying to load requires a C++ runtime environment, but Microsoft does not include one in their operating systems by default.
Install the Latest [Microsoft Visual C++ Redistributable Version](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170#latest-microsoft-visual-c-redistributable-version).
## Starting the server
To start the API server, click **Start**.
To stop the API server, click **Stop**.
## Viewing API server logs
The API server records all incoming traffic for the currently running server and displays relevant information in the **Server Logs**. To view your server logs, click **Logs** on the **Model Servers** page.
The server logs provide information for the following metrics:
* **System information**: Provides information about your system's hardware capabilities.
* **Build information**: Provides information about the version of the API server you are using.
* **Chat template**: Shows the sequence of messaging defined for the system.
* **Server listening**: Displays the server address and port number being listened to.
* **Slot information**: Displays the number of available slots for the server. Each slot is able to manage one user API request at a time.
* **Task information**: Displays information such as time spent processing and responding to a request, the request task ID, and which slot fulfilled the task.
Once the server is stopped, the log is saved to the following location with a timestamped filename (for example, `_api-server.log`), so you can efficiently locate specific server logs if necessary:
For Windows machines, you can find the logs here: `C:\Users\\AppData\Roaming\anaconda-desktop\logs`
For macOS machines, you can find the logs here: `~/Library/Logs/anaconda-desktop/`
## Connecting applications to servers
Connecting your application to a model server requires setting two environment variables: the `base_url`, which directs your API calls to the server hosting your model, and the `api_key`, which authenticates your requests.
Follow the steps below to find these variables' values and save them as environment variables in your application.
From the **Model Servers** page:
1. Note your **Server Address** and **Server Port** (the default is `http://localhost:8080`). This will be your `base_url`.
2. (Optional) Set an **API Key**. You can use any unique string value. If you don't set an API key, you don't need to provide an `api_key` in your application.
3. [Load a model](#loading-a-model) and click **Start**.
Set your credentials as environment variables:
```powershell Windows Powershell theme={null}
$env:MODEL_SERVER_BASE_URL=""
$env:ANACONDA_API_KEY=""
```
```bash macOS/ Linux terminal theme={null}
export MODEL_SERVER_BASE_URL=""
export ANACONDA_API_KEY=""
```
Replace \ with the server address and port you copied from the **Model Servers** page.
Replace \ with the API key you created on the **Model Servers** page.
Setting environment variables is considered a best practice for security. However, for local testing and development, you can add the credentials directly in your code instead.
***
Environment variables set directly in the terminal will only last as long as the terminal session is active. To make these environment variables permanent, follow your system's standard process for persisting environment variables.
### OpenAI API compatibility
Anaconda Desktop model servers are OpenAI API compatible for chat completions. See OpenAI's [Chat Completion API documentation](https://developers.openai.com/api/reference/chat-completions/overview) for more information on making chat completion requests.
Make sure the model server is [running](#starting-the-server) before making API requests.
***
The `model` parameter is required but can be left as an empty string since the model is already loaded on the server.
```python theme={null}
from openai import OpenAI
import os
# Initialize client with environment variables
client = OpenAI(
base_url=os.environ.get('MODEL_SERVER_BASE_URL'),
api_key=os.environ.get('ANACONDA_API_KEY')
)
# Make a chat completion request example
response = client.chat.completions.create(
model="",
messages=[{"role": "user", "content": "Hello!"}],
max_completion_tokens=150,
temperature=0.7
)
print(response.choices[0].message.content)
```
If you did not set an API key, change `api_key=os.environ.get('ANACONDA_API_KEY')` to `api_key=""`.