LlamaIndex

Install the llama-index-llms-openai package:

pip install llama-index-llms-openai

Only pip install packages in your conda environment once all other packages and their dependencies have been installed. For more information on installing pip packages in your conda environment, see Installing pip packages.

Here is a minimal setup example for using LlamaIndex with Anaconda AI:

from anaconda_ai.integrations.llama_index import AnacondaModel

llm = AnacondaModel(
    model='OpenHermes-2.5-Mistral-7B_q4_k_m'
)

# Or with custom server configuration:
llm = AnacondaModel(
    model='OpenHermes-2.5-Mistral-7B_q4_k_m',
    temperature=0.7,
    extra_options={
        'ctx_size': 4096,
        'n_gpu_layers': 20,
        'port': 8080
    }
)

To use an already running server, pass server/<server-name> as the model:

llm = AnacondaModel('server/my-server')

The AnacondaModel class supports the following arguments:

For more information on using the llama-index-llms-openai package, see the official documentation.