> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon SageMaker

export const GCell = ({children, className}) => <div className={`grid-table-cell ${className || ""}`} role="cell">
    {children}
  </div>;

export const GTH = ({children, className}) => <div className={`grid-table-th ${className || ""}`} role="columnheader">
    {children}
  </div>;

export const GRow = ({children}) => <div className="grid-table-row" role="row">{children}</div>;

export const GBody = ({children}) => <div className="grid-table-body" role="rowgroup">{children}</div>;

export const GHead = ({children}) => <div className="grid-table-head" role="rowgroup">{children}</div>;

export const GTable = ({children, className, cols}) => <div className={`grid-table not-prose overflow-hidden rounded-2xl ${className || ""}`} style={{
  "--grid-table-cols": cols
}} role="table">
    {children}
  </div>;

The SageMaker integration lets you deploy models from the Anaconda Platform model catalog to Amazon SageMaker real-time endpoints using a custom inference container. The container serves an OpenAI-compatible API backed by [llama.cpp](https://github.com/ggml-org/llama.cpp) and handles SageMaker's `/ping` and `/invocations` protocol.

## Prerequisites

* Docker with NVIDIA GPU support ([nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))
* Access to Anaconda Platform and an [API key](/docs/anaconda-platform/user/api-keys)
* An AWS account with SageMaker access
* A SageMaker execution IAM role
* An Amazon ECR repository to push the container image to

## Install

Install the `sagemaker` extra for `anaconda-ai`:

```bash theme={null}
pip install 'anaconda-ai[sagemaker]'
```

## Build the container

Clone the `anaconda-sagemaker-runtime` repository and build the image:

```bash theme={null}
git clone https://github.com/anaconda/anaconda-sagemaker-runtime
cd anaconda-sagemaker-runtime

docker build \
  -t anaconda-sagemaker-runtime:latest \
  -f Dockerfile \
  .
```

## Push to ECR

Authenticate with ECR, then tag and push the image:

```bash theme={null}
AWS_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
AWS_REGION=us-east-1
ECR_REPO=anaconda-sagemaker-runtime

# Create the repository if it does not exist
aws ecr create-repository \
  --repository-name ${ECR_REPO} \
  --region ${AWS_REGION} 2>/dev/null || true

# Authenticate
aws ecr get-login-password --region ${AWS_REGION} \
  | docker login --username AWS --password-stdin \
    ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com

# Tag and push
docker tag anaconda-sagemaker-runtime:latest \
  ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest

docker push \
  ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest
```

## Deploy and invoke

<Tabs>
  <Tab title="Python SDK">
    Use `AnacondaModel` to deploy a model from the catalog and invoke the endpoint:

    ```python theme={null}
    from anaconda_ai.integrations.sagemaker import AnacondaModel
    import json

    IMAGE_URI = "<account_id>.dkr.ecr.<region>.amazonaws.com/anaconda-sagemaker-runtime:latest"

    model = AnacondaModel(
        model_id="Qwen2.5-7B-Instruct/Q4_K_M",
        image_uri=IMAGE_URI,
    )

    endpoint = model.deploy(instance_type="ml.g5.2xlarge")

    response = endpoint.invoke(
        body=json.dumps({
            "messages": [{"role": "user", "content": "What is conda?"}],
            "max_tokens": 256,
        }),
        content_type="application/json",
    )
    print(json.loads(response.body))

    endpoint.delete()
    ```

    <Note>
      Deleting an endpoint does not delete the underlying deployable model. The model registration persists in SageMaker and can be redeployed at any time from the AWS console or using native AWS tools. You only need to run `model.stage()` and `model.build()` once per model.
    </Note>

    To pre-stage the model to S3 for a faster cold start (approximately 4 minutes versus 6 minutes), pass `stage=True`:

    ```python theme={null}
    endpoint = model.deploy(instance_type="ml.g5.2xlarge", stage=True)
    ```

    You can also call the staging, registration, and deployment steps explicitly:

    ```python theme={null}
    model.stage()   # upload GGUF to S3 via CodeBuild
    model.build()   # register SageMaker Model resource
    endpoint = model.deploy(instance_type="ml.g5.2xlarge")
    ```
  </Tab>

  <Tab title="CLI">
    ```bash theme={null}
    IMAGE_URI="<account_id>.dkr.ecr.<region>.amazonaws.com/anaconda-sagemaker-runtime:latest"

    # Deploy — container downloads model from the catalog at startup
    anaconda ai deploy Qwen2.5-7B-Instruct/Q4_K_M --image-uri $IMAGE_URI

    # Deploy with S3 staging for faster cold start
    anaconda ai deploy Qwen2.5-7B-Instruct/Q4_K_M --image-uri $IMAGE_URI --stage

    # Register model only, without creating an endpoint
    anaconda ai deploy Qwen2.5-7B-Instruct/Q4_K_M --image-uri $IMAGE_URI --build-only

    # Stage a model to S3 without deploying
    anaconda ai stage Qwen2.5-7B-Instruct/Q4_K_M

    # List staged models
    anaconda ai stage --list
    ```

    Run `anaconda ai deploy --help` to see all available options.
  </Tab>
</Tabs>

## llama-server tuning

Use these options to configure the inference server's context size, parallelism, and attention behavior.

<Tabs>
  <Tab title="Python SDK">
    ```python theme={null}
    model = AnacondaModel(
        model_id="Qwen2.5-7B-Instruct/Q4_K_M",
        image_uri=IMAGE_URI,
        ctx_size=16384,
        parallel=8,
        flash_attn=True,
        cache_type_k="q8_0",
    )
    ```
  </Tab>

  <Tab title="CLI">
    ```bash theme={null}
    anaconda ai deploy Qwen2.5-7B-Instruct/Q4_K_M --image-uri $IMAGE_URI \
        --ctx-size 16384 --parallel 8 --flash-attn --cache-type-k q8_0
    ```
  </Tab>
</Tabs>

## Environment variables

### Required for catalog download

<Note>
  The following environment variables are required when the container downloads the model from the catalog at startup.

  ***

  When using `stage=True` or `model.stage()`, the model is pre-staged to S3 and these variables are not required.
</Note>

<GTable cols="35% 65%">
  <GHead>
    <GRow>
      <GTH>Variable</GTH>
      <GTH>Description</GTH>
    </GRow>
  </GHead>

  <GBody>
    <GRow>
      <GCell>`ANACONDA_MODEL_ID`</GCell>
      <GCell>Model and quantization. For example, `Qwen2.5-7B-Instruct/Q4_K_M`</GCell>
    </GRow>

    <GRow>
      <GCell>`ANACONDA_AUTH_API_KEY`</GCell>
      <GCell>Your Package Security Manager (On-prem) API key</GCell>
    </GRow>
  </GBody>
</GTable>

### Optional

<GTable cols="30% 15% 55%">
  <GHead>
    <GRow>
      <GTH>Variable</GTH>
      <GTH>Default</GTH>
      <GTH>Description</GTH>
    </GRow>
  </GHead>

  <GBody>
    <GRow>
      <GCell>`ANACONDA_AUTH_DOMAIN`</GCell>
      <GCell>`anaconda.com`</GCell>
      <GCell>Anaconda API domain</GCell>
    </GRow>

    <GRow>
      <GCell>`LLAMA_ARG_CTX_SIZE`</GCell>
      <GCell>`8192`</GCell>
      <GCell>Context window size in tokens</GCell>
    </GRow>

    <GRow>
      <GCell>`LLAMA_ARG_PARALLEL`</GCell>
      <GCell>`4`</GCell>
      <GCell>Number of parallel inference sequences</GCell>
    </GRow>

    <GRow>
      <GCell>`LLAMA_ARG_N_GPU_LAYERS`</GCell>
      <GCell>`999`</GCell>
      <GCell>GPU layers to offload (`999` = all)</GCell>
    </GRow>

    <GRow>
      <GCell>`INFERENCE_TIMEOUT_SECONDS`</GCell>
      <GCell>`300`</GCell>
      <GCell>Maximum seconds to wait for a response</GCell>
    </GRow>

    <GRow>
      <GCell>`HEALTH_TIMEOUT_SECONDS`</GCell>
      <GCell>`600`</GCell>
      <GCell>Maximum seconds to wait for llama-server startup</GCell>
    </GRow>

    <GRow>
      <GCell>`LOG_REQUEST_BODY`</GCell>
      <GCell>`0`</GCell>
      <GCell>Set to `1` to log request bodies to CloudWatch Logs</GCell>
    </GRow>
  </GBody>
</GTable>

<Warning>
  Enabling `LOG_REQUEST_BODY` writes all prompts to CloudWatch Logs. Ensure log retention is configured appropriately for your data policy.
</Warning>

Any [llama-server argument](https://github.com/ggml-org/llama.cpp/tree/master/tools/server#usage) that supports an environment variable can be passed as `LLAMA_ARG_*`. For example: `LLAMA_ARG_BATCH=4096`, `LLAMA_ARG_FLASH_ATTN=on`.

## Supported instance types

GPU is required. Recommended instance families:

<GTable cols="20% 15% 15% 50%">
  <GHead>
    <GRow>
      <GTH>Instance</GTH>
      <GTH>GPU</GTH>
      <GTH>VRAM</GTH>
      <GTH>Suitable for</GTH>
    </GRow>
  </GHead>

  <GBody>
    <GRow>
      <GCell>`ml.g4dn.xlarge`</GCell>
      <GCell>Tesla T4</GCell>
      <GCell>16 GB</GCell>
      <GCell>Models up to \~8B Q4</GCell>
    </GRow>

    <GRow>
      <GCell>`ml.g5.2xlarge`</GCell>
      <GCell>A10G</GCell>
      <GCell>24 GB</GCell>
      <GCell>Models up to \~13B Q4</GCell>
    </GRow>

    <GRow>
      <GCell>`ml.g5.12xlarge`</GCell>
      <GCell>4× A10G</GCell>
      <GCell>96 GB</GCell>
      <GCell>Models up to \~70B Q4</GCell>
    </GRow>

    <GRow>
      <GCell>`ml.g5.48xlarge`</GCell>
      <GCell>8× A10G</GCell>
      <GCell>192 GB</GCell>
      <GCell>Largest models</GCell>
    </GRow>
  </GBody>
</GTable>

## Model identifier format

For models in the Anaconda Platform model catalog, models are identified using the format:

```text theme={null}
<ModelName>/<Quantization>
```

Anaconda Platform supports the following quantization methods: `Q4_K_M`, `Q5_K_M`, `Q6_K`, `Q8_0`.

Examples:

```text theme={null}
Qwen2.5-7B-Instruct/Q4_K_M
Llama-3.2-3B-Instruct/Q5_K_M
```

See the Model Catalog for available models and their quantizations.

## Next steps

Once your endpoint is in service, you're back to standard AWS. You can monitor and manage the endpoint, view startup and inference logs in CloudWatch, and configure auto-scaling or other post-creation settings using the AWS CLI or Boto3. See the [Amazon SageMaker real-time inference documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) for details.