> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks integration

Integrating Databricks with Anaconda Package Security Manager enables organizations to maintain security and compliance while leveraging the power of both platforms.

For data science teams working in regulated environments, this integration provides essential security controls over Python package usage. Your organization can enforce security policies and maintain consistent environments across development and production. This helps prevent the use of unauthorized or vulnerable packages while providing comprehensive audit trails of package usage across your Databricks workspaces.

This guide explains how to set up a secure, customized Python environment in Databricks using packages from Anaconda's Package Security Manager.

## Prerequisites

Before starting, ensure you have:

* Administrator permissions for your Package Security Manager instance
* Docker installed on your local machine
* A Databricks workspace with admin privileges

## Setup and configuration

**1. Create a Policy**

1. Log in to your Package Security Manager instance.
2. Select  **Policies** from the left-hand navigation.
3. Click  **Create Policy**.
4. Name your policy `databricks`, provide a brief description, and then click **Next**.
5. Set your policy to allow packages with `noarch` or `linux-64` architectures only, then click **Next**.
6. Skip setting CVE rules and exclusions by clicking **Next** to progress the policy configuration.
7. Click **Create Policy** to save your policy.

> [!TIP]
>
> For more information on policies, see [Policies](https://anaconda.com/docs/psm-on-prem/6.8.0/admin/policies.md).

**2. Create a channel for Anaconda's packages**

1. Select  **My Channels** from the left-hand navigation.
2. Click  **Add Channel**.
3. Name your channel `databricks-anaconda`.
4. Set your channel to  **Private**.
5. Open the **Assign Policy** dropdown and select your *databricks* policy.
6. Click **Submit**.

> [!TIP]
>
> For more information on channels, see [Channels](https://anaconda.com/docs/psm-on-prem/6.8.0/user/channel.md).

**3. Create a mirror for Anaconda's packages**

1. Open the  **Manage** dropdown menu and select *Freeze*, then click **Freeze**.

2. Open the  **Manage** dropdown menu again and select *Create Mirror*.

3. Complete the [Create mirror](https://anaconda.com/docs/psm-on-prem/6.8.0/admin/mirrors.md#creating-a-mirror) form with the following configurations:
   * **Name**: databricks-anaconda-packages
   * **External Source URL**: `https://repo.anaconda.com/pkgs/main`
   * **Mirror Type**: conda

4. Verify that the **Run now** checkbox is selected.

5. Verify that the **Assign Mirror Policy** toggle reads `APPLY CHANNEL POLICY`.

6. Allow the mirror time to complete.

   > [accordion] How do I know when my mirror is complete?
   >
   > You can view the status of in-progress mirrors from the from the channel's **Mirrors** tab. In-progress mirrors display which step is currently being performed with a brief description of the operation. When a mirror completes, the status updates to **Completed** and the date and time of completion is displayed.
   >
   > ![The channel's Mirror tab spotlighting the status of an in-progress mirror](https://mintcdn.com/anaconda-29683c67/EzP2zCM6A9Fxpzmy/images/psm-on-prem/ap_mirror_status.png?fit=max\&auto=format\&n=EzP2zCM6A9Fxpzmy\&q=85\&s=a4ea18591d4fdc4e4729e465a966c511)

   > [!NOTE]
   >
   > Administrators can view the status of all mirrors from the  **Mirrors** page.

7. Open the  **Manage** dropdown and select *Unfreeze*, then click **Unfreeze**.

> [!TIP]
>
> For more information on mirroring, see [Mirrors](https://anaconda.com/docs/psm-on-prem/6.8.0/admin/mirrors.md).

**4. Create a channel for conda-forge packages**

1. Select  **My Channels** from the left-hand navigation.
2. Click  **Add Channel**.
3. Name your channel `databricks-conda-forge`.
4. Set your channel to  **Private**.
5. Open the **Assign Policy** dropdown and select your *databricks* policy.
6. Click **Submit**.

**5. Create a mirror for conda-forge packages**

1. Open the  **Manage** dropdown menu and select *Freeze*, then click **Freeze**.
2. Open the  **Manage** dropdown menu again and select *Create Mirror*.
3. Complete the [Create mirror](https://anaconda.com/docs/psm-on-prem/6.8.0/admin/mirrors.md#creating-a-mirror) form with the following configurations:
   * **Name**: databricks-conda-forge-packages
   * **External Source URL**: `https://conda.anaconda.org/conda-forge`
   * **Mirror Type**: conda
4. Verify that the **Run now** checkbox is selected.
5. Verify that the **Assign Mirror Policy** toggle reads `APPLY CHANNEL POLICY`.
6. Allow the mirror time to complete.
7. Open the  **Manage** dropdown and select *Unfreeze*, then click **Unfreeze**.

**6. Generate a token for your channels**

1. Select  **Tokens** from the left-hand navigation.
2. Click  **Generate Token**.
3. Name your token `databricks-token`.
4. Set an expiration date for your token that makes sense for your use case.
5. Open the **Type** dropdown menu and select **Resources**.
6. Open the **Channel** dropdown menu and select your `databricks-anaconda` channel.
7. Open the **Permission** dropdown menu and select **Read**.
8. Click  **Add Resource**.
9. Open the **Channel** dropdown menu and select your `databricks-conda-forge` channel.
10. Open the **Permission** dropdown menu and select **Read**.
11. Click **Create**.
12. Your token displays in the upper-right corner of the page. Copy the token and save it in a secure location.

> [!TIP]
>
> For more information on tokens, see [Tokens](https://anaconda.com/docs/psm-on-prem/6.8.0/user/auth_token.md).

**7. Build a Custom Docker Image**

To create a secure Python environment in Databricks, you'll need to build a custom Docker image. This image includes your conda-based environment and can be used when launching a cluster through Databricks Container Services.

For more information, see [Customize containers with Databricks Container Service](https://docs.databricks.com/aws/en/compute/custom-containers) and [GitHub - databricks/containers](https://github.com/databricks/containers?tab=readme-ov-file).

1. Create a directory on your local machine called `dcs-conda` by running the following command:

   ```sh theme={null}
   mkdir dcs-conda
   ```

2. Enter your new `dcs-conda` directory and create a `Dockerfile` file inside the `dcs-conda` directory:

   ```sh theme={null}
   cd dcs-conda
   vi Dockerfile
   ```

3. Add the following content to the `Dockerfile` file, depending on your Databricks Runtime version:

   ```dockerfile 16.4-LTS highlight {27, 28} expandable theme={null}
   # Replace <FQDN> with the fully qualified domain name of your Package Security Manager instance
   # Replace <TOKEN> with the resource token you just generated

   FROM ubuntu:24.04 AS builder
   RUN apt-get update && apt-get install --yes \
       wget \
       libdigest-sha-perl \
       bzip2 \
       gcc \
       python3-dev \
       libpq-dev \
       libcairo2-dev \
       libdbus-1-dev \
       libgirepository1.0-dev \
       libsnappy-dev \
       git \
       maven && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-py311_25.1.1-2-Linux-x86_64.sh -O miniconda.sh && \
       /bin/bash miniconda.sh -b -p /databricks/conda && \
       rm miniconda.sh
   FROM databricksruntime/minimal:16.4-LTS
   RUN apt-get update && apt-get install --yes git && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   COPY --from=builder /databricks/conda /databricks/conda
   COPY env.yml /databricks/.conda-env-def/env.yml
   RUN /databricks/conda/bin/conda config --system --prepend channels https://<FQDN>/api/repo/t/<TOKEN>/databricks-anaconda && \
       
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/main && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/r
   RUN /databricks/conda/bin/conda env create --file /databricks/.conda-env-def/env.yml && \
       ln -s /databricks/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
   RUN /databricks/conda/bin/conda config --system --set channel_priority strict && \
       /databricks/conda/bin/conda config --system --set always_yes True
   RUN rm -f /root/.condarc
   ENV GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
   ENV DEFAULT_DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ENV DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ```

   ```dockerfile 15.4-LTS highlight {27, 28} expandable theme={null}
   # Replace <FQDN> with the fully qualified domain name of your Package Security Manager instance
   # Replace <TOKEN> with the resource token you just generated

   FROM ubuntu:22.04 AS builder
   RUN apt-get update && apt-get install --yes \
       wget \
       libdigest-sha-perl \
       bzip2 \
       gcc \
       python3-dev \
       libpq-dev \
       libcairo2-dev \
       libdbus-1-dev \
       libgirepository1.0-dev \
       libsnappy-dev \
       git \
       maven && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-py311_25.1.1-2-Linux-x86_64.sh -O miniconda.sh && \
       /bin/bash miniconda.sh -b -p /databricks/conda && \
       rm miniconda.sh
   FROM databricksruntime/minimal:15.4-LTS
   RUN apt-get update && apt-get install --yes git && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   COPY --from=builder /databricks/conda /databricks/conda
   COPY env.yml /databricks/.conda-env-def/env.yml
   RUN /databricks/conda/bin/conda config --system --prepend channels https://<FQDN>/api/repo/t/<TOKEN>/databricks-anaconda && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/main && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/r
   RUN /databricks/conda/bin/conda env create --file /databricks/.conda-env-def/env.yml && \
       ln -s /databricks/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
   RUN /databricks/conda/bin/conda config --system --set channel_priority strict && \
       /databricks/conda/bin/conda config --system --set always_yes True
   RUN rm -f /root/.condarc
   ENV GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
   ENV DEFAULT_DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ENV DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ```

   ```dockerfile 14.3-LTS highlight {27, 28} expandable theme={null}
   # Replace <FQDN> with the fully qualified domain name of your Package Security Manager instance
   # Replace <TOKEN> with the resource token you just generated

   FROM ubuntu:22.04 AS builder
   RUN apt-get update && apt-get install --yes \
       wget \
       libdigest-sha-perl \
       bzip2 \
       gcc \
       python3-dev \
       libpq-dev \
       libcairo2-dev \
       libdbus-1-dev \
       libgirepository1.0-dev \
       libsnappy-dev \
       git \
       maven && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-py311_25.1.1-2-Linux-x86_64.sh -O miniconda.sh && \
       /bin/bash miniconda.sh -b -p /databricks/conda && \
       rm miniconda.sh
   FROM databricksruntime/minimal:14.3-LTS
   RUN apt-get update && apt-get install --yes git && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   COPY --from=builder /databricks/conda /databricks/conda
   COPY env.yml /databricks/.conda-env-def/env.yml
   RUN /databricks/conda/bin/conda config --system --prepend channels https://<FQDN>/api/repo/t/<TOKEN>/databricks-anaconda && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/main && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/r
   RUN /databricks/conda/bin/conda env create --file /databricks/.conda-env-def/env.yml && \
       ln -s /databricks/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
   RUN /databricks/conda/bin/conda config --system --set channel_priority strict && \
       /databricks/conda/bin/conda config --system --set always_yes True
   RUN rm -f /root/.condarc
   ENV GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
   ENV DEFAULT_DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ENV DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ```

   ```dockerfile 13.3-LTS highlight {27, 28} expandable theme={null}
   # Replace <FQDN> with the fully qualified domain name of your Package Security Manager instance
   # Replace <TOKEN> with the resource token you just generated

   FROM ubuntu:22.04 AS builder
   RUN apt-get update && apt-get install --yes \
       wget \
       libdigest-sha-perl \
       bzip2 \
       gcc \
       python3-dev \
       libpq-dev \
       libcairo2-dev \
       libdbus-1-dev \
       libgirepository1.0-dev \
       libsnappy-dev \
       git \
       maven && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-py311_25.1.1-2-Linux-x86_64.sh -O miniconda.sh && \
       /bin/bash miniconda.sh -b -p /databricks/conda && \
       rm miniconda.sh
   FROM databricksruntime/minimal:13.3-LTS
   RUN apt-get update && apt-get install --yes git && \
       apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   COPY --from=builder /databricks/conda /databricks/conda
   COPY env.yml /databricks/.conda-env-def/env.yml
   RUN /databricks/conda/bin/conda config --system --prepend channels https://<FQDN>/api/repo/t/<TOKEN>/databricks-anaconda && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/main && \
       /databricks/conda/bin/conda config --system --remove channels https://repo.anaconda.com/pkgs/r
   RUN /databricks/conda/bin/conda env create --file /databricks/.conda-env-def/env.yml && \
       ln -s /databricks/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
   RUN /databricks/conda/bin/conda config --system --set channel_priority strict && \
       /databricks/conda/bin/conda config --system --set always_yes True
   RUN rm -f /root/.condarc
   ENV GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
   ENV DEFAULT_DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ENV DATABRICKS_ROOT_CONDA_ENV="dcs-conda"
   ```

4. Create an `env.yml` file inside the `dcs-conda` directory:

   ```sh theme={null}
   vi env.yml
   ```

5. Add the following content to the `env.yml` file:

   ```yml expandable highlight {6, 7} theme={null}
   # Replace <FQDN> with the fully qualified domain name of your Package Security Manager instance
   # Replace <TOKEN> with the resource token you generated earlier

   name: dcs-conda
   channels:
     - https://<FQDN>/api/repo/t/<TOKEN>/databricks-anaconda
   dependencies:
     - python=3.11
     - databricks-sdk
     - grpcio
     - grpcio-status
     - ipykernel
     - ipython
     - jedi
     - jinja2
     - matplotlib
     - nomkl
     - numpy
     - pandas
     - pip
     - pyarrow
     - pyccolo
     - setuptools
     - six
     - traitlets
     - wheel
   ```

   > [!NOTE]
   >
   > Please check the recommended package versions in the [System environment](https://docs.databricks.com/aws/en/release-notes/runtime/15.4lts#system-environment) section of the Databricks Runtime release notes and compatibility documentation.

6. Build the Docker image:

   ```sh theme={null}
   docker build -t dcs-conda:15.4-psm .
   ```

7. Tag and push your custom image to a Docker registry by running the following commands:

   ```sh theme={null}
   docker tag dcs-conda:15.4-psm anaconda/dcs-conda:15.4-psm
   docker push anaconda/dcs-conda:15.4-psm
   ```

**8. Launch a Cluster using Databricks Container Service**

> [!WARNING]
>
> Clients must be authorized to access Databricks resources using a Databricks account with appropriate permissions. Without proper access, CLI commands and REST API calls will fail. Permissions can be configured by a workspace administrator.
>
> > [accordion] Enabling Databricks Container Service
> >
> > Use the [Databricks CLI](https://docs.databricks.com/aws/en/compute/custom-containers) to enable Databricks Container Service. In a JSON request body, specify `enableDcs` to `true`, as in the following example:
> >
> > ```py theme={null}
> > databricks workspace-conf set-status --json '{"enableDcs": "true"}'
> > ```

> [!NOTE]
>
> Databricks recommends using OAuth for authorization instead of Personal Access Tokens (PATs). OAuth tokens refresh automatically and reduce security risks associated with token leaks or misuse. For more information, see [Authorizing access to Databricks resources](https://docs.databricks.com/aws/en/dev-tools/auth/).

1. Open your Databricks workspace.

2. Select **Compute** from the left-hand navigation, then click **Create compute**.

3. On the **New compute** page, specify the **Cluster Name**.

4. Under **Performance**, set the **Databricks Runtime Version** to a version that supports Databricks Container Service. For example - **Runtime: 15.4-LTS**.

   ![databricks container service — compute, new compute page](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-conda-cluster.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=913c887bb5d797e42b91a0b5b07ccd89)

   > [!NOTE]
   >
   > This version is under long-term support (LTS). For more information, see [Databricks support lifecycles](https://docs.databricks.com/aws/en/release-notes/runtime/databricks-runtime-ver).
   >
   > ***
   >
   > `Databricks Runtime for Machine Learning` [does not support Databricks Container Service](https://docs.databricks.com/aws/en/compute/custom-containers#limitations).

5. Open the **Advanced options** dropdown and click the **Spark** tab.

6. Add the following Spark configurations:

   ```sh theme={null}
   spark.databricks.isv.product anaconda-psm # Must always be added, regardless of other settings
   spark.databricks.driverNfs.enabled false
   ```

   > [!NOTE]
   >
   > To access volumes on Databricks Container Service, add the following configuration to the compute's **Spark config** field as well: `spark.databricks.unityCatalog.volumes.enabled true`.

   ![databricks container service — compute, advanced options, spark tab](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-spark_2.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=7f08af279a407441128a131a0be0901f)

7. Click the **Docker** tab.

8. Select the **Use your own Docker container** checkbox.

9. Enter your custom Docker image in the **Docker Image URL** field.

   ![databricks container service — compute, advanced options, docker tab](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-docker-image_2.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=b9f3e0af683bcd04105f5510461663c3)

   > [accordion] Docker Image URL examples
   >
   > * Docker Hub - `<organization>/<repository>:<tag>`
   > * Amazon ECR - `<aws-account-id>.dkr.ecr.<region>.amazonaws.com/<repository>:<tag>`
   > * Azure Container Registry - `<your-registry-name>.azurecr.io/<repository-name>:<tag>`

10. Open the **Authentication** dropdown and select an authentication method.

11. Click **Create compute**.

**9. Create a Notebook and connect it to your cluster**

1. Click **New** in the top-left corner, then click **Notebook**.
2. Specify a name for the notebook.
3. Click **Connect**, then select your cluster from the resource list.

   ![databricks notebook workspace — connect cluster resources](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-notebook.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=48d0d2dffacb96c5d251d3fc3bd451be)

**10. Verify your conda installation**

1. In your notebook, run one of the following commands to check that conda is installed:

   ```py theme={null}
   !conda --help
   ```

   ```py theme={null}
   %sh conda --help
   ```

   > [!NOTE]
   >
   > Both commands run shell code from the notebook. `!conda --help` runs the command in the current shell. `%sh conda --help` starts a subshell, which is useful for multi-line scripts, but might not have the same environment or path.

   ![databricks notebook, verifying conda installation](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-conda-verify.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=f4b550292cc8f8f4beca49f0a79c9abc)

2. In your notebook, run the following command to check your source channels:

   ```py theme={null}
   !conda config --show channels
   ```

   ![databricks notebook, viewing conda channels](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-conda-show-channels.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=be61ce6451853141579b13e04ad687dc)

**11. Install MLflow from your Anaconda organization channel**

MLflow is available through your Anaconda organization channel for use in your Databricks environment.

1. In your notebook, install MLflow from your Anaconda organization channel:

   ```py theme={null}
   !conda install mlflow
   ```

   > [!NOTE]
   >
   > This command installs MLflow and all of its dependencies from your Package Security Manager channel.

   ![databricks notebook, installing mlflow using conda](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-mlflow-install.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=313226e63532146a8f75ea4cdd6e6ea4)

2. In your notebook, verify the installation:

   ```py theme={null}
   import mlflow
   print("MLflow: " + mlflow.__version__)
   ```

   ![databricks notebook, verifying mlflow installation](https://mintcdn.com/anaconda-29683c67/ghAcs1Rv_eiLtpni/images/anaconda-platform/databricks-mlflow-verify.png?fit=max\&auto=format\&n=ghAcs1Rv_eiLtpni\&q=85\&s=a26914fa6e7b0a72ca674f19449a059d)