Mirroring channels and packages

Data Science & AI Workbench enables you to create local copies of a repositories so users can access packages from a centralized, on-premise location. This process is called mirroring. You can mirror the full content of a repository, or include only specific packages or types of packages from the repository in your mirror. You can also create mirrors in an air-gapped network improve performance and security.

You can mirror an online repository, or you can use a tarball containing package data to populate a channel in Workbench.

Prerequisites:

• Follow the steps for administration server setup on your current machine to install the necessary tools for mirroring.
• Configure the Workbench CLI.

It can take several hours to mirror an entire repository, depending on its size.

Creating a conda mirror

The basic steps for creating a conda mirror are:

1. Prepare your mirror configuration file.

2. Log in to the Workbench CLI.

3. If necessary, create a channel in the internal Workbench repository.

4. Initiate the mirror by running the following command:

   anaconda-mirror-ae5 --file /path/to/<mirror.yaml>
   
   Append --dry-run to the command to see what actions would be taken by the mirror, without performing actual modifications.

Preparing your mirror configuration file

Create a <mirror>.yaml file that details the configurations for the mirror.

You can name this file whatever you’d like. Anaconda recommends naming it the same as the channel you are mirroring to.

Basic configurations

Define source channel locations, package platforms, and destination/storage location details. Manage package formats, clean up outdated packages, and test configurations without applying changes by including these configurations.

Parameter	Description
channels	List of URLs for channels you want to mirror from. If a short channel name is supplied, Workbench uses its system-level .condarc file’s channel_alias: value to complete the channel URL.
platforms	List of platforms you want to mirror packages for. For example, win-64 or linux-64. If no value is supplied, the mirror will include packages for all platforms available on the source channel.
dest_channel	The short name for the internal Workbench repository channel you are mirroring to. The rest of the channel URL is automatically completed by anaconda-enterprise-cli for you.
dest_site	The web address or path where you want to store your mirrored packages. The specific formatting and necessity of this value depends on the type of destination repository. For more information, see repository-specific configurations.
format_policy	Determines how the mirror manages .conda and .tar.bz2 files: prefer-conda or prefer-tarbz2 — Mirror one package format over the other if both are available for a package. If the preferred type is unavailable, the other file type is still downloaded. only-conda or only-tarbz2 — Mirror packages that are available in the preferred file format only. transmute-conda or transmute-tarbz2 — Convert packages to the preferred format as necessary. Requires the conda-package-handling package to function. keep-both — Mirror both file types for all available packages. Defaults to prefer-conda for repositories that support .conda formatting and only-tarbz2 for those that do not. If your repository does not support .conda formatting, Anaconda recommends installing the conda-package-handling package and using the transmute-tarbz2 option.
clean	true / false - If true, removes packages from the destination channel that are not on the source channel when updating. Default: false (to ensure packages are not inadvertently removed)
dry_run	true / false - If true, outputs what actions would be taken by the mirror, without performing actual modifications. Default: false

Filtering configurations

Fine-tune which packages are included in the mirror. Specify versions of Python or R packages that your packages should be compatible with, include only specific packages, or exclude packages by name and license family type.

Parameter	Description
python_versions	A comma-separated list of Python versions. Restricts all Python packages and packages that depend on Python to these versions.
r_versions	A comma-separated list of R versions. Restricts all R packages and packages that depend on R to these versions.
pkg_list	List of package names or valid MatchSpec strings. If supplied, only the specified packages will be mirrored, not their dependencies. Cannot be paired with license_exclude, exclude, or include.
license_exclude	List of license families to exclude from the mirror. To see a list of valid license families, use the anaconda-mirror-ae5 --help command. Cannot be paired with pkg_list.
exclude	List of package names or valid MatchSpec strings to exclude. Cannot be paired with pkg_list.
include	List of package names or valid MatchSpec strings to override the mirror’s other filters and include these packages even if they would otherwise be filtered out. Cannot be paired with pkg_list.

For more information about MatchSpec, see package match specifications.

Advanced configurations

Configure repository authentication, enforce platform restrictions, and manage SSL verification for secure connections.

Parameter	Description
username / password	Supplies credentials for repository authentication. For more information, see repository-specific configurations.
strict_platforms	true / false - If true, excludes noarch from the mirror. Default: false (all platforms use noarch)
max_attempts	Number of retry attempts for failed connections. Default: 5
max_failures	Number of failed transactions before stopping. Default: 100
verify_ssl	true / false - Enables or disables SSL verification. Default: true

If Workbench is installed in a proxied environment, see Configuring conda in Workbench for information on setting the NO_PROXY variable.

Repository-specific configurations

JFrog Artifactory

For Artifactory destinations, the dest_site can be a repository hostname, or a full URL.

Hostname

If you supply the hostname only, anaconda-mirror interprets the channel path as:

https://<dest_site>/artifactory/<dest_channel>

URL

If you supply a URL, anaconda-mirror appends /artifactory/dest_channel to it to complete the channel path. For example, if you set the dest_site to https://example.site.com, the full path is interpreted as this:

https://example.site.com/artifactory/<dest_channel>

If your mirror is not at this location, further pathing can be included in the URL to reach the correct location. For example, if you set dest_site as https://example.site.com/artifactory/repo/subpath/, it will interpret the path literally and append the dest_channel value to the end:

https://example.site.com/artifactory/repo/subpath/<dest_channel>

To authenticate to a JFrog Artifactory repository

• Configure the username and password values in your .yaml file to contain your credentials. If both values are supplied, they are delivered using basic HTTP authentication. You can substitute an access token for your password if necessary.
• Configure just the password value in your .yaml file. This is delivered as a bearer token using the Authorization: Bearer header. This must be an access token.
• Configure your .netrc file to store your username and password for the repository. These values are delivered using basic HTTP authentication.

S3 bucket

For Simple Storage Service (S3) buckets, the channel path is a concatenation of the dest_site and dest_channel values.

For example, if you were mirroring to an S3 bucket, your dest_site would be set to <bucket_name>/full/path/to/ and the full channel path is interpreted as:

<bucket_name>/full/path/to/<dest_channel>

Authentication to an S3 source is currently controlled entirely by the environment. For example, you can use the aws CLI tool to configure the target region and authenticate. You might want to use the AWS_PROFILE environment variable to select among multiple configurations.

Local

Much like the S3 bucket, the local repository channel path consists of a concatenation of the dest_site and dest_channel values.

No authentication is necessary for local repositories.

anaconda-enterprise-cli

The dest_site value defaults to the <SITE_NAME> value established when you configure the workbench CLI. If you have only configured the CLI to be able to access one site (that is, your Workbench instance), there is no need to specify this value.

Authentication is handled when you log in to the CLI.

Example conda and R mirrors

Here are some example mirror .yaml files you can use to mirror some common repositories:

Anaconda’s main channel (full)

dest_channel: main
channels:
   - https://repo.anaconda.com/pkgs/main
platforms:
   - linux-64

Anaconda’s R channel (full)

dest_channel: r
channels:
   - https://repo.anaconda.com/pkgs/r
platforms:
   - linux-64

Air-gapped network mirror

dest_channel: anaconda
channels:
   - /file/path/to/unpacked/repository/packages
platforms:
   - linux-64

Mirroring a PyPI repository

The full PyPI mirror size is currently close to 10TB, so ensure that your file storage location has sufficient disk space before proceeding.

Because anaconda-mirror does not handle .pip package formatting, mirrors for PyPI repositories containing such packages are managed by the anaconda-enterprise-cli tool.

The steps are identical to creating a conda mirror:

1. Prepare your mirror configuration file.

2. Log in to the Workbench CLI.

3. If necessary, create a channel in the internal Workbench repository.

4. Initiate the mirror by running the following command:

   anaconda-enterprise-cli mirror pypi --config pypi-mirror.yaml
   

This command loads the packages on https://pypi.org into the user’s account.

Mirrored packages can be viewed at https://<FQDN>/repository/pypi/pypi/simple/, replacing <FQDN> with the fully qualified domain name of your installation of Workbench. (The second pypi in the url should match the user configuration value described below.)

PyPI configurations:

PyPI mirror .yaml configuration values consist of the following:

Parameter	Description
user	The local user under which the PyPI packages are imported. Default: pypi
pkg_list	List of package names to mirror. If supplied, only the specified packages will be mirrored, not their dependencies. Cannot be paired with blocklist or allowlist.
allowlist	List of package names to mirror. If supplied, only the specified packages will be mirrored, not their dependencies. Cannot be paired with pkg_list.
blocklist	List of package names to skip. Packages listed here are not mirrored. Cannot be paired with pkg_list.
latest_only	If supplied, only the latest package versions are mirrored. Default: false
remote_url	The URL of the PyPI mirror. /pypi is appended to build the XML RPC API URL, /simple for the simple index and /pypi/{package}/{version}/json for the JSON API. Default: https://pypi.python.org/
xml_rpc_api_url	A custom value for XML RPC URL. If this value is present, it takes precedence over the URL built using remote_url Default: null.
simple_index_url	A custom value for the simple index URL. If this value is present, it takes precedence over the URL built using remote_url. Default: null.
use_xml_rpc	Whether to use the XML RPC API as specified by PEP381. If this is set to true, the XML RPC API is used to determine which packages to check. Otherwise, the script falls back to the simple index. If the XML RPC fails, the simple index is used. Default: true
use_serial	If set to true, uses the serial number provided by the XML RPC API. Only packages updated since the last serial saved are checked. If this is set to false, all PyPI packages are checked for updates. Default: true
create_org	Creates the mirror user as an organization instead of a regular user account. All superusers are added to the Owners group of the organization. Default: false

All mirrored PyPI-like channels are publicly available to pull packages from both inside and outside Workbench (no authentication is required).

Example PyPI mirror (partial)

allowlist:
   - requests
   - six
   - numpy
   - simplejson
latest_only: true
remote_url: https://pypi.org/
use_xml_rpc: true

Configuring pip

To configure pip to use this new mirror, create pip.conf as follows:

# Replace <WORKBENCH_URL> with the actual URL to your Workbench instance
[global]
index-url=<WORKBENCH_URL>/repository/pypi/pypi/simple/

To configure Workbench sessions and deployments to automatically use the pip.conf, run the following command.

anaconda-enterprise-cli spark-config --config /etc/pip.conf pip.conf

For more specific information on configuring pip, see the official pip documentation.