Azure (dagster-azure)¶

Utilities for using Azure Storage Accounts with Dagster. This is mostly aimed at Azure Data Lake Storage Gen 2 (ADLS2) but also contains some utilities for Azure Blob Storage.

NOTE: This package is incompatible with dagster-snowflake! This is due to a version mismatch between the underlying azure-storage-blob package; dagster-snowflake has a transitive dependency on an old version, via snowflake-connector-python.

dagster_azure.adls2.adls2_resource ResourceDefinition[source]¶

Config Schema:

storage_account (dagster.StringSource):

The storage account name.

credential (selector):

The credentials with which to authenticate.

Config Schema:

sas (dagster.StringSource):

SAS token for the account.

key (dagster.StringSource):

Shared Access Key for the account

Resource that gives ops access to Azure Data Lake Storage Gen2.

The underlying client is a DataLakeServiceClient.

Attach this resource definition to a JobDefinition in order to make it available to your ops.

Example

from dagster import job, op
from dagster_azure.adls2 import adls2_resource

@op(required_resource_keys={'adls2'})
def example_adls2_op(context):
    return list(context.resources.adls2.adls2_client.list_file_systems())

@job(resource_defs={"adls2": adls2_resource})
def my_job():
    example_adls2_op()

Note that your ops must also declare that they require this resource with required_resource_keys, or it will not be initialized for the execution of their compute functions.

You may pass credentials to this resource using either a SAS token or a key, using environment variables if desired:

resources:
  adls2:
    config:
      storage_account: my_storage_account
      # str: The storage account name.
      credential:
        sas: my_sas_token
        # str: the SAS token for the account.
        key:
          env: AZURE_DATA_LAKE_STORAGE_KEY
        # str: The shared access key for the account.

class dagster_azure.adls2.FakeADLS2Resource(account_name, credential='fake-creds')[source]¶

Stateful mock of an ADLS2Resource for testing.

Wraps a mock.MagicMock. Containers are implemented using an in-memory dict.

class dagster_azure.blob.AzureBlobComputeLogManager(storage_account, container, secret_key=None, local_dir=None, inst_data=None, prefix='dagster', upload_interval=None, default_azure_credential=None)[source]¶

Logs op compute function stdout and stderr to Azure Blob Storage.

This is also compatible with Azure Data Lake Storage.

Users should not instantiate this class directly. Instead, use a YAML block in dagster.yaml such as the following:

compute_logs:
  module: dagster_azure.blob.compute_log_manager
  class: AzureBlobComputeLogManager
  config:
    storage_account: my-storage-account
    container: my-container
    credential: sas-token-or-secret-key
    default_azure_credential:
      exclude_environment_credential: true
    prefix: "dagster-test-"
    local_dir: "/tmp/cool"
    upload_interval: 30

Parameters:

• storage_account (str) – The storage account name to which to log.

• container (str) – The container (or ADLS2 filesystem) to which to log.

• secret_key (Optional[str]) – Secret key for the storage account. SAS tokens are not supported because we need a secret key to generate a SAS token for a download URL.

• default_azure_credential (Optional[dict]) – Use and configure DefaultAzureCredential. Cannot be used with sas token or secret key config.

• local_dir (Optional[str]) – Path to the local directory in which to stage logs. Default: dagster._seven.get_system_temp_directory().

• prefix (Optional[str]) – Prefix for the log file keys.

• upload_interval – (Optional[int]): Interval in seconds to upload partial log files blob storage. By default, will only upload when the capture is complete.

• inst_data (Optional[ConfigurableClassData]) – Serializable representation of the compute log manager when newed up from config.

dagster_azure.adls2.adls2_pickle_io_manager IOManagerDefinition[source]¶

Config Schema:

adls2_file_system (dagster.StringSource):

ADLS Gen2 file system name

adls2_prefix (dagster.StringSource, optional):

Default Value: ‘dagster’

Persistent IO manager using Azure Data Lake Storage Gen2 for storage.

Serializes objects via pickling. Suitable for objects storage for distributed executors, so long as each execution node has network connectivity and credentials for ADLS and the backing container.

Assigns each op output to a unique filepath containing run ID, step key, and output name. Assigns each asset to a single filesystem path, at “<base_dir>/<asset_key>”. If the asset key has multiple components, the final component is used as the name of the file, and the preceding components as parent directories under the base_dir.

Subsequent materializations of an asset will overwrite previous materializations of that asset. With a base directory of “/my/base/path”, an asset with key AssetKey([“one”, “two”, “three”]) would be stored in a file called “three” in a directory with path “/my/base/path/one/two/”.

Example usage:

1. Attach this IO manager to a set of assets.

from dagster import asset, repository, with_resources
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource

@asset
def asset1():
    # create df ...
    return df

@asset
def asset2(asset1):
    return df[:5]

@repository
def repo():
    return with_resources(
        [asset1, asset2],
        resource_defs={
            "io_manager": adls2_pickle_io_manager.configured(
                {"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
            ),
            "adls2": adls2_resource,
        },
    )
)

2. Attach this IO manager to your job to make it available to your ops.

from dagster import job
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource

@job(
    resource_defs={
        "io_manager": adls2_pickle_io_manager.configured(
            {"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
        ),
        "adls2": adls2_resource,
    },
)
def my_job():
    ...