Managing code locations in Dagster Cloud#

In this guide, we'll cover the requirements for Dagster code, how to add code in Dagster Cloud, and how to add code using the dagster-cloud CLI.

Understanding code locations#

A code location specifies a single Python package or file that defines your Dagster code. When you add a code location in Dagster Code, you're instructing the agent where to find your code.

When you add or update a code location, the agent uses the location configuration to load your code and upload metadata about your jobs to Dagster Cloud. Each full deployment - for example, prod - can include code from one or more code locations.

Note that, unlike Dagster Open Source, Dagster Cloud doesn't require a workspace.yaml file. Instead, you use the Dagster Cloud API to configure your workspace. You can still create a workspace.yaml file if you want to load your code in an open-source Dagit instance, but doing so won't affect how your code is loaded in Dagster Cloud.

Dagster Cloud code requirements#

To work with Dagster Cloud, your Dagster code:

• Must be loaded from a single entry point, either a Python file or package. This entry point can load repositories from other files or packages.

• Must run in an environment where the dagster and dagster-cloud 0.13.2 or later Python packages are installed.

• If using Hybrid Deployment:

  • And you're using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent, your code must be packaged into a Docker image and pushed to a registry your agent can access. Dagster Cloud doesn't need access to your image - your agent only needs to be able to pull it.

    Additionally, the Dockerfile for your image doesn't need to specify an entry point or command. These will be supplied by the agent when it runs your code using your supplied image.

  • And you're using a local agent, your code must be in a Python environment that can be accessed on the same machine as your agent.

Additionally, note that:

• Your code doesn't need to use the same version of Dagster as your agent
• Different code locations can use different versions of Dagster

Managing code locations in Dagster Cloud#

• Adding code locations
• Modifying code locations
• Redeploying code locations
• Deleting code locations

Adding code locations#

1. Sign in to your Dagster Cloud account.

2. Click Deployment.

3. Click + Add code location. This will open a YAML editor with a schema describing the acceptable fields:

   

4. In the editor, define the code location's configuration:

   Property	Description
   code_source	Set this key to either python_file: or package_name:to specify where to find your code.
   executable_path	Optional. Define a specific Python executable if your code should run in a certain Python environment. If left undefined, the code will run using the default dagster command-line entry-point.
   image	Required if not using a local agent. Specifies a Docker image for use with containerized agents.
   working_directory	Specifies the directory to use to resolve relative Python imports while loading your code.
   attribute	Specifies only a specific Dagster repository should be loaded.
   container_context	Optional. For agent versions 0.14.9 and later. Customizes the code location for a specific execution environment. Refer to the Agent documentation for info on available configuration options for each agent type, including declaring environment variables and secrets.

   For example, the following config specifies that a code location should include a secret named my_secret and run in a k8s namespace (my_namespace) whenever the Kubernetes agent creates a pod for the location:

   location_name: cloud-examples
   image: dagster/dagster-cloud-examples:latest
   code_source:
     package_name: dagster_cloud_examples
   container_context:
     k8s:
       namespace: my_namespace
       env_secrets:
         - my_secret
   

5. When finished, click Add code location.

The agent will attempt to load your code and send its metadata to Dagster Cloud. Note: This may take some time.

Once your code has loaded, the location will show a green Loaded status and jobs will appear in Dagster Cloud. If the agent is unable to load your code, the location will show an error with more information.

Modifying code locations#

To modify a code location, click the dropdown menu to the right of the location. In the menu, click Modify:

After a code location is updated, the agent will perform a rolling update of your code and jobs will update in Dagster Cloud. Note: Updating code won't interrupt any currently launched runs.

Redeploying code locations#

To reload your code and upload job metadata to Dagster Cloud without modifying the code location, click the Redeploy button:

For example, if the agent was unable to pull your image due to a permissions issue that's since been addressed, clicking Redeploy will tell the agent to try again.

Deleting code locations#

To delete a code location, click the dropdown menu to the right of the location. In the menu, click Remove:

When prompted, confirm the deletion.

Managing code locations using the dagster-cloud CLI#

You can also use the dagster-cloud workspace CLI commands to:

• Add and update code locations
• Delete code locations
• Sync the workspace and remote

These commands perform the same underlying operations as editing your code locations in the Deployment tab in Dagster Cloud. Refer to the dagster-cloud CLI guide for more info and installation instructions.

Adding and updating code locations#

You can add or update a code location with the add-location command. For example, to add our public example image, you can run:

# Set up YAML file for example location
cat > example_location.yaml <<EOL
location_name: cloud-examples
image: dagster/dagster-cloud-examples:latest
code_source:
  package_name: dagster_cloud_examples
EOL

dagster-cloud workspace add-location --from example_location.yaml

Instead of creating a YAML file, you may also pass these values inline as command line options:

dagster-cloud workspace add-location test_location \
    --image dagster/dagster-cloud-examples:latest \
    --package-name dagster_cloud_examples

You may also selectively overwrite parts of your YAML input using inline options:

dagster-cloud workspace add-location \
    --from example_location.yaml \
    --image dagster/dagster-cloud-examples:1d9c5d

The arguments to the add-location and update-location commands are similar to the keys in the YAML config editor in the Deployment tab. To see all available options, run:

dagster-cloud workspace add-location --help

Note: If updating an existing code location, the full set of information about the location must be specified even if only one piece of configuration is modified.

Deleting code locations#

Delete an existing code location from Dagster Cloud using the delete-location command:

dagster-cloud workspace delete-location <LOCATION_NAME>

Syncing the workspace#

You can also keep the YAML configuration for your entire workspace in a file and use the dagster-cloud sync command to reconcile the workspace config in Dagster Cloud with that local file.

For example, if you have the following dagster_cloud.yaml file:

locations:
  - location_name: machine-learning
    image: myregistry/dagster-machine-learning:mytag
    code_source:
      package_name: dagster_cloud_machine_learning
    executable_path: /my/folder/python_executable
    attribute: my_repo
  - location_name: data-eng
    image: myregistry/dagster-data-eng:myothertag
    code_source:
      python_file: repo.py
    working_directory: /my/folder/working_dir/

Reconcile the above file with Dagster Cloud's remote workspace by running:

dagster-cloud workspace sync -w dagster_cloud.yaml