Amazon ECS agent configuration reference#

This guide is applicable to Dagster Cloud.

This reference describes the various configuration options Dagster Cloud currently supports for Amazon ECS agents.


Per-location configuration#

When adding a code location to Dagster Cloud with an Amazon ECS agent, you can use the container_context key on the location configuration to add additional ECS-specific configuration that will be applied to any ECS tasks associated with that code location.

Note: If you're using the Dagster Cloud Github action, the container_context key can also be set for each location in your dagster_cloud.yaml file.

The following example dagster_cloud.yaml file illustrates the available fields:

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      ecs:
        env_vars:
          - DATABASE_NAME=staging
          - DATABASE_PASSWORD
        secrets:
          - name: "MY_API_TOKEN"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
          - name: "MY_PASSWORD"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
        secrets_tags:
          - "my_tag_name"
        server_resources: # Resources for code servers launched by the agent for this location
          cpu: 256
          memory: 512
        run_resources: # Resources for runs launched by the agent for this location
          cpu: 4096
          memory: 16384

Environment variables and secrets#

Using the container_context.ecs.env_vars and container_context.ecs.secrets properties, you can configure environment variables and secrets for a specific code location.

# dagster_cloud.yaml

container_context:
  ecs:
    env_vars:
      - DATABASE_NAME=testing
      - DATABASE_PASSWORD
    secrets:
      - name: "MY_API_TOKEN"
        valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
      - name: "MY_PASSWORD"
        valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
    secrets_tags:
      - "my_tag_name"
PropertyDescription
container_context.ecs.env_varsA list of keys or key-value pairs to include in the task. If a value is not specified, the value will be pulled from the agent task.
In the example above, FOO_ENV_VAR will be set to foo_value and BAR_ENV_VAR will be set to whatever value it has in the agent task.
container_context.ecs.secretsIndividual secrets specified using the same structure as the ECS API.
container_context.ecs.secrets_tagsA list of tag names. Each secret tagged with any of those tag names in AWS Secrets Manager will be included in the launched tasks as environment variables. The name of the environment variable will be the name of the secret, and the value of the environment variable will be the value of the secret.

Refer to the following guides for more info about environment variables:


Per-job configuration: Resource limits#

You can use job tags to customize the CPU and memory of every run for that job:

from dagster import job, op

@op()
def my_op(context):
  context.log.info('running')

@job(
  tags = {
    "ecs/cpu": "256",
    "ecs/memory": "512",
  }
)
def my_job():
  my_op()

Fargate tasks only support certain combinations of CPU and memory.

If the ecs/cpu or ecs/memory tags are set, they will override any defaults set on the code location or the deployment.


Per-deployment configuration#

This section describes the properties of the dagster.yaml configuration file used by Amazon ECS agents. Typically, this file is created by the CloudFormation template that deploys the agent and can be found within the agent task definition's command.

To change these properties, edit the CloudFormation template and redeploy the CloudFormation stack.

instance_class:
  module: dagster_cloud
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token: <Agent Token String>
  deployment: <Deployment Name>
  branch_deployments: <true|false>

user_code_launcher:
  module: dagster_cloud.workspace.ecs
  class: EcsUserCodeLauncher
  config:
    cluster: <Cluster Name>
    subnets:
      - <Subnet Id 1>
      - <Subnet Id 2>
    security_group_ids:
      - <Security Group ID>
    service_discovery_namespace_id: <Service Discovery Namespace Id>
    execution_role_arn: <Task Execution Role Arn>
    task_role_arn: <Task Role Arn>
    log_group: <Log Group Name>
    launch_type: <"FARGATE"|"EC2">
    server_process_startup_timeout: <Timeout in seconds>
    server_resources:
      cpu: <CPU value>
      memory: <Memory value>
    run_resources:
      cpu: <CPU value>
      memory: <Memory value>

dagster_cloud_api properties#

PropertyDescription
dagster_cloud_api.agent_tokenAn agent token for the agent to use for authentication.
dagster_cloud_api.deploymentThe name of a full deployment for the agent to serve.
dagster_cloud_api.branch_deploymentsWhether the agent should serve all branch deployments.

user_code_launcher properties#

PropertyDescription
config.clusterThe name of an ECS cluster with a Fargate or EC2 capacity provider.
config.launch_typeAn ECS launch type to use for your launched ECS tasks. The following are currently supported:
  • FARGATE
  • EC2 - Note: Using this launch type requires you to have an EC2 capacity provider installed and additional operational overhead to run the agent.
config.subnetsAt least one subnet is required. Dagster Cloud tasks require a route to the internet so they can access our API server. How this requirement is satisfied depends on the type of subnet provided:
  • Public subnets - The ECS agent will assign each task a public IP address. Note that ECS tasks on EC2 launched within public subnets do not have access to the internet, so a public subnet will only work for Fargate tasks.
  • Private subnets - The ECS agent assumes you've configured a NAT gateway with an attached NAT gateway. Tasks will not be assigned a public IP address.
config.security_group_idsA list of security groups to use for tasks launched by the agent.
config.service_discovery_namespace_idThe name of a private DNS namespace.

The ECS agent launches each code location as its own ECS service. The agent communicates with these services via AWS CloudMap service discovery.
config.execution_role_arnThe ARN of the Amazon ECS task execution IAM role. This role allows ECS to interact with AWS resources on your behalf, such as getting an image from ECR or pushing logs to CloudWatch.

Note: This role must include a trust relationship that allows ECS to use it.
config.task_role_arnThe ARN of the Amazon ECS task IAM role. This role allows the containers running in the ECS task to interact with AWS.

Note: This role must include a trust relationship that allows ECS to use it.
config.log_groupThe name of a CloudWatch log group.
config.server_process_startup_timeoutThe amount of time, in seconds, to wait for code to import when launching a new service for a code location. If your code takes an unusually long time to load and results in timeouts in the Deployment tab, you can increase this setting above the default. Note This setting isn't applicable to the time it takes for a job to execute.
  • Default - 180 (seconds)
config.server_resourcesThe resources that the agent should allocate to the ECS service for each code location that it creates. If set, must be a dictionary with a cpu and/or memory key. Note: Fargate tasks only support certain combinations of CPU and memory.
config.run_resourcesThe resources that the agent should allocate to the ECS task that it creates for each run. If set, must be a dictionary with a cpu and/or memory key. Note: Fargate tasks only support certain combinations of CPU and memory.