AWS ECR for Environments

Datalayer Environments are the images users and platform administrators build for their runtimes and sandboxes. They are pushed to AWS ECR, while the images of the Datalayer services stay in the Docker registry.

One Clouder command deploys everything the registry needs, from an empty AWS account to a plane that pulls from it:

clouder aws ecr-environments deploy

Nothing is created in the AWS console, and nobody runs terraform by hand: the command drives the Terraform root under terraform/environments-registry itself.

What gets deployed

With the default options:

Piece	Name	What it is for
Enhanced scanning	every repository under environments/	Amazon Inspector scans each push, and promotion reads the findings
Base channel repositories	environments/base/python-cpu, environments/base/python-cuda	The approved bases every environment builds from
KMS encryption key	alias/datalayer-environments-ecr	Encrypts every repository under environments/
KMS signing key	alias/datalayer-environments-signing	cosign signs each image with it, and the operator verifies the signature before it starts a pod
IAM user datalayer-environments-builder	Secret ecr-environments-builder in datalayer-durable	Creates repositories, pushes, signs and reads its images' scan findings, from the durable worker
IAM user datalayer-environments-reader	Secret ecr-environments-reader in datalayer-api	Reads images and scan findings, from Runtimes
IAM user datalayer-environments-puller	Secret ecr-environments-puller in datalayer-runtimes	Pulls, through the refresher
IAM role datalayer-environments-base-reader	assumed by the builder only	Reads environments/base/ for an E2B or Daytona build's base pull, never a user's image (D-18)
IAM user datalayer-environments-modal-base-reader	Secret ecr-environments-modal-base-reader in datalayer-durable	The same read as the role above, held as a real key: Modal's own Image.from_aws_ecr never accepts an assumed session, only a permanent key or its own OIDC (D-18)
CronJob ecr-environments-refresher	in datalayer-runtimes	Rewrites the pull secret ecr-environments every 6 hours

Every policy is scoped to the environments/ prefix. A user environment gets its repository, environments/u/<owner>/<environment>, from the builder the first time it is built, and platform environments live under environments/platform/. Tags are immutable, except in the BuildKit caches under environments/cache/, and images are pulled by digest.

The builder's policy statements are listed on the BuildKit page. That includes ReadFindings, which every build's scan read needs. A registry deployed before clouder commit ee85c7a lacks that statement, and the same page shows how to add it.

Prerequisites

1. AWS credentials allowed to create ECR, KMS and IAM resources in the account. clouder aws info shows the account and identity the command will use, and clouder aws configure lists the ways to set them.
2. Terraform, or docker. Without a terraform binary, the command runs the pinned hashicorp/terraform image, with ~/.aws mounted read-only and the AWS_* variables passed by name.
3. kubectl pointing at the plane, unless you pass --skip-kubernetes.
4. docker for the final check, unless you pass --skip-check. cosign is used when it is installed; otherwise the check runs its pinned image.

Registry scanning is one per region

An AWS account has one registry scanning configuration per region, and applying the registry replaces it. So the command reads it first:

• Enhanced scanning that already covers environments/, such as a rule for *, is left as it is.
• A configuration that scans anything else is never replaced silently. The command stops and names what it scans; pass each of those filters with --extra-scan-filter to keep them, or pass --no-manage-registry-scanning to leave the configuration alone.
• A region with no scanning rules gets enhanced scanning for environments/.

Deploy

clouder aws ecr-environments deploy --region us-east-1 --context <plane-context>

The command:

1. Prints the AWS account and the Kubernetes context it is about to change, and asks before going on.
2. Writes the Terraform variables from its options into clouder.auto.tfvars, plans, shows the plan, and applies it once you confirm.
3. Creates an access key for every principal that has none — builder, puller, reader, modal-base-reader — each in a file only you can read under ~/.clouder/ecr-environments/<workspace>/keys. Keys are never printed.
4. Creates the three Secrets by server-side apply, so no key is copied into an annotation.
5. Installs the refresher in each runtime namespace and runs it once, so the ecr-environments pull secret exists straight away.
6. Runs the check.
7. Prints the two lines to add to the datalayerrc of each plane that launches environments:

export DATALAYER_ECR_ENVIRONMENTS_REGION=us-east-1
export DATALAYER_ECR_ENVIRONMENTS_REGISTRY=123456789012.dkr.ecr.us-east-1.amazonaws.com

--yes answers every question, for an unattended run. Running deploy again is safe: it plans no changes when nothing changed, keeps the key files it finds, and applies the Secrets and the refresher again.

Option	Default	Notes
--region	DATALAYER_ECR_ENVIRONMENTS_REGION, then us-east-1
--project-name	datalayer	The first part of every IAM and KMS name
--repository-prefix	environments	A scratch prefix tries the setup beside a real one
--kms-deletion-window	30	Days a destroyed KMS key stays recoverable, 7 to 30; 7 suits a scratch registry
--base-channel	python-cpu and python-cuda	Repeat it for several
--keys-dir	~/.clouder/ecr-environments/<workspace>/keys
--builder-namespace	DATALAYER_DURABLE_NAMESPACE, then datalayer-durable
--reader-namespace	datalayer-api
--runtime-namespace	datalayer-runtimes	Repeat it for several
--kubeconfig, --context	the current context
--skip-kubernetes, --skip-check	off

Each registry, a project and a prefix, is a Terraform workspace of its own, <project>-<prefix>: datalayer-environments by default. Its state lives under terraform/environments-registry/terraform.tfstate.d/<workspace>/, its variables in workspaces/<workspace>.tfvars, and its key files under ~/.clouder/ecr-environments/<workspace>/keys. So a scratch registry sits beside the real one, and destroying it leaves the real one alone. Commands that do not plan (outputs, rotate-keys, secrets, refresher, check) act on the registry planned last, or on the one --workspace names.

The state is local to the machine that ran deploy. Keep it: git ignores it, and without it the next run plans to create what already exists.

Check

clouder aws ecr-environments check

check uses the principals' own keys, not yours, to prove what the platform relies on:

Step	Proves
Log in, create environments/u/clouder-check/probe, push a probe image	The builder creates and pushes under the prefix
Sign with the KMS key	The builder signs
Log in, pull by digest, verify the signature	The puller pulls and verifies
Read the scan	The reader sees the enhanced scanning findings
Create a repository outside the prefix	Refused to the builder
Read the probe as the base-reader	Refused: that role reads bases only

It removes the probe image when it is done. A scan can take a few minutes, and --scan-timeout sets how long to wait. A key IAM has just created is refused for a while as it propagates, so a check run straight after deploy created the keys waits up to two minutes for them.

cosign signs with --tlog-upload=false and verifies with --insecure-ignore-tlog=true. The signature of a private environment is not written to the public transparency log, where anyone could read its digest and its repository. The key is the trust anchor.

Step by step

Every step of deploy is also a command of its own:

Command	Does
clouder aws ecr-environments plan	Plans into tfplan, and with --json also writes tfplan.json
clouder aws ecr-environments apply	Applies the saved tfplan
clouder aws ecr-environments outputs	Shows the registry, the key ARNs and the IAM names
clouder aws ecr-environments rotate-keys	Creates a new key per principal, or deletes the older ones with --retire-old
clouder aws ecr-environments secrets	Creates the Secrets from the key files
clouder aws ecr-environments refresher	Installs and runs the refresher
clouder aws ecr-environments check	Runs the check
clouder aws ecr-environments destroy	Removes one registry: its repositories and images, its access keys, and what Terraform made

In CI

Plan and apply are split, so a person or a policy can approve in between:

# Keep tfplan and tfplan.json as artifacts of this stage.
clouder aws ecr-environments plan --json
# After the approval: applies exactly that plan.
clouder aws ecr-environments apply

plan takes the same Terraform options as deploy. Git ignores both files, and they should stay out of it: a plan can carry sensitive values.

A scratch registry

Try the setup, or prove it, beside the real registry, in the same account:

clouder aws ecr-environments deploy \
  --project-name datalayer-scratch --repository-prefix environments-scratch \
  --kms-deletion-window 7 --skip-kubernetes

Remove it when you are done:

clouder aws ecr-environments destroy --workspace datalayer-scratch-environments-scratch

destroy shows what it removes and asks first; --confirm <workspace> answers for an unattended run. In order, it deletes:

1. The repositories under the prefix, with their images.
2. The access keys of the four principals, since IAM will not delete a user that still has one.
3. What Terraform made.

Then it deletes the workspace, its variables and its key files. It refuses the default workspace. It also refuses a registry holding user or platform environment images, unless you pass --delete-environment-images. KMS keys are scheduled for deletion and stay recoverable until their window ends.

Proved on a scratch registry

Before a plane relied on it, the setup was proved under the environments-scratch prefix with the principals' own keys:

What	Result
buildctl pushing by digest with an SBOM and mode=max provenance	Accepted: ECR keeps an OCI image index with the linux/amd64 image and an attestation-manifest beside it
A BuildKit registry cache exported with image-manifest=true,oci-mediatypes=true	Accepted
The same cache tag exported by a second build	Refused: cannot be overwritten because the tag is immutable
Enhanced scanning of an image carrying Jinja2 2.10 and urllib3 1.26.4	15 CVEs on those two packages, read by the reader
The base-reader, assumed by the builder, reading a base and pulling it with a login token made from its session	Passed. Sessions last up to an hour, the module's base_reader_session_seconds; 12 hours is refused. The token lasts 12 hours whatever the session's length
cosign signing with the KMS key, and verifying as the puller	Passed
The base-reader, assumed with a session policy, reading a user image	Refused with AccessDeniedException

The build that was proved:

buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
  --opt attest:sbom= --opt attest:provenance=mode=max \
  --output type=image,name=<registry>/environments/u/<owner>/<environment>:v<n>-<build>,push=true,oci-mediatypes=true \
  --export-cache type=registry,ref=<registry>/environments/cache/u/<owner>:<tag>,mode=max,image-manifest=true,oci-mediatypes=true \
  --metadata-file metadata.json

So the builder creates each cache repository, environments/cache/u/<owner>, with mutable tags: every build exports its cache under the same tag, and nothing is launched from a cache. Environment repositories keep immutable tags, and a runtime starts from the digest in metadata.json, never from a tag.

The ECR quotas that matter here, from Service Quotas: 100,000 repositories, 100,000 images per repository, 1,000 tags per image, and 10 PutImage requests a second, the limit a burst of concurrent builds meets first. All but the tags per image are adjustable.

Kubernetes later

A registry can be deployed before any plane uses it. Deploy with --skip-kubernetes, and the Secrets and the refresher wait. When a plane is ready, with the key files of its workspace on the machine and the plane's kubeconfig context:

# The builder's Secret in the durable namespace, the reader's in datalayer-api,
# the puller's in each runtime namespace, by server-side apply.
clouder aws ecr-environments secrets --workspace datalayer-environments --context <plane-context>
# The CronJob that rewrites the ecr-environments pull secret every 6 hours, run once.
clouder aws ecr-environments refresher --workspace datalayer-environments --context <plane-context>

Both ask before writing into the context. Run deploy again without --skip-kubernetes to do both in one go: it plans no changes and keeps the keys it has.

A bug only a real cluster caught, fixed 2026-09-14

secrets and refresher apply through kubectl apply --server-side -f -, fed several manifests joined by ---. The first run against a real plane (r1) refused outright: invalid character '-' in numeric literal. The manifests were being written as JSON, one document per ----separated piece — but JSON has no multi-document form, so kubectl reads the dashes as the start of a number the moment it commits to parsing the stream as JSON rather than YAML. Fixed by writing the stream as YAML instead (--- is exactly what -f - expects there); each manifest is still built as the same plain dict, only how it is serialized changed. The existing unit tests never caught this because their own recorder decoded each ----piece as its own JSON object rather than validating the whole stream as one — fixed the same way, with a regression test asserting the real stream parses as YAML and fails as JSON.

Rotate keys

IAM allows two keys per user, so a rotation never leaves a plane without a working key:

# 1. New key files for every principal.
clouder aws ecr-environments rotate-keys
# 2. The Secrets carry the new keys.
clouder aws ecr-environments secrets

Once the pods that read the Secrets have restarted, retire the old keys:

clouder aws ecr-environments rotate-keys --retire-old

Either command takes --principal to act on one at a time: --principal builder, --principal puller, --principal reader or --principal modal_base_reader.

A principal added to the code after your last deploy has no key yet

deploy only creates a key for a principal that has none (step 3 above), and only when it runs — a principal that joins the code later, such as modal_base_reader on 2026-09-18, is not retroactively given one just by its Terraform resource landing. secrets --principal modal_base_reader run before that key exists refuses outright:

/home/.../keys/modal_base_reader.env is missing: run `clouder aws ecr-environments rotate-keys` first.

The message says exactly what to do. Two ways to get there:

• Run deploy again. It is idempotent — it plans no changes when Terraform already matches, keeps every key file it already has, and ensure_keys fills in exactly the ones missing, modal_base_reader included. This is the simplest choice when you have not already split plan/apply apart.
• Or run the two steps by hand, if you already applied separately (the CI split further down) and would rather not re-run the whole thing:

clouder aws ecr-environments rotate-keys --principal modal_base_reader
clouder aws ecr-environments secrets --principal modal_base_reader

Either way, finish with a rollout of whatever reads the new Secret — p reup datalayer-durable for modal_base_reader, whose Secret only that worker's environments queue mounts.

Another plane

A plane that launches environments without building them needs only the pull secret. With the puller's key file on the machine:

clouder aws ecr-environments secrets --principal puller --context <other-plane>
clouder aws ecr-environments refresher --context <other-plane> \
  --registry 123456789012.dkr.ecr.us-east-1.amazonaws.com

An expired pull secret

An ECR token lasts 12 hours and the refresher rewrites it every 6, so a single failed run goes unnoticed. When runtimes on environments/ images fail with ImagePullBackOff and an event such as no basic auth credentials or authorization token has expired, look at the refresher:

kubectl -n datalayer-runtimes get cronjob ecr-environments-refresher
kubectl -n datalayer-runtimes get pods -l app.kubernetes.io/part-of=datalayer-environments
kubectl -n datalayer-runtimes logs <refresher-pod> --all-containers

and run it now:

kubectl -n datalayer-runtimes create job --from=cronjob/ecr-environments-refresher ecr-environments-refresher-manual

If the password container fails, the puller's key is wrong or deleted, and clouder aws ecr-environments check names the step that breaks. If the secret container answers 403, the refresher's Role was changed, and clouder aws ecr-environments refresher applies it again.