☰ 🧱 Datalayer BuildKit

datalayer-buildkit is the build pool for Environments. It runs one rootless BuildKit daemon, buildkitd. The durable environments worker is its only client, and reaches it over mutual TLS.

Every Environment build runs here, in two steps: the resolver's solve, then the image build. The image build runs a Dockerfile that Datalayer generates from the resolved lock. For a dockerfile source it starts from the author's own Dockerfile, its FROM pinned to a digest, and installs the lock over it.

What gets deployed

Piece	Namespace	What it is for
Deployment datalayer-buildkit, 1 replica, strategy: Recreate	datalayer-builds	Rootless buildkitd, moby/buildkit:v0.33.0-rootless pinned by digest
Service datalayer-buildkit, port 1234	datalayer-builds	The TLS listener, tcp://datalayer-buildkit.datalayer-builds.svc.cluster.local:1234
PVC datalayer-buildkit-cache, 100 Gi	datalayer-builds	The layer cache, kept across restarts. Build contexts go after 24 h. Bases, layers and uv's cache are kept 30 days, with 30 GB always kept, at most 80 GB used and 15% left free (values.gcPolicy)
ClusterIssuers datalayer-buildkit-selfsigned and datalayer-buildkit-ca	cluster-wide	A private CA that signs this pool's two certificates and nothing else
Certificate datalayer-buildkit-ca, Secret datalayer-buildkit-ca-secret	datalayer-cert-manager	The CA, valid 1 year
Certificate datalayer-buildkit-server, Secret datalayer-buildkit-server-tls	datalayer-builds	buildkitd's server certificate, valid 90 days
Certificate datalayer-buildkit-client, Secret datalayer-buildkit-client-tls	datalayer-durable	The durable worker's client certificate, valid 90 days
Container egress-proxy in the same pod	datalayer-builds	Squid (ubuntu/squid, by digest) on 127.0.0.1:3128, as uid 13: the pod's only way out, to values.egressProxy.allowedDomains alone. See Egress
Init container egress-firewall	datalayer-builds	Sets the pod's firewall before anything else starts, then exits. Root with NET_ADMIN only; not privileged
NetworkPolicy datalayer-buildkit	datalayer-builds	Ingress from datalayer-durable on port 1234. Egress to DNS and port 443, never the cloud metadata address. Not enforced by Flannel: the proxy and the firewall are what hold egress

cert-manager renews the leaf certificates 15 days before they expire, and the CA 30 days before.

The client certificate lives in datalayer-durable because a Certificate's Secret is always created in the Certificate's own namespace, and the worker's pod has to mount it.

The chart is plane/etc/helm-private/charts/datalayer-buildkit.

Prerequisites

1. cert-manager, deployed with plane up datalayer-cert-manager. The CA Secret is created in cert-manager's cluster resource namespace, datalayer-cert-manager.

2. A build node, labelled and tainted by hand. Pick the node, then:

   kubectl label node <node> role.datalayer.io/build=true
   kubectl taint node <node> datalayer.io/build=true:NoSchedule

   The chart never applies either. A taint decides what else may run on a node, so it is the operator's call. Only buildkitd, Falco and the image pruner tolerate this taint. Falco has to reach the build node, because that is where untrusted build steps run. On r1 the build node is r1-node-7-fb74.

3. The environments registry, deployed with clouder aws ecr-environments deploy. buildkitd holds no AWS credential of its own: the durable worker mints an ECR token for each build and hands it over for that build only.

Deploy

plane up datalayer-buildkit

plane up checks the first two prerequisites before it installs anything. If cert-manager or a labelled build node is missing, it stops and prints the commands above. A labelled node without the taint only gets a warning, since the pool still works there.

Then point durable at the pool. Add this to the plane's rc, and run plane up datalayer-durable:

export DATALAYER_BUILDKIT_ADDR="tcp://datalayer-buildkit.datalayer-builds.svc.cluster.local:1234"

The durable chart mounts datalayer-buildkit-client-tls into the environments queue and sets DATALAYER_BUILDKIT_TLSCERT, DATALAYER_BUILDKIT_TLSKEY and DATALAYER_BUILDKIT_TLSCACERT from it. The mount is optional, so the worker starts without the Secret, but every build then fails at the TLS handshake.

Runtime Environments gives the full deploy order: registry, then this pool, then the environments queue.

Verify

kubectl -n datalayer-builds rollout status deployment/datalayer-buildkit
kubectl -n datalayer-builds exec deploy/datalayer-buildkit -- \
  buildctl --addr=unix:///run/user/1000/buildkit/buildkitd.sock debug workers
kubectl get certificate -A | grep datalayer-buildkit

The three certificates should all be True. These commands prove the daemon. Only a real build proves the TLS path from durable.

The builder's AWS permissions

The durable worker builds as the IAM user datalayer-environments-builder. Its key is in the Secret ecr-environments-builder in datalayer-durable. The user has one inline policy, also named datalayer-environments-builder, with six statements:

Statement	Allows
Login	ecr:GetAuthorizationToken
EnvironmentRepositories	Creating, pushing to, reading and deleting repositories under environments/
EncryptRepositories	Using the ECR encryption key
Sign	Signing with the signing key
HandBasesToAProviderBuild	Assuming datalayer-environments-base-reader for a managed-provider build
ReadFindings	inspector2:ListFindings and inspector2:ListCoverage

Every build needs ReadFindings. The build's attest step reads its image's scan from Amazon Inspector, and without this statement every build fails there with AccessDeniedException on inspector2:ListCoverage.

The Terraform module has included it since clouder commit ee85c7a. A registry deployed before that lacks it. Re-run clouder aws ecr-environments deploy, or add the statement by hand with AWS credentials for the registry's account:

aws iam get-user-policy --user-name datalayer-environments-builder \
  --policy-name datalayer-environments-builder --query PolicyDocument --output json > builder-policy.json
python3 - <<'EOF'
import json
d = json.load(open("builder-policy.json"))
d["Statement"] = [s for s in d["Statement"] if s.get("Sid") != "ReadFindings"] + [
    {"Sid": "ReadFindings", "Effect": "Allow",
     "Action": ["inspector2:ListFindings", "inspector2:ListCoverage"], "Resource": "*"}]
json.dump(d, open("builder-policy.json", "w"), indent=2)
EOF
aws iam put-user-policy --user-name datalayer-environments-builder \
  --policy-name datalayer-environments-builder --policy-document file://builder-policy.json

Check that the policy now lists six statements, ending with ReadFindings:

aws iam get-user-policy --user-name datalayer-environments-builder \
  --policy-name datalayer-environments-builder --query 'PolicyDocument.Statement[].Sid' --output text

On r1 the statement was added by hand on 2026-09-14. Confirm the worker itself can read the scans:

kubectl -n datalayer-durable exec deploy/datalayer-durable-environments -- python -c "
import boto3
c = boto3.client('inspector2', region_name='us-east-1')
c.list_coverage(maxResults=1); c.list_findings(maxResults=1); print('ok')"

Restarting it safely

The pool is one replica with strategy: Recreate. A build's layer cache lives with the pod that made it, so a rolling update would only run two builders with two half-warm caches.

The cost is that restarting it kills the build it is running. That happens with plane up datalayer-buildkit on a changed chart, with plane down, and with a node drain that evicts it. Before restarting it on purpose, check for builds in flight:

# With the runtimes plane's Solr forwarded (plane pf-solr):
curl -s -G 'http://localhost:8983/solr/environment-builds/select' \
  --data-urlencode 'q=status_s:(queued OR running) AND variant_s:datalayer' \
  --data-urlencode 'fl=id,status_s' | jq '.response'

There is no API route that lists builds across owners, so the check reads the collection. numFound: 0 means nothing is running. Otherwise, wait for each build to reach succeeded, failed or cancelled. Any build still running when the daemon goes ends failed with DL_ENV_PROVIDER_ERROR, which is retryable. A retry starts again from resolving, because nothing about a killed solve can be resumed. The daemon has no drain signal of its own: this check is the whole mechanism.

Tear down

plane down datalayer-buildkit

This deletes the release, and the layer cache PVC with it, so the next build starts cold. It also deletes the three certificate Secrets: cert-manager created them, so Helm does not delete them. The durable worker keeps running, and its builds fail until the pool is back. The build node keeps its label and taint.

Egress

A build reaches the package indexes, the snapshot mirrors and the registries, and nothing else, the cloud metadata address included. Flannel enforces no NetworkPolicy, so the pod holds it itself:

• The firewall. A RUN step runs in buildkitd's network namespace, which is the pod's (rootlesskit --net=host). The init container sets that namespace's OUTPUT chain (DL_EGRESS, IPv4 and IPv6) to let through loopback, replies, and the proxy's own user (uid 13) on DNS, 80 and 443, and to refuse everything else. buildkitd and its steps run as other users, so they cannot reach the network, not even DNS, except through the proxy. A step cannot change the firewall: it is root only inside rootlesskit's user namespace, which does not own the pod's network namespace.
• The proxy. Squid allows CONNECT on 443, and plain HTTP on 80, to allowedDomains only, and answers anything else 403. It caches nothing and logs every request to its container's output, TCP_TUNNEL for one it let through and TCP_DENIED for one it refused, with the host.
• Who uses it. buildkitd pulls and pushes through HTTP(S)_PROXY. A build's steps get the same address as build args, which the durable environments worker adds to every buildctl call when DATALAYER_BUILDKIT_PROXY is set (http://127.0.0.1:3128 in its chart, code-sandboxes 1.9.36 or later).

To let builds reach another host, a private index or conda channel for instance, add it to egressProxy.allowedDomains for the plane and run plane up datalayer-buildkit. A leading dot takes the domain and its subdomains.

To see what the proxy refused:

kubectl -n datalayer-builds logs deploy/datalayer-buildkit -c egress-proxy | grep TCP_DENIED

Drilled on r1 on 2026-09-18, with builds sent to this buildkitd over mTLS from a pod holding the worker's client certificate:

A RUN step that asks for	Through the proxy	Answer
https://pypi.org/simple/pip/	no	Could not resolve host: no DNS for a step
https://1.1.1.1/	no	Could not connect to server, refused in 0 ms
http://169.254.169.254/latest/meta-data/	no	Could not connect to server
https://pypi.org/simple/pip/	yes	HTTP 200
https://example.com/	yes	CONNECT tunnel failed, response 403
http://169.254.169.254/latest/meta-data/	yes	HTTP 403 from the proxy

Troubleshooting

Each of these was found standing the pool up on r1, and the chart or its clients now carry the fix.

Symptom	Cause	Fix
The pod stays Pending for CPU	A whole-core request fit on no r1 node, all at 76–96% CPU requested	The chart requests 500m and sets no CPU limit, so a build can burst
permission denied on the cache's lock file	A new PVC is mounted owned by root	fsGroup: 1000 with fsGroupChangePolicy: OnRootMismatch
rootlesskit cannot create a user namespace	The default seccomp and AppArmor profiles block it	Unconfined seccomp and AppArmor, while the container stays non-root and unprivileged
A RUN step fails mounting /proc with operation not permitted	Rootless per-step sandboxing needs systempaths=unconfined, which Kubernetes cannot set	--oci-worker-no-process-sandbox, as in BuildKit's own examples/kubernetes/pod.rootless.yaml. procMount: Unmasked does not help
Clients fail the TLS handshake	The daemon accepts mTLS only	code-sandboxes 1.9.1 or later, which passes --tlscert, --tlskey and --tlscacert to buildctl
A build stops at attest with AccessDeniedException on inspector2:ListCoverage	The builder policy lacks ReadFindings	See the builder's AWS permissions
A pull or a step fails with 403 or CONNECT tunnel failed, response 403	The host is not on egressProxy.allowedDomains. From AWS, Docker Hub serves blobs out of its S3 bucket in the caller's region, not Cloudflare, which is how the first pull through the proxy failed	Read the host from TCP_DENIED in the proxy's log and add it, see Egress
Every first resolve after an idle spell takes about 3 minutes, and buildctl du shows almost nothing cached	GC rules with no limit, which reclaim everything: keepBytes = 0, or a key BuildKit does not know, such as keepBytesFreePercent	BuildKit's own keys (keepDuration, reservedSpace, maxUsedSpace, minFreeSpace). Check what buildkitd parsed with buildctl debug workers --verbose

Known gaps

• The NetworkPolicy is not enforced. r1 and prod1 both run plain Flannel, which enforces no NetworkPolicy. Egress is held by the pod's own proxy and firewall instead; the policy stays for a plane whose network plugin enforces it.
• A build naming a host that is not on the list fails at the proxy. An index or a channel a spec names is not added to the allowlist for it: the plane's values are.
• The taint did not empty the build node on its own. A taint stops new pods, not running ones. r1-node-7-fb74 was drained by hand on 2026-09-18, one pod at a time, and now runs only buildkitd, Falco, the image pruner and the per-node daemons.
• The r1 rc carries AWS credentials for another account. Sourcing datalayerrc-r1 exports AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for account 575108930674, while the Environments registry is in 773842031886. Anything run after sourcing it uses the wrong account: clouder aws ecr-environments plan planned a whole new registry there, and a base channel release was refused at docker push. That is why r1's ReadFindings statement was added by hand. Run registry commands in a shell without the rc, or unset those variables first, and check with aws sts get-caller-identity.