Skip to main content

☰ 🥷 Datalayer Operator

KubernetesService

Datalayer Operator ensures that the requested services and resources are available. Datalayer Operator is a Kubernetes operator that constantly monitors the system and takes action so that the desired state is met on the Kubernetes cluster. It is responsible for managing the state of the Datalayer services.

- 🛣️ Manages the Routes to secure and direct the incoming request to the correct service or pod.
- 🐋 Manages the Docker Images building and publication in the Docker registry.
- 💼 Manages the Runtime Contents.
- 🌳 Manages the Runtime Environments
- 🎱 Manages the Runtime Pools.
- 📎 Provisions the Volumes and mounts the Contents attachments — Home Folders, Volumes, local bridges — into the Runtime pods, and tells Contents how each came up.

Please note that the Runtimes Service is responsible to deliver the ⚪ Runtimes to the user, so it is not the duty of the Datalayer Operator to serve Runtimes.

A subset of the Operator methods is exposed via REST endpoints for internal consumption (not visible from external clients).

Deploy Datalayer Operator​

plane up datalayer-operator

After the upgrade: re-apply the Runtime Contents and the Environments​

Since Environments select their Runtime Contents by uid, the Operator spawns nothing from an Environment still applied in the earlier shape (contents: [{name: ...}]), and nothing from one whose uids match no RuntimeContent in the cluster — every pool pod on it is refused before it is created. Upgrading the image is therefore not the whole upgrade. Apply the definitions first (each carries its spec.uid), then the Environments:

kubectl apply -n datalayer-runtimes -f plane/etc/specs/runtime-contents/
kubectl apply -n datalayer-runtimes -f plane/etc/specs/runtime-environments/prod/
kubectl get runtime-contents -n datalayer-runtimes -o custom-columns=NAME:.metadata.name,UID:.spec.uid,TYPE:.spec.type
kubectl get runtime-environments -n datalayer-runtimes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.contents[*].uid}{"\n"}{end}'

Every content must show a UID and every Environment a list of uids; a name in an Environment's contents is the old shape.

When no pool pod comes up. The Operator now says why, at startup and on every reconcile, on its own logger:

plane logs datalayer-operator | grep -E "startup_check|pool_reconcile|add_runtime_pod"

[startup_check] Environment 'X' still selects its contents by name … or … selects a Runtime Content the cluster cannot deliver … names the spec to re-apply; [startup_hydrate] runtime contents (name: uid) lists what the cluster defines; [add_runtime_pod] FAILED carries the traceback of a spawn that failed for any other reason. The pool retries a minute later, so a re-applied spec is picked up without a restart.

Cloud Credentials (AWS/Azure)​

For Operator scaling/executor workflows, credentials are consumed through a Kubernetes Secret referenced by the Operator Deployment.

  • plane up datalayer-operator configures Helm values to create and mount ${RELEASE}-cloud-credentials.
  • plane down datalayer-operator removes the release and the generated ${RELEASE}-cloud-credentials Secret.
  • If you manage credentials outside Plane, keep operator.cloudCredentials.enabled=true, set operator.cloudCredentials.create=false, and point operator.cloudCredentials.secretName to your existing Secret.

For clusters created with clouder kubeadm, the recommended flow is to generate a cluster-scoped Helm values JSON and let Plane use it automatically.

  1. Ensure Azure is configured in Clouder.
clouder azure configure
  1. Generate operator cloud credentials JSON for your cluster.
clouder azure helm-values --cluster my-cluster

By default this creates:

~/.clouder/kubeadm/r1/datalayer-operator-azure.json
  1. Deploy the operator.
plane up datalayer-operator

plane/datalayer_plane/sbin/up.sh now checks the current kube context and, when present, automatically applies:

~/.clouder/kubeadm/<current-context>/datalayer-operator-azure.json
  1. Optional: override the file path explicitly.
export DATALAYER_OPERATOR_AZURE_VALUES_FILE=/path/to/datalayer-operator-azure.json
plane up datalayer-operator
  1. Optional: deploy with Helm directly.
helm upgrade \
  --install datalayer-operator \
  oci://${DATALAYER_HELM_REGISTRY_HOST}/datalayer-charts/operator \
  --namespace datalayer-runtimes \
  --create-namespace \
  --values ~/.clouder/kubeadm/r1/datalayer-operator-azure.json

The generated JSON includes:

  • operator.cloudCredentials.azure.tenantId
  • operator.cloudCredentials.azure.clientId
  • operator.cloudCredentials.azure.clientSecret
  • operator.cloudCredentials.azure.subscriptionId
  • operator.cloudCredentials.azure.resourceGroup
plane ls

Check the availability of the Datalayer CRDs.

kubectl get crd runtime-contents.datalayer.io
kubectl explain runtime-contents.spec
kubectl get crd runtime-environments.datalayer.io
kubectl explain runtime-environments.spec
kubectl get crd runtime-pools.datalayer.io
kubectl explain runtime-pools.spec

Check the availability of the Datalayer Operator Pods.

kubectl get pods -n datalayer-runtimes -l app=operator

Check the logs of the Datalayer Operator Pods.

kubectl logs -n datalayer-runtimes -l app=operator -f

Runtime Contents and Environments​

A RuntimeContent defines something an Environment can bring into a runtime: a git repository, a subpath of the platform shared filesystem (nfs), or a S3 bucket. Every content carries a stable identity, spec.uid (a ULID assigned once, when the content is defined), and Environments select contents by uid, never by name — a content can be renamed or re-described without every Environment that uses it quietly pointing elsewhere.

apiVersion: datalayer.io/v1
kind: RuntimeContent
metadata:
  name: jakevdp-sklearn-tutorial-content
  namespace: datalayer-runtimes
spec:
  uid: 01M0YX0MXYRP0Q29YWYRE5THZE
  type: git
  description: Jake VanderPlas scikit-learn tutorial
  source: https://github.com/jakevdp/sklearn_tutorial.git
  revision: 5098cee2a638c56c311aca0c18987e407fe127fd
FieldMeaning
uidThe identity Environments select by. Required.
typegit, nfs or s3.
sourceThe repository URI (git) or the bucket name (s3).
subPathThe subpath of the shared filesystem (nfs).
revisionThe commit sha a git content is checked out at. A git content without a revision is refused at launch: a tutorial that changes under a user is not the tutorial they were promised.
sha256Optional. The digest of git archive --format=tar <revision>; when set, the checkout is verified and the runtime pod fails to start on a mismatch.
permissionsThe content's own access, ro (default) or rw.
secretThe Kubernetes Secret holding the bucket credentials (s3).

An Environment selects what it brings, and where:

apiVersion: datalayer.io/v1
kind: RuntimeEnvironment
spec:
  contents:
    - uid: 01M0YX0MXYD8YZVMJW016KTB4M # nfs-models-oss-content
      mount: /home/jovyan/models/datalayer-oss
      permissions: ro
    - uid: 01M0YX0MXYRP0Q29YWYRE5THZE # jakevdp-sklearn-tutorial-content
      mount: /home/jovyan/tutorials/sklearn-tutorial

The permissions of a selection narrows the content's own access and is never wider than it; absent means the content's own. The name after # is a comment for the reader — the Operator resolves names, types and revisions from the definitions when it lists Environments.

At launch, the Operator:

  • resolves every selected uid against its index of RuntimeContent definitions (the kopf cache, with the API server as fallback). A uid nothing defines, or a git content with no revision, refuses the launch with an error naming the uid — a runtime never starts with less than its Environment promised;
  • mounts only the selected entries. On a pod spawned for the launch, each type is rendered into the spec: a git content is checked out at its pinned revision by an init container (alpine/git) that fails the pod when the revision is missing or the digest does not match; a nfs content is a subpath of the shared filesystem claim; a s3 content is its Datashim claim. On a pooled pod none of that is possible — the spec was fixed before the Environment was known — so the same three become gateway grants (git, shared-folder or nfs, cloud-storage) applied after the pod is running. A content the gateway cannot deliver refuses the pool and spawns instead, so a runtime never starts short of what its Environment promised;
  • publishes the resolved contents on the pod, in the contents.datalayer.io/attachments annotation the Contents attachments already use, as delivery: environment entries carrying mount_path, mode and provider_resource_id (the RuntimeContent uid). The sandbox manifest therefore lists everything the runtime mounts, wherever it came from. These entries have no Contents attachment record behind them: nothing is prepared, reported or revoked for them.

A platform administrator changes what an Environment brings through the Runtimes gateway (PUT /api/runtimes/v1/environments/<name>/contents), which proxies to the Operator's PUT /api/operator/v1/environments/<name>/contents with a body of the form {"contents": [{"uid": "...", "mount": "/home/jovyan/...", "permissions": "ro"}]}. The Operator resolves every uid before writing, patches the RuntimeEnvironment spec, and answers 422 naming a uid nothing defines, a git content nobody pinned, or a mount path that is not a unique absolute path.

kubectl get runtime-contents -n datalayer-runtimes
kubectl get runtime-environments -n datalayer-runtimes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.contents[*].uid}{"\n"}{end}'

Contents attachments​

The Contents service keeps the catalog — what an account may bring into a sandbox — and the Operator is the one process that turns an attachment into something a pod can read. Contents never talks to Kubernetes; the Operator never decides who may attach what. Three things are the Operator's, in the order they happen.

Volumes: the PersistentVolumeClaims​

A Volume source in the Contents catalog is a PVC in datalayer-runtimes, made by the Operator on Contents' request. The request is not inline any more: creating a Volume source records a volume-provision operation that the Contents worker claims and drives against the Operator's REST endpoints — POST /api/operator/v1/volumes to create, GET /volumes/{uid} until the claim is Bound, DELETE /volumes/{uid} as the compensation when every attempt is spent. The calls carry the owner in x-forwarded-user and the Operator's own key (DATALAYER_OPERATOR_API_KEY); the Volume's uid is the source's uid, so a claim is always traceable to its catalog entry. A source is trusted — ready, mountable — only once the claim is bound. When a pod mounts a Volume the Operator records the assignment on the PVC and clears it when the runtime is deleted, so a Volume in use is never re-attached elsewhere.

kubectl get pvc -n datalayer-runtimes

At launch: rendering the attachments​

Runtimes launches a sandbox that has attachments with POST /api/operator/v1/runtimes carrying content_attachments — the attachments it just had Contents prepare for that pod name (see Runtimes). The Operator refuses the launch (4xx) when an attachment is not in preparing state, names another sandbox or another provider than datalayer, or when two of them share a mount path or one is not an absolute path. Then, per delivery:

AttachmentWhat the pod gets
Home Folder (a files source, stamped by Runtimes)subpaths of the platform shared filesystem claim (DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME), one per membership the caller has — their own, each organization, each team — at /home/jovyan/<handle>; an init container creates the folders and hands them to the runtime user. The list of memberships is resolved by Runtimes from IAM and travels on the pod as runtime-pools.datalayer.io/home-folder-mounts. Contents reads the same claim at the same path, which is why the two services' mount paths must agree. With the Node Mount Gateway on, the same folders reach the same paths through a pooled pod instead — see below
mount of a Volumethe Volume's PVC, at the attachment's mount_path (default /home/jovyan/volumes/<volume name>), read-only when the mode is ro. A Volume of another account refuses the launch; a required Volume that cannot be found refuses it, an optional one is dropped
local-bridge (a person's own folder over the relay)an inline csi: volume of the local.csi.datalayer.io driver — content-bridge-<n>, attributes bridge-uid, sandbox-uid, mount-mode, relay-url — with nodePublishSecretRef pointing at the Secret bridge-<bridge uid> (key mount-token, label contents.datalayer.io/bridge-uid). The Secret is created right after the pod and owned by it, so the token goes when the pod goes; the pod spec names the bridge and never carries the token. The Local CSI driver on the node dials the relay and binds the filesystem before the container starts. An attachment Contents did not prepare a bridge session for is reported failed with BRIDGE_NOT_PREPARED: a required one refuses the launch, an optional one is left out — never rendered as an empty directory
environment (brought by the Environment, see above)the pinned git checkout, the shared-filesystem subpath or the Datashim claim; no Contents record, nothing reported
anything elserefused when required, dropped when optional

The rendered list is written on the pod as the contents.datalayer.io/attachments annotation. It is the record: a pod spec rebuilt from it renders the same mounts, the Operator's cache is rebuilt from it at restart, and the runtime's API description lists what the pod actually mounts rather than something re-derived from billing labels.

kubectl get pod -n datalayer-runtimes <pod> -o jsonpath='{.metadata.annotations.contents\.datalayer\.io/attachments}'
kubectl get secret -n datalayer-runtimes -l contents.datalayer.io/bridge-uid

Reporting to Contents​

The Operator tells Contents how each attachment came up, with PATCH /api/contents/v1/attachments/{uid}/status under DATALAYER_OPERATOR_API_KEY (the attachments:status scope):

  • ready once the pod runs — for the Home Folder and the Volumes. A local-bridge is not reported ready by the Operator: the volume is there, but the bytes are readable only once the bridge connects, and that is the relay's to say;
  • failed, with a code and a message, for what could not be mounted. A required attachment whose report cannot be delivered ends the launch: the runtime is deleted (attachment_failed) and the caller gets 502;
  • revoked when the runtime is deleted, after the Volume assignments are cleared — so the catalog never lists a sandbox that is gone;
  • ready again for every attachment of a pod the Operator finds when it (re)starts and indexes the cluster, so a restart of the Operator does not leave attachments stuck in preparing.

Environment entries are skipped in all four: nothing prepared them and there is no record to update.

Configuration​

VariableMeaning
DATALAYER_CONTENTS_URLWhere the Operator reports (operator.env in the chart; default the in-cluster datalayer-contents-svc, https://r1.datalayer.run in the code).
DATALAYER_OPERATOR_API_KEYThe Operator's identity to Contents — one key per calling process, so it can be rotated without touching Runtimes' and the audit trail says which acted. Required as soon as one attachment exists: without it the report raises.
DATALAYER_SHARED_FS_VOLUME_CLAIM_NAMEThe shared filesystem claim the Home Folders live on. Contents mounts the same claim at the same path (DATALAYER_SHARED_FS_MOUNT_PATH); a mismatch makes a path recorded by one unresolvable in the other.
DATALAYER_NODE_MOUNT_GATEWAY_ENABLEDWhether pooled pods carry the Node Mount Gateway and a Home Folder launch is served from the pool. Off by default. Turn it on after the node agent is deployed: pods would otherwise carry the volume and wait for mounts nobody makes.
DATALAYER_NODE_MOUNT_GATEWAY_READY_TIMEOUT_SECONDSHow long the Operator waits for the node agent to report a grant applied (default 20). Past it the launch is reported without its folders, loudly.
DATALAYER_NODE_MOUNT_GATEWAY_MAX_MOUNTS_PER_PODThe most folders one sandbox may be granted (default 32).

Pool or its own Pod: which launches pay for a cold start​

A Pod's volumes are fixed when it is created, so a launch that mounts something used to need a Pod of its own — scheduling, an image pull, the init containers and a Jupyter startup probe — while a launch that mounts nothing cost one PATCH against a prewarmed Pod.

With DATALAYER_NODE_MOUNT_GATEWAY_ENABLED that is no longer true of the Home Folder. What decides, per launch:

The launchWhere the Pod comes from
Mounts nothingThe pool.
Home Folder only (the home-folder capability, or a files attachment), gateway on, a pooled Pod carrying the gatewayThe pool, with a hot attach: the Operator writes the mount set on the Pod and the node agent binds it in.
Home Folder only, gateway offIts own Pod.
A Volume, a Local Mount, or anything else the gateway cannot serveIts own Pod, whatever the gateway setting. A Volume has a PVC nothing staged on the node; a Local Mount is an inline CSI volume fixed at creation.
Home Folder only, gateway on, but the pool holds only Pods built before itIts own Pod. Handing out a pre-gateway Pod would start a runtime whose folders never arrive.

The hot-attach path writes runtime-pools.datalayer.io/node-mount-gateway-mounts on the Pod and waits for the node agent's answer on …/gateway-mounts-ready — the hash it asked for, not merely a recent answer — then asks the companion to link /home/jovyan/<handle> to the gateway path. The mount set is resolved from the memberships Runtimes resolved against IAM with the caller's own credentials, never from what the request named. The whole node side is on the Local CSI driver page.

Endpoints, for a runtime that is already running:

GET    /api/operator/v1/runtimes/{runtime}/mounts            # granted, and what is applied
POST   /api/operator/v1/runtimes/{runtime}/mounts            # mount the caller's home folders
DELETE /api/operator/v1/runtimes/{runtime}/mounts/{target}   # take one out

POST takes attachment uids, never records: Runtimes resolves and prepares them through Contents with the caller's credentials, exactly as at launch, and drops any content_attachments a request carries. A client that could hand over an attachment record could hand over a bridge session it does not own.

It answers 409 when the gateway is off in this deployment or the Pod was created without it, 422 for an attachment the gateway cannot serve, and 504 when the node agent did not apply the mounts in time — never 200 for a mount nobody confirmed.

What can be attached to a running Runtime: a Home Folder, a Local Mount, and a mounted bucket — the last two only where their path is under /home/jovyan. A Volume cannot — its PVC is not staged on the node — and neither can a Local Mount asking for any other absolute path, because the gateway delivers under the home folder. A creation-time mount can be anywhere and still can; the refusal names the reason (LOCAL_BRIDGE_PATH_UNSUPPORTED) rather than mounting somewhere the caller did not ask for. A hot-attached Local Mount's token goes into the Pod-owned bridge-<uid> Secret, which the Operator creates before it writes the grant, so the agent never looks for a Secret that is not there. A mounted bucket's STS session goes into mount-<attachment uid> the same way.

Keeping a mounted bucket alive​

A bucket's session expires and a runtime does not. refresh_mount_sessions runs in the Operator every DATALAYER_MOUNT_SESSION_REFRESH_INTERVAL_SECONDS (60 by default, and off entirely when the gateway is off): it finds every session Secret within ten minutes of expiry by its contents.datalayer.io/mount-attachment-uid label, asks Contents for a newer one at POST /attachments/{uid}/mount-session, and replaces the Secret in place. The node agent's credential endpoint re-reads it and the filesystem never stops running — nothing is unmounted and no open file handle is disturbed.

The same pass also heals a rebuilt Pod. A Pod spec rebuilt from its annotations carries the grant — writing the mount set on the Pod is the point of it — but not the session, which is deliberately never written there, and the Secret was owned by the Pod that is gone. So a grant with no Secret is treated as a session to mint: the loop finds it, mints one, and creates the Secret owned by the new Pod. The node agent retries the grant on its next reconcile and the bucket comes back. A Pod that is terminating is skipped — its Secret is gone because the Pod is going, and minting one would create a credential for something already leaving.

A Secret whose expiry cannot be read is refreshed anyway: a session nobody can date is one nobody can trust, and re-minting costs one API call. One that cannot be re-minted is logged and skipped; that mount keeps working until its session ends and then reports itself through the health path.

kubectl -n datalayer-runtimes get secret -l contents.datalayer.io/mount-attachment-uid
kubectl -n datalayer-api logs deploy/datalayer-operator | grep 'mounted bucket session'

Tear Down Datalayer Operator​

If needed, tear down.

plane down datalayer-operator