☰ 📎 Datalayer Node Mounts

KubernetesNode DaemonSet shippedLocal CSI driver retired (audit 82)

Everything Datalayer mounts on a Kubernetes node comes from one privileged DaemonSet, datalayer-node-mounts. Since audit 82 it runs one node service, the Node Mount Gateway (--node-mount-gateway-only), because a node has one mount table, and two components owning it is how a leaked mount goes unnoticed.

The Node Mount Gateway mounts home folders, datasets, NFS exports, Git checkouts, buckets, and a folder of a user's own computer, over the Contents local bridge (how), into a Pod that is already running, so a launch that mounts content is served from the prewarmed pool instead of a cold Pod.

note

The Local CSI driver, local.csi.datalayer.io, is retired (audit 82). It once delivered that local folder as an inline CSI volume at Pod creation. The gateway has taken the job over, over the same Contents bridge relay, and the local folder is the one capability that changed hands.

Each service has its own page. This one is about the DaemonSet: the image, the chart, the deployment, the checks, and how a folder of a user's computer reaches a sandbox today.

A folder of a user's own computer, now

What was retired is only the delivery. The CSI driver got the folder into a sandbox by one means: the Operator rendered an inline local.csi.datalayer.io volume on the Pod when it was created, and kubelet asked the driver to publish it. That rendering is gone from the Operator, and the chart no longer deploys the CSIDriver object or its registrar (driver.gatewayOnly: true, which plane up forces).

Everything that carries the files is still there, and unchanged:

• the local client on the person's computer (the CLI, Desktop, VS Code or JupyterLab), which owns the folder and serves it;
• the Contents bridge relay, datalayer-contents-bridge on 9402, published at wss://<run host>/bridges/<bridge uid> through the API host's ingress, which pairs the two ends and forwards frames it cannot read;
• the bridge filesystem, clouder.csi.bridge_mount (fusepy, from code-sandboxes[bridge]), which turns those frames into a mounted folder.

The Node Mount Gateway now starts that same filesystem, as a local-bridge process grant, into a Pod that is already running.

1. The person mounts a folder from their client (Local Mounts in the user manual). Contents records an attachment with delivery: local-bridge and opens a bridge session: a bridge uid, a short-lived mount token and a session key. The folder lands at /home/datalayer/<name>, and only there (LOCAL_MOUNT_PATH_UNSUPPORTED otherwise).
2. Contents asks for the mount, as the person, through Runtimes' POST /runtimes/{name}/mounts, the moment the session exists (the mount token comes from the session, so no earlier moment would do). A sandbox not launched yet (404) gets the attachment with its launch instead.
3. The Operator writes a local-bridge grant on the Pod's annotation, and a Secret owned by the Pod holding the mount token, the relay URL and the session key. The grant names the Secret, never the values.
4. Both ends dial out to the relay. The client already has; the gateway on that node reads the grant, reads the Secret (refusing any Secret the Pod does not own), and starts the bridge filesystem with the mount token and the session key in its environment, never in its argv. The relay URL, not a secret, is an argument, and must be on relay.host. The relay pairs the two by bridge uid and forwards frames sealed end to end with the session key: it cannot read them. The person's computer opens no inbound port.
5. The filesystem mounts at the grant's target inside the gateway's tree for that Pod, and the mount propagates into the sandbox, where it appears at /home/datalayer/<name>, owned by the sandbox's user (--allow-other, --uid 1000 --gid 100).

Ending it. datalayer contents unmount, the Unmount button and DELETE /api/contents/v1/bridges/{uid} end the session and take the folder out of the sandbox: Contents asks Runtimes to detach it, and the gateway stops the filesystem before removing its mount point, so no process is left serving a path nothing can reach. A filesystem that dies instead leaves the mount answering errors, reported as NODE_MOUNT_GATEWAY_MOUNT_DEAD on the Pod's ready annotation: the Pod is degraded while another of its mounts is fine, and failed when that was its only one.

What a deployment needs for it, all on the runtimes plane. plane up reads the same variables for the DaemonSet and the Operator, so export them once in the cluster's datalayerrc and deploy both:

Piece	Setting
Contents, with the relay	The Secret datalayer-contents-bridge in datalayer-api, with the keys secret and api-key. plane up datalayer-contents turns the relay on exactly when it exists, and wires processes.bridge.enabled, DATALAYER_CONTENTS_BRIDGE_SECRET and DATALAYER_CONTENTS_BRIDGE_API_KEY from it. See Contents.
The Operator, writing grants	DATALAYER_NODE_MOUNT_GATEWAY_ENABLED, DATALAYER_NODE_MOUNT_GATEWAY_CREDENTIALS and DATALAYER_NODE_MOUNT_GATEWAY_LOCAL_BRIDGES, all true. With either of the last two off, the Operator refuses the grant with GATEWAY_KIND_OFF.
This DaemonSet, running the filesystem	DATALAYER_NODE_MOUNTS_ENABLED, DATALAYER_NODE_MOUNT_GATEWAY_ENABLED, _CREDENTIALS and _LOCAL_BRIDGES, all true, and DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME set. relay.host, derived from DATALAYER_CONTENTS_URL, must be the relay's host, and the NetworkPolicy must let the agent reach it (DATALAYER_LOCAL_CSI_RELAY_CIDR, _PORT). The image carries code-sandboxes[bridge] and user_allow_other in /etc/fuse.conf.

Without the gateway, a Datalayer runtime cannot take a local mount, and nothing refuses it up front. Runtimes' launch gate (422 LOCAL_BRIDGE_UNSUPPORTED) judges external sandboxes only. On Datalayer, an Operator with the gateway or its local-bridge switches off logs GATEWAY_KIND_OFF and launches the sandbox without the folder. A mount asked for a running sandbox is answered 409 (gateway off) or 422 (switches off); Contents logs the answer and the session stays open, so the person's client waits for a sandbox end that never joins (datalayer contents mounts shows it).

External sandboxes need none of this. An E2B, Modal or Daytona sandbox has no node agent: it runs the same bridge filesystem itself when its environment lists the fuse feature, against the same relay.

Why the DaemonSet is not called local-csi any more

It was, when the Local CSI driver was the only thing in it and a laptop's folder the only thing it mounted. The gateway joined it and mounted everything else, and since audit 82 it mounts the laptop's folder too, while the driver is retired.

Renaming the package after CSI would have been worse. The gateway is deliberately not a CSI driver: a PVC cannot attach to a Pod that is already running, which is the whole reason the gateway exists. Calling the package datalayer-csi would have named it after the mechanism it was built to avoid. So the retired driver keeps local.csi.datalayer.io, and the package is named for what it does: Datalayer's mounts, on a node.

The gateway and the retired driver

They share an image, a chart and an upgrade. By default the DaemonSet is gateway-only (driver.gatewayOnly: true, which plane up forces), which always starts the gateway agent. nodeMountGateway.enabled is what makes that agent work: the Pod RBAC, the API server egress, the shared claim and the mount-kind switches. Deploy the DaemonSet only with the gateway on. Without it, the agent cannot list Pods and mounts nothing, and clouder node-mounts verify fails the claim and patch checks.

	Node Mount Gateway	Local CSI driver, retired (audit 82)
What it mounts	Home folders, datasets, NFS exports, Git checkouts, buckets, and a folder on a user's own computer (over the Contents bridge relay)	Once: a folder on a user's own computer (that delivery is now a gateway grant)
How a Pod gets it	A grant on the Pod's annotation, applied while the Pod is already running	Was an ephemeral inline CSI volume the Operator rendered at Pod creation
When it can act	Any point in the Pod's life, and revocable	Pod creation only, which is exactly the limitation the gateway removed
Is it CSI?	No, deliberately: a PVC cannot attach to a running Pod, which is why the gateway exists	Yes, local.csi.datalayer.io
Switch	nodeMountGateway.enabled, off by default	None that brings it back into use: driver.gatewayOnly: false (a direct Helm install only) renders its objects, and no Operator renders a volume for them

What ships in Clouder

Piece	Where
Image	plane/etc/dockerfiles/datalayer-node-mounts → datalayer/node-mounts:0.2.4
Chart	plane/etc/helm-private/charts/datalayer-node-mounts
Install	plane up datalayer-node-mounts (needs DATALAYER_NODE_MOUNTS_ENABLED=true)
Check	clouder node-mounts status, clouder node-mounts verify, locally or over SSH to a kubeadm master with --cluster
Entrypoint	python -m clouder.csi --node-id=$(NODE_ID) --node-mount-gateway-only, plus --node-mount-gateway, --shared-root, --node-mount-gateway-root and the kind switches when nodeMountGateway.enabled, and --relay-host from relay.host

The tag lives in three places: the Makefile's IMAGE, the chart's driver.image, and the --set driver.image= in up.sh. The up.sh one wins on every plane up.

The image is Clouder, built with:

• the csi extra;
• code-sandboxes[bridge], the bridge FUSE filesystem (without the extra, every local mount fails FUSE_UNAVAILABLE);
• mount-s3 (Mountpoint for S3);
• fuse3, util-linux mount, git and curl, and user_allow_other in /etc/fuse.conf.

The chart

Objects datalayer-node-mounts ships, all in datalayer-runtimes:

• a DaemonSet on the nodes labelled role.datalayer.io/runtime=true (driver.nodeSelector), with the privileged driver container, /var/lib/kubelet/pods mounted at the same path Bidirectional, and /dev/fuse. The upstream csi-node-driver-registrar sidecar and the /var/lib/kubelet/plugins/local.csi.datalayer.io socket mount are rendered only when driver.gatewayOnly is false (retired; audit 82);
• a ServiceAccount with a ClusterRole limited to nodes: get and events: create, patch, with no Secrets. The gateway adds pods: get, list, watch, patch to it when it is on, and reading a Secret is a separate, namespaced Role;
• a CSIDriver local.csi.datalayer.io, rendered only when driver.gatewayOnly is false (retired; audit 82) (details);
• a NetworkPolicy (networkPolicy.enabled) allowing egress to DNS, to the bridge relay port, and, when the gateway is on, to the API server;
• a PrometheusRule (monitoring.prometheusRule, off by default because it needs the Prometheus operator's CRD and an unappliable manifest fails the whole release).

The values divide this way: nodeMountGateway.* is the gateway's; driver.* (image, health port, log level, node selector, resources) is the one container's; relay.* and driver.allowInsecureRelay are what the gateway's local-bridge filesystem dials and may dial; monitoring.* and apiServer.* are the DaemonSet's. Only registrar.* and driver.gatewayOnly: false belong to the retired driver alone.

Deploy

The DaemonSet joins the system layer after the shared filesystem and before the Operator: the Operator writes the grants the gateway applies, and must not write them before an agent is there to apply them.

Plane

Off by default: it is privileged and runs on every runtimes node, so a cluster serving no gateway mounts should not have it. Turn it on only together with the gateway.

Everything below reads from the cluster's environment, so set that once — a datalayerrc per cluster is the usual place — and source it rather than exporting by hand:

source ~/.datalayer/datalayerrc-<cluster>

1. Build and push the image. The chart pulls ${DATALAYER_DOCKER_REGISTRY}/node-mounts:0.2.4, which does not exist until you build it:

cd $DATALAYER_SERVICES_HOME/plane/etc/dockerfiles/datalayer-node-mounts
make build push

It takes two named build contexts — Clouder with its csi extra, and code-sandboxes for the bridge filesystem — resolved from DATALAYER_HOME, so neither needs to be on PyPI. A release should pin MOUNTPOINT_URL to a Mountpoint version rather than taking latest.

2. Deploy.

plane up datalayer-node-mounts

up.sh pins the image and forces driver.gatewayOnly=true, so a change to the chart's defaults cannot bring the retired driver back. The command it prints afterwards, kubectl get csidriver local.csi.datalayer.io, answers NotFound, which is expected.

3. Check before moving on. verify is a preflight, not a smoke test: it reports what is wrong on the node while it is still cheap to fix.

kubectl -n datalayer-runtimes rollout status daemonset/datalayer-node-mounts
clouder node-mounts verify

The environment up.sh reads:

Variable	Sets	Notes
DATALAYER_NODE_MOUNTS_ENABLED	whether to deploy at all	Off by default. DATALAYER_LOCAL_CSI_ENABLED is still read when it is unset
DATALAYER_NODE_MOUNT_GATEWAY_ENABLED	nodeMountGateway.enabled	The same variable the Operator reads, so one export turns on both ends of the gateway
DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME	nodeMountGateway.sharedFilesystemClaim	Required with the gateway. Asking for one without the other fails the deploy rather than installing an agent with nothing to bind
DATALAYER_KUBERNETES_API_SERVER_CIDR	apiServer.cidr	The control plane, for the NetworkPolicy
DATALAYER_KUBERNETES_API_SERVER_PORT	apiServer.port	See the warning below
DATALAYER_NODE_MOUNT_GATEWAY_CREDENTIALS, _BUCKETS, _LOCAL_BRIDGES, _REPOSITORIES	the four mount-kind switches	All off unless set. The Operator reads the same names
DATALAYER_LOCAL_CSI_RELAY_CIDR, _PORT	relay.cidr, relay.port (443 by default)	The Contents bridge relay that the gateway's local-bridge filesystem dials. The names are older than the gateway

relay.host is derived from DATALAYER_CONTENTS_URL (or DATALAYER_RUNTIMES_URL).

The API server port is not 443 on kubeadm

apiServer.port defaults to 443, which is the port pods use to reach the kubernetes.default.svc Service. Most CNIs evaluate a NetworkPolicy after the Service is resolved, so the destination is the endpoint behind it — and a kubeadm control plane answers on 6443.

Leaving the default on such a cluster produces the worst failure this component has: the DaemonSet is healthy, the chart is right, nothing logs an error, and the agent's Pod watch is blocked — so it mounts nothing, for ever. Set DATALAYER_KUBERNETES_API_SERVER_PORT=6443 there. kubectl cluster-info prints the port your control plane is on.

clouder kubeadm setup

This path does not install a working DaemonSet today

clouder kubeadm setup --node-mounts installs the chart in Step 7, but:

• its default image, datalayer/node-mounts:0.1.0, predates the --node-mount-gateway-only flag the chart passes;
• its install script ends on kubectl get csidriver local.csi.datalayer.io, which the gateway-only chart no longer creates, so the step is reported failed and node_mounts_ready is recorded false;
• --node-mounts-gateway needs a shared claim that no option passes, so it always fails;
• it has no option for the credentials or localBridges switches, so it can never serve a local folder.

Use plane up datalayer-node-mounts, from the Plane tab.

The whole sequence, end to end

The DaemonSet is one step of several, and the order matters — turning the Operator's half on before the agent exists gives you pooled Pods carrying a gateway volume and waiting for mounts nobody makes. Everything here reads the cluster's environment, so source it once:

source ~/.datalayer/datalayerrc-<cluster>

# 1. The claim every home folder lives on, if it is not already there.
plane up datalayer-shared-filesystem

# 2. The images. Neither exists in the registry until it is built.
(cd $DATALAYER_SERVICES_HOME/plane/etc/dockerfiles/datalayer-node-mounts && make build push)
(cd $DATALAYER_SERVICES_HOME/plane/etc/dockerfiles/datalayer-contents     && make build push)

# 3. For Local Mounts: DATALAYER_NODE_MOUNT_GATEWAY_CREDENTIALS=true and
#    DATALAYER_NODE_MOUNT_GATEWAY_LOCAL_BRIDGES=true in the rc (the DaemonSet
#    and the Operator read both), and the relay's Secret, once:
kubectl create secret generic datalayer-contents-bridge -n datalayer-api \
  --from-literal=secret="$(openssl rand -base64 48 | tr -d '\n')" \
  --from-literal=api-key=<the relay's service key>

# 4. The node agent, BEFORE the Operator.
plane up datalayer-node-mounts
kubectl -n datalayer-runtimes rollout status daemonset/datalayer-node-mounts
clouder node-mounts verify

# 5. Contents, which serves the Home Folder browser from the same claim name.
plane up datalayer-contents
kubectl -n datalayer-api rollout status deploy/datalayer-contents

# 6. The Operator, which writes the grants the agent applies.
plane up datalayer-operator
clouder node-mounts verify        # now expects both halves on

Contents and the runtimes each mount a claim of that name in their own namespace, both on one directory of the filesystem (One directory, a claim per namespace).

If the Contents deploy carried a schema change, move the stored documents after it — the command is idempotent and resumes if interrupted, and the reindex has the detail:

kubectl exec -n datalayer-api deploy/datalayer-contents -- datalayer-contents-reindex --dry-run
kubectl exec -n datalayer-api deploy/datalayer-contents -- datalayer-contents-reindex

Then prove it end to end: a launch that mounts the Home Folder should come from the pool rather than a cold Pod. That is Turning it on, and proving it works on the gateway's page, with the measurement to take.

Verify

kubectl -n datalayer-runtimes rollout status daemonset/datalayer-node-mounts
kubectl -n datalayer-runtimes logs daemonset/datalayer-node-mounts -c driver --tail=50
clouder node-mounts status          # the DaemonSet, and the "Node Mount Gateway" table of mounts
clouder node-mounts status --json   # each node's gateway state under pods[].gateway
clouder node-mounts verify          # propagation, RBAC, the claim, and anything left behind

On a gateway-only DaemonSet, status prints CSIDriver …: missing and leaves its Bridges and Volumes columns empty: those are the retired driver's, and that is expected. A local folder the gateway serves is listed in the "Node Mount Gateway" table.

Proving a local mount end to end:

# On the person's computer, with a Datalayer sandbox running:
datalayer contents mount ./project --sandbox SANDBOX_UID --path /home/datalayer/project
datalayer contents mounts             # the session, and whether the sandbox end joined

# On the cluster: the Pod's answer, and its Pod-owned Secret.
kubectl get pod -n datalayer-runtimes POD \
  -o jsonpath='{.metadata.annotations.runtime-pools\.datalayer\.io/node-mount-gateway-ready}'
kubectl get secret -n datalayer-runtimes -l contents.datalayer.io/bridge-uid=BRIDGE_UID
clouder node-mounts status            # the "Node Mount Gateway" table lists `project`

# In the sandbox: ls -la /home/datalayer/project. Then end it:
datalayer contents unmount BRIDGE_UID

Proving the gateway's other kinds is on the Node Mount Gateway page.

The full node suite (test_csi_kernel.py skips itself where it cannot run):

cd clouder
python -m pytest clouder/tests/test_csi_driver.py clouder/tests/test_csi_server.py \
  clouder/tests/test_csi_bridge_mount.py clouder/tests/test_csi_bridge_processes.py \
  clouder/tests/test_csi_bucket_processes.py clouder/tests/test_csi_dataset_materializer.py \
  clouder/tests/test_csi_chart.py clouder/tests/test_node_mounts_cli.py \
  clouder/tests/test_csi_node_mount_gateway.py clouder/tests/test_csi_node_mount_gateway_agent.py \
  clouder/tests/test_csi_linux.py clouder/tests/test_csi_kernel.py \
  clouder/tests/test_csi_git_materializer.py clouder/tests/test_csi_main_wiring.py -q

Where to go next

• Node Mount Gateway — what it can mount and how, the grant annotation, and the credential rules.
• Shared File System — the claim the gateway binds home folders from.
• Local CSI driver — retired: the record of the CSI path the gateway replaced.