☰ 🗂️ Datalayer Shared File System

Datalayer user data is stored on a Shared File System. The File System Storage can be implemented as a:

• AWS EFS mounted through the EFS CSI driver.
• Azure Files NFS mounted through the Azure CSI driver.
• Ceph Filesystem deployed and managed by Rook.

The Shared File System and the Node Mount Gateway

The two are often deployed together and are easy to mix up. They do different jobs, and only one depends on the other.

• The Shared File System is where the bytes are. It is one directory of one filesystem, reached through an RWX claim named by DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME in each namespace that needs it (One directory, a claim per namespace): every Home Folder (a person's, a team's, an organization's), the administrators' datasets, public/, datalayer/, ai-models/ and tmp/. It decides the layout and the ownership, 1000:100 with 0775 folders, and plane up datalayer-shared-filesystem creates the top-level directories.
• The Node Mount Gateway is one way those bytes reach a sandbox. It is an agent in the datalayer-node-mounts DaemonSet. It mounts this claim once per node, then binds a sub-path of it into a runtime Pod that is already running: a Home Folder for a files grant, or an administrator's dataset for a shared-folder grant. Because it acts after the Pod starts, a launch that mounts folders can take a prewarmed Pod instead of a cold one.

	Shared File System	Node Mount Gateway
What it is	Storage: a PVC on EFS, Azure Files NFS or CephFS	A node service that mounts things into running Pods
Needs the other	No	Yes, for Home Folders and datasets: nodeMountGateway.sharedFilesystemClaim names this claim, and the install fails without it
Default	Deployed when DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME is set; plane up skips it otherwise	Off (nodeMountGateway.enabled: false)
Creates folders	The top-level directories, at provisioning	A Home Folder on its first mount, never a dataset folder
Also mounts	Nothing else	Sources that are not on this claim: NFS exports, git checkouts, object-store buckets, local bridges

With the gateway off, the Operator mounts the same claim into each runtime Pod with one subPath per Home Folder, fixed when the Pod is created. Turning the gateway on or off migrates nothing. Both paths reach the same bytes in the same layout (Two ways in). Contents serves its Home Folder browser from the same directory, so a file it writes is the file a sandbox reads.

Deploy order: this claim first; then datalayer-node-mounts with the gateway and DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME set to this claim's name; then the Operator with DATALAYER_NODE_MOUNT_GATEWAY_ENABLED=true. The gateway page explains why the agent goes before the Operator: Turning it on, and proving it works.

The boundary is the same on both paths. The gateway resolves every source beneath this claim and refuses one that walks outside it. What a sandbox can reach is what was mounted into it (The mount set is the boundary).

Who mounts the claim

The claim is read and written by the components below, and they must agree on the layout or a path recorded by one is unresolvable in another. Every release up.sh passes the claim to (*.sharedFsPVC) mounts it too: ai-agents, ai-inference, mcp-servers, spacer, the Library and the Data Server, all in datalayer-api, through that namespace's claim on the same directory.

Component	How it mounts it
Runtime Pods (Operator)	Whole claim, in the companion container, at DATALAYER_SHARED_FS_MOUNT_PATH.
datalayer-contents	The whole directory, through the datalayer-api claim, at the same path.
Node Mount Gateway	Whole claim, once per node, in the datalayer-node-mounts DaemonSet, at nodeMountGateway.sharedFilesystemMountPath.
plane provisioning	Creates the top-level directories and their ownership.

Each component adds one more detail beyond where it mounts:

• Runtime Pods (Operator): the runtime container gets its folders one of two ways — from the gateway when it is on, or, when it is off, one subPath per home folder, at /home/datalayer/<handle>.
• datalayer-contents: serves the Home Folder browser from the bytes a sandbox mounts, and writes a transfer's working copy into the same home/{users|organizations|teams}/{uid}/ folder.
• Node Mount Gateway: binds sub-paths of the claim into Pods that are already running — home folders from home/, and Environment datasets from wherever an administrator put them.

One directory, a claim per namespace

A PVC belongs to a namespace, so the runtimes side and the API side each hold one, both named $DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME. They are two claims on one directory of the filesystem, which is what makes a file Contents writes the file a sandbox reads.

On AWS the directory is /datalayer-shared/shared on the EFS filesystem, and each claim reaches it through an access point of its own rooted there. Two settings arrange that, and both matter:

• the aws-efs StorageClass pins every claim to that one directory — basePath: /datalayer-shared, subPathPattern: shared and ensureUniqueDirectory: "false". Without the last two, the driver appends the volume's own name and each namespace lands in a directory the other cannot see;
• DATALAYER_AWS_EFS_ACCESS_POINT_RUNTIMES and _API name the access points the claims bind to, and DATALAYER_AWS_EFS_FILE_SYSTEM_ID the filesystem they are on. A cluster whose CSI controller carries no AWS credentials of its own cannot create access points while provisioning, and these let the deploy bind ones that already exist. With them unset, the claims are provisioned dynamically and the StorageClass still puts them in one place.

To check that both claims really are one directory:

for ns in datalayer-runtimes datalayer-api; do
  pv=$(kubectl get pvc -n $ns "$DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME" -o jsonpath='{.spec.volumeName}')
  echo "$ns $(kubectl get pv "$pv" -o jsonpath='{.spec.csi.volumeHandle}')"
done
kubectl exec -n datalayer-api deploy/datalayer-contents -c api -- ls /mnt/shared-fs
kubectl exec -n datalayer-runtimes ds/datalayer-node-mounts -c driver -- ls /mnt/shared-fs

The access points differ, one per namespace; both list the same entries, and a file written through one is there through the other.

The layout

/mnt/shared-fs/
  home/
    users/<user uid>/
    organizations/<organization uid>/
    teams/<team uid>/
  public/  datalayer/  datasets/  ai-models/  tmp/

ai-models/ is root:root, 0555. The others are 1000:100, created 775 and then chmod g+ws, so what is created beneath them keeps the group. Contents also writes objects/, staging/ and published/ at the root of the claim it mounts.

The names under /home/datalayer are the account handles, not the uids, and they are produced in one place — datalayer_common.home_folders — because the Operator, Contents and the gateway all have to call one folder by one name. A team is named after its organization and itself joined by a double underscore (datalayer__research).

Two ways in, and why the claim only sees one of them

A subPath volume mount and a gateway bind are the same bytes reached differently, and the difference is when. A subPath is part of the Pod's spec, so it is fixed when the Pod is created — which is why a pooled Pod, built before anyone knew whose folder it would carry, cannot have one. The gateway binds after the Pod is running, which is what makes a warm start possible.

Nothing about the claim changes between the two. The layout is the same, the ownership is the same, and a path written by one is read by the other — which is the property that lets a deployment turn the gateway on, and off again, without migrating anything. plane provisioning creates the same directories either way.

The subPath rendering is kept as the gateway-off path, and it is still what a deployment gets by default. Deleting it belongs with making the gateway mandatory, which has not happened.

The mount set is the boundary

Every runtime runs as 1000:100 and the folders are created 0775, so the shared filesystem gives every sandbox the same POSIX identity. What a sandbox cannot reach is what was not mounted into it — the mount set is the access decision, not a file permission. That is why the set is resolved in Runtimes against IAM with the caller's own credentials, and why a client naming a folder in a request is never a way to mount it. It holds for a gateway bind exactly as it holds for a subPath mount: the gateway refuses a source that does not resolve beneath this claim, and it never takes a path from anywhere but the Pod annotation the Operator wrote.

AWS EFS

Define the storage provider as aws.

DATALAYER_STORAGE_PROVIDER=aws

Prerequisites

The backing EFS infrastructure is provisioned by platform automation before plane up datalayer-shared-filesystem runs. DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME must be set, since it names the claim in each namespace, and DATALAYER_STORAGE_PROVIDER must be aws (it defaults to ceph).

You need:

• The AWS EFS CSI driver installed in the cluster.
• The aws-efs StorageClass, which plane up lays down from the chart and recreates when its parameters drift — a StorageClass parameter cannot be edited in place.
• DATALAYER_AWS_EFS_FILE_SYSTEM_ID, and DATALAYER_AWS_EFS_ACCESS_POINT_RUNTIMES / _API where the access points already exist. With the filesystem id set, the deploy makes no AWS call; without it, up.sh provisions the filesystem and its mount target, which needs a kubeadm cluster whose metadata carries a networking block and AWS credentials in the shell.

The CSI controller needs AWS credentials only to create an access point while provisioning a claim. Binding claims to access points named in the environment asks nothing of it, which is what a cluster whose nodes carry no EFS instance profile needs.

Kubeadm (Clouder)

On Kubeadm clusters created with Clouder, clouder kubeadm setup now bakes all AWS EFS prerequisites:

• Ensures node IAM profile usage is aligned across kubeadm nodes.
• Ensures required managed policies are attached to the node role:
  • AmazonEBSCSIDriverPolicy
  • AmazonEFSCSIDriverPolicy
• Ensures IMDSv2 hop limit is set to 2 on kubeadm nodes.
• Installs AWS EFS CSI driver and creates the aws-efs StorageClass.

Validate the resulting objects:

# Verify the StorageClass exists
kubectl get storageclass aws-efs

# Verify EFS CSI driver objects
kubectl -n kube-system get deployment efs-csi-controller
kubectl -n kube-system get daemonset efs-csi-node

# Verify node IMDS settings and IAM profile on EC2 (run from an AWS-authenticated shell)
aws ec2 describe-instances \
  --instance-ids <instance-id> \
  --query 'Reservations[].Instances[].{Hop:MetadataOptions.HttpPutResponseHopLimit,Profile:IamInstanceProfile.Arn}' \
  --output table

EKS / Terraform

Use your Terraform/cluster automation workflow to provision EFS and aws-efs before deploying datalayer-shared-filesystem.

What the CSI controller needs credentials for

The driver asks AWS for one thing: creating an access point, while it provisions a claim. Mounting one needs no credential — that is an NFS mount through the filesystem's mount targets — so a cluster whose claims are already bound keeps working whatever the controller can or cannot authenticate as.

efs-csi-controller takes its credentials from the instance profile of the node it runs on, read over IMDS. Two things have to be true of that node, and clouder kubeadm setup arranges both: the instance profile is attached and its role carries AmazonEFSCSIDriverPolicy, and the IMDSv2 hop limit is 2, because a pod's request to the metadata service crosses one hop more than the host's.

Where neither is true, the claim never binds and the event says why:

ProvisioningFailed  Failed to create Access point in File System fs-… :
  get identity: get credentials: failed to refresh cached credentials,
  no EC2 IMDS role found

Which nodes can answer for the driver, and where it runs:

aws ec2 describe-instances --region "$AWS_REGION" \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].[InstanceId,MetadataOptions.HttpPutResponseHopLimit,IamInstanceProfile.Arn,Tags[?Key==`Name`]|[0].Value]' \
  --output text
kubectl -n kube-system get pods -l app=efs-csi-controller -o wide

A node with None for its profile, or a hop limit of 1, cannot give the controller an identity. A cluster grown after setup, or one whose workers were replaced, ends up there. r1 is in that state: only the master carries r1-efs-csi-profile with hop limit 2, and the controller runs on workers that carry neither.

What such a cluster does instead. Its claims bind to access points made out of band and named in the plane's rc (DATALAYER_AWS_EFS_ACCESS_POINT_RUNTIMES and _API), which asks nothing of the controller. One per namespace, both rooted at the same directory:

aws efs create-access-point --region "$AWS_REGION" \
  --file-system-id "$DATALAYER_AWS_EFS_FILE_SYSTEM_ID" \
  --client-token datalayer-shared-runtimes \
  --posix-user "Uid=0,Gid=0" \
  --root-directory 'Path=/datalayer-shared/shared,CreationInfo={OwnerUid=0,OwnerGid=0,Permissions=700}'

Keep the ids in the rc. A cluster rebuilt without them provisions a claim per namespace again, and Contents stops seeing what the sandboxes see.

To have the controller provision on its own again, give it a node with the identity. Either attach the instance profile to the workers and let a pod reach IMDS, which is the state clouder kubeadm setup leaves behind:

aws ec2 associate-iam-instance-profile --region "$AWS_REGION" \
  --instance-id <worker id> --iam-instance-profile Name=<cluster>-efs-csi-profile
aws ec2 modify-instance-metadata-options --region "$AWS_REGION" \
  --instance-id <worker id> --http-put-response-hop-limit 2 --http-endpoint enabled

or schedule it where the profile already is, on the control plane:

kubectl -n kube-system patch deploy efs-csi-controller --type=strategic -p '{"spec":{"template":{"spec":{
  "nodeSelector":{"node-role.kubernetes.io/control-plane":""},
  "tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"},
                 {"key":"efs.csi.aws.com/agent-not-ready","operator":"Exists"},
                 {"key":"node-role.kubernetes.io/control-plane","operator":"Exists","effect":"NoSchedule"}]}}}}'

An access point's POSIX user is 0:0, so every operation through it acts as root whatever a pod runs as. That is what lets Contents, which runs as nobody, write a home folder owned by 1000:100.

Deploy the shared filesystem with the following command.

plane up datalayer-shared-filesystem

You can check the AWS EFS-backed shared filesystem with the following commands.

kubectl get pvc -A --field-selector metadata.name=$DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME
# NAMESPACE            NAME                          STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
# datalayer-api        datalayer-shared-filesystem   Bound    pvc-…    1Ti        RWX            aws-efs
# datalayer-runtimes   datalayer-shared-filesystem   Bound    pvc-…    1Ti        RWX            aws-efs
kubectl get storageclass aws-efs

A shared-fs-prepare Pod is launched to create the initial folders in the shared filesystem. The volume is mounted on /mnt/shared-fs. It sleeps 1000 s after its script and then Completes, so the kubectl exec lines below work only in its first 17 minutes. To run it again: kubectl delete pod shared-fs-prepare -n datalayer-runtimes && plane up datalayer-shared-filesystem.

kubectl logs shared-fs-prepare -n datalayer-runtimes
# + mkdir -pm 775 /mnt/shared-fs/home
# + chmod g+ws /mnt/shared-fs/home
# + chown 1000:100 /mnt/shared-fs/home
# … the same for public, datalayer, datasets, ai-models and tmp
# + chown -R root:root /mnt/shared-fs/ai-models
# + chmod -R 555 /mnt/shared-fs/ai-models
kubectl exec shared-fs-prepare -n datalayer-runtimes -it -- mount | grep /mnt/shared-fs
# fs-xxxxxx.efs.<region>.amazonaws.com:/ on /mnt/shared-fs type nfs4 (...)
kubectl exec shared-fs-prepare -n datalayer-runtimes -it -- ls /mnt/shared-fs
kubectl exec shared-fs-prepare -n datalayer-runtimes -it -- sh
kubectl delete pod shared-fs-prepare -n datalayer-runtimes

On AWS/EFS, chown runs in shared-fs-prepare as on other providers. This relies on EFS access points being provisioned with root identity mapping (uid=0, gid=0) by platform automation.

Validate ownership behavior:

kubectl logs shared-fs-prepare -n datalayer-runtimes --tail=120
# Look for chown lines for each directory and ai-models

kubectl exec shared-fs-prepare -n datalayer-runtimes -- \
  stat -c "%u:%g %a %n" /mnt/shared-fs/ai-models /mnt/shared-fs/home
# 0:0 555 /mnt/shared-fs/ai-models
# 1000:100 2775 /mnt/shared-fs/home

For a full from-scratch AWS validation, tear down and reprovision everything:

# 1) Remove the Helm release resources
plane down datalayer-shared-filesystem

# 2) Delete old PVC/PV and StorageClass objects
kubectl delete pvc datalayer-shared-filesystem -n datalayer-runtimes --ignore-not-found
kubectl delete pvc datalayer-shared-filesystem -n datalayer-api --ignore-not-found
for PV in $(kubectl get pv -o jsonpath='{range .items[?(@.spec.claimRef.name=="datalayer-shared-filesystem")]}{.metadata.name}{"\n"}{end}'); do
  kubectl patch pv "$PV" --type=merge -p '{"metadata":{"finalizers":[]}}' >/dev/null 2>&1 || true
  kubectl delete pv "$PV" --ignore-not-found
done
kubectl delete storageclass aws-efs --ignore-not-found

# 3) Delete old EFS file system (delete access points and mount targets first)
EFS_ID=<old-fs-id>
for AP in $(aws efs describe-access-points --file-system-id "$EFS_ID" --query 'AccessPoints[].AccessPointId' --output text); do
  aws efs delete-access-point --access-point-id "$AP"
done
for MT in $(aws efs describe-mount-targets --file-system-id "$EFS_ID" --query 'MountTargets[].MountTargetId' --output text); do
  aws efs delete-mount-target --mount-target-id "$MT"
done
aws efs delete-file-system --file-system-id "$EFS_ID"
aws efs wait file-system-deleted --file-system-id "$EFS_ID"

# 4) Reprovision from plane automation
plane up datalayer-shared-filesystem
kubectl wait --for=condition=Bound pvc/datalayer-shared-filesystem -n datalayer-runtimes --timeout=240s
kubectl wait --for=condition=Bound pvc/datalayer-shared-filesystem -n datalayer-api --timeout=240s
kubectl delete pod shared-fs-prepare -n datalayer-runtimes --force --grace-period=0 --ignore-not-found
plane up datalayer-shared-filesystem
kubectl wait --for=condition=Ready pod/shared-fs-prepare -n datalayer-runtimes --timeout=240s

# 5) Verify StorageClass identity mapping and prepare logs
kubectl get storageclass aws-efs -o yaml | grep -E 'uid:|gid:|fileSystemId|provisioningMode'
kubectl logs shared-fs-prepare -n datalayer-runtimes --tail=120

Tear down when you don't need the Shared Filesystem anymore.

danger

plane down deletes both PVCs, and their PVs have reclaimPolicy: Delete: the access points and the users' data under them go. The next plane up creates new, empty directories.

plane down datalayer-shared-filesystem

Azure Files NFS

Define the storage provider as azure.

DATALAYER_STORAGE_PROVIDER=azure

Prerequisites

The azure-nfs StorageClass must exist before running plane up datalayer-shared-filesystem. The helm chart creates the PVCs but does not create the StorageClass for Azure — it is expected to be provisioned at the infrastructure level.

Kubeadm (Clouder)

On Kubeadm clusters created with Clouder, clouder kubeadm setup already creates the azure-nfs StorageClass with the correct Azure subscription, resource group, and location baked in. No additional steps are needed.

# Verify the StorageClass exists
kubectl get storageclass azure-nfs

AKS / Manual

On AKS or other clusters, create the StorageClass manually. On AKS, the subscriptionID, resourceGroup, and location parameters can be omitted (auto-detected from IMDS).

cat <<'EOF' | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-nfs
provisioner: file.csi.azure.com
allowVolumeExpansion: true
parameters:
  shareName: datalayer-shared-filesystem
  skuName: Premium_LRS
  protocol: nfs
  # Required on non-AKS clusters (Kubeadm, etc.):
  # subscriptionID: "<subscription-id>"
  # resourceGroup: "<resource-group>"
  # location: "<region>"
EOF

Deploy

Plane

plane up datalayer-shared-filesystem

Verify:

kubectl get pvc -A --field-selector metadata.name=$DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME
kubectl get storageclass azure-nfs
kubectl logs shared-fs-prepare -n datalayer-runtimes

Tear down when you don't need the Shared Filesystem anymore.

danger

plane down deletes both PVCs. Whether the share's data survives depends on the StorageClass's reclaimPolicy; with Delete, it does not.

plane down datalayer-shared-filesystem

Ceph

Deploy

Define the storage provider as ceph.

DATALAYER_STORAGE_PROVIDER=ceph

The installation is a 3 steps process.

1. Install the Rook Operator for Ceph.

plane up datalayer-ceph-operator

This installs the Rook Operator pod. Check the availabilty of the Rook Operator with the following command.

kubectl get pod -n datalayer-storage -l app=rook-ceph-operator

2. Install a Ceph Cluster.

plane up datalayer-ceph-cluster

This installs the Ceph Cluster (you will still need to create the effective Ceph Filesystem on the Cluster). You can check the Ceph Cluster with the following commands.

kubectl get cephcluster $DATALAYER_RUN_HOST -n datalayer-storage -w
# NAME                DATADIRHOSTPATH   MONCOUNT   AGE     PHASE         MESSAGE                 HEALTH   EXTERNAL   FSID
# oss.datalayer.run   /var/lib/rook     3          8s    Progressing   Detecting Ceph version
# oss.datalayer.run   /var/lib/rook     3          3m59s   Progressing   Configuring the Ceph Cluster                          f54e114c-6947-4811-b8b3-3b4240c4931c
# oss.datalayer.run   /var/lib/rook     3          3m59s   Progressing   Configuring Ceph Mons                                 f54e114c-6947-4811-b8b3-3b4240c4931c
# oss.datalayer.run   /var/lib/rook     3          4m24s   Progressing   Configuring Ceph Mgr(s)                               f54e114c-6947-4811-b8b3-3b4240c4931c
# oss.datalayer.run   /var/lib/rook     3          4m25s   Progressing   Configuring Ceph OSDs                                 f54e114c-6947-4811-b8b3-3b4240c4931c
# oss.datalayer.run   /var/lib/rook     3          4m46s   Progressing   Processing OSD 0 on node "aks-ce..."                  f54e114c-6947-4811-b8b3-3b4240c4931c
# oss.datalayer.run   /var/lib/rook     3          4m57s   Ready         Cluster created successfully            HEALTH_OK              f54e114c-6947-4811-b8b3-3b4240c4931c
# 
kubectl describe cephcluster $DATALAYER_RUN_HOST -n datalayer-storage 

The cluster is now equipped to provision Filesystems (see following step).

warning

Be sure to wait for the cluster to be ready and healthy. This steps has proven to be quite highly dependent on your nodes sizing especially to get OSDs up and running.

We recommend 5 nodes with 4 CPU and 16 GB each.

You can see the cluster status in the Grafana dashboard Ceph Cluster entry.

You can also use the native Ceph dashboard. Run a port-forward proxy, get the password, and login with the admin user on http://localhost:7000.

plane pf-ceph

3. Create the Shared Filesystem PVCs for the user data, in datalayer-runtimes and datalayer-api, with the following command.

plane up datalayer-shared-filesystem

You can check the Ceph Filesystem with the following commands.

kubectl get cephfilesystem shared-filesystem -n datalayer-storage
kubectl get cephfilesystemsubvolumegroups shared-filesystem-csi -n datalayer-storage

A shared-fs-prepare Pod has been launched and is responsible to create the storage initial folders. The shared filesystem is mounted on /mnt/shared-fs.

kubectl logs shared-fs-prepare -n datalayer-runtimes
# + mkdir -pm 775 /mnt/shared-fs/home
# + chmod g+ws /mnt/shared-fs/home
# + chown 1000:100 /mnt/shared-fs/home
# … the same for public, datalayer, datasets, ai-models and tmp
# + chown -R root:root /mnt/shared-fs/ai-models
# + chmod -R 555 /mnt/shared-fs/ai-models
kubectl exec shared-fs-prepare -n datalayer-runtimes -it -- mount | grep /mnt/shared-fs
# 10.0.110.3:6789,10.0.202.233:6789,10.0.44.146:6789:/volumes/csi/csi-vol-a66fdf1a-7046-4bed-8d21-846d9b84f4df/23dde9c7-67f3-40a1-8329-49d93cd6a8de on /mnt/shared-fs type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=shared-filesystem)
kubectl exec shared-fs-prepare -n datalayer-runtimes -it -- sh
kubectl delete pod shared-fs-prepare -n datalayer-runtimes

What is deployed?

The first piece is the Rook Ceph Operator. Then we create through a Custom Resource a Ceph Cluster using PVC as storage backend; see cephClusterSpec.storage.storageClassDeviceSets. This is important as it defines the real amount of available storage (that will contain all data replication).

As part of that spec, the second important point is the provisioning of a Ceph FileSystem (see cephFileSystems). In particular, it defines the number of data and metadata replication as well as the metadata server.

The final part creates a Ceph shared storage through PVC on top of the Ceph FileSystem to be mounted in the Remote Runtime nodes.

Configuration

Ceph Operator Configuration

The Ceph Operator configuration is defined in plane/etc/helm-private/charts/datalayer-ceph-operator/values.yaml. You can find more information about the available values in the rook documentation.

Ceph Cluster Configuration

The Ceph Cluster configuration is defined in plane/etc/helm-private/charts/datalayer-ceph-cluster/values.yaml. You can find more information about the available values in the rook documentation.

Of particular importance are the configuration for the OSDs cephClusterSpec.storage.storageClassDeviceSets (the real storage) and the provisioned Ceph storage cephFileSystems (e.g. a Filesystem).

note

The deployment examples can provide direction to tune a configuration depending on the scenario.

Ceph Storage Configuration

The users ceph storage is defined on top of the provisioned Filesystem as a PVC through a custom storageClassName (itself defined in the ceph cluster configuration).

That chart also defines an pod to set up the default content of the volume.

Teardown

Make sure to check all resources are deleted before moving to the next steps. If you don't it is highly probable something will go wrong and you will have to delete resource manually (including editing specs to remove finalizers).

We advice you to read the teardown documentation before running the following commands.

1. Delete Shared Filesystem

danger

You will loose the complete users data.

plane down datalayer-shared-filesystem

You may need to manually remove the following objects in case of up/down not working (edit them to remove the finalizers).

# Remove the finalizers.
for ns in datalayer-runtimes datalayer-api; do
  kubectl edit pvc $DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME -n $ns
  kubectl delete pvc $DATALAYER_SHARED_FS_VOLUME_CLAIM_NAME -n $ns
done
# Remove the finalizers.
kubectl edit cephfilesystemsubvolumegroups shared-filesystem-csi -n datalayer-storage
kubectl delete cephfilesystemsubvolumegroups shared-filesystem-csi -n datalayer-storage
# Remove the finalizers.
kubectl edit cephfilesystem shared-filesystem -n datalayer-storage
kubectl delete cephfilesystem shared-filesystem -n datalayer-storage

2. Delete Ceph Cluster

plane down datalayer-ceph-cluster

note

By default the Filesystem is not deleted; see cephFileSystems[0].preserveFilesystemOnDelete.

To delete the Filesystem, you need to remove the finalizer.

kubectl edit cephcluster $DATALAYER_RUN_HOST -n datalayer-storage
kubectl delete cephcluster $DATALAYER_RUN_HOST -n datalayer-storage

3. Delete Ceph Operator

plane down datalayer-ceph-operator

Tips and Tricks for Ceph

Monitoring

The default values are configuring the gathering of metrics by the Prometheus instance handles in datalayer-observer.

Grafana dashboards for Ceph are also provisioned in datalayer-observer and should be populated out-of-the-box with the Ceph metrics.

Handling Ceph OSDs

Refer to the Rook documentation to remove or add OSDs.

Scaling Ceph Global Storage

The Rook documentation describes how to scale up OSDs vertically and horizontally.

tip

On the Web, the recommendation is to prefer growing horizontally rather than vertically.