Skip to main content

☰ 🛜 Datalayer Datashim

Deprecated
Retired

🚨 RETIRED — Datashim no longer provisions anything. It served exactly one thing here — the RuntimeContent specs of type: s3, which it turned into a Dataset, then a PVC, then a mount on a Runtime Pod created with it. That is now the Node Mount Gateway's job: it binds the bucket into a Pod that is already running with Mountpoint for S3, reading the content's own secret at pod time and holding an STS session that expires, instead of writing a long-lived S3 key into a Dataset spec.

The change is in the Operator: content_operator.py no longer creates a Dataset for an s3 content (it only caches and validates the RuntimeContent), and the spawn path renders no csi-s3 PVC. The runtime-contents.datalayer.io CRD stays applicable — only its Datashim delivery is gone.

This page is kept for the record and for removing Datashim from a cluster that still runs it — see TearDown. The comparison, and what retiring it bought, is on the gateway's page.

Deploy​

Deprecated

The Operator no longer provisions through Datashim (audit 83), so no runtime depends on it any more. The install steps below still describe the chart as the plane tooling ships it today — removing Datashim from that tooling is the remaining cleanup. To take it out of a cluster that still runs it, skip to TearDown.

Datashim needs to be deployed in the cloud to benefit from the Runtime Content features.

helm repo add datashim https://datashim-io.github.io/datashim
helm repo update

Install Datashim.

plane up datalayer-datashim

Check the Datashim Pods.

kubectl get pods -n datalayer-runtimes | grep -E 'dataset-operator|csi-s3|csi-nfs|datashim'
# csi-s3-xxxxx                        2/2     Running   0          30s
# csi-s3-yyyyy                        2/2     Running   0          30s
# dataset-operator-zzzzz              1/1     Running   0          20s

# release information
helm list -A | grep datalayer-datashim
# datalayer-datashim   datalayer-runtimes   ...   deployed

# webhook resources
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep -i datashim
note

plane up datalayer-datashim deploys the release in the datalayer-runtimes namespace.

If you run kubectl get pods -n datalayer-datashim, you may see no resources and incorrectly conclude Datashim is down.

S3 Secret​

Create the secret for S3 access so it can be reused in Runtime Environments.

kubectl create secret generic \
  s3-secret \
  --from-literal=access_key_id=$AWS_ACCESS_KEY_ID \
  --from-literal=secret_access_key=$AWS_SECRET_ACCESS_KEY \
  --from-literal=region=$AWS_DEFAULT_REGION \
  -n datalayer-runtimes
kubectl describe secret s3-secret -n datalayer-runtimes

Namespace Label​

Datashim uses a mutating webhook with a namespaceSelector that requires the label monitor-pods-datasets=enabled on any namespace where pods should receive automatic dataset volume mounts.

The plane up datalayer-datashim command automatically labels the datalayer-runtimes namespace. For other namespaces (e.g. default for testing), add the label manually:

kubectl label namespace default monitor-pods-datasets=enabled
caution

Without this label, the datashim webhook will not inject volume mounts into pods, even if the pods have the correct dataset.0.id labels and the Dataset/PVC exist.

Validation​

Validate the configuration with the creation of an example Dataset.

Create a Dataset​

cat <<EOF | kubectl apply -f -
apiVersion: datashim.io/v1alpha1
kind: Dataset
metadata:
  name: example-dataset
spec:
  local:
    type: COS
    accessKeyID: $AWS_ACCESS_KEY_ID
    secretAccessKey: $AWS_SECRET_ACCESS_KEY
    endpoint: https://s3.$AWS_DEFAULT_REGION.amazonaws.com
    bucket: datalayer-dev
    region: $AWS_DEFAULT_REGION
EOF

Verify the Dataset and PVC are created:

kubectl describe dataset example-dataset
kubectl get pvc example-dataset
# NAME              STATUS   VOLUME                                     CAPACITY        ACCESS MODES   STORAGECLASS
# example-dataset   Bound    pvc-c26adf05-...                           9765625000Ki    RWX            csi-s3

Label the namespace​

kubectl label namespace default monitor-pods-datasets=enabled

Mount the Dataset in a Pod​

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    dataset.0.id: example-dataset
    dataset.0.useas: mount
spec:
  containers:
    - name: nginx
      image: nginx
EOF

Verify the mount is injected and accessible:

kubectl get pod nginx
kubectl exec nginx -it -- ls /mnt/datasets/example-dataset

Clean up​

kubectl delete pod nginx
kubectl delete dataset example-dataset

TearDown​

If needed, tear down.

plane down datalayer-datashim