☰ 🥷 Datalayer Operator

KubernetesService

Datalayer Operator ensures that the requested services and resources are available. Datalayer Operator is a Kubernetes operator that constantly monitors the system and takes action so that the desired state is met on the Kubernetes cluster. It is responsible for managing the state of the Datalayer services.

• Manage the defined 🛣️ Routes to secure and direct the incoming request to the correct service or pod.
• Manage the defined 🐋 Docker Images building and publication in the Docker registry.
• Manage the defined 💼 Runtime Contents, 🌳 Runtime Environments and 🎱 Runtime Pools.

Please note that the Jupyter Service is responsible to deliver the ⚪ Runtimes to the user, so it is not the duty of the Datalayer Operator to serve Runtimes.

A subset of the Operator methods is exposed via REST endpoints for internal consumption (not visible from external clients).

Deploy Datalayer Operator

Plane

plane up datalayer-operator

Helm

export RELEASE=datalayer-operator
export NAMESPACE=datalayer-runtimes
helm upgrade \
  --install $RELEASE \
  oci://${DATALAYER_HELM_REGISTRY_HOST}/datalayer-charts/operator \
  --create-namespace \
  --namespace $NAMESPACE \
  --set operator.image="${DATALAYER_DOCKER_REGISTRY}/operator:0.1.1" \
  --set operator.certificateIssuer="letsencrypt" \
  --set operator.crds="true" \
  --set operator.cloudCredentials.enabled="true" \
  --set operator.cloudCredentials.create="true" \
  --set operator.cloudCredentials.secretName="${RELEASE}-cloud-credentials" \
  --set operator.cloudCredentials.aws.accessKeyId="${AWS_ACCESS_KEY_ID}" \
  --set operator.cloudCredentials.aws.secretAccessKey="${AWS_SECRET_ACCESS_KEY}" \
  --set operator.cloudCredentials.aws.defaultRegion="${AWS_DEFAULT_REGION}" \
  --set operator.cloudCredentials.azure.tenantId="${AZURE_TENANT_ID}" \
  --set operator.cloudCredentials.azure.clientId="${AZURE_CLIENT_ID}" \
  --set operator.cloudCredentials.azure.clientSecret="${AZURE_CLIENT_SECRET}" \
  --set operator.cloudCredentials.azure.subscriptionId="${AZURE_SUBSCRIPTION_ID}" \
  --set operator.cloudCredentials.azure.resourceGroup="${AZURE_RESOURCE_GROUP}" \
  --set operator.env.DATALAYER_DOCKER_REGISTRY="${DATALAYER_DOCKER_REGISTRY}" \
  --set operator.env.DATALAYER_RUN_HOST="${DATALAYER_RUN_HOST}" \
  --set operator.env.DATALAYER_OPERATOR_API_KEY="${DATALAYER_OPERATOR_API_KEY}" \
  --set operator.env.DATALAYER_IAM_HOST="${DATALAYER_IAM_HOST}" \
  --set operator.env.DATALAYER_IAM_API_KEY="${DATALAYER_IAM_API_KEY}" \
  --set operator.env.DATALAYER_AUTHZ_ENGINE="${DATALAYER_AUTHZ_ENGINE}" \
  --set operator.env.DATALAYER_CERT_ISSUER="${DATALAYER_CERT_ISSUER}" \
  --set operator.env.DATALAYER_INGRESS_CLASS_NAME="${DATALAYER_INGRESS_CLASS_NAME}" \
  --timeout 5m

Terraform

cd terraform
terraform init
terraform apply
./generated/clouder-Kubeadm-setup.sh
export KUBECONFIG=~/.clouder/kubeadm/<cluster-name>/kubeconfig
./generated/services/deploy-datalayer-operator.sh

Cloud Credentials (AWS/Azure)

For Operator scaling/executor workflows, credentials are consumed through a Kubernetes Secret referenced by the Operator Deployment.

• plane up datalayer-operator configures Helm values to create and mount ${RELEASE}-cloud-credentials.
• plane down datalayer-operator removes the release and the generated ${RELEASE}-cloud-credentials Secret.
• If you manage credentials outside Plane, keep operator.cloudCredentials.enabled=true, set operator.cloudCredentials.create=false, and point operator.cloudCredentials.secretName to your existing Secret.

Recommended Azure Flow (Clouder Kubeadm)

For clusters created with clouder kubeadm, the recommended flow is to generate a cluster-scoped Helm values JSON and let Plane use it automatically.

1. Ensure Azure is configured in Clouder.

clouder azure configure

2. Generate operator cloud credentials JSON for your cluster.

clouder azure helm-values --cluster my-cluster

By default this creates:

~/.clouder/kubeadm/r1/datalayer-operator-azure.json

3. Deploy the operator.

plane up datalayer-operator

plane/datalayer_plane/sbin/up.sh now checks the current kube context and, when present, automatically applies:

~/.clouder/kubeadm/<current-context>/datalayer-operator-azure.json

4. Optional: override the file path explicitly.

export DATALAYER_OPERATOR_AZURE_VALUES_FILE=/path/to/datalayer-operator-azure.json
plane up datalayer-operator

5. Optional: deploy with Helm directly.

helm upgrade \
  --install datalayer-operator \
  oci://${DATALAYER_HELM_REGISTRY_HOST}/datalayer-charts/operator \
  --namespace datalayer-runtimes \
  --create-namespace \
  --values ~/.clouder/kubeadm/r1/datalayer-operator-azure.json

The generated JSON includes:

• operator.cloudCredentials.azure.tenantId
• operator.cloudCredentials.azure.clientId
• operator.cloudCredentials.azure.clientSecret
• operator.cloudCredentials.azure.subscriptionId
• operator.cloudCredentials.azure.resourceGroup

Plane

plane ls

Helm

helm ls -A

Check the availability of the Datalayer CRDs.

kubectl get crd runtime-contents.datalayer.io
kubectl explain runtime-contents.spec

kubectl get crd runtime-environments.datalayer.io
kubectl explain runtime-environments.spec

kubectl get crd runtime-pools.datalayer.io
kubectl explain runtime-pools.spec

Check the availability of the Datalayer Operator Pods.

kubectl get pods -n datalayer-runtimes -l app=operator

Check the logs of the Datalayer Operator Pods.

kubectl logs -n datalayer-runtimes -l app=operator -f

Tear Down Datalayer Operator

If needed, tear down.

Plane

plane down datalayer-operator

Helm

export RELEASE=datalayer-operator
export NAMESPACE=datalayer-runtimes
helm delete $RELEASE --namespace $NAMESPACE