☰ 🔎 Datalayer Solr

Apache Solr Operator

To deploy Solr, you first need to deploy the Solr Operator.

helm repo add apache-solr https://solr.apache.org/charts
helm repo update

Plane

plane up datalayer-solr-operator

Helm

cat << 'EOF' > /tmp/values.yaml
nodeSelector:
  role.datalayer.io/system: "true"
zookeeper-operator:
  nodeSelector:
    role.datalayer.io/system: "true"
EOF
export RELEASE=datalayer-solr-operator
export NAMESPACE=datalayer-solr-operator
kubectl create \
  -n $NAMESPACE \
  -f https://solr.apache.org/operator/downloads/crds/v0.9.1/all-with-dependencies.yaml
helm upgrade \
  --install $RELEASE \
  apache-solr/solr-operator \
  --version 0.9.1 \
  --create-namespace \
  --namespace $NAMESPACE \
  --values /tmp/values.yaml \
  --timeout 5m

Terraform

cd terraform
terraform init
terraform apply
./generated/clouder-Kubeadm-setup.sh
export KUBECONFIG=~/.clouder/kubeadm/<cluster-name>/kubeconfig
./generated/services/deploy-datalayer-solr-operator.sh

Check the availability of the Solr CRDs.

kubectl explain solrcloud.spec.zookeeperRef.provided.config
kubectl explain solrcloud.spec.zookeeperRef.provided.persistence
kubectl explain solrcloud.spec.zookeeperRef.provided.persistence.spec

Plane

plane ls

Helm

helm ls -A

Check the availability of the Solr Operator Pods.

kubectl get pods -n datalayer-solr-operator -l control-plane=solr-operator

Datalayer Solr Cluster

Prepare a AWS S3 bucket for the Solr backups (the DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME environment variable should contain that name) and create a secret with write access to that S3 bucket.

kubectl create secret generic aws-creds \
  --from-literal=access-key-id=$AWS_ACCESS_KEY_ID \
  --from-literal=secret-access-key=$AWS_SECRET_ACCESS_KEY \
  --namespace=datalayer-solr
kubectl describe secret aws-creds -n datalayer-solr

Create a secret for the Solr authentication.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: solr-basic-auth
  namespace: datalayer-solr
type: kubernetes.io/basic-auth
stringData:
  username: ${DATALAYER_SOLR_USERNAME}
  password: ${DATALAYER_SOLR_PASSWORD}
EOF
kubectl describe secret solr-basic-auth -n datalayer-solr
# Secret for the datalayer-api namespace.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: solr-basic-auth
  namespace: datalayer-api
type: kubernetes.io/basic-auth
stringData:
  username: ${DATALAYER_SOLR_USERNAME}
  password: ${DATALAYER_SOLR_PASSWORD}
EOF
kubectl describe secret solr-basic-auth -n datalayer-api

You are now ready to create a Solr cluster. ensure the DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME and DATALAYER_SOLR_BACKUP_S3_BUCKET_REGION variables with the name and region of the S3 bucket for the backups.

You can also configure the size of the cluster, a 3 nodes replica cluster is what is defined in the following spec.

Version compatibility note Use Solr 9.10.0 with Solr Operator 0.9.1 in this setup. Solr 10.x currently fails to start with the operator-generated solr.xml due to the genericCoreNodeNames parameter in the <solrcloud> section, which causes startup probe failures and pod restart loops. If you previously deployed with 10.x, patch the SolrCloud back to 9.10.0 and wait for the StatefulSet rollout to complete.

Plane

kubectl apply -f $PLANE_HOME/etc/specs/solr/datalayer-solr.yaml

Bash

cat <<EOF | kubectl apply -f -
apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: solr-datalayer
  namespace: datalayer-solr
spec:
  dataStorage:
    persistent:
      reclaimPolicy: Retain
      pvcTemplate:
        spec:
          resources:
            requests:
              storage: "200Gi"
  replicas: 3
  solrImage:
    tag: 9.10.0
  solrJavaMem: "-Xms1g -Xmx5g"
  solrModules:
    - s3-repository
  additionalLibs:
    - "/opt/solr/contrib/s3-repository/lib"
  backupRepositories:
    - name: s3
      s3:
        bucket: ${DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME}
        region: ${DATALAYER_SOLR_BACKUP_S3_BUCKET_REGION}
        credentials:
          accessKeyIdSecret:
            name: aws-creds
            key: access-key-id
          secretAccessKeySecret:
            name: aws-creds
            key: secret-access-key
  customSolrKubeOptions:
    podOptions:
      nodeSelector:
        role.datalayer.io/solr: "true"
      envVars:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: aws-creds
            key: access-key-id
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: aws-creds
            key: secret-access-key
      - name: AWS_DEFAULT_REGION
        value: us-east-1
      resources:
        limits:
          memory: "3G"
        requests:
          cpu: "65m"
          memory: "1G"
  zookeeperRef:
    provided:
      zookeeperPodPolicy:
        nodeSelector:
          role.datalayer.io/solr: "true"
        resources:
          limits:
            memory: "1G"
          requests:
            cpu: "65m"
            memory: "156Mi"
      persistence:
        reclaimPolicy: Delete
        spec:
          resources:
            requests:
              storage: "5Gi"
      replicas: 3
  solrOpts: "-Dsolr.autoSoftCommit.maxTime=10000"
  solrGCTune: "-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8"
  solrSecurity:
    authenticationType: "Basic"
    basicAuthSecret: "solr-basic-auth"
    probesRequireAuth: false
EOF

Check the creation of the Solr Cluster Pods. It may take multiple minutes to get a completely up-and-running Solr cluster, mainly due to the time to provision the required storage.

kubectl get solrclouds -n datalayer-solr -w
# NAME             VERSION   TARGETVERSION   DESIREDNODES   NODES   READYNODES   UPTODATENODES   AGE
# solr-datalayer   9.10.0                    3              3       3            3               79s
kubectl get pods -n datalayer-solr -w
kubectl describe pods -n datalayer-solr | grep Node

Create Datalayer Solr Collections

Solr collections are organized into two sets that can be initialized independently.

Core Collections (17)

Platform collections used by the main Datalayer services:

Collection	Purpose
ai-agents	AI agents service records
api-keys	IAM authentication tokens
contacts	Contact information
credits	Usage credits
datasources	Data source definitions
events	Agent lifecycle and system event records
iam	Identity and access management
inbounds	Inbound integrations
invites	User invitations
library	Content library
notifications	User-facing notifications records
outbounds	Outbound integrations
secrets	Encrypted secrets metadata
spaces	Workspace spaces
success	Success tracking
tool-approvals	Tool approval requests and decision state
usage	Platform usage metrics

Note: The notifications collection stores user-facing notifications, while events stores lifecycle and system events.

Runtimes Collections (2)

Collections specific to the runtimes subsystem:

Collection	Purpose
runtime-checkpoints	Runtime checkpoint records (CRIU snapshots metadata)
sandbox-snapshots	Runtime snapshot records

Source of Truth

The collection arrays are defined in solr-collections.sh:

• SOLR_COLLECTIONS_CORE — Core platform collections
• SOLR_COLLECTIONS_RUNTIMES — Runtimes collections
• SOLR_COLLECTIONS — All collections combined (union of the above two)

The solr-init script prompts which collection set to create. This is useful when adding the runtimes collections to an existing cluster without re-creating the core ones.

Plane

plane solr-init
# Select:
#   1) Core       - Platform collections (17 collections)
#   2) Runtimes   - Runtimes collections (2 collections)
#   3) All        - All collections (19 collections)
#
# The init pod is created, logs are streamed, and the pod is auto-cleaned.

Bash

Access Datalayer Solr

To connect from you host to the Solr user interface, add the following entries in your /etc/hosts file.

# /etc/hosts
127.0.0.1      solr-datalayer-solrcloud-0.solr-datalayer-solrcloud-headless.datalayer-solr solr-datalayer-solrcloud-1.solr-datalayer-solrcloud-headless.datalayer-solr solr-datalayer-solrcloud-2.solr-datalayer-solrcloud-headless.datalayer-solr

127.0.0.1      solr-datalayer-solrcloud-zookeeper-0.solr-datalayer-solrcloud-zookeeper-headless.datalayer-solr.svc.cluster.local solr-datalayer-solrcloud-zookeeper-1.solr-datalayer-solrcloud-zookeeper-headless.datalayer-solr.svc.cluster.local solr-datalayer-solrcloud-zookeeper-2.solr-datalayer-solrcloud-zookeeper-headless.datalayer-solr.svc.cluster.local

Launch two kubectl port-forward to the Kubernetes cluster.

# open http://localhost:8983/solr
# open http://solr-datalayer-solrcloud-0.solr-datalayer-solrcloud-headless.datalayer-solr:8983/solr
kubectl port-forward -n datalayer-solr service/solr-datalayer-solrcloud-zookeeper-client 2181:2181 &
kubectl port-forward -n datalayer-solr service/solr-datalayer-solrcloud-headless 8983:8983 &
wait

You will need Java and Apache Solr available on your system as well as the configuration files.

export ZK_HOST=localhost:2181
export SOLR_HOME=/opt/solr
export SOLR_AUTH_TYPE="basic"
export SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:${DATALAYER_SOLR_PASSWORD}"
export PATH=$SOLR_HOME/bin:$PATH
export YELLOW='\x1b[33m'
export RESET='\x1b[0m'

for COLLECTION in ai-agents contacts credits datasources events iam api-keys inbounds invites library notifications outbounds secrets spaces success tool-approvals usage runtime-checkpoints sandbox-snapshots
do
    echo
    echo -e $YELLOW"Creating Solr collection $COLLECTION"$RESET
    echo
    $SOLR_HOME/bin/solr create -c $COLLECTION -shards 3 -replicationFactor 3 -d $PLANE_HOME/etc/dockerfiles/datalayer-solr/config -p 8983
done

Backup Datalayer Solr

Solr collections are backed up to an AWS S3 bucket using the Solr Operator SolrBackup CRD. The backup relies on the s3 repository configured in the SolrCloud spec (see Datalayer Solr Cluster above).

Three backup definitions are available, matching the collection sets:

File	Scope	CR Name
datalayer-backup-s3.yaml	All 19 collections	datalayer-solr-collection-backups
datalayer-backup-s3-core.yaml	17 core collections	datalayer-solr-collection-backups
datalayer-backup-s3-runtimes.yaml	2 runtimes collections	datalayer-solr-collection-backups

All backups run every 10 minutes and retain up to 200 snapshots.

Prerequisites

• The aws-creds secret must exist in the datalayer-solr namespace (see Datalayer Solr Cluster).
• The DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME and DATALAYER_SOLR_BACKUP_S3_BUCKET_REGION environment variables must be set.

Apply the Backup Schedule

Plane

# Prompt for backup set (core / runtimes / all).
plane solr-backups-apply
# Apply a specific backup spec directly.
plane solr-backups-apply core
plane solr-backups-apply runtimes
plane solr-backups-apply all

Bash

# Example: All collections.
cat <<EOF | kubectl apply -f -
apiVersion: solr.apache.org/v1beta1
kind: SolrBackup
metadata:
  name: datalayer-solr-collection-backups
  namespace: datalayer-solr
spec:
  repositoryName: s3
  solrCloud: solr-datalayer
  collections:
    - ai-agents
    - contacts
    - credits
    - datasources
    - events
    - iam
    - api-keys
    - inbounds
    - invites
    - library
    - notifications
    - runtime-checkpoints
    - sandbox-snapshots
    - outbounds
    - secrets
    - spaces
    - success
    - tool-approvals
    - usage
  recurrence:
    schedule: "*/10 * * * *" # every 10 minutes
    maxSaved: 200
EOF

Inspect Solr Collection and Shard Health

Use the Solr status command to inspect available collections and list shard and replica details directly from the Solr API (localhost:8983) through the Solr pod.

# Inspect all collections.
plane solr-status

# Inspect one collection.
plane solr-status iam

Monitor Backups

# List all backups.
kubectl get solrbackups -n datalayer-solr
# Describe a specific backup for detailed status.
kubectl describe solrbackup datalayer-solr-collection-backups -n datalayer-solr
# Plane summary command (analyzes command outputs and prints a summary).
plane solr-backups-status

Diagnose Failed Backups

When a backup fails, plane solr-backups-status reports:

• top-level backup status (running, completed, failed)
• per-collection status (running, successful, failed)
• per-collection S3 prefix (s3Prefix) to verify whether backup artifacts exist
• a failedProcessDebug command to inspect Solr Operator logs

In some runs, SolrBackup status does not expose a detailed root-cause message for failed collections. In that case the failure reason can remain generic even though the failure is real.

Practical verification workflow:

# 1) Confirm failed collections in the backup summary.
plane solr-backups-status

# 2) Check whether expected S3 backup content exists for a failed collection.
# Example for iam:
aws s3 ls s3://datalayer-backups-solr/datalayer-solr-collection-backups-iam/iam/index/

# 3) Inspect Solr operator logs for additional context.
kubectl logs -n datalayer-solr-operator $(kubectl get pods -n datalayer-solr-operator -o name | grep datalayer-solr-operator- | grep -v zookeeper | head -n 1 | sed 's#pod/##') --since=2h | grep -E 'datalayer-solr-collection-backups|error|failed|exception' -i

# 4) Inspect collection/shard/replica state.
plane solr-status iam

If the S3 path is empty for a failed collection (for example datalayer-solr-collection-backups-iam/iam/index/), that confirms the backup artifact was not produced, even when no explicit root-cause string is present in SolrBackup.status.

Clean Stale Backup Async Tasks

When Solr Operator logs show Task with the same requestid already exists for a collection backup, clear the stored async task response for the affected collection(s).

Use plane solr-backups-clean-tasks to clean request IDs for a selected scope (core, runtimes, all, or a single collection).

# Prompt for scope (core / runtimes / all / single collection).
plane solr-backups-clean-tasks

# Clean by predefined set.
plane solr-backups-clean-tasks core
plane solr-backups-clean-tasks runtimes
plane solr-backups-clean-tasks all

# Clean one collection.
plane solr-backups-clean-tasks collection library
# or shorthand:
plane solr-backups-clean-tasks library

Equivalent direct Solr API command for a single collection:

COLLECTION=library
kubectl exec -n datalayer-solr solr-datalayer-solrcloud-0 -c solrcloud-node -- \
  curl -s -u "solr:${DATALAYER_SOLR_PASSWORD}" \
  "http://localhost:8983/solr/admin/collections?action=DELETESTATUS&requestid=datalayer-solr-collection-backups-${COLLECTION}&wt=json"

If needed, flush all stored async statuses:

kubectl exec -n datalayer-solr solr-datalayer-solrcloud-0 -c solrcloud-node -- \
  curl -s -u "solr:${DATALAYER_SOLR_PASSWORD}" \
  "http://localhost:8983/solr/admin/collections?action=DELETESTATUS&flush=true&wt=json"

Stop (Remove) Backup Schedules

Use plane solr-backups-stop to delete the SolrBackup definitions with the same collection-set menu (core, runtimes, all).

Plane

# Prompt for backup set to remove (core / runtimes / all).
plane solr-backups-stop
# Remove a specific backup definition directly.
plane solr-backups-stop core
plane solr-backups-stop runtimes
plane solr-backups-stop all

Bash

# Equivalent kubectl delete commands.
kubectl delete -f $PLANE_HOME/etc/specs/solr/backups/datalayer-backup-s3-core.yaml --ignore-not-found
kubectl delete -f $PLANE_HOME/etc/specs/solr/backups/datalayer-backup-s3-runtimes.yaml --ignore-not-found
kubectl delete -f $PLANE_HOME/etc/specs/solr/backups/datalayer-backup-s3.yaml --ignore-not-found

Restore Datalayer Solr

Restore Solr collections from an S3 backup using the plane solr-restore command. The restore calls the Solr Collections API RESTORE action asynchronously for each collection.

When restoring multiple collections, the script prompts which collection set to restore (core, runtimes, or all). Specifying a single collection on the command line bypasses the prompt.

Prerequisites

• The DATALAYER_SOLR_PASSWORD environment variable must be set.
• A valid backup must exist in the S3 repository (check with kubectl get solrbackups -n datalayer-solr).

Restore Collections

The default backup name is datalayer-solr-collection-backups (matching the SolrBackup CRD name).

Plane

# Restore collections (prompts for set: core / runtimes / all).
plane solr-restore
# Restore only a specific collection (no prompt).
plane solr-restore datalayer-solr-collection-backups iam

Bash

# Restore a single collection via kubectl exec into a Solr pod.
# The backup name per collection is: <backup-name>-<collection>
kubectl exec -n datalayer-solr solr-datalayer-solrcloud-0 -- \
  curl -s -u "solr:${DATALAYER_SOLR_PASSWORD}" \
  "http://localhost:8983/solr/admin/collections?action=RESTORE&name=datalayer-solr-collection-backups-iam&collection=iam&location=/&repository=s3&async=iam-restore"

Monitor Restore Progress

Restore operations are asynchronous. Use the solr-restore-status script to check a selected collection set (core, runtimes, or all), or query a single collection.

Plane

# Prompt for status set (core / runtimes / all).
plane solr-restore-status
# Check a single collection.
plane solr-restore-status iam

Bash

# Check the status of a single restore request.
kubectl exec -n datalayer-solr solr-datalayer-solrcloud-0 -- \
  curl -s -u "solr:${DATALAYER_SOLR_PASSWORD}" \
  "http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS&requestid=iam-restore"

Scale Datalayer Solr

Scale Solr replicas if needed.

kubectl scale \
  --replicas=5 \
  solrcloud/solr-datalayer \
  -n datalayer-solr

Tear Down Datalayer Solr

Tear down the created Solr Cloud if needed.

kubectl delete solrcloud solr-datalayer -n datalayer-solr
kubectl get solrcloud -A

Tear down the Solr Operator if needed.

Plane

plane down datalayer-solr-operator

Helm

export RELEASE=datalayer-solr-operator
export NAMESPACE=datalayer-solr-operator
helm delete $RELEASE --namespace $NAMESPACE
kubectl delete \
  -n $NAMESPACE \
  -f https://solr.apache.org/operator/downloads/crds/v0.9.1/all-with-dependencies.yaml

Known Limitations

• Solr Operator SolrBackup.status may mark collection backups as failed without exposing a concrete per-collection root-cause message.
• In that case, treat plane solr-backups-status as the operational source of truth for failure detection and use the reported s3Prefix values to verify artifact presence in S3.
• An empty prefix (for example s3://datalayer-backups-solr/datalayer-solr-collection-backups-iam/iam/index/) confirms artifact creation failed even when the CR status reason is generic.
• For root-cause investigation, use the failedProcessDebug command from plane solr-backups-status and inspect collection health with plane solr-status <collection>.