Skip to main content

☰ 🔎 Datalayer Solr

Apache Solr Operator​

To deploy Solr, you first need to deploy the Solr Operator.

helm repo add apache-solr https://solr.apache.org/charts
helm repo update
plane up datalayer-solr-operator

Check the availability of the Solr CRDs.

kubectl explain solrcloud.spec.zookeeperRef.provided.config
kubectl explain solrcloud.spec.zookeeperRef.provided.persistence
kubectl explain solrcloud.spec.zookeeperRef.provided.persistence.spec
plane ls

Check the availability of the Solr Operator Pods.

kubectl get pods -n datalayer-solr-operator -l control-plane=solr-operator

Datalayer Solr Cluster​

Prepare a AWS S3 bucket for the Solr backups (the DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME environment variable should contain that name) and create a secret with write access to that S3 bucket.

kubectl create secret generic aws-creds \
  --from-literal=access-key-id=$AWS_ACCESS_KEY_ID \
  --from-literal=secret-access-key=$AWS_SECRET_ACCESS_KEY \
  --namespace=datalayer-solr
kubectl describe secret aws-creds -n datalayer-solr

Create a secret for the Solr authentication.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: solr-basic-auth
  namespace: datalayer-solr
type: kubernetes.io/basic-auth
stringData:
  username: ${DATALAYER_SOLR_USERNAME}
  password: ${DATALAYER_SOLR_PASSWORD}
EOF
kubectl describe secret solr-basic-auth -n datalayer-solr
# Secret for the datalayer-api namespace.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: solr-basic-auth
  namespace: datalayer-api
type: kubernetes.io/basic-auth
stringData:
  username: ${DATALAYER_SOLR_USERNAME}
  password: ${DATALAYER_SOLR_PASSWORD}
EOF
kubectl describe secret solr-basic-auth -n datalayer-api

You are now ready to create a Solr cluster. ensure the DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME and DATALAYER_SOLR_BACKUP_S3_BUCKET_REGION variables with the name and region of the S3 bucket for the backups.

You can also configure the size of the cluster, a 3 nodes replica cluster is what is defined in the following spec.

Version compatibility note Use Solr 9.10.0 with Solr Operator 0.9.1 in this setup. Solr 10.x currently fails to start with the operator-generated solr.xml due to the genericCoreNodeNames parameter in the <solrcloud> section, which causes startup probe failures and pod restart loops. If you previously deployed with 10.x, patch the SolrCloud back to 9.10.0 and wait for the StatefulSet rollout to complete.

kubectl apply -f $PLANE_HOME/etc/specs/solr/datalayer-solr.yaml

Check the creation of the Solr Cluster Pods. It may take multiple minutes to get a completely up-and-running Solr cluster, mainly due to the time to provision the required storage.

kubectl get solrclouds -n datalayer-solr -w
# NAME             VERSION   TARGETVERSION   DESIREDNODES   NODES   READYNODES   UPTODATENODES   AGE
# solr-datalayer   9.10.0                    3              3       3            3               79s
kubectl get pods -n datalayer-solr -w
kubectl describe pods -n datalayer-solr | grep Node

Create Datalayer Solr Collections​

Solr collections are organized into three sets that can be initialized independently: Core, AI and Runtimes. Every collection belongs to exactly one of them — 18 / 6 / 8, 32 in total. A collection is never listed in two sets to make it visible from two planes.

Core Collections (18)​

Platform collections used by the main Datalayer services:

CollectionPurpose
api-keysIAM authentication tokens
contactsContact information
creditsUsage credits
eventsAgent lifecycle and system event records
iamIdentity and access management
inboundsInbound integrations
invitesUser invitations
libraryContent library
notificationsUser-facing notifications records
oauth-clientsOAuth client registrations
oauth-codesOAuth authorization codes
oauth-grantsOAuth grants
outboundsOutbound integrations
schedulesScheduled jobs
secretsEncrypted secrets metadata
spacesWorkspace spaces
successSuccess tracking
usagePlatform usage metrics

Note: The notifications collection stores user-facing notifications, while events stores lifecycle and system events.

AI Collections (6)​

What the AI services own — ai-agents and the Jupyter MCP Server gateway:

CollectionOwnerPurpose
ai-agentsai-agentsAI agents service records
evalsai-agentsEvaluation definitions and results
tool-approvalsai-agentsTool approval requests and decision state
mcp-gatewayJupyter MCP ServerHandles (notebook, toolset and sandbox bindings), the worker directory, rate windows and fired alerts
mcp-tasksJupyter MCP ServerTask projection (mcp_task roots, bounded mcp_task_output children)
mcp-auditJupyter MCP ServerImmutable audit events, one per call and decision

These six were Core until 2026-08-28, placed by the rule that a service's collections live in the set initialized by the plane where that service runs — and both AI services do run on the platform plane. That rule answers where a collection is created and leaves what it belongs to unsaid. These arrive with the AI services, a cluster running without those services has no use for them, and they are recovered with each other rather than with iam and spaces. So they are their own set, and the deployment plane still decides routing while ownership decides placement.

The set is the ai recovery scope (backup resource datalayer-solr-ai-backups). The three mcp-* collections form the narrower mcp scope inside it (datalayer-solr-mcp-backups), so the gateway can be recovered on its own and on a tighter schedule — the way the four Contents collections form the contents scope inside the Runtimes set.

Retention​

mcp-audit is append-only and never updated in place; rows age out on the retention window the gateway reports in GET /api/mcp/v1/audit (retention_days). mcp-gateway holds short-lived, compare-and-set records with their own TTLs — bindings expire on BINDING_TTL_SECONDS, rate windows sooner — so it stays small and hot and must not contend with the other two.

Runtimes Collections (8)​

Collections specific to the runtimes plane, Contents included — the Contents service is deployed there, beside the NFS that backs the Home Folder and the Volumes:

CollectionPurpose
rayRay service records
runtime-registrationsRuntime registrations
runtime-checkpointsRuntime checkpoint records (CRIU snapshots metadata)
sandbox-snapshotsRuntime snapshot records
contentsContent sources, grants, revisions and attachments
content-objectsContents object/version and materialization metadata
content-operationsContents jobs, transfers, sync, query and reconciliation state
content-auditImmutable Contents audit and provenance events

The inventory is 18 Core plus 6 AI plus 8 Runtimes collections, 32 in total. Always verify the arrays and manifests rather than relying on these numbers alone (scripts/check_solr_collections_parity.py in the Services repository asserts them against both canonical lists, and refuses a collection that appears in two sets by naming it).

All four Contents collections — contents, content-objects, content-operations and content-audit — belong to the Runtimes set, and together they form one operational recovery scope. The Contents service is deployed on the runtimes plane, beside the NFS that backs the Home Folder and the Volumes, so its collections initialize with that plane: a plane brought up with the Runtimes set has everything datalayer-contents reads and writes. Omitting one of them leaves the worker crashing at startup with Unknown collection, which is how content-operations was found missing. The contents scope stays available to backup, status, task cleanup and restore commands; it does not create duplicate collections.

The Contents mapping​

datalayer_solr/contents_mappings.json records every Contents document family: its type_s, the fields it writes, its nested children, the sorts it pages by and the fields carrying an expiry. It is generated from the DAO codecs and queries by scripts/generate_contents_mappings.py and verified by a test, so it describes what the code does rather than what it did when someone last updated a table by hand. Regenerate it in the same change that adds a field:

cd services/solr && python3 scripts/generate_contents_mappings.py

Cursor pagination follows Solr's rule: a cursorMark query must sort on the uniqueKey (id) as a tie breaker, or Solr answers HTTP 400. Contents sorts by public identity and appends id asc — uid asc,id asc for sources — so the page order stays meaningful and the cursor stays legal.

The collections use the shared managed configset and its typed dynamic fields (*_s, *_t, *_dt, numeric and boolean variants). Application-facing UIDs are ULIDs stored in exact UID fields; Solr's internal id remains important for optimistic updates and nested blocks but is not the user-facing identity. Bounded grants and attachment status details may use nested documents. Unbounded revisions, objects, operations and audit events are independent root documents linked by exact *_uid fields.

Source of Truth​

The collection arrays are defined in solr-collections.sh:

  • SOLR_COLLECTIONS_CORE — Core platform collections
  • SOLR_COLLECTIONS_RUNTIMES — Runtimes collections
  • SOLR_COLLECTIONS_CONTENTS — Contents recovery set across both groups
  • SOLR_COLLECTIONS — All collections combined (union of the above two)

The solr-init script prompts which collection set to create. This is how you add a set to a cluster that already has the others without issuing a create against every collection it already holds — bringing the Jupyter MCP Server up on an existing platform plane, say. Creating a collection that already exists fails and is skipped, so any set is safe to re-run; pick the narrowest one that covers what is missing.

plane solr-init
# Select:
#   1) Core       - Platform collections (18 collections)
#   2) Runtimes   - Runtimes collections (8 collections, Contents included)
#   3) All        - All collections (32 collections)
#   4) AI         - AI collections (6 collections: ai-agents, evals,
#                   tool-approvals, mcp-gateway, mcp-tasks, mcp-audit)
#
# The init pod is created, logs are streamed, and the pod is auto-cleaned.

Backup Datalayer Solr​

Solr collections are backed up to an AWS S3 bucket using the Solr Operator SolrBackup CRD. The backup relies on the s3 repository configured in the SolrCloud spec (see Datalayer Solr Cluster above).

Six backup definitions are available. Contents, AI and MCP have distinct resource names so their coordinated schedules can run beside the full backup:

FileScopeCR Name
datalayer-backup-s3.yamlAll 32 collectionsdatalayer-solr-collection-backups
datalayer-backup-s3-ai.yamlThe 6 AI collectionsdatalayer-solr-ai-backups
datalayer-backup-s3-core.yaml18 Core collectionsdatalayer-solr-collection-backups
datalayer-backup-s3-runtimes.yaml8 Runtimes collections, the four Contents ones includeddatalayer-solr-collection-backups
datalayer-backup-s3-contents.yaml4 Contents collectionsdatalayer-solr-contents-backups
datalayer-backup-s3-mcp.yaml3 Jupyter MCP Server collectionsdatalayer-solr-mcp-backups

All backups run every 10 minutes and retain up to 200 snapshots.

Prerequisites​

  • The aws-creds secret must exist in the datalayer-solr namespace (see Datalayer Solr Cluster).
  • The DATALAYER_SOLR_BACKUP_S3_BUCKET_NAME and DATALAYER_SOLR_BACKUP_S3_BUCKET_REGION environment variables must be set.

Apply the Backup Schedule​

# Prompt for backup set (core / runtimes / contents / all).
plane solr-backups-apply
# Apply a specific backup spec directly.
plane solr-backups-apply core
plane solr-backups-apply runtimes
plane solr-backups-apply contents
plane solr-backups-apply all

Inspect Solr Collection and Shard Health​

Use the Solr status command to inspect available collections and list shard and replica details directly from the Solr API (localhost:8983) through the Solr pod.

# Inspect all collections.
plane solr-status

# Inspect one collection.
plane solr-status iam

Monitor Backups​

# List all backups.
kubectl get solrbackups -n datalayer-solr
# Describe a specific backup for detailed status.
kubectl describe solrbackup datalayer-solr-collection-backups -n datalayer-solr
# Plane summary command (analyzes command outputs and prints a summary).
plane solr-backups-status
# Inspect only the coordinated Contents backup resource.
plane solr-backups-status contents

Diagnose Failed Backups​

When a backup fails, plane solr-backups-status reports:

  • top-level backup status (running, completed, failed)
  • per-collection status (running, successful, failed)
  • per-collection S3 prefix (s3Prefix) to verify whether backup artifacts exist
  • a failedProcessDebug command to inspect Solr Operator logs

In some runs, SolrBackup status does not expose a detailed root-cause message for failed collections. In that case the failure reason can remain generic even though the failure is real.

Practical verification workflow:

# 1) Confirm failed collections in the backup summary.
plane solr-backups-status

# 2) Check whether expected S3 backup content exists for a failed collection.
# Example for iam:
aws s3 ls s3://datalayer-backups-solr/datalayer-solr-collection-backups-iam/iam/index/

# 3) Inspect Solr operator logs for additional context.
kubectl logs -n datalayer-solr-operator $(kubectl get pods -n datalayer-solr-operator -o name | grep datalayer-solr-operator- | grep -v zookeeper | head -n 1 | sed 's#pod/##') --since=2h | grep -E 'datalayer-solr-collection-backups|error|failed|exception' -i

# 4) Inspect collection/shard/replica state.
plane solr-status iam

If the S3 path is empty for a failed collection (for example datalayer-solr-collection-backups-iam/iam/index/), that confirms the backup artifact was not produced, even when no explicit root-cause string is present in SolrBackup.status.

Clean Stale Backup Async Tasks​

When Solr Operator logs show Task with the same requestid already exists for a collection backup, clear the stored async task response for the affected collection(s).

Use plane solr-backups-clean-tasks to clean request IDs for a selected scope (core, runtimes, contents, all, or a single collection).

# Prompt for scope (core / runtimes / contents / all / single collection).
plane solr-backups-clean-tasks

# Clean by predefined set.
plane solr-backups-clean-tasks core
plane solr-backups-clean-tasks runtimes
plane solr-backups-clean-tasks contents
plane solr-backups-clean-tasks all

# Clean one collection.
plane solr-backups-clean-tasks collection library
# or shorthand:
plane solr-backups-clean-tasks library

Equivalent direct Solr API command for a single collection:

COLLECTION=library
kubectl exec -n datalayer-solr solr-datalayer-solrcloud-0 -c solrcloud-node -- \
  curl -s -u "solr:${DATALAYER_SOLR_PASSWORD}" \
  "http://localhost:8983/solr/admin/collections?action=DELETESTATUS&requestid=datalayer-solr-collection-backups-${COLLECTION}&wt=json"

If needed, flush all stored async statuses:

kubectl exec -n datalayer-solr solr-datalayer-solrcloud-0 -c solrcloud-node -- \
  curl -s -u "solr:${DATALAYER_SOLR_PASSWORD}" \
  "http://localhost:8983/solr/admin/collections?action=DELETESTATUS&flush=true&wt=json"

Stop (Remove) Backup Schedules​

Use plane solr-backups-stop to delete the SolrBackup definitions with the same collection-set menu (core, runtimes, contents, all).

# Prompt for backup set to remove (core / runtimes / contents / all).
plane solr-backups-stop
# Remove a specific backup definition directly.
plane solr-backups-stop core
plane solr-backups-stop runtimes
plane solr-backups-stop contents
plane solr-backups-stop all

Restore Datalayer Solr​

Restore Solr collections from an S3 backup using the plane solr-restore command. The restore calls the Solr Collections API RESTORE action asynchronously for each collection.

When restoring multiple collections, the script prompts which collection set to restore (core, runtimes, contents, or all). Specifying a single collection on the command line bypasses the prompt.

Prerequisites​

  • The DATALAYER_SOLR_PASSWORD environment variable must be set.
  • A valid backup must exist in the S3 repository (check with kubectl get solrbackups -n datalayer-solr).

Restore Collections​

Core, Runtimes and all-collection restores use datalayer-solr-collection-backups. The Contents scope automatically uses its independent datalayer-solr-contents-backups resource.

# Restore collections (prompts for set: core / runtimes / contents / all).
plane solr-restore
# Restore the coordinated Contents set from its dedicated backup.
plane solr-restore contents
# Restore only a specific collection (no prompt).
plane solr-restore datalayer-solr-collection-backups iam

Monitor Restore Progress​

Restore operations are asynchronous. Use the solr-restore-status script to check a selected collection set (core, runtimes, contents, or all), or query a single collection.

# Prompt for status set (core / runtimes / contents / all).
plane solr-restore-status
# Check all four Contents restore requests.
plane solr-restore-status contents
# Check a single collection.
plane solr-restore-status iam

For the coordinated metadata and managed-object recovery procedure, validation steps and rollback rules, see Continuity.

Scale Datalayer Solr​

Scale Solr replicas if needed.

kubectl scale \
  --replicas=5 \
  solrcloud/solr-datalayer \
  -n datalayer-solr

Tear Down Datalayer Solr​

Tear down the created Solr Cloud if needed.

kubectl delete solrcloud solr-datalayer -n datalayer-solr
kubectl get solrcloud -A

Tear down the Solr Operator if needed.

plane down datalayer-solr-operator

Known Limitations​

  • Solr Operator SolrBackup.status may mark collection backups as failed without exposing a concrete per-collection root-cause message.
  • In that case, treat plane solr-backups-status as the operational source of truth for failure detection and use the reported s3Prefix values to verify artifact presence in S3.
  • An empty prefix (for example s3://datalayer-backups-solr/datalayer-solr-collection-backups-iam/iam/index/) confirms artifact creation failed even when the CR status reason is generic.
  • For root-cause investigation, use the failedProcessDebug command from plane solr-backups-status and inspect collection health with plane solr-status <collection>.