Skip to main content

Continuity

Disaster Recovery​

To prevent any disaster, ensure you take the recommended backup of the Datalayer services.

tip

Contact us for any question. We are here to help.

Backup​

Take recurring backups with a S3 SolrBackup.

kubectl apply -f $PLANE_HOME/etc/specs/solr/backups/datalayer-backup-s3.yaml
kubectl get solrbackups -n datalayer-solr -w
# NAMESPACE        NAME                                CLOUD            STARTED   FINISHED   SUCCESSFUL   NEXTBACKUP   AGE
# datalayer-solr   datalayer-solr-collection-backups  solr-datalayer   44s                                            44s
kubectl describe solrbackups -n datalayer-solr
# Backup schedule is every 10 minutes (*/10 * * * *), maxSaved=200.
# Example check for a collection backup artifact path.
aws s3 ls s3://datalayer-backups-solr/datalayer-solr-collection-backups-iam/iam/index/
# If empty while the collection is reported as failed, backup artifact creation failed.

# Backup status summary with failed collections and S3 prefixes.
plane solr-backups-status

# Solr shard/replica status for one collection.
plane solr-status iam
kubectl delete -f $PLANE_HOME/etc/specs/solr/backups/datalayer-backup-s3.yaml

Troubleshooting Note​

In some failed backup runs, SolrBackup.status does not include a specific root-cause message for each failed collection. When this happens:

  1. Use plane solr-backups-status to identify failed collections and their s3Prefix values.
  2. Verify the S3 prefix content directly (empty prefix confirms artifact was not written).
  3. Use the failedProcessDebug command printed by plane solr-backups-status to inspect operator logs.
  4. Use plane solr-status <collection> to inspect current shard and replica health.

Simulate a Disaster​

Delete Solr documents.

for COLLECTION in accounts credits invites spaces tweets usage
do
    curl http://localhost:8983/solr/$COLLECTION/update?commitWithin=500 -d '{ delete: { query: "*:*" } }'
done

... or delete and re-create empty collections before running the restore.

# export SOLR_HOME=.../opt/solr # Install Solr on your local machine https://solr.apache.org (you will need java - sudo apt install default-jre)
for COLLECTION in accounts credits invites spaces tweets usage
do
    curl http://localhost:8983/solr/$COLLECTION/update?commitWithin=500 -d '{ delete: { query: "*:*" } }'
    curl http://localhost:8983/solr/admin/collections?action=DELETE -d "name=${COLLECTION}"
    $SOLR_HOME/bin/solr create -c $COLLECTION -shards 3 -replicationFactor 3 -d $PLANE_HOME/etc/dockerfiles/datalayer-solr/config/ -p 8983 -force
done

Restore from a Disaster​

Run the effective restore.

# Restore https://solr.apache.org/guide/8_11/collectimanagement.html#restore
#
# backupId=10& # You may want to force the backupId, if not provided, the last one (most recent one) will be used.
#
for COLLECTION in accounts credits invites spaces tweets
do
    echo
    echo Restoring Solr "$COLLECTION" collection
    echo ---------------------------------------
    curl http://localhost:8983/solr/admin/collections -d '
action=RESTORE&
repository=s3&
collection='"$COLLECTION"'&
location=s3:/&
name=datalayer-solr-collection-backups-'"$COLLECTION"''
done
#
curl http://localhost:8983/solr/accounts/select?q=*:*
# curl http://localhost:8983/solr/accounts/update?commitWithin=500 -d '{ delete: { query: "*:*" } }'