Skip to main content

☰ 🐘 Datalayer PostgreSQL

Datalayer runs PostgreSQL through the CloudNativePG operator. The primary use case is a highly-available PostgreSQL cluster used as the persisted memory backend for the Datalayer AI agents via mem0ai, which stores agent memories in a pgvector vector store.

Build the pgvector Image​

mem0ai requires the vector (pgvector) extension. Datalayer publishes a CloudNativePG-compatible operand image that bundles it, built on top of ghcr.io/cloudnative-pg/postgresql so it stays compatible with the operator. The Dockerfile lives in etc/dockerfiles/datalayer-postgresql-pgvector.

# Example private registry namespace:
export DATALAYER_DOCKER_REGISTRY=<registry-host>/datalayer
cd $PLANE_HOME/etc/dockerfiles/datalayer-postgresql-pgvector
make build
make verify   # asserts pgvector is present in the image
make push

This publishes ${DATALAYER_DOCKER_REGISTRY}/datalayer-postgresql-pgvector:17 and ${DATALAYER_DOCKER_REGISTRY}/datalayer-postgresql-pgvector:latest. The memory cluster uses :17 by default. Override the PostgreSQL major version with make build POSTGRES_MAJOR=16.

CloudNativePG Operator​

To deploy PostgreSQL, you first need to deploy the CloudNativePG operator. It ships the Cluster, Backup, ScheduledBackup and related CRDs and reconciles the PostgreSQL clusters.

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
plane up datalayer-postgresql-operator

Version note Chart 0.29.0 ships CloudNativePG 1.30.0. The chart version is decoupled from the operator (app) version — check helm search repo cnpg/cloudnative-pg --versions before pinning.

Check the availability of the PostgreSQL CRDs.

kubectl get crd | grep cnpg.io
# backups.postgresql.cnpg.io
# clusters.postgresql.cnpg.io
# databases.postgresql.cnpg.io
# poolers.postgresql.cnpg.io
# scheduledbackups.postgresql.cnpg.io

Check the availability of the operator pod.

kubectl rollout status deployment -n datalayer-postgresql-operator
kubectl get pods -n datalayer-postgresql-operator

Optionally install the cnpg kubectl plugin for richer diagnostics (kubectl cnpg status ...).

curl -sSfL \
  https://github.com/cloudnative-pg/cloudnative-pg/raw/main/hack/install-cnpg-plugin.sh | \
  sudo sh -s -- -b /usr/local/bin

Datalayer PostgreSQL Memory Cluster​

The memory cluster hosts the mem0 database owned by the mem0 role, with the vector (pgvector) extension enabled at bootstrap. The operator automatically generates a connection secret named datalayer-postgresql-agent-memories-app.

Operator and Runtime Wiring​

datalayer-operator (running in datalayer-runtimes) reads PostgreSQL memory credentials from the CloudNativePG app secret in datalayer-postgresql and injects them into agent-runtimes pods. This keeps memory traffic internal to the cluster by default (datalayer-postgresql-agent-memories-rw.datalayer-postgresql.svc.cluster.local).

Default integration settings:

VariableDefault
DATALAYER_POSTGRESQL_AGENT_MEMORIES_NAMESPACEdatalayer-postgresql
DATALAYER_POSTGRESQL_AGENT_MEMORIES_SECRETdatalayer-postgresql-agent-memories-app
DATALAYER_POSTGRESQL_AGENT_MEMORIES_HOSTdatalayer-postgresql-agent-memories-rw.datalayer-postgresql.svc.cluster.local
DATALAYER_POSTGRESQL_AGENT_MEMORIES_PORT5432
DATALAYER_POSTGRESQL_AGENT_MEMORIES_DATABASEmem0
DATALAYER_POSTGRESQL_AGENT_MEMORIES_COLLECTIONagent_memories

If DATALAYER_POSTGRESQL_AGENT_MEMORIES_URI is not explicitly set, the operator builds it from the discovered username/password and host/port/db defaults.

Per-User Memory Isolation​

Memories are stored in a single shared mem0 database and agent_memories collection. Isolation is logical: every memory is partitioned by a trusted user_id derived from the runtime owner's personal account, never by the agent id and never from any value supplied by the model or caller. Memories are scoped to a user's personal account (organization and team scoping are not used yet).

The operator injects the trusted identity into each agent-runtimes pod from the reservation labels, so a pod can never impersonate another user:

VariableSource labelPurpose
DATALAYER_USER_UIDruntime-pools.datalayer.io/user-uidAuthenticated user id.
DATALAYER_USER_HANDLEruntime-pools.datalayer.io/user-account-handleHuman-readable handle (fallback).

The agent-runtimes memory gateway resolves these into a single effective key user_id (the personal account, for example user-123) and passes it to mem0ai as the partition key. The agent id remains a secondary scope for agent-private memories. Set AGENT_RUNTIMES_MEMORY_USER_ID to override the resolved identity (mainly for local development).

Because pgvector stores these scope fields in the payload JSONB column, the cluster bootstrap creates expression indexes for the hot filters:

CREATE INDEX IF NOT EXISTS agent_memories_user_id_idx
  ON agent_memories ((payload->>'user_id'));
CREATE INDEX IF NOT EXISTS agent_memories_agent_id_idx
  ON agent_memories ((payload->>'agent_id'));

pgvector image mem0ai requires the vector (pgvector) extension. Datalayer publishes a CloudNativePG-compatible image that bundles it, ${DATALAYER_DOCKER_REGISTRY}/datalayer-postgresql-pgvector:17, built from etc/dockerfiles/datalayer-postgresql-pgvector (see Build the pgvector image). This is the default operand image; override it with DATALAYER_POSTGRESQL_IMAGE. If DATALAYER_DOCKER_REGISTRY is not set, Plane falls back to ${DATALAYER_DOCKER_REGISTRY_HOST}/datalayer.

The cluster is configured through the following environment variables (defaults shown):

VariableDefaultPurpose
DATALAYER_POSTGRESQL_INSTANCES3Number of instances (1 primary + replicas).
DATALAYER_POSTGRESQL_STORAGE_SIZE10GiPersistent storage per instance.
DATALAYER_POSTGRESQL_IMAGE${DATALAYER_DOCKER_REGISTRY}/datalayer-postgresql-pgvector:17Operand image bundling pgvector.
DATALAYER_POSTGRESQL_AGENT_MEMORIES_DATABASEmem0mem0ai database name.
DATALAYER_POSTGRESQL_AGENT_MEMORIES_OWNERmem0mem0ai owning role.
DATALAYER_POSTGRESQL_AGENT_MEMORIES_VECTOR_DIMS1536pgvector dimensions for agent_memories.vector.
DATALAYER_POSTGRESQL_IMAGE_PULL_SECRETreg-credsImage pull secret in datalayer-postgresql namespace.
DATALAYER_POSTGRESQL_BACKUP_S3_BUCKET_NAME(unset)Enables S3 backups when set.
DATALAYER_POSTGRESQL_BACKUP_S3_BUCKET_REGIONus-east-1S3 bucket region.
DATALAYER_POSTGRESQL_BACKUP_RETENTION30dBackup retention policy.
DATALAYER_POSTGRESQL_BACKUP_SCHEDULE0 0 */6 * * *Backup cron (with seconds field).

How the Memory Database Is Created​

Run

plane reg-creds-create
plane postgresql-agent-memories-create

It will create a CloudNativePG Cluster with a bootstrap initdb block that creates the memory database and owner role on first initialization:

bootstrap:
  initdb:
    database: ${DATALAYER_POSTGRESQL_AGENT_MEMORIES_DATABASE:-mem0}
    owner: ${DATALAYER_POSTGRESQL_AGENT_MEMORIES_OWNER:-mem0}
    postInitApplicationSQL:
      - CREATE EXTENSION IF NOT EXISTS vector
      - CREATE TABLE IF NOT EXISTS agent_memories (id UUID PRIMARY KEY, vector vector(${DATALAYER_POSTGRESQL_AGENT_MEMORIES_VECTOR_DIMS:-1536}), payload JSONB)
      - ALTER TABLE agent_memories OWNER TO ${DATALAYER_POSTGRESQL_AGENT_MEMORIES_OWNER:-mem0}
      - GRANT ALL PRIVILEGES ON TABLE agent_memories TO ${DATALAYER_POSTGRESQL_AGENT_MEMORIES_OWNER:-mem0}
      - CREATE INDEX IF NOT EXISTS agent_memories_text_lemmatized_idx ON agent_memories USING gin(to_tsvector('simple', payload->>'text_lemmatized'))
      - CREATE INDEX IF NOT EXISTS agent_memories_user_id_idx ON agent_memories ((payload->>'user_id'))
      - CREATE INDEX IF NOT EXISTS agent_memories_agent_id_idx ON agent_memories ((payload->>'agent_id'))

This means:

  • The memory database (default mem0) is created automatically.
  • The owning role (default mem0) is created automatically.
  • The vector extension is enabled automatically for mem0ai.
  • The agent_memories table is created automatically with id, vector, and payload columns.
  • The agent_memories table ownership and privileges are granted to the memory owner role (default mem0).
  • The agent_memories_text_lemmatized_idx, agent_memories_user_id_idx, and agent_memories_agent_id_idx indexes are created automatically.
  • CloudNativePG also generates the application secret datalayer-postgresql-agent-memories-app with connection credentials (username, password, uri).

After the cluster reaches ready state, plane postgresql-agent-memories-create also runs an idempotent schema reconciliation against the primary. This guarantees that the agent_memories table and indexes are present even when bootstrap SQL was changed between releases.

Note: these bootstrap settings apply when the cluster data directory is initialized. If you change DATALAYER_POSTGRESQL_AGENT_MEMORIES_DATABASE, DATALAYER_POSTGRESQL_AGENT_MEMORIES_OWNER, or DATALAYER_POSTGRESQL_AGENT_MEMORIES_VECTOR_DIMS on an existing cluster, re-create a fresh cluster (or provision roles/databases manually) to apply the new values.

You can verify the memory database setup with:

# 1) Read the generated URI and export the generated password.
kubectl get secret datalayer-postgresql-agent-memories-app -n datalayer-postgresql -o jsonpath='{.data.uri}' | base64 -d
export PGSQL_PASSWORD="$(kubectl get secret datalayer-postgresql-agent-memories-app -n datalayer-postgresql -o jsonpath='{.data.password}' | base64 -d)"

# 2) Port-forward to the primary.
plane pf-postgresql

# 3) From another terminal, connect and verify DB/user/extension.
# sslmode=disable avoids a TLS-teardown RST that would drop the port-forward
# (traffic already rides the encrypted kubectl port-forward tunnel).
psql "postgres://mem0:${PGSQL_PASSWORD}@localhost:5432/mem0?sslmode=disable" -c "SELECT current_database(), current_user;"
psql "postgres://mem0:${PGSQL_PASSWORD}@localhost:5432/mem0?sslmode=disable" -c "SELECT extname FROM pg_extension WHERE extname='vector';"
plane postgresql-agent-memories-create

A checked-in reference spec is available at $PLANE_HOME/etc/specs/postgresql/datalayer-postgresql-agent-memories.yaml.

Monitor the PostgreSQL Cluster​

It may take a few minutes for the cluster to become fully healthy, mostly due to storage provisioning.

plane postgresql-agent-memories-status

Backup the PostgreSQL Cluster​

Backups are written to an AWS S3 bucket using CloudNativePG's Barman object store. They rely on the backup.barmanObjectStore stanza, which plane postgresql-agent-memories-create adds automatically when DATALAYER_POSTGRESQL_BACKUP_S3_BUCKET_NAME is set.

Prerequisites​

Create the aws-creds secret with write access to the bucket in the datalayer-postgresql namespace.

kubectl create secret generic aws-creds \
  --from-literal=access-key-id=$AWS_ACCESS_KEY_ID \
  --from-literal=secret-access-key=$AWS_SECRET_ACCESS_KEY \
  --namespace=datalayer-postgresql

Set the bucket variables and (re)create the cluster so it declares the object store.

export DATALAYER_POSTGRESQL_BACKUP_S3_BUCKET_NAME=my-datalayer-postgresql-backups
export DATALAYER_POSTGRESQL_BACKUP_S3_BUCKET_REGION=us-east-1
plane postgresql-agent-memories-create

Schedule and Trigger Backups​

# Apply the recurring backup schedule.
plane postgresql-agent-memories-backup
# Trigger a one-off backup immediately.
plane postgresql-agent-memories-backup now

Restore the PostgreSQL Cluster​

CloudNativePG restores into a new cluster that bootstraps from the source cluster's object store — the source is never mutated. Optionally provide a point-in-time recovery (PITR) target.

After restore readiness, plane postgresql-agent-memories-restore runs the same idempotent schema reconciliation so agent_memories and its indexes are recreated automatically when restoring from older backups.

# Restore the latest backup into datalayer-postgresql-agent-memories-restore.
plane postgresql-agent-memories-restore
# Restore into a named cluster at a specific point in time.
plane postgresql-agent-memories-restore datalayer-postgresql-agent-memories-pitr "2026-07-31 10:00:00+00"

Access the PostgreSQL Cluster​

The operator exposes a read-write service (always the current primary) and a read-only service (replicas).

ServiceEndpoint
Read-writedatalayer-postgresql-agent-memories-rw.datalayer-postgresql.svc.cluster.local:5432
Read-onlydatalayer-postgresql-agent-memories-ro.datalayer-postgresql.svc.cluster.local:5432

Read the auto-generated application credentials.

kubectl get secret datalayer-postgresql-agent-memories-app -n datalayer-postgresql -o jsonpath='{.data.username}' | base64 -d
kubectl get secret datalayer-postgresql-agent-memories-app -n datalayer-postgresql -o jsonpath='{.data.password}' | base64 -d
kubectl get secret datalayer-postgresql-agent-memories-app -n datalayer-postgresql -o jsonpath='{.data.uri}' | base64 -d

Port-forward to the primary for local access.

plane pf-postgresql

Use as mem0ai Memory​

mem0ai uses pgvector as its vector store. Point it at the memory cluster's read-write service and the mem0 database.

from mem0 import Memory

config = {
    "vector_store": {
        "provider": "pgvector",
        "config": {
            "host": "datalayer-postgresql-agent-memories-rw.datalayer-postgresql.svc.cluster.local",
            "port": 5432,
            "dbname": "mem0",
            "user": "mem0",          # from datalayer-postgresql-agent-memories-app secret
            "password": "...",       # from datalayer-postgresql-agent-memories-app secret
            "collection_name": "agent_memories",
        },
    },
}

memory = Memory.from_config(config)

Teardown​

# Delete the memory cluster (keeps the operator).
plane postgresql-agent-memories-terminate
# Remove the operator.
plane down datalayer-postgresql-operator

Cheat Sheet​

plane postgresql-help