☰ Datalayer System
Datalayer System provides technical utilities and facilities needed by the other services.
Ingress
Kubernetes
Cert Manager
Deploy Datalayer Certificate Manager
Home
Kubernetes
Observer
🚨 DEPRECATED - The Datalayer Observer is deprecated and will be removed in a future release. It is replaced by the Datalayer OTEL service that only deploys the OpenTelemetry Collector and relies on the services to be instrumented and to send their telemetry to it.
OTEL
Datalayer OTEL service provides long-term observability storage and querying for the Datalayer Platform. It collects traces, metrics and logs via Apache Pulsar, stores them as Parquet files using Apache DataFusion, and exposes a REST + WebSocket API for querying.
Shared File System
Datalayer user data is stored on a Shared File System. The File System Storage can be implemented as a:
OpenFGA
Follow the OpenFGA installation guide.
Vault
We are using HashiCorp Vault.
Pulsar
Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud.
Kafka
Apache Kakfa is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Solr
Apache Solr Operator
PostgreSQL
Datalayer runs PostgreSQL through the CloudNativePG operator. The primary use case is a highly-available PostgreSQL cluster used as the persisted memory backend for the Datalayer AI agents via mem0ai, which stores agent memories in a pgvector vector store.
Datasets
GPU CUDA
Install CUDA Operator
Datashim
Datashim needs to be deployed in the cloud to benefit from the Runtime Content features.
Inbounds
Check the availability of the Inbounds Pods.
Outbounds
Check the availability of the Outbounds Pods.