☰ Datalayer System

KubernetesService

Datalayer System provides technical utilities and facilities needed by the other services.

🚨 DEPRECATED - The Datalayer Observer is deprecated and will be removed in a future release. It is replaced by the Datalayer OTEL service that only deploys the OpenTelemetry Collector and relies on the services to be instrumented and to send their telemetry to it.

📄️ OTEL

Datalayer OTEL service provides long-term observability storage and querying for the Datalayer Platform. It collects traces, metrics and logs via Apache Pulsar, stores them as Parquet files using Apache DataFusion, and exposes a REST + WebSocket API for querying.

📄️ Shared File System

Datalayer user data is stored on a Shared File System. The File System Storage can be implemented as a:

📄️ OpenFGA

Follow the OpenFGA installation guide.

📄️ Vault

We are using HashiCorp Vault.

📄️ Pulsar

Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud.

📄️ Kafka

Apache Kakfa is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.