☰ 🔭 Datalayer OTEL

Datalayer OTEL service provides long-term observability storage and querying for the Datalayer Platform. It collects traces, metrics and logs via Apache Pulsar, stores them as Parquet files using Apache DataFusion, and exposes a REST + WebSocket API for querying.

The datalayer-otel Helm chart deploys two components into the datalayer-otel namespace:

OTEL Collector – An OpenTelemetry Collector Contrib instance that receives OTLP data (gRPC :4317, HTTP :4318) from instrumented services and exports it to Apache Pulsar topics.
FastAPI Query Service – A Python FastAPI application that consumes OTLP data from Pulsar, stores it in Parquet files (via DataFusion), and serves a REST API for querying traces, metrics and logs.

  ┌──────────────┐   OTLP gRPC/HTTP   ┌────────────────────┐
  │  Services    │ ──────────────────► │  OTEL Collector   │
  │  (13 svcs)   │                     │  (datalayer-otel) │
  └──────────────┘                     └────────┬──────────┘
                                                │  Pulsar
                                      ┌─────────▼──────────┐
                                      │   Apache Pulsar    │
                                      │   otel-traces      │
                                      │   otel-metrics     │
                                      │   otel-logs        │
                                      └─────────┬──────────┘
                                                │  consume
                                      ┌─────────▼──────────┐
                                      │  FastAPI Service   │
                                      │  DataFusion Store  │
                                      │  (Parquet files)   │
                                      └─────────┬──────────┘
                                                │  REST / WS
                                      ┌─────────▼──────────┐
                                      │  UI / CLI clients  │
                                      └────────────────────┘

info

This is separate from the datalayer-observer stack (deployed in the datalayer-observer namespace) which runs Tempo, Prometheus and Loki for Grafana-based dashboards. The datalayer-otel service provides a dedicated Datalayer-native API for observability data, with SQL query capabilities via DataFusion.

Deploy Datalayer OTEL

Plane
Helm
Terraform

plane up datalayer-otel

export RELEASE=datalayer-otel
export NAMESPACE=datalayer-otel
helm upgrade \
    --install $RELEASE \
    $PLANE_HOME/etc/helm-private/charts/datalayer-otel \
    --create-namespace \
    --namespace $NAMESPACE \
    --set fastapi.image="${DATALAYER_DOCKER_REGISTRY:-datalayer}/datalayer-otel:0.1.0" \
    --set fastapi.jwtSecret="${DATALAYER_JWT_SECRET}" \
    --set fastapi.jwtIssuer="${DATALAYER_JWT_ISSUER}" \
    --set fastapi.jwtAlgorithm="${DATALAYER_JWT_ALGORITHM:-HS256}" \
    --set fastapi.jwtCacheValidate="${DATALAYER_JWT_CACHE_VALIDATE}" \
    --set fastapi.iamApiKey="${DATALAYER_IAM_API_KEY}" \
    --set fastapi.corsOrigin="${DATALAYER_CORS_ORIGIN:-*}" \
    --set fastapi.pulsar.url="${DATALAYER_PULSAR_URL:-pulsar://pulsar-broker:6650}" \
    --set fastapi.pulsar.batchTimeoutSeconds="${DATALAYER_OTEL_PULSAR_BATCH_TIMEOUT_SECONDS:-3}" \
    --set fastapi.retention.days="${DATALAYER_OTEL_RETENTION_DAYS:-30}" \
    --set fastapi.logfire.apiKey="${DATALAYER_LOGFIRE_API_KEY}" \
    --set fastapi.logfire.project="${DATALAYER_LOGFIRE_PROJECT:-starter-project}" \
    --set fastapi.logfire.url="${DATALAYER_LOGFIRE_URL:-https://logfire-us.pydantic.dev}" \
    --set fastapi.logfire.sendToLogfire="${DATALAYER_LOGFIRE_SEND_TO_LOGFIRE:-true}" \
    --set collector.pulsar.endpoint="${DATALAYER_PULSAR_URL:-pulsar://pulsar-broker:6650}" \
    --timeout 5m

cd terraform
terraform init
terraform apply
./generated/clouder-Kubeadm-setup.sh
export KUBECONFIG=~/.clouder/kubeadm/<cluster-name>/kubeconfig
./generated/services/deploy-datalayer-otel.sh

Check the availability of the OTEL Pods.

kubectl get pods -n datalayer-otel

API Endpoints

The FastAPI query service exposes the following endpoints under /api/otel/v1:

Endpoint	Method	Auth	Description
`/ping`	GET	No	Health check
`/traces/`	GET	JWT	List / query traces
`/metrics/`	GET	JWT	Query metrics
`/logs/`	GET	JWT	Query logs
`/query/sql`	POST	JWT	Run ad-hoc SQL via DataFusion
`/system/stats`	GET	Admin	Storage statistics
`/ws`	WS	Token	Real-time WebSocket stream
`/dashboards/`	GET, POST	JWT	The built-in MCP and orchestration dashboards, then the account's own
`/dashboards/{id}`	GET, PUT, DELETE	JWT	One dashboard; built-ins are read-only (`409`)
`/dashboards/{id}/data`	GET	JWT	Every panel, answered under the caller's account scope

API docs are available at https://<RUN_HOST>/api/otel/v1/docs (Swagger) and /api/otel/v1/redoc (ReDoc).

Dashboards

The MCP gateway keeps its metrics catalogue and serves no telemetry route: this service is where its signals are read, so its dashboards live here, beside the platform's others. Seven are built in. Six are the gateway's — service levels, the four SLIs the app's Observability view and datalayer mcp metrics read, and one per marketed MCP area: calls and refusals, runs, sandboxes by provider, data access, agents by organization — generated from a copy of the gateway's catalogue (datalayer_otel/dashboards.py) that a contract test holds equal to the gateway's telemetry.CATALOGUE. The seventh, orchestration, is generated the same way from a copy of agent-runtimes' orchestration_measures.CATALOGUE: the measures the execution store records wherever executions are held, in the agents service and in durable. An instrument a writer adds without a panel fails a test rather than going undrawn; a panel naming a label its writer does not emit fails at import.

A dashboard is a list of panels. A panel is one metric, optionally split by one of its labels (by) and narrowed to some label values (where, as {"recovery": "disconnected"}), summarised one way: sum, last, avg, max or count. The orchestration dashboard narrows its settled executions to those that lost sight of their worker and to those whose worker was lost, each split by how they ended, which is what the share that completed is read from.

How a panel reads its series

Services export counters as running totals, the OpenTelemetry default: every export repeats everything counted since the process started, so a service handling one call a minute reports 1, 2, 3, 4 and 5 over five minutes. Adding those reports up says 15; the answer is 5. A panel therefore reads each series — one process's running total for one set of labels — and only then combines the series:

a counter is how much its running total went up over the window: its last reading minus its reading when the window opened, and all of it when its process started inside the window;
a change reported as a delta adds up;
a gauge, or an up-down counter, is its latest reading;
a duration is its total over its count.

sum adds the series' values; avg and max are over series, except for a duration, whose avg is its total time over its total count; count is how many things were counted, or how many series a gauge has. p50, p95 and p99 read a percentile from a histogram's buckets: each series' bucket increases over the window, added across processes, and the rank interpolated within the bucket it falls in, as Prometheus's histogram_quantile does. A rank in the last bucket, which has no upper bound, reads as that bucket's lower bound. A panel whose aggregation is not its metric's default names it in its id (mcp-call-duration-p95).

The consumer keeps what makes a point part of a series: start_time_unix_nano (when its running total started, so a restarted process is a new series rather than a counter that went down), aggregation_temporality (cumulative or delta), is_monotonic (a sum that only goes up), and for an explicit-bucket histogram its bucket_counts and explicit_bounds. Points stored before those columns existed read as running totals from an unknown start. Each panel is two aggregate queries, one row per series, so a window of any length is read whole; the points a chart draws — each export's step — are what limit can cut, and points_truncated says when it did.

GET /dashboards/{id}/data?start=&end=&account_uid= answers every panel under the caller's resolved account scope, the same scope every other query here gets. That is what "agents by organization" means: the same panels, read as one organization by passing its account_uid. A panel whose query fails answers with its error and the other panels still answer.

Whose account a point is under decides who sees it. The gateway exports with a service credential, so its platform-level points — workers, dependencies, jobs — are filed under the gateway's own account, and a platform administrator asking for a built-in MCP dashboard with no account_uid reads that account rather than their personal one, where the gateway has never written a point. The orchestration dashboard is never redirected: its points carry usage_account_uid, the account whose executions they count. The account is found in the store, from the instruments only the gateway writes about itself, not configured. A call made in an organization carries usage_account_uid as a label, and the consumer files that point under the organization instead: that is what "agents by organization" reads when an owner passes their organization's account_uid, or opens it from the console. Naming an account always means that account, and a custom dashboard is never redirected.

People's own dashboards are JSON files under the data directory, one per dashboard per account ({data_dir}/dashboards/{account}/{id}.json); dashboards are few, small and edited by hand, so there is no table to migrate. A custom panel may chart any metric this service holds — the other services report here too. Built-in ids cannot be taken, updated or deleted.

CLI

The datalayer-otel package also ships a CLI for operating and querying the service:

datalayer-otel serve         # Start the FastAPI server
datalayer-otel traces        # List / get traces
datalayer-otel metrics       # Query metrics
datalayer-otel logs          # Query logs
datalayer-otel query         # Run ad-hoc SQL via DataFusion
datalayer-otel stats         # Show storage statistics
datalayer-otel smoke-test    # Send traces/metrics/logs and query them back
datalayer-otel logfire       # Send test spans/logs to Logfire
datalayer-otel flush         # Force-flush buffered data
datalayer-otel services      # List observed service names

All query commands authenticate via DATALAYER_API_KEY (Bearer token).

Environment Variables

FastAPI Query Service

Authentication (via `datalayer_common`)

Variable	Required	Default	Description
`DATALAYER_JWT_SECRET`	Yes	–	Shared secret for JWT token validation
`DATALAYER_JWT_ISSUER`	Yes	–	Expected JWT token issuer (e.g. `https://id.datalayer.run`)
`DATALAYER_JWT_ALGORITHM`	No	`HS256`	JWT signing algorithm
`DATALAYER_JWT_CACHE_VALIDATE`	No	`false`	Cache JWT validation results
`DATALAYER_IAM_API_KEY`	Yes	–	Internal service-to-service API key

Service Configuration

Variable	Required	Default	Description
`DATALAYER_OTEL_PORT`	No	`7800`	Port the FastAPI server listens on
`DATALAYER_CORS_ORIGIN`	No	`*`	Allowed CORS origin (used by `datalayer_common`)

DataFusion / Storage

Variable	Required	Default	Description
`DATALAYER_OTEL_DATAFUSION_DATA_DIR`	No	`/var/lib/datalayer-otel/data`	Path to Parquet data directory
`DATALAYER_OTEL_DATAFUSION_MAX_ROWS_PER_FILE`	No	`100000`	Max rows per Parquet file before rotation
`DATALAYER_OTEL_RETENTION_DAYS`	No	`30`	Number of days to retain Parquet files

Apache Pulsar

Variable	Required	Default	Description
`DATALAYER_OTEL_PULSAR_URL`	No	`pulsar://pulsar-broker:6650`	Pulsar broker URL
`DATALAYER_OTEL_PULSAR_TRACES_TOPIC`	No	`persistent://public/default/otel-traces`	Pulsar topic for traces
`DATALAYER_OTEL_PULSAR_METRICS_TOPIC`	No	`persistent://public/default/otel-metrics`	Pulsar topic for metrics
`DATALAYER_OTEL_PULSAR_LOGS_TOPIC`	No	`persistent://public/default/otel-logs`	Pulsar topic for logs
`DATALAYER_OTEL_PULSAR_SUBSCRIPTION`	No	`datalayer-otel-consumer`	Pulsar subscription name
`DATALAYER_OTEL_PULSAR_BATCH_SIZE`	No	`1000`	Number of messages per batch
`DATALAYER_OTEL_PULSAR_BATCH_TIMEOUT_SECONDS`	No	`3`	Max seconds before a partial batch is flushed

Logfire (optional)

Variable	Required	Default	Description
`DATALAYER_LOGFIRE_API_KEY`	No	`""`	Logfire write token
`DATALAYER_LOGFIRE_PROJECT`	No	`starter-project`	Logfire project name
`DATALAYER_LOGFIRE_URL`	No	`https://logfire-us.pydantic.dev`	Logfire base URL
`DATALAYER_LOGFIRE_SEND_TO_LOGFIRE`	No	`true`	Whether to send data to Logfire cloud

OTEL Collector

The collector is configured via the Helm chart ConfigMap. The following Helm values control its behavior:

Helm Value	Default	Description
`collector.image`	`otel/opentelemetry-collector-contrib:0.117.0`	Collector container image
`collector.otlp.grpcPort`	`4317`	OTLP gRPC listen port
`collector.otlp.httpPort`	`4318`	OTLP HTTP listen port
`collector.pulsar.endpoint`	`pulsar://pulsar-broker:6650`	Pulsar broker endpoint
`collector.pulsar.topics.traces`	`persistent://public/default/otel-traces`	Pulsar traces topic
`collector.pulsar.topics.metrics`	`persistent://public/default/otel-metrics`	Pulsar metrics topic
`collector.pulsar.topics.logs`	`persistent://public/default/otel-logs`	Pulsar logs topic

Instrumented Services

The platform services that send telemetry to the OTEL Collector use these environment variables (set via up.sh and the datalayer_common.instrumentation module):

Variable	Required	Default	Description
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	Yes	`http://datalayer-otel-otel-collector-svc.datalayer-otel.svc.cluster.local:4317`	Collector gRPC endpoint for traces
`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`	Yes	`http://datalayer-otel-otel-collector-svc.datalayer-otel.svc.cluster.local:4317`	Collector gRPC endpoint for metrics
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`	Yes	`http://datalayer-otel-otel-collector-svc.datalayer-otel.svc.cluster.local:4317`	Collector gRPC endpoint for logs
`DATALAYER_OTEL_API_KEY`	No	`""`	Bearer token attached to OTLP export requests
`OTEL_PYTHON_LOG_LEVEL`	No	`info`	Python OTEL SDK log level
`OTEL_SDK_DISABLED`	No	`false`	Set to `true` to disable the OTEL SDK entirely

Endpoint Values

For a standard deployment, all three OTEL_EXPORTER_OTLP_*_ENDPOINT variables point to the same collector:

# The datalayer-otel collector in the datalayer-otel namespace
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://datalayer-otel-otel-collector-svc.datalayer-otel.svc.cluster.local:4317"
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://datalayer-otel-otel-collector-svc.datalayer-otel.svc.cluster.local:4317"
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://datalayer-otel-otel-collector-svc.datalayer-otel.svc.cluster.local:4317"

If the datalayer-observer stack is also deployed, services can point to the observer collector instead (or in addition) — that collector forwards to Tempo, Prometheus and Loki for Grafana dashboards:

# The datalayer-observer collector in the datalayer-observer namespace
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://datalayer-collector-collector.datalayer-observer.svc.cluster.local:4317"
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://datalayer-collector-collector.datalayer-observer.svc.cluster.local:4317"
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://datalayer-collector-collector.datalayer-observer.svc.cluster.local:4317"

Service names

Each service names itself when it installs the providers (datalayer_common.instrumentation.instrument(app, "<name>")), and that name becomes the service.name resource attribute every span, metric and log record carries — it is what datalayer-otel services lists and what the SQL examples group by:

ai-agents, ai_inference, contents, growth, iam, inbounds, library, manager, runtimes, scheduler, spacer, status, success, support. The Operator and the Runtimes companion pass their name in from their configuration rather than hard-coding it.

MCP Server gateway and workers

The MCP Server gateway is addressed at the collector by the same OTEL_* variables as the other services — its chart sets all three endpoints, DATALAYER_OTEL_API_KEY, OTEL_PYTHON_LOG_LEVEL and OTEL_SDK_DISABLED, and the gateway validates the endpoint at startup and reports it on GET /api/mcp/readyz. Telemetry is listed there and never probed: no user operation waits on the collector, and its availability is never a readiness condition for the gateway.

Two service.names are reserved for it: datalayer-mcp-server for the gateway replicas, and datalayer-mcp-worker for the per-user jupyter-mcp-server processes a replica runs — a worker's last lines are worth querying after the process is gone, which is why it gets a name of its own rather than being folded into the gateway's.

Nothing arrives from the gateway yet

The gateway does not install an OpenTelemetry SDK provider: it neither calls datalayer_common.instrumentation.instrument nor runs under opentelemetry-instrument. Its instruments are therefore created against the global no-op meter, no service.name is set, and no span, metric or log record from the gateway or its workers reaches this collector. The endpoints are configured so the addressing is right the day the provider is installed. Until then the gateway is read through its own logs, /api/mcp/readyz and GET /api/mcp/v1/operations/workers.

The MCP metric catalog

The names are declared in one place — datalayer_mcp_server/telemetry.py — so a dashboard and an alert can be written against names that do not drift. Labels are closed and bounded: a tool, an outcome, a dependency, a refusal reason. A user, an agent, an organization or a sandbox is never a metric label.

Instrument	Kind	Labels
`mcp.calls`	counter	`tool`, `outcome` — declared, not written yet
`mcp.refusals`	counter	`reason` (`scope`, `item`, `attribution_failed`, `header_body_mismatch`, `access_check_failed`, `unknown_handle`, `sandbox_lost`, `runtimes_unavailable`, `rate_limited:user`, `rate_limited:agent`)
`mcp.workers`	gauge	`state=running`
`mcp.worker_start_seconds`	histogram	`provider=process`
`mcp.forwards`	counter	`outcome` (`ok`, `unreachable`) — cross-replica hops
`mcp.dependency.duration`	histogram	`dependency` (`iam`, `store`, `spacer`, `runtimes`, `contents`), `outcome`
`mcp.dependency.timeouts`	counter	`dependency`
`mcp.readiness.failures`	counter	`dependency`
`mcp.sandbox.lost`	counter	`provider`, `then` (`reported`, `relaunched`)

The durable.* instruments are written by the durable execution service under service.name datalayer-durable, and sandbox.launch_seconds by the Runtimes path; both are listed, with their alert rules, on the durable service page. This sentence used to say they belonged to a service that was not deployed, which stopped being true with the durable chart and was left standing.

Correlation

Spans are not emitted by the gateway yet, so none of the fields below can be queried today. They are the set the gateway will carry, and the set to write a dashboard or a support query against once it does — recorded here so a name is agreed before it is used:

Field	On	Meaning
`mcp.method` / `mcp.tool`	spans, and `mcp.calls`	The JSON-RPC method and the tool name
`mcp.binding.uid`	spans, logs	The handle the call resolved: `nb_…` a notebook, `sb_…` a sandbox session, `ts_…` a toolset
`mcp.task.id`	spans, logs	The run. A task outliving its request links its later spans to the initiating trace, so "show me this run" is a query by task id rather than by one trace id
`sandbox.uid` / `sandbox.provider`	spans	The Runtimes `pod_name` of the session's sandbox, and which provider holds it
`workflow.id` / `workflow.engine`	spans	The durable run behind a task
`evals.launch_uid` / `evals.run_uid` / `evals.case_id` / `evals.subject`	the `evals.task` span	One benchmark task a team worked, and the root of the trace its whole team tree runs in; the task result's `trace_id` is this trace. Written by durable, its page says more
`contents.source.uid`	spans	The content source a call reached
`user.uid`, `client.id`, `org.uid`	spans only	Identity, on access-controlled spans — never on a metric, where the label set must stay bounded

The client's traceparent (SEP-414, sent in _meta) is meant to be continued through gateway, worker, Runtimes and Contents; the gateway does not read it today. IAM already reads a request's traceparent into the trace_id of the audit rows it writes, so its security decisions can be joined to a run once the gateway carries the trace.

Dashboards and alerts

There are none for MCP yet: no dashboard on this service, no alert rules on the SLOs, and no PrometheusRule in the gateway's chart. A gateway outage is caught today by the ordinary Kubernetes deployment, ingress and certificate alerts. The SLIs the alerts will read — availability of POST /mcp, p95 call duration for the synchronous tools, task success rate, and p95 sandbox launch time per provider — need mcp.calls, mcp.call.duration and sandbox.launch_seconds, none of which is written yet.

Connecting from the Internet (Public Endpoints)

When the datalayer-otel service is deployed behind a public ingress (e.g. https://prod1.datalayer.run), external clients can send OTLP data and query the REST API over the internet.

Environment Variables (Public Internet)

When sending telemetry or querying the OTEL service from outside the cluster, use these environment variables:

Variable	Required	Default	Description
`DATALAYER_RUN_URL`	Yes	`https://prod1.datalayer.run`	Base URL of the Datalayer platform
`DATALAYER_API_KEY`	Yes	–	JWT or API key for authentication (used as Bearer token)
`DATALAYER_OTLP_URL`	No	`${DATALAYER_RUN_URL}/api/otel/v1/otlp`	OTLP/HTTP collector endpoint for sending signals
`DATALAYER_OTEL_URL`	No	`${DATALAYER_RUN_URL}`	REST API base URL for querying traces/metrics/logs
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	No	–	Standard OTEL SDK env var (set to `${DATALAYER_OTLP_URL}` for external use)
`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`	No	–	Standard OTEL SDK env var (set to `${DATALAYER_OTLP_URL}` for external use)
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`	No	–	Standard OTEL SDK env var (set to `${DATALAYER_OTLP_URL}` for external use)

Internal vs. Public Endpoints

Internal services (inside the cluster) use gRPC on port 4317 via the cluster-local service name. External clients use OTLP/HTTP via the public ingress — gRPC is not available over the public endpoint.

Sending OTLP Signals (OTLP/HTTP)

The OTEL Collector is exposed at https://<RUN_HOST>/api/otel/v1/otlp.
External clients use OTLP/HTTP (not gRPC) and authenticate with a Bearer token:

# Public OTLP endpoint for external / internet clients
export DATALAYER_RUN_URL="https://prod1.datalayer.run"
export DATALAYER_API_KEY="<your-jwt-or-api-key>"

# OTLP/HTTP endpoints (used by the core otel example generator)
export DATALAYER_OTLP_URL="${DATALAYER_RUN_URL}/api/otel/v1/otlp"
# Individual signal endpoints:
#   POST ${DATALAYER_OTLP_URL}/v1/traces
#   POST ${DATALAYER_OTLP_URL}/v1/logs
#   POST ${DATALAYER_OTLP_URL}/v1/metrics

Querying the REST API

export DATALAYER_RUN_URL="https://prod1.datalayer.run"
export DATALAYER_API_KEY="<your-jwt-or-api-key>"

# Query traces
curl -H "Authorization: Bearer ${DATALAYER_API_KEY}" \
  "${DATALAYER_RUN_URL}/api/otel/v1/traces/?limit=10"

# Query logs
curl -H "Authorization: Bearer ${DATALAYER_API_KEY}" \
  "${DATALAYER_RUN_URL}/api/otel/v1/logs/?limit=10"

# Query metrics
curl -H "Authorization: Bearer ${DATALAYER_API_KEY}" \
  "${DATALAYER_RUN_URL}/api/otel/v1/metrics/?limit=10"

# Run SQL on DataFusion
curl -X POST -H "Authorization: Bearer ${DATALAYER_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT service_name, COUNT(*) as cnt FROM spans GROUP BY service_name ORDER BY cnt DESC LIMIT 10"}' \
  "${DATALAYER_RUN_URL}/api/otel/v1/query/sql"

WebSocket (Real-Time Stream)

# Connect to the WebSocket stream (authenticates via query param)
wscat -c "wss://prod1.datalayer.run/api/otel/v1/ws?token=${DATALAYER_API_KEY}"

Core OTEL Example

The core otel example uses these environment variables to connect to a public deployment:

# Point the example at the public Datalayer platform
export DATALAYER_RUN_URL="https://prod1.datalayer.run"
export DATALAYER_API_KEY="<your-jwt-or-api-key>"

# Optional: override the OTLP target (defaults to ${DATALAYER_RUN_URL}/api/otel/v1/otlp)
# export DATALAYER_OTLP_URL="https://prod1.datalayer.run/api/otel/v1/otlp"

# Optional: override the OTEL REST query URL (defaults to DATALAYER_RUN_URL)
# export DATALAYER_OTEL_URL="https://prod1.datalayer.run"

# Start the example
cd examples/otel
uvicorn app.main:app --reload --port 8600

The generator (generator.py) resolves the OTLP endpoint in this order:

DATALAYER_OTLP_URL — explicit OTLP collector URL
DATALAYER_OTEL_RUN_URL or DATALAYER_RUN_URL + /api/otel/v1/otlp
https://prod1.datalayer.run/api/otel/v1/otlp — production fallback

The UI (vite.config.ts) resolves the REST + WebSocket URLs from DATALAYER_RUN_URL (default https://prod1.datalayer.run).

Smoke Test

You can verify the full pipeline end-to-end (send → Pulsar → DataFusion → query) using:

# Via the datalayer CLI (from datalayer-core)
datalayer otel smoke-test --url https://prod1.datalayer.run --token $DATALAYER_API_KEY

# Via the datalayer-otel CLI (from the otel service itself)
datalayer-otel smoke-test --url https://prod1.datalayer.run --token $DATALAYER_API_KEY

This sends test traces, metrics and logs, waits for ingestion, then queries them back and runs SQL queries on the DataFusion tables.

Deploy Datalayer OTEL​

API Endpoints​

Dashboards​

How a panel reads its series​

CLI​

Environment Variables​

FastAPI Query Service​

Authentication (via datalayer_common)​

Service Configuration​

DataFusion / Storage​

Apache Pulsar​

Logfire (optional)​

OTEL Collector​

Instrumented Services​

Service names​

MCP Server gateway and workers​

The MCP metric catalog​

Correlation​

Dashboards and alerts​

Connecting from the Internet (Public Endpoints)​

Environment Variables (Public Internet)​

Sending OTLP Signals (OTLP/HTTP)​

Querying the REST API​

WebSocket (Real-Time Stream)​

Core OTEL Example​

Smoke Test​