☰ 🛠️ Datalayer MCP Server

KubernetesREST APIMCP

The hosted MCP gateway: one endpoint — https://mcp.datalayer.run/mcp — that AI agents (Claude Code, Claude Desktop, Codex, Cursor, VS Code, Windsurf, Cline, any MCP client) connect to in order to read, edit and execute the Notebooks their user is allowed to reach, and to run code in Datalayer sandboxes.

The tool vocabulary comes from the open source jupyter-mcp-server; this service adds identity, tenancy, per-item permissions, rate limiting, session-to-sandbox affinity and cross-replica routing around it.

The gateway runs on the platform plane, in datalayer-api, beside IAM and Spacer. It owns no notebook, no sandbox and no content: every one of those is an authenticated call to the service that does own it, made with the caller's own exchanged token. It keeps only what a request must be able to find again on another replica — handles, the worker directory and rate windows — and it keeps those in Solr, not in the pod.

Processes and ports

Process	Purpose	Port
Gateway	The one container of the Deployment — see below for what it serves	4404 (PORT)
Worker	One jupyter-mcp-server process per user, started by the gateway on demand, reached over loopback only	an ephemeral loopback port per worker

The gateway container serves:

• the MCP transport on /mcp
• the RFC 9728 metadata
• the /api/mcp/* REST routes
• scope and item enforcement
• the rate limiter
• the worker directory
• cross-replica forwarding

Both run from the one image. The gateway is python -m datalayer_mcp_server.main (uvicorn, one worker process, proxy headers trusted because TLS is terminated at the ingress); a worker is python -m datalayer_mcp_server.worker_main start --transport streamable-http pointed at Spacer for documents and at Runtimes for sandboxes.

Why a process per user

The open source server keeps state per process: a singleton configuration, HTTP clients with a token baked into their session, the notebooks that have been used. All of that is correct for one person and none of it is safe to share, so the boundary used is the one the operating system already gives — a process. Workers are keyed by user, not by agent: two agents of the same person share a process, which is safe because what each agent may do was already decided in the gateway, by scope, before anything was proxied. Two people never share one.

A worker holds no credential of its own. The gateway forwards the caller's Authorization header on every request, and clears DATALAYER_API_KEY, DATALAYER_TOKEN, DOCUMENT_TOKEN, CODE_SANDBOX_TOKEN and JUPYTER_TOKEN out of the child's environment before putting back the caller's own exchanged token — so a credential configured on the gateway is never handed to a user's worker.

Workers are reclaimed after DATALAYER_MCP_WORKER_IDLE_TIMEOUT (default 900 s) and evicted least-recently-used when DATALAYER_MCP_MAX_WORKERS (default 50 per replica) is reached. Neither the reaper nor the eviction stops a worker whose session has a task working: the task store is asked first, and when it cannot answer the worker is kept.

Two modes, and which one you are running

This service is two things at once. It is an MCP server that answers a client, and it is the platform surface around one: Solr-backed bindings and tasks, a durable engine, a scheduler, an organization policy fetched from IAM on every call.

Every item in that second list is a dependency that can be missing, misconfigured or slow — and when one is, what a person sees is "Claude cannot connect".

DATALAYER_MCP_STANDALONE keeps the first and turns off the second.

	Standalone	Platform
Tools — notebooks, cells, sandboxes	unchanged	unchanged
Authentication and scopes	unchanged	unchanged
Bindings, tasks, rate windows	in this process	mcp-gateway, mcp-tasks
Audit	in this process	mcp-audit
Durable runs	the in-process fake (durable: false)	Durable
Periodic jobs (retention, alerts)	not scheduled	on a lease
Organization policy	not fetched	from IAM, cached, enforced
Replicas	one	as many as the HPA gives you

The tools are not what standalone turns off

A worker still starts, still opens notebooks through Spacer and still launches sandboxes through Runtimes, because those are the tools. What goes is everything that exists to make several replicas agree with each other, which one replica does not need.

It is not a lesser gateway. It is the same gateway with nothing behind it to go wrong — which is the right thing to deploy while that machinery is being brought up, and the right thing for anybody running one of these outside Datalayer's own cluster.

Standalone means one replica

Every record lives in the process that made it. A second pod would have its own idea of every binding and every task, and a session would work or not depending on which one answered. The chart ships replicaCount: 1 alongside the flag; raise it back to 2 in the same change that turns the flag off, never before.

Two things it deliberately does not touch. Authentication — standalone is about what is behind the gateway, never about who may reach it, and a flag that quietly stopped checking tokens would be a very bad thing to have on by default. And the audit, which is still written, to this process rather than to Solr: a record nobody keeps is not the same as one kept in one place.

The flag also wins over an explicit DATALAYER_MCP_STORE=solr. A deployment that asked for no platform machinery and still names Solr has contradicted itself, and honouring the narrower answer is what keeps the flag meaning "nothing behind this to go wrong".

Turning it off

Setting DATALAYER_MCP_STANDALONE: "false" does not on its own move the records out of the process. It removes the check that kept the deployment to one replica; DATALAYER_SOLR_ZK_HOST is what actually decides, and the chart ships it empty.

So the flag off, autoscaling.enabled back on, and that host still empty gives you two pods with separate bindings, tasks, rate windows and alert state. What a person then sees is use_notebook answered by one pod and read_cell by the other: a truthful "no such notebook" for a notebook that is open. It works about half the time, and every health check stays green throughout. Of all the ways this service can be misconfigured it is the only one nobody sees.

Three things say so, and none of them infers it from another:

• the chart refuses to render more than one replica whose records stay in the process, by the same rule the process follows;
• startup reports it wherever the chart is not — every local run — as a configuration problem naming both halves of the remedy. Not fatal: one replica on memory is a perfectly good gateway, and refusing to start would take that away;
• readyz answers records beside mode, because they are different questions and platform on memory is the interesting answer.

The change that turns the flag off

Set DATALAYER_SOLR_ZK_HOST in the same change as DATALAYER_MCP_STANDALONE: "false" and autoscaling.enabled: true. Then check readyz before sending traffic — "records": "memory" on a platform gateway means the pods do not share a session, whatever else the page says.

On a plane, that change is two lines in the plane's rc file rather than a helm invocation of your own: plane up already passes the Solr host, credentials and every service URL from the rc, and since 2026-09-05 it also passes the mode and the replica count, defaulting to what the chart did on its own.

export DATALAYER_MCP_STANDALONE=false   # default: true
export DATALAYER_MCP_REPLICAS=2         # default: 1
plane reup datalayer-mcp-server
curl -s https://r1.datalayer.run/api/mcp/readyz | jq '{mode, records, replica}'

records must read solr and two consecutive healthz answers should name two replicas. prod1 made this change on 2026-09-05: two replicas, records in Solr, the acceptance run's cross-replica checks passing — and the audit ledger, which had been in the process and gone with every restart, in mcp-audit from that day. The mcp-* collections must already exist (plane solr-init, choice 4) AI); the durable service is a separate matter — with DATALAYER_DURABLE_URL empty the platform gateway keeps the in-process fake and readyz says so under workflows, which is not a readiness failure but is not durability either.

helm upgrade ... --set mcpServer.env.DATALAYER_MCP_STANDALONE=false \
                 --set mcpServer.autoscaling.enabled=true

Error: DATALAYER_MCP_STANDALONE is off and the gateway would still keep its
records in the process: DATALAYER_MCP_STORE names memory, or it names nothing
and DATALAYER_SOLR_ZK_HOST is empty. [...]

Which one is running

curl -s https://r1.datalayer.run/api/mcp/readyz | jq '{mode, records}'
{
  "mode": "standalone",
  "records": "memory"
}

mode is what the deployment asked for; records is where the bindings, tasks, audit rows and alert state actually went. They agree in standalone, which always keeps them in the process. In platform mode they can differ, and that difference is the failure above.

Ask this first when a task is missing or a policy is not applying — for most of those questions standalone is the answer, and working it out from an empty list is how somebody spends an afternoon. The startup log says it too:

🎯 Standalone: the MCP endpoint and the tools, with no durable engine, no
scheduler, no organization policy and every record in this process.

Where it sits, and what it calls

Dependency	Plane	Why
IAM	platform	Issues and signs the tokens this gateway verifies locally; the RFC 8693 exchange that turns a gateway token into one the rest of the platform accepts. Gates readiness
Solr (mcp-gateway)	platform	Handles, the worker directory and rate windows. Gates readiness
Spacer	platform	Notebooks and the per-notebook permission check
Runtimes	runtimes	Sandboxes: launch, look up, terminate — always through the Runtimes API, never a Kubernetes call and never a provider SDK
Contents	runtimes	The per-source permission check, at /api/contents/v1/sources/{uid}/permissions
AI Agents	platform	Best-effort activity events, so what an agent does appears in the feed
OTEL	platform	Where telemetry is addressed. Never probed, never a readiness condition

Contents and Runtimes are cross-plane: they run on the runtimes plane and are reached at their configured public URLs (https://r1.datalayer.run in production), never at an assumed same-cluster service name. The gateway refuses to start on a malformed URL, and reports a missing one through readiness instead — a deployment that does without a dependency still serves the rest and says which tools will not work. DATALAYER_CONTENTS_URL has no fallback and is rejected when it names the IAM host.

IAM must be deployed with the same DATALAYER_MCP_SERVER_URL value used here: IAM stamps it into the audience of every OAuth token, and this service refuses a token naming anything else.

Routes and authentication boundaries

The Auth column is the boundary on who may call a route at all; Notes is what it returns or how it behaves once a caller is let through.

Surface	Route	Auth	Notes
MCP	POST /mcp	An OAuth 2.1 access token whose aud is this gateway, or a personal access token	Refused before the body is read when the scope does not cover the call
Protected-resource metadata	GET /.well-known/oauth-protected-resource, …/oauth-protected-resource/mcp	None (RFC 9728)	The document belongs to the resource, and it is where a 401 from /mcp sends a client
Server Card	GET /.well-known/mcp-server	None, deliberately	Read before a client authenticates — it is what lets a registry list this server and a client decide whether authenticating is worth it
Audit	GET /api/mcp/v1/audit	The caller's own rows; organization_security_auditor or owner for the organization's	Cursor-paged, with the retention window in retention_days
Activity	GET /api/mcp/v1/activity	The caller's own token	What the home page's Running now grid reads: each client with its last_call (tool, item, time, decision, outcome) and its sandbox count
Liveness	GET /api/mcp/healthz	None — the kubelet has no token	—
Readiness	GET /api/mcp/readyz	None	200 ready, 503 not; the body reports every dependency on its own line
Version	GET /api/mcp/version	None	—
Toolsets	GET /toolsets	The same token as /mcp	What this deployment declares and what a query string would activate — see Choosing the tools in the URL
Handles	GET /api/mcp/v1/bindings, DELETE /api/mcp/v1/bindings/{uid}	The caller's own token	A DELETE of an sb_… handle needs code:execute and terminates the sandbox through Runtimes
Notebook tasks	GET /api/mcp/v1/notebooks/{uid}/tasks	notebooks:read plus the item check	Reads mcp-tasks, so every replica answers the same
Tasks	GET /api/mcp/v1/tasks	notebooks:read, scoped to the caller	?org= widens it to the organization, decided by the same rule as the audit. Open tasks only — a finished one is the audit's
One task	GET /api/mcp/v1/tasks/{uid}, DELETE … to cancel, POST …/input to answer one waiting on a person, GET …/events for its changes as server-sent events	notebooks:read to read, code:execute to cancel or answer	A task that is not yours is 404 rather than 403: saying it exists but is not yours is saying it exists
Policy	GET /api/mcp/v1/policy	The caller's own effective policy, rule by rule, with the layer that decided each — platform, organization, team or personal	?org= widens it by the same rule as the audit; ?agent= reads it as one connected client rather than as the caller, which is what the console's Preview as and datalayer mcp policy --agent ask for
Alerts	GET /api/mcp/v1/alerts, POST …/{uid}/acknowledge	An organization's owners and security auditors, the same rule as the audit	Acknowledging is idempotent; somebody else's organization's alert is 404, because the uid of an alert is enough to know one fired
Audit forwarding	GET /api/mcp/v1/audit/forwarding	The same rule again	Says whether the audit is reaching the organization's own system of record — never the destination or the secret
Organization usage	GET /api/mcp/v1/organizations/{uid}/usage	The same roles as the overview	Spend, running sandboxes and the per-minute cap, each beside the limit in force, plus the day's credits by agent — see The two sandbox quotas and Which agent spent it
Organization overview	GET /api/mcp/v1/organizations/{uid}/overview	An organization owner or security auditor, ?team= to narrow	The enterprise console's first page: agents, spend, tasks and alerts for one organization
Worker directory	GET /api/mcp/v1/operations/workers	The platform_admin role, and nothing else	—
Sandbox ended	POST /api/mcp/v1/operations/runtime-ended	Runtimes' own key	How an external sandbox's loss reaches the gateway — see below
Workflow backend	GET /api/mcp/v1/operations/workflows	The platform_admin role	Says whether a run would survive this pod: durable: true is Durable, false is the in-process fallback
Periodic work	GET /api/mcp/v1/operations/jobs	The platform_admin role	Answers for the replica that happens to serve the request — see Periodic work before reading the numbers
OpenAPI	GET /api/mcp/v1/docs, /api/mcp/v1/openapi.json	None	—

On audit forwarding. Forwarding never fails the call it describes — holding up a tool call because a customer's SIEM is down would be the worse trade — so a failure is invisible unless something reports it, and a silently dropped audit record looks exactly like nothing having happened.

The audit-forwarding route above is what reports it, and the console's Audit page now renders it: a line when it is working, and the destination's own words when it is not. If you alert on one thing here, alert on healthy going false — the counters alone cannot tell a failing destination from a quiet week.

configured is not healthy

An organization that has set a destination and had nothing shipped to it yet reports configured: false with counters at zero and no error. That reads exactly like a working destination on a quiet week, which is why the two are separate fields rather than one — and for an organization that did set a destination, "never attempted" is the worse of the two states.

Why runtime-ended exists

A Datalayer runtime can be asked about — the gateway polls Runtimes and finds it gone. A Daytona, E2B or Modal sandbox that its provider reaped, whose credentials expired, or that idled out is known to Runtimes and to nobody else. Without this webhook the binding stays active and the next call fails somewhere far from the cause, with the agent believing its session is fine.

The sessions are marked lost, not closed: lost carries the reason and the resource state, and is what SANDBOX_LOST reports to an agent holding the handle. A session created with on_lost: relaunch gets its replacement on its next call rather than here — a lifecycle event is news, not an instruction to spend money.

The ingress publishes on r1.datalayer.run: the /mcp, /.well-known/oauth-protected-resource and /api/mcp prefixes, and the exact path /.well-known/mcp-server. Nothing else on the host is routed to this service.

A route the app serves and the ingress does not publish is a route that does not exist from outside, and it fails as a 404 that looks like a missing feature rather than a missing rule — the Server Card was served correctly and unreachable for exactly that reason. When adding a .well-known document, add the ingress path in the same change.

The card needs the chart re-applied

The /.well-known/mcp-server path was added to the ingress on 2026-08-28. A deployment created before that serves the card in-pod and answers 404 on the host until p reup datalayer-mcp-server applies the new rule. Check which you have by comparing the two:

curl -s -o /dev/null -w "%{http_code}\n" https://mcp.datalayer.run/.well-known/mcp-server
kubectl -n datalayer-api exec deploy/datalayer-mcp-server -- \
  curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:4404/.well-known/mcp-server

404 outside and 200 inside is the missing ingress rule, not a missing card.

The two credentials

An OAuth 2.1 access token is a user's authority narrowed to what they approved for one agent: it carries scope, client_id and an aud naming this gateway. A personal access token is the user acting as themselves: no aud, no scope, and therefore every scope the gateway knows — notebooks:read, notebooks:write, code:execute, sandboxes:manage, data:read, data:write, tools:use. Both are Datalayer JWTs signed with the shared secret, so verification is local and no round trip to IAM sits on the request path.

code:execute and sandboxes:manage are separate because they are different questions. The first is running code on a sandbox; the second is deciding which sandbox, keeping it, sharing it and ending it — each spending credits on a decision the agent made. launch_sandbox, use_sandbox, terminate_sandbox and snapshot_sandbox require it; list_sandboxes does not, since seeing what you have is not managing it; and the implicit launch a session does when it has none stays under code:execute, because an agent granted "run this" and given nowhere to run it would otherwise be refused for asking to do exactly what it was granted.

Deploy IAM before the gateway when this scope arrives

Every agent authorized before sandboxes:manage existed loses those three tools until it is re-authorized. The refusal says so and names the remedy. An agent re-authorizing against an IAM that cannot yet grant the scope gets a grant that still does not work, so IAM goes first.

A test holds the ends together — because a mapping typo otherwise fails as a refusal, and a refusal is what a correctly configured agent also gets.

Three lists have to agree, and only checking two is how this scope shipped unreachable:

List	Where	If a scope is missing
TOOL_POLICIES	gateway	The tool asks for nothing and anyone may call it
SCOPES_SUPPORTED	gateway	No client can request it. It is what the protected-resource metadata publishes, so it is the only thing a client reads to know what to ask for — the consent screen never shows it, no token carries it, and the tool is refused for everybody
RESOURCE_SCOPE_NAMES	IAM	The authorization server refuses the request

sandboxes:manage was in the first and the third and not the second, so launch_sandbox, use_sandbox and terminate_sandbox were unreachable for every agent — with a message that reads like the caller's own fault. Fixed in 99020fa4; the test now checks tools against what is advertised, in both directions.

A client's own URL may narrow what it gets: …/mcp?scopes=notebooks:read,code:execute.

A service agent key is the third, and the only one that is not a JWT. It authenticates an agent that is a principal rather than a person's proxy — see Service agents below. It looks like dla_sa_…, which is how the gateway tells it from a token before sending it anywhere: an access token forwarded to IAM as a key would be a token in a log line on a service that had no business seeing it. The scopes travel on the resource of the protected-resource document, and IAM reads them off it rather than trusting what the client asked for. The OpenID Connect scopes (openid, profile, email, offline_access) pass through so one sign-in answers both questions; none of them grants anything here.

A token bound to a key is not accepted as a bearer. RFC 9449 puts a cnf claim on a token to say only the holder of a named key may use it, and the proof travels in a DPoP header. Nothing here verifies such a proof yet, and until it does the safe reading of a bound token arriving as a plain bearer is a refusal, not "close enough" — accepting it would make a stolen bound token work exactly as well as a stolen unbound one, and the binding a promise the platform makes and does not keep. The refusal says the token is bound and what to do, rather than that it is not valid, which is true and useless because the holder retries the same token. An empty cnf is not a binding, and this platform mints no bound tokens, so the only holder of one got it elsewhere.

Because the aud names this gateway, the token cannot be forwarded — every other Datalayer API refuses it, which is the point. RFC 8693 token exchange is the way across: the gateway proves to IAM with DATALAYER_IAM_API_KEY that it is a Datalayer service and receives a token for another audience, same user, never more scope. Exchanged tokens are cached until shortly before they expire. Without that key, OAuth callers reach nothing.

Enforcement is layered and deny-by-default. enforcement.py reads Mcp-Method and Mcp-Name and refuses a call the rate limit, the scope or the organization's, team's or personal policy does not allow before the body is read — a call announced in headers skipped the policy until 2026-09-11 — then cross-validates the body against the headers; a tool absent from the policy table is treated as requiring the strongest scope. A caller that cannot be attributed to a user is refused (-32001), never served. The per-item read/write/execute decision is asked of Spacer or Contents on every call and is never cached.

JSON-RPC error	Meaning
-32000	Refused: scope, item permission, or an unknown handle
-32001	The call could not be attributed, or an access check could not be made — fail closed
-32002	Rate limited; the data carries retry_after_ms
-32010	SANDBOX_LOST, with the runtime's ResourceState
-32011	Runtimes could not be reached to launch or check a sandbox

Client registration is IAM's: clients are registered by Client ID Metadata Document — a client_id that is a URL IAM fetches, validates and caches, advertised as client_id_metadata_document_supported in the RFC 8414 metadata — with Dynamic Client Registration kept as the deprecated fallback. See IAM.

A desktop client's redirect port is not fixed

IAM matches a redirect URI exactly against the client's document — the whole string as the client wrote it, never a prefix, never a different scheme, path, query or fragment — except the port of a loopback address, where any port is accepted.

That exception is RFC 8252 §7.3 and it is not optional. Claude Code, Claude Desktop and every other native client listen on whatever port the operating system hands them at the moment they start the flow: http://localhost:3118/ callback today, something else tomorrow. No document can list those ports — the client does not know them when it is written.

Matching the port too refuses the client on its own callback:

{"error":"invalid_request",
 "error_description":"Redirect URI [http://localhost:3118/callback] is not registered for this client."}

which reads to whoever hit it as the server being broken. If you see that error, check whether the host and path match the document — those are still compared exactly, and 127.0.0.1 is not localhost as far as matching is concerned.

Sessions and sandboxes

What a person means by "my session" is the sandbox their code has been running in. That sandbox is the session, and it is an sb_… handle in mcp-gateway:

• one active sandbox per (user, client_id) unless a client holds several handles on purpose;
• every execution of the session resolves the binding first and targets its sandbox_uid, on the replica whose worker holds the connection;
• the Runtimes pod_name is derived from the binding uid, so a retried launch or two replicas racing cannot make two sandboxes for one session;
• a sandbox that is gone ends the call with SANDBOX_LOST and the runtime's ResourceState. Only a binding created with on_lost: relaunch gets a replacement, and it says so in the result. There is no silent relaunch;
• launch_sandbox on a session that already holds a sandbox answers that one — after confirming with Runtimes that it is still there, never off the record alone;
• a binding follows the runtime's reservation; closing it ends the session's open tasks.

GET /api/mcp/v1/bindings lists a caller's handles in every state, lost included — a session whose sandbox is gone is exactly what an operator and a user want to see, with why.

What a tool says about itself

A client reads a tool's annotations before it calls it, and decides two things from them: whether to ask the person first — a destructive tool usually earns a prompt — and whether a call that timed out is safe to send again, which only an idempotent one is. Left unset, the cautious defaults answer: not idempotent, open world. Those are the wrong answers for a listing, and until 2026-09-05 they were the answers for every tool this service adds to the worker: twenty-two tools with a title and at most one hint, so a client retried none of the reads and asked before list_notebooks.

Every tool of this service now answers all four — readOnlyHint, destructiveHint, idempotentHint, openWorldHint — and a test holds the rule over the tool list as it is actually served, not as the open source project ships it (tests/test_every_tool_says_what_it_is.py). The reads say read-only, idempotent, closed world; what keeps a version (snapshot_notebook, save_query_as_revision) says not idempotent, because a retry keeps one more; query_source says destructive and not idempotent, because a statement is whatever the caller wrote; and nothing this service adds says open world, since every one of them talks to one known deployment.

Two tools replace an open source one under its own name — use_notebook, so a notebook can be named as well as identified, and launch_sandbox, so an agent is not offered a dozen providers it does not choose between. They are held to a different rule: they say exactly what the original said, its annotations and its output schema both, taken from the original at the moment it is replaced (replacing.py). Written -> Any, the two had been the only tools on a worker that advertised no output at all, which the open source contract that every tool says what it returns caught the moment a checkout carried this service.

Borrowed toolset tools (below) say what their source said. Contents reads the annotations off each upstream tool into its manifest, in the wire names, and the gateway hands the four hints on under its own title — a borrowed tool is named for its source, so a title in the manifest is dropped, and so is anything that is not one of the four booleans. A source that says nothing leaves the cautious defaults to answer for its tool, which is the honest outcome: the gateway knows nothing the source did not say.

Choosing the tools in the URL

A connection to /mcp gets every tool this deployment serves — around fifty, whoever connected and whatever for. That is the right default and a poor answer for a client that came to do one thing: a model choosing between fifty tools chooses worse than one choosing between twelve, and each one costs context on every turn.

A client says what it came for in the URL it connects to:

URL	What it gets
https://mcp.datalayer.run/mcp	The default toolsets — everything, here
…/mcp?only=notebooks	The notebook tools and nothing else
…/mcp?only=spaces,notebooks	Two of them
…/mcp?spaces	The defaults and spaces, for a toolset that is off by default
…/mcp?without=sandboxes	The defaults, less one

Four toolsets, all on by default:

Toolset	Tools	What it holds
notebooks	15	The open source server's own: opening a notebook, reading and editing cells, running them. list_notebooks and use_notebook are the Datalayer-narrowed ones, whichever toolsets are picked
spaces	6	list_spaces, list_space_notebooks, find_notebook, list_notebook_versions, snapshot_notebook, restore_notebook
sandboxes	25	The lifecycle (launch_sandbox, use_sandbox, terminate_sandbox, snapshot_sandbox, share_sandbox, unshare_sandbox) and what rides with it: the Contents tools, the MCP-source bridge (list_toolsets, enable_toolset, disable_toolset), list_secrets, and the benchmark readers
library	3	search_library, get_library_artifact, read_library_artifact — Datalayer's published library, which answers without a credential

Three of the open source server's own tools are never served here: list_files, list_kernels and connect_to_jupyter assume a local Jupyter, and a gateway pointed at spaces has nothing for them to talk to. They are taken off each built server rather than left to fail — see What a tool says about itself.

GET /toolsets answers what there is and what a URL would activate, which is how a client is told the names rather than sent to read this page. It takes the same query string, so ?only=spaces says what that connection would get, and it is authenticated like /mcp:

curl -H "Authorization: Bearer $TOKEN" https://mcp.datalayer.run/toolsets

{"toolsets": [{"name": "sandboxes", "description": "Launch, use and terminate code sandboxes.", "default": true, "always": false, "active": true}, …],
 "active": ["library", "notebooks", "sandboxes", "spaces"], "unknown": []}

A name nobody declared comes back in unknown and is served anyway — a stale name in an agent's configuration should not take the connection down — and logged.

A toolset is a subject, not a package: sandboxes holds the open source lifecycle tools (launch_sandbox, use_sandbox, …) and what this service adds around them, because both distributions declare that name. Asking for it gives a set that can be used on its own.

Read once per connection, not per call. MCP is a session: a client lists the tools when it opens one and works from that list, so a server that changed its tools mid-session would have clients calling tools that are gone and never seeing the ones that are. The URL is the one place a client can say what it wants before the session exists.

A worker builds one server per selection and keeps it, so two clients asking for the same toolsets share one; the toolsets a connection asked for are what its tools/list answers and what its calls may reach. A tool left out of the selection is not merely hidden: it is not on that server, and calling it is "unknown tool".

Where the tools come from

The worker is the open source jupyter-mcp-server, and everything above it arrives as a plugin. Four distributions publish on the reactor.mcp.extensions entry-point group, and the worker loads whatever is installed:

Distribution	Publishes	Offers
jupyter-mcp-server	jupyter-mcp-server (built in)	The notebooks toolset — its own 18 tools, offered as contributions since 2.2.1
jupyter-mcp-sandboxes	sandboxes	The four lifecycle tools, in the sandboxes toolset since 0.2.5
datalayer_mcp_server (this service)	spaces, sandboxes-datalayer, library	Everything Datalayer-specific, and the narrowings of use_notebook, list_notebooks and launch_sandbox

Three operational consequences:

• a toolset is added by installing a distribution, not by editing the gateway — but the image pins the floors, so a new toolset is an image build;
• a narrowing is declared by name, so use_notebook taking a notebook's name rather than a path survives whatever order the entry points load in. Before 2.2, it depended on how the names happened to sort;
• a tool that cannot work here is taken off the built server, which is why list_files is absent rather than failing.

The mechanism is documented upstream in Extensions.

note

Two different things are called a toolset here. The ones above are named sets of this deployment's own tools, picked in the URL. The ones below are a Contents source's tools, borrowed for a session by calling enable_toolset — a different mechanism, decided by Contents rather than by the URL, and arriving as <source>__<tool> after the session has started.

Toolsets and Contents

Two families of tools reach data, and both call the Contents service with the caller's own exchanged token. Contents decides; the gateway owns none of it.

Toolsets are a source's own tools, borrowed. list_toolsets shows which sources lend any; enable_toolset mints a Contents MCP session — which is what scopes what may be called and where results may be written — and registers the source's tools as <source>__<tool> with the source's own schema, then says the tool list changed. A call goes to /mcp-sessions/{uid}/calls with an idempotency key derived from what the call is, so a retry after a timeout is answered the first result rather than run again. pending-approval comes back as input_required naming the approval a person has to decide.

Contents tools are the catalogue, attachments, queries and revisions. Three rules worth knowing:

• No tool returns bulk bytes. A query answers its uid, its state and datalayer://queries/{uid}/results. The rows are an Arrow stream a client fetches when it wants them.
• An attachment belongs to the session's sandbox, keyed on the sandbox's uid. A session with no sandbox is told to launch one; a Contents tool never launches one itself, because attaching is not executing and a sandbox nobody asked for is a sandbox somebody is billed for.
• Secrets are named, never read. list_secrets answers names, and no tool here accepts or returns a value.

Cancelling the task that is waiting for a query cancels the query too: the tool registers the Contents cancel as the task's interrupt. Without it the Datasource keeps executing while the task says cancelled.

Scope	What it buys
data:read	The catalogue, a source, a query's state, a secret's name, and a borrowed tool's call
data:write	Attaching, detaching, saving a revision — and enable_toolset, which is where writing artifacts into your storage is consented to, once
tools:use	Borrowing a source's tools at all

Bulk data is named, never carried

Three resources say where data is rather than handing it over, so a large result costs an agent's context nothing:

Resource	What it answers
datalayer://queries/{uid}/results	A Flight ticket where Contents mints one, and the HTTPS Arrow stream beside it — the same bytes for a client that cannot carry gRPC
datalayer://objects/{uid}	What an object is, its versions, and the URL its bytes are at
datalayer://transfers/{uid}	A transfer's progress

All three cost data:read and are cached privately: two callers get different answers, and a shared cache would hand somebody else's data to whoever asked next. The bytes themselves are fetched from Contents by the client, with its own credential, and are never proxied through the gateway.

The library, which anyone may read

Three tools search what Datalayer has published — search_library, get_library_artifact, read_library_artifact — against DATALAYER_LIBRARY_URL, which defaults to the IAM host because that is where the library is served.

They work without a credential, and that is the design. The library's public routes answer an anonymous browser, so an agent asked to "find a notebook that plots a parabola" can answer it. When the request does carry a Datalayer identity it is exchanged and passed on, and the only difference is whether results say the caller orbits them. A caller whose exchange fails searches anonymously rather than being refused — a toolset that required a token would be refusing the thing it exists for.

Consequences worth knowing before you read the audit:

• A library call is priced at notebooks:read, the weakest read scope, rather than left to deny-by-default. Charging code:execute to search a public catalogue would refuse a read-only agent the one thing it is certainly allowed to do.
• Nothing here writes. Publishing is a decision a person makes about their own work, in the web application, in a dialog that shows them what will appear where. All three tools are annotated read-only, and a test holds that over the served list.
• Every result carries the artifact's public URL, built by the library from DATALAYER_CDN_URL. If that is unset there, results arrive without a url and an agent answers with uids a person cannot open — the symptom appears here and the cause is in the library's configuration.
• A kind the library does not hold is refused by name, so an agent that asked for "spreadsheets" is told what there is instead of silently being given everything.

Spaces, and finding a notebook

A person's notebooks live in spaces, and an agent that has to open one walks from the space to the notebook. Five tools cover it, and which one to reach for is a matter of how much is being asked for:

Tool	Answers	When
list_spaces	Every space the caller reaches: uid, name, handle, and how many notebooks it holds	First, when nothing is known
list_space_notebooks	One space's notebooks, and the space it resolved to	When the space is known — its uid, handle or name
list_notebooks	Every notebook of every space, each carrying space and space_uid	When the space is not known, and the account is small enough for one answer
find_notebook	Which notebook a name means, or the candidates when several could	When a person said a name out loud
use_notebook	Opens one, by name or by uid	Once the notebook is decided

The uid is what travels. A space's uid from list_spaces is what list_space_notebooks takes, and every notebook carries the space_uid of the space it is in — a name is not an identity, and two spaces may carry one name. A name or a handle is accepted too: it is resolved exactly on the uid, then the handle, then the name, and a word that matches several spaces comes back as candidates rather than a guess, the way find_notebook answers for notebooks.

Why one space at a time matters. list_notebooks answers for every space at once, which is the right answer for an account with one space and a poor one for an account with many: a client that caps what it shows silently drops the tail, and the notebook that was asked for is the one that did not arrive. Scoping the listing to a space is how that is avoided, and the reason the space-scoped tool exists.

A sandbox has two names

A sandbox is identified by a ULID — that is its sandbox_uid, and it is what Contents validates and keys attachments on. Its Pod name is a deployment detail: the Operator names a runtime runtime-<ulid>, and this gateway derives sb-<ulid> from a session handle (sb_<ULID>) because a Pod name has to be a DNS label. An external sandbox is external-<provider>-sb-<ulid>.

Until 2026-09-04 the gateway stored the Pod name in a field called sandbox_uid and handed it on — to _meta, to the task projection, to the durable engine and through Runtimes to Contents, which refused it. The binding stores runtime_name now and derives the uid from its own handle, so there is one stored fact and no second copy to disagree with it. Every Runtimes call takes the name; _meta carries both. datalayer_common.runtime_names.sandbox_uid_of is the one converter that knows every name the platform mints, and answers "" rather than a name it cannot convert.

The first execution binds the session

A session's first execute_code, execute_cell or insert_execute_code_cell with no sandbox bound gets the deployment's default, launched by the gateway itself — reserved under the session's identity, counted against the budget, created through Runtimes and bound with what the runtime said it can do — exactly as launch_sandbox with no arguments would. A quota refusal there is answered as BUDGET_EXCEEDED, not as a failed call.

On prod1 it did not launch anything until 2026-09-05. The gateway's own Runtimes client sent environment_name; the operator's CreateRuntimeRequest takes environment: {name} — an object, not the bare name — and requires a credits_limit, so every first execution was refused 422, a runtimes_unavailable to the agent, which reads as an outage. Three fixes in a row, each found by the next 422: the field's name, its shape, and the credits limit, which the gateway now computes the way the worker's own launcher does — the environment's burning_rate × 60 × DATALAYER_SANDBOX_RESERVATION_MINUTES (default 10), read off the caller's environment catalogue, so the two launch paths reserve the same. A limit the caller set is kept; an environment the catalogue cannot name sets none, and the operator's refusal then says what is missing rather than the gateway inventing a number. The one-process gate (test_the_default_sandbox_gate.py) could see none of this: its Runtimes was a double that accepted whatever it was sent. use_sandbox and restart_notebook launch nothing: one selects a sandbox, the other has nothing to restart.

Every execution the gateway forwards names the session's sandbox in the request's _meta under io.datalayer/sandbox. The worker's before-call hook makes that sandbox its active one, attaching by name when this worker never launched it — so execution lands on the session's sandbox on whichever replica serves the call, and a shared sandbox is reached the same way by its grantee's worker.

Sharing a sandbox

A sandbox can be shared with other people, teams, organizations and service agents, at one of three nested levels: view reads outputs, update may change files, execute may run code. The record and the decision live on Runtimes: share_sandbox and unshare_sandbox are worker-side tools that carry the caller's own credential to PUT /runtimes/{name}/sharing, and Runtimes settles who owns a runtime by asking the operator as the caller. Both cost sandboxes:manage: who else may use a sandbox is its owner's decision, and an agent granted "run this" may not hand it around.

Two names, and the route takes the second. A sandbox has the name it was launched under — what every tool takes — and the runtime's own uid, which is what Runtimes' routes take. Until 2026-09-05 share_sandbox sent the first to a route that wanted the second: Runtimes asked the operator for a runtime called shared-drill, found none, and the owner was told only the owner may share their own sandbox. The worker translates the alias through its manager now, and a name it does not know is passed through, so an owner may name the runtime directly. Underneath, code-sandboxes' Datalayer sandbox kept the uuid it drew at construction as its id after the runtime existed; it takes the runtime's uid at launch now, which reaches a worker with the next code-sandboxes release.

From the console, as well as from an agent. The bound-sandboxes panel of the MCP dashboard has a Share action beside Terminate, and the sandbox details dialog a Manage sharing button; both open the same access dialog that content uses, against GET|PUT /runtimes/{name}/sharing directly. Runtimes answers an access document in exactly the shape the dialog reads, so there is no translation between them and nothing to keep in step.

The name is the runtime's here too, and sandboxSharingUrl is where that is written down for both surfaces. It answers nothing for a sandbox with no runtime behind it — a browser or local kernel, or a binding still reserving — so those are offered no button rather than a button that always fails. A person who is not the owner gets Runtimes' 403, and the dialog says so rather than drawing an empty list of grants, which would read as a sandbox shared with nobody.

Sharing works, and it took three fixes to get there.

Runtimes wrote the runtime's expiry straight into a Solr date field, and the operator answers epoch seconds as a string — '1788671641.000'. A date field that is not a date makes Solr refuse the whole document, so every share answered 500 Database Request Error for a user, a team, an organization and an agent alike: the field at fault was the one that never varied. Every test of the document builder called it without an expiry, which is how the value that reaches Solr in production never reached it in a test.

A share naming nobody is refused, and that is live. share_sandbox with no principals used to PUT what was already stored and answer "sharing updated. They can use it with use_sandbox" — so a typo'd parameter, an empty list or a uid that came back blank all read as a share that happened, and the owner found out only when the grantee was refused. An unshare naming nobody keeps its meaning: revoke everybody at that level.

The image needs agent-runtimes 1.2 — released on 6 September 2026

agent-runtimes 1.1.0 pinned opentelemetry-semantic-conventions<0.62, and opentelemetry-sdk 1.44.0 requires exactly 0.65b0. So the OpenTelemetry 1.40 → 1.44 bump could not be installed beside it and the image would not build with both. 1.2.0 lifts the pin, and the running pod carries agent-runtimes 1.2.0, opentelemetry-sdk 1.44.0 and opentelemetry-semantic-conventions 0.65b0.

Kept because of how it failed, which is the part that costs an afternoon: pip reported a conflict about mcp[cli] and fastmcp-slim, which is where it gave up rather than what was wrong. Resolve those two in a clean environment and they install together fine; add the OpenTelemetry constraint and the real conflict names itself. An old agent-runtimes in the build environment brings it straight back.

A revoked share is not a lost sandbox

After unshare_sandbox, a grantee's next call used to be refused "the sandbox is gone: absent — its variables and files are lost". It was not gone. The owner had stopped sharing it, and those are two different truths with two different things to do about them: one says the work is lost, the other says ask for the share again.

Two causes. The liveness check ran before the share check, and a grantee cannot see the runtime at all once the share is taken back, because Runtimes answers a runtime to its owner and nobody else. And swapping them was not enough: the share answer is believed for thirty seconds, so a share revoked a moment ago still read as held and the call fell through to the liveness check anyway.

So a grantee told absent has its share re-read without the cache. Absent from a non-owner is not evidence of absence. A share that still holds leaves the loss standing, and a binding the caller owns never takes this path.

Walked on prod1 and r1 on 2026-09-06. An owner's first execution gets the default sandbox; the share with a service agent is stored; Runtimes answers that agent view/update/execute and names the owner; the agent attaches by name and runs code; its binding carries shared_from and on_lost: fail; and the call after unshare_sandbox is refused "no longer shared with you".

A shared sandbox is genuinely shared. The owner sets a variable and the grantee reads it back. Before execution followed the session's sandbox the grantee got a NameError, which looked like a decision about namespaces and was nothing of the kind: its code was running on another pod of the pool.

A grantee calls use_sandbox with the sandbox's runtime name, or with the owner's handle. The gateway asks Runtimes what the caller may do, and a view is the least that makes a binding — one of the grantee's own, naming the owner in shared_from, never on_lost: relaunch (a relaunch would be a sandbox of the grantee's own). The worker attaches to the runtime by name with the grantee's token, which Runtimes now answers for a grantee.

The grant is checked on every call. Each tool's access — read, write, execute from its policy — needs the matching level, and the answer is believed for thirty seconds at most. A tool the level does not cover is refused SANDBOX_ACCESS_REVOKED with reason: insufficient; a grant that is gone altogether is reason: revoked, the binding is closed, and the next call is refused without asking again. Both name the owner and say launch_sandbox is the way to a sandbox of one's own. Counted as access_revoked in mcp.refusals.

Letting go is never refused. use_sandbox with no name drops the session's selection and goes back to Jupyter kernels. It names no sandbox, so neither the grant nor the sandbox's liveness is checked: a caller whose share has been taken back, or whose sandbox is gone, is exactly the caller who most needs that call to work. Naming a sandbox is a selection and is checked as one.

That was worth fixing rather than documenting around. Until 2026-09-08 the release was checked like any other selection, so a session bound to a revoked sandbox had no way out: the release answered the revocation error, and launch_sandbox — the remedy the revocation message itself names — answered "this session already has a live sandbox", because the close that is supposed to free the session was being written with a call that could not work and whose TypeError went to a debug line. Only terminate_sandbox followed by launch_sandbox with replace: true recovered such a session, which a grantee cannot do on somebody else's sandbox at all. If you see a session stuck between those two answers, it is running an image from before that date.

The same answer had a second way of being wrong, found on 2026-09-09 with a runtime deleted straight at Runtimes: the binding's record still said open, "live" was read off the record, and launch_sandbox answered the handle of a sandbox that no longer existed — nothing launched, nothing raised — while the use_sandbox that followed found it gone and said "call launch_sandbox". From gateway 0.0.8 the existing sandbox is confirmed against Runtimes before a launch answers it: gone is marked lost and the launch goes ahead; a binding created with on_lost: relaunch is replaced and the launch is answered the replacement; and replace: true does not relaunch a gone one only to close it.

Found on the same drill, one call later: use_sandbox naming the alias of the sandbox the session had just launched was taken for a share — the "already adopted" check knew only shared bindings by runtime name — and a second binding was made on the same runtime, with on_lost: fail, so the on_lost: relaunch the launch had asked for was gone from the session's current binding without anybody saying so. From gateway 0.0.9 a session's own live binding, by alias or runtime name, is what a use_sandbox selects; adoption is only ever of somebody else's sandbox. A session that shows two live bindings on one runtime is running an image from before that.

And one call after that, the replacement itself: on_lost: relaunch launches from the binding's launch_spec, and a sandbox launched over the wire had its environment recorded there under a key the relaunch does not read — so Runtimes refused every such relaunch with 422 body.environment: Field required, and no sandbox launched through launch_sandbox had ever actually been relaunched. From gateway 0.0.10 the spec has one vocabulary, environment_name. A binding created before 0.0.10 still carries the old key: its relaunch fails the same way, the session is answered SANDBOX_LOST with that refusal, and launch_sandbox is the way on — nothing reads the old key.

A subscriber is told, or nobody is

An agent that has opened a notebook can ask to hear when it changes — resources/subscribe on notebook://<handle> at 2025-11-25, a subscriptions/listen filter at 2026-07-28 — and the worker holds a persistent connection to the collaborative document so that a person's edit in JupyterLab is announced too, not only the agent's own.

None of it was delivered until 2026-09-05, and the server reported that it had been. The registry of subscribed sessions was weak-keyed so that a session going away would take its subscriptions with it; nothing else holds an SDK ServerSession, so every subscription was collected between the request that made it and the next call. publish then returned True anyway, because the 2026-07-28 bus had accepted the event — and that bus knows nothing of the 2025-11-25 client waiting on its stream. Measured against prod1 first (subscribe accepted, listen stream open, nothing ever arriving), then reproduced in one process against a raw client, where the verdict is on the wire rather than in a client library.

Three things were wrong, and all three are fixed in the open source server: the registry holds its sessions strongly, with removal said out loud — on unsubscribe, on a send that fails, and past a ceiling of 256 sessions, oldest first; the legacy half runs whether or not a modern bus exists, where before an early return skipped it entirely on any SDK without subscriptions/listen; and a subscriber that cannot be told is logged rather than dropped in silence. The test that holds it (tests/test_a_subscriber_ really_hears.py) runs a real server over real TCP, reads the wire, and deliberately keeps no reference of its own to the session — a test that keeps one passes against the broken code, which is how this stayed broken while its unit tests were green.

Measured on prod1 with 2.1.6, 2026-09-05. A 2025-11-25 client opens a notebook through the gateway, subscribes to notebook://<handle> and holds its standalone GET stream; a person editing the same document through the Spacer collaboration socket reaches it in 0.97 s, and the agent's own edit in 1.83 s. Read both off the raw wire and through an SDK client, and against a pod that was asked what it was running first.

The notification names the cell

resources/updated carries notebook://<handle>/cells/<id> as well as notebook://<handle>, so an agent watching one cell of a hundred-cell notebook refetches one cell.

As well as, never instead of. A client subscribed to the notebook asked about the notebook, and a deleted cell names nobody — its id went with it, and there is nothing left to read. The notebook frame is what every subscriber can rely on; the cell frames are an addition on top of it.

Both halves say which cell. A writing tool already resolves an index to a cell id and attaches it to its result, for the agent's benefit; the publisher reads it back off that result rather than off the tool's arguments, because an argument may be an index and an index is not something anybody can subscribe to. The watcher reads it off the pycrdt events: a change inside a cell arrives with the cell's index at the head of its path, and an inserted cell — which is how a moved cell appears — carries its own id in the delta. Inserting a cell names no cell, deliberately: nobody was subscribed to a cell that did not exist.

One edit is one notification again. It used to be two: the tool published on its way out, and the same edit then arrived back over the watcher's persistent connection carrying no origin, looking like somebody else's, and was announced again. Naming the cell would have made that two identical cell frames. So a watched notebook's tools hand their news to the watcher's own debounce instead of publishing it, and the tool's edit and the watcher's sight of it collapse into one frame — which picks up a person's concurrent edit in the same window for free. An unwatched notebook is unaffected: its tools publish immediately, because nothing else is going to.

The news never takes the edit with it. The watchers are one object for the process and outlive the loop they were started on; folding onto a closed loop raises, and until it was caught it raised inside the result wrapper — every writing tool answering Event loop is closed after doing its work perfectly well. An agent told its edit failed when it had not makes the edit again. So fold refuses a watch whose loop is closed or is not the one running now, and the announcement is wrapped besides: the edit is done and the answer is in hand by the time any of it runs.

Worth saying because the unit tests were green through all of it. What found it was the integration suite against a real Jupyter server, and what made it findable was comparing the failures against the same suite at HEAD — a suite with twenty pre-existing failures tells you nothing until you know which twenty.

Each URI of a burst is published as its own attempt. Inside one try, a frame that failed took the rest of the burst with it and made a publish that had reached somebody answer that nobody was told — and the caller counts that answer, reading False as "nothing is listening".

A debounce must not decide anything by reading a clock. Two Windows CI jobs failed on a suite green everywhere else, and the cause is not really Windows: asyncio runs a timer once the loop's time is within one clock resolution of its deadline, and on Windows that resolution is about 16 ms — so a 50 ms debounce fires with its own deadline still in the future. The check for "a later timer has taken over this burst" read that deadline, decided the firing was premature and declined, and the notification never arrived. It is a generation counter now: arming advances it, a firing carries the number it was armed with, and only the current one announces. The same race exists on Linux; a finer resolution merely makes it rare.

The debounce has a ceiling (MAX_WAIT_SECONDS, 2 s) now that the server's own edits feed it. A debounce with no ceiling is starvation waiting for a fast enough typist: every keystroke pushes the timer back and a subscriber hears nothing until the typing stops, which in a notebook two agents are working in may be never.

Measured on prod1 with 2.1.7, 2026-09-05. A 2025-11-25 client subscribes to the Welcome notebook and to two of its cells by id, and overwrites one: off the raw wire, notebook://welcome and notebook://welcome/cells/<id>, one frame each, nothing for the other subscribed cell. Then a person edits that same cell through the Spacer collaboration socket, and the subscriber hears the notebook and that cell in 0.35 s — a frame naming the cell somebody typed in, from a change this server did not make.

The image asserts watchers.fold at build time, beside the assertions for the extensions and the sandbox client. A wheel too old to fold now fails the build rather than deploying into a silence nobody would look for.

Where the cell ids come from

Not from read_notebook, which answers a text summary. A client is handed a cell id on read_cell's own _meta, under io.jupyter-mcp/cell_id, put there by the resolver that turned an index into a cell.

An open source fix is not deployed by deploying

The image builds this service from source — the build context is k8s/services, so pip install ./mcp-server is the gateway package — and takes the open source jupyter_mcp_server the worker runs from PyPI, through the dependency jupyter-mcp-server>=2.2.1. So a fix in the open source repository is invisible here until that wheel is released, and a redeploy will look exactly like a fix that did not work. It cost half an hour on 2026-09-05: the subscription fix passed in process, the walk against prod1 failed identically afterwards, and the ingress and the proxy were both exonerated before the pod was asked what it was actually running.

kubectl exec -n datalayer-api deploy/datalayer-mcp-server -- \
  python -c "import importlib.metadata as m; print(m.version('jupyter-mcp-server'))"

Ask the pod that before believing an open source change is live.

A disconnected agent stops within seconds, not within the hour

Disconnecting an agent revokes its grant, and the refresh token stops working at once. The access token it already holds was another matter: this gateway verifies the signature, the issuer, the expiry and the audience, and asked nobody whether the consent behind the token still stood — so every call that token made went through until it expired, up to an hour. The disconnect route says as much in its own docstring, so the trade was deliberate; it is not what a person means by disconnect.

IAM now stamps grant_uid on the tokens it mints from a grant and answers, to platform services only, whether that grant is live. The gateway asks before it judges a call and remembers the answer for ten seconds (LIVENESS_TTL_SECONDS), so the question costs a round trip occasionally rather than on every call. The honest claim is within seconds, not "on the very next request": the alternative buys a fraction of a second and spends a round trip on every call in the platform.

A refusal is grant_revoked in mcp.refusals, and the agent is told the connection was disconnected and to authenticate again.

The card goes too. GET /api/mcp/v1/activity lists the clients that called today from the audit rows, and a disconnected agent had called today — so its card outlived the disconnection until midnight, saying the opposite of what the person had just done. The route now asks IAM, with the caller's own token, which of their agents are still connected (GET /api/iam/v1/oauth/connected-agents, the Settings page's own route) and drops a delegated client that is not among them; the answer is cached with the rest of the panel, five seconds. A personal access token and a service agent are never on that list and are never dropped by it, and an IAM that does not answer lists everyone — an outage must not read as "all disconnected". The agent's sandboxes stay in the sessions panel until they are reaped: a disconnection ends the agent's authority, not the runtime.

Two things it deliberately does not do:

• A token that names no grant is never asked about. A personal access token and a service-agent key are not grants and were never in question.
• An IAM that cannot be reached does not refuse. A token good in every other way is stopped only when IAM says the grant is gone. An outage that logged every agent out would be a worse failure than the hour this shortens, and it is the behaviour that was in place before the question existed. A failure is not remembered as an answer either, so one blip cannot let a revoked agent run for the whole window.

External sandboxes

A sandbox at Daytona, E2B or Modal is bound like any other: the same handle, the same SANDBOX_LOST, the same quota. Two things differ.

Transport. Runtimes reaches an external sandbox either through the provider's HTTPS and WebSocket ingress to a real Jupyter server (provider-ingress), or through the provider's own execution API (direct). A direct sandbox has no Jupyter server behind it, so every tool that reaches for one — files, kernels, notebook cells — has nothing to reach. The gateway keeps the runtime's latest word on a live binding, the transport among it, and refuses those tools with transport_direct, naming execute_code as what works and launch_sandbox as the way to a sandbox with a Jupyter server. Counted in mcp.refusals.

Cancelling a running cell. A task's execute step runs the code in a thread, which nothing outside that thread can stop, so stopping the wait for a cell is not stopping the cell. tasks/cancel therefore calls the interrupt the tool registered before it cancels the handle: the sandbox's own, which for a Datalayer runtime reaches the kernel as POST /api/kernels/{id}/interrupt on the Jupyter server that runtime is. A ten-minute cell a person had already given up on used to keep computing until it finished on its own.

Measured on prod1 on 2026-09-07, a cell looping for two minutes and cancelled ten seconds in: the next execute_code on that session answers in 0.6 seconds, where it took 60.8 before — a free kernel answers in about a second, and anything longer is the cancelled cell still running. Worth stating as a measurement rather than a description, because the task said cancelled in both cases: the status was never the thing that told you whether the work had stopped.

The cancel also waits a bounded two seconds for the interrupted tool to hand back what the cell printed, and only then cancels it. Without that the tool was killed before its own completion path could run, so a cancelled cell answered with no outputs at all — in the task and in the notebook — though it had been printing for twenty seconds. Re-measured after the fix: eleven lines and the KeyboardInterrupt in both, and the cancel itself returning in 1.4 seconds. A tool that registered no interrupt, or a provider that answered that it could not deliver one, is cancelled at once rather than waited on.

When the cell is run through the runtime's HTTP /execute route instead of over the kernel WebSocket — the durable path a worker in MCP_SERVER mode takes so the outputs survive its own death — the cancel reaches it the same way: a DELETE /api/kernels/{id}/requests/{request_id} on the runtime (jupyter-server-nbmodel ≥ 0.2.9), which interrupts the cell and leaves the kernel alive for the next one. It is best effort: a runtime on an older image without the route answers 404, the run continues as it did before, and the worker only stops waiting — so the behaviour degrades to exactly the pre-0.2.9 one rather than failing. Requires the runtime image to carry jupyter-server-nbmodel ≥ 0.2.9 and the gateway to run jupyter-mcp-server ≥ 2.1.13.

Under the durable engine (milestone 2) the same interrupt is driven from the other side: the worker asks the engine every two seconds whether the run was cancelled — from any replica, since the engine's record is shared — and interrupts the same way the moment it was.

The follower, and why it never ran

Every execution the gateway forwards names the session's sandbox in the request's _meta, and a hook in the worker makes that sandbox the active one — attaching by name when this worker never launched it, which is what lets an execution land on whichever replica serves it.

It asked for an attribute that does not exist. The hook read manager off the sandboxes extension; the extension publishes its registry as sandboxes, a property it exposes precisely so nothing downstream reaches for _manager. So getattr answered None and the hook returned without doing anything, on every execution, on every deployment.

What that looked like on prod1 on 2026-09-05: the gateway launched a session's default sandbox and the binding came back active, and print(6*7) answered no output and no error. A tool that fails silently cannot be told apart from a program that printed nothing.

Neither package's tests could see it. The extension's do not know this gateway exists; this gateway's faked the manager and never asked where a real one comes from. The test that holds it now reads the attribute name off the real extension class, so a rename on either side fails rather than returning quietly. Re-walked after the fix, print(6*7) answers 42.

A second defect sat behind it, in code-sandboxes: DatalayerSandbox._adopt marked a sandbox started because the sandbox is running, and left agent_runtimes with no kernel client, so the first execution answered Kernel client is not started. from_id connects now; list_all does not, because listing thirty runtimes must not open thirty kernel connections, and run_code connects on demand. That half reaches the image with the next code-sandboxes release.

code-sandboxes 1.4.3 carries that half, and the image asserts it at build time beside the other floors: a wheel too old to connect to a runtime it adopted fails the build rather than deploying into silence.

The listing agrees with the session

list_sandboxes answers about the worker's registry — what that process launched. The session's sandbox is the gateway's record, and the two differ whenever the gateway launched it or another replica did. So a session whose execute_code had just run was told by list_sandboxes that it had no sandbox, while launch_sandbox said it already had one.

The gateway completes the answer now: the session's sandbox is added to what the worker knew, never substituted for it, because the worker's entries carry which notebooks are attached and dropping those would trade one wrong answer for another.

Three places, and the first two looked right

The same fact had to be wired through the decoration, the lookup, and _wanted — the gate deciding whether this layer looks at an answer at all, where a False forwards the body byte for byte. The lookup reached for self.sessions on the rewriter rather than on the layer it was handed, and the except beneath it swallowed the AttributeError. Each of the first two passed its unit tests and answered an unchanged listing on prod1. All three are held by tests that run the thing rather than read its source.

Walked on prod1 on 2026-09-05, with every binding cleared first. A session that has never had a sandbox: list_sandboxes answers [], print(6*7) answers 42, list_sandboxes then names the session's sandbox as active and running, and launch_sandbox says the session already has one. Three tools, one answer, and state kept across calls.

A sandbox that is finished is not offered either. The worker calls every entry in its registry running without asking anyone, and never forgets one, so a terminated sandbox stayed in the listing as active: true — from the tool an agent is told to consult for the names you may use. The gateway takes out what it positively knows is finished, and only that: an entry matching no binding of the caller's is left alone, because a worker may have launched it for itself and dropping on ignorance trades a stale row for a missing one.

Who writes the outputs

The runtime image carries jupyter-server-nbmodel, enabled, and its /api/kernels/<id>/execute route is served: read from inside a deployed runtime on 2026-09-05, POST there answers 202 Accepted.

Nothing goes through it yet. The worker drives the kernel itself over a websocket and writes the outputs into the document, so the outputs stop when the worker stops. Executing through the runtime's own route would put the writing on the far side of that loss, which is what "a task survives the process driving it, with the outputs the runtime already wrote still in the notebook" asks for.

Probing that route

The kernel id in the path must match Jupyter's own uuid pattern. With a made-up id Tornado never reaches the route and answers 404, which reads exactly like an extension that was never loaded. Ask /api/kernels for a real id first.

Execution follows the session's sandbox

Drilled on prod1 and r1 on 2026-09-06. Restart the gateway deployment under a live session and the runtime pod is the same pod on the other side, with the same creation time; the next call resolves the same sandbox handle and is answered. Neither the pod nor the binding is a casualty of a deploy.

The session's memory is not, and chasing that found something larger.

Runtimes come from a warm pool: a pod keeps its own name and is claimed by a logical runtime. The pod claimed by the session's runtime is stable across a rollout. But socket.gethostname() inside the kernel names a different pool pod — before the rollout and again after, a different one each time. The session's own sandbox is running, and nothing runs on it.

The steering has never worked on this SDK. The gateway stamps io.datalayer/sandbox on the request and the worker's hook is meant to read it back off ctx.request_context.meta. Measured against a real MCP 2 server — a _meta carrying the key and a tool that prints what it got — that meta is None by the time a tool sees it. So the hook read nothing on every request, and the worker ran the code on whatever its manager happened to hold.

Three fixes are deployed, each necessary. The extractor reads a plain mapping, because RequestParamsMeta is a TypedDict and not a model with extras. The gateway also sends the name in an x-datalayer-sandbox header, which is what actually crosses — the proxy forwards the scope's headers verbatim, the way traceparent already travels — read by one ASGI middleware in the worker. And that middleware no longer sits inside the telemetry wrapper, which returns early when no collector is configured: where a session's code runs must not depend on whether spans are exported.

The last link was a name. Extensions are registered under their manifest name and the follower asked for the entry-point name, so the lookup answered None and the follower gave up — silently, because each of its four ways of doing nothing was a bare return. Three deployments went on guessing between them. Each now says which, and the next deploy answered it in one line:

🧭 execute_code names sandbox [01m1twfv…] and there is no sandbox manager to follow it with

A line per decision costs one log entry per execution, and is why this took one more deploy rather than five.

Walked on prod1 and r1 on 2026-09-06, and it holds. The session's binding names a runtime; the kernel's own socket.gethostname() is the pod that runtime claimed. Restart the gateway under the session and afterwards it is the same runtime, the same pod, and the variable set beforehand is still there.

So a deploy no longer empties a session's memory, and a grantee attaching to a shared sandbox reaches the sandbox rather than a pool pod of its own.

The notebook walk passes with it. On prod1 the same day: use_notebook opens the notebook, insert_cell adds a code cell, execute_cell answers cell says 42 in a second and names the session's sandbox, read_cell shows the output in the notebook with execution count 1, and execute_code then reads the cell's variable back. A notebook cell and ad-hoc code share one kernel on one sandbox, which is the design's first rule walked rather than argued.

A notebook survives the worker that opened it

The gateway's notebook binding lives in the shared store: it survives a rollout, a restart, and a move to another replica. The worker's connection does not — it is a NotebookManager entry and a websocket to the document server, in one process's memory. Those two lifetimes are not the same, and until 2026-09-06 they were assumed to be.

So a worker restarted under a live session held no notebooks, and every cell tool refused:

Error executing tool read_cell: Notebook 'reconnect' is not connected.

— while the gateway's own binding still named the notebook perfectly well. Measured on prod1 across a gateway rollout: [('QAXYM5', 'reconnect', '01KW96PZYRVKZXT8FS644GFD0D')] before it and [('QAXYM5', 'reconnect')] after, with read_cell refusing in between. The session was not broken; the process that had opened the notebook was gone, and nothing reopened it.

Same shape as the sandbox above, and the same fix. The gateway names the notebook on every call that acts on one — the alias the worker knows it by, and the item it resolves to — in x-datalayer-notebook and x-datalayer-notebook-path, beside x-datalayer-sandbox. One ASGI middleware in the worker reads all three into context variables, and a hook reopens the notebook when the worker serving the call does not hold it.

Reopening takes nothing away. In MCP_SERVER mode use_notebook starts no kernel — it makes the document connection — so a reopened notebook lands on the session's own sandbox exactly as it did before. That is the follower's job, one seam over, and the two compose, in that order:

📓 Reopened notebook [reopen] from [01KW96PZYRVKZXT8FS644GFD0D], for insert_execute_code_cell on this worker
🧭 insert_execute_code_cell already runs on the session's sandbox [01m1tzb6j…]

use_notebook is not reopened for — it is the call that opens one — and neither is unuse_notebook, which would open one only to close it again.

Walked on prod1 on 2026-09-06. A session opens a notebook, runs a cell that sets a variable, and the gateway deployment is restarted under it. On the other side the binding is unchanged, insert_execute_code_cell runs on the same notebook and answers the notebook is back: set before the rollout — the notebook reopened, the sandbox followed, and the variable still in the kernel that was never restarted.

The reading half holds with it. A cell executed before the rollout is read back after it, through a worker that never opened the notebook:

=====Cell 40 | type: code | execution count: 1=====
kept = 'written before the rollout'
print(kept)
written before the rollout

read_notebook and execute_cell answer on the same notebook afterwards. The outputs the runtime already wrote survive the loss of the process that was driving them.

Asking for a task used to wedge the worker

Discovery advertises io.modelcontextprotocol/tasks, so a client may ask for a long-running call to become a task. Until 6 September 2026 the first client that did never got an answer — and neither did anything else that worker was asked for afterwards. On a freshly restarted deployment, with nothing else running:

1 plain          :    7.2s  {"jsonrpc": "2.0", "id": 1, "result": …}
2 plain again    :    0.6s  {"jsonrpc": "2.0", "id": 2, "result": …}
3 ASKING A TASK  :   45.0s  !! ReadTimeout
4 plain after    :   45.3s  !! ReadTimeout
5 plain after    :   45.3s  !! ReadTimeout

Not the task's own coroutine stuck: the event loop. The worker's main thread sat in sock_alloc_send_pskb — a blocking socket send with nobody reading — while its other seventeen threads waited on futexes. Nothing said so: /api/mcp/v1/operations/workers reported the worker healthy throughout, serving: 1, running: true, and only a pod restart cleared it.

The worker silenced itself by talking. Its stdout is a pipe the gateway reads, and the relay that reads it exists precisely so somebody empties it: otherwise the kernel buffer fills, the next write blocks, and because logging is synchronous from whichever thread logs — the event loop included — the worker stops serving anything at all.

The relay had stopped doing its job. Its loop sat inside contextlib.suppress(CancelledError, Exception), so any exception ended it in silence — including the ValueError asyncio raises for a line past its 64KiB reader limit. And a worker produced such lines: building the Solr task repository installs a ZooKeeper cluster-state watcher that logs every collection, shard and replica at INFO on every update.

worker output, by second:  1 @18:26:19   5 @18:26:24   25 @18:26:25   27 @18:26:26
605,129 bytes in seven seconds, longest line 62,793

Both halves are fixed. The relay survives a line of any length, says when it drops one, and says so if it ever does stop instead of vanishing; its reader limit is 1MiB rather than 64KiB. And the worker does not emit that line: pysolr and kazoo are quietened to WARNING, so a Solr failure still comes through and the cluster state does not.

A worker that has gone quiet can be asked where it is

Send it SIGUSR1 and every thread's Python stack goes to its stderr, which the gateway copies into its own log:

kubectl exec -n datalayer-api <pod> -- kill -USR1 <worker pid>

If nothing appears, that is itself the answer: the dump writes to the same pipe, so silence means the pipe is the problem. Then read /proc/<pid>/task/*/wchan — a main thread that is not in ep_poll is a blocking call that should have been in a thread.

Worth keeping for the shape rather than the line: anything that blocks a worker's loop takes the whole worker down, not the request that made it, and the directory cannot tell.

A task nobody owns is not a task everybody owns

_task_for is the one place a task's ownership is decided, and three routes ask it: reading a task, cancelling it, and answering one that is waiting. It read

owner = str(task.get("user_uid") or "")
if owner and owner != caller.user_uid:
    ...refuse...

so a row with an empty user_uid skipped the check entirely and was answered — and cancelled — for any caller holding notebooks:read.

Every row was in that state. The projection read initiating_user_uid, the Solr field the owner is stored in, rather than initiating_user, the key the codec decodes to. A task written on prod1 on 7 September 2026 with initiating_user='01JV1VE1…' came back over GET /api/mcp/v1/tasks/{uid} as "user_uid": "".

Either defect alone was harmless: an unenforced check over a value that was always present, or a lost value under a check that enforced it. Together they were an open door.

Both halves are fixed — the projection reads the key that arrives, and an owner that cannot be established is a refusal rather than a pass. Worth keeping for the shape: an empty value is not permission. The same sentence covers the sandbox grant that named nobody and reported success, one section up.

Snapshots, and why there is no restore tool

A sandbox is the one place a session's state is not recoverable. The notebook is in the Spacer, the binding is in mcp-gateway, a lost sandbox can be relaunched — but what is loaded inside one, the data read and the packages installed and the model fitted, lives nowhere else. That is what on_lost: relaunch means when it says the replacement is empty.

snapshot_sandbox saves it. It costs sandboxes:manage, not code:execute: it writes storage the account is billed for and produces something another session can be started from, so an agent granted "run this" may run it and may not leave a copy behind. datalayer://snapshots is the catalogue, fetched with the caller's own token — a service credential there would list everybody's saved working state — and it costs data:read, because finding out which snapshots exist is not making one.

Restoring is launch_sandbox(snapshot_name=…), not a tool of its own. A launch is the door where the gateway reserves the session, counts the sandbox against the quota and checks the provider against policy. A restore_snapshot beside it would be a second way to obtain a sandbox and a second place to remember all three — and the gateway intercepts by tool name, so the one it did not know about is the one that skips the checks.

restore_notebook is not that tool. It restores a notebook version — the Spacer's kept copy of a notebook's content — and touches no sandbox. snapshot_notebook, list_notebook_versions and restore_notebook resolve a name the way use_notebook does and hand the uid to the Spacer with the caller's exchanged credential, so the Spacer decides who may and its 403 reaches the agent in words. A restore keeps what it replaces first, so it can itself be undone. They cost notebooks:read for the listing and notebooks:write for the two that change history — a notebook's history, not a sandbox's state, which is why they sit beside the editing tools and not here.

A sandbox started from a snapshot cannot also ask for on_lost: relaunch, and that pair is refused. A relaunch is built from what the binding recorded — the alias, the provider, the environment — and a snapshot is not among them: the name an agent gives is resolved to a snapshot at launch time, and the same name can point at a different snapshot later. A relaunch is also silent, so the alternative is a sandbox that comes back empty after being asked for a saved state, found whenever somebody next looks for a variable. Launch with on_lost: fail and launch again with the same snapshot_name when you are told the sandbox is gone.

Only sandboxes whose Environment declares the snapshot capability are offered the tool at all: it is filtered out of tools/list for a session whose sandbox cannot do it, and a call to it anyway is refused naming the capability. An agent on Modal never sees it — an external provider's environment never gets snapshot from the shared vocabulary.

A capability nobody declares hides the tool from everybody

This is per Environment, like home-folder, and it is opt-in. Every platform Environment shipped in plane/etc/specs/runtime-environments declares it; an Environment added without it will not offer snapshot_sandbox to any session, and nothing about that reads as an error — the tool is simply absent from tools/list.

A deployment can take it away again: a snapshot is written to the shared filesystem, so the Operator turns snapshot off where SHARED_FS_VOLUME_CLAIM_NAME is unset, exactly as it does home-folder. On such a deployment no session reports the capability however the Environments are written, and a launch that requires it is refused rather than given a sandbox that cannot snapshot.

server/discover is where to look. Under io.jupyter-mcp/capabilities it answers three things:

Field	Says
known	Every capability name the shared vocabulary defines — what exists at all
knownFamilies	Prefixes with one member per thing they name. contents. is the only one: an Environment that brings a Content declares contents.<uid>, which no fixed list can hold
capabilities	What this session's sandbox reported, which is empty when the session has no sandbox

A tool missing while its capability is in known and not in capabilities is an Environment that has not declared it, and not a broken gateway.

Discovery is what puts a client on the modern wire

supportedVersions in the server/discover result is what a client adopts: it intersects that list with the modern versions it knows and takes the newest. An answer it cannot parse sends the client back to initialize, which negotiates 2025-11-25 — and long-running calls do not exist on that wire. A tools/call carrying task is answered with a task, the client validates that answer as an ordinary tool result, and it is rejected for having no content.

Until 2026-09-08 the field was sent under the wrong name, so every SDK client fell back and no conformant client could use the durable path at all, while readyz correctly reported running execute_code durably. If a client of yours cannot create tasks, check the protocol version it negotiated before anything else — 2025-11-25 explains it completely, and readiness will not.

And every answer must say what kind it is

resultType is required on the 2026-07-28 wire, and tools/call is a union discriminated on it: complete for a finished result, task for the receipt of a call that became one. The gateway assembles its own answers rather than letting the SDK dump them — refusals, a launch_sandbox it short-circuits, a cached not modified — and until 2026-09-08 none of them carried it. A modern client raised on each one before its caller saw anything, which reads as the gateway being down rather than as a shape it got wrong.

Fixed in datalayer-mcp-server:0.0.4. On the older wire the field's absence means complete, so nothing about this changes for a 2025-11-25 client — which is also why it went unnoticed for as long as the only clients were on that wire.

:::

What it stores

Three Solr collections belong to this service, all in the AI set beside what ai-agents owns, and all three are one recovery unit — the mcp scope, nested inside the wider ai one so the gateway can be restored without the rest of the AI plane. See Solr and Continuity.

Collection	Written today by	Holds
mcp-gateway	the gateway, and IAM	Handles (nb_… notebooks, sb_… sandbox sessions, ts_… toolsets), the worker directory, rate windows, fired alerts, the scheduler's leases — and IAM's MCP policies and alert rules
mcp-tasks	read by the gateway	The task projection. GET /api/mcp/v1/notebooks/{uid}/tasks and the Running now panel answer from this collection, so every replica gives the same answer. Tasks are created by whatever runs them — Durable — so the gateway reads and ends them rather than writing new ones
mcp-audit	the gateway	The append-only audit trail, one row per call and decision, served by GET /api/mcp/v1/audit. Rows carry decision, outcome, trace_id, replica, protocol_version, refusal_reason, duration_ms and a hash of the arguments rather than the arguments

mcp-gateway holds eight document families, told apart by type_s:

type_s	Written by	Holds
mcp_binding	the gateway	A handle returned to an agent, and its lifecycle
mcp_worker	the gateway	Which replica holds the connection for one user's sandbox
mcp_rate_window	the gateway	A per-subject counter over one window
mcp_alert_event	the gateway	A rule that fired: what, when, whose, acknowledged by
mcp_lease	the gateway	Which replica is doing one periodic job right now
mcp_policy	IAM	One layer's rules for one subject — see Policy layers
mcp_alert_rule	IAM	What an organization asked to be told about — see Alert rules
mcp_audit_settings	IAM	How long an organization keeps its audit and where a copy goes — see Audit settings

The last three are IAM's to write and the gateway never writes them; they live here rather than in an IAM collection because a rule and the events it produces are one question, and answering it across two collections is a join nobody would write.

Every record carries a version, so a write that names the version it read fails when the record moved in between, and every record carries an expiry. Nothing here is a cache: it is what a request must be able to find again on another pod.

The mapping is generated, versioned and fingerprinted

MCP_MAPPING_VERSION is 9. The mapping in mcp_mappings.json is generated from the DAOs' own codecs and a test fails when the checked-in copy disagrees with the code, so a field added without a version bump cannot ship. Every version so far has been additive — a document written under an earlier one reads back correctly — so no deployment needs a reindex to take an upgrade.

The store is chosen by DATALAYER_MCP_STORE. Unset, Solr is used when DATALAYER_SOLR_ZK_HOST is configured and an in-memory store otherwise — which is right for plane local and wrong in a cluster, where it would give every replica its own idea of who holds what.

One rule decides it, for the bindings, the audit ledger, the task projection and the alert state together. They are separate collections but never separate answers: a deployment whose audit went to Solr while its bindings stayed in the process would be a gateway half in each mode, and nothing would say so. readyz reports the answer as records, and Turning it off is what to read before changing it.

plane solr-backups-apply mcp      # the three collections as one backup unit
plane solr-backups-status mcp
plane solr-restore mcp

plane solr-backups-apply ai       # the wider unit: these three and ai-agents'

A refusal is a decision, not an error

decision records what policy did. A call for a tool that does not exist is decision=allowed, outcome=is_error: nothing refused it, and the worker answered with an error. A call naming a handle bound to nobody is decision=refused, refusal_reason=unknown_handle, and never reaches a worker. Reading the first as a missing refusal is the mistake to avoid when auditing.

Deploy

The gateway needs IAM and Spacer to be up, and the three mcp-* collections to exist. Create them with plane solr-init and choose 4) AI, which creates only that set rather than issuing a create against every Core collection the cluster already has. Without mcp-gateway the gateway starts, serves /api/mcp/healthz, and never becomes ready: readyz reports store … Unknown collection: mcp-gateway, the pods stay 0/1, and the ingress answers no available server. Build and push the image from the Services repository:

cd plane/etc/dockerfiles/datalayer-mcp-server
make build-dev
make push

The build asserts that a worker could start, so a missing piece fails the build rather than showing up hours later as every user getting the wrong tool list. It checks four things:

• the spaces, sandboxes and library extensions are on the reactor.mcp.extensions entry point, and the open source sandboxes one with them — four names, not three;
• datalayer-core, code-sandboxes and the provider SDKs are new enough, and the traceback hook installs;
• a server built the way a worker builds one carries execute_cell (which is only true of a base that offers its own tools as contributions) and the three toolsets' tools;
• ?only=spaces leaves launch_sandbox out and ?only=sandboxes keeps it — the URL selection works, rather than being present and inert.

The last two are outcomes rather than version numbers on purpose: a floor that is satisfied and a feature that does not work look identical from a pip list.

Plane

plane up datalayer-mcp-server
kubectl rollout status deployment/datalayer-mcp-server -n datalayer-api

Terraform

cd terraform
terraform init
terraform apply
./generated/clouder-Kubeadm-setup.sh
export KUBECONFIG=~/.clouder/kubeadm/<cluster-name>/kubeconfig
./generated/services/deploy-datalayer-mcp-server.sh

up.sh passes DATALAYER_MCP_SERVER_URL (default https://mcp.datalayer.run/mcp), the IAM, Spacer, Runtimes and Contents URLs, the Solr identity, the JWT settings and the OTLP endpoints. It leaves the scaling values, the worker limits and the rate limits at their chart defaults. plane up runs a fixed helm upgrade, so anything it does not pass is changed in the chart's values.yaml (or with a helm upgrade --reuse-values --set of your own) rather than on the plane command line.

Local development

plane local starts the gateway beside IAM, Spacer, Runtimes and Contents on http://localhost:4404, validates /api/mcp/healthz and prints the claude mcp add datalayer-local --transport http http://localhost:4404/mcp line to connect a client to it. It runs on the in-memory store.

Verify

The acceptance run

tests/test_acceptance.py drives a deployed gateway the way an agent does — both protocol versions, a handle across replicas, the audit export, the activity panel, a disconnection, a trace — and skips loudly rather than passing vacuously: every skip names the variable that would unlock it, and asking for the file with nothing configured is a failure, not a green run.

export DATALAYER_MCP_ACCEPTANCE_URL=https://r1.datalayer.run
export DATALAYER_MCP_E2E_TOKEN=…            # the run's own credential; a PAT will do
export DATALAYER_MCP_E2E_SECOND_TOKEN=…     # an OAuth token of the same person's, from a real
                                            # consent — it is the agent that gets disconnected
export DATALAYER_MCP_ACCEPTANCE_REPLICAS=2  # say so only when the deployment has two
pytest tests/test_acceptance.py -v

The second token has to be a grant — a personal access token names none and cannot be disconnected — and it is spent by the run: the disconnection revokes it. Mint another for the next run. A client that still performs the handshake (2025-11-25) opens a session first; without one the SDK answers 400 Missing session ID, which is the gateway being right.

kubectl get pods,service,ingress -n datalayer-api -l app=mcp-server
kubectl logs -n datalayer-api -l app=mcp-server

The gateway is served on a host of its own, so it has a certificate of its own, declared by the chart rather than inferred from ingress annotations — the ingress deliberately carries no cert-manager.io/* annotation, because two owners for one secret is how a certificate ends up repeatedly reissued.

kubectl describe certificate r1.datalayer.run-datalayer-api-cert -n datalayer-api
kubectl get certificate,certificaterequest,order,challenge -n datalayer-api | grep mcp

Ready: True means the certificate is issued. While it is still being issued the browser reports the site as not secure, and the events of the resource say what the ACME challenge is waiting for — most often DNS for the new host not yet resolving to the ingress.

curl -s https://r1.datalayer.run/api/mcp/version
curl -s https://r1.datalayer.run/api/mcp/readyz | jq '.dependencies[] | {name, required, ready, detail}'
curl -s https://mcp.datalayer.run/.well-known/oauth-protected-resource/mcp
curl -s https://mcp.datalayer.run/.well-known/mcp-server | jq '{name, version, protocolVersions}'

The third is the document an MCP client reads to discover where to authenticate. If it does not answer, no agent can connect. The fourth is the Server Card: what this server is, before anyone authenticates.

Calling it by hand

A tools/list at 2025-11-25 is an ordinary JSON-RPC POST:

curl -s https://mcp.datalayer.run/mcp \
  -H "Authorization: Bearer ${DATALAYER_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2025-11-25" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

2026-07-28 is not the same request with a different header. There is no initialize handshake, so each call carries what the handshake used to establish, and repeats its method — and its tool name, for tools/call — in headers so a proxy can route without parsing a body it may not read:

curl -s https://mcp.datalayer.run/mcp \
  -H "Authorization: Bearer ${DATALAYER_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -H "MCP-Method: tools/list" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{"_meta":{
        "io.modelcontextprotocol/protocolVersion":"2026-07-28",
        "io.modelcontextprotocol/clientCapabilities":{}}}}'

Omitting any of it is a 400 that names what is missing, and it is the server being correct:

Omission	Answer
no params._meta envelope	params._meta must be an object carrying the required … envelope keys
no MCP-Method header	mcp-method header does not match the request body's method
no MCP-Name on a tools/call	mcp-name header does not match the request body's 'name' parameter

Reading those as a gateway that refuses 2026-07-28 is the mistake to avoid; they are the gateway enforcing the version it advertises.

Authentication end to end

An unauthenticated request must be refused and must say where to authenticate:

curl -i https://mcp.datalayer.run/mcp

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="Datalayer",
  resource_metadata="https://mcp.datalayer.run/.well-known/oauth-protected-resource/mcp"

A 401 without that header means a client will fail instead of starting the OAuth flow. With a personal access token:

curl -H "Authorization: Bearer ${DATALAYER_TOKEN}" \
  https://r1.datalayer.run/api/mcp/v1/bindings

Scaling and the drain

The gateway is one Deployment whose replicas are interchangeable, and the chart says so.

Setting	Value	Why
mcpServer.replicaCount	2	The floor. Applied only when autoscaling is off — with the HPA enabled the Deployment omits replicas so the two do not fight
mcpServer.autoscaling	enabled: false, minReplicas: 2, maxReplicas: 8, targetCPUUtilizationPercentage: 70	Off by default; see below
mcpServer.podDisruptionBudget	enabled, minAvailable: 1	A node drain never empties the service
mcpServer.topologySpread	enabled	maxSkew: 1 across topology.kubernetes.io/zone and kubernetes.io/hostname, ScheduleAnyway so a small cluster still schedules rather than leaving a replica Pending
strategy.rollingUpdate	maxSurge: 1, maxUnavailable: 0	A rolling upgrade is the ordinary case; the new pod is ready before an old one goes
mcpServer.terminationGracePeriodSeconds	900	Long enough to finish the streamed responses a pod is holding
node affinity	role.datalayer.io/api=true (required)	The API node pool
pod annotation	karpenter.sh/do-not-disrupt: "true"	Karpenter does not consolidate a pod holding sessions

On mcpServer.autoscaling. The chart refuses to render if it is switched on while the records stay in the process — standalone mode, or memory as the store. minReplicas wins over replicaCount, which is how two standalone pods came to be running, each with its own idea of every session. Switched on, it is an autoscaling/v2 HPA on CPU with scaleDown.stabilizationWindowSeconds: 600, so a quiet minute does not take a pod away from the sessions on it.

The HPA scales on CPU only. The plan's request-rate and running-worker (mcp.workers{state=running}) targets are written into the template as a comment and are not wired: they need an external-metrics adapter over the OTEL service, which is not deployed.

What the first measurement showed (prod1, 2026-09-05). The load harness (tests/load/) at 25 agents for 90 s against 1, 2 and 4 replicas gave 3.99, 4.22 and 3.91 calls/s at a p95 of 11.8, 11.6 and 12.7 s — flat, and for a reason that is not the gateway's. Every agent in a run carries one credential, one credential is one user, and a user is one worker process: the ceiling is that worker's kernel, wherever the gateway puts it, and no number of replicas moves it. What the run does prove is that the session path holds through the gateway — the p95 did not degrade when every call to the one worker was forwarded across four pods. A measurement of capacity growing needs a credential per agent, which is a fixture the platform does not hand out yet. The run also counted 5% HTTP 500s, all one defect: a session's last_used_at touch is a compare-and-set, it was retried once, and under twenty-five agents on one session the retry lost too. It re-reads up to five times now, and a touch that keeps losing answers what was written last rather than failing the call. And getting the harness as far as a number found the four defects listed under The first execution binds the session and No affinity at the edge, none of which a one-process test could see.

The two disruption runs, same day and shape (25 agents, two replicas, the disruption twenty seconds in): losing one pod cost 2 × 502 and 5 in-flight timeouts out of 393 calls — that pod's requests in flight and little else; the survivor forgot the dead pod's directory entries on the first refused forward, and the agents' handle kept working. A rolling restart cost 23 × 502 and two timeouts out of 385: the pod being replaced kept serving requests forwarded to it while its worker children were already dying. A drain that stops accepting forwards before anything else, and re-registers nothing, is what closes that gap; the grace period alone does not.

What the load test measured, and what it could not

Run on prod1 on 2026-09-07: 50 agents for 120 seconds at each of 1, 2 and 4 replicas, each agent doing a whole sequence — discover, list, open a notebook, read a cell, execute code — rather than a stateless read.

replicas  calls    ok  failed  calls/s      p50      p95      p99
       1    842   468     374     6.66    12.5s    23.4s    28.4s
       2    478   476       2     3.01    13.5s    29.2s    44.9s
       4    466   460       6     3.47    13.2s    31.6s    38.5s

Successful calls are flat — 468, 476, 460. compare.py refused to report a factor at all, naming the baseline's 374 failures and a p95 that moved between runs, which is the harness doing its job rather than failing.

The reason is not a scaling defect: a worker is one process per user, and the proxy routes to the replica holding it. The harness drives every agent with one token, so every tool call lands on that one replica or hops to it, and a fourth replica adds a hop rather than capacity. Only server/discover — one step of six — is answered without a worker. Measuring capacity per replica needs many callers, not many connections; with one token the question cannot be asked.

Losing a pod under load behaves, and after the readiness fix it is perfect. Delete one of two while 30 agents work: before the fix, 9 failures out of 581, all connection errors; after it, 629 calls and 629 answers, no failures at all, with the same thirteen active handles before and after each time. The claim allows losing that pod's in-flight requests; it now loses none.

A rolling upgrade under load does not, quite. Same load, roll the deployment: 15 failures out of 596, nine of them 502, with handles again thirteen before and thirteen after. Nothing is lost — but "no request dropped" is not true today.

Fixing the readiness probe below did not change that: run again on the same day with the fix deployed, the number was 15 of 596 exactly, all 502 — while the pod-deletion drill beside it went from 9 failures to none. So the probe was one real cause and this is another, which is worth stating because they looked like one problem.

Why a rollout drops requests: nothing holds the door. The Deployment has no lifecycle.preStop hook, so SIGTERM reaches the process at the same moment the endpoints controller starts removing the pod from the Service — and endpoint removal has to propagate to every kube-proxy and the ingress before traffic stops arriving. Requests routed during that window reach a pod that has begun draining, and the ingress answers 502.

The remedy is the usual one and belongs in the chart rather than the code: a preStop that sleeps a few seconds, so the pod keeps serving while the removal propagates and only then hears SIGTERM. The gateway's own drain is not the problem — it is correct and it starts too early.

There is no autoscaler running, and that is the default

prod1 has no HorizontalPodAutoscaler and the Deployment carries an explicit replicas: 2. That matches the chart, whose autoscaling.enabled is false — this page said enabled until 7 September 2026, which is the error. So the scale-up and scale-down behaviour above is not what is running anywhere until somebody turns it on, and turning it on requires moving the records out of the process first; the chart enforces that rather than trusting it.

A readiness probe must not queue behind the work

The load test found this, and it is the more useful half of what it found.

/api/mcp/readyz used to answer by probing: four service pings and four Solr reads, eight network round trips, on the request path. The kubelet runs it every ten seconds with timeoutSeconds: 5. Under load that budget is gone — not because a dependency is slow, but because the answer waits behind the requests it is reporting on:

Warning  Unhealthy  1s (x19 over 4m16s)  kubelet
  Readiness probe failed: Get "http://10.244.1.232:4404/api/mcp/readyz":
  context deadline exceeded

while the same endpoint answered instantly when called from inside that pod. Kubernetes takes such a replica out of the Service and sheds its share onto the others, which meet the same wall — the shape of the 371 503s in the one-replica run above.

The answer is kept now, and refreshed behind the requests rather than in front of them: readyz serves the last result immediately and starts a background refresh when it has aged past five seconds, one refresh at a time. A refresh that fails leaves the previous answer standing, because reporting not ready over a check that could not be made would take a working replica out of service for being busy — the same failure through another door. dependencies_age_seconds on the response says how old the verdict is.

Measured after deploying it: nineteen probe failures became zero. Both pods went through a 30-agent run and a rolling upgrade with no Unhealthy event at all. What it did not do is change the rollout's dropped requests, which is the section above.

No affinity at the edge

The ingress round-robins. There is no cookie affinity and there are no single-replica idle timeouts, because affinity is a directory lookup, not a cookie: every worker is written into mcp-gateway keyed by user and by sandbox session, naming its replica and an address. A replica that is asked for a worker another replica holds forwards the request there over the cluster network, carrying x-datalayer-mcp-hop; a replica that sees that header serves the request itself, so two stale entries can never bounce a request forever.

A replica that is not there is forgotten on the first refused connection. Until 2026-09-05 an unreachable target answered 502 replica_unreachable and said to retry — and kept saying it: a pod that went with a rollout stopped refreshing its entries, but they lived on for DATALAYER_MCP_WORKER_ENTRY_TTL_SECONDS (default 180 s), and every session that pod held was a 502 for three minutes. That is not "losing that pod's in-flight requests and nothing else", which is the property the scale-out measurement exists to prove; prod1's first initialize after leaving standalone mode met exactly it. Now the replica that could not reach the target drops the directory entries that named it — the session's and the user's — and serves the request itself, which is what the entry expiring would have meant, three minutes later. The connection fails before any of the body is sent, so the fallback has the whole request. Expiry remains the backstop for an entry nobody asks about; 502 worker_unreachable remains the answer when the local worker cannot be reached either.

The ingress keeps proxy-read-timeout and proxy-send-timeout at 900 to match the pod's drain, and proxy-buffering: off so a streamed response is not collected at the edge. These are nginx.ingress.kubernetes.io/* annotations, as on the other Datalayer charts; under the default datalayer-traefik class they are inert and the equivalent limits are Traefik entrypoint settings.

POD_IP is what lets a forward leave the pod

A replica registers itself at DATALAYER_MCP_REPLICA_ADDRESS, or at http://$POD_IP:4404 when POD_IP is set, and at http://127.0.0.1:4404 otherwise. The chart's Deployment sets POD_IP and POD_NAME from the downward API, which is what the cross-replica acceptance check on prod1 (2026-09-05, two replicas, a handle issued by one and read six times through the other) relied on. Without it every replica registers the loopback address and a forward lands back on the pod that sent it — which then serves the request itself, correctly but on the wrong worker, so a session mid-execution can be answered by a second sandbox connection. A chart of your own needs the same:

# values.yaml of the datalayer-mcp-server chart
mcpServer:
  envValueFrom:
    POD_IP:
      fieldRef:
        fieldPath: status.podIP

POD_NAME needs nothing: the replica names itself from HOSTNAME, which Kubernetes already sets to the pod name.

What a pod does when it is told to stop

1. uvicorn stops accepting new connections and lets the in-flight requests and streamed responses finish.
2. The supervisor marks itself draining, so /api/mcp/readyz answers 503 with "draining": true.
3. Every worker-directory entry of this replica is given an expiry of now, so no other replica forwards a request to a pod on its way out.
4. Every worker on the pod is stopped and waited on, up to ten seconds each.

Step 4 stops every worker, including one under a working task — only the idle reaper and the capacity eviction consult the task store first. Until runs live in a durable engine rather than in the worker's connection, a rolling upgrade ends the executions in flight on the pod being replaced. maxSurge: 1 / maxUnavailable: 0 and the 900 s grace keep that to the sessions of one pod at a time.

Handles, sessions and rate windows survive all of it: they are in Solr, and a replica restarts empty and is serving in seconds.

kubectl scale deployment/datalayer-mcp-server -n datalayer-api --replicas=4
kubectl get hpa datalayer-mcp-server -n datalayer-api
kubectl get pdb datalayer-mcp-server -n datalayer-api

To see which replica holds which worker, as a platform_admin:

curl -H "Authorization: Bearer ${DATALAYER_TOKEN}" \
  https://r1.datalayer.run/api/mcp/v1/operations/workers | jq

Health and readiness

Route	Answers
/api/mcp/healthz	Liveness: the process is up. Version and replica name, nothing else
/api/mcp/readyz	Readiness: may traffic be sent here. 200 / 503, plus every dependency on its own line

The body also carries mode — standalone or platform, what the deployment asked for — and records — memory or solr, where the bindings, tasks, audit rows and alert state actually went. Ask for both. They agree in standalone; in platform mode "records": "memory" means the replicas do not share a session, and it is the one misconfiguration every other check here reports as healthy. See Turning it off.

Readiness is gated on what every request needs: IAM (to verify a token) and the store (to find a handle). Spacer, Runtimes, Contents, the audit ledger, the task projection and the workflow backend are reported and never gate — an outage in Contents must read as "Contents unavailable" in the Contents tools, not as a gateway that stopped serving notebooks. OTEL is listed and never probed: telemetry is never a readiness dependency. A draining replica is not ready.

The workflows line says where durable work goes, and whether it can be accepted. The gateway falls back to the in-process fake whenever DATALAYER_DURABLE_URL is unset — the rule the store and the ledger already follow — so a deployment in platform mode with that variable empty keeps every workflow in one process and loses them on restart, while mode: platform says the opposite. That reads as not-ready, because it is a misconfiguration rather than an intent; the same fake in standalone mode is what that deployment asked for, and reads as ready with a detail saying nothing survives a restart. A real datalayer-durable is ready when it says it can accept work — it answers that honestly, so a launched engine whose system database cannot be read says durable: false and nothing else, and a gateway that called it ready anyway would repeat the failure one layer up. The service's none engine — what plane local runs — answers durable: false, serving: true: it takes work in its own process and keeps nothing, and the gateway routes to it, with the detail saying "not durable" so the line reads as what it is rather than as an outage. None of this gates: a durable outage costs the long-running calls and leaves everything else working, and gating readiness on it would turn that into a total outage.

The store, audit and tasks lines each name which implementation answered, not only whether it did, and that is the point of them. Each lives in its own Solr collection, so the bindings collection being healthy says nothing about the other two — a deployment whose task collection is missing would otherwise serve a green page and lose every long-running call. And a line reading MemoryTaskStore is perfectly healthy and means tasks do not survive a restart and are invisible to the other replicas, which is the same warning "records": "memory" gives above and worth reading twice.

Each dependency probe is bounded at three seconds (PROBE_TIMEOUT_SECONDS), and they run concurrently, so a hung dependency is reported as hung rather than hanging the endpoint. That bound is also the endpoint's worst case: one unreachable dependency — including an optional one, which does not gate readiness — costs the full three seconds before readyz answers 200.

Give the probe more time than the endpoint takes

The Kubernetes default timeoutSeconds is 1, which is shorter than readyz is designed to take. Left at the default, the kubelet cuts the probe off before the endpoint can answer:

Readiness probe failed: Get "http://10.244.2.137:4404/api/mcp/readyz":
  context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The pods stay 0/1 with a healthy process inside them and the ingress answers no available server, while curl from inside the pod returns 200. The chart states the budget rather than inheriting it:

# values.yaml of the datalayer-mcp-server chart
mcpServer:
  readinessPath: /api/mcp/readyz
  readinessTimeoutSeconds: 5   # must exceed PROBE_TIMEOUT_SECONDS (3)
  livenessTimeoutSeconds: 2    # healthz reaches nothing

Raise both together if PROBE_TIMEOUT_SECONDS ever rises. This bit on prod1 when datalayer-runtimes — an optional dependency — was failing its own ping, which is how an optional dependency being down took the gateway out of service entirely.

To see which dependency is slow rather than guessing:

kubectl -n datalayer-api exec deploy/datalayer-mcp-server -- \
  python -c "import urllib.request,json;\
print(json.dumps(json.load(urllib.request.urlopen(\
'http://127.0.0.1:4404/api/mcp/readyz')),indent=2))"

Each line names the dependency, whether it is required, whether it is configured, whether it is ready, and the detail — ready, timed out, or the store's own message such as Unknown collection: mcp-gateway.

A configuration mistake is one log line at the top of the log, not a permission error an hour later: a malformed URL is fatal and the process refuses to start naming the setting; a missing one is an error line and a readiness report.

Configuration

Variable	Purpose
DATALAYER_MCP_SERVER_URL	The resource identifier of this gateway: what a token's aud must name, what the protected-resource metadata publishes, and what the ingress host is derived from — so the host and the audience cannot drift apart. Default https://mcp.datalayer.run/mcp
DATALAYER_MCP_VERIFY_AUDIENCE	Refuse a token issued for another resource; default true. Turning it off is for a local run against a token minted before the resource URL was settled, never for a deployment
DATALAYER_IAM_URL	The authorization server: token verification metadata, the RFC 8693 exchange, and the readiness probe. Required
DATALAYER_IAM_API_KEY	Proves to IAM that this is a Datalayer service, which is what lets it exchange a caller's gateway token for one the rest of the platform accepts. Without it, OAuth callers reach nothing. DATALAYER_OPERATOR_API_KEY is accepted as a fallback. Supply it through a Kubernetes Secret
DATALAYER_SPACER_URL	Where notebooks and their permissions live; defaults to DATALAYER_IAM_URL
DATALAYER_RUNTIMES_URL	Where a sandbox is asked for. Provider-neutral: the same URL launches an internal runtime or an external one, and the gateway never sees which SDK is behind it. Defaults to DATALAYER_IAM_URL
DATALAYER_CONTENTS_URL	Where Contents runs — the runtimes plane, at its own public URL (https://r1.datalayer.run). No fallback, and a value equal to DATALAYER_IAM_URL is reported as a misconfiguration: a gateway that does not know where Contents is has no Contents, and readiness says so
DATALAYER_LIBRARY_URL	Where the public library is searched, for search_library, get_library_artifact and read_library_artifact; defaults to DATALAYER_IAM_URL, which is where it is served. Leaving it out does not turn the tools off. Its routes answer anonymously, so these tools work for a caller with no Datalayer identity — what a credential adds is only whether the caller orbits a result
DATALAYER_AI_AGENTS_URL	Where activity events are posted, so what an agent does appears in the feed; defaults to DATALAYER_IAM_URL. Every post is best-effort and off the request path
DATALAYER_MCP_STANDALONE	Run the MCP endpoint with none of the platform machinery behind it — see Two modes. true/1/yes/on; anything else, including false, is off. It wins over DATALAYER_MCP_STORE below, and it requires replicaCount: 1. Turning it off does not by itself move the records out of the process — set DATALAYER_SOLR_ZK_HOST in the same change, or the chart refuses to render. See Turning it off
DATALAYER_MCP_STORE	solr in a deployment, memory for a single local process. Unset, Solr is used when DATALAYER_SOLR_ZK_HOST is set and memory otherwise. Any other value refuses to start. Ignored under DATALAYER_MCP_STANDALONE
DATALAYER_SOLR_ZK_HOST	The ZooKeeper ensemble the mcp-gateway DAO connects through
DATALAYER_SOLR_USERNAME / DATALAYER_SOLR_PASSWORD	The Solr identity, supplied through Kubernetes Secrets
DATALAYER_MCP_MAX_WORKERS	The most worker processes one replica runs at once; default 50. Reached, the least recently used idle worker without a working task is stopped to make room, and a caller who cannot be given one is refused 503 with the reason. This is the density limit of a replica — capacity is added by adding pods, not by a bigger pod
DATALAYER_MCP_WORKER_IDLE_TIMEOUT	How long a worker may sit unused before it is reclaimed; default 900 seconds. A worker whose session has a task working is never idle
DATALAYER_MCP_WORKER_START_TIMEOUT	How long to wait for a new worker to answer before giving up on it and quoting its last output; default 30 seconds
DATALAYER_MCP_WORKER_ENTRY_TTL_SECONDS	How long a worker-directory entry lives after its last heartbeat; default 180. A replica that is gone stops refreshing, its entries expire, and the caller gets a fresh worker rather than a dead address
DATALAYER_MCP_REPLICA_ADDRESS	The address other replicas forward to. Defaults to http://$POD_IP:4404, or loopback when POD_IP is unset — see the warning above
DATALAYER_MCP_BINDING_TTL_SECONDS	How long a handle lives after it was last used, refreshed on every use; default 86400
DATALAYER_MCP_RESERVATION_TTL_SECONDS	How long a reservation may sit before another caller may take the session back; default 300. A reservation says "a launch is in flight", which is a question of seconds — it once inherited the binding's day-long lifetime, so a launch that never concluded blocked every later launch for that session for a day. abandon() returns it when the gateway sees the failure; this bounds the case where nothing sees it because the replica died mid-launch
DATALAYER_MCP_POLICY_TTL	How long a policy layer is cached; default 30 seconds. Deliberately short: a policy is what somebody changes in order to stop an agent, and a revocation that takes ten minutes to arrive is one the person watching will assume did not work. See Policy layers
DATALAYER_MCP_AUDIT_RETENTION_DAYS	Per-plan audit retention, as plan:days pairs — free:30,pro:365. A malformed pair is skipped rather than raising, because this is read at import and a typo must not be a crash loop; a non-positive value is skipped too, since audit readable for no time at all can only be a mistake
DATALAYER_DURABLE_URL	Where Durable is, at its own public URL on the runtimes plane. Left empty the gateway falls back to an in-process workflow backend that reports durable: false — correct for a cluster that has not deployed it, and the thing not to mistake for one that has
DATALAYER_DURABLE_API_KEY	What the gateway proves itself to Durable with. Falls back to DATALAYER_MCP_SERVER_API_KEY, which is the name Durable knows this caller by
DATALAYER_MCP_DURABLE_TOOLS	Which tools run on the durable engine instead of in the worker that accepted the call. Empty — the default — keeps every task on the worker, which is what a deployment does until somebody chooses otherwise. A comma-separated subset of what the gateway vouches for, or all for whatever that is; today it vouches for execute_code alone. The allowlist is in code and this may only narrow it: a name it does not know is dropped with a warning rather than honoured, so a typo routes less rather than routing something nobody chose. GET /api/mcp/readyz says which tools are routed, because an engine that is ready and receives nothing reads exactly like one that is working
DATALAYER_MCP_CALLS_PER_MINUTE_PER_USER	Tool calls one user may make in a minute; default 300, 0 no limit. The refusal is a JSON-RPC error carrying retry_after_ms, and the windows live in mcp-gateway so the limit means the same thing however many pods answer
DATALAYER_MCP_CALLS_PER_MINUTE_PER_AGENT	The same, per client_id; default 120
DATALAYER_MCP_DEPENDENCY_TIMEOUT	How long a call to a platform dependency may take before it is a timeout; default 10 seconds
DATALAYER_SANDBOX_ENVIRONMENT	The Datalayer environment a sandbox is launched into; default ai-agents-env. Set it where a cluster offers different ones, such as a GPU environment — the environments differ per cluster, and picking one that does not exist fails at the first cell rather than at startup
JUPYTER_MCP_TOKEN_VERIFIER_CLASS	How the worker authenticates clients; datalayer_mcp_server.verifier:DatalayerTokenVerifier, which accepts OAuth access tokens and personal access tokens
DATALAYER_CORS_ORIGIN	The browser origins allowed on the REST routes, comma-separated; default *. See below — this variable took the console down once
PORT	The port uvicorn listens on inside the pod; default 4404, and what the chart's mcpServer.port must agree with
LOG_LEVEL	Accepted in either spelling; an unknown value falls back to INFO rather than stopping the service
DATALAYER_MCP_LOG_COLOR	Whether the request log is coloured; default true, and NO_COLOR turns it off
standard DATALAYER_JWT_* variables	DATALAYER_JWT_SECRET, _ALGORITHM, _ISSUER, _CACHE_VALIDATE — token verification, shared with IAM
standard OTEL_* variables	OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, DATALAYER_OTEL_API_KEY, OTEL_PYTHON_LOG_LEVEL, OTEL_SDK_DISABLED. Validated and reported by health; see Observability for what is exported today

One variable is set by the gateway rather than read by it: JUPYTER_MCP_STATEFUL=true is put into every worker's environment. The open source server defaults to stateless Streamable HTTP; a worker holds one user's session and must not, so the gateway states it rather than relying on a default it does not own.

Do not place the IAM API key, the JWT secret or the Solr password in chart values. Use mcpServer.envValueFrom with Kubernetes Secret references.

DATALAYER_CORS_ORIGIN, and how it fails

The console reads this gateway from the browser, so every REST route is subject to CORS. When the origin is wrong the failure does not look like a CORS failure: the browser refuses the request before sending it, the fetch rejects with no status, and the page says "The MCP server did not answer" — while the gateway is up and answering.

Set it to the origins that actually load the console, comma-separated:

env:
  - name: DATALAYER_CORS_ORIGIN
    value: "https://datalayer.ai,https://prod1.datalayer.run"

Include every origin that loads the console, the development server among them. Working against a deployed plane from http://localhost:3063 is the normal way to develop here, and that origin is as much a browser origin as the production one:

DATALAYER_CORS_ORIGIN: "https://datalayer.ai,https://prod1.datalayer.run,http://localhost:3063"

A localhost origin on a production gateway

It is a real widening: with allow_credentials, any page served from that port on any developer's machine may call this gateway with the caller's token. That is the trade for developing against a deployed plane, and it is worth making deliberately — on a plane with real customer data, prefer a separate development deployment to adding localhost to this list.

An empty value is not an unset one

DATALAYER_CORS_ORIGIN: "" is not the same as leaving it out. The default applies to a variable that is absent; a variable set to the empty string is a value, and it used to become an allow-list of one entry that matches no origin — refusing every browser there is.

That is exactly what happened: /api/mcp/healthz answered 200, /api/mcp/v1/activity answered a correct 401, and every preflight answered 400 Disallowed CORS origin. A blank value now falls back to the default instead, but a Helm value that renders empty is still worth spotting.

* turns credentials off, necessarily

The CORS specification forbids a wildcard origin on a credentialed request, and Starlette honours it by answering the wildcard without the credential header — which the browser then drops. So a wildcard cannot be combined with allow_credentials, and the gateway turns credentials off when the wildcard is in force rather than sending a pair no browser accepts.

For anything that has to carry credentials, name the origins.

To check a deployment from a terminal, ask for the preflight the browser would:

curl -si -X OPTIONS https://r1.datalayer.run/api/mcp/v1/activity \
  -H 'Origin: https://datalayer.ai' \
  -H 'Access-Control-Request-Method: GET' \
  -H 'Access-Control-Request-Headers: authorization' | head -3

200 with an access-control-allow-origin naming your origin is right. 400 Disallowed CORS origin is this variable.

Logs

plane logs datalayer-mcp-server
kubectl logs -n datalayer-api -l app=mcp-server -f

The gateway logs one line per proxied call, naming the user and the agent that asked — a person may have several connected, and both matter when reading back why something was refused — plus a ⇢ line whenever a request is forwarded to another replica, and the reason for every refusal. A worker's stdout is relayed into the gateway's log as it arrives, with the last two hundred lines kept so that a worker which dies has its explanation quoted in the death message. The startup banner names the resource URL, the replica, where notebooks, sandboxes and Contents are, which store is in use and where telemetry is addressed.

Log lines do not carry trace_id or a task id yet: the correlated structured logging of the plan is not implemented.

Observability

The chart addresses the OTEL collector with the standard OTEL_* variables, and the metric catalog is declared in one place — datalayer_mcp_server/telemetry.py — with deliberately bounded labels: a tool, an outcome, a dependency, a refusal reason. A user, an agent, an organization or a sandbox is never a metric label; those belong in access-controlled spans and in the audit rows.

Every audit row names the trace its call ran in. The row's trace_id is what the Observability view and datalayer mcp trace start from. It was read from the client's traceparent alone, and most MCP clients send none — so until 2026-09-05 their rows carried no trace id while the gateway's own spans for the same call were exported under a trace nothing named, and the view had an empty state for every one of them. A client's traceparent still comes first (its span is the parent of ours); without one, the row takes the server span the request is served in, which is the trace the worker's spans hang under too, since the call to the worker carries its context. A malformed header is no header, and outside any span — telemetry off, a test process — the row names nothing rather than inventing an id.

And the span is filed where the person can read it. The OTEL service scopes every query to an account and files a span under its usage_account_uid attribute, else under the exporter's — this gateway's service account. So a person querying the trace their audit row names got 404 for spans that existed. mcp.request and mcp.policy now carry the organization the call was made in, else the person: a per-person value is fine on a span, which is one access-controlled record, where it is forbidden on a metric (see account_labels against span_account_labels). The ASGI root span, the http send/receive spans and the worker's stay under the gateway's account — the person sees the gateway and policy spans of their call, a platform administrator (?account_uid=) sees the whole trace. Measured on prod1: the row has its trace id within 4 s of the call, and the span is queryable under the person's account after 19–40 s, which is the collector's and the consumer's batching. The route is /api/otel/v1/traces/{id} without a trailing slash — with one it is a 404 for a trace the listing route just returned, which datalayer-core's client sent until 2026-09-05.

How the spans are read. The Observability view draws a trace twice: as the named tree it always was, and above it as bars laid against one clock — proportional, nested by parent_span_id, coloured by service. The tree gives the durations, the bars give the shape, and the gap between a parent's bar and its children's is time the call spent waiting rather than working, which a table of durations hides. The same drawing sits under the output on a run's own page. Span names are labelled by prefix, not by service, because one service emits spans for more than one stage: mcp.request is Gateway, mcp.policy Policy, sandbox.* Sandbox, durable.* Workflow and runtimes.* Runtime. Workflow and Runtime are kept apart on purpose — a durable run that is slow is either waiting on the engine or waiting on the sandbox, and those are different services to go and look at. A span name nothing recognises is drawn and timed with no stage label rather than a wrong one.

Turning execution over to the engine

Deploying Durable does not move any work to it. The gateway calls it for cancel, signal and operations, and until DATALAYER_MCP_DURABLE_TOOLS names a tool it calls start for nothing: every task runs in the worker that accepted it, which is what a task did before Durable existed and what it goes on doing after.

DATALAYER_MCP_DURABLE_TOOLS=execute_code moves that one tool. The gateway answers such a call itself — with the task handle the client polls on — and never forwards it to the worker, which is what makes "both ran it" a state the design cannot reach rather than one a setting protects against.

Every reason not to route is a fallback to the worker, never a refusal of the call: a synchronous call, a session with no sandbox, a call carrying an idempotency key (only the worker's task store can recognise a retry of one), an engine that cannot be named. Declining costs a task that does not survive a pod restart. Starting one wrongly costs a cell executed twice, which is why the engine's own execute step is marked not idempotent.

execute_code and not execute_cell: execute_code carries the code in its own arguments, so what the engine runs is what the client sent. execute_cell names a cell, and routing it would make the gateway a second reader of which cell is meant against a document that can change between the read and the run — a run of the wrong code, reported as a success.

The Live pane asks /api/mcp/v1/operations/workflows how the durable engine is and lists the runs still working. The engine has three states, not two: healthy, unhealthy — the engine says so — and unreachable, which is this gateway not being able to ask. That is why the route answers instead of returning 500, and collapsing the two would send an operator to the engine when the problem is the network to it. A healthy engine with a queue stays healthy and reports the depth; a queue is a queue, not a fault. durable.* spans will appear in these traces when a run first goes through the durable service — nothing starts one yet, so today those rows draw nothing.

The table below is generated from telemetry.CATALOGUE, which a test holds against the instruments actually created — in both directions — and against the label values actually passed. It was hand-copied once and drifted three ways: it listed ten of these twenty, it called mcp.calls "declared, not written yet" after it had a call site, and it named five of the mcp.refusals reasons. A refusals panel built from that would have drawn scope and rate refusals, missed every policy, quota and capability refusal, and looked complete.

dependency is one of (audit, contents, iam, otel, runtimes, spacer, store, tasks, workflows) everywhere it appears. workflows is where durable work goes — datalayer-durable, or the in-process fake — and it is the dependency that decides whether a long-running call survives anything at all.

The worker exports too, since 2026-09-05, and under the caller. It exported nothing before: mcp-worker was absent from traces/services/list under every account, so a trace showed the gateway's request and policy spans and stopped at the boundary where the work happens. The open source server already emits a span per tool call and per kernel execution through a hook handler — into a file, which is right for its own tests and useless here — so worker_main installs OTLP providers as mcp-worker and registers that handler against them. The export credential is the platform's, so the supervisor names the account: DATALAYER_USAGE_ACCOUNT_UID on the worker's environment becomes the datalayer.usage_account_uid resource attribute the OTEL consumer files by. Without it every worker span lands under the gateway's service account, where the person whose call it was cannot read it.

And one call is one trace. The gateway injects W3C traceparent from the mcp.request span into every forward, and the worker extracts it — the seam is the server's own streamable_http_app, whose app is wrapped once in the standard ASGI middleware, because this image has no global Starlette instrumentor and the open source serve() builds its own app. A single list_notebooks on prod1 is ten spans in one trace: mcp.request and mcp.policy from jupyter-mcp-server, POST /mcp and tools/call list_notebooks from mcp-worker. Before this the view showed the gateway's two and then an orphan.

An organization reads its agents' work, not only its gateway's. A worker is one process per user, so its exporter can only ever name the person — and a call made in an organization belongs to the organization, which is where the gateway files its own spans. The two disagreed, so the same trace had its halves under two accounts. The account now travels on the request, the only place it is known (x-datalayer-usage-account-uid), and a span processor stamps every span the worker makes: the OTEL service files each span on its own attributes, so stamping only the request would leave the work itself under the wrong account. Walked on prod1 with a service agent, whose token carries org_uid: one call, nine spans, all readable under the organization.

The gateway exports with its service credential, so the OTEL service files every point under the gateway's own account. Five instruments carry one more label when the call was made in an organization: usage_account_uid, the organization's uid, which the OTEL service files that point under instead — so an organization owner reads their agents' calls, tasks and refusals on the OTEL service's "agents by organization" dashboard, and a platform administrator reads the platform's on the gateway's own account. Organizations only: a personal-scope call carries no account label. A per-person label would be a per-person time series, the one label these conventions forbid.

Instrument	Kind	Labels	What it says
mcp.calls	counter	tool, method, outcome; usage_account_uid when the call was made in an organization	Tool calls the gateway forwarded
mcp.call.duration	histogram	tool; usage_account_uid when the call was made in an organization	How long a tool call took, end to end, in seconds — the latency SLI, its p95 read from the buckets
mcp.tasks	counter	status; usage_account_uid when the call was made in an organization	Tasks that reached a terminal state — a rate, because an SLO is written against one
mcp.task.duration	histogram	tool; usage_account_uid when the call was made in an organization	How long a task ran before reaching a terminal state
sandbox.launch_seconds	histogram	provider, environment	How long a sandbox took to become usable
mcp.dependency.ready	gauge	dependency	1 or 0 per dependency, so an alert need not wait for a readiness scrape
mcp.refusals	counter	reason (see the reason values below); usage_account_uid when the call was made in an organization	Calls refused before they reached a worker
mcp.limit.unenforced	counter	limit — allowedProviders, environmentCapabilities, gpuHoursPerMonth, maxConcurrentSandboxes, maxCreditsPerDay, storageBytes; reason — incomplete, unreadable	A limit not applied because the figure it compares against could not be read
mcp.workers	gauge	state	Worker processes on this replica, set on every directory heartbeat
mcp.worker_start_seconds	histogram	provider	How long a worker took to answer after being started
mcp.forwards	counter	outcome	Requests forwarded to another replica
mcp.dependency.duration	histogram	dependency, outcome	How long one call to a dependency took
mcp.dependency.timeouts	counter	dependency	Calls to a dependency that ran out of time
mcp.readiness.failures	counter	dependency	Readiness probes that found a required dependency down
mcp.sandbox.lost	counter	provider, then	Sessions whose sandbox was found gone
mcp.audit.writes	counter	decision	Audit rows written to the ledger
mcp.audit.write_failures	counter	decision	Audit rows that could not be written — the one number that says the security record has a hole in it
mcp.audit.deleted	counter	reason	Audit rows deleted past their retention
mcp.jobs	counter	job, outcome	Periodic job ticks — skipped is the ordinary outcome on a replica without the lease
mcp.job.duration	histogram	job	How long one periodic job took

mcp.refusals's reason values: access_check_failed, access_revoked, approval_unavailable, attribution_failed, capability_missing, environment_capability, execution, grant_revoked, header_body_mismatch, item, policy_unavailable, relaunch_from_snapshot, runtimes_unavailable, sandbox_lost, scope, transport_direct, unknown_handle, policy:…, rate_limited:…, quota:….

Two of those are worth a panel of their own. approval_unavailable is a tool a policy asks a person to approve on a deployment with no durable engine to hold the call on — it reads to whoever hit it as an agent being blocked, and it is a deployment missing a piece. execution is a task token asking about a notebook its execution's manifest does not name, decided by the execution layer rather than by any policy layer, so no Policies page explains it.

Exporting depends on one setting

telemetry.start_exporting installs the SDK providers through datalayer_common.instrumentation.instrument, which also brings the ASGI server span — so a client's traceparent is continued rather than replaced, and the calls to IAM, Spacer, Contents and Runtimes hang under the request that caused them. The gateway registers as jupyter-mcp-server and a worker's relayed lines as mcp-worker, which is why the Instrumented Services list shows two.

The gateway's service.name is not the deployment's name, and deliberately: renaming it when the service became datalayer-mcp-server would have orphaned every trace, dashboard and alert that already names it. The worker's was jupyter-mcp-worker and was renamed, because nothing had been filed under it — it had been exporting for a day.

Without OTEL_EXPORTER_OTLP_ENDPOINT it exports nothing, on purpose: the SDK's default is localhost:4317, which in a deployment with no collector is a closed port the exporter would retry against forever on a background thread. It says so once at startup — 📡 Telemetry is not exported — rather than retrying quietly. OTEL_SDK_DISABLED=true turns it off explicitly.

Nothing here can stop the gateway. An unreachable collector, a missing SDK or a provider that raises is logged and the gateway serves without telemetry; telemetry is never a readiness dependency. Which does mean a misconfigured endpoint looks identical to a healthy one from outside, so check the startup line rather than assuming.

The dashboards are the OTEL service's built-ins (see the OTEL service's Dashboards): service levels — the four SLIs, which the Observability view and datalayer mcp metrics read rather than adding metric points up, since the gateway exports running totals — and one per marketed area. The alert rules run in the gateway (jobs.evaluate_alerts) over its own stores: open and failed tasks, audit write failures, agent spend and budgets. Still missing: a PrometheusRule in this chart. A gateway outage is caught today by the ordinary Kubernetes deployment and ingress alerts, not by an MCP-specific one.

Policy layers

What an agent may do is decided by five layers, narrowest last. The gateway fetches the last four from IAM on the way to a call and caches each for thirty seconds.

Layer	Written by	Applies to
platform	nobody — it is the token's own scopes	Every call. No layer below can add a scope the grant did not carry
organization	an organization owner	Tokens carrying that org_uid
team	a team owner, or the owner of the organization above it	Tokens carrying that team_uid
personal	the person themselves	Tokens with no org_uid. A person does not loosen or tighten what their organization decided about work done in its name
execution	nobody — IAM computes it from the task grant the token was exchanged from	Tokens carrying task_grant_uid: an orchestration worker's. It narrows whatever the layers above allow to what the execution's manifest names

The execution layer is read differently from the stored ones (ORCHESTRATOR.md, O1-17). A stored layer nobody wrote narrows nothing. An execution layer that cannot be read, or whose grant has ended (live: false), refuses every call, because the person's layers are exactly what the manifest narrows. It is asked in standalone mode too, because a task token implies the platform's IAM. Its toolAllowlist admits only the tools the manifest's references reach, and an empty one admits none. Its authorizedResources are checked wherever a call's notebook is known:

• use_notebook binds a notebook only when the manifest names it;
• a cell tool acts only on a bound notebook the manifest names with that access (read, or write for edits and execution);
• a cell tool about the active notebook is refused while none of the execution's notebooks is bound.

Those refusals carry reason: execution and decided_by: execution. A token with task_grant_uid and no scope is granted nothing, where a token with neither is a person's whole authority. GET /api/mcp/v1/policy folds the layer in the same way, so the Policies page says what is enforced.

Which organization an agent acts in

org_uid on the token is what all of this keys on — the layers here, the quotas, the alerts and the audit scoping. It is chosen at consent and stamped by IAM into the access token as a top-level claim, beside scope and client_id.

Chosen rather than inferred: somebody may belong to three organizations, and which one an agent acts in is a fact about that grant. A token naming none is personal scope — what a person with no organizations gets, and what every OAuth token was before IAM issued this claim.

The claim is earned, never taken

IAM checks membership when the authorization code is issued, and a refused claim never reaches the store. A person who could name an organization they do not belong to would read its policy, spend against its budget and write rows into its audit.

A team additionally needs membership of the team and must be inside the named organization — a team uid in a request is a claim about a team, not proof of which organization it is in.

It rides on the grant, so a refresh mints the same scope: a refreshed token that quietly lost its organization would move an agent from its employer's policy to no policy at all, with nobody asked. The token exchange carries it for the same reason.

Where the choice is made: the approval screen asks, above the scope list, whenever the person belongs to at least one organization — a single connection acts in one place, so connecting the same client twice is how somebody covers two. Personal scope is first and preselected, because acting in an organization applies its policy, spends its budget and writes its audit, which is a thing to opt into rather than be defaulted into. The list is fetched from IAM rather than read from the application's own store: this page is a surface for a decision IAM makes and checks.

# What is in force for this token, rule by rule, with who decided each
curl -sH "Authorization: Bearer $DATALAYER_TOKEN" \
  https://r1.datalayer.run/api/mcp/v1/policy | jq '.rules'

# Writing one, as an organization owner
curl -X PUT -H "Authorization: Bearer $DATALAYER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"toolDenylist": ["execute_cell"], "maxCallsPerMinute": 30}' \
  https://iam.datalayer.run/api/iam/v1/mcp-policies/organization/$ORG_UID

The team layer is written on the console's Teams page — by a team owner for their own team, or by an organization owner for any of them, since an owner who could not narrow one of their own teams would have to remove the team to do it.

The personal layer is written by each person at /settings/policies, over the same form and the same six rules. It only narrows: a personal value wider than an organization's changes nothing, and the page says so beside the field rather than refusing it — the same layer governs that person's personal-scope work, where no organization narrows anything, so a refusal would make their own policy unwritable because of their employer.

Four rules are enforced. Nothing else may be stored — IAM refuses a rule the gateway does not act on, so a policy page cannot promise something that never happens, and toolDenyList is rejected as a typo rather than kept as a setting that does nothing.

Rule	Type	Combines by
toolDenylist	list of tool names	Union — a layer may add a denial, never lift one
toolAllowlist	list of tool names	Intersection. An empty list as written is not an allowlist; a half-filled setting must not refuse everything
allowedClients	list of CIMD URLs or hostnames	Every layer's list must admit the client. An empty list is not an allowlist, same reason
maxCallsPerMinute	whole number, at least 1	The smallest wins
maxCreditsPerDay	a positive number of credits	The smallest wins
maxConcurrentSandboxes	whole number, at least 1	The smallest wins as a reported figure; enforced per scope — see A team's concurrency is counted against the team
gpuHoursPerMonth	a positive number of GPU-hours	The smallest wins; enforced per scope, and only on a launch onto an environment that has a GPU — see The GPU-hour limit counts more than agents
storageBytes	a positive whole number of bytes	The smallest wins; enforced per scope on every launch, because every sandbox can write — see The storage limit is checked on every launch

Owners write these on the Policy page of the organization's MCP tab, or through the API above. The page's fields are generated from the same rule list the gateway enforces, so it cannot offer a rule that would be thrown away at the write; a security auditor reads the page but does not get the form.

A write carries the version that was read, so two owners editing at once get a 409 rather than one silently overwriting the other. The page shows that as a banner with a "read it again" button — the losing draft is still on screen, and a toast would vanish before it could be acted on.

maxCallsPerMinute: 0 is refused, and why

The gateway reads a non-positive cap as no cap. A zero written to stop an organization's agents would lift its limit instead — the one value whose stored meaning is the opposite of its plain reading. To stop an agent, revoke its grant or deny the tools.

Service agents

An agent that belongs to an organization rather than to a person: a nightly pipeline, a CI job, a bot. It holds its own key, spends the organization's budget under its own name, and appears in the audit as agent_uid rather than under whoever happened to create it.

The reason to have them rather than telling people to use a personal token: an organization whose pipeline runs under an engineer's grant loses the pipeline when the engineer leaves, and its spend shows against somebody who was asleep. Naming the agent as itself makes revoking, budgeting and auditing it ordinary operations rather than archaeology.

From the CLI, which is where an operator setting up a pipeline already is:

datalayer mcp service-agents create $ORG_UID \
  --name "nightly ingest" --scopes "runtimes:read data:read"
datalayer mcp service-agents list $ORG_UID
datalayer mcp service-agents rotate $ORG_UID $AGENT_UID
datalayer mcp service-agents revoke $ORG_UID $AGENT_UID

rotate and revoke confirm first (--yes to skip), because rotation breaks whatever holds the old key with no grace period. --output json puts the key in a field rather than in a sentence, for a pipeline that captures it. Or over HTTP:

# Create one. The key is in this answer and in no other.
curl -X POST -H "Authorization: Bearer $DATALAYER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name": "nightly ingest", "scopes": "runtimes:read data:read"}' \
  https://iam.datalayer.run/api/iam/v1/organizations/$ORG_UID/mcp-service-agents

# What acts in this organization, revoked agents included
curl -sH "Authorization: Bearer $DATALAYER_TOKEN" \
  https://iam.datalayer.run/api/iam/v1/organizations/$ORG_UID/mcp-service-agents \
  | jq '.agents[] | {name: .name_t, scopes: .scopes_s, revoked: .revoked_b}'

# Rotate. The previous key stops working with this call.
curl -X POST -H "Authorization: Bearer $DATALAYER_TOKEN" \
  https://iam.datalayer.run/api/iam/v1/organizations/$ORG_UID/mcp-service-agents/$AGENT_UID/rotate

# Revoke. The agent stays listed, for its audit.
curl -X POST -H "Authorization: Bearer $DATALAYER_TOKEN" \
  https://iam.datalayer.run/api/iam/v1/organizations/$ORG_UID/mcp-service-agents/$AGENT_UID/revoke

The key is shown once and cannot be recovered

IAM stores a SHA-256 of it and has no way back. Somebody who loses a key rotates; there is no read that returns it.

That is deliberate. The alternative is a credential that spends an organization's budget sitting readable in a console for the life of the organization, liftable by anybody who can reach the page. One support conversation is the cheaper end of that trade.

A team named on a service agent must be this organization's

IAM checks it, and refuses a team it cannot read rather than admitting it — the opposite of the rule elsewhere in this system, where an unreadable dependency lets the work through.

The reason for the inversion: nothing is running, nobody is blocked, and what is being decided is which organization a long-lived credential belongs to. An agent filed under another organization's team would work, while spending their budget and counting against their sandbox limit.

Who may manage them: organization owners and platform_admin, and nobody else — not a team owner, unlike an alert rule scoped to their own team. A rule watches; a service agent holds a credential that spends the organization's budget and carries its name, so delegating it is delegating the organization's authority rather than administration of a slice.

Organization members may read the list, revoked agents included. An agent spending their organization's budget is one they will ask about, and a revoked agent hidden from the listing is invisible to whoever is deciding whether it is still needed while its audit rows still name it.

What an agent may be granted is a subset of the OAuth scopes: code:execute, notebooks:read, notebooks:write, runtimes:read, runtimes:write, data:read, sandboxes:manage. The scopes about a person — reading their profile, acting as them — mean nothing for a principal that is not one, and accepting them would grant something that silently does nothing. An agent with no scopes is refused outright: it is a key that fails at its first call, issued as though it worked.

The first three were missing until 2026-09-05. A service agent could be handed a sandbox — sandboxes:manage — and could not run a line on it, because execute_code is the one tool that costs code:execute and the gateway takes an agent's scopes as they are; a pipeline that runs a notebook with its own key, which is what a service agent is for, needs all three. Found by walking the shared-sandbox scenario on prod1 with a real service agent.

How a key is checked, and what that costs

The gateway cannot verify a key by itself the way it verifies a JWT, so every call carrying one is a lookup against IAM's POST /api/iam/v1/mcp-service-agents/authenticate. That endpoint is for the gateway alone — it takes a credential and answers who it is, so anything that may call it may test keys against it — and the gateway proves itself with DATALAYER_SERVICE_API_KEY in X-API-Key before the key in the body is looked at.

Successful lookups are cached	60 seconds. An agent in a loop would otherwise put IAM in its inner loop
Refusals are cached	Never. A reinstated key, or one whose rotation raced a call, must not keep failing for a minute with nothing to say why — and a cached refusal makes the cache a place to park a negative answer for a real key
IAM unreachable	Refused. Not the last good answer

A revocation takes effect within a minute, not instantly

That is the honest cost of the cache, and it is why the window is a minute rather than an hour.

Note the difference from the policy cache, which does keep its last answer through an IAM outage: an outage is not a policy change, and reading it as "no restrictions" would leave every agent briefly unconstrained. Authentication is the other way round. Serving a stale principal during an outage makes the revocation window as long as the outage — revoke a leaked key, IAM has a bad afternoon, and the key works all afternoon. A refusal is the failure somebody can see.

A principal missing its organization is refused

An empty org_uid is personal scope, which is not what a service agent is: it would run under no policy, spend against nobody's budget and write its audit where nobody looks. The gateway refuses rather than running it that way.

The two sandbox quotas

maxCreditsPerDay and maxConcurrentSandboxes bound different things, and an organization usually wants one of them: the first is what may be spent in a day, the second what may run at once. "Ten agents, each on a small box" is a sentence about the second.

Both are checked before a launch and nowhere else. A call on a sandbox that already exists costs nothing new, and refusing every one would read as an outage rather than as "your agents are out of budget for today" — which is the one somebody can act on. A session that already holds a sandbox is answered it without the budget being consulted, since refusing there would stop an agent using what it is already paying for.

At the budget, not over it: a budget of 100 that admits the launch taking you to 140 is a budget of 140.

A team's concurrency is counted against the team

The one place a quota does not behave like the rest of the policy. Every other rule narrows: the effective maxCallsPerMinute is the smallest any layer set, one number, checked once. A counted quota cannot work that way, because the count belongs to a scope.

So both are checked, each against its own count:

Limit set by	Counted over
The organization	Every sandbox in the organization
The team	Only that team's sandboxes

Taking the team's limit and the organization's count would refuse a team running nothing because other teams are busy — a refusal its owner can neither understand nor act on. Taking the team's count against the organization's limit would let ten teams of nine walk past a cap of ten. Checking both is what "a team has its own quota within the organization's" has to mean to be usable.

The organization's answer comes first when both are full: it is the one that holds however the team is configured, and the one an owner rather than a team owner acts on.

A sandbox with no team belongs to no team, and is counted only for the organization. Counting it would make the first team to launch carry the whole organization's teamless work against its own cap.

The daily credit budget works the same way, and for the same reason: a budget is a sum, and a sum belongs to whoever is billed for it. A team is a billing entity with its own delegated wallet, so agent_credits(team_uid) is a real figure rather than a share of the organization's. Each scope's budget is weighed against that scope's own spend, organization first.

Both limits are checked at the reservation too

The gateway checks them before a launch, and that is not the only door: a client reserving directly, or another service doing it, would otherwise walk past a limit an organization set — while the limit looked enforced, because it is, on the path most people take.

The storage limit is checked on every launch

storageBytes refuses a launch when the scope has stored as many bytes as it is allowed. Unlike gpuHoursPerMonth, there is no condition on the environment, and that is not an oversight: a GPU-hour limit has nothing to say about a CPU launch because a CPU launch cannot use a GPU, while a storage limit has something to say about every launch because every sandbox can write — there is no environment without a home folder. Conditioning it on the contents a launch attaches would miss the writes it makes anyway.

It counts every live object version, not every object. An object's recorded size is its current version's; a scope keeping a hundred versions of a one-gigabyte file stores a hundred gigabytes. Deleting the newest version of a file does not free what the older ones hold, and the refusal says so.

The number comes from Contents rather than from IAM, which is the opposite of how the credits budget is read and for the matching reason: credits are IAM's ledger and there is exactly one of it, while the bytes are Contents' and it is the only service that can count them. GET /api/contents/v1/sources/storage-usage/{principal_uid} lists the principal's sources and sums the versions in them with one json.facet — bounded reads, because an organization's object count has no bound and paging it on a launch path would be slowest for exactly the tenant a quota is bought for.

The rule is in bytes rather than gigabytes. A number whose unit is implied is a number somebody eventually reads as the other unit, and this is compared against a figure reported in bytes; the refusal converts for the reader.

A figure that could not be read, or that Contents reports as short — a principal with more sources than one reading walks — lets the launch through and counts mcp.limit.unenforced{limit="storageBytes"}, as every limit on a launch path behaves.

The GPU-hour limit counts more than agents

gpuHoursPerMonth refuses a launch onto an environment that has a GPU when the scope has used its hours. Two things about it will otherwise be discovered from a refusal.

It counts every GPU-hour billed to the scope, not only the ones an agent asked for. A notebook somebody ran by hand on a GPU is in this number. The agent dimension lives on a child document in the usage ledger (client_id_s and agent_uid_s under metadata), and a child field cannot be a clause of a facet over a parent field — so the sum that makes a monthly window safe is the one that cannot be narrowed. It is the stricter reading, which is the right direction for a quota, and the refusal says so rather than leaving somebody to reconcile it against what their agents did.

The window is a trailing 30 days, not the calendar month. A limit that resets on the 1st can be spent twice inside 48 hours, once on each side of the reset, with the scope under its limit at every instant — the same reason maxCreditsPerDay is a trailing day. It will not line up with an invoice period, and it is not meant to.

Why it is not expressed in credits: the credits a GPU hour costs differ per environment, so neither number can be derived from the other. An organization that has bought forty GPU-hours a month cannot say that as a credit budget.

The number comes from gpu_hours_f, which Runtimes writes onto a reservation when it ends — the only moment both halves are known, since the GPU count has been on it since it was made and the duration exists only then. IAM sums a window with one json.facet at GET /api/iam/v1/mcp-agent-gpu-hours/{org_uid}, so the answer is one row back whatever the window holds. That matters: the paged approach used for credits reports itself short past 2000 records, and an organization busy enough to pass 2000 reservations in a month is exactly the one a GPU quota is bought for — it would have failed the quota open for the customers who paid for it.

A launch onto a CPU environment is never refused by this rule and never even asks IAM for the sum. When it cannot be told whether an environment has a GPU — an unreadable catalogue, or one the environment is missing from — the launch goes ahead and mcp.limit.unenforced{limit="gpuHoursPerMonth"} counts it, as every limit on a launch path behaves.

IAM applies maxConcurrentSandboxes and maxCreditsPerDay when the reservation is made, reading its own policy and its own ledger with no round trip. Only to agent reservations, identified by the client_id_s/agent_uid_s dimension: both are rules about agents, and refusing a person launching their own runtime would refuse somebody under a rule that was never about them. Only the organization's layer, since both figures are per billing entity — a team's number against an entity-wide count or spend would refuse a team holding nothing, and the gateway, which knows which team a caller acts for, is where the team layer belongs.

The count is asked before the spend: the count is a rows=0 read and the spend reads the day's usage records, so an organization already at its sandbox cap does not pay for a lookup to learn what the count knew.

A refusal at either is a 409 naming both numbers and who can raise the limit.

The two doors have to agree

Otherwise the same launch is refused at one and admitted at the other, and which one you hit is an implementation detail of the client you happen to be using. So the reservation follows the gateway exactly: at the budget rather than over it, and neither an unreadable budget nor an unreadable spend refuses.

A short sum does not refuse either, and it is the sharper case. It always reads as under budget, so the tempting move is to enforce on a partial figure that is already over the limit — but that figure is not the number. The reason it is unknown is that rows were dropped, and the total could be anywhere above it. Both cases are logged, because a limit that is not being enforced looks exactly like one nobody set.

One unreadable count does not disable the other

Each count that cannot be read is skipped with a line in the log, and the other still holds. A team at its cap is still at its cap when the organization's count times out.

The daily credit budget is still organization-only: agent_credits sums a billing entity, and a team's share of a ledger is a different question from a team's count of running sandboxes.

GET /api/mcp/v1/organizations/{uid}/usage puts the use beside the limit — spend from IAM's ledger, running sandboxes from mcp-gateway, the limits from the policy. It is the page for "are we near a limit, and which one".

Unknown is not absent

A figure the page could not read is reported unknown by name rather than left out. Both render as no number beside a heading and they mean opposite things: one is "no limit set", the other is "we cannot tell you what your limit is". A reader who confuses them either relaxes about a quota they are close to, or chases one that does not exist.

An unknown figure gets no fraction either — a share of a number nobody read is a number somebody will act on. And one unreadable figure does not lose the page: it is opened when something is already wrong.

No route on this gateway writes a quota. Limits are policy rules and IAM owns that document; a second way to write it would be a second answer to what an organization is allowed. The usage page names where they are set.

Which agent spent it

The same answer carries byAgent: the day's credits grouped by the principal that spent them, biggest spender first. A budget says whether to act. It does not say what to turn off, and that is the question an administrator at ninety percent of a limit is actually asking.

It costs no extra query. IAM was already reading the per-agent usage records in order to sum them, and discarding the dimension that says who — so "what did this organization spend" had an answer and "which agent spent it" did not. The grouping happens over the rows the total is summed from, in the same read, so the two cannot disagree across the seconds between two queries.

The key is agent_uid, then client_id

Service agents reach Runtimes through the gateway and carry the gateway's client id. Grouped on that, every one of an organization's pipelines collapses into a single row costing the sum of all of them, with nothing to say which one to look at. So a row is keyed on agent_uid where there is one, and on client_id — claude, cursor — for a person's delegated agent.

Where the dimension shows up

The same two keys answer three questions in three places, and they are worth knowing together because a report that disagrees with a table is usually one of them reading the pair in the other order:

Surface	Reads
GET …/organizations/{uid}/usage	byAgent, the day's credits grouped
The console's Usage page	The same answer, biggest spender first
The Reservations table in the web UI	A Started by column per running runtime

The reservations table gets it for free: IAM already puts the metadata on every usage record it serves, so the dimension was arriving in the browser long before anything displayed it.

agent_uid_s is read before client_id_s, everywhere

A service agent reaching Runtimes through the gateway carries the gateway's client id. Read the other way round, every one of an organization's pipelines is the same agent — one row costing the sum of all of them, with nothing to say which to look at.

This holds in IAM's grouping, in the gateway's answer and in the table. If you add a fourth reader, read them in that order.

The runtimes table does not have this column

RuntimesTables builds its rows from the runtime record, which the Operator assembles from pod labels and annotations — the reservation's metadata is not among them. A Started by column there needs the dimension propagated onto the pod at launch, across the gateway, Runtimes and the Operator.

The reservations table has it because that view reads usage records, which is where the dimension lives.

A reservation carrying no agent dimension is somebody's own work, not an agent's. It is skipped rather than filed under unknown, which reads in a table like an agent by that name; it stays in the organization's total, which is the bill.

An unreadable total carries no breakdown

Where IAM summed fewer records than it holds, the total is unknown and byAgent is empty — not partial. The parts under an unreadable whole make every agent look cheap and the reader concludes the budget is fine, which is the one direction a short sum always errs in and the one nobody checks.

The concurrency count is asked first. It is a Solr read this process already makes, where the budget is a round trip to IAM — so an organization at its sandbox limit does not pay for a spend lookup to be told what the count already knew. It is counted with rows=0, so an organization with two hundred sandboxes costs what one with two costs.

The refusal carries its own JSON-RPC code and retryable: false — retrying tomorrow works, retrying now does not, and a client that cannot tell will loop. It names the layer that set the budget, so a team owner who set 20 is not sent to the organization's settings page to find out why.

Neither quota refuses on a number it could not read

The opposite of what a budget implies, and deliberate. An organization whose agents stop working because a billing lookup timed out is a worse failure than one that spends a little over on the day IAM was down.

An incomplete spend figure is treated the same way and for a sharper reason: a sum missing rows always reads as under budget, so enforcing on one refuses launches somebody is entitled to or admits ones they are not — and which depends on rows nobody saw. A sandbox count that could not be read is the same trade.

Both are logged. A budget that is not being enforced otherwise looks exactly like one nobody set.

Two allowlists that do not overlap admit nothing

An organization allowing read_cell and a team allowing execute_cell between them allow no tool at all. Each reads fine alone. GET /api/mcp/v1/policy says so in the rule's reason rather than rendering an empty list, because the administrator is looking at two sensible settings.

Reading is wider than writing on purpose: an organization member may see what their organization's policy is, because "why was my agent refused" is a question every member of it will ask, and an answer only owners can see is an answer nobody gets. Every change writes an IAM audit row naming who, when and which rules — by name, never by value.

A limit that stopped being enforced

mcp.limit.unenforced is the one metric here that counts something not happening, and it is the one to alert on.

The gateway does not refuse when it cannot read the figure a limit compares against: an organization whose agents stop launching because a billing lookup timed out is a worse failure than one that spends a little over. That is the right call, and it has a consequence — the limit is still on the Policy page, the refusals just stop, and an organization whose budget is no longer being enforced looks exactly like one that never set a budget.

It is not mcp.refusals

A refusal is the limit working. Folding the two together would make a gateway that refuses nothing because it can read nothing look like a gateway with nothing to refuse.

The two reasons are different problems:

reason	What happened	What to do
unreadable	The lookup failed — IAM down, Solr down, a timeout	Look at the dependency; mcp.dependency.ready says which
incomplete	The ledger holds more rows than one reading sums	The organization is busy enough to exceed the read bound, which is exactly the one the limit was set for

incomplete is the sharper of the two. A quota that fails open there fails open for the customers who bought it and holds for everybody else — a quiet failure of the worst shape, since the page still shows the limit.

Alert rules

An McpAlertRule is an IAM organization setting: a condition, a comparison, a threshold and a window. The gateway evaluates every enabled rule on the platform once a minute, records a firing in mcp_alert_event and delivers it as an ai-agents notification.

None of it worked on a deployment until 2026-09-05

Driven end to end on prod1 for the first time, and it took three fixes — each of which left every test green:

• IAM could not store a rule at all. It validated one by importing the gateway's own datalayer_mcp_server.alerts.Rule, and that package is not in IAM's image: POST /mcp-alert-rules/{org} answered 500 ModuleNotFoundError. The vocabulary lives in datalayer_common.mcp_alerts now — both services read the same closed sets, and IAM's suite fails if it imports any other service's package.
• POST /api/mcp/v1/alerts/test refused the caller's own organization. It used the token's organization and never the one asked about, so every personal access token — an administrator's included — was told "this token names none", while the listing route beside it accepted ?org= from the same caller.
• Every sli.latency rule read as unreadable. The reading called a window incomplete whenever the ledger answered a cursor, and Solr answers a fresh cursorMark for any page that returned rows: eighteen rows were read as "more than two thousand". The tick logged could not evaluate 1 rules once a minute and nothing fired.

With those three, the walk is: a service agent makes calls (its token carries org_uid; a personal one does not, and an alert is an organization's), alerts/test answers readable, value 848 ms, would_fire, and the tick raises the alert within 42 s.

# What this organization watches
curl -sH "Authorization: Bearer $DATALAYER_TOKEN" \
  https://iam.datalayer.run/api/iam/v1/mcp-alert-rules/$ORG_UID | jq '.rules'

# Watching for more than 20 open tasks, told at most once an hour
curl -X POST -H "Authorization: Bearer $DATALAYER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"condition":"tasks.open","operator":"gt","threshold":20,
       "severity":"warning","window_seconds":3600}' \
  https://iam.datalayer.run/api/iam/v1/mcp-alert-rules/$ORG_UID

Field	Values
condition	tasks.open, tasks.failed, audit.write_failures, spend.credits, spend.budget_fraction, sli.availability, sli.latency, sandbox.lost, dependency.down
operator	gt, gte, lt, lte, eq
threshold	a real number — nan and inf are refused, since nan > x is false for every x and such a rule never fires
severity	info, warning, critical — delivered as info, warning, error
window_seconds	whole seconds. Also the most often this rule may fire
scope_kind / scope_uid	organization, or team / user / agent with the uid of the thing. A scoped rule without a uid is refused: it would be measured over the whole organization under a name saying otherwise
enabled	a rule switched off is kept, not deleted — silencing an alert for a migration should not lose the threshold somebody tuned

IAM refuses a rule the evaluator could not evaluate — through the closed sets both services read from datalayer_common.mcp_alerts, rather than by importing the evaluator (which it cannot: that package is not in its image) or by restating its rules (which would drift). A rule that cannot be evaluated is a condition nobody is watching, and that looks exactly like a condition that never happens.

Before writing a rule, ask what it would see:

curl -X POST -H "Authorization: Bearer $DATALAYER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"org":"'$ORG_UID'","condition":"sli.latency","operator":"gt",
       "threshold":500,"window_seconds":3600}' \
  https://r1.datalayer.run/api/mcp/v1/alerts/test
# {"readable":true,"value":848.0,"would_fire":true,"detail":""}

readable is the answer worth having: a rule on something nothing can read never fires, and never firing is exactly what a correctly quiet rule looks like. The organization is the one named in the body — an owner or security auditor may name their own, a platform administrator any, and nobody else.

Every condition has a reader

There is no longer a condition that validates and then reads unreadable on every tick. The sli.* pair read the audit rather than telemetry — the audit is never sampled, which makes it the more complete source, and it is already scoped by organization and bounded by time.

A reading that could not be taken is still unreadable rather than zero: a reader answering zero would turn "nobody is watching this" into "this never happens". Read the tick's counters, not the empty alerts list — an evaluator whose unreadable is climbing looks exactly like a platform with no problems.

The two platform conditions

sandbox.lost and dependency.down watch what mcp.sandbox.lost and mcp.dependency.ready export. Those metrics go to OTEL, where a dashboard reads them; an exported counter cannot be read back inside the process that exports it, so a rule naming either used to be refused at the point of writing it — the two things an operator most wants to be paged about.

sandbox.lost is counted in the shared store beside the audit write failures, and for the same reason: a session is lost on whichever replica held it, and a per-process count would report a third of the losses three times.

Relaunched losses count

A provider losing sandboxes steadily and quietly relaunching them is invisible from the outside precisely because it recovers — which is when an alert earns its keep. If you want only the losses somebody noticed, read the audit; the rule counts both.

dependency.down is how many configured dependencies are not answering — the same set the mcp.dependency.ready gauge describes. One nobody configured is neither ready nor down, and counting it would page somebody for a deployment choice.

It is a platform reading answered per organization, deliberately: the gateway is what an organization's agents talk to, and "Contents is down" is their outage whoever caused it. One probe serves the whole tick, so two organizations cannot disagree in the same pass about whether IAM is up, and four HTTP probes are not multiplied by the number of rules watching them.

A probe that raised is cached as the failure it is

Retrying it per rule turns one outage into a burst of timeouts at the moment the platform can least afford them. And it reads unreadable, never zero — a failed probe read as "nothing is down" is an alert that goes quiet exactly when the platform is in trouble.

Use spend.budget_fraction rather than spend.credits for a "tell me at 80%" rule. Written as spend.credits > 80 beside a maxCreditsPerDay of 100, the threshold is a second copy of the budget that nothing keeps in sync — raise the budget to 500 and the rule now fires at 16%, with nothing saying so. The fraction's denominator is the budget in force, narrowed across the layers, so the rule follows the setting instead of shadowing it. An organization with no budget reads unreadable, because a fraction of no budget is not a number.

spend.credits is asked of IAM, not counted here. The credits ledger is IAM's and there is exactly one of it; a count kept beside it would be a second answer to "what did this cost" that disagrees with the invoice. The query is windowed, restricted to the agent dimension — summing the organization's whole bill would alert on somebody running a notebook by hand — and bounded.

A short spend sum is reported unreadable, never as a number

If IAM summed fewer usage records than it holds, the figure is short, and a short spend figure always reads as under budget: 90 credits of an unknown larger number is under a threshold of 100, and the alert firing nothing is the alert working exactly as designed while the budget is blown. The gateway reads complete: false as "could not tell" and the tick counts it unreadable.

audit.write_failures is the one worth understanding, because the gateway's audit is best-effort on purpose: a tool call must not fail because Solr did. What that buys is a notebook edit that works during an outage; what it costs is a ledger with a gap, and "the audit was not written" is the one failure the audit itself cannot record. A failed write bumps a counter in the shared store, per organization, which is what this condition reads. A store it cannot read is reported unreadable rather than zero — that store is the one whose outage produces the failures, so zero would say the audit is fine while it is being lost.

tasks.failed counts over the rule's own window. Two rules on the same condition with different windows are two different questions, and a count taken over all of history only grows — so an alert on it would fire once and then every window for ever.

Owners write rules on the Alerts page of the organization's MCP tab, or through IAM's API. The page's dropdowns are generated from the evaluator's own CONDITIONS, OPERATORS and SEVERITIES, so a rule written there cannot be one the evaluator refuses; a 422 from the API carries the evaluator's own words about what is wrong.

sli.latency is listed and not selectable, because nothing reads it yet. A rule on a condition with no reader is stored, evaluated every minute, counted unreadable and never fires — which looks configured, and is the silence alerting exists to break. A test in the gateway's suite holds the console's list against alerts.CONDITIONS and jobs.alert_readers() in both directions, so adding a reader without offering it, or offering one without a reader, fails CI rather than shipping quietly.

Being told

A firing is delivered as an ai-agents notification, POSTed to alert_webhook_url if one is set, emailed to alert_emails if any are, and posted to alert_slack_webhook_url if Slack is configured.

Every copy is attempted independently. An organization that asked for a webhook and an email must not lose the email because the webhook was down — they are different people finding out — and the failure names which copy was lost.

The notification goes first and separately. It is the delivery somebody will look for, and it must not be lost because a Slack workspace was down. The webhook follows; a failure is logged, counted undelivered, and does not undo the notification that already arrived.

Slack is its own destination, not the generic webhook pointed at Slack. The transport is the same; the body is not. Slack renders a generic JSON POST as one grey line of text, with the reading, the threshold and the scope run together in a sentence somebody has to parse. Given Block Kit it renders a coloured attachment — good/warning/danger by severity, so a workspace's own theme applies — with the reading, the severity and the scope as separate fields. That is the difference between glancing at a phone and opening a laptop.

Two fields rather than one, because an organization that runs a Slack channel and a PagerDuty endpoint wants each to get what it can render. IAM refuses a alert_slack_webhook_url that is not a hooks.slack.com or hooks.slack-gov.com URL: anything else sent Block Kit receives an attachments array it does not read and a text it might, so a URL pasted into the wrong field would "work", badly, and look configured.

A 200 from Slack is not always a delivery

Slack answers 200 with an error body for some failures — a webhook whose channel was archived among them. The response body is checked as well as the status, and anything but ok is counted undelivered.

A channel nobody reads answering 200 is exactly the quiet failure alerting exists to avoid: the alert would be recorded as delivered, and the first anybody knew of the gap would be the incident it was meant to warn about.

Email is sent off the event loop: SMTP is a synchronous conversation with somebody else's server, and this runs on the scheduler's tick, so a slow mail host would otherwise hold up every other organization's alert. It opts into raise_on_failure on the shared sender, which swallows by default — right for a signup confirmation, wrong for mail that is the delivery, since a send that failed and was swallowed is an alert reported as delivered and never received. That flag also fires when no SMTP is configured, which is the commonest failure and the one that looks most like success.

The webhook is not retried, unlike audit forwarding. That is a copy of a record and this is a copy of a notice — retrying a notice against somebody else's endpoint turns their outage into a loop against them, while the thing that matters is already in the app and in the alerts list.

The body leads with text, shaped as a sentence, because Slack and most chat relays render that field and ignore the rest. The structure is underneath for an endpoint that reads it. No credential is in it: this goes somewhere outside the platform, and nothing here should be usable from there.

Once per condition per window. The idempotency key is the rule, the scope and which window we are in, so a condition true for an hour is one event and one notification, not sixty — and two replicas ticking together write one event between them and send one notification for it, because delivery follows the record rather than the firing.

A delivery that fails is counted undelivered and logged; the event is in the alerts list either way. That log line is the only place the difference between an alert and a row shows up, since the list looks identical.

Audit settings

How long an organization keeps its MCP audit, and where a copy goes. Both are IAM's to write and this gateway's to act on, on the two schedules above.

# Keep a year, and send a copy to the organization's SIEM
curl -X PUT -H "Authorization: Bearer $DATALAYER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"retention_days": 365,
       "destination_kind": "https",
       "destination_url": "https://siem.example/datalayer"}' \
  https://iam.datalayer.run/api/iam/v1/mcp-audit-settings/$ORG_UID

Field	Meaning
retention_days	Between 7 and 3650. Rows past it are deleted by the hourly sweep, and rows a sweep deletes do not come back
destination_kind	https or s3
destination_url	An https:// or s3:// URL. http is refused: audit rows carry who did what for which organization, and that does not go on the wire in the clear
destination_secret_ref	Names a signing secret in IAM's store. Refused today — see below
alert_webhook_url	An https:// endpoint a fired alert is POSTed to, beside the in-app notification. http is refused: an alert names the organization and what its agents did
alert_emails	Who is emailed when a rule fires, comma-separated, at most 20. An alert list is people who want to know, not a mailing list
alert_slack_webhook_url	A Slack incoming webhook. Must be a hooks.slack.com or hooks.slack-gov.com URL — anything else belongs in alert_webhook_url, which sends a generic body. Audited as set/unset, never the URL: an incoming webhook is a credential

Owners set the three alert destinations under Where alerts go on the console's Alerts page. The form sends only the fields it owns and only those that changed: IAM merges this document, and retention and SIEM forwarding are set elsewhere by other people, so a form that posted its whole shape would clear them.

A PUT that clears forwarding used to drop the alert destinations

Fixed in be55d2b6. validate_settings answered as soon as it saw destination_kind and destination_url both empty, so alert_webhook_url, alert_slack_webhook_url and alert_emails in the same request were discarded without a word — a 200 and "Audit settings updated." over a request that was half thrown away.

It bit exactly the common case: an organization that forwards nothing to a SIEM and still wants to be told when a rule fires. If you configured alert destinations alongside a cleared forwarding destination before that commit, re-apply them — they were never stored.

An organization owner writes them. Reading needs the role that reads the audit itself — owner or organization_security_auditor — because where an organization's audit is sent is part of its security posture. The auditor may read and not write: shortening a retention deletes rows, and changing a destination sends the audit somewhere else, which are decisions about the organization rather than observations of it.

The two settings merge: they are set by different people at different times, and making somebody resend a destination in order to change a retention is how a destination gets resent wrong.

An organization with no settings is neither swept nor forwarded

This is deliberate and it is the whole reason both jobs sat unscheduled for a milestone. Retention is not derived from the plan catalogue: resolving every organization's plan means walking every organization and asking Stripe about each, and a sweep that deletes an enterprise customer's audit because a billing lookup timed out is the worst failure in this area.

So the gateway acts on organizations that decided. Absent means untouched, and an organization with settings but no retention_days is skipped too — it configured forwarding and said nothing about deleting. Keeping too much audit is recoverable; deleting it early is not.

Signed forwarding is not available yet

destination_secret_ref is refused at the write. GET /secrets/values resolves the account from the caller, so the gateway asking with its own service key gets its own secrets rather than the organization's, and there is no service-side route yet.

Stored, it would be worse than refused: the forwarder declines to deliver a batch it cannot sign, so an organization that set it would forward nothing for ever having configured forwarding. Leave it empty and forward unsigned over https, which works today.

Periodic work

Some of what the gateway owes an organization has no caller: audit past its retention has to be deleted, and alert rules have to be evaluated. Every replica runs the scheduler, and a lease in mcp-gateway decides which one actually does each job.

curl -sH "Authorization: Bearer $DATALAYER_API_KEY" \
  https://r1.datalayer.run/api/mcp/v1/operations/jobs | jq

or datalayer mcp jobs.

Read the counts per replica, never summed

The answer is from whichever replica served the request, and its skipped is the ordinary outcome: only one replica holds a job's lease at a time, so every other replica skips every tick. A high skipped on one replica is the scheduler working exactly as designed.

skipped climbing on every replica at once is the different thing worth finding — that is the lease store refusing everybody, and the jobs are not running anywhere. There is deliberately no aggregate view, because an aggregate is what would hide it.

Job	Every	What it does
audit-retention	hour	Deletes mcp-audit rows past each organization's retention, never one an open export still holds. See Audit settings for which organizations
task-reconciler	10 minutes	Makes mcp-tasks agree with the engine. durable.reconcile runs at startup too, which catches a gateway killed mid-projection; this catches a projection write that failed while the gateway stayed up, where the task says working until something looks
audit-forwarding	5 minutes	Ships each organization's audit to where it asked. The one job here that is not idempotent — two replicas forwarding is the customer's SIEM receiving every row twice — so the lease matters most for this one
alerts	minute	Reads every enabled McpAlertRule from IAM in one call, evaluates each, records a firing and delivers it. See Alert rules

Both are registered only where the records are in Solr. Standalone schedules neither and says so at startup, because a job that quietly does not run looks identical to one that runs and finds nothing.

A job that fails is counted, named and left on its schedule; the lease is released either way, so one bad tick does not cost the next hour as well. A job that wedges is stopped after its timeout rather than holding the lease until the pod is replaced.

Conformance

@modelcontextprotocol/conformance is the specification's own test suite, written by the people who define the protocol. It is the only test of this gateway that can say we misread the spec rather than that we disagree with ourselves.

npm install --no-save --prefix /tmp/conf @modelcontextprotocol/conformance
CONFORMANCE_BIN=/tmp/conf/node_modules/.bin/conformance \
  pytest tests/test_conformance.py -v

The fixture starts a standalone gateway on loopback, runs the suite against it and fails on any scenario that is not in tests/conformance-baseline.yaml. Without the npm package the tests skip, saying so.

How the credential gets in, and what was refused

The suite takes --url, --scenario, --suite, --expected-failures, --output-dir, --spec-version and --verbose — and nothing that sends a header. Every /mcp route needs a bearer token, so pointed straight at this gateway the suite fails every scenario including ping, which measures that authentication works. True, and not what conformance means.

The obvious fix is a switch that turns authentication off for the test run. It was refused: that means shipping a mode in which this gateway serves unauthenticated, and that is the kind of switch which escapes into a deployment.

Instead the fixture starts the gateway, so it holds DATALAYER_JWT_SECRET and mints a token — signature, issuer and expiry all verified by the ordinary code path, no bypass — and a small proxy in tests/ attaches it. The server under test is in exactly the configuration it is deployed in.

What it finds

At 2025-11-25, nine of thirty scenarios pass and twenty-one are baselined, in three kinds:

Kind	Why
Fixtures (12)	The scenario asks for test_simple_prompt or test://static-text by name — things only the suite's reference server has. Adding them would put a test fixture in a product
Unimplemented (3)	completion/complete and the log notifications that follow logging/setLevel
Client capabilities (4)	Elicitation and sampling are the server asking the client for something. A tool that stops to ask a question is one an agent cannot run unattended

None of the twenty-one is this gateway claiming something it cannot do

initialize advertises no completions capability. The suite runs those scenarios anyway, without gating on what the server said it supports — so a -32601 Method not found there is the honest answer to a request the client was told not to make.

resources/subscribe is no longer in that list. Read against prod1 on 2026-09-05, initialize answers resources: {subscribe: true, listChanged: false} and logging: {} — the baseline said the opposite of both, and a capability advertised as true is a promise the -32601 defence does not cover. Subscribe and unsubscribe are implemented and, since the same day, actually deliver: see A subscriber is told, or nobody is.

Read the baseline before reading twenty-one failures as twenty-one bugs. Each entry names what it is waiting for; an entry is a debt with a name, not a silence, and a new failure outside the baseline blocks the release.

The suite carries scenarios for 2025-03-26, 2025-06-18 and 2025-11-25 and none for 2026-07-28, which this gateway also serves. The harness asks for that version, gets no scenarios, and skips saying so — a run that executes nothing must not be read as a pass. When the suite ships them the skip becomes a run with no change here.

What is not here yet

Named plainly, because an operator reading a route should know what stands behind it.

Most of this list is off in standalone anyway

The deployment ships with DATALAYER_MCP_STANDALONE=true, which turns off the durable engine, the scheduled jobs and the policy fetch by design rather than by omission. The entries below describe the platform mode; check readyz's mode before chasing any of them — and its records before believing the mode, since turning the flag off does not by itself move anything out of the process.

• Durable runs. mcp-tasks is read and /api/mcp/v1/tasks/* is served, but the engine that creates tasks — Durable — is deployed separately and is not required. Without it the gateway falls back to an in-process workflow backend that reports durable: false, and a run does not survive the pod. GET /api/mcp/v1/operations/workflows says which you have; do not read an empty task list as "nothing is running" until it says durable: true.
• The organization and personal policy layers. /api/mcp/v1/policy answers, and the platform layer in it is real — the tool table narrowed by the token's scopes, which is what decides every refusal the gateway makes today. The layers are enforced: a tool denylist or allowlist, a client allowlist and a per-minute cap refuse the call and name the layer that decided — and IAM serves them now, at /api/iam/v1/mcp-policies/{scope}/{uid}. See Policy layers for what may be written and by whom. A layer only ever narrows. A team cannot re-admit what its organization denied, and a cap is taken as the smallest of the platform's, the organization's and the team's — a layer that could raise a limit would be a way to buy more of the platform by writing a setting. Two empties are not restrictions: an empty tool allowlist and an empty client allowlist admit everything, because a setting somebody created and has not filled in would otherwise refuse every agent in the organization, and the first conclusion anyone draws from that is that the gateway is down.
• DPoP. Not enforced, and deliberately not built ahead of the Agent Identity WG's finalized profile.
• Signed audit forwarding. Unsigned over https and to an S3-compatible store both work and are scheduled. A signing secret cannot be resolved by this gateway yet, so destination_secret_ref is refused rather than stored — see Audit settings.
• Alert delivery beyond the app. The rules exist, the evaluator runs on its minute tick and a firing reaches ai-agents as a notification — see Alert rules. Email, the organization's webhook and Slack through it are not built. And four of the six conditions have no reader: audit.write_failures, spend.credits and the two sli.* need IAM's ledger and the OTEL service, so a rule naming one of those is counted unreadable every tick and never fires. That is the honest outcome — a reader answering zero would turn "nobody is watching this" into "this never happens" — but it is not the same as being watched.
• The load test that would state a per-replica capacity is not in the repository, so the numbers behind the HPA targets are unmeasured.

See the CLI for what datalayer mcp can answer today.

/api/mcp/v1/executions/* is gone

It was an alias for the task routes while those were being built, backed by a per-process registry that lost its contents when a pod was replaced. Use /api/mcp/v1/tasks/*, which reads mcp-tasks and answers the same on every replica. Nothing hand-written called the alias.

The CLI

datalayer mcp reads this gateway with the caller's own token, so an administrator sees what an administrator may and a user sees their own. It is the fastest way to answer most of the questions on this page without curl.

Command	Answers
datalayer mcp setup <client>	Writes an MCP client's configuration for this endpoint — claude-code, and the other registered clients
datalayer mcp agents	The agents connected to your account
datalayer mcp bindings list	The handles you hold: notebooks, toolsets, sandboxes — lost ones included, with why
datalayer mcp bindings terminate <uid>	Ends one, terminating the sandbox through Runtimes
datalayer mcp tasks list / describe / cancel / input	The runs the agents started, and answering one that is waiting on a person
datalayer mcp audit	The rows: your own, or the organization's with the role for it
datalayer mcp policy	The effective policy for your token, rule by rule, with the layer that decided each
datalayer mcp toolsets list	The sources that lend tools, and which have a session open
datalayer mcp toolsets enable <source> / disable <session>	Open or revoke a Contents MCP session, so an agent can call a source's tools
datalayer mcp sandboxes sharing <name>	Who a sandbox of yours is shared with
datalayer mcp sandboxes share <name> --user/--team/--organization/--agent [--level]	Share it; --replace makes the lists given the whole grant at that level
datalayer mcp sandboxes unshare <name> [--level]	Take a share back; naming nobody revokes everybody at the level
datalayer mcp sandboxes permissions <name>	What you may do with a sandbox, yours or somebody else's
datalayer mcp alerts list / ack	The rules that fired. ack names who acknowledged rather than showing a tick — "who saw this" is the question
datalayer mcp activity	What each connected client last did
datalayer mcp forwarding	Whether the audit is reaching the organization's own system of record
datalayer mcp jobs	The periodic work, per replica — read Periodic work before the numbers
datalayer mcp trace / metrics / logs	What the OTEL service holds, which is everything the gateway exports

Test after a deploy

# The gateway, its dependencies and its store
curl -s https://r1.datalayer.run/api/mcp/version
# platform + solr, or standalone + memory; platform + memory is the silent one
curl -s https://r1.datalayer.run/api/mcp/readyz | jq '{mode, records}'
curl -s https://r1.datalayer.run/api/mcp/readyz | jq '.dependencies[] | {name, required, ready}'
curl -s https://mcp.datalayer.run/.well-known/oauth-protected-resource/mcp | jq
curl -i -s https://mcp.datalayer.run/mcp | grep -i www-authenticate
# Scaling: two replicas, a budget, a spread
kubectl get deployment,hpa,pdb -n datalayer-api -l app=mcp-server
kubectl get pods -n datalayer-api -l app=mcp-server -o wide   # on different nodes
# As a user
datalayer mcp setup claude-code                # writes the client configuration
datalayer mcp bindings list                    # the handles you hold, sandboxes included
# As a platform administrator
curl -H "Authorization: Bearer ${DATALAYER_TOKEN}" \
  https://r1.datalayer.run/api/mcp/v1/operations/workers | jq '.items[] | {replica, user_uid}'

Then connect a real client and run one tool: claude mcp add datalayer --transport http https://mcp.datalayer.run/mcp, authenticate through the browser flow, and ask it to list your notebooks. A 401 that carries WWW-Authenticate, a consent screen, and a tool result is the whole authentication path exercised end to end.

Tear down

plane down datalayer-mcp-server

Removing the release removes the Deployment, Service, Ingress, HPA, PDB and the r1.datalayer.run Certificate. It does not delete the mcp-gateway, mcp-tasks or mcp-audit collections, their Solr backups, the OAuth grants IAM holds for the clients that connected, or any sandbox a session left running.

Before tearing down a cluster that agents are connected to:

• terminate the live sessions, or let their reservations expire — a sandbox outlives the gateway, and the binding that named it goes with the release;
• keep the mcp backup unit if the audit trail matters, and follow Continuity rather than deleting collections;
• revoke the OAuth grants in IAM if the endpoint is not coming back, so a client is refused rather than left retrying against a host that no longer resolves.

OpenAPI Specification

The OpenAPI (Swagger) specification is available online.