Skip to main content

☰ 📚 Datalayer Library

KubernetesREST API

Datalayer Library provides a searchable index of artifacts useful for data analysis.

- Cell.
- Notebook.
- Environment.
- Runtime.

Dataset publications are owned and indexed by the Contents service; the Library surfaces them alongside the items above.

Deploy Datalayer Library​

plane up datalayer-library

Check the availability of the Datalayer Library Pods.

kubectl get pods -n datalayer-api -l app=library

Check the logs of the Datalayer Library Pods.

kubectl logs -n datalayer-api -l app=library

Check the availability of the Datalayer Library Certificate.

kubectl describe certificate ${DATALAYER_RUN_HOST}-datalayer-api-cert-secret -n datalayer-api

Check the availability of the Datalayer Library Endpoints.

open https://${DATALAYER_RUN_HOST}/api/library/version
open https://${DATALAYER_RUN_HOST}/api/library/v1/ping

Environments published on a runtimes plane​

Environment publications live with Runtimes on the runtimes plane (r1); the Library and its Solr on prod1. Runtimes projects each card through POST /api/library/v1/internal/environments/sync, authenticated with its own key, and the Library reads the publications back from Runtimes' open GET /api/runtimes/v1/environment-publications when it reindexes. Two settings, from the prod1 rc through up.sh:

  • DATALAYER_RUNTIMES_LIBRARY_API_KEY: the key Runtimes presents. The same value is in the r1 rc, for Runtimes. Unset, the route refuses everybody with 401.
  • DATALAYER_LIBRARY_ENVIRONMENT_SOURCES: the Runtimes to read publications from, comma-separated. up.sh defaults it to DATALAYER_RUNTIMES_URL. Unset, the Library reads its own Solr, which on prod1 has no environment collection: a reindex then stops on the sweep.

POST /api/library/v1/admin/reindex?kind=environment rebuilds the environment cards alone from those Runtimes; kind without uid used to rebuild every kind. The whole setup, its deploy order and its checks are on the Environments page.

Tear Down Datalayer Library​

If needed, tear down.

plane down datalayer-library

What was made of an artifact​

Beside each artifact's orbits, the Library keeps the derivations that came of it: type_s:derivation documents in the library collection. Each one has a deterministic id (derivation-<kind>-<from>-<to>), so recording the same derivation twice writes one edge. Each edge names:

  • the source (from_uid_s) and the version taken, when there is one (from_version_i);
  • the result (to_uid_s);
  • the kind (kind_s): fork, reproduce, compare, cite or revise;
  • the artifact type, and who made it (actor_uid_s).

The service that makes the result writes the edge through datalayer_solr.library.add_derivation. Today a benchmark clone in ai-agents writes a fork.

GET /api/library/v1/items/{id}/derivations?offset=&max= answers anybody, signed in or not, for a public artifact, and 404 for any other. It returns:

  • counts: every derivation, by kind;
  • derivations: newest first, naming only the results that are public themselves, each with kind, fromVersion, derivedAt, and the item as a search result;
  • total: how many edges there are.

A clone somebody keeps private adds one to the count and is never named. The public pages draw these as Reused by.

OpenAPI Specification​

The OpenAPI (Swagger) specification is available online.

Social publication index and Bluesky feed​

Native social publication state is stored as bounded social_share documents in the Library Solr collection. The documents contain provider identifiers, status and public counters, never OAuth credentials. Public reads expose only published Bluesky records whose Library item is still public:

  • GET /api/library/v1/social/publications for the /bluesky page;
  • GET /api/library/v1/public/items/{uid}/publications for artifact backlinks;
  • GET /.well-known/did.json, describeFeedGenerator, and getFeedSkeleton for the did:web:datalayer.ai feed generator.

The chart routes /.well-known/did.json and /xrpc/app.bsky.feed.* to the Library. DATALAYER_SOCIAL_BLUESKY_ENABLED=false is the immediate feed kill switch.

When DATALAYER_SOCIAL_REFRESH_ENABLED=true, the chart's single Library replica runs the bounded refresh every two minutes. A deployment can instead call the internal endpoint from a CronJob by configuring a dedicated DATALAYER_SOCIAL_REFRESH_TOKEN; the generated chart uses concurrencyPolicy: Forbid. Both paths read batches of at most 25 posts from the public AppView. One missing response is tolerated; two confirmed misses hide the publication. Only an automated provider_post_missing hide may be automatically restored. Privacy and moderation hides remain hidden until their owner clears them.

Native publication uses deterministic post and publication record keys. The Library persists pending before calling IAM, uploads the scrubbed result, writes the post, then writes ai.datalayer.library.publication with a strong reference to that post. If the second write fails, it compensates the post and records whether cleanup is confirmed or still failed. Unpublish hides locally before deleting the publication record and post.

Immutable share-card retention is disabled until operations configures the approved positive DATALAYER_SHARE_CARD_RETENTION_DAYS. Once enabled, a daily bounded job removes expired Solr snapshots first, then garbage-collects only image objects referenced by neither another card nor a current Library item.

Native-share cleanup evidence has an independent period, DATALAYER_SOCIAL_TOMBSTONE_RETENTION_DAYS, also disabled at 0 until an approved audit period is deployed. Its bounded daily job removes only deleted records whose remote cleanup is confirmed; pending and failed cleanup evidence is retained regardless of age. Every candidate's family and cleanup state are re-read immediately before deletion, preventing a stale retention listing from erasing evidence changed by reconciliation.