Bluesky and social publishing

Datalayer has three deliberately different social boundaries:

1. anonymous share links open LinkedIn, X or Bluesky's own composer;
2. LinkedIn native publishing uses a user's connected token held by IAM;
3. Bluesky native publishing uses AT Protocol OAuth and records owned by the author in their own repository.

Library owns immutable cards, publication state, feed reads and refresh. IAM owns provider sessions and every provider API call. Provider credentials must never appear in Solr, a browser response, a URL, a cookie, or logs.

The user-facing half is documented at Integrations › Bluesky.

Bringing it up, in order

Nothing below is optional, and the order matters: the feed cannot be pinned before its record exists, and the record names a service that must already answer.

1. Secrets, through Helm only. The Library→IAM key and the refresh token are Helm-owned Secrets (socialRefresh.secret, social.libraryServiceSecret). Never create either by hand; rotate by setting a new non-secret rotationNonce and redeploying. Configure a dedicated random DATALAYER_SOCIAL_REFRESH_TOKEN in the plane rc — never a user token or a provider secret — then DATALAYER_SOCIAL_REFRESH_ENABLED=true.
2. Deploy IAM, including the /bluesky/oauth routes.
3. Deploy the Library, including /s/, /.well-known/did.json and /xrpc/app.bsky.feed.*, and the two-minute refresh CronJob.
4. Deploy the OTEL service, which evaluates the refresh alerts (below).
5. Deploy the landing UI.
6. Route the four paths at the CDN (below).
7. Publish the DNS TXT record so the lexicon resolves from its NSID: _lexicon.library.datalayer.ai → did=did:plc:5wcl7w4vyvswopgbn7qgku6c.
8. Publish the two AT Protocol records (below).
9. Connect the @datalayer.ai account in Settings → AT Protocol, which is what lets an administrator seed publications.
10. Validate (below).

DATALAYER_SOCIAL_BLUESKY_ENABLED=false removes every entry from Datalayer's Bluesky surfaces without deleting author-owned records.

Rebuilding the services

The Library and the OTEL service are built from a clean export and deployed the usual way:

source ~/.datalayer/datalayerrc-prod1
cd $PLANE_HOME/etc/dockerfiles/datalayer-library && make build-dev push
cd $PLANE_HOME/etc/dockerfiles/datalayer-otel    && make build-dev push

Both charts pull Always, so when the tag has not changed a kubectl rollout restart is enough and leaves the release alone. A chart change — new values, new environment — needs p reup <service>. Confirm what is actually running before believing a deploy:

kubectl get pods -n datalayer-api  -o jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[0].imageID}{"\n"}{end}' | grep library
kubectl get pods -n datalayer-otel -o jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[0].imageID}{"\n"}{end}'

The CDN routes

Four path patterns on the datalayer.ai distribution go to the plane; the SPA stays the default behavior:

Path	Served by
/s/*	Library — the immutable share cards
/bluesky/oauth/*	IAM — client metadata, JWKS, the one-use start link and the callback
/.well-known/did.json	Library — did:web:datalayer.ai
/xrpc/app.bsky.feed.*	Library — the feed generator

None of them may fall back to the SPA: a crawler that receives index.html for /s/… draws the site's generic card instead of the item's.

The SPA's status is content-negotiated

On SPA routes such as /bluesky, the edge answers 200 to a request that asks for HTML and 404 to one sending Accept: */* — which is curl's default. A page that renders in a browser can therefore look missing from a script. The routed paths above are unaffected: they are served by the plane and answer 200 to any client.

The settings page is the AT Protocol page

In the application the connection lives at Settings → AT Protocol (/settings/atprotocol; the earlier /settings/bluesky redirects). Bluesky is one network on the protocol, and the page works with any server that speaks it, so the page, its icon and IAM's return address say AT Protocol; the IAM provider id stays bluesky, and the feed, the share entry and the user documentation say Bluesky where they mean the network. DATALAYER_BLUESKY_SETTINGS_URL defaults to https://datalayer.ai/settings/atprotocol since IAM 1.1.21.

Publishing the AT Protocol records

Two records live in the @datalayer.ai repository. Without them the feed endpoints answer and nobody can reach them: the feed is invisible in the Bluesky app, and a reader has no schema to validate a publication against.

Record	Collection / rkey	What it does
Lexicon schema	com.atproto.lexicon.schema/ai.datalayer.library.publication	Lets any app validate Datalayer's publication record
Feed generator	app.bsky.feed.generator/datalayer-ai	Makes Datalayer AI findable and pinnable

These are not written by Datalayer. The platform's OAuth client is scoped to app.bsky.feed.post and ai.datalayer.library.publication only, so that connecting an account never gives Datalayer the run of a repository. Neither record is in those collections, so they are published once, by the account holder, with an app password:

# bsky.app → Settings → App Passwords, while signed in as @datalayer.ai
cd $DATALAYER_HOME/src/landings/datalayer/ui
BLUESKY_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx \
  npm run publish:social-atproto -- --avatar-url https://assets.datalayer.tech/datalayer-square.png
npm run check:social-atproto

The feed's picture is optional: --avatar <file> takes a local PNG or JPEG, --avatar-url <url> fetches one, and leaving both out keeps whatever the feed has. Whichever is used, it is read before anything is written — an earlier run wrote the lexicon and then died on a path that was not there, leaving the feed record unwritten.

npm run check:social-atproto reads the records from the repository's own server, resolved from its DID, rather than from the public AppView: the AppView indexes app.bsky.* and answers RecordNotFound for a lexicon schema that is published perfectly well.

The publisher reads the schema straight from the Library's own lexicons/ai.datalayer.library.publication.json, so the published record cannot drift from the service that writes publications, and it uses putRecord: running it again replaces rather than duplicates, which is also how the feed's description or avatar is changed later. Revoke the app password when the run is done. --dry-run prints both records and writes nothing.

The feed's identity, and changing it

What	Value	Set in
Name shown in Bluesky	Datalayer AI	FEED_DISPLAY_NAME in landings/datalayer/ui/scripts/publish-social-atproto.mjs
Description	Results published from Datalayer notebooks…	FEED_DESCRIPTION, same script
Record key	datalayer-ai	FEED_RKEY in the publisher and in check-social-atproto.mjs
Feed URI the service answers for	at://did:plc:5wcl7w4vyvswopgbn7qgku6c/app.bsky.feed.generator/datalayer-ai	FEED_URI in library/datalayer_library/api/social.py
Feed service	did:web:datalayer.ai	the record's did; served by /.well-known/did.json
Public address	https://bsky.app/profile/did:plc:5wcl7w4vyvswopgbn7qgku6c/feed/datalayer-ai	the landing's Follow the feed link (views/social/BlueskyFeed.tsx)

To rename the feed or change its description or picture, edit the script's constant and run the publisher again — putRecord replaces the record in place, and the AppView picks the change up within seconds:

cd $DATALAYER_HOME/src/landings/datalayer/ui
BLUESKY_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx npm run publish:social-atproto
curl -s "https://public.api.bsky.app/xrpc/app.bsky.feed.getFeedGenerator?feed=at://did:plc:5wcl7w4vyvswopgbn7qgku6c/app.bsky.feed.generator/datalayer-ai" \
  | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d["view"]["displayName"], d["isOnline"], d["isValid"])'
# → Datalayer AI True True

Changing the record key is a migration, not an edit: the publisher, the checker, the Library's FEED_URI (a Library release) and the landing's link all name it, and readers who pinned the old key keep the old one. The key was moved from datalayer-notebooks to datalayer-ai once, on 23 September 2026, before anyone could pin it.

How a result reaches the feed

A native publish is a saga the Library runs, through IAM, into the author's own repository. Each attempt is a share with its own idempotency key and a status: pending → published, or failed; later deleting → deleted.

1. Capture — the landing renders the chosen output to a PNG in the browser (html-to-image, on the output's own node), at most 1 MB.
2. Blob — the Library uploads it: POST /api/iam/v1/social/bluesky/blobs → com.atproto.repo.uploadBlob on the author's PDS.
3. Post — app.bsky.feed.post under a TID record key, the text being the finding followed by the artifact's tags as #tag hashtags, each with its app.bsky.richtext.facet#tag facet. Tags that would pass 300 graphemes are left out. The PDS validates this record (validate: true).
4. Publication record — ai.datalayer.library.publication under its own TID, pointing at the post (strongRef). The PDS does not validate it: validate: true requires a lexicon the PDS knows and it does not know ours; the Library validates it against LibraryPublication first.
5. Index — the Library marks the share published and indexes the publication; the refresh job updates its counts every two minutes.
6. Feed — Bluesky asks https://datalayer.ai/xrpc/app.bsky.feed.getFeedSkeleton for …/datalayer-ai, and the Library answers with the published posts.

If step 4 fails after step 3, the Library deletes the post again and marks the share failed with remote_cleanup_status confirmed; if that deletion fails too, failed, which the public page offers to clean up. Unpublishing deletes the publication record, then the post.

Every IAM call is authorised twice: the Library's service key (X-API-Key) and a single-use delegation it signs for one user and one operation. A record write or delete names its exact target in the delegation — <share>/<collection>/<rkey> — and IAM refuses any other record.

When a publish fails

The dialog says Bluesky could not publish this result; the cause is in the logs (p logs datalayer-library, p logs datalayer-iam). Read which call failed, then:

Library log	Cause	Fix
POST …/blobs 401, nothing in IAM's log	IAM refused the Library's service key: the two pods hold different DATALAYER_LIBRARY_IAM_API_KEY	Compare them by hash (below). The IAM chart generates the secret datalayer-library-iam-social and marks it helm.sh/resource-policy: keep; before that, every plane reup datalayer-iam re-minted it. kubectl -n datalayer-api rollout restart deploy/datalayer-library picks up the current one
POST …/blobs 401, IAM logs the user lookup	The author's session could not be refreshed	IAM logs Bluesky token refresh at … refused with …. The refresh sends the authorization server's own DPoP nonce (as_dpop_nonce) and answers one use_dpop_nonce; anything else means reconnecting in Settings → AT Protocol
PUT …/records 403	The delegation's target does not match the record	Library and IAM disagree on the target form; deploy them together, IAM first
PUT …/records 400/502	The PDS refused the record	IAM logs Bluesky com.atproto.repo.putRecord refused with 400: <error> — <message>. Invalid TID string is a record key of the wrong shape; Unknown lexicon type is validate: true on our own collection

Comparing the service key without printing it:

for p in $(kubectl -n datalayer-api get pods -o name | grep -E 'datalayer-(iam|library)-'); do
  printf '%s ' "$p"
  kubectl -n datalayer-api exec "${p#pod/}" -- sh -c 'printf %s "$DATALAYER_LIBRARY_IAM_API_KEY"' | sha256sum | cut -c1-12
done

Validating a release

The checks live beside the plan in landings/datalayer/ui, and their ledgers in landings/datalayer/ui/socials/:

Command	What it proves
npm run check:social-release	The DID document, OAuth metadata, JWKS, feed generator endpoint and _lexicon TXT record all answer
npm run check:social-atproto	Both repository records exist and are well formed
npm run check:social-public-pages	The public pages render on desktop and phone
npm run check:social-rollback	The routed paths are intact and a rollback payload exists
npm run check:social-owners / -scope	The release has named owners and a frozen measurement scope
npm run check:social-seeds / -evidence / -campaign	The seed, evidence and campaign ledgers are complete rather than merely valid — add -- --require-complete

When a connection is refused

POST /api/iam/v1/social/bluesky/connect answers 400 with The Bluesky authorization request was refused: <error> — <description>, and IAM logs the same line as Bluesky PAR at <issuer> refused with <status>: …. The error and description are the authorization server's own words; the settings page shows them too.

Two of them have a known meaning:

• invalid_client_metadata — Unable to obtain client metadata for …, with no reason after it. The authorization server could not fetch https://datalayer.ai/bluesky/oauth/client-metadata.json within its ten seconds. It fetches that document from IAM before it can answer the pushed request, so IAM has to serve it while its own request is in flight. It once could not: require_auth wrapped every handler in a coroutine and the synchronous connect handler blocked the whole single-worker server on its httpx.post. That is fixed in datalayer_common.auth.require_auth (a synchronous handler now runs in the threadpool), and guarded by common/tests/test_require_auth_keeps_sync_handlers_off_the_loop.py. To check for a recurrence, time the document while a connect is running:

  curl -s -o /dev/null -w '%{time_total}s\n' \
    https://prod1.datalayer.run/bluesky/oauth/client-metadata.json

  Ten seconds is the deadlock; under a second is healthy. A bare POST https://bsky.social/oauth/par with our client_id and no client_assertion is the other half of the picture: it should answer client authentication method "private_key_jwt" required a "client_assertion", which means the server fetched and validated the document and got as far as asking for our key.

• … Validation of "<field>" failed … The document was fetched but rejected. Compare it with the shape in Bringing it up, in order.

{"detail":"The Bluesky transaction cookie does not match."} at the callback, after a sign-in that worked, means the browser never kept the transaction cookie. It is a __Host- cookie, and a browser stores one of those only when it is Secure, carries no Domain and has Path=/; it was once set with Path=/bluesky/oauth/ and silently dropped. set_transaction_cookie in datalayer_iam/api/bluesky_oauth.py owns the attributes, and iam/tests/test_bluesky_transaction_cookie.py holds the rule. CloudFront's DatalayerSocialOAuthOrigin origin-request policy forwards that one cookie on /bluesky/oauth/*, and nothing else — check it with aws cloudfront get-origin-request-policy if the callback still cannot see it.

"The Bluesky connection could not be loaded" on the settings page right after a successful callback was GET /api/iam/v1/social/bluesky/profile answering 502. The session had been written to Vault under a hash of the DID and read back under a hash of what the account record held — and Solr escapes what it stores, so the record holds did\:plc\:… and the two hashes never met; hvac then raised on the missing path instead of reading it as "no session". Since IAM 1.1.20 the session is keyed by the user (datalayer/iam/social/bluesky/<uid>/session, the DID inside it), a session found under the old key is adopted on first read, a missing Vault path reads as None (datalayer_vault.read_vault_path), and an unexpected failure behind a 502 is logged with its traceback (_error in datalayer_iam/api/bluesky_oauth.py). Tests: iam/tests/test_bluesky_session_path.py.

The sign-in page that follows a successful pushed request is Bluesky's, and it takes the account's main password: an app password is refused there with Wrong identifier or password. App passwords are only for the com.atproto.server.createSession publisher above.

Then read the surfaces directly:

curl -fsS https://datalayer.ai/.well-known/did.json
curl -fsS https://datalayer.ai/xrpc/app.bsky.feed.describeFeedGenerator
curl -fsS --get \
  --data-urlencode 'feed=at://did:plc:5wcl7w4vyvswopgbn7qgku6c/app.bsky.feed.generator/datalayer-ai' \
  https://datalayer.ai/xrpc/app.bsky.feed.getFeedSkeleton
curl -fsS https://prod1.datalayer.run/api/library/v1/social/publications

Check that private items, pending/failed/deleted shares, and moderation-hidden shares are absent. The feed accepts anonymous reads and optional AT Protocol service authentication. A presented JWT is verified as ES256 or ES256K against the issuer DID's #atproto key, with the exact feed-generator audience and lxm, a bounded lifetime, and a single-use jti. DID resolution is bounded, cached, non-redirecting and restricted to public hosts. Invalid present authentication is refused rather than silently downgraded to an anonymous read.

Ambiguous LinkedIn create

LinkedIn post creation has no provider idempotency key. If Library records provider_timeout_ambiguous, do not retry the create request. Compare the connected member's activity with the attempted text, image and time window, then have a platform administrator record exactly one result:

curl -fsS -X POST \
  -H "Authorization: Bearer ${DATALAYER_ADMIN_TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
    "userUid": "<user uid>",
    "idempotencyKey": "<original idempotency key>",
    "outcome": "published",
    "postUrn": "urn:li:share:<provider id>",
    "evidence": "LinkedIn activity lookup <UTC timestamp>"
  }' \
  https://prod1.datalayer.run/api/library/v1/admin/shares/linkedin/reconcile

Use "outcome": "absent" and omit postUrn only after confirming the post was not created. The endpoint accepts only the ambiguous pending state, validates the URN, stores the administrator UID, evidence and timestamp, refuses conflicting terminal evidence, and never submits a post to LinkedIn.

Refresh and deletion drill

The refresh is bounded to 100 records per run and batches 25 URIs per AppView request. The single Library replica runs it every two minutes; an external CronJob must use concurrencyPolicy: Forbid to prevent overlapping jobs. A provider failure increments the failure count and backs off; it does not hide the post. One confirmed missing post also remains visible. Two consecutive confirmed misses hide it with reason provider_post_missing.

For a pre-release drill:

1. publish one non-production test record;
2. observe it in the feed, /bluesky, and its artifact backlink;
3. delete it from the Bluesky app;
4. verify the first refresh keeps it and the second hides it;
5. make another artifact private and verify all dynamic Datalayer surfaces omit it immediately;
6. exercise the kill switch and restore it;
7. inspect pending, failed, deleting and hidden records before ending the drill.

Never claim that hiding or deletion retracts copies cached or reposted by a network.

Observability and alerts

Build the dashboard from versioned OTEL events, not HTTP crawler counts alone:

• share.attempted → public_page.viewed → clone.clicked → signup.completed → clone.completed, split by validated network;
• linkedin.connect.*, linkedin.published, linkedin.publish_failed and linkedin.unpublished;
• bluesky.connect.*, bluesky.published, bluesky.publish_failed and bluesky.unpublished;
• feed.requested, refresh failures, records hidden after two misses, and pending remote cleanup.

The refresh alerts

Two things can go wrong with the refresh, and both are silent: a run that fails, and a refresh that stops happening. Both are alerts of the Datalayer OTEL service — there is no PrometheusRule and no rule evaluator to install, because the signal is already here. The Library emits one social.refresh.run event per run, carrying its outcome and counts, and the OTEL service evaluates the two rules over those events on its own sixty-second loop.

Alert	Fires when	Severity
social_refresh_failed	A run reported failures, or raised, inside the window (default 10 minutes)	warning
social_refresh_stale	No successful run inside the freshness window (default 10 minutes, five missed runs) — including a refresh that never started, or one whose every run skipped its slot	critical

A run that skipped its two-minute slot refreshed nothing: another replica held the lease. It is neither a success nor a failure, so a deployment where every run skips reads as stale rather than healthy. The replica that did the work reports its own success to the same service.

Read the current state as a platform admin:

curl -fsS -H "Authorization: Bearer ${DATALAYER_API_KEY}" \
  https://prod1.datalayer.run/api/otel/v1/social/alerts

It answers runs_seen, seconds_since_success and both alerts with their windows. The service log carries one line when an alert starts firing and one when it recovers, and nothing in between. The windows are fastapi.socialAlerts.staleAfterSeconds and failedWindowSeconds in the datalayer-otel chart; fastapi.socialAlerts.enabled turns the evaluation off. To rehearse them, stop the CronJob past the freshness window for the stale alert, and make one run fail for the other.

Alert on the rest through the same pipeline: a growing pending/deleting backlog, OAuth callback failures, Vault read failures, and feed 5xx responses. Raw event-level attribution expires after 30 days; aggregate-only metrics may be kept for 13 months once the privacy owner approves that retention policy.

Rollback

Disable native Bluesky discovery first with the kill switch, leaving records and sessions intact for diagnosis. Disable the refresh CronJob next. Rolling back the UI never requires deleting IAM sessions or Solr publication state. Provider-side deletion is a separate, explicit author action and must not be used as a deployment rollback mechanism.

Publishing the two repository records is not part of a rollback: they are the author's, they describe a schema and a service rather than any content, and removing them only makes the feed unfindable.

Glossary

• AT URI — the stable at:// address of a repository record.
• DID — the decentralized identifier that owns an AT Protocol repository; a handle is display metadata and may change.
• DPoP — proof-of-possession attached to OAuth token and repository calls; provider nonces are persisted and retried with a fresh proof.
• Feed generator — the service returning ordered post AT URIs from getFeedSkeleton; the AppView hydrates those URIs for the client.
• Lexicon — the AT Protocol schema for a record or XRPC method. Datalayer's publication record is ai.datalayer.library.publication.
• PDS — the user's Personal Data Server, discovered from their DID and used for blob and repository operations.
• PAR — Pushed Authorization Request, used before redirecting the browser to provider consent.
• Strong reference — the post's AT URI and CID captured together so the publication record names one immutable post revision.