System Design

Overview

Wanderland implements an executable knowledge graph where markdown documents serve as both the storage format and the programming language. The architecture comprises four core subsystems: the Token Stream (document representation), the Compilation Pipeline (cache levels), the Graph Engine (navigation and linking), and the Provenance System (verification tracking).

The Token Stream

A Wanderland document is a stream of tokens with three categories:

• Prose Tokens: Markdown text rendered for human consumption
• Fence Tokens: Executable regions delimited by triple backticks
• Slot Tokens: Graph edges encoded in YAML blocks within cardinal sections (North/South/East/West)

Fences are the executable boundary. A fence declares:

• Language: The execution runtime (python, yaml, bash, sql, etc.)
• Attributes: Key-value metadata including execution flags, service bindings, labels
• Content: The code or data to process

This mirrors the structure of a binary: instructions (prose), relocatable symbols (fences), and linkage information (slots).

The Compilation Pipeline

Documents progress through five cache levels, analogous to compilation stages:

Level	Name	Transformation
L0	seed	Raw tokens from file (source)
L1	sprout	Variables substituted (preprocessing)
L2	stalk	Components expanded, includes resolved (compilation)
L3	data	Fences executed, external data fetched (linking)
L4	clouds	Middleware transforms applied (optimization)

The critical insight is L3 as linking: fence execution fetches data from external systems (AWS APIs, JIRA, databases) and splices results into the stream. The fence is a hole. The linker fills it.

Each level caches independently. Cache invalidation is trivial: delete any level and it regenerates from below. The system is self-healing—poke a hole anywhere and it fills from the source of truth.

The Graph Engine

Documents are nodes in a directed graph. Edges are encoded in cardinal sections using a spatial metaphor:

Direction	Semantic	Traversal Meaning
North	Parent/overview	Move up to broader context
South	Child/detail	Move down to finer granularity
East	Next/forward	Move forward in sequence
West	Previous/backward	Move backward in sequence

This convention enables predictable navigation: "go north" always means "zoom out," while "go south" means "drill down." East/West provide linear traversal through sequences like paper sections or workflow steps.

Navigation is query execution. Moving from node A to node B:

• Resolves the target node
• Executes edge middleware (guards, transforms)
• Renders the target at the requested cache level
• Accumulates context for downstream processing

The graph supports multiple addressing modes:

• Slug: Direct node identifier (my-node)
• Path: Dot-notation into node structure (my-node.section.fence.key)
• Fence ID: Direct fence addressing (my-node:my-label)

The Middleware Layer

Middleware operates at L3.5—after fence execution but before caching. This is Kafka consumer semantics applied to documents:

Fence Output → [Normalizer] → [Cleaner] → [Extractor] → Cached Result

Middleware is:

• Composable: Chain transformations in sequence
• Conditional: Pattern matching determines execution
• Stateless: Each transformer is pure
• Extensible: Add processors without modifying core

Edge middleware extends this to navigation: transitions between nodes trigger middleware that can guard, transform, or accumulate context.

The Provenance System

Every document and fence tracks verification state independently:

State	Symbol	Meaning
Unverified	🔴	Content exists but no human has reviewed
Reviewed	🟡	Human has reviewed, no hash recorded
Verified	🟢	Human has verified, hash recorded

Multi-granularity tracking: Documents contain fences, and each tracks status independently. A document marked 🟡 (reviewed) may contain fences marked 🔴 (unverified). This is intentional—it allows targeted verification. A reviewer might approve prose content while marking executable fences for specialist review. There is no automatic cascade; status propagation is a policy choice, not a system constraint.

Drift detection: When content changes after verification, hash comparison detects the mismatch. The system can automatically downgrade status (🟢→🔴) or flag for re-review. This provides continuous verification without external tooling.

Inline visibility: Provenance sections auto-inject on save via filesystem hooks. The verification state is always visible inline, not reconstructed from external audit logs. When you read a document, you see its trust level immediately.

North

slots:
- slug: wanderland-paper
  context:
  - Parent paper node
  - Paper parent to system design section

East

slots:
- slug: oculus-cache-architecture
  context:
  - Detailed cache level documentation
- slug: lebowski-corollary
  context:
  - The rug pattern - source of truth architecture
- slug: middleware-pipeline-pattern
  context:
  - Middleware implementation details
- slug: wanderland-paper-implementation
  context:
  - Section sequence

West

slots:
- slug: wanderland-paper-related-work
  context:
  - Previous section
  - Section sequence

Provenance

Document

• Status: 🔴 Unverified

Fences

wanderland-paper-system-design-the-middleware-layer-fence-0

• Status: 🔴 Unverified

wanderland-paper-system-design-north-fence-0

• Status: 🔴 Unverified

wanderland-paper-system-design-east-fence-0

• Status: 🔴 Unverified

wanderland-paper-system-design-west-fence-0

• Status: 🔴 Unverified