mcp-memory/design.md

MCP Memory Server Architecture & Workflow Design

1. Core Architecture

The mcp-memory system runs as a single, continuously running background daemon natively on Windows. It manages the state of the Antigravity knowledge graph and serves as a universal backend for both Windows and WSL environments.

Why this design?

• Cross-OS I/O Optimization: Prevents the WSL agent from performing slow, heavy filesystem writes against the mounted Windows C:\ drive.
• Concurrency & Locking: A single daemon holds the lock on the mcp_memory JSON files, preventing data corruption and eliminating complex delta-reconciliation between parallel WSL and Windows processes.
• Dual Transport System: Utilizes both stdio and HTTP transports concurrently. The local Windows agy instance connects natively via standard stdio, while the Axum HTTP server provides synchronous, non-blocking access for WSL clients and external scripts.

2. Server Transport & Endpoints (Axum)

The daemon uses the axum and rust-mcp-axum crates, binding to 0.0.0.0:3000 to serve the network.

Standard MCP Endpoints

• GET /sse: The Server-Sent Events (SSE) endpoint. Antigravity clients connect here to keep a one-way pipe open for receiving pushed responses and event notifications from the server.
• POST /messages: The JSON-RPC endpoint. Clients use this to send tool calls and resource reads up to the server.

Custom Integration Endpoints

• GET /: The Brain Monitor live HTML dashboard (returns real-time metrics UI).
• GET /api/stats: Returns a JSON snapshot of current entity, relation, task, and tech debt counts.
• GET /gate/verify: A lightweight, deterministic endpoint used by external scripts to verify if an action (like a git push) is authorized based on the current state.

3. Client Connections

The Antigravity configurations (mcp_config.json) utilize the dual-transport system:

• Windows agy: Spawns and connects to the local daemon natively via stdio subprocess execution.
• WSL (Linux) agy: Connects to the running Windows HTTP daemon via the host network proxy, e.g., http://127.0.0.1:3000/sse.

4. Git Integration (Global Wrapper)

Instead of relying on localized per-repository Git hooks (like .git/hooks/pre-push), the system leverages a global bash/PowerShell alias wrapper for the git command. This intercepts git commands universally across the OS.

Workflow Example (Push Safety Gate)

1. The user types git push.
2. The global wrapper intercepts the command.
3. It makes a synchronous HTTP request to the local daemon: curl -s http://localhost:3000/gate/verify.
4. If the endpoint returns 200 OK (indicating the PrePushAuditor subagent has verified that unit tests pass and history is squashed), the push proceeds.
5. If not 200 OK, the wrapper blocks the push and alerts the user to fix tests or run gsquash.

Benefits of the Wrapper Approach

• Universal Enforcement: The push safety gate is protected across all repositories automatically, without copying hook scripts.
• No File Locks: Uses safe, lightweight, parallelizable HTTP requests rather than executing the Rust binary directly.
• Action Logging: The wrapper can be seamlessly extended to log actions (like checkout or commit) directly into the knowledge graph in real-time.

5. Storage Architecture & Persistence (WAL)

The daemon utilizes a Write-Ahead Logging (WAL) architecture for robust, high-performance state management, replacing fragmented delta-file reconciliation and massive synchronous JSON rewrites.

Key Principles:

• Append-Only Log: Every state change (e.g., adding an entity, updating a task) is instantly appended to a sequential, append-only log file (wal.log) before in-memory state is altered.
• Crash Resilience: Eliminates data corruption. If the daemon is forcefully terminated, the system guarantees zero data loss by replaying the WAL against the last known valid checkpoint upon restart.
• Asynchronous Checkpointing: Master JSON store files are no longer rewritten on every tool call. Checkpointing (flushing memory to the master JSON files) is pushed to a background thread to run periodically or upon graceful shutdown, drastically reducing disk I/O.

6. Domain Models & Component Stores

While the core architecture relies on a unified Knowledge Graph (Entities, Relations, Observations), the daemon leverages a modular, thread-safe generic Store pattern for specialized domains. Currently implemented persistent stores include:

• Audit Ledger & Tasks: Tracks agent actions and active background tasks.
• Context & Handoffs: Sticky notes, Session Summaries, Handoff Memos, and Context Workspaces.
• Engineering Tracking: ADRs (Architecture Decision Records), Snippets, Error Fixes, Tech Debt, and PR Checklists.
• Environment State: Pinned Files, Env Fingerprints, Milestones, and Environments.

7. Background Maintenance (Reconcile Worker)

The daemon runs a continuous asynchronous background task ( econcile_worker) on a 5-second polling loop responsible for:

• State Reconciliation: Reading the append-only wal.jsonl file, squashing the mutations into the master knowledge graph, and cleanly truncating the WAL.
• Ledger Pruning: Automatically truncating the audit_ledger.json to keep only the last 7 days of activity, with a hard cap of 1,000 records to prevent infinite bloat.
• Ephemeral Data Cleanup: Automatically expiring and purging sticky_notes.json that are older than 24 hours.

8. The Gate System (Push Safety Verification)

The binary includes dedicated CLI subcommands (gate set and gate verify) that interact with a persistent gates.json store to enforce safety policies (like ensuring tests pass before a git push).

• mcp-memory-stub gate set: Records an authorization status (authorized, blocked, or pending) for a specific target and namespace, alongside optional failure reasons and parameters.
• mcp-memory-stub gate verify: Evaluates a pending action against the gate store. It returns exit code if authorized, 1 if explicitly blocked, and 2 if no gate record exists. It also supports a --consume flag to immediately revoke the authorization after a successful check.

9. Operational Configuration & Paths

The physical storage location of the knowledge graph and all persistent stores is strictly controlled by the MCP_MEMORY_STORE_DIR environment variable.

• Default Path: If not set, the daemon defaults to ~/.gemini/mcp_memory.
• Port Binding: The Axum HTTP server strictly binds to .0.0.0:3000.

10. Exposed MCP Capabilities (Tools)

The server implements the Model Context Protocol (MCP) by exposing a vast suite of tools via the JSON-RPC interface, categorized broadly into:

• Graph Management: create_entities, create_relations, merge_entities, ead_graph, etc.
• Task & Context Tracking: add_task, add_sticky_note, add_session_summary, etc.
• Engineering & DevOps: log_code_change, log_error_fix, log_tech_debt, add_pr_checklist_item.
• Environment & Workspaces: egister_environment, save_context_workspace, pin_file.

11. Concurrency & Thread Safety

With the introduction of the Dual Transport System, the daemon must safely handle simultaneous read/write requests from both Stdio (Windows agy) and HTTP (WSL agy) clients.

• Shared State: The entire server operates on a cloned Arc.
• Locking Mechanism: The unified Knowledge Graph and modular Store components are protected by RwLock primitives.
• Safe Mutation: Store modifications utilize closure-based modify(|store| { ... }) methods to ensure locks are safely acquired, mutations applied, and file writes executed sequentially without deadlocking the asynchronous Tokio runtimes.

12. Lifecycle & Startup Management (Server/Stub Architecture)

The mcp-memory daemon employs a self-healing, morphing Server/Client paradigm to allow multiple concurrent agy sessions without port collisions or file lock contention.

The Windows Background Daemon

Every time a Windows gy session starts, it blindly spawns mcp-memory-stub.exe as a proxy. The proxy expects the mcp-memory-server.exe daemon to be running natively on port 3000.

WSL (Linux) Client Lifecycle (Permanent Stub)

To provide a seamless experience without complex configuration drift, the WSL environment utilizes a native Linux binary (/home/riz/.local/bin/mcp-memory-stub) that acts as a Permanent Stub.

• Transparent Proxying: The WSL agy CLI spawns this Linux binary via standard stdio. The binary immediately proxies all stdio JSON-RPC requests over HTTP to the Windows Leader at http://127.0.0.1:3000/messages, handling SSE streams transparently.
• Startup via Interop: Upon launch, the Linux Stub pings the Windows host. If port 3000 is dead, the Linux binary automatically executes WSL interop (cmd.exe /c start /B C:\Users\reazul.ashraf\.local\bin\mcp-memory.exe) to silently wake up the Windows Leader before commencing the proxy loop.
• Zero I/O Penalty: This ensures the Linux binary never directly touches the Windows NTFS files, reserving all heavy disk operations for the native Windows host.

13. Cargo Workspace & Binary Artifacts

To optimize for different environments, the codebase is structured as a Cargo Workspace containing two distinct crates:

1. mcp-memory-server (The "Full-Fat" Daemon)

• Path: server/
• Size/Complexity: Heavy (contains Axum, MCP SDK, JSON parsing, Tokio runtime).
• Role: This is the primary background daemon. It binds to .0.0.0:3000, holds file locks, and manages the graph.
• Windows Behavior: It is designed to run in the background as a standalone service.

2. mcp-memory-stub (The Ultra-Lightweight Proxy)

• Path: stub/
• Size/Complexity: Extremely light (only relies on eqwest and okio).
• Role: A dedicated, OS-agnostic proxy binary used strictly for routing stdio JSON-RPC traffic over HTTP to a remote Leader. Windows gy clients point directly to this binary to bypass loading the heavy Server daemon into memory.
• WSL Behavior: Compiled as a Linux native binary (x86_64-unknown-linux-musl). When executed by WSL agy, it acts as a transparent proxy to http://127.0.0.1:3000. It can also execute wake_cmd (e.g., WSL interop) to silently wake the Windows host if the Leader is offline.