mcp-memory/design.md

# MCP Memory Server Architecture & Workflow Design

## 1. Core Architecture
The `mcp-memory` system runs as a **single, continuously running background daemon natively on Windows**. It manages the state of the Antigravity knowledge graph and serves as a universal backend for both Windows and WSL environments.

### Why this design?
*   **Cross-OS I/O Optimization:** Prevents the WSL agent from performing slow, heavy filesystem writes against the mounted Windows `C:\` drive.
*   **Concurrency & Locking:** A single daemon holds the lock on the `mcp_memory` JSON files, preventing data corruption and eliminating complex delta-reconciliation between parallel WSL and Windows processes.
*   **Dual Transport System:** Utilizes both `stdio` and HTTP transports concurrently. The local Windows `agy` instance connects natively via standard `stdio`, while the Axum HTTP server provides synchronous, non-blocking access for WSL clients and external scripts.

## 2. Server Transport & Endpoints (Axum)
The daemon uses the `axum` and `rust-mcp-axum` crates, binding to `0.0.0.0:3000` to serve the network.

### Standard MCP Endpoints
*   `GET /ws`: The WebSocket endpoint for efficient, low-latency, full-duplex JSON-RPC communication (preferred for proxies and the UI dashboard).
*   `GET /sse`: The Server-Sent Events (SSE) endpoint. Antigravity clients connect here to keep a one-way pipe open for receiving pushed responses.
*   `POST /messages`: The JSON-RPC endpoint. Clients use this to send tool calls and resource reads up to the server when connected via SSE.

### Dashboard REST API
The server hosts a rich Single Page Application (SPA) natively on the root route, backed by a suite of REST endpoints:
*   `GET /`: The Brain Monitor live HTML dashboard SPA.
*   `GET /api/graph`: Returns the full node/edge topology for the interactive physics-simulated canvas.
*   `GET /api/tasks` & `POST /api/tasks/{id}/complete`: Drives the actionable Kanban board.
*   `GET /api/sticky` & `GET /api/search`: Powers the sticky notes tab and global fuzzy search UI.
*   `GET /api/stats`: Returns a JSON snapshot of current entity, relation, task, and tech debt counts.

### Git & IDE Integration Endpoints
*   `UDP /nvim/telemetry`: A one-way connectionless datagram listener that Neovim instances hit on `FocusGained` or `BufEnter` to globally broadcast the user's active file to the UI and Agent.
*   `GET /gate/verify`: A lightweight, deterministic endpoint used by external scripts (like a `git push` wrapper) to verify if an action is authorized based on the current state.

## 3. Client Connections
The Antigravity configurations (`mcp_config.json`) utilize the dual-transport system:
*   **Windows `agy`:** Spawns and connects to the local daemon natively via `stdio` subprocess execution.
*   **WSL (Linux) `agy`:** Connects to the running Windows HTTP daemon via the host network proxy, e.g., `http://127.0.0.1:3000/sse`.

## 4. Git Integration (Global Wrapper)
Instead of relying on localized per-repository Git hooks (like `.git/hooks/pre-push`), the system leverages a **global bash/PowerShell alias wrapper** for the `git` command. This intercepts `git` commands universally across the OS.

### Workflow Example (Push Safety Gate)
1.  The user types `git push`.
2.  The global wrapper intercepts the command.
3.  It makes a synchronous HTTP request to the local daemon: `curl -s http://localhost:3000/gate/verify`.
4.  If the endpoint returns `200 OK` (indicating the `PrePushAuditor` subagent has verified that unit tests pass and history is squashed), the push proceeds.
5.  If not `200 OK`, the wrapper blocks the push and alerts the user to fix tests or run `gsquash`.

### Benefits of the Wrapper Approach
*   **Universal Enforcement:** The push safety gate is protected across all repositories automatically, without copying hook scripts.
*   **No File Locks:** Uses safe, lightweight, parallelizable HTTP requests rather than executing the Rust binary directly.
*   **Action Logging:** The wrapper can be seamlessly extended to log actions (like `checkout` or `commit`) directly into the knowledge graph in real-time.
## 5. Storage Architecture & Persistence (Redb LSM-Tree)
The daemon has completely eliminated raw JSON file sprawl and fragmented delta-file reconciliation. It now utilizes a pure-Rust, embedded Key-Value engine (`redb`) that implements a robust Log-Structured Merge-Tree (LSM-tree) architecture.

### Key Principles:
*   **Embedded Database Engine:** All structured components (Tasks, Snippets, Tech Debt, Checklists, etc.) are stored as binary-encoded values inside a unified `redb` database file (`store.redb`).
*   **ACID Compliance & File Locks:** The Windows daemon holds an exclusive read-write lock on the database file, guaranteeing zero data corruption, race conditions, or lock contention during concurrent access.
*   **Asynchronous Checkpointing:** The core Knowledge Graph (Entities, Relations, Observations) still utilizes a Write-Ahead Logging (WAL) pattern (`wal.jsonl`) and a master snapshot (`master.json`) to allow safe, lock-free memory mutations which are reconciled in the background.

## 6. Domain Models & Component Stores
The system leverages a modular, thread-safe generic `Store<T>` abstraction that automatically transparently serializes and deserializes native Rust structs directly into the underlying `redb` tables.
Currently implemented persistent stores include:
*   **Audit Ledger & Tasks:** Tracks agent actions and active background tasks.
*   **Context & Handoffs:** Sticky notes, Session Summaries, Handoff Memos, and Context Workspaces.
*   **Engineering Tracking:** ADRs (Architecture Decision Records), Snippets, Error Fixes, Tech Debt, and PR Checklists.
*   **Environment State:** Pinned Files, Env Fingerprints, Milestones, and Environments.
*   **Safety Gates:** Authorized execution gates (Push Safety).

## 7. Full-Text & Semantic Search Engine (Tantivy + FastEmbed)
To support blazing-fast, intelligent semantic retrieval across the sprawling knowledge graph, the daemon embeds **Tantivy** (a full-text search engine inspired by Apache Lucene) alongside **FastEmbed** (a local ONNX runtime for vector embeddings).
*   **The `MemoryIndex`:** Whenever the graph or auxiliary stores mutate, a background thread dynamically rebuilds the Tantivy index (`tantivy_index/` dir) and computes semantic vectors.
*   **Global Omni-Search:** This architecture powers the `omni_search` tool, allowing subagents to instantly fuzzy-search and semantically rank documents across Entities, Tasks, Snippets, Error Fixes, and ADRs simultaneously in milliseconds, without loading massive JSON arrays into RAM.

## 8. Webhook Telemetry & Passive Ingestion
The server features a suite of webhook listeners that passively ingest development activity to build context without requiring human copy-pasting:
*   **Terminal Ingestion (`/terminal/telemetry`):** Shell hooks (PowerShell/Nushell) silently POST command execution history and exit codes, allowing agents to read recent stack traces via the `memory://terminal/recent` resource.
*   **IDE Telemetry (`/nvim/telemetry`):** A connectionless UDP datagram listener receives focus events from Neovim, broadcasting the user's active file to the UI and Agent instantly.

## 9. The Gate System (Push Safety Verification)
The binary supports a flexible safety gate authorization system, accessible both via the HTTP API and CLI subcommands.
*   **API / CLI set**: Records an authorization status (`authorized`, `blocked`, or `pending`) for a specific target and namespace. Can be invoked via `POST /gate/set` (JSON) or `mcp-memory-stub gate set`.
*   **API / CLI verify**: Evaluates a pending action against the gate store. It returns standard HTTP status codes (`200 OK`, `403 Forbidden`, `404 Not Found`) via `GET /gate/verify?action=...` or POSIX exit codes (0, 1, 2) via `mcp-memory-stub gate verify`. Both support a `consume` parameter/flag to immediately revoke the authorization after a successful check.

## 9. Operational Configuration & Paths
The physical storage location of the knowledge graph and all persistent stores is strictly controlled by the MCP_MEMORY_STORE_DIR environment variable.
*   **Default Path:** If not set, the daemon defaults to ~/.gemini/mcp_memory.
*   **Port Binding:** The Axum HTTP server strictly binds to .0.0.0:3000.

## 10. Exposed MCP Capabilities (Tools)
The server implements the Model Context Protocol (MCP) by exposing a vast suite of tools via the JSON-RPC interface, categorized broadly into:
*   **Graph Management:** create_entities, create_relations, merge_entities, 
ead_graph, etc.
*   **Task & Context Tracking:** add_task, add_sticky_note, add_session_summary, etc.
*   **Engineering & DevOps:** log_code_change, log_error_fix, log_tech_debt, add_pr_checklist_item.
*   **Environment & Workspaces:** 
egister_environment, save_context_workspace, pin_file.

## 11. Concurrency & Thread Safety
With the introduction of the Dual Transport System, the daemon must safely handle simultaneous read/write requests from both Stdio (Windows agy) and HTTP (WSL agy) clients.
*   **Shared State:** The entire server operates on a cloned Arc<MemoryState>.
*   **Locking Mechanism:** The unified Knowledge Graph and modular Store<T> components are protected by RwLock primitives. 
*   **Safe Mutation:** Store modifications utilize closure-based modify(|store| { ... }) methods to ensure locks are safely acquired, mutations applied, and file writes executed sequentially without deadlocking the asynchronous Tokio runtimes.

## 12. Lifecycle & Startup Management (Server/Stub Architecture)
The mcp-memory daemon employs a strict Server/Client paradigm to maintain separation of concerns. There are no dual roles or morphing executables.

### The Windows Background Daemon (`mcp-memory-server.exe`)
The server executable is responsible exclusively for running the Axum HTTP and WebSockets daemon and managing the Knowledge Graph. It does not contain any proxy logic. If port 3000 is already in use by an existing server instance, it gracefully exits instead of attempting to run.

*   **Non-Blocking Startup & Indexing:** Upon launch, the server uses a `tokio::spawn` task to rebuild the Tantivy index in the background. This ensures the Axum HTTP server binds immediately to port 3000, allowing instantaneous liveness checks via the `/ping` endpoint without waiting for the synchronous I/O operations of the index rebuild to complete.
*   **Robust Background Detachment:** To ensure the server survives the death of the parent shell that spawns it (such as a deployment script), it uses Windows `WScript.Shell` (via `start_server.ps1`) to spawn the binary with a strictly hidden window style (`SW_HIDE`). This completely detaches it from the parent process tree, preventing `CTRL_CLOSE_EVENT` signals from killing the background daemon when the deploying terminal closes.

### The Client Proxy (`mcp-memory-stub.exe`)
All Antigravity sessions (Windows and WSL) use the lightweight `mcp-memory-stub` as their proxy. The stub connects to the server via WebSockets and acts as the bridge for standard `stdio` JSON-RPC traffic.
*   **Startup via Interop/Spawn:** Upon launch, the stub attempts to connect to the Windows host on port 3000. If the server is offline, the stub automatically executes a spawn command (e.g., executing `Start-Process` natively, or via WSL interop) to silently wake up the Windows Leader before commencing the proxy loop.
*   **MPSC Queue Resilience:** The stub utilizes an asynchronous multi-producer, single-consumer (MPSC) channel queue. If the Windows Leader daemon restarts or momentarily drops, the proxy buffers incoming JSON-RPC tool calls and infinitely retries them until the connection is restored. This guarantees **zero message loss** and **zero thread leaks** without crashing the active `agy` session.
*   **Zero I/O Penalty (WSL):** This ensures the Linux binary never directly touches the Windows NTFS files, reserving all heavy disk operations for the native Windows host.
## 13. Cargo Workspace & Binary Artifacts
To optimize for different environments, the codebase is structured as a Cargo Workspace containing two distinct crates:

### 1. mcp-memory-server (The "Full-Fat" Daemon)
*   **Path:** server/
*   **Size/Complexity:** Heavy (contains Axum, MCP SDK, JSON parsing, Tokio runtime).
*   **Role:** This is the primary background daemon. It binds to .0.0.0:3000, holds file locks, and manages the graph.
*   **Windows Behavior:** It is designed to run in the background as a standalone service.

### 2. mcp-memory-stub (The Ultra-Lightweight Proxy)
*   **Path:** stub/
*   **Size/Complexity:** Extremely light (only relies on 
eqwest and 	okio).
*   **Role:** A dedicated, OS-agnostic proxy binary used strictly for routing stdio JSON-RPC traffic over HTTP to a remote Leader. Windows gy clients point directly to this binary to bypass loading the heavy Server daemon into memory.
*   **WSL Behavior:** Compiled as a Linux native binary (x86_64-unknown-linux-musl). When executed by WSL agy, it acts as a transparent proxy to http://127.0.0.1:3000. It can also execute wake_cmd (e.g., WSL interop) to silently wake the Windows host if the Leader is offline.



## 14. Neovim Integration Architecture & "God Mode"
To enable seamless pair-programming inside Neovim, the daemon integrates with Neovim using two complementary systems: a Webhook Telemetry pipeline and dedicated MCP binaries.

### Telemetry Pipeline (Last Focused Wins)
A lightweight Lua script (`gemini-integration.lua`) is loaded into Neovim, which fires an asynchronous UDP datagram to `127.0.0.1:3002` (fire-and-forget via native libuv) whenever the user focuses a buffer or moves the cursor. The server then writes this data (including `session_id`, `file`, `line`, and `col`) to both the Windows and WSL `active_nvim.txt` files and broadcasts it over WebSockets.

### Native MCP Binaries (win-nvim & linux-nvim)
The project compiles two standalone, highly-performant binaries that implement the MCP JSON-RPC protocol over Stdio and bridge it directly to Neovim's Msgpack-RPC Named Pipes/Sockets. These binaries avoid hardcoding infinite tools by utilizing a "God Mode" escape hatch.

Exposed Neovim Tools:
*   **
vim_get_active_buffer & 
vim_get_cursor**: Read file state.
*   **
vim_goto_line & 
vim_set_diagnostics**: Manipulate IDE state.
*   **
vim_get_visual_selection**: Read exact highlight coordinates (handles mode dynamically).
*   **
vim_list_buffers**: Discover unsaved work and context.
*   **
vim_get_diagnostics**: Read live LSP errors dynamically instead of requiring a compiler.
*   **
vim_execute_lua ("God Mode")**: The ultimate fallback tool. Evaluates raw Lua scripts inside the active Neovim instance and returns JSON. This prevents the need to continuously recompile the Rust server whenever a new Neovim capability is required.

## 15. Build & Deployment Strategy
Because the background server operates as an always-on Windows daemon, standard recompilation and file-copying strategies will fail due to active Windows OS file locks. 

### Randomized Lock Bypassing
The \uild.ps1\ deployment pipeline intercepts locked \.exe\ files by appending a unique, timestamped/randomized suffix (e.g., \mcp-memory-server.exe.12345.old\) when forcing a \Move-Item\. This guarantees that rapid sequential deployments (where a previous \.old\ file might still be locked by a zombie process) never silently fail or collide.

### Dynamic Versioning
To trace binary provenances during rapid deployment cycles, all binaries embed dynamic versioning directly at compile time (via \uild.rs\ and \uild_template.rs\). The injected \APP_VERSION\ environment variable combines the static Cargo \ersion\ with the live \git\ short hash and UTC timestamp, allowing the CLI \--version\ commands and the HTTP \/api/version\ endpoints to guarantee exactly which iteration of the code is actively executing.
## 16. Testing Architecture (Native Rust E2E)
Historically, the project relied on a complex Python testing suite (\pytest\ + \mcp_client.py\) to validate the server over HTTP/SSE. This has been fully deprecated in favor of **Native Rust End-to-End Testing**.
*   **Unit Tests:** Handlers and business logic are tested directly inside \server/src/handlers.rs\ using native \	okio::test\ constructs.
*   **E2E Tests:** Integration and full-system tests run via \stub/tests/e2e.rs\ and \win-nvim/tests/integration_test.rs\, ensuring type safety, faster execution, and eliminating Python environment dependencies.
## 17. Automated Git Context Binding (git2)
Instead of forcing the LLM client to manually run `git rev-parse HEAD` and pass `git_branch` / `git_commit` arguments for every single code change, the server integrates the native **`git2`** C bindings. 
When engineering endpoints (`log_code_change`, `log_error_fix`, `log_tech_debt`) are invoked, the server asynchronously discovers the surrounding Git repository, peels the HEAD reference, and automatically injects the current commit hash, commit message, and branch name directly into the stored entities and audit logs. This guarantees airtight VCS traceability without wasting LLM tokens or relying on the agent's memory.

## 18. Upcoming Enhancements

### A. Graph Summarization & Decay
To prevent context window bloat, the Redb engine will enforce TTLs on ephemeral nodes (`StickyNotes`, minor `Tasks`). Furthermore, an `archive_routine` prompt will allow subagents to automatically compress older `SessionSummaries` into dense milestone retrospectives.

### B. Proactive Subagent Triggers
The daemon will eventually gain the ability to autonomously spawn background subagents (like the `BugDiagnostician`) when specific events are detected in the webhook telemetry, rather than relying strictly on the human user to initiate the agent.

## 11. Advanced Clipboard Capabilities
The memory server bypasses typical cross-OS Linux/Windows clipboard restrictions by proxying operations from the WSL stub back to the Windows Axum host. 
It supports reading and writing rich formats natively to the Windows Host OS using the clipboard-win and rboard crates:
*   **Plain & Rich Text:** CF_UNICODETEXT and CF_HTML are supported for rich copying/pasting.
*   **File Drops (CF_HDROP):** The server can parse file lists copied from Windows Explorer, and can inversely synthesize file drops into the clipboard from absolute paths.
*   **Images (CF_BITMAP):** The server natively rasterizes clipboard bitmaps to JPEG on read, and can write raw RgbaImage buffers back to the clipboard on write.
*   **Developer Tooling:** read_file_skeleton (AST), get_active_worktree_context (Git), get_recent_logs, toggle_clipboard_watch_mode.