From c88d312c8c147b87c60937f62e8d82bbd7b150ae Mon Sep 17 00:00:00 2001 From: Riz Ashraf Date: Mon, 14 Sep 2026 04:13:51 +0100 Subject: [PATCH] docs: align design docs with redb, tantivy, and native rust e2e testing architectures --- design.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/design.md b/design.md index 1b711ca..82d903c 100644 --- a/design.md +++ b/design.md @@ -40,28 +40,27 @@ Instead of relying on localized per-repository Git hooks (like `.git/hooks/pre-p * **Universal Enforcement:** The push safety gate is protected across all repositories automatically, without copying hook scripts. * **No File Locks:** Uses safe, lightweight, parallelizable HTTP requests rather than executing the Rust binary directly. * **Action Logging:** The wrapper can be seamlessly extended to log actions (like `checkout` or `commit`) directly into the knowledge graph in real-time. -## 5. Storage Architecture & Persistence (WAL) -The daemon utilizes a Write-Ahead Logging (WAL) architecture for robust, high-performance state management, replacing fragmented delta-file reconciliation and massive synchronous JSON rewrites. +## 5. Storage Architecture & Persistence (Redb LSM-Tree) +The daemon has completely eliminated raw JSON file sprawl and fragmented delta-file reconciliation. It now utilizes a pure-Rust, embedded Key-Value engine (`redb`) that implements a robust Log-Structured Merge-Tree (LSM-tree) architecture. ### Key Principles: -* **Append-Only Log:** Every state change (e.g., adding an entity, updating a task) is instantly appended to a sequential, append-only log file (wal.log) *before* in-memory state is altered. -* **Crash Resilience:** Eliminates data corruption. If the daemon is forcefully terminated, the system guarantees zero data loss by replaying the WAL against the last known valid checkpoint upon restart. -* **Asynchronous Checkpointing:** Master JSON store files are no longer rewritten on every tool call. Checkpointing (flushing memory to the master JSON files) is pushed to a background thread to run periodically or upon graceful shutdown, drastically reducing disk I/O. +* **Embedded Database Engine:** All structured components (Tasks, Snippets, Tech Debt, Checklists, etc.) are stored as binary-encoded values inside a unified `redb` database file (`store.redb`). +* **ACID Compliance & File Locks:** The Windows daemon holds an exclusive read-write lock on the database file, guaranteeing zero data corruption, race conditions, or lock contention during concurrent access. +* **Asynchronous Checkpointing:** The core Knowledge Graph (Entities, Relations, Observations) still utilizes a Write-Ahead Logging (WAL) pattern (`wal.jsonl`) and a master snapshot (`master.json`) to allow safe, lock-free memory mutations which are reconciled in the background. ## 6. Domain Models & Component Stores -While the core architecture relies on a unified Knowledge Graph (Entities, Relations, Observations), the daemon leverages a modular, thread-safe generic Store pattern for specialized domains. +The system leverages a modular, thread-safe generic `Store` abstraction that automatically transparently serializes and deserializes native Rust structs directly into the underlying `redb` tables. Currently implemented persistent stores include: * **Audit Ledger & Tasks:** Tracks agent actions and active background tasks. * **Context & Handoffs:** Sticky notes, Session Summaries, Handoff Memos, and Context Workspaces. * **Engineering Tracking:** ADRs (Architecture Decision Records), Snippets, Error Fixes, Tech Debt, and PR Checklists. * **Environment State:** Pinned Files, Env Fingerprints, Milestones, and Environments. +* **Safety Gates:** Authorized execution gates (Push Safety). -## 7. Background Maintenance (Reconcile Worker) -The daemon runs a continuous asynchronous background task ( -econcile_worker) on a 5-second polling loop responsible for: -* **State Reconciliation:** Reading the append-only wal.jsonl file, squashing the mutations into the master knowledge graph, and cleanly truncating the WAL. -* **Ledger Pruning:** Automatically truncating the audit_ledger.json to keep only the last 7 days of activity, with a hard cap of 1,000 records to prevent infinite bloat. -* **Ephemeral Data Cleanup:** Automatically expiring and purging sticky_notes.json that are older than 24 hours. +## 7. Full-Text Search Engine (Tantivy) +To support blazing-fast, intelligent semantic retrieval across the sprawling knowledge graph, the daemon embeds **Tantivy**, a full-text search engine (inspired by Apache Lucene). +* **The `MemoryIndex`:** Whenever the graph or auxiliary stores mutate, a background thread dynamically rebuilds the Tantivy index (`tantivy_index/` dir). +* **Global Omni-Search:** This architecture powers the `omni_search` tool, allowing subagents to instantly fuzzy-search and rank documents across Entities, Tasks, Snippets, Error Fixes, and ADRs simultaneously in milliseconds, without loading massive JSON arrays into RAM. ## 8. The Gate System (Push Safety Verification) The binary supports a flexible safety gate authorization system, accessible both via the HTTP API and CLI subcommands. @@ -149,4 +148,8 @@ Because the background server operates as an always-on Windows daemon, standard The \uild.ps1\ deployment pipeline intercepts locked \.exe\ files by appending a unique, timestamped/randomized suffix (e.g., \mcp-memory-server.exe.12345.old\) when forcing a \Move-Item\. This guarantees that rapid sequential deployments (where a previous \.old\ file might still be locked by a zombie process) never silently fail or collide. ### Dynamic Versioning -To trace binary provenances during rapid deployment cycles, all binaries embed dynamic versioning directly at compile time (via \uild.rs\ and \uild_template.rs\). The injected \APP_VERSION\ environment variable combines the static Cargo \ ersion\ with the live \git\ short hash and UTC timestamp, allowing the CLI \--version\ commands and the HTTP \/api/version\ endpoints to guarantee exactly which iteration of the code is actively executing. \ No newline at end of file +To trace binary provenances during rapid deployment cycles, all binaries embed dynamic versioning directly at compile time (via \uild.rs\ and \uild_template.rs\). The injected \APP_VERSION\ environment variable combines the static Cargo \ ersion\ with the live \git\ short hash and UTC timestamp, allowing the CLI \--version\ commands and the HTTP \/api/version\ endpoints to guarantee exactly which iteration of the code is actively executing. +## 16. Testing Architecture (Native Rust E2E) +Historically, the project relied on a complex Python testing suite (\pytest\ + \mcp_client.py\) to validate the server over HTTP/SSE. This has been fully deprecated in favor of **Native Rust End-to-End Testing**. +* **Unit Tests:** Handlers and business logic are tested directly inside \server/src/handlers.rs\ using native \ okio::test\ constructs. +* **E2E Tests:** Integration and full-system tests run via \stub/tests/e2e.rs\ and \win-nvim/tests/integration_test.rs\, ensuring type safety, faster execution, and eliminating Python environment dependencies. \ No newline at end of file