docs: align design docs with redb, tantivy, and native rust e2e testing architectures

design.md

@@ -40,28 +40,27 @@ Instead of relying on localized per-repository Git hooks (like `.git/hooks/pre-p
 *   **Universal Enforcement:** The push safety gate is protected across all repositories automatically, without copying hook scripts.
 *   **No File Locks:** Uses safe, lightweight, parallelizable HTTP requests rather than executing the Rust binary directly.
 *   **Action Logging:** The wrapper can be seamlessly extended to log actions (like `checkout` or `commit`) directly into the knowledge graph in real-time.
-## 5. Storage Architecture & Persistence (WAL)
+## 5. Storage Architecture & Persistence (Redb LSM-Tree)
-The daemon utilizes a Write-Ahead Logging (WAL) architecture for robust, high-performance state management, replacing fragmented delta-file reconciliation and massive synchronous JSON rewrites.
+The daemon has completely eliminated raw JSON file sprawl and fragmented delta-file reconciliation. It now utilizes a pure-Rust, embedded Key-Value engine (`redb`) that implements a robust Log-Structured Merge-Tree (LSM-tree) architecture.
 
 ### Key Principles:
-*   **Append-Only Log:** Every state change (e.g., adding an entity, updating a task) is instantly appended to a sequential, append-only log file (wal.log) *before* in-memory state is altered.
+*   **Embedded Database Engine:** All structured components (Tasks, Snippets, Tech Debt, Checklists, etc.) are stored as binary-encoded values inside a unified `redb` database file (`store.redb`).
-*   **Crash Resilience:** Eliminates data corruption. If the daemon is forcefully terminated, the system guarantees zero data loss by replaying the WAL against the last known valid checkpoint upon restart.
+*   **ACID Compliance & File Locks:** The Windows daemon holds an exclusive read-write lock on the database file, guaranteeing zero data corruption, race conditions, or lock contention during concurrent access.
-*   **Asynchronous Checkpointing:** Master JSON store files are no longer rewritten on every tool call. Checkpointing (flushing memory to the master JSON files) is pushed to a background thread to run periodically or upon graceful shutdown, drastically reducing disk I/O.
+*   **Asynchronous Checkpointing:** The core Knowledge Graph (Entities, Relations, Observations) still utilizes a Write-Ahead Logging (WAL) pattern (`wal.jsonl`) and a master snapshot (`master.json`) to allow safe, lock-free memory mutations which are reconciled in the background.
 
 ## 6. Domain Models & Component Stores
-While the core architecture relies on a unified Knowledge Graph (Entities, Relations, Observations), the daemon leverages a modular, thread-safe generic Store<T> pattern for specialized domains. 
+The system leverages a modular, thread-safe generic `Store<T>` abstraction that automatically transparently serializes and deserializes native Rust structs directly into the underlying `redb` tables.
 Currently implemented persistent stores include:
 *   **Audit Ledger & Tasks:** Tracks agent actions and active background tasks.
 *   **Context & Handoffs:** Sticky notes, Session Summaries, Handoff Memos, and Context Workspaces.
 *   **Engineering Tracking:** ADRs (Architecture Decision Records), Snippets, Error Fixes, Tech Debt, and PR Checklists.
 *   **Environment State:** Pinned Files, Env Fingerprints, Milestones, and Environments.
+*   **Safety Gates:** Authorized execution gates (Push Safety).
 
-## 7. Background Maintenance (Reconcile Worker)
+## 7. Full-Text Search Engine (Tantivy)
-The daemon runs a continuous asynchronous background task (
+To support blazing-fast, intelligent semantic retrieval across the sprawling knowledge graph, the daemon embeds **Tantivy**, a full-text search engine (inspired by Apache Lucene).
-econcile_worker) on a 5-second polling loop responsible for:
+*   **The `MemoryIndex`:** Whenever the graph or auxiliary stores mutate, a background thread dynamically rebuilds the Tantivy index (`tantivy_index/` dir).
-*   **State Reconciliation:** Reading the append-only wal.jsonl file, squashing the mutations into the master knowledge graph, and cleanly truncating the WAL.
+*   **Global Omni-Search:** This architecture powers the `omni_search` tool, allowing subagents to instantly fuzzy-search and rank documents across Entities, Tasks, Snippets, Error Fixes, and ADRs simultaneously in milliseconds, without loading massive JSON arrays into RAM.
-*   **Ledger Pruning:** Automatically truncating the audit_ledger.json to keep only the last 7 days of activity, with a hard cap of 1,000 records to prevent infinite bloat.
-*   **Ephemeral Data Cleanup:** Automatically expiring and purging sticky_notes.json that are older than 24 hours.
 
 ## 8. The Gate System (Push Safety Verification)
 The binary supports a flexible safety gate authorization system, accessible both via the HTTP API and CLI subcommands.
@@ -150,3 +149,7 @@ The \uild.ps1\ deployment pipeline intercepts locked \.exe\ files by appending
 
 ### Dynamic Versioning
 To trace binary provenances during rapid deployment cycles, all binaries embed dynamic versioning directly at compile time (via \uild.rs\ and \uild_template.rs\). The injected \APP_VERSION\ environment variable combines the static Cargo \ersion\ with the live \git\ short hash and UTC timestamp, allowing the CLI \--version\ commands and the HTTP \/api/version\ endpoints to guarantee exactly which iteration of the code is actively executing.
+## 16. Testing Architecture (Native Rust E2E)
+Historically, the project relied on a complex Python testing suite (\pytest\ + \mcp_client.py\) to validate the server over HTTP/SSE. This has been fully deprecated in favor of **Native Rust End-to-End Testing**.
+*   **Unit Tests:** Handlers and business logic are tested directly inside \server/src/handlers.rs\ using native \	okio::test\ constructs.
+*   **E2E Tests:** Integration and full-system tests run via \stub/tests/e2e.rs\ and \win-nvim/tests/integration_test.rs\, ensuring type safety, faster execution, and eliminating Python environment dependencies.