feat(embedding,vision): candle embeddings with offline fallback, on-demand clipboard vision capture, and concurrency audit

instructions.md

@@ -21,20 +21,25 @@ To prevent graph fragmentation and ensure seamless LLM context retrieval:
 
 ## 2. Consolidated Smart Tools Architecture
 
+> [!IMPORTANT] The Two-Tier Context Paradigm
+> * **Tier 1 (Static Markdown)**: Repository rules, constraints, architectural patterns, and developer preferences are maintained directly in static git-tracked files (`rules/*.md`, `instructions.md`) and system prompts. This guarantees 0ms turn-0 availability without relying on proactive agent tool retrieval.
+> * **Tier 2 (Telemetry & Ephemeral DB)**: High-frequency operational history—the Code Change Ledger (`audit_ledger`), terminal command history, compiler error fixes (`log_error_fix`), active tasks, and preflight context—is handled by the MCP Memory Server and surfaced via the Brain Monitor Web UI and MCP resources.
+
+
 The server consolidates granular single-purpose tools into domain-named smart tools. Always prefer the consolidated tools over legacy aliases:
 
 * **`tasks`**: Complete task lifecycle management.
-  - `action: "add"`: Create a new task (requires `title`, optional `description`, `git_branch`, `parent_id`, `dependencies`).
+  - `action: "add"`: Create a new task (requires `title`, optional `description`, `git_branch`, `repo_name`, `priority: "low" | "medium" | "high" | "urgent"`, `assigned_agent`, `verification_command`, `parent_id`, `dependencies`).
   - `action: "update"`: Update task status (requires `id`, `status: "pending" | "completed" | "cancelled"`).
   - `action: "delete"`: Delete task and child tasks (requires `id`).
   - `action: "list"`: List active tasks (optional `git_branch`, `summary_level: "compact" | "detailed" | "full"`, `max_tokens`).
   - `action: "set_criteria"`: Set acceptance criteria (requires `id`, `criteria: Vec<String>`).
-  - `action: "verify"`: Verify criteria met (requires `id`).
+  - `action: "verify"`: Verify criteria met (requires `id`, optional `proof`).
 
 * **`milestones`**: Milestone tracking.
-  - `action: "add"`: Create milestone (requires `title`).
-  - `action: "update"`: Update milestone status (requires `id`, `status`).
-  - `action: "list"`: List milestones.
+  - `action: "add"`: Create milestone (requires `title`, optional `namespace`, `target_date`, `description`, `deliverables: Vec<String>`, `repo_name`).
+  - `action: "update"`: Update milestone status (requires `id`, `status: "active" | "completed" | "cancelled"`).
+  - `action: "list"`: List milestones (optional `namespace`).
 
 * **`sticky_notes`**: Ephemeral scratchpad notes with TTL.
   - `action: "add"`: Add note (requires `content`, optional `ttl_seconds`, `session_only`).
@@ -43,48 +48,48 @@ The server consolidates granular single-purpose tools into domain-named smart to
   - `action: "clear"`: Clear all sticky notes.
 
 * **`handoff_memos`**: Session handoff notes for future agents.
-  - `action: "leave"`: Leave a memo (requires `content`).
+  - `action: "leave"`: Leave a memo (requires `content`, optional `vcs_revision`, `repo_name`, `git_branch`, `blockers: Vec<String>`, `action_items: Vec<String>`, `expires_at`).
   - `action: "read"`: Read active handoff memos.
   - `action: "clear"`: Clear memos.
 
 * **`pinned_files`**: Focus file working set.
-  - `action: "pin"`: Pin file to focus set (requires `path`).
+  - `action: "pin"`: Pin file to focus set (requires `path`, optional `pin_reason`, `symbol_or_line`, `repo_name`, `git_branch`).
   - `action: "unpin"`: Unpin file from focus set (requires `path`).
-  - `action: "list"`: List pinned files.
+  - `action: "list"`: List pinned files (optional `namespace`).
 
 * **`context_workspaces`**: Workspace context state snapshots.
-  - `action: "save"`: Save context workspace (requires `name`).
+  - `action: "save"`: Save context workspace (requires `name`, optional `pinned_files`, `active_task_ids`, `description`, `git_branch`, `vcs_revision`, `repo_name`).
   - `action: "load"`: Restore saved context workspace (requires `name`).
   - `action: "list"`: List saved context workspaces.
   - `action: "delete"`: Delete saved context workspace (requires `name`).
   - `action: "diff"`: Compare two saved context workspaces (requires `name`, `other_name`).
 
 * **`pr_checklist`**: Pre-commit and PR checklist.
-  - `action: "add"`: Add checklist item (requires `description`).
+  - `action: "add"`: Add checklist item (requires `description`, optional `category: "Security" | "Testing" | "Formatting" | ...`, `automated_check`, `is_checked`, `repo_name`).
   - `action: "get"`: Get PR checklist items.
   - `action: "clear"`: Clear PR checklist.
 
 * **`snippets`**: Reusable code snippet vault.
-  - `action: "store"`: Store snippet (requires `query` as name, optional `language`, `code`, `description`, `tags`).
+  - `action: "store"`: Store snippet (requires `query` as name, optional `language`, `code`, `description`, `tags`, `origin_file`, `line_range`, `repo_name`).
   - `action: "search"`: Search snippet vault (optional `query`, `tags`, `hybrid: true`).
   - `action: "delete"`: Delete snippet (requires `id`).
   - `action: "tag"`: Attach classification tags (requires `id`, `tags: Vec<String>`).
 
 * **`decisions`**: Architectural Decision Records (ADRs).
-  - `action: "log"`: Log ADR (requires `title`, optional `status`, `context`, `decision`, `consequences`).
+  - `action: "log"`: Log ADR (requires `title`, optional `status: "accepted" | "proposed" | "deprecated" | "superseded"`, `context`, `decision`, `consequence`, `author`, `affected_components: Vec<String>`, `alternatives_considered: Vec<String>`, `supersedes`, `repo_name`).
   - `action: "query"`: Query ADRs (optional `query`).
   - `action: "delete"`: Delete ADR (requires `id`).
 
 * **`tech_debt`**: Engineering debt backlog.
-  - `action: "log"`: Log debt item (requires `description`, optional `ideal_solution`, `git_commit`, `git_branch`, `symbol_references`, `line_range`).
+  - `action: "log"`: Log debt item (requires `description`, optional `ideal_solution`, `file_path`, `line_range`, `workaround`, `effort_estimate`, `severity: "low" | "medium" | "high" | "critical"`, `git_commit`, `git_branch`, `symbol_references`, `repo_name`).
   - `action: "resolve"`: Resolve debt item (requires `id`).
   - `action: "list"`: List debt items (optional `include_resolved`).
 
 * **`environment`**: Infrastructure and requirements tracking.
-  - `action: "update_fingerprint"`: Update tool versions.
+  - `action: "update_fingerprint"`: Update tool versions (requires `tool_versions: Map<String, String>`, optional `repo_name`).
   - `action: "read_fingerprint"`: Read tool versions fingerprint.
-  - `action: "log_requirement"`: Log environment variable requirement (requires `key`).
-  - `action: "register"`: Register target environment (requires `name`).
+  - `action: "log_requirement"`: Log environment variable requirement (requires `key`, `description`, `is_secret`, optional `default_value`, `validation_regex`, `repo_name`).
+  - `action: "register"`: Register target environment (requires `name`, `url`, optional `description`, `requires_vpn`, `env_type: "dev" | "staging" | "qa" | "prod"`, `healthcheck_endpoint`, `ssh_host`, `repo_name`).
   - `action: "get_details"`: Read full environment details.
 
 * **`clipboard`**: OS Clipboard management.
@@ -94,6 +99,26 @@ The server consolidates granular single-purpose tools into domain-named smart to
 
 ---
 
+## 3. VCS & SVN Agnosticism & Multi-Repo Provenance
+
+To support diverse enterprise repositories (Git, Subversion / SVN, Mercurial / Hg, Monorepos):
+* **`vcs_type`**: Designates the VCS engine (`"git"`, `"svn"`, `"hg"`, `"perforce"`, or `"none"`).
+* **`vcs_revision`**: Agnostic commit hash or SVN revision identifier (e.g., `"r12458"`, `"3e4f7a9"`).
+* **`upstream_url`**: Canonical remote repository URL (e.g. `https://svn.corp/repo/trunk`, `git@bitbucket.org:org/repo.git`).
+* **`repo_name`**: Logical project or repository identifier allowing multiple repositories to share or partition memory namespaces cleanly without collision.
+* **Audit Ledger (`log_code_change`)**: Enriched with `vcs_type`, `vcs_revision`, `upstream_url`, `author`, `diff_summary`, and extensible `metadata: HashMap<String, String>`.
+
+---
+
+## 4. Terminal & Process Telemetry
+
+The server ingests and tracks active terminal commands and sessions:
+* **Active Terminals**: Tracks PIDs, shell interpreters (`pwsh`, `bash`, `nu`, `zsh`), current working directories (`cwd`), command exit codes, and timestamps.
+* **Terminal History Endpoint**: `/terminal/history` exposes recent shell commands and output streams to dashboard and LLMs to prevent lost shell context.
+* **Quality Gate Enforcement**: `GateRecord` captures pre-flight and pre-push validation passes with `gate_type`, `enforcer`, `status`, `validation_log`, and `repo_name`.
+
+---
+
 ## 4. High-Signal Tool Responses & Performance Guidelines
 
 To optimize context usage, response times, and LLM reasoning efficiency:
@@ -157,8 +182,16 @@ To maintain maximum security, speed, and cross-platform reliability:
 ---
 
 ## 10. High-Performance Concurrency & Resilience Guarantees
+* **Explicit Fail-Fast Persistence Safety**: Replaced silent fallback to temporary databases (`/tmp/mcp_store_fallback_*`) with an explicit open retry and fail-fast panic unless `MCP_ALLOW_TMP_FALLBACK=1` is explicitly set, preventing silent data loss.
+* **Async Mutex Deadlock Elimination**: Converted shared state and Neovim connection locks (`shutdown_tx`, `NVIM_CONN`, `ACTIVE_SOCKET`, `HEADLESS_PROC`) to `tokio::sync::Mutex` to prevent worker thread pool starvation across `.await` points.
+* **Telemetry Session Deduplication & Channel Pruning**: Added `LAST_SESSION` in-memory state deduplication for UDP telemetry writes (eliminating disk I/O thrashing) and distinguished WebSocket `TrySendError::Full` backpressure vs `TrySendError::Closed` client pruning.
+* **Graph Adjacency Indexing**: Leverages `KnowledgeGraph::build_adjacency_map` to build $O(1)$ lookup adjacency lists for fast BFS shortest path graph queries.
+* **Atomic Store Write Lock Minimization**: `Store::modify` and `Store::modify_async` release write guards immediately after in-memory state mutations, serializing JSON payloads under read locks to allow non-blocking concurrent readers.
+* **Async Commit Index Reader Auto-Reload**: `MemoryIndex::commit()` automatically triggers `reader.reload()` upon completing background commits, guaranteeing immediate visibility of newly indexed document terms.
 * **Zero-Allocation NDJSON Reader**: `mcp-stdio` reclaims line buffers via `std::mem::take` and in-place trimming to eliminate heap allocations during high-frequency NDJSON message parsing.
-* **Micro-Batched Fastembed Inference**: `generate_embeddings_async` chunks text arrays into 32-item micro-batches inside `spawn_blocking`, eliminating RAM/CPU spikes during batch indexing.
+* **Dynamic Character Micro-Batched Fastembed Inference**: `generate_embeddings_async` dynamically batches text arrays using a 16,000 character budget ceiling inside `spawn_blocking`, preventing heap spikes during vector indexing while keeping ONNX SIMD execution saturated.
+* **Bounded Telemetry Detail Records**: Activity and terminal telemetry buffers enforce a 4,000 character truncation ceiling on log details (`ActivityRecord`, `TerminalHistory`) to prevent unbounded RAM growth under high RPC throughput.
+* **Zero-Allocation Stream Formatting**: Graph condensation loops (`condense_graph_worker`) format node/relation subgraphs into stream buffers via `std::fmt::Write`, eliminating intermediate String allocation overhead.
 * **Single-Handle OS Clipboard Retries & Image Downsampling**: `WriteClipboardHandler` initializes OS clipboard handles once per operation and downsamples images exceeding $2048 \times 2048$ resolution before writing raw RGBA bytes.
 * **Zero Transaction Drop Persistence Guarantee**: `Store::modify` automatically spawns an async task to execute `push_async` with channel backpressure if `push` encounters queue saturation, ensuring zero data loss under spike write loads.
 * **Non-Blocking Read Lock Sticky Notes**: `StickyNotesHandler` (`action = "read"`) queries notes using shared read locks, executing write pruning only when expired items exist.
@@ -183,5 +216,16 @@ To maintain maximum security, speed, and cross-platform reliability:
 * **Directory Tree Depth Safeguard**: `ReadDirectoryArchitectureHandler` caps directory recursion at depth 10 to prevent stack overflow on deep or cyclic directory structures.
 * **Deterministic Total-Order Score Ranking**: `OmniSearchHandler` uses `f64::total_cmp` for Reciprocal Rank Fusion (RRF) score sorting, guaranteeing deterministic NaN-safe search result ordering.
 * **RPC Timeout Memory Hygiene**: `nvim-core` maintains request hygiene by removing pending request entries from static RPC maps upon timeout or channel drop, eliminating orphan memory leaks.
+* **Embedding Input Safeguard**: `generate_embedding_async` returns explicit errors for empty/0-length text inputs instead of returning empty vectors, preventing downstream vector dimension mismatches during cosine similarity calculations.
+* **Path Traversal Security Guards**: `validate_safe_path` enforces path canonicalization and rejects relative parent traversal components (`..`) across file and process log handlers (`GetRecentLogsTool`, `WatchProcessLogsTool`).
+* **Watcher Map Memory Eviction**: Proactive daemon file watcher in `watcher.rs` caps `last_processed` map size at 1,000 entries and purges entries older than 10 minutes to prevent monotonic memory leakage.
+* **Comprehensive Serde Casing Aliases**: All 11 consolidated tool action enums (`TaskAction`, `MilestoneAction`, `PinnedFileAction`, `ContextWorkspaceAction`, `PrChecklistAction`, `SnippetAction`, `DecisionAction`, `TechDebtAction`, `EnvAction`, `ClipboardAction`, `HandoffMemoAction`) include serde alias attributes supporting `snake_case`, `camelCase`, `PascalCase`, and uppercase variants for maximum LLM casing resilience.
+* **Two-Phase Graph Condensation**: `condense_graph_worker` uses a 2-phase commit (non-destructive `read_with` -> graph insert -> prune by timestamp/content) to prevent data loss if summarization or graph insertion fails.
+* **Store Write Lock Minimization**: `Store::modify` and `Store::modify_async` unblock concurrent readers during JSON serialization by releasing the write lock immediately after mutating memory state.
+* **Redb Database Lock Retry Backoff**: `init_db` retries transient Redb lock contention with exponential backoff (3 attempts, 150ms delay) before falling back.
+* **Offloaded Background Index Rebuilds**: `MemoryState::rebuild_index` offloads graph snapshot cloning and Tantivy document re-indexing into `tokio::task::spawn_blocking` to avoid stalling async event loops.
+* **Broadcast Watch-Based Shutdown Channels**: Background workers utilize `tokio::sync::watch` for broadcast shutdown notifications without consuming cancellation signals.
+* **Consolidated Neovim Tool Suite (v2)**: The Neovim server exposes 7 consolidated domain tools (`nvim_buffer`, `nvim_window`, `nvim_view`, `nvim_diagnostics`, `nvim_visual`, `nvim_execute_lua`, `nvim_system`) with comprehensive action dispatching.
+* **Fallback Vector Search Parity**: In-memory vector search fallback indexes Knowledge Graph entities, observations, and error fixes when external vector databases are unavailable.