feat(memory): système de mémoire projet model-agnostic (L14, LOT A+B+C)

Base de connaissance persistante par projet, indépendante de tout modèle/CLU
et de git. Cadrage archi en §14.5 (ARCHITECTURE.md), cycle Archi→Dev→Test.

LOT A — étage 1 (.md, source de vérité)
  - domaine: entité Memory (+ MemorySlug, MemoryType, MemoryFrontmatter,
    MemoryLink, MemoryIndexEntry), liens [[slug]], index MEMORY.md dérivé
  - port MemoryStore + MemoryError, adapter FsMemoryStore (.ideai/memory/)
  - application: 7 use cases (Create/Update/List/Get/Delete/ReadIndex/
    ResolveLinks), From<MemoryError> for AppError
  - app-tauri: commandes + DTO, events MemorySaved/MemoryDeleted
  - suppression de la variante morte DomainError::MalformedFrontmatter

LOT B — rappel adaptatif (étage 1)
  - port MemoryRecall + MemoryQuery, adapter NaiveMemoryRecall (troncature
    au budget de tokens, court-circuit budget-0), use case RecallMemory

LOT C — étage 2 vectoriel (structure complète, zéro dépendance lourde)
  - port Embedder + EmbedderError, profils déclaratifs EmbedderProfile/
    EmbedderStrategy (embedder.json)
  - VectorMemoryRecall (cosinus, cache .ideai/memory/.index/ gitignoré)
  - AdaptiveMemoryRecall (bascule pure should_use_vector), défaut none
  - HashEmbedder (déterministe, tests), StubEmbedder (onnx/server/api)

Tests: 57 binaires verts, build + clippy --workspace sans warning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-08 08:47:23 +02:00
parent 3ed0f6b45f
commit 98a8b7292a
30 changed files with 4978 additions and 14 deletions

View File

@ -0,0 +1,277 @@
//! Tests for [`NaiveMemoryRecall`] (LOT B, étage 1): the default, dependency-free
//! [`MemoryRecall`] that reads the aggregated index via a composed [`MemoryStore`]
//! and truncates the entries to fit the query's token budget.
//!
//! Budget semantics under test (mirroring the adapter doc): each entry costs
//! `ceil((title.len() + hook.len()) / 4)` tokens; entries are taken in index order,
//! accumulating cost; the first entry that would exceed the budget — and every one
//! after it — is dropped. `token_budget == 0` ⇒ empty; an empty/absent memory ⇒
//! empty, never an error.
//!
//! Two backings are exercised:
//! - the **real** [`FsMemoryStore`] over an in-memory [`FileSystem`] mock (same
//! `MemFs` shape as `memory_store.rs`), giving end-to-end coverage of the
//! index round-trip feeding recall;
//! - a tiny **counting fake** `MemoryStore` to assert the `budget == 0`
//! short-circuit performs *no* `read_index` call.
use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use domain::markdown::MarkdownDoc;
use domain::memory::{Memory, MemoryFrontmatter, MemoryIndexEntry, MemoryLink, MemorySlug, MemoryType};
use domain::ports::{
DirEntry, FileSystem, FsError, MemoryError, MemoryQuery, MemoryRecall, MemoryStore, RemotePath,
};
use domain::project::ProjectPath;
use infrastructure::{FsMemoryStore, NaiveMemoryRecall};
// ---------------------------------------------------------------------------
// In-memory FileSystem mock (same minimal shape as `memory_store.rs`).
// ---------------------------------------------------------------------------
#[derive(Default)]
struct MemFs {
files: Mutex<HashMap<String, Vec<u8>>>,
}
impl MemFs {
fn arc() -> Arc<Self> {
Arc::new(Self::default())
}
}
#[async_trait]
impl FileSystem for MemFs {
async fn read(&self, path: &RemotePath) -> Result<Vec<u8>, FsError> {
self.files
.lock()
.unwrap()
.get(path.as_str())
.cloned()
.ok_or_else(|| FsError::NotFound(path.as_str().to_string()))
}
async fn write(&self, path: &RemotePath, data: &[u8]) -> Result<(), FsError> {
self.files
.lock()
.unwrap()
.insert(path.as_str().to_string(), data.to_vec());
Ok(())
}
async fn exists(&self, path: &RemotePath) -> Result<bool, FsError> {
Ok(self.files.lock().unwrap().contains_key(path.as_str()))
}
async fn create_dir_all(&self, _path: &RemotePath) -> Result<(), FsError> {
Ok(())
}
async fn list(&self, _path: &RemotePath) -> Result<Vec<DirEntry>, FsError> {
Ok(Vec::new())
}
async fn symlink(&self, _src: &RemotePath, _dst: &RemotePath) -> Result<(), FsError> {
Ok(())
}
}
// ---------------------------------------------------------------------------
// Builders
// ---------------------------------------------------------------------------
fn root() -> ProjectPath {
ProjectPath::new("/proj").unwrap()
}
/// A note whose index `title` is its slug (length `slug.len()`) and whose `hook`
/// is `desc` — so the recall cost is `ceil((slug.len() + desc.len()) / 4)`.
fn note(slug: &str, desc: &str) -> Memory {
Memory::new(
MemoryFrontmatter {
name: MemorySlug::new(slug).unwrap(),
description: desc.to_string(),
r#type: MemoryType::Project,
},
MarkdownDoc::new("# body"),
)
.unwrap()
}
/// Saves notes (in order) through the real store and returns a recall over it.
async fn recall_over(notes: &[Memory]) -> NaiveMemoryRecall {
let store = Arc::new(FsMemoryStore::new(MemFs::arc()));
for n in notes {
store.save(&root(), n).await.unwrap();
}
NaiveMemoryRecall::new(store)
}
fn query(budget: usize) -> MemoryQuery {
MemoryQuery {
text: "anything".to_string(),
token_budget: budget,
}
}
fn slugs(entries: &[MemoryIndexEntry]) -> Vec<String> {
entries.iter().map(|e| e.slug.as_str().to_string()).collect()
}
// ---------------------------------------------------------------------------
// Empty / absent memory ⇒ empty list, never an error.
// ---------------------------------------------------------------------------
#[tokio::test]
async fn absent_index_recalls_empty_without_error() {
let recall = NaiveMemoryRecall::new(Arc::new(FsMemoryStore::new(MemFs::arc())));
let out = recall.recall(&root(), &query(1000)).await.unwrap();
assert!(out.is_empty(), "absent memory must recall empty: {out:?}");
}
// ---------------------------------------------------------------------------
// Budget 0 ⇒ empty list (and short-circuits before touching the store).
// ---------------------------------------------------------------------------
#[tokio::test]
async fn zero_budget_recalls_empty() {
let recall = recall_over(&[note("aaaa", "bbbb"), note("cccc", "dddd")]).await;
let out = recall.recall(&root(), &query(0)).await.unwrap();
assert!(out.is_empty(), "budget 0 must recall empty: {out:?}");
}
/// A `MemoryStore` that counts `read_index` calls and panics on every other
/// method — proving the recall path only reads the index, and that a `0` budget
/// short-circuits before any read.
#[derive(Default)]
struct CountingStore {
read_index_calls: AtomicUsize,
}
#[async_trait]
impl MemoryStore for CountingStore {
async fn list(&self, _root: &ProjectPath) -> Result<Vec<Memory>, MemoryError> {
panic!("recall must not call list")
}
async fn get(&self, _root: &ProjectPath, _slug: &MemorySlug) -> Result<Memory, MemoryError> {
panic!("recall must not call get")
}
async fn save(&self, _root: &ProjectPath, _memory: &Memory) -> Result<(), MemoryError> {
panic!("recall must not call save")
}
async fn delete(&self, _root: &ProjectPath, _slug: &MemorySlug) -> Result<(), MemoryError> {
panic!("recall must not call delete")
}
async fn read_index(
&self,
_root: &ProjectPath,
) -> Result<Vec<MemoryIndexEntry>, MemoryError> {
self.read_index_calls.fetch_add(1, Ordering::SeqCst);
Ok(Vec::new())
}
async fn resolve_links(
&self,
_root: &ProjectPath,
_slug: &MemorySlug,
) -> Result<Vec<MemoryLink>, MemoryError> {
panic!("recall must not call resolve_links")
}
}
#[tokio::test]
async fn zero_budget_short_circuits_no_read_index() {
let store = Arc::new(CountingStore::default());
let recall = NaiveMemoryRecall::new(store.clone());
let out = recall.recall(&root(), &query(0)).await.unwrap();
assert!(out.is_empty());
assert_eq!(
store.read_index_calls.load(Ordering::SeqCst),
0,
"budget 0 must short-circuit before reading the index"
);
}
#[tokio::test]
async fn nonzero_budget_reads_index_once() {
let store = Arc::new(CountingStore::default());
let recall = NaiveMemoryRecall::new(store.clone());
let _ = recall.recall(&root(), &query(10)).await.unwrap();
assert_eq!(store.read_index_calls.load(Ordering::SeqCst), 1);
}
// ---------------------------------------------------------------------------
// Sufficient budget ⇒ all entries, in index order.
// ---------------------------------------------------------------------------
#[tokio::test]
async fn ample_budget_returns_all_entries_in_index_order() {
// Saved order: alpha, beta, gamma → index order is the same.
let recall = recall_over(&[
note("alpha", "first hook"),
note("beta", "second hook"),
note("gamma", "third hook"),
])
.await;
let out = recall.recall(&root(), &query(100_000)).await.unwrap();
assert_eq!(slugs(&out), vec!["alpha", "beta", "gamma"]);
}
// ---------------------------------------------------------------------------
// Intermediate budget ⇒ exact truncation at the cumulative-cost boundary.
//
// Three notes, each title.len()==4 and hook.len()==4 ⇒ 8 chars ⇒ ceil(8/4)==2
// tokens apiece. Cumulative costs: 2, 4, 6.
// ---------------------------------------------------------------------------
#[tokio::test]
async fn intermediate_budget_truncates_at_exact_boundary() {
let notes = [note("aaaa", "bbbb"), note("cccc", "dddd"), note("eeee", "ffff")];
let recall = recall_over(&notes).await;
// Budget 1 < 2 ⇒ first entry already exceeds ⇒ none.
assert!(recall.recall(&root(), &query(1)).await.unwrap().is_empty());
// Budget 2 == cost(1) ⇒ exactly the first entry; second (would reach 4) dropped.
assert_eq!(slugs(&recall.recall(&root(), &query(2)).await.unwrap()), vec!["aaaa"]);
// Budget 3 < 4 ⇒ still only the first.
assert_eq!(slugs(&recall.recall(&root(), &query(3)).await.unwrap()), vec!["aaaa"]);
// Budget 4 == cost(1)+cost(2) ⇒ first two; third (would reach 6) dropped.
assert_eq!(
slugs(&recall.recall(&root(), &query(4)).await.unwrap()),
vec!["aaaa", "cccc"]
);
// Budget 5 < 6 ⇒ still the first two.
assert_eq!(
slugs(&recall.recall(&root(), &query(5)).await.unwrap()),
vec!["aaaa", "cccc"]
);
// Budget 6 == total ⇒ all three.
assert_eq!(
slugs(&recall.recall(&root(), &query(6)).await.unwrap()),
vec!["aaaa", "cccc", "eeee"]
);
}
// ---------------------------------------------------------------------------
// ceil rounding: an odd character count rounds *up* (5 chars ⇒ 2 tokens).
// ---------------------------------------------------------------------------
#[tokio::test]
async fn entry_cost_rounds_up_via_ceil() {
// slug "ab" (2) + hook "c" (1) = 3 chars ⇒ ceil(3/4) == 1 token.
// slug "de" (2) + hook "fgh" (3) = 5 chars ⇒ ceil(5/4) == 2 tokens.
let recall = recall_over(&[note("ab", "c"), note("de", "fgh")]).await;
// Budget 1 covers only the 1-token first entry; the 2-token second is dropped.
assert_eq!(slugs(&recall.recall(&root(), &query(1)).await.unwrap()), vec!["ab"]);
// Budget 2 still cannot fit the second on top of the first (1 + 2 = 3 > 2).
assert_eq!(slugs(&recall.recall(&root(), &query(2)).await.unwrap()), vec!["ab"]);
// Budget 3 fits both (1 + 2 == 3).
assert_eq!(
slugs(&recall.recall(&root(), &query(3)).await.unwrap()),
vec!["ab", "de"]
);
}

View File

@ -0,0 +1,278 @@
//! Tests for [`FsMemoryStore`] against an **in-memory mock** [`FileSystem`]:
//! `save` writes the `<slug>.md` AND the `MEMORY.md` index line; get/list/delete;
//! upsert idempotence (same slug => one index line); malformed frontmatter
//! surfaces as [`MemoryError::Frontmatter`]; and `resolve_links` ignores broken
//! links.
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use domain::markdown::MarkdownDoc;
use domain::memory::{Memory, MemoryFrontmatter, MemorySlug, MemoryType};
use domain::ports::{DirEntry, FileSystem, FsError, MemoryError, MemoryStore, RemotePath};
use domain::project::ProjectPath;
use infrastructure::FsMemoryStore;
/// A minimal in-memory [`FileSystem`] mock: a path -> bytes map behind a mutex.
/// `create_dir_all`/`symlink`/`list` are no-ops or derived; only read/write/exists
/// matter for the memory store.
#[derive(Default)]
struct MemFs {
files: Mutex<HashMap<String, Vec<u8>>>,
}
impl MemFs {
fn arc() -> Arc<Self> {
Arc::new(Self::default())
}
fn raw(&self, path: &str) -> Option<String> {
self.files
.lock()
.unwrap()
.get(path)
.map(|b| String::from_utf8(b.clone()).unwrap())
}
/// Directly seeds a file (to inject a malformed note bypassing the store).
fn put(&self, path: &str, content: &str) {
self.files
.lock()
.unwrap()
.insert(path.to_string(), content.as_bytes().to_vec());
}
}
#[async_trait]
impl FileSystem for MemFs {
async fn read(&self, path: &RemotePath) -> Result<Vec<u8>, FsError> {
self.files
.lock()
.unwrap()
.get(path.as_str())
.cloned()
.ok_or_else(|| FsError::NotFound(path.as_str().to_string()))
}
async fn write(&self, path: &RemotePath, data: &[u8]) -> Result<(), FsError> {
self.files
.lock()
.unwrap()
.insert(path.as_str().to_string(), data.to_vec());
Ok(())
}
async fn exists(&self, path: &RemotePath) -> Result<bool, FsError> {
Ok(self.files.lock().unwrap().contains_key(path.as_str()))
}
async fn create_dir_all(&self, _path: &RemotePath) -> Result<(), FsError> {
Ok(())
}
async fn list(&self, _path: &RemotePath) -> Result<Vec<DirEntry>, FsError> {
Ok(Vec::new())
}
async fn symlink(&self, _src: &RemotePath, _dst: &RemotePath) -> Result<(), FsError> {
Ok(())
}
}
fn root() -> ProjectPath {
ProjectPath::new("/proj").unwrap()
}
fn slug(s: &str) -> MemorySlug {
MemorySlug::new(s).unwrap()
}
fn note(slug_str: &str, hook: &str, ty: MemoryType, body: &str) -> Memory {
Memory::new(
MemoryFrontmatter {
name: MemorySlug::new(slug_str).unwrap(),
description: hook.to_string(),
r#type: ty,
},
MarkdownDoc::new(body),
)
.unwrap()
}
#[tokio::test]
async fn missing_index_lists_empty() {
let store = FsMemoryStore::new(MemFs::arc());
assert!(store.list(&root()).await.unwrap().is_empty());
assert!(store.read_index(&root()).await.unwrap().is_empty());
}
#[tokio::test]
async fn save_writes_md_and_index_line() {
let fs = MemFs::arc();
let store = FsMemoryStore::new(fs.clone());
store
.save(&root(), &note("alpha", "the hook", MemoryType::Project, "# Body"))
.await
.unwrap();
// The note `.md` exists with frontmatter + body.
let md = fs.raw("/proj/.ideai/memory/alpha.md").expect("note written");
assert!(md.starts_with("---\n"));
assert!(md.contains("name: alpha"));
assert!(md.contains("description: the hook"));
assert!(md.contains("type: project"));
assert!(md.contains("# Body"));
// The index has the header + the line.
let index = fs.raw("/proj/.ideai/memory/MEMORY.md").expect("index written");
assert!(index.starts_with("# Memory Index"));
assert!(index.contains("- [alpha](alpha.md) — the hook"));
}
#[tokio::test]
async fn get_roundtrips_the_saved_note() {
let store = FsMemoryStore::new(MemFs::arc());
let m = note("topic-x", "hook", MemoryType::Feedback, "content [[other]]");
store.save(&root(), &m).await.unwrap();
let got = store.get(&root(), &slug("topic-x")).await.unwrap();
assert_eq!(got, m);
}
#[tokio::test]
async fn get_missing_is_not_found() {
let store = FsMemoryStore::new(MemFs::arc());
assert!(matches!(
store.get(&root(), &slug("nope")).await.unwrap_err(),
MemoryError::NotFound
));
}
#[tokio::test]
async fn list_returns_all_saved_notes() {
let store = FsMemoryStore::new(MemFs::arc());
store
.save(&root(), &note("a", "ha", MemoryType::User, "A"))
.await
.unwrap();
store
.save(&root(), &note("b", "hb", MemoryType::Project, "B"))
.await
.unwrap();
let mut slugs: Vec<_> = store
.list(&root())
.await
.unwrap()
.into_iter()
.map(|m| m.slug().as_str().to_string())
.collect();
slugs.sort();
assert_eq!(slugs, vec!["a".to_string(), "b".to_string()]);
}
#[tokio::test]
async fn save_is_idempotent_upsert_single_index_line() {
let fs = MemFs::arc();
let store = FsMemoryStore::new(fs.clone());
store
.save(&root(), &note("dup", "hook v1", MemoryType::Project, "v1"))
.await
.unwrap();
store
.save(&root(), &note("dup", "hook v2", MemoryType::Project, "v2"))
.await
.unwrap();
// Content was replaced.
assert_eq!(
store
.get(&root(), &slug("dup"))
.await
.unwrap()
.body
.as_str(),
"v2"
);
// Exactly one index entry for the slug.
let entries = store.read_index(&root()).await.unwrap();
assert_eq!(entries.len(), 1);
assert_eq!(entries[0].hook, "hook v2");
// And the raw index has only one matching line.
let index = fs.raw("/proj/.ideai/memory/MEMORY.md").unwrap();
assert_eq!(index.matches("- [dup]").count(), 1);
}
#[tokio::test]
async fn delete_removes_index_line_and_is_not_found_twice() {
let store = FsMemoryStore::new(MemFs::arc());
store
.save(&root(), &note("gone", "h", MemoryType::User, "x"))
.await
.unwrap();
store.delete(&root(), &slug("gone")).await.unwrap();
assert!(store.list(&root()).await.unwrap().is_empty());
assert!(matches!(
store.delete(&root(), &slug("gone")).await.unwrap_err(),
MemoryError::NotFound
));
}
#[tokio::test]
async fn malformed_frontmatter_surfaces_as_error() {
let fs = MemFs::arc();
// Seed the index so the slug is listed, plus a malformed note file.
fs.put(
"/proj/.ideai/memory/MEMORY.md",
"# Memory Index\n\n- [bad](bad.md) — h\n",
);
fs.put(
"/proj/.ideai/memory/bad.md",
"no frontmatter fence here\njust body",
);
let store = FsMemoryStore::new(fs);
assert!(matches!(
store.get(&root(), &slug("bad")).await.unwrap_err(),
MemoryError::Frontmatter(_)
));
// list propagates the same error.
assert!(matches!(
store.list(&root()).await.unwrap_err(),
MemoryError::Frontmatter(_)
));
}
#[tokio::test]
async fn resolve_links_ignores_broken_links() {
let store = FsMemoryStore::new(MemFs::arc());
// `target` exists; `ghost` does not.
store
.save(&root(), &note("target", "h", MemoryType::Project, "T"))
.await
.unwrap();
store
.save(
&root(),
&note(
"src",
"h",
MemoryType::Project,
"links to [[target]] and [[ghost]]",
),
)
.await
.unwrap();
let resolved = store.resolve_links(&root(), &slug("src")).await.unwrap();
let targets: Vec<_> = resolved
.into_iter()
.map(|l| l.target.as_str().to_string())
.collect();
assert_eq!(targets, vec!["target".to_string()]);
}
#[tokio::test]
async fn resolve_links_missing_source_is_not_found() {
let store = FsMemoryStore::new(MemFs::arc());
assert!(matches!(
store.resolve_links(&root(), &slug("nope")).await.unwrap_err(),
MemoryError::NotFound
));
}

View File

@ -0,0 +1,626 @@
//! Tests for the étage-2 vector recall (LOT C, §14.5.3):
//! - [`should_use_vector`] — the pure routing decision (full truth table);
//! - [`AdaptiveMemoryRecall`] — the étage-1/étage-2 switch, with naïve equivalence
//! on strategy `none`, routing by size/budget, and the never-fail-hard fallback;
//! - [`VectorMemoryRecall`] — semantic ranking with the deterministic
//! [`HashEmbedder`], budget truncation, the derived `vectors.json` cache and its
//! embedder-id/dimension invalidation, and `StubEmbedder` best-effort behaviour;
//! - [`HashEmbedder`] / [`StubEmbedder`] embedder contracts;
//! - [`embedder_from_profile`] strategy → embedder mapping.
//!
//! Everything is in-memory and deterministic (`MemFs` + `HashEmbedder`); no real
//! I/O, no network.
use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use domain::markdown::MarkdownDoc;
use domain::memory::{Memory, MemoryFrontmatter, MemoryIndexEntry, MemorySlug, MemoryType};
use domain::ports::{
DirEntry, Embedder, EmbedderError, FileSystem, FsError, MemoryQuery, MemoryRecall, MemoryStore,
RemotePath,
};
use domain::profile::{EmbedderProfile, EmbedderStrategy};
use domain::project::ProjectPath;
use infrastructure::{
embedder_from_profile, should_use_vector, AdaptiveMemoryRecall, FsMemoryStore, HashEmbedder,
NaiveMemoryRecall, StubEmbedder, VectorMemoryRecall,
};
// ---------------------------------------------------------------------------
// In-memory FileSystem mock (same shape as `memory_recall.rs` / `memory_store.rs`),
// with raw read access to assert on the derived `vectors.json`.
// ---------------------------------------------------------------------------
#[derive(Default)]
struct MemFs {
files: Mutex<HashMap<String, Vec<u8>>>,
}
impl MemFs {
fn arc() -> Arc<Self> {
Arc::new(Self::default())
}
fn raw(&self, path: &str) -> Option<String> {
self.files
.lock()
.unwrap()
.get(path)
.map(|b| String::from_utf8(b.clone()).unwrap())
}
}
#[async_trait]
impl FileSystem for MemFs {
async fn read(&self, path: &RemotePath) -> Result<Vec<u8>, FsError> {
self.files
.lock()
.unwrap()
.get(path.as_str())
.cloned()
.ok_or_else(|| FsError::NotFound(path.as_str().to_string()))
}
async fn write(&self, path: &RemotePath, data: &[u8]) -> Result<(), FsError> {
self.files
.lock()
.unwrap()
.insert(path.as_str().to_string(), data.to_vec());
Ok(())
}
async fn exists(&self, path: &RemotePath) -> Result<bool, FsError> {
Ok(self.files.lock().unwrap().contains_key(path.as_str()))
}
async fn create_dir_all(&self, _path: &RemotePath) -> Result<(), FsError> {
Ok(())
}
async fn list(&self, _path: &RemotePath) -> Result<Vec<DirEntry>, FsError> {
Ok(Vec::new())
}
async fn symlink(&self, _src: &RemotePath, _dst: &RemotePath) -> Result<(), FsError> {
Ok(())
}
}
const VECTORS_PATH: &str = "/proj/.ideai/memory/.index/vectors.json";
// ---------------------------------------------------------------------------
// A counting wrapper embedder: delegates to an inner embedder and counts the
// number of *texts* it was asked to embed — to assert the derived cache avoids
// recomputing note vectors on a second recall.
// ---------------------------------------------------------------------------
struct SpyEmbedder {
inner: HashEmbedder,
embedded_texts: AtomicUsize,
}
impl SpyEmbedder {
fn new(id: &str, dim: usize) -> Self {
Self {
inner: HashEmbedder::new(id, dim),
embedded_texts: AtomicUsize::new(0),
}
}
}
#[async_trait]
impl Embedder for SpyEmbedder {
fn id(&self) -> &str {
self.inner.id()
}
async fn embed(&self, texts: &[String]) -> Result<Vec<Vec<f32>>, EmbedderError> {
self.embedded_texts.fetch_add(texts.len(), Ordering::SeqCst);
self.inner.embed(texts).await
}
fn dimension(&self) -> usize {
self.inner.dimension()
}
}
// ---------------------------------------------------------------------------
// Builders.
// ---------------------------------------------------------------------------
fn root() -> ProjectPath {
ProjectPath::new("/proj").unwrap()
}
/// A note whose index line is `<slug> <hook>` (title == slug). The vector recall
/// embeds exactly `"{slug} {hook}"`, so picking distinct vocabulary words in
/// `slug`/`hook` makes cosine similarity to a chosen query deterministic.
fn note(slug: &str, hook: &str) -> Memory {
Memory::new(
MemoryFrontmatter {
name: MemorySlug::new(slug).unwrap(),
description: hook.to_string(),
r#type: MemoryType::Project,
},
MarkdownDoc::new("# body"),
)
.unwrap()
}
fn query(text: &str, budget: usize) -> MemoryQuery {
MemoryQuery {
text: text.to_string(),
token_budget: budget,
}
}
fn slugs(entries: &[MemoryIndexEntry]) -> Vec<String> {
entries.iter().map(|e| e.slug.as_str().to_string()).collect()
}
async fn store_with(notes: &[Memory], fs: Arc<MemFs>) -> Arc<FsMemoryStore> {
let store = Arc::new(FsMemoryStore::new(fs));
for n in notes {
store.save(&root(), n).await.unwrap();
}
store
}
// ===========================================================================
// 1. should_use_vector — pure truth table.
// ===========================================================================
#[test]
fn should_use_vector_none_strategy_is_always_false() {
// Even when size > budget, a `none` strategy never routes to the vector path.
assert!(!should_use_vector(0, 0, EmbedderStrategy::None));
assert!(!should_use_vector(100, 10, EmbedderStrategy::None));
assert!(!should_use_vector(10, 100, EmbedderStrategy::None));
assert!(!should_use_vector(usize::MAX, 0, EmbedderStrategy::None));
}
#[test]
fn should_use_vector_non_none_iff_size_exceeds_budget() {
for strategy in [
EmbedderStrategy::LocalOnnx,
EmbedderStrategy::LocalServer,
EmbedderStrategy::Api,
] {
// size > budget ⇒ true.
assert!(should_use_vector(11, 10, strategy), "{strategy:?}: 11>10");
assert!(should_use_vector(1, 0, strategy), "{strategy:?}: 1>0");
// size == budget ⇒ false (exact boundary).
assert!(!should_use_vector(10, 10, strategy), "{strategy:?}: 10==10");
assert!(!should_use_vector(0, 0, strategy), "{strategy:?}: 0==0");
// size < budget ⇒ false.
assert!(!should_use_vector(9, 10, strategy), "{strategy:?}: 9<10");
}
}
// ===========================================================================
// 2. HashEmbedder — determinism, dimension, L2 normalisation.
// ===========================================================================
#[tokio::test]
async fn hash_embedder_is_deterministic() {
let e = HashEmbedder::new("h", 64);
let a = e.embed(&["hello world".to_string()]).await.unwrap();
let b = e.embed(&["hello world".to_string()]).await.unwrap();
assert_eq!(a, b, "same text must give the same vector");
}
#[tokio::test]
async fn hash_embedder_dimension_is_consistent() {
let e = HashEmbedder::new("h", 48);
assert_eq!(e.dimension(), 48);
let out = e
.embed(&["alpha beta".to_string(), "gamma".to_string()])
.await
.unwrap();
assert_eq!(out.len(), 2, "one vector per input, order preserved");
assert!(out.iter().all(|v| v.len() == 48), "every vector is dimension-long");
}
#[tokio::test]
async fn hash_embedder_vectors_are_l2_normalised() {
let e = HashEmbedder::new("h", 64);
let out = e
.embed(&["one two three four".to_string()])
.await
.unwrap();
let norm = out[0].iter().map(|x| x * x).sum::<f32>().sqrt();
assert!((norm - 1.0).abs() < 1e-5, "norm ≈ 1, got {norm}");
}
#[tokio::test]
async fn hash_embedder_empty_text_yields_zero_vector_no_panic() {
// No tokens ⇒ zero vector (norm 0, left unnormalised). Must not panic / NaN.
let e = HashEmbedder::new("h", 16);
let out = e.embed(&["".to_string()]).await.unwrap();
assert_eq!(out[0].len(), 16);
assert!(out[0].iter().all(|x| *x == 0.0));
}
// ===========================================================================
// 3. StubEmbedder — always Unsupported, never panics.
// ===========================================================================
#[tokio::test]
async fn stub_embedder_returns_unsupported() {
let e = StubEmbedder::new("stub", 32, "localOnnx");
assert_eq!(e.dimension(), 32);
let err = e.embed(&["x".to_string()]).await.unwrap_err();
assert!(matches!(err, EmbedderError::Unsupported(_)), "got {err:?}");
}
// ===========================================================================
// 4. embedder_from_profile — strategy → embedder mapping.
// ===========================================================================
#[tokio::test]
async fn embedder_from_profile_maps_each_strategy() {
// none ⇒ no embedder (recall stays naïve).
assert!(embedder_from_profile(&EmbedderProfile::none()).is_none());
// localOnnx / localServer / api ⇒ a StubEmbedder (Unsupported), dimension kept.
for strategy in [
EmbedderStrategy::LocalOnnx,
EmbedderStrategy::LocalServer,
EmbedderStrategy::Api,
] {
let profile =
EmbedderProfile::new("e", "E", strategy, None, None, None, 24).unwrap();
let embedder = embedder_from_profile(&profile)
.unwrap_or_else(|| panic!("{strategy:?} must yield an embedder"));
assert_eq!(embedder.dimension(), 24);
let err = embedder.embed(&["t".to_string()]).await.unwrap_err();
assert!(
matches!(err, EmbedderError::Unsupported(_)),
"{strategy:?} stub must be Unsupported, got {err:?}"
);
}
}
// ===========================================================================
// 5. VectorMemoryRecall — semantic ranking, budget, cache.
// ===========================================================================
/// Builds a vector recall over `notes` with the given embedder, sharing one fs.
fn vector_recall(
embedder: Arc<dyn Embedder>,
store: Arc<FsMemoryStore>,
fs: Arc<MemFs>,
) -> VectorMemoryRecall {
VectorMemoryRecall::new(embedder, store, fs)
}
#[tokio::test]
async fn vector_recall_ranks_closest_note_first() {
// Notes have disjoint vocabularies; the query shares all tokens with `beta`,
// so `beta` must rank first (cosine == 1 for it, ~0 for the others).
let fs = MemFs::arc();
let notes = [
note("alpha", "apple orange grape"),
note("beta", "kiwi mango papaya"),
note("gamma", "carrot potato onion"),
];
let store = store_with(&notes, fs.clone()).await;
let embedder = Arc::new(HashEmbedder::new("hash", 256));
let recall = vector_recall(embedder, store, fs);
// Query identical to beta's index line ("beta kiwi mango papaya").
let out = recall
.recall(&root(), &query("beta kiwi mango papaya", 100_000))
.await
.unwrap();
assert_eq!(out.first().map(|e| e.slug.as_str()), Some("beta"), "ranked: {:?}", slugs(&out));
// All three notes fit the ample budget.
assert_eq!(out.len(), 3);
}
#[tokio::test]
async fn vector_recall_truncates_to_budget() {
// The closest note ("beta apricot apricot") must survive a tight budget while
// the rest are pruned. Cost of an entry = ceil((title+hook chars)/4).
let fs = MemFs::arc();
let notes = [
note("beta", "kiwi"), // "beta kiwi" → 8 chars → 2 tokens
note("alpha", "apple orange grape"), // larger
note("gamma", "carrot potato onion"), // larger
];
let store = store_with(&notes, fs.clone()).await;
let embedder = Arc::new(HashEmbedder::new("hash", 256));
let recall = vector_recall(embedder, store, fs);
// Budget 2 fits only the top-ranked `beta` (cost 2); the next would exceed.
let out = recall.recall(&root(), &query("beta kiwi", 2)).await.unwrap();
assert_eq!(slugs(&out), vec!["beta"], "budget must keep only the closest");
}
#[tokio::test]
async fn vector_recall_zero_budget_and_empty_memory_are_empty() {
let fs = MemFs::arc();
let embedder = Arc::new(HashEmbedder::new("hash", 64));
// Empty memory ⇒ empty, no error.
let empty_store = Arc::new(FsMemoryStore::new(fs.clone()));
let recall = vector_recall(embedder.clone(), empty_store, fs.clone());
assert!(recall
.recall(&root(), &query("anything", 1000))
.await
.unwrap()
.is_empty());
// Zero budget ⇒ empty, even with notes present.
let store = store_with(&[note("a", "x"), note("b", "y")], fs.clone()).await;
let recall = vector_recall(embedder, store, fs);
assert!(recall
.recall(&root(), &query("a x", 0))
.await
.unwrap()
.is_empty());
}
#[tokio::test]
async fn vector_recall_writes_and_reuses_the_derived_cache() {
let fs = MemFs::arc();
let notes = [note("alpha", "apple"), note("beta", "banana")];
let store = store_with(&notes, fs.clone()).await;
let spy = Arc::new(SpyEmbedder::new("spy", 128));
let recall = VectorMemoryRecall::new(spy.clone(), store, fs.clone());
// First recall: embeds 2 note texts + 1 query text = 3.
let _ = recall.recall(&root(), &query("alpha apple", 100_000)).await.unwrap();
let after_first = spy.embedded_texts.load(Ordering::SeqCst);
assert_eq!(after_first, 3, "2 notes + 1 query embedded on a cold cache");
// The derived cache file now exists with both slugs and the embedder id.
let doc = fs.raw(VECTORS_PATH).expect("vectors.json written");
assert!(doc.contains("\"embedderId\": \"spy\""), "tagged with embedder id: {doc}");
assert!(doc.contains("\"alpha\""), "alpha vector cached: {doc}");
assert!(doc.contains("\"beta\""), "beta vector cached: {doc}");
// Second recall: note vectors come from the cache; only the query is embedded.
let _ = recall.recall(&root(), &query("beta banana", 100_000)).await.unwrap();
let delta = spy.embedded_texts.load(Ordering::SeqCst) - after_first;
assert_eq!(delta, 1, "cache reuse ⇒ only the query is re-embedded, got {delta}");
}
#[tokio::test]
async fn vector_recall_invalidates_cache_on_embedder_id_change() {
let fs = MemFs::arc();
let notes = [note("alpha", "apple"), note("beta", "banana")];
let store = store_with(&notes, fs.clone()).await;
// Warm the cache with embedder "first".
let first = Arc::new(SpyEmbedder::new("first", 128));
VectorMemoryRecall::new(first.clone(), store.clone(), fs.clone())
.recall(&root(), &query("alpha apple", 100_000))
.await
.unwrap();
assert!(fs.raw(VECTORS_PATH).unwrap().contains("\"embedderId\": \"first\""));
// A different embedder id must invalidate ⇒ recompute all note vectors.
let second = Arc::new(SpyEmbedder::new("second", 128));
VectorMemoryRecall::new(second.clone(), store, fs.clone())
.recall(&root(), &query("beta banana", 100_000))
.await
.unwrap();
// 2 notes recomputed + 1 query embedded against the new embedder.
assert_eq!(
second.embedded_texts.load(Ordering::SeqCst),
3,
"id change must invalidate and recompute note vectors"
);
// The file is now tagged with the new embedder id.
assert!(fs.raw(VECTORS_PATH).unwrap().contains("\"embedderId\": \"second\""));
}
#[tokio::test]
async fn vector_recall_invalidates_cache_on_dimension_change() {
let fs = MemFs::arc();
let notes = [note("alpha", "apple"), note("beta", "banana")];
let store = store_with(&notes, fs.clone()).await;
// Warm the cache at dimension 64.
let dim64 = Arc::new(SpyEmbedder::new("same-id", 64));
VectorMemoryRecall::new(dim64, store.clone(), fs.clone())
.recall(&root(), &query("alpha apple", 100_000))
.await
.unwrap();
// Same id, different dimension ⇒ invalidation, all note vectors recomputed.
let dim128 = Arc::new(SpyEmbedder::new("same-id", 128));
VectorMemoryRecall::new(dim128.clone(), store, fs.clone())
.recall(&root(), &query("beta banana", 100_000))
.await
.unwrap();
assert_eq!(
dim128.embedded_texts.load(Ordering::SeqCst),
3,
"dimension change must invalidate and recompute"
);
assert!(fs.raw(VECTORS_PATH).unwrap().contains("\"dimension\": 128"));
}
#[tokio::test]
async fn vector_recall_with_stub_embedder_errors_without_panic() {
// The vector recall surfaces the embedder error as a MemoryError (the Adaptive
// switch turns that into a naïve fallback — covered below). Here we just pin
// that StubEmbedder is best-effort: an Err, never a panic.
let fs = MemFs::arc();
let store = store_with(&[note("alpha", "apple")], fs.clone()).await;
let stub = Arc::new(StubEmbedder::new("stub", 64, "localOnnx"));
let recall = VectorMemoryRecall::new(stub, store, fs);
let res = recall.recall(&root(), &query("alpha apple", 1000)).await;
assert!(res.is_err(), "stub embedder ⇒ Err on the vector path, got {res:?}");
}
// ===========================================================================
// 6. AdaptiveMemoryRecall — the étage-1/étage-2 switch.
// ===========================================================================
/// Assembles an [`AdaptiveMemoryRecall`] over `notes` for the given strategy and
/// embedder, sharing one in-memory fs / store.
async fn adaptive_over(
notes: &[Memory],
strategy: EmbedderStrategy,
embedder: Arc<dyn Embedder>,
) -> (Arc<MemFs>, AdaptiveMemoryRecall) {
let fs = MemFs::arc();
let store = store_with(notes, fs.clone()).await;
let naive: Arc<dyn MemoryRecall> = Arc::new(NaiveMemoryRecall::new(store.clone()));
let vector: Arc<dyn MemoryRecall> =
Arc::new(VectorMemoryRecall::new(embedder, store.clone(), fs.clone()));
let adaptive = AdaptiveMemoryRecall::new(naive, vector, store, strategy);
(fs, adaptive)
}
#[tokio::test]
async fn adaptive_none_strategy_matches_naive_exactly() {
let notes = [
note("alpha", "apple orange grape"),
note("beta", "kiwi mango papaya"),
note("gamma", "carrot potato onion"),
];
let fs = MemFs::arc();
let store = store_with(&notes, fs.clone()).await;
let naive = NaiveMemoryRecall::new(store.clone());
// Adaptive with strategy `none` (embedder present but never used).
let naive_dyn: Arc<dyn MemoryRecall> = Arc::new(NaiveMemoryRecall::new(store.clone()));
let vector_dyn: Arc<dyn MemoryRecall> = Arc::new(VectorMemoryRecall::new(
Arc::new(HashEmbedder::new("hash", 64)),
store.clone(),
fs.clone(),
));
let adaptive = AdaptiveMemoryRecall::new(naive_dyn, vector_dyn, store, EmbedderStrategy::None);
// For several budgets, adaptive(none) == naive, byte-for-byte.
for budget in [0usize, 3, 6, 100, 100_000] {
let q = query("kiwi mango papaya", budget);
let expected = naive.recall(&root(), &q).await.unwrap();
let got = adaptive.recall(&root(), &q).await.unwrap();
assert_eq!(got, expected, "strategy none must equal naive at budget {budget}");
}
}
#[tokio::test]
async fn adaptive_small_memory_routes_to_naive_order() {
// Memory smaller than the budget ⇒ naïve path ⇒ index order, even with a
// non-none strategy. The query would, on the vector path, reorder by
// similarity; index order proves naïve was used.
let notes = [
note("alpha", "apple"),
note("beta", "banana"),
note("gamma", "grape"),
];
let (_fs, adaptive) = adaptive_over(
&notes,
EmbedderStrategy::LocalServer,
Arc::new(HashEmbedder::new("hash", 256)),
)
.await;
// Huge budget ⇒ memory_size <= budget ⇒ naïve ⇒ index order alpha,beta,gamma.
let out = adaptive
.recall(&root(), &query("gamma grape", 100_000))
.await
.unwrap();
assert_eq!(slugs(&out), vec!["alpha", "beta", "gamma"], "naïve keeps index order");
}
#[tokio::test]
async fn adaptive_large_memory_routes_to_vector_ranking() {
// Memory larger than the budget + non-none strategy ⇒ vector path ⇒ the note
// closest to the query ranks first (not index order).
let notes = [
note("alpha", "apple orange grape"),
note("beta", "kiwi mango papaya"),
note("gamma", "carrot potato onion"),
];
let (_fs, adaptive) = adaptive_over(
&notes,
EmbedderStrategy::LocalOnnx,
Arc::new(HashEmbedder::new("hash", 256)),
)
.await;
// Budget smaller than the total memory size forces the vector route; the query
// matches `gamma`, so `gamma` must be first (≠ index order, where it is last).
let total = {
let store = FsMemoryStore::new(MemFs::arc());
for n in &notes {
store.save(&root(), n).await.unwrap();
}
infrastructure::index_token_size(&store.read_index(&root()).await.unwrap())
};
assert!(total > 4, "sanity: memory must exceed the chosen budget");
let out = adaptive
.recall(&root(), &query("carrot potato onion", total - 1))
.await
.unwrap();
assert_eq!(
out.first().map(|e| e.slug.as_str()),
Some("gamma"),
"vector path ranks the closest note first: {:?}",
slugs(&out)
);
}
#[tokio::test]
async fn adaptive_falls_back_to_naive_when_embedder_fails() {
// Large memory + non-none strategy routes to the vector path; with a
// StubEmbedder that path errors, and Adaptive must degrade to naïve — never a
// hard error. Result therefore equals the naïve recall (index order, truncated).
let notes = [
note("alpha", "apple orange grape"),
note("beta", "kiwi mango papaya"),
note("gamma", "carrot potato onion"),
];
let fs = MemFs::arc();
let store = store_with(&notes, fs.clone()).await;
let naive_ref = NaiveMemoryRecall::new(store.clone());
let naive_dyn: Arc<dyn MemoryRecall> = Arc::new(NaiveMemoryRecall::new(store.clone()));
let vector_dyn: Arc<dyn MemoryRecall> = Arc::new(VectorMemoryRecall::new(
Arc::new(StubEmbedder::new("stub", 64, "localOnnx")),
store.clone(),
fs.clone(),
));
let adaptive =
AdaptiveMemoryRecall::new(naive_dyn, vector_dyn, store.clone(), EmbedderStrategy::Api);
let total = infrastructure::index_token_size(&store.read_index(&root()).await.unwrap());
let q = query("carrot potato onion", total - 1);
let got = adaptive.recall(&root(), &q).await.expect("fallback must not error");
let expected = naive_ref.recall(&root(), &q).await.unwrap();
assert_eq!(got, expected, "embedder failure must degrade to the naïve result");
assert!(!got.is_empty(), "the degraded result still returns entries");
}
#[tokio::test]
async fn adaptive_empty_memory_and_zero_budget_are_empty() {
// Liskov common contract: empty list, never an error, both paths.
let fs = MemFs::arc();
let empty_store = Arc::new(FsMemoryStore::new(fs.clone()));
let naive: Arc<dyn MemoryRecall> = Arc::new(NaiveMemoryRecall::new(empty_store.clone()));
let vector: Arc<dyn MemoryRecall> = Arc::new(VectorMemoryRecall::new(
Arc::new(HashEmbedder::new("hash", 64)),
empty_store.clone(),
fs.clone(),
));
let adaptive =
AdaptiveMemoryRecall::new(naive, vector, empty_store, EmbedderStrategy::LocalOnnx);
// Empty memory.
assert!(adaptive
.recall(&root(), &query("anything", 1000))
.await
.unwrap()
.is_empty());
// Zero budget.
assert!(adaptive
.recall(&root(), &query("anything", 0))
.await
.unwrap()
.is_empty());
}