Momento/EMBEDDING-VALIDATION-TASK.md

Embedding Model Validation & Search Robustness

Context

pgvector supports max 2000 dimensions for HNSW/IVFFlat indexes. The app must validate embedding models and gracefully handle dimension mismatches.

Tasks

1. Revert dimension from 2560 back to 1536

• All files changed in commit e09ea3a need reverting: 1536 everywhere
• This includes: schema.prisma (both), migration.sql, embedding.service.ts, validate route, scripts, tests

2. Add embedding dimension validation in admin settings

File: memento-note/lib/ai/services/embedding.service.ts

• After generating an embedding, check its dimension
• Add a method validateEmbeddingModel() that:
  • Generates a test embedding
  • Checks dimension count
  • Returns { valid: boolean, dimensions: number, warning?: string }
  • If dimensions > 2000: warning "This model produces {N} dimensions. pgvector indexes support max 2000 dimensions. Semantic search will use sequential scan (slower for large note collections)."
  • If dimensions != current DB vector dimension: warning "Dimension mismatch: model produces {N}d but DB stores {M}d. You need to reindex all notes."

File: memento-note/app/api/admin/embeddings/validate/route.ts

• Use the new validateEmbeddingModel() method
• Return dimension info in the API response

File: memento-note/app/api/admin/settings/route.ts (or wherever embedding model is saved)

• After saving a new embedding model, call validateEmbeddingModel()
• Store the warning in the response so the frontend can display it

3. Make semantic search robust

File: memento-note/lib/ai/services/semantic-search.service.ts

• In vectorSearch(): after generating query embedding, check dimension matches DB (1536). If not, log warning and return [] (fallback to FTS)
• In _doSearch(): the existing try/catch already calls _ftsFallback(). Make sure this works.

4. Update the migration SQL

File: memento-note/prisma/migrations/20260512120000_pgvector_and_fts_search/migration.sql

• Keep vector(1536) as the target type
• The migration should work correctly

5. Commit

• Git add and commit with descriptive message
• Do NOT push