# Embedding Model Validation & Search Robustness ## Context pgvector supports max 2000 dimensions for HNSW/IVFFlat indexes. The app must validate embedding models and gracefully handle dimension mismatches. ## Tasks ### 1. Revert dimension from 2560 back to 1536 - All files changed in commit e09ea3a need reverting: 1536 everywhere - This includes: schema.prisma (both), migration.sql, embedding.service.ts, validate route, scripts, tests ### 2. Add embedding dimension validation in admin settings File: `memento-note/lib/ai/services/embedding.service.ts` - After generating an embedding, check its dimension - Add a method `validateEmbeddingModel()` that: - Generates a test embedding - Checks dimension count - Returns { valid: boolean, dimensions: number, warning?: string } - If dimensions > 2000: warning "This model produces {N} dimensions. pgvector indexes support max 2000 dimensions. Semantic search will use sequential scan (slower for large note collections)." - If dimensions != current DB vector dimension: warning "Dimension mismatch: model produces {N}d but DB stores {M}d. You need to reindex all notes." File: `memento-note/app/api/admin/embeddings/validate/route.ts` - Use the new validateEmbeddingModel() method - Return dimension info in the API response File: `memento-note/app/api/admin/settings/route.ts` (or wherever embedding model is saved) - After saving a new embedding model, call validateEmbeddingModel() - Store the warning in the response so the frontend can display it ### 3. Make semantic search robust File: `memento-note/lib/ai/services/semantic-search.service.ts` - In `vectorSearch()`: after generating query embedding, check dimension matches DB (1536). If not, log warning and return [] (fallback to FTS) - In `_doSearch()`: the existing try/catch already calls `_ftsFallback()`. Make sure this works. ### 4. Update the migration SQL File: `memento-note/prisma/migrations/20260512120000_pgvector_and_fts_search/migration.sql` - Keep vector(1536) as the target type - The migration should work correctly ### 5. Commit - Git add and commit with descriptive message - Do NOT push