41596c2035
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 5s
43 lines
2.1 KiB
Markdown
43 lines
2.1 KiB
Markdown
# Embedding Model Validation & Search Robustness
|
|
|
|
## Context
|
|
pgvector supports max 2000 dimensions for HNSW/IVFFlat indexes. The app must validate embedding models and gracefully handle dimension mismatches.
|
|
|
|
## Tasks
|
|
|
|
### 1. Revert dimension from 2560 back to 1536
|
|
- All files changed in commit e09ea3a need reverting: 1536 everywhere
|
|
- This includes: schema.prisma (both), migration.sql, embedding.service.ts, validate route, scripts, tests
|
|
|
|
### 2. Add embedding dimension validation in admin settings
|
|
File: `memento-note/lib/ai/services/embedding.service.ts`
|
|
- After generating an embedding, check its dimension
|
|
- Add a method `validateEmbeddingModel()` that:
|
|
- Generates a test embedding
|
|
- Checks dimension count
|
|
- Returns { valid: boolean, dimensions: number, warning?: string }
|
|
- If dimensions > 2000: warning "This model produces {N} dimensions. pgvector indexes support max 2000 dimensions. Semantic search will use sequential scan (slower for large note collections)."
|
|
- If dimensions != current DB vector dimension: warning "Dimension mismatch: model produces {N}d but DB stores {M}d. You need to reindex all notes."
|
|
|
|
File: `memento-note/app/api/admin/embeddings/validate/route.ts`
|
|
- Use the new validateEmbeddingModel() method
|
|
- Return dimension info in the API response
|
|
|
|
File: `memento-note/app/api/admin/settings/route.ts` (or wherever embedding model is saved)
|
|
- After saving a new embedding model, call validateEmbeddingModel()
|
|
- Store the warning in the response so the frontend can display it
|
|
|
|
### 3. Make semantic search robust
|
|
File: `memento-note/lib/ai/services/semantic-search.service.ts`
|
|
- In `vectorSearch()`: after generating query embedding, check dimension matches DB (1536). If not, log warning and return [] (fallback to FTS)
|
|
- In `_doSearch()`: the existing try/catch already calls `_ftsFallback()`. Make sure this works.
|
|
|
|
### 4. Update the migration SQL
|
|
File: `memento-note/prisma/migrations/20260512120000_pgvector_and_fts_search/migration.sql`
|
|
- Keep vector(1536) as the target type
|
|
- The migration should work correctly
|
|
|
|
### 5. Commit
|
|
- Git add and commit with descriptive message
|
|
- Do NOT push
|