feat: migrate semantic search to pgvector + full-text search
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m12s

Replace JSON-string embeddings with native pgvector(1536) storage and
add PostgreSQL full-text search (tsvector/GIN) with Reciprocal Rank Fusion
for hybrid keyword + semantic ranking.

Changes:
- NoteEmbedding.embedding: String → vector(1536) via pgvector
- NoteEmbedding: added updatedAt for reindex tracking
- Note: added tsv (tsvector) with auto-update trigger for FTS
- semantic-search.service: hybrid FTS + vector search with RRF fusion
- embedding.service: toVectorString() for pgvector SQL literals
- Removed JS-side cosine similarity loops (now DB-side via <=>)
- Added HNSW index on NoteEmbedding.embedding (cosine distance)
- Added GIN index on Note.tsv for FTS queries

Schema migration in: prisma/migrations/20260512120000_pgvector_and_fts_search/

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Antigravity
2026-05-12 07:03:56 +00:00
parent 92c3a6f307
commit 03e6a62b80
43 changed files with 4024 additions and 786 deletions

View File

@@ -27,14 +27,18 @@ export async function POST(req: NextRequest) {
}
})
// 2. Clean up NoteEmbeddings that don't have a corresponding Note (shouldn't happen with Cascade, but good for cleanup)
const orphanedEmbeddings = await prisma.noteEmbedding.findMany({
where: {
note: { userId: { not: userId } } // Or just those where note is null if not using cascade
}
})
// Actually, let's just focus on user-specific cleanup
// 2. Clean up NoteEmbeddings that don't have a corresponding Note
const orphanedEmbeddings: Array<{ id: string }> = await prisma.$queryRawUnsafe(
`SELECT e.id FROM "NoteEmbedding" e
LEFT JOIN "Note" n ON n.id = e."noteId"
WHERE n.id IS NULL`
)
if (orphanedEmbeddings.length > 0) {
await prisma.$executeRawUnsafe(
`DELETE FROM "NoteEmbedding" WHERE id = ANY(${`ARRAY['${orphanedEmbeddings.map(e => e.id).join("','")}']`}::text[])`
)
}
// 3. Remove note history entries for notes that were deleted (cascade should handle this, but let's be safe)