feat: migrate semantic search to pgvector + full-text search
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m12s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m12s
Replace JSON-string embeddings with native pgvector(1536) storage and add PostgreSQL full-text search (tsvector/GIN) with Reciprocal Rank Fusion for hybrid keyword + semantic ranking. Changes: - NoteEmbedding.embedding: String → vector(1536) via pgvector - NoteEmbedding: added updatedAt for reindex tracking - Note: added tsv (tsvector) with auto-update trigger for FTS - semantic-search.service: hybrid FTS + vector search with RRF fusion - embedding.service: toVectorString() for pgvector SQL literals - Removed JS-side cosine similarity loops (now DB-side via <=>) - Added HNSW index on NoteEmbedding.embedding (cosine distance) - Added GIN index on Note.tsv for FTS queries Schema migration in: prisma/migrations/20260512120000_pgvector_and_fts_search/ Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -27,14 +27,18 @@ export async function POST(req: NextRequest) {
|
||||
}
|
||||
})
|
||||
|
||||
// 2. Clean up NoteEmbeddings that don't have a corresponding Note (shouldn't happen with Cascade, but good for cleanup)
|
||||
const orphanedEmbeddings = await prisma.noteEmbedding.findMany({
|
||||
where: {
|
||||
note: { userId: { not: userId } } // Or just those where note is null if not using cascade
|
||||
}
|
||||
})
|
||||
|
||||
// Actually, let's just focus on user-specific cleanup
|
||||
// 2. Clean up NoteEmbeddings that don't have a corresponding Note
|
||||
const orphanedEmbeddings: Array<{ id: string }> = await prisma.$queryRawUnsafe(
|
||||
`SELECT e.id FROM "NoteEmbedding" e
|
||||
LEFT JOIN "Note" n ON n.id = e."noteId"
|
||||
WHERE n.id IS NULL`
|
||||
)
|
||||
|
||||
if (orphanedEmbeddings.length > 0) {
|
||||
await prisma.$executeRawUnsafe(
|
||||
`DELETE FROM "NoteEmbedding" WHERE id = ANY(${`ARRAY['${orphanedEmbeddings.map(e => e.id).join("','")}']`}::text[])`
|
||||
)
|
||||
}
|
||||
|
||||
// 3. Remove note history entries for notes that were deleted (cascade should handle this, but let's be safe)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user