Keep/docs/integration-architecture.md

Integration Architecture - Memento Project

Overview

Document describing how the two parts of the Memento project (keep-notes web application and mcp-server) integrate and communicate with each other and external systems.

---

System Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                     External Systems                        │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────┐  │
│  │    User   │  │    AI    │  │   N8N    │  │  Email  │  │
│  │  Browser │  │Assistant │  │ Workflow │  │ Service │  │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬────┘  │
│       │             │             │              │         │
└───────┼─────────────┼─────────────┼──────────────┼─────────┘
        │             │             │              │
        ↓             ↓             ↓              ↓
┌───────────────────────────────────────────────────────────────┐
│                      Memento System                           │
├───────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌─────────────────────────────────────────────────────┐    │
│  │          keep-notes (Next.js Web App)               │    │
│  │                                                        │    │
│  │  Frontend (React/Next.js)                            │    │
│  │  ├─ User Interface (20+ components)                  │    │
│  │  ├─ State Management (Context + Hooks)              │    │
│  │  └─ Client-Side Routing                               │    │
│  │                                                        │    │
│  │  Backend (Next.js API Routes + Server Actions)       │    │
│  │  ├─ REST API Endpoints (12 routes)                   │    │
│  │  ├─ Server Actions (mutations)                       │    │
│  │  ├─ Authentication (NextAuth)                        │    │
│  │  ├─ AI Integration (Vercel AI SDK)                  │    │
│  │  └─ Cron Jobs (reminders)                            │    │
│  │                                                        │    │
│  │  ┌──────────────────────────────────────────────┐    │    │
│  │  │         Prisma ORM Layer                      │    │    │
│  │  │  ┌──────────────────────────────────────┐   │    │    │
│  │  │  │    SQLite Database (dev.db)          │   │    │    │
│  │  │  │                                        │   │    │    │
│  │  │  │  - User                                 │   │    │    │
│  │  │  │  - Note                                 │   │    │    │
│  │  │  │  - Label                                │   │    │    │
│  │  │  │  - Account, Session, etc.             │   │    │    │
│  │  │  │                                        │   │    │    │
│  │  │  └──────────────────────────────────────┘   │    │    │
│  │  └──────────────────────────────────────────────┘    │    │
│  └────────────────────────────────────────────────────┘    │
│                              ↕                             │
│                         Shared Data                         │
│                                                                │
│  ┌─────────────────────────────────────────────────────┐    │
│  │          mcp-server (Express MCP Server)            │    │
│  │                                                        │    │
│  │  MCP Protocol Layer                                    │    │
│  │  ├─ Tools (9 tools: create_note, search, etc.)      │    │
│  │  ├─ Request/Response Handling                         │    │
│  │  └─ Error Handling                                    │    │
│  │                                                        │    │
│  │  ┌──────────────────────────────────────────────┐    │    │
│  │  │         Prisma Client (Shared)               │    │    │
│  │  └──────────────┬───────────────────────────────┘    │    │
│  │                 │ Direct Access                     │    │
│  │                 ↓                                   │    │
│  │  ┌──────────────────────────────────────────────┐    │    │
│  │  │    ../keep-notes/prisma/dev.db (Same File)  │◄───┼────┘
│  │  └──────────────────────────────────────────────┘    │
│  └────────────────────────────────────────────────────┘    │
│                                                                │
└────────────────────────────────────────────────────────────┘

---

Integration Points

1. Database Integration (Primary)

Type: Direct File Access Protocol: SQLite file sharing Direction: Both read/write

keep-notes → Database:

// lib/prisma.ts
import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
// Direct connection to prisma/dev.db

mcp-server → Database:

// index.js
const prisma = new PrismaClient({
  datasources: {
    db: {
      url: `file:${join(__dirname, '../keep-notes/prisma/dev.db')}`
    }
  }
});
// Connects to SAME database file

Data Consistency:

• Both parts see same data immediately
• No replication lag
• Single source of truth
• SQLite handles concurrent reads
• Single writer limitation (acceptable for current scale)

Implications:

• ✅ Real-time data sync
• ✅ Simple architecture
• ✅ No API needed between parts
• ⚠️ Must be on same filesystem (or network mount)
• ⚠️ SQLite concurrency limits
• ⚠️ Single point of failure (database file)

---

2. HTTP Integration (Future/Optional)

Current: No HTTP communication between keep-notes and mcp-server

Potential HTTP Integration:

keep-notes → HTTP → mcp-server → Prisma → DB

Use Cases:

• Remote deployment (separate servers)
• MCP server as API gateway
• Webhook forwarding

Current State: Not implemented (database-mediated preferred)

---

3. User Authentication Integration

keep-notes:

• NextAuth.js v5 for authentication
• Session stored in Session table (DB)
• Protected routes via middleware

mcp-server:

• NO authentication currently
• Direct database access
• Trusts client environment

Security Implications:

• ⚠️ MCP server has full DB access
• ⚠️ No authentication layer
• ✅ Acceptable for local/localhost
• ❌ Not secure for public deployment

Recommendation: Add API key auth for MCP server in production

---

Data Flow Patterns

Pattern 1: Web App Data Flow

User Browser
    ↓ HTTP Request
Next.js Route Handler
    ↓
Server Action or API Route
    ↓
Prisma Query (read/write)
    ↓
SQLite Database
    ↓
Prisma Response
    ↓
UI Update (React re-render)

Pattern 2: MCP Server Data Flow

AI Assistant / N8N
    ↓ MCP Protocol (stdio)
MCP Server (index.js)
    ↓ Tool Invocation
Prisma Query (read/write)
    ↓
SQLite Database (shared file)
    ↓
Parse & Format Data
    ↓ MCP Response (stdio)
AI Assistant / N8N

Pattern 3: External Integrations

Email Service:

keep-notes → nodemailer → SMTP Server → User Email

AI Provider:

keep-notes → Vercel AI SDK → OpenAI/Ollama API → AI Response

N8N Workflow:

N8N → MCP Protocol → mcp-server → Prisma → DB

---

Service Boundaries

keep-notes Responsibilities

User-Facing:

• Web UI (React components)
• User authentication
• Note management UI
• Label management UI
• Settings & admin panel

Business Logic:

• Note CRUD operations
• Label CRUD operations
• Auto-tagging (AI)
• Search functionality
• Reminder scheduling

Data Access:

• Prisma ORM
• SQLite queries
• File uploads
• AI integrations

External Communication:

• Email sending (nodemailer)
• AI provider APIs (OpenAI, Ollama)
• User authentication (NextAuth)

mcp-server Responsibilities

Protocol-Specific:

• MCP protocol implementation
• Tool registration
• Request/response handling
• Error handling

Data Access:

• Direct Prisma access
• Note operations via MCP tools
• Label queries
• Search operations

No Direct User Interface - Protocol-only service

---

Communication Protocols

keep-nets → User

Protocol: HTTP/WebSocket Format: HTML/JSON Authentication: NextAuth session (cookies)

Examples:

• Browser → GET / → Next.js SSR → HTML
• Browser → POST /api/notes → JSON response
• Browser → Server Action → Mutation → Revalidate

keep-notes → External Services

Email:

• Protocol: SMTP
• Library: nodemailer
• Purpose: Password reset, reminders

AI Providers:

• Protocol: HTTP
• Libraries: Vercel AI SDK
• Providers: OpenAI API, Ollama (local)

mcp-server → MCP Clients

Protocol: Model Context Protocol (MCP) Transport: Stdio (standard input/output) Format: JSON-RPC over stdio Authentication: None (currently)

Clients:

• AI Assistants (ChatGPT, Claude)
• N8N workflows
• Custom automation scripts

---

Deployment Topologies

Current: Local Development

Same Machine
├── keep-notes (npm run dev)
│   └── http://localhost:3000
├── mcp-server (npm start)
│   └── stdio://
└── SQLite DB (shared file)
    └── keep-notes/prisma/dev.db

Production Option 1: Single Server

Single Server (Docker/VM)
├── keep-notes (Node.js process)
├── mcp-server (Node.js process)
└── SQLite DB (shared file)

Pros: Simple, low latency Cons: Single point of failure

Production Option 2: Microservices

├── Web Server (keep-notes)
│   └── Docker container
├── MCP Server (mcp-server)
│   └── Docker container
└── Database Server (PostgreSQL)
    └── Connection pooling

Pros: Scalable, resilient Cons: More complex, requires PostgreSQL migration

Production Option 3: Cloud Native

├── Vercel/Netlify (keep-notes frontend)
├── Cloud Run/AWS Lambda (MCP server)
└── Cloud SQL (PostgreSQL)

Pros: Highly scalable, managed Cons: Vendor lock-in, higher cost

---

API Contract Between Parts

Currently: NO Direct HTTP API

Integration Method: Shared database Communication: Database-mediated Synchronization: Immediate (via shared DB)

Future HTTP API (if needed):

keep-notes exposes internal API:
  GET /api/notes
  POST /api/notes
  etc.

mcp-server could call these:
  fetch('http://localhost:3000/api/notes')

Current Preference: Database sharing (simpler, faster)

---

Authentication Flow

User Authentication (keep-notes)

1. User enters credentials in LoginForm
2. POST /api/auth/signin
3. NextAuth validates credentials
4. Creates session in DB
5. Sets HTTP-only cookie
6. Redirects to dashboard

MCP Authentication (mcp-server)

1. MCP client connects via stdio
2. No authentication required
3. Full database access granted
4. Executes tools directly

Security Note: This is acceptable for local/trusted environments but should be enhanced for production deployment.

---

Data Synchronization

Consistency Model

Type: Strong consistency via shared database Mechanism: SQLite ACID properties Lag: Zero (immediate)

Conflict Resolution:

• Last write wins (SQLite default)
• Application-level locking for critical operations

Caching:

• React Cache (server components)
• No explicit cache invalidation needed (DB is source of truth)

---

Error Handling Across Parts

keep-notes Errors

Database Errors:

try {
  const note = await prisma.note.create({ ... })
} catch (error) {
  return NextResponse.json(
    { success: false, error: 'Failed to create note' },
    { status: 500 }
  )
}

Action Errors:

• Form validation errors (Zod)
• User-facing error messages
• Graceful degradation

mcp-server Errors

Tool Execution Errors:

try {
  const note = await prisma.note.create({ ... })
} catch (error) {
  throw new McpError(
    ErrorCode.InternalError,
    `Tool execution failed: ${error.message}`
  )
}

Error Types:

• InvalidRequest: Bad parameters
• InternalError: DB/Server errors
• MethodNotFound: Unknown tool

---

Monitoring & Observability

Current State: Minimal

keep-notes:

• Console logging
• Playwright test reports

mcp-server:

• Stderr logging
• No structured logging

Recommended Enhancements

1. Distributed Tracing

• Trace requests across both parts
• Correlation IDs
• Performance monitoring

2. Centralized Logging

• Structured JSON logs
• Log aggregation (ELK, Loki)
• Error tracking (Sentry)

3. Metrics

• Request/response times
• Database query performance
• Tool execution statistics

4. Health Checks

• /health endpoint for keep-notes
• Health check for mcp-server
• Database connectivity checks

---

Scaling Considerations

Current Limitations (SQLite)

Concurrency:

• Multiple readers OK
• Single writer (limitation)
• Lock contention possible under load

Capacity:

• File-based storage
• Manual backups required
• No automatic failover

Scaling Strategies

Horizontal Scaling:

• Web App: Multiple instances behind load balancer
• MCP Server: Multiple instances (round-robin)
• Database: Migrate to PostgreSQL

Vertical Scaling:

• More CPU cores
• More RAM (for caching)
• Faster SSD (for I/O)

Database Migration:

// Change datasource in schema.prisma
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

// Update adapters
// - @prisma/adapter-pg

---

Security Architecture

Current Security Posture

keep-notes:

• ✅ NextAuth authentication
• ✅ Password hashing (bcrypt)
• ✅ Session management
• ✅ Protected routes
• ✅ CSRF protection (NextAuth)
• ⚠️ No rate limiting
• ⚠️ No brute force protection

mcp-server:

• ✅ Input validation (schemas)
• ✅ SQL injection protection (Prisma)
• ❌ No authentication
• ❌ No authorization
• ❌ No rate limiting

Security Recommendations

Immediate:

1. Add API key authentication to mcp-server
2. Restrict mcp-server to localhost/VPN
3. Add rate limiting to both services
4. Implement brute force protection

Future:

1. HTTPS/TLS for all communications
2. Input sanitization
3. Output encoding
4. Security headers (CSP, HSTS, etc.)
5. Regular security audits

---

Integration Testing

Test Scenarios

1. Data Consistency

• Create note in keep-notes → Verify visible in mcp-server
• Update via mcp-server → Verify visible in keep-notes

2. Concurrent Access

• Simultaneous writes to same note
• Race condition testing
• Lock behavior verification

3. Error Propagation

• Database connection failures
• Invalid data handling
• Graceful degradation

Testing Tools

Integration Tests:

• Playwright (E2E for keep-notes)
• Custom MCP client tests
• Database state verification

Load Tests:

• Concurrent tool execution
• Database query performance
• Memory leak detection

---

Deployment Integration

Docker Compose (Recommended)

version: '3.8'
services:
  keep-notes:
    build: ./keep-notes
    ports:
      - "3000:3000"
    volumes:
      - db-data:/app/prisma
    environment:
      - DATABASE_URL=file:/app/prisma/dev.db

  mcp-server:
    build: ./mcp-server
    volumes:
      - db-data:/app/db
    depends_on:
      - keep-notes
    environment:
      - DATABASE_URL=file:/app/db/dev.db

volumes:
  db-data:

Benefits:

• Single command deployment
• Shared volume for database
• Environment configuration
• Easy local development

---

Summary

The Memento project uses a database-mediated integration pattern:

Strengths:

• ✅ Simple architecture
• ✅ Real-time data sync
• ✅ No API complexity
• ✅ Easy local development
• ✅ Strong consistency

Considerations:

• ⚠️ Single database file (scalability)
• ⚠️ No authentication in MCP server
• ⚠️ SQLite concurrency limits
• ⚠️ Manual backup required

Ideal For:

• Single-user or small team deployments
• Local/self-hosted environments
• Applications with modest write volume
• Projects prioritizing simplicity

Future Evolution:

• Add authentication layer
• Migrate to PostgreSQL for scaling
• Implement HTTP API between parts
• Add caching layer (Redis)
• Implement message queue for async ops