Keep/docs/integration-architecture.md

# Integration Architecture - Memento Project

## Overview

Document describing how the two parts of the Memento project (keep-notes web application and mcp-server) integrate and communicate with each other and external systems.

---

## System Architecture Diagram

```
┌─────────────────────────────────────────────────────────────┐
│                     External Systems                        │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────┐  │
│  │    User   │  │    AI    │  │   N8N    │  │  Email  │  │
│  │  Browser │  │Assistant │  │ Workflow │  │ Service │  │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬────┘  │
│       │             │             │              │         │
└───────┼─────────────┼─────────────┼──────────────┼─────────┘
        │             │             │              │
        ↓             ↓             ↓              ↓
┌───────────────────────────────────────────────────────────────┐
│                      Memento System                           │
├───────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌─────────────────────────────────────────────────────┐    │
│  │          keep-notes (Next.js Web App)               │    │
│  │                                                        │    │
│  │  Frontend (React/Next.js)                            │    │
│  │  ├─ User Interface (20+ components)                  │    │
│  │  ├─ State Management (Context + Hooks)              │    │
│  │  └─ Client-Side Routing                               │    │
│  │                                                        │    │
│  │  Backend (Next.js API Routes + Server Actions)       │    │
│  │  ├─ REST API Endpoints (12 routes)                   │    │
│  │  ├─ Server Actions (mutations)                       │    │
│  │  ├─ Authentication (NextAuth)                        │    │
│  │  ├─ AI Integration (Vercel AI SDK)                  │    │
│  │  └─ Cron Jobs (reminders)                            │    │
│  │                                                        │    │
│  │  ┌──────────────────────────────────────────────┐    │    │
│  │  │         Prisma ORM Layer                      │    │    │
│  │  │  ┌──────────────────────────────────────┐   │    │    │
│  │  │  │    SQLite Database (dev.db)          │   │    │    │
│  │  │  │                                        │   │    │    │
│  │  │  │  - User                                 │   │    │    │
│  │  │  │  - Note                                 │   │    │    │
│  │  │  │  - Label                                │   │    │    │
│  │  │  │  - Account, Session, etc.             │   │    │    │
│  │  │  │                                        │   │    │    │
│  │  │  └──────────────────────────────────────┘   │    │    │
│  │  └──────────────────────────────────────────────┘    │    │
│  └────────────────────────────────────────────────────┘    │
│                              ↕                             │
│                         Shared Data                         │
│                                                                │
│  ┌─────────────────────────────────────────────────────┐    │
│  │          mcp-server (Express MCP Server)            │    │
│  │                                                        │    │
│  │  MCP Protocol Layer                                    │    │
│  │  ├─ Tools (9 tools: create_note, search, etc.)      │    │
│  │  ├─ Request/Response Handling                         │    │
│  │  └─ Error Handling                                    │    │
│  │                                                        │    │
│  │  ┌──────────────────────────────────────────────┐    │    │
│  │  │         Prisma Client (Shared)               │    │    │
│  │  └──────────────┬───────────────────────────────┘    │    │
│  │                 │ Direct Access                     │    │
│  │                 ↓                                   │    │
│  │  ┌──────────────────────────────────────────────┐    │    │
│  │  │    ../keep-notes/prisma/dev.db (Same File)  │◄───┼────┘
│  │  └──────────────────────────────────────────────┘    │
│  └────────────────────────────────────────────────────┘    │
│                                                                │
└────────────────────────────────────────────────────────────┘
```

---

## Integration Points

### 1. Database Integration (Primary)

**Type:** Direct File Access
**Protocol:** SQLite file sharing
**Direction:** Both read/write

**keep-notes → Database:**
```typescript
// lib/prisma.ts
import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
// Direct connection to prisma/dev.db
```

**mcp-server → Database:**
```javascript
// index.js
const prisma = new PrismaClient({
  datasources: {
    db: {
      url: `file:${join(__dirname, '../keep-notes/prisma/dev.db')}`
    }
  }
});
// Connects to SAME database file
```

**Data Consistency:**
- Both parts see same data immediately
- No replication lag
- Single source of truth
- SQLite handles concurrent reads
- Single writer limitation (acceptable for current scale)

**Implications:**
- ✅ Real-time data sync
- ✅ Simple architecture
- ✅ No API needed between parts
- ⚠️ Must be on same filesystem (or network mount)
- ⚠️ SQLite concurrency limits
- ⚠️ Single point of failure (database file)

---

### 2. HTTP Integration (Future/Optional)

**Current:** No HTTP communication between keep-notes and mcp-server

**Potential HTTP Integration:**
```
keep-notes → HTTP → mcp-server → Prisma → DB
```

**Use Cases:**
- Remote deployment (separate servers)
- MCP server as API gateway
- Webhook forwarding

**Current State:** Not implemented (database-mediated preferred)

---

### 3. User Authentication Integration

**keep-notes:**
- NextAuth.js v5 for authentication
- Session stored in `Session` table (DB)
- Protected routes via middleware

**mcp-server:**
- **NO authentication** currently
- Direct database access
- Trusts client environment

**Security Implications:**
- ⚠️ MCP server has full DB access
- ⚠️ No authentication layer
- ✅ Acceptable for local/localhost
- ❌ Not secure for public deployment

**Recommendation:** Add API key auth for MCP server in production

---

## Data Flow Patterns

### Pattern 1: Web App Data Flow

```
User Browser
    ↓ HTTP Request
Next.js Route Handler
    ↓
Server Action or API Route
    ↓
Prisma Query (read/write)
    ↓
SQLite Database
    ↓
Prisma Response
    ↓
UI Update (React re-render)
```

### Pattern 2: MCP Server Data Flow

```
AI Assistant / N8N
    ↓ MCP Protocol (stdio)
MCP Server (index.js)
    ↓ Tool Invocation
Prisma Query (read/write)
    ↓
SQLite Database (shared file)
    ↓
Parse & Format Data
    ↓ MCP Response (stdio)
AI Assistant / N8N
```

### Pattern 3: External Integrations

**Email Service:**
```
keep-notes → nodemailer → SMTP Server → User Email
```

**AI Provider:**
```
keep-notes → Vercel AI SDK → OpenAI/Ollama API → AI Response
```

**N8N Workflow:**
```
N8N → MCP Protocol → mcp-server → Prisma → DB
```

---

## Service Boundaries

### keep-notes Responsibilities

**User-Facing:**
- Web UI (React components)
- User authentication
- Note management UI
- Label management UI
- Settings & admin panel

**Business Logic:**
- Note CRUD operations
- Label CRUD operations
- Auto-tagging (AI)
- Search functionality
- Reminder scheduling

**Data Access:**
- Prisma ORM
- SQLite queries
- File uploads
- AI integrations

**External Communication:**
- Email sending (nodemailer)
- AI provider APIs (OpenAI, Ollama)
- User authentication (NextAuth)

### mcp-server Responsibilities

**Protocol-Specific:**
- MCP protocol implementation
- Tool registration
- Request/response handling
- Error handling

**Data Access:**
- Direct Prisma access
- Note operations via MCP tools
- Label queries
- Search operations

**No Direct User Interface** - Protocol-only service

---

## Communication Protocols

### keep-nets → User

**Protocol:** HTTP/WebSocket
**Format:** HTML/JSON
**Authentication:** NextAuth session (cookies)

**Examples:**
- Browser → `GET /` → Next.js SSR → HTML
- Browser → `POST /api/notes` → JSON response
- Browser → Server Action → Mutation → Revalidate

### keep-notes → External Services

**Email:**
- Protocol: SMTP
- Library: nodemailer
- Purpose: Password reset, reminders

**AI Providers:**
- Protocol: HTTP
- Libraries: Vercel AI SDK
- Providers: OpenAI API, Ollama (local)

### mcp-server → MCP Clients

**Protocol:** Model Context Protocol (MCP)
**Transport:** Stdio (standard input/output)
**Format:** JSON-RPC over stdio
**Authentication:** None (currently)

**Clients:**
- AI Assistants (ChatGPT, Claude)
- N8N workflows
- Custom automation scripts

---

## Deployment Topologies

### Current: Local Development

```
Same Machine
├── keep-notes (npm run dev)
│   └── http://localhost:3000
├── mcp-server (npm start)
│   └── stdio://
└── SQLite DB (shared file)
    └── keep-notes/prisma/dev.db
```

### Production Option 1: Single Server

```
Single Server (Docker/VM)
├── keep-notes (Node.js process)
├── mcp-server (Node.js process)
└── SQLite DB (shared file)
```

**Pros:** Simple, low latency
**Cons:** Single point of failure

### Production Option 2: Microservices

```
├── Web Server (keep-notes)
│   └── Docker container
├── MCP Server (mcp-server)
│   └── Docker container
└── Database Server (PostgreSQL)
    └── Connection pooling
```

**Pros:** Scalable, resilient
**Cons:** More complex, requires PostgreSQL migration

### Production Option 3: Cloud Native

```
├── Vercel/Netlify (keep-notes frontend)
├── Cloud Run/AWS Lambda (MCP server)
└── Cloud SQL (PostgreSQL)
```

**Pros:** Highly scalable, managed
**Cons:** Vendor lock-in, higher cost

---

## API Contract Between Parts

### Currently: NO Direct HTTP API

**Integration Method:** Shared database
**Communication:** Database-mediated
**Synchronization:** Immediate (via shared DB)

**Future HTTP API (if needed):**

```
keep-notes exposes internal API:
  GET /api/notes
  POST /api/notes
  etc.

mcp-server could call these:
  fetch('http://localhost:3000/api/notes')
```

**Current Preference:** Database sharing (simpler, faster)

---

## Authentication Flow

### User Authentication (keep-notes)

```
1. User enters credentials in LoginForm
2. POST /api/auth/signin
3. NextAuth validates credentials
4. Creates session in DB
5. Sets HTTP-only cookie
6. Redirects to dashboard
```

### MCP Authentication (mcp-server)

```
1. MCP client connects via stdio
2. No authentication required
3. Full database access granted
4. Executes tools directly
```

**Security Note:** This is acceptable for local/trusted environments but should be enhanced for production deployment.

---

## Data Synchronization

### Consistency Model

**Type:** Strong consistency via shared database
**Mechanism:** SQLite ACID properties
**Lag:** Zero (immediate)

**Conflict Resolution:**
- Last write wins (SQLite default)
- Application-level locking for critical operations

**Caching:**
- React Cache (server components)
- No explicit cache invalidation needed (DB is source of truth)

---

## Error Handling Across Parts

### keep-notes Errors

**Database Errors:**
```typescript
try {
  const note = await prisma.note.create({ ... })
} catch (error) {
  return NextResponse.json(
    { success: false, error: 'Failed to create note' },
    { status: 500 }
  )
}
```

**Action Errors:**
- Form validation errors (Zod)
- User-facing error messages
- Graceful degradation

### mcp-server Errors

**Tool Execution Errors:**
```javascript
try {
  const note = await prisma.note.create({ ... })
} catch (error) {
  throw new McpError(
    ErrorCode.InternalError,
    `Tool execution failed: ${error.message}`
  )
}
```

**Error Types:**
- `InvalidRequest`: Bad parameters
- `InternalError`: DB/Server errors
- `MethodNotFound`: Unknown tool

---

## Monitoring & Observability

### Current State: Minimal

**keep-notes:**
- Console logging
- Playwright test reports

**mcp-server:**
- Stderr logging
- No structured logging

### Recommended Enhancements

**1. Distributed Tracing**
- Trace requests across both parts
- Correlation IDs
- Performance monitoring

**2. Centralized Logging**
- Structured JSON logs
- Log aggregation (ELK, Loki)
- Error tracking (Sentry)

**3. Metrics**
- Request/response times
- Database query performance
- Tool execution statistics

**4. Health Checks**
- `/health` endpoint for keep-notes
- Health check for mcp-server
- Database connectivity checks

---

## Scaling Considerations

### Current Limitations (SQLite)

**Concurrency:**
- Multiple readers OK
- Single writer (limitation)
- Lock contention possible under load

**Capacity:**
- File-based storage
- Manual backups required
- No automatic failover

### Scaling Strategies

**Horizontal Scaling:**
- **Web App:** Multiple instances behind load balancer
- **MCP Server:** Multiple instances (round-robin)
- **Database:** Migrate to PostgreSQL

**Vertical Scaling:**
- More CPU cores
- More RAM (for caching)
- Faster SSD (for I/O)

**Database Migration:**
```prisma
// Change datasource in schema.prisma
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

// Update adapters
// - @prisma/adapter-pg
```

---

## Security Architecture

### Current Security Posture

**keep-notes:**
- ✅ NextAuth authentication
- ✅ Password hashing (bcrypt)
- ✅ Session management
- ✅ Protected routes
- ✅ CSRF protection (NextAuth)
- ⚠️ No rate limiting
- ⚠️ No brute force protection

**mcp-server:**
- ✅ Input validation (schemas)
- ✅ SQL injection protection (Prisma)
- ❌ No authentication
- ❌ No authorization
- ❌ No rate limiting

### Security Recommendations

**Immediate:**
1. Add API key authentication to mcp-server
2. Restrict mcp-server to localhost/VPN
3. Add rate limiting to both services
4. Implement brute force protection

**Future:**
1. HTTPS/TLS for all communications
2. Input sanitization
3. Output encoding
4. Security headers (CSP, HSTS, etc.)
5. Regular security audits

---

## Integration Testing

### Test Scenarios

**1. Data Consistency**
- Create note in keep-notes → Verify visible in mcp-server
- Update via mcp-server → Verify visible in keep-notes

**2. Concurrent Access**
- Simultaneous writes to same note
- Race condition testing
- Lock behavior verification

**3. Error Propagation**
- Database connection failures
- Invalid data handling
- Graceful degradation

### Testing Tools

**Integration Tests:**
- Playwright (E2E for keep-notes)
- Custom MCP client tests
- Database state verification

**Load Tests:**
- Concurrent tool execution
- Database query performance
- Memory leak detection

---

## Deployment Integration

### Docker Compose (Recommended)

```yaml
version: '3.8'
services:
  keep-notes:
    build: ./keep-notes
    ports:
      - "3000:3000"
    volumes:
      - db-data:/app/prisma
    environment:
      - DATABASE_URL=file:/app/prisma/dev.db

  mcp-server:
    build: ./mcp-server
    volumes:
      - db-data:/app/db
    depends_on:
      - keep-notes
    environment:
      - DATABASE_URL=file:/app/db/dev.db

volumes:
  db-data:
```

**Benefits:**
- Single command deployment
- Shared volume for database
- Environment configuration
- Easy local development

---

## Summary

The Memento project uses a **database-mediated integration pattern**:

**Strengths:**
- ✅ Simple architecture
- ✅ Real-time data sync
- ✅ No API complexity
- ✅ Easy local development
- ✅ Strong consistency

**Considerations:**
- ⚠️ Single database file (scalability)
- ⚠️ No authentication in MCP server
- ⚠️ SQLite concurrency limits
- ⚠️ Manual backup required

**Ideal For:**
- Single-user or small team deployments
- Local/self-hosted environments
- Applications with modest write volume
- Projects prioritizing simplicity

**Future Evolution:**
- Add authentication layer
- Migrate to PostgreSQL for scaling
- Implement HTTP API between parts
- Add caching layer (Redis)
- Implement message queue for async ops