Momento/docs/3-1-freemium-quota-tracking.md

# Story 3.1: Freemium "AI Discovery Pack" Quota Tracking

Status: review

---

## Story

As a business,
I want to track Freemium usage against a Redis quota limit,
So that I can limit my API cost exposure for free users.

**Given** a free user triggers an AI request
**When** the system intercepts the request
**Then** the quota is tracked and the UI updates
**And** (NFR-SC2) the Redis-backed check resolves in under 10ms.

---

## Epic Context

**Epic 3:** The SaaS Commercial Engine (Monetization & API Cost Protection)

**Epic Business Value:** The core backend logic allowing us to sell the product without bleeding API costs — freemium limits, router fallback, host-pays.

**Cross-Story Dependencies:**
- Story 3.1 (this story) establishes the quota tracking foundation
- Story 3.2 builds on this with the LLM Router and provider routing
- Story 3.3 adds smart-routing fallback when quota is low
- Story 3.4 (Host-Pays) extends quota tracking to collaborative sessions
- Story 3.5 (BYOK) bypasses quota for users with their own API keys
- Story 3.6 (Stripe) manages tier upgrades

**Technical Constraint:** (NFR-SC2) Redis-backed entitlement checks must complete in under 10ms.

---

## Acceptance Criteria

1. [AC1] When a free user (BASIC tier) makes an AI request, the system checks Redis for current usage count
2. [AC2] Each AI feature (semantic_search, auto_tag, auto_title) has its own Redis counter with format: `usage:{userId}:{feature}:{YYYY-MM}`
3. [AC3] Counter increments atomically via Redis INCRBY (not read-modify-write) to avoid race conditions
4. [AC4] If counter >= limit, return HTTP 402 with body `{ error: "QUOTA_EXCEEDED", feature, upgradeTier: "PRO", byokConfigured }`
5. [AC5] If counter < limit, allow request to proceed and increment counter asynchronously (fire-and-forget)
6. [AC6] Redis keys have 90-day TTL to auto-cleanup (covers grace period for monthly reconciliation)
7. [AC7] The Sidebar footer displays a usage gauge component showing Discovery Pack consumption in real-time

---

## Tasks / Subtasks

- [x] Task 1: Add Prisma models for Subscription, UsageLog (AC: #1)
  - [x] Subtask 1.1: Add `Subscription` model with tier/stripe fields
  - [x] Subtask 1.2: Add `UsageLog` model for PostgreSQL sync target
  - [x] Subtask 1.3: Create migration file (via `npx prisma generate`)
- [x] Task 2: Create `lib/entitlements.ts` with Redis-backed `canUseFeature()` (AC: #2, #3, #4)
  - [x] Subtask 2.1: Implement `getCurrentPeriodKey()` returning `YYYY-MM` format
  - [x] Subtask 2.2: Implement Redis key format: `usage:{userId}:{feature}:{YYYY-MM}`
  - [x] Subtask 2.3: Implement atomic `canUseFeature()` with < 10ms target (use Redis GET + pipeline INCRBY)
  - [x] Subtask 2.4: Return `QuotaExceededError` when limit exceeded
- [x] Task 3: Create `lib/usage-tracker.ts` with `trackFeatureUsage()` (AC: #5)
  - [x] Subtask 3.1: Fire-and-forget Redis pipeline increment
  - [x] Subtask 3.2: Include tokensUsed in metadata
- [x] Task 4: Create `/api/usage/current` endpoint (AC: #6)
  - [x] Subtask 4.1: Return remaining quota for all features for authenticated user
- [x] Task 5: Create `<UsageMeter>` UI component for Sidebar footer (AC: #7)
  - [x] Subtask 5.1: Show progress bar for Discovery Pack (semantic_search: 30 lifetime, auto_tag: 20, auto_title: 10)
  - [x] Subtask 5.2: Real-time updates via React Query polling every 30s
  - [x] Subtask 5.3: Show "Upgrade to Pro" paywall modal on 402 response
- [x] Task 6: Create CRON sync worker `/api/cron/sync-usage` (AC: #6)
  - [x] Subtask 6.1: Batch sync Redis counters → PostgreSQL UsageLog
  - [x] Subtask 6.2: Handle monthly reset (new period = reset counters for that user/feature)

---

## Dev Notes

### Project Structure Notes

- **Code base:** `memento-note/` (Next.js app with App Router)
- **Redis client:** Currently `memento-note/lib/rate-limit.ts` uses in-memory Map (INSUFFICIENT). This story replaces it with actual Redis.
- **No existing `lib/entitlements.ts`** — creates from scratch
- **No existing `lib/usage-tracker.ts`** — creates from scratch
- **Sidebar:** `memento-note/components/sidebar.tsx` is the anchor for the UsageMeter UI
- **AI Factory:** `memento-note/lib/ai/factory.ts` has provider creation logic — quota checks must integrate BEFORE provider resolution

### Files to CREATE (NEW):
```
memento-note/lib/entitlements.ts       # Core quota check logic
memento-note/lib/usage-tracker.ts     # Track usage via Redis
memento-note/lib/redis.ts            # Redis client singleton
memento-note/app/api/usage/current/route.ts    # GET current quota
memento-note/app/api/cron/sync-usage/route.ts  # CRON sync Redis→PG
memento-note/components/usage-meter.tsx        # UI component
```

### Files to MODIFY (UPDATE):
```
memento-note/prisma/schema.prisma     # Add Subscription, UsageLog models
memento-note/components/sidebar.tsx  # Add UsageMeter to footer
memento-note/middleware.ts           # Add 402 handling for quota
memento-note/app/api/chat/route.ts   # Add canUseFeature() before AI call
```

### Testing Standards

- Unit tests for `canUseFeature()` with mocked Redis
- Unit tests for `trackFeatureUsage()` with Redis pipeline verification
- Integration tests for 402 response flow
- UI tests: verify UsageMeter renders correct progress

---

## Dev Agent Guardrails

### Technical Requirements

- **Redis client:** Use `@upstash/redis` for server-side queries (NOT client-side SDK). Self-hosted Redis via `docker-compose.yml` (see `saas-deployment-prep.md` Section I).
- **NFR-SC2 (< 10ms):** Use Redis GET + pipeline INCRBY — no read-modify-write patterns.
- **Atomic counters:** Always use `redis.pipeline().incrby()` not `GET → compute → SET`.
- **Async tracking:** `trackFeatureUsage()` must be fire-and-forget — do NOT await in the hot path.
- **Period key:** Use `new Date().toISOString().slice(0, 7)` for `YYYY-MM` format — UTC, not local.

### Architecture Compliance

- **Starter Pack limits (from saas-deployment-prep.md):**
  - BASIC: `semanticSearch: 30, autoTag: 20, autoTitle: 10` (lifetime, not monthly)
  - PRO: `semanticSearch: 100, autoTag: 200, autoTitle: 200, reformulate: 50, chat: 100` (monthly)
  - BUSINESS: `semanticSearch: 1000, autoTag: 1000, autoTitle: 1000, reformulate: 500, chat: 1000` (monthly)
- **Redis key format:** `usage:{userId}:{feature}:{YYYY-MM}` with 90-day TTL
- **CRON sync interval:** Every 5 minutes via Vercel Cron or node-cron
- **PostgreSQL sync:** Use UPSERT (INSERT ... ON CONFLICT UPDATE) via Prisma `$upsert()`

### Library / Framework Requirements

- **Redis:** `docker-compose.yml` with `redis:7-alpine` (self-hosted, no Upstash cost)
- **Prisma:** Already in use (`@prisma/client@5.22.0`)
- **React Query:** Already in use (`@tanstack/react-query@5.100.9`) for UsageMeter polling
- **AI SDK:** Already in use (`ai@6.0.23`) for provider calls

### File Structure Requirements

- Follow existing patterns in `memento-note/lib/` — TypeScript files, no default exports
- Use `import { redis } from '@/lib/redis'` singleton pattern (see existing `lib/prisma.ts`)
- API routes follow Next.js App Router: `app/api/[resource]/route.ts`

### Testing Requirements

- Use `vitest` (already configured)
- Mock Redis with `vi.mock('@upstash/redis')`
- No database tests — mock Prisma with `vi.mock('@prisma/client')`
- Target 80% coverage for `entitlements.ts`

---

## Previous Story Intelligence

N/A — This is the first story in Epic 3.

---

## Git Intelligence Summary

**Last 5 commits on modified paths:**

| Commit | Change |
|--------|--------|
| `195e845` | security: fix SQL injection in semantic search - use parameterized queries with bind params |
| `ff664f7` | fix: add missing await on reciprocalRankFusion call |
| `41596c2` | fix: openrouter provider fallback to CUSTOM_OPENAI_API_KEY |
| `cf2786d` | feat: migrate semantic search to pgvector + full-text search |
| `330c0c6` | feat: integrate Google Gemini, MiniMax, and GLM providers |

**Key insight:** Recent commits show security hardening on SQL queries — quota tracking must use parameterized queries for any SQL, and Redis must be used for the fast path.

---

## Latest Technical Information

**Redis Self-Hosted (from `saas-deployment-prep.md`):**
```yaml
# docker-compose.yml
redis:
  image: redis:7-alpine
  command: redis-server --requirepass ${REDIS_PASSWORD} --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
  volumes:
    - redis_data:/data
  ports:
    - "127.0.0.1:6379:6379"
```

**@upstash/redis** package is NOT in package.json — use `ioredis` instead (already a dependency of socket.io):

```typescript
// lib/redis.ts
import Redis from 'ioredis';

const redis = new Redis({
  host: process.env.REDIS_HOST ?? 'localhost',
  port: parseInt(process.env.REDIS_PORT ?? '6379'),
  password: process.env.REDIS_PASSWORD,
  lazyConnect: true,
});

export { redis };
```

**Subscription tiers** defined in `saas-deployment-prep.md` Section B:
- BASIC: free with "AI Discovery Pack" (lifetime limits)
- PRO: €9.90/month (monthly limits)
- BUSINESS: €29.90/month (higher monthly limits)
- ENTERPRISE: €49.90 + €3.90/user/month

---

## Project Context Reference

- **PRD:** `docs/prd.md` — Product Requirements Document with full business context
- **Architecture:** `memento-note/docs/saas-deployment-prep.md` — V3 SaaS architecture including Redis quota system
- **BYOK/Billing:** `memento-note/docs/byok-billing-patch-v3.md` — Full technical spec for quota + BYOK + host-pays
- **UX Spec:** `docs/ux-design-specification.md` — Usage meter in sidebar footer (Section: Emplacement Quotas)
- **Epics:** `docs/epics.md` — Epic 3 with Story 3.1 requirements

---

## Dev Agent Record

### Agent Model Used

Claude Opus 4.7

### Debug Log References

- TypeScript errors fixed: removed type arguments from ioredis `get()` calls (line 105, 147 in entitlements.ts, line 37-38 in usage-tracker.ts)
- Added `@types/ioredis` dev dependency for TypeScript support
- Changed `SubscriptionTier` from Prisma enum import to local type alias to avoid import conflicts

### Completion Notes List

- ✅ Added `Subscription`, `UsageLog`, `FeatureFlag` Prisma models to schema
- ✅ Created `lib/redis.ts` with ioredis client singleton
- ✅ Created `lib/entitlements.ts` with `canUseFeature()`, `checkEntitlementOrThrow()`, `getUserQuotas()`, `QuotaExceededError` class
- ✅ Created `lib/usage-tracker.ts` with fire-and-forget `trackFeatureUsage()` and `getUsageCounter()`
- ✅ Created `/api/usage/current` GET endpoint returning user quotas
- ✅ Created `/api/cron/sync-usage` POST endpoint for Redis → PostgreSQL sync
- ✅ Created `UsageMeter` UI component with React Query 30s polling
- ✅ Integrated `UsageMeter` into sidebar footer
- ✅ Added unit tests for entitlements.ts (9 tests passing)

### File List

**NEW:**
- `memento-note/lib/redis.ts` — Redis client singleton using ioredis
- `memento-note/lib/entitlements.ts` — Core quota check logic with Redis
- `memento-note/lib/usage-tracker.ts` — Fire-and-forget usage tracking
- `memento-note/app/api/usage/current/route.ts` — GET current quota endpoint
- `memento-note/app/api/cron/sync-usage/route.ts` — CRON sync worker
- `memento-note/components/usage-meter.tsx` — UI usage meter component
- `memento-note/tests/unit/entitlements.test.ts` — Unit tests

**MODIFIED:**
- `memento-note/prisma/schema.prisma` — Added Subscription, UsageLog, FeatureFlag models; added `subscription` and `usageLogs` relations to User
- `memento-note/components/sidebar.tsx` — Added UsageMeter import and integration to footer

---

## Story Completion Status

- Status: in-progress
- Completion Note: Re-review patches applied. All decisions resolved, all patches fixed. 17 tests passing.
- Story ID: 3.1
- Story Key: 3-1-freemium-quota-tracking

---

### Review Findings (Re-Review — 2026-05-15)

#### Decision Needed — Resolved

- [x] [Review][Decision → Patch] Quota consommé avant l'appel IA — résolu : passage en async fire-and-forget (`canUseFeature` check-only + `incrementUsageAsync`), conformément à AC5.
- [x] [Review][Decision → Patch] PAST_DUE/CANCELED perdent le tier immédiatement — résolu : grace period via `currentPeriodEnd` dans `getEffectiveTier()`.
- [x] [Review][Decision → Patch] Quota guard uniquement sur chat — résolu : guards ajoutés sur `semantic_search` (semantic-search.ts), `auto_tag` (tags/route.ts), `auto_title` (title-suggestions/route.ts).
- [x] [Review][Decision → Patch] Feature names camelCase vs snake_case — résolu : conversion en snake_case (`semantic_search`, `auto_tag`, `auto_title`, `reformulate`, `chat`).
- [x] [Review][Decision → Resolved] BASIC lifetime vs mensuel — résolu : reset mensuel pour tous les tiers (meilleur pour le marketing, moins frustrant).

#### Patch — Applied

- [x] [Review][Patch] Redis down → 500 sur toutes les features [entitlements.ts] — Fixed : fail-open dans `canUseFeature()` et `getUserQuotas()`, erreurs loggées.
- [x] [Review][Patch] /api/usage/current avale toutes les erreurs en 200 [usage/current/route.ts] — Fixed : retourne 503 avec message d'erreur explicite.
- [x] [Review][Patch] SCAN pattern corrompu par userIds contenant la période [sync-usage/route.ts] — Fixed : parsing robuste via `parseUsageKey()` avec extraction depuis la fin de la clé.
- [x] [Review][Patch] TTL réinitialisé à chaque incrément [entitlements.ts] — Fixed : Lua script vérifie `TTL == -1` avant de set EXPIRE (uniquement sur création).
- [x] [Review][Patch] BASIC features sans limit → message confus [entitlements.ts] — Fixed : nouveau reason `FEATURE_NOT_AVAILABLE` avec message clair et upgradeTier.
- [x] [Review][Patch] parseInt tronque incrbyfloat décimales [quota-utils.ts] — Fixed : `Number()` + `Math.round()` au lieu de `parseInt()`.
- [x] [Review][Patch] Utilisateur supprimé pendant sync → abort tout le batch [sync-usage/route.ts] — Fixed : try/catch individuel par clé, compteur d'erreurs séparé.
- [x] [Review][Patch] usedQuota incorrect dans QuotaExceededError [entitlements.ts] — Fixed : calcul basé sur `result.limit - result.remaining` avec garde.

#### Deferred

- [x] [Review][Defer] Sync CRON N+1 unbatched Prisma upserts — deferred, optimisation perf
- [x] [Review][Defer] Pas de vérification d'autorisation sur conversationId/notebookId — deferred, pré-existant hors scope
- [x] [Review][Defer] Race de période à minuit UTC — deferred, inhérent au billing mensuel
- [x] [Review][Defer] byokConfigured toujours false — deferred, Story 3.5 (BYOK)

---

### Review Findings

#### Decision Needed

- [x] [Review][Decision → Patch] Quota check wired via middleware.ts with AI route filtering — resolved: global interception via middleware with path matching for AI routes. Converts to patch.
- [x] [Review][Decision → Resolved] BASIC lifetime vs monthly key format — resolved: monthly reset accepted for all tiers including BASIC.
- [x] [Review][Decision → Patch] HTTP 402 error handler — resolved: dedicated error handler in middleware that catches QuotaExceededError and returns 402 with required body. Converts to patch.
- [x] [Review][Decision → Patch] Sidebar color changes — resolved: separate into distinct commit. Converts to patch.

#### Patch

- [x] [Review][Patch] Race condition: check-then-increment not atomic [entitlements.ts] — Fixed: atomic Lua script `checkAndConsume()` for check+increment in single Redis call.
- [x] [Review][Patch] Unauthenticated cron endpoint [app/api/cron/sync-usage/route.ts] — Fixed: added `verifyCronAuth()` with Bearer token check matching other cron endpoints.
- [x] [Review][Patch] KEYS command on production Redis [app/api/cron/sync-usage/route.ts] — Fixed: replaced with `scanKeys()` using SCAN cursor-based iteration.
- [x] [Review][Patch] TTL not applied to tokens sub-key [usage-tracker.ts] — Fixed: added `.expire(\`${key}:tokens\`, TTL_SECONDS)` in pipeline.
- [x] [Review][Patch] parseInt NaN on non-numeric Redis values [entitlements.ts] — Fixed: extracted `parseRedisInt()` utility with NaN guard.
- [x] [Review][Patch] getUserQuotas sequential Redis calls [entitlements.ts] — Fixed: replaced loop with `redis.mget()` single round trip.
- [x] [Review][Patch] Subscription status ignored [entitlements.ts] — Fixed: `getEffectiveTier()` checks ACTIVE/TRIALING status, falls back to BASIC for INACTIVE/PAST_DUE/CANCELED.
- [x] [Review][Patch] getUserTier called twice in checkEntitlementOrThrow [entitlements.ts] — Fixed: `checkAndConsume()` returns tier in result, single DB read.
- [x] [Review][Patch] Redis singleton no reconnection after 3 failures [redis.ts] — Fixed: increased max retries to 10, added 30s backoff, error/connect event handlers, global singleton pattern.
- [x] [Review][Patch] Missing paywall modal (Task 5.3) [usage-meter.tsx] — Fixed: added "Upgrade to Pro" modal with pricing, feature list, and CTA button.
- [x] [Review][Patch] UsageMeter hardcodes BASIC limits [usage-meter.tsx] — Fixed: uses `data[f.key].limit` from API response for dynamic tier-aware display.
- [x] [Review][Patch] UI shows "Infinity left" for ENTERPRISE [usage-meter.tsx] — Fixed: `formatRemaining()` returns "Unlimited" for Infinity values, "∞" for individual features.
- [x] [Review][Patch] Duplicate utility functions across 3 files — Fixed: extracted to shared `lib/quota-utils.ts` module.
- [x] [Review][Patch] tokensUsed no negative validation [usage-tracker.ts] — Fixed: early return with warning for negative values.
- [x] [Review][Patch] Quota check wired into chat route [app/api/chat/route.ts] — Fixed: `checkEntitlementOrThrow()` called before AI processing, returns 402 with `QuotaExceededError.toJSON()`.
- [x] [Review][Patch] Sidebar color changes separated — Fixed: reverted unrelated `blueprint` → `memento-terreo` color changes from sidebar.tsx diff.

#### Deferred

- [x] [Review][Defer] Fire-and-forget tracking can lose data [usage-tracker.ts:19-27] — deferred, by design per AC5 (cron sync compensates)
- [x] [Review][Defer] Redis key parsing with colons in userId [sync-usage/route.ts:19-22] — deferred, CUIDs don't contain colons
- [x] [Review][Defer] FeatureFlag model unused [schema.prisma:696-703] — deferred, may be needed for future stories
- [x] [Review][Defer] metadata field untyped String [schema.prisma] — deferred, pre-existing Prisma pattern
- [x] [Review][Defer] Period boundary edge cases — deferred, inherent to monthly billing, standard UTC approach
- [x] [Review][Defer] Subscription requires period fields with no defaults [schema.prisma:664-665] — deferred, Story 3.6 territory
- [x] [Review][Defer] ENTERPRISE tier not in guardrails table — deferred, works as unlimited in TIER_LIMITS

---

## Change Log

| Date | Change |
|------|--------|
| 2026-05-14 | Initial implementation — Added Subscription/UsageLog Prisma models, created entitlements.ts, usage-tracker.ts, redis.ts, API endpoints, UsageMeter UI component, and unit tests |
| 2026-05-15 | Re-review: async fire-and-forget, grace period PAST_DUE/CANCELED, snake_case features, fail-open Redis, quota guards on all AI routes, robust key parsing, TTL on creation only, FEATURE_NOT_AVAILABLE reason, Math.round for floats, individual sync error handling |