Momento/docs/3-1-freemium-quota-tracking.md

Story 3.1: Freemium "AI Discovery Pack" Quota Tracking

Status: review

---

Story

As a business, I want to track Freemium usage against a Redis quota limit, So that I can limit my API cost exposure for free users.

Given a free user triggers an AI request When the system intercepts the request Then the quota is tracked and the UI updates And (NFR-SC2) the Redis-backed check resolves in under 10ms.

---

Epic Context

Epic 3: The SaaS Commercial Engine (Monetization & API Cost Protection)

Epic Business Value: The core backend logic allowing us to sell the product without bleeding API costs — freemium limits, router fallback, host-pays.

Cross-Story Dependencies:

• Story 3.1 (this story) establishes the quota tracking foundation
• Story 3.2 builds on this with the LLM Router and provider routing
• Story 3.3 adds smart-routing fallback when quota is low
• Story 3.4 (Host-Pays) extends quota tracking to collaborative sessions
• Story 3.5 (BYOK) bypasses quota for users with their own API keys
• Story 3.6 (Stripe) manages tier upgrades

Technical Constraint: (NFR-SC2) Redis-backed entitlement checks must complete in under 10ms.

---

Acceptance Criteria

1. [AC1] When a free user (BASIC tier) makes an AI request, the system checks Redis for current usage count
2. [AC2] Each AI feature (semantic_search, auto_tag, auto_title) has its own Redis counter with format: usage:{userId}:{feature}:{YYYY-MM}
3. [AC3] Counter increments atomically via Redis INCRBY (not read-modify-write) to avoid race conditions
4. [AC4] If counter >= limit, return HTTP 402 with body { error: "QUOTA_EXCEEDED", feature, upgradeTier: "PRO", byokConfigured }
5. [AC5] If counter < limit, allow request to proceed and increment counter asynchronously (fire-and-forget)
6. [AC6] Redis keys have 90-day TTL to auto-cleanup (covers grace period for monthly reconciliation)
7. [AC7] The Sidebar footer displays a usage gauge component showing Discovery Pack consumption in real-time

---

Tasks / Subtasks

• Task 1: Add Prisma models for Subscription, UsageLog (AC: #1)
  • Subtask 1.1: Add Subscription model with tier/stripe fields
  • Subtask 1.2: Add UsageLog model for PostgreSQL sync target
  • Subtask 1.3: Create migration file (via npx prisma generate)
• Task 2: Create lib/entitlements.ts with Redis-backed canUseFeature() (AC: #2, #3, #4)
  • Subtask 2.1: Implement getCurrentPeriodKey() returning YYYY-MM format
  • Subtask 2.2: Implement Redis key format: usage:{userId}:{feature}:{YYYY-MM}
  • Subtask 2.3: Implement atomic canUseFeature() with < 10ms target (use Redis GET + pipeline INCRBY)
  • Subtask 2.4: Return QuotaExceededError when limit exceeded
• Task 3: Create lib/usage-tracker.ts with trackFeatureUsage() (AC: #5)
  • Subtask 3.1: Fire-and-forget Redis pipeline increment
  • Subtask 3.2: Include tokensUsed in metadata
• Task 4: Create /api/usage/current endpoint (AC: #6)
  • Subtask 4.1: Return remaining quota for all features for authenticated user
• Task 5: Create <UsageMeter> UI component for Sidebar footer (AC: #7)
  • Subtask 5.1: Show progress bar for Discovery Pack (semantic_search: 30 lifetime, auto_tag: 20, auto_title: 10)
  • Subtask 5.2: Real-time updates via React Query polling every 30s
  • Subtask 5.3: Show "Upgrade to Pro" paywall modal on 402 response
• Task 6: Create CRON sync worker /api/cron/sync-usage (AC: #6)
  • Subtask 6.1: Batch sync Redis counters → PostgreSQL UsageLog
  • Subtask 6.2: Handle monthly reset (new period = reset counters for that user/feature)

---

Dev Notes

Project Structure Notes

• Code base: memento-note/ (Next.js app with App Router)
• Redis client: Currently memento-note/lib/rate-limit.ts uses in-memory Map (INSUFFICIENT). This story replaces it with actual Redis.
• No existing lib/entitlements.ts — creates from scratch
• No existing lib/usage-tracker.ts — creates from scratch
• Sidebar: memento-note/components/sidebar.tsx is the anchor for the UsageMeter UI
• AI Factory: memento-note/lib/ai/factory.ts has provider creation logic — quota checks must integrate BEFORE provider resolution

Files to CREATE (NEW):

memento-note/lib/entitlements.ts       # Core quota check logic
memento-note/lib/usage-tracker.ts     # Track usage via Redis
memento-note/lib/redis.ts            # Redis client singleton
memento-note/app/api/usage/current/route.ts    # GET current quota
memento-note/app/api/cron/sync-usage/route.ts  # CRON sync Redis→PG
memento-note/components/usage-meter.tsx        # UI component

Files to MODIFY (UPDATE):

memento-note/prisma/schema.prisma     # Add Subscription, UsageLog models
memento-note/components/sidebar.tsx  # Add UsageMeter to footer
memento-note/middleware.ts           # Add 402 handling for quota
memento-note/app/api/chat/route.ts   # Add canUseFeature() before AI call

Testing Standards

• Unit tests for canUseFeature() with mocked Redis
• Unit tests for trackFeatureUsage() with Redis pipeline verification
• Integration tests for 402 response flow
• UI tests: verify UsageMeter renders correct progress

---

Dev Agent Guardrails

Technical Requirements

• Redis client: Use @upstash/redis for server-side queries (NOT client-side SDK). Self-hosted Redis via docker-compose.yml (see saas-deployment-prep.md Section I).
• NFR-SC2 (< 10ms): Use Redis GET + pipeline INCRBY — no read-modify-write patterns.
• Atomic counters: Always use redis.pipeline().incrby() not GET → compute → SET.
• Async tracking: trackFeatureUsage() must be fire-and-forget — do NOT await in the hot path.
• Period key: Use new Date().toISOString().slice(0, 7) for YYYY-MM format — UTC, not local.

Architecture Compliance

• Starter Pack limits (from saas-deployment-prep.md):
  • BASIC: semanticSearch: 30, autoTag: 20, autoTitle: 10 (lifetime, not monthly)
  • PRO: semanticSearch: 100, autoTag: 200, autoTitle: 200, reformulate: 50, chat: 100 (monthly)
  • BUSINESS: semanticSearch: 1000, autoTag: 1000, autoTitle: 1000, reformulate: 500, chat: 1000 (monthly)
• Redis key format: usage:{userId}:{feature}:{YYYY-MM} with 90-day TTL
• CRON sync interval: Every 5 minutes via Vercel Cron or node-cron
• PostgreSQL sync: Use UPSERT (INSERT ... ON CONFLICT UPDATE) via Prisma $upsert()

Library / Framework Requirements

• Redis: docker-compose.yml with redis:7-alpine (self-hosted, no Upstash cost)
• Prisma: Already in use (@prisma/client@5.22.0)
• React Query: Already in use (@tanstack/react-query@5.100.9) for UsageMeter polling
• AI SDK: Already in use (ai@6.0.23) for provider calls

File Structure Requirements

• Follow existing patterns in memento-note/lib/ — TypeScript files, no default exports
• Use import { redis } from '@/lib/redis' singleton pattern (see existing lib/prisma.ts)
• API routes follow Next.js App Router: app/api/[resource]/route.ts

Testing Requirements

• Use vitest (already configured)
• Mock Redis with vi.mock('@upstash/redis')
• No database tests — mock Prisma with vi.mock('@prisma/client')
• Target 80% coverage for entitlements.ts

---

Previous Story Intelligence

N/A — This is the first story in Epic 3.

---

Git Intelligence Summary

Last 5 commits on modified paths:

Commit	Change
195e845	security: fix SQL injection in semantic search - use parameterized queries with bind params
ff664f7	fix: add missing await on reciprocalRankFusion call
41596c2	fix: openrouter provider fallback to CUSTOM_OPENAI_API_KEY
cf2786d	feat: migrate semantic search to pgvector + full-text search
330c0c6	feat: integrate Google Gemini, MiniMax, and GLM providers

Key insight: Recent commits show security hardening on SQL queries — quota tracking must use parameterized queries for any SQL, and Redis must be used for the fast path.

---

Latest Technical Information

Redis Self-Hosted (from saas-deployment-prep.md):

# docker-compose.yml
redis:
  image: redis:7-alpine
  command: redis-server --requirepass ${REDIS_PASSWORD} --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
  volumes:
    - redis_data:/data
  ports:
    - "127.0.0.1:6379:6379"

@upstash/redis package is NOT in package.json — use ioredis instead (already a dependency of socket.io):

// lib/redis.ts
import Redis from 'ioredis';

const redis = new Redis({
  host: process.env.REDIS_HOST ?? 'localhost',
  port: parseInt(process.env.REDIS_PORT ?? '6379'),
  password: process.env.REDIS_PASSWORD,
  lazyConnect: true,
});

export { redis };

Subscription tiers defined in saas-deployment-prep.md Section B:

• BASIC: free with "AI Discovery Pack" (lifetime limits)
• PRO: €9.90/month (monthly limits)
• BUSINESS: €29.90/month (higher monthly limits)
• ENTERPRISE: €49.90 + €3.90/user/month

---

Project Context Reference

• PRD: docs/prd.md — Product Requirements Document with full business context
• Architecture: memento-note/docs/saas-deployment-prep.md — V3 SaaS architecture including Redis quota system
• BYOK/Billing: memento-note/docs/byok-billing-patch-v3.md — Full technical spec for quota + BYOK + host-pays
• UX Spec: docs/ux-design-specification.md — Usage meter in sidebar footer (Section: Emplacement Quotas)
• Epics: docs/epics.md — Epic 3 with Story 3.1 requirements

---

Dev Agent Record

Agent Model Used

Claude Opus 4.7

Debug Log References

• TypeScript errors fixed: removed type arguments from ioredis get() calls (line 105, 147 in entitlements.ts, line 37-38 in usage-tracker.ts)
• Added @types/ioredis dev dependency for TypeScript support
• Changed SubscriptionTier from Prisma enum import to local type alias to avoid import conflicts

Completion Notes List

• ✅ Added Subscription, UsageLog, FeatureFlag Prisma models to schema
• ✅ Created lib/redis.ts with ioredis client singleton
• ✅ Created lib/entitlements.ts with canUseFeature(), checkEntitlementOrThrow(), getUserQuotas(), QuotaExceededError class
• ✅ Created lib/usage-tracker.ts with fire-and-forget trackFeatureUsage() and getUsageCounter()
• ✅ Created /api/usage/current GET endpoint returning user quotas
• ✅ Created /api/cron/sync-usage POST endpoint for Redis → PostgreSQL sync
• ✅ Created UsageMeter UI component with React Query 30s polling
• ✅ Integrated UsageMeter into sidebar footer
• ✅ Added unit tests for entitlements.ts (9 tests passing)

File List

NEW:

• memento-note/lib/redis.ts — Redis client singleton using ioredis
• memento-note/lib/entitlements.ts — Core quota check logic with Redis
• memento-note/lib/usage-tracker.ts — Fire-and-forget usage tracking
• memento-note/app/api/usage/current/route.ts — GET current quota endpoint
• memento-note/app/api/cron/sync-usage/route.ts — CRON sync worker
• memento-note/components/usage-meter.tsx — UI usage meter component
• memento-note/tests/unit/entitlements.test.ts — Unit tests

MODIFIED:

• memento-note/prisma/schema.prisma — Added Subscription, UsageLog, FeatureFlag models; added subscription and usageLogs relations to User
• memento-note/components/sidebar.tsx — Added UsageMeter import and integration to footer

---

Story Completion Status

• Status: in-progress
• Completion Note: Re-review patches applied. All decisions resolved, all patches fixed. 17 tests passing.
• Story ID: 3.1
• Story Key: 3-1-freemium-quota-tracking

---

Review Findings (Re-Review — 2026-05-15)

Decision Needed — Resolved

• [Review][Decision → Patch] Quota consommé avant l'appel IA — résolu : passage en async fire-and-forget (canUseFeature check-only + incrementUsageAsync), conformément à AC5.
• [Review][Decision → Patch] PAST_DUE/CANCELED perdent le tier immédiatement — résolu : grace period via currentPeriodEnd dans getEffectiveTier().
• [Review][Decision → Patch] Quota guard uniquement sur chat — résolu : guards ajoutés sur semantic_search (semantic-search.ts), auto_tag (tags/route.ts), auto_title (title-suggestions/route.ts).
• [Review][Decision → Patch] Feature names camelCase vs snake_case — résolu : conversion en snake_case (semantic_search, auto_tag, auto_title, reformulate, chat).
• [Review][Decision → Resolved] BASIC lifetime vs mensuel — résolu : reset mensuel pour tous les tiers (meilleur pour le marketing, moins frustrant).

Patch — Applied

• [Review][Patch] Redis down → 500 sur toutes les features [entitlements.ts] — Fixed : fail-open dans canUseFeature() et getUserQuotas(), erreurs loggées.
• [Review][Patch] /api/usage/current avale toutes les erreurs en 200 [usage/current/route.ts] — Fixed : retourne 503 avec message d'erreur explicite.
• [Review][Patch] SCAN pattern corrompu par userIds contenant la période [sync-usage/route.ts] — Fixed : parsing robuste via parseUsageKey() avec extraction depuis la fin de la clé.
• [Review][Patch] TTL réinitialisé à chaque incrément [entitlements.ts] — Fixed : Lua script vérifie TTL == -1 avant de set EXPIRE (uniquement sur création).
• [Review][Patch] BASIC features sans limit → message confus [entitlements.ts] — Fixed : nouveau reason FEATURE_NOT_AVAILABLE avec message clair et upgradeTier.
• [Review][Patch] parseInt tronque incrbyfloat décimales [quota-utils.ts] — Fixed : Number() + Math.round() au lieu de parseInt().
• [Review][Patch] Utilisateur supprimé pendant sync → abort tout le batch [sync-usage/route.ts] — Fixed : try/catch individuel par clé, compteur d'erreurs séparé.
• [Review][Patch] usedQuota incorrect dans QuotaExceededError [entitlements.ts] — Fixed : calcul basé sur result.limit - result.remaining avec garde.

Deferred

• [Review][Defer] Sync CRON N+1 unbatched Prisma upserts — deferred, optimisation perf
• [Review][Defer] Pas de vérification d'autorisation sur conversationId/notebookId — deferred, pré-existant hors scope
• [Review][Defer] Race de période à minuit UTC — deferred, inhérent au billing mensuel
• [Review][Defer] byokConfigured toujours false — deferred, Story 3.5 (BYOK)

---

Review Findings

Decision Needed

• [Review][Decision → Patch] Quota check wired via middleware.ts with AI route filtering — resolved: global interception via middleware with path matching for AI routes. Converts to patch.
• [Review][Decision → Resolved] BASIC lifetime vs monthly key format — resolved: monthly reset accepted for all tiers including BASIC.
• [Review][Decision → Patch] HTTP 402 error handler — resolved: dedicated error handler in middleware that catches QuotaExceededError and returns 402 with required body. Converts to patch.
• [Review][Decision → Patch] Sidebar color changes — resolved: separate into distinct commit. Converts to patch.

Patch

• [Review][Patch] Race condition: check-then-increment not atomic [entitlements.ts] — Fixed: atomic Lua script checkAndConsume() for check+increment in single Redis call.
• [Review][Patch] Unauthenticated cron endpoint [app/api/cron/sync-usage/route.ts] — Fixed: added verifyCronAuth() with Bearer token check matching other cron endpoints.
• [Review][Patch] KEYS command on production Redis [app/api/cron/sync-usage/route.ts] — Fixed: replaced with scanKeys() using SCAN cursor-based iteration.
• [Review][Patch] TTL not applied to tokens sub-key [usage-tracker.ts] — Fixed: added .expire(\${key}:tokens`, TTL_SECONDS)` in pipeline.
• [Review][Patch] parseInt NaN on non-numeric Redis values [entitlements.ts] — Fixed: extracted parseRedisInt() utility with NaN guard.
• [Review][Patch] getUserQuotas sequential Redis calls [entitlements.ts] — Fixed: replaced loop with redis.mget() single round trip.
• [Review][Patch] Subscription status ignored [entitlements.ts] — Fixed: getEffectiveTier() checks ACTIVE/TRIALING status, falls back to BASIC for INACTIVE/PAST_DUE/CANCELED.
• [Review][Patch] getUserTier called twice in checkEntitlementOrThrow [entitlements.ts] — Fixed: checkAndConsume() returns tier in result, single DB read.
• [Review][Patch] Redis singleton no reconnection after 3 failures [redis.ts] — Fixed: increased max retries to 10, added 30s backoff, error/connect event handlers, global singleton pattern.
• [Review][Patch] Missing paywall modal (Task 5.3) [usage-meter.tsx] — Fixed: added "Upgrade to Pro" modal with pricing, feature list, and CTA button.
• [Review][Patch] UsageMeter hardcodes BASIC limits [usage-meter.tsx] — Fixed: uses data[f.key].limit from API response for dynamic tier-aware display.
• [Review][Patch] UI shows "Infinity left" for ENTERPRISE [usage-meter.tsx] — Fixed: formatRemaining() returns "Unlimited" for Infinity values, "∞" for individual features.
• [Review][Patch] Duplicate utility functions across 3 files — Fixed: extracted to shared lib/quota-utils.ts module.
• [Review][Patch] tokensUsed no negative validation [usage-tracker.ts] — Fixed: early return with warning for negative values.
• [Review][Patch] Quota check wired into chat route [app/api/chat/route.ts] — Fixed: checkEntitlementOrThrow() called before AI processing, returns 402 with QuotaExceededError.toJSON().
• [Review][Patch] Sidebar color changes separated — Fixed: reverted unrelated blueprint → memento-terreo color changes from sidebar.tsx diff.

Deferred

• [Review][Defer] Fire-and-forget tracking can lose data [usage-tracker.ts:19-27] — deferred, by design per AC5 (cron sync compensates)
• [Review][Defer] Redis key parsing with colons in userId [sync-usage/route.ts:19-22] — deferred, CUIDs don't contain colons
• [Review][Defer] FeatureFlag model unused [schema.prisma:696-703] — deferred, may be needed for future stories
• [Review][Defer] metadata field untyped String [schema.prisma] — deferred, pre-existing Prisma pattern
• [Review][Defer] Period boundary edge cases — deferred, inherent to monthly billing, standard UTC approach
• [Review][Defer] Subscription requires period fields with no defaults [schema.prisma:664-665] — deferred, Story 3.6 territory
• [Review][Defer] ENTERPRISE tier not in guardrails table — deferred, works as unlimited in TIER_LIMITS

---

Change Log

Date	Change
2026-05-14	Initial implementation — Added Subscription/UsageLog Prisma models, created entitlements.ts, usage-tracker.ts, redis.ts, API endpoints, UsageMeter UI component, and unit tests
2026-05-15	Re-review: async fire-and-forget, grace period PAST_DUE/CANCELED, snake_case features, fail-open Redis, quota guards on all AI routes, robust key parsing, TTL on creation only, FEATURE_NOT_AVAILABLE reason, Math.round for floats, individual sync error handling