feat(quality): A3 — L1 LLM judge via API (5 chunks, 0.0003 USD/job)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s

L1 quality layer — uses a cheap LLM via the OpenAI-compatible API to
validate translation quality. Designed to be the SECOND line of defense
after L0 (script detection, length, pattern).

Architecture:
  - sampler.py — picks 5 representative chunks per job (longest first,
    skips L0-failed indices, skips too-short or identical pairs)
  - llm_judge.py — OpenAI-compatible client, binary verdict per chunk
    (accurate / fluent / correct_language / no_leaks), JSON output,
    hard timeout, defensive (never raises), cost estimation built in
  - pipeline.py — defensive wrapper that integrates both, never breaks
    a translation job, always logs a structured event

Integration:
  - 5 feature flags in config.py (QUALITY_L1_ENABLED, _LOG_ONLY, etc.)
  - QUALITY_L1_LOG_ONLY=true by default: log-only mode, verdict NEVER
    blocks or retries a job
  - Reuses the chunks extracted by L0 (no double work)
  - Passes the set of L0-failed indices so L1 doesn't re-judge them
  - Wrapped in try/except so a misconfigured L1 NEVER breaks a job

Default config: deepseek-chat via DeepSeek API
  - Cost: ~0.0003 USD per job (5 chunks)
  - Speed: typically 1-2s per call, hard ceiling at 8s
  - Easy to swap: just set L1_JUDGE_BASE_URL and L1_JUDGE_MODEL

LLM judge is intentionally a SEPARATE model from the translator
(self-evaluation bias mitigation — Meta/Stanford papers 2024-2025).

Tests:
  test_sampler.py — 9 tests covering the sampling strategy
  test_llm_judge.py — 22 tests covering init, parsing, mocked API,
    cost estimation, env factory
  test_l1_pipeline.py — 6 tests covering the wrapper
  Total new: 37 tests, all pass
  Grand total quality+format: 264 tests passing (0 regression)

  All 36 new tests + 111 L0 tests + 117 existing translator tests = 264

Phase 1 (observation) for 2 weeks. Then QUALITY_L1_LOG_ONLY=false
to enable auto-retry via the fallback chain.
This commit is contained in:
2026-07-14 16:39:47 +02:00
parent 5ae1587428
commit 4d466699fd
10 changed files with 1184 additions and 15 deletions

View File

@@ -104,6 +104,30 @@ MAX_CONCURRENT_TRANSLATIONS=5
QUALITY_L0_ENABLED=false
QUALITY_L0_SAMPLE_SIZE=20
# ============== Quality Layer (L1) ==============
# Track A3 of the dev plan — API-based LLM judge.
# Sends 5 sampled chunks per job to a cheap LLM (deepseek-chat by default)
# and logs the verdict (pass/fail) but does NOT modify the file or job
# status. Log-only by default for the first 2 weeks of observation.
# After that, set QUALITY_L1_LOG_ONLY=false to enable auto-retry.
#
# Cost: ~$0.0003 per job with deepseek-chat. ~$0.001 with gpt-4o-mini.
QUALITY_L1_ENABLED=false
QUALITY_L1_LOG_ONLY=true
QUALITY_L1_SAMPLE_SIZE=5
QUALITY_L1_MIN_CHUNKS=10
QUALITY_L1_TIMEOUT_SEC=8.0
# L1 judge configuration (any OpenAI-compatible endpoint).
# DeepSeek is the default (cheapest, ~$0.14/M input tokens).
L1_JUDGE_API_KEY=
L1_JUDGE_BASE_URL=https://api.deepseek.com/v1
L1_JUDGE_MODEL=deepseek-chat
# L1_JUDGE_MODEL=gpt-4o-mini
# L1_JUDGE_BASE_URL=https://api.openai.com/v1
# L1_JUDGE_MODEL=google/gemini-2.5-flash-lite
# L1_JUDGE_BASE_URL=https://openrouter.ai/api/v1
# ============== Cleanup Service ==============
# Enable automatic file cleanup
CLEANUP_ENABLED=true