feat(quality): A3 — L1 LLM judge via API (5 chunks, 0.0003 USD/job)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s
L1 quality layer — uses a cheap LLM via the OpenAI-compatible API to
validate translation quality. Designed to be the SECOND line of defense
after L0 (script detection, length, pattern).
Architecture:
- sampler.py — picks 5 representative chunks per job (longest first,
skips L0-failed indices, skips too-short or identical pairs)
- llm_judge.py — OpenAI-compatible client, binary verdict per chunk
(accurate / fluent / correct_language / no_leaks), JSON output,
hard timeout, defensive (never raises), cost estimation built in
- pipeline.py — defensive wrapper that integrates both, never breaks
a translation job, always logs a structured event
Integration:
- 5 feature flags in config.py (QUALITY_L1_ENABLED, _LOG_ONLY, etc.)
- QUALITY_L1_LOG_ONLY=true by default: log-only mode, verdict NEVER
blocks or retries a job
- Reuses the chunks extracted by L0 (no double work)
- Passes the set of L0-failed indices so L1 doesn't re-judge them
- Wrapped in try/except so a misconfigured L1 NEVER breaks a job
Default config: deepseek-chat via DeepSeek API
- Cost: ~0.0003 USD per job (5 chunks)
- Speed: typically 1-2s per call, hard ceiling at 8s
- Easy to swap: just set L1_JUDGE_BASE_URL and L1_JUDGE_MODEL
LLM judge is intentionally a SEPARATE model from the translator
(self-evaluation bias mitigation — Meta/Stanford papers 2024-2025).
Tests:
test_sampler.py — 9 tests covering the sampling strategy
test_llm_judge.py — 22 tests covering init, parsing, mocked API,
cost estimation, env factory
test_l1_pipeline.py — 6 tests covering the wrapper
Total new: 37 tests, all pass
Grand total quality+format: 264 tests passing (0 regression)
All 36 new tests + 111 L0 tests + 117 existing translator tests = 264
Phase 1 (observation) for 2 weeks. Then QUALITY_L1_LOG_ONLY=false
to enable auto-retry via the fallback chain.
This commit is contained in:
15
config.py
15
config.py
@@ -77,6 +77,21 @@ class Config:
|
||||
QUALITY_L0_SAMPLE_SIZE = int(os.getenv("QUALITY_L0_SAMPLE_SIZE", "20"))
|
||||
|
||||
|
||||
# ============== Quality Layer (L1) ==============
|
||||
# Track A3 of the dev plan — API-based LLM judge.
|
||||
# Observability first; the verdict is logged but never used to retry
|
||||
# or block a job (QUALITY_L1_LOG_ONLY=true). After 2 weeks of
|
||||
# monitoring, set QUALITY_L1_LOG_ONLY=false to enable auto-retry.
|
||||
QUALITY_L1_ENABLED = os.getenv("QUALITY_L1_ENABLED", "false").lower() == "true"
|
||||
QUALITY_L1_LOG_ONLY = os.getenv("QUALITY_L1_LOG_ONLY", "true").lower() == "true"
|
||||
# Number of chunks sampled per job. 5 is the sweet spot (cost vs coverage).
|
||||
QUALITY_L1_SAMPLE_SIZE = int(os.getenv("QUALITY_L1_SAMPLE_SIZE", "5"))
|
||||
# Skip the check if the document has fewer than this many chunks.
|
||||
QUALITY_L1_MIN_CHUNKS = int(os.getenv("QUALITY_L1_MIN_CHUNKS", "10"))
|
||||
# Hard ceiling on the L1 call (seconds). Anything longer is a skip.
|
||||
QUALITY_L1_TIMEOUT_SEC = float(os.getenv("QUALITY_L1_TIMEOUT_SEC", "8.0"))
|
||||
|
||||
|
||||
# ============== API Configuration ==============
|
||||
API_TITLE = "Document Translation API"
|
||||
API_VERSION = "1.0.0"
|
||||
|
||||
Reference in New Issue
Block a user