feat(quality): A3 — L1 LLM judge via API (5 chunks, 0.0003 USD/job)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s

L1 quality layer — uses a cheap LLM via the OpenAI-compatible API to
validate translation quality. Designed to be the SECOND line of defense
after L0 (script detection, length, pattern).

Architecture:
  - sampler.py — picks 5 representative chunks per job (longest first,
    skips L0-failed indices, skips too-short or identical pairs)
  - llm_judge.py — OpenAI-compatible client, binary verdict per chunk
    (accurate / fluent / correct_language / no_leaks), JSON output,
    hard timeout, defensive (never raises), cost estimation built in
  - pipeline.py — defensive wrapper that integrates both, never breaks
    a translation job, always logs a structured event

Integration:
  - 5 feature flags in config.py (QUALITY_L1_ENABLED, _LOG_ONLY, etc.)
  - QUALITY_L1_LOG_ONLY=true by default: log-only mode, verdict NEVER
    blocks or retries a job
  - Reuses the chunks extracted by L0 (no double work)
  - Passes the set of L0-failed indices so L1 doesn't re-judge them
  - Wrapped in try/except so a misconfigured L1 NEVER breaks a job

Default config: deepseek-chat via DeepSeek API
  - Cost: ~0.0003 USD per job (5 chunks)
  - Speed: typically 1-2s per call, hard ceiling at 8s
  - Easy to swap: just set L1_JUDGE_BASE_URL and L1_JUDGE_MODEL

LLM judge is intentionally a SEPARATE model from the translator
(self-evaluation bias mitigation — Meta/Stanford papers 2024-2025).

Tests:
  test_sampler.py — 9 tests covering the sampling strategy
  test_llm_judge.py — 22 tests covering init, parsing, mocked API,
    cost estimation, env factory
  test_l1_pipeline.py — 6 tests covering the wrapper
  Total new: 37 tests, all pass
  Grand total quality+format: 264 tests passing (0 regression)

  All 36 new tests + 111 L0 tests + 117 existing translator tests = 264

Phase 1 (observation) for 2 weeks. Then QUALITY_L1_LOG_ONLY=false
to enable auto-retry via the fallback chain.
This commit is contained in:
2026-07-14 16:39:47 +02:00
parent 5ae1587428
commit 4d466699fd
10 changed files with 1184 additions and 15 deletions

View File

@@ -1360,28 +1360,70 @@ async def _run_translation_job(
# L0 quality checks. NEVER blocks the job, NEVER modifies the file.
# Enabled by feature flag QUALITY_L0_ENABLED (default: false).
# ------------------------------------------------------------------
quality_samples: list = [] # captured here for L1 to reuse
l0_failed_indices: set = set() # captured for L1 sampling
if getattr(config, "QUALITY_L0_ENABLED", False):
try:
from services.quality import run_l0_check, extract_sample
samples = extract_sample(
quality_samples = extract_sample(
output_path,
file_extension,
max_samples=getattr(config, "QUALITY_L0_SAMPLE_SIZE", 20),
)
translated_chunks = [s["translated"] for s in samples]
run_l0_check(
translated_chunks = [s["translated"] for s in quality_samples]
l0_result = run_l0_check(
source_chunks=[""] * len(translated_chunks), # L0 doesn't need source
translated_chunks=translated_chunks,
target_lang=target_lang,
job_id=job_id,
file_extension=file_extension,
)
# Build the set of indices that L0 flagged as bad
if not l0_result.passed and l0_result.samples:
l0_failed_indices = {s["index"] for s in l0_result.samples}
except Exception as l0_err:
# Quality L0 must NEVER break a job. Log and continue.
logger.warning(
f"Job {job_id}: quality L0 layer failed: {l0_err}"
)
# ------------------------------------------------------------------
# Quality L1 layer (Track A3 — API-based LLM judge, observability first)
# Sends a small sample of chunks to a cheap LLM (deepseek-chat by
# default) and logs a binary verdict (pass/fail). Log-only by
# default — the verdict NEVER blocks a job, NEVER triggers a retry.
# After 2 weeks of observation, set QUALITY_L1_LOG_ONLY=false to
# enable auto-retry.
# Cost: ~$0.0003/job (deepseek) or ~$0.001/job (gpt-4o-mini).
# ------------------------------------------------------------------
if getattr(config, "QUALITY_L1_ENABLED", False) and quality_samples:
try:
from services.quality import run_l1_check
translated_chunks_for_l1 = [s["translated"] for s in quality_samples]
# The samples are extracted from the OUTPUT, not the source —
# we don't have the source here. L1 still works because the
# judge can spot language confusion, gibberish, repetition,
# and prompt leaks without the source. (Source IS used for
# the "accurate" check, so the verdict on that dimension is
# conservative when no source is available — we expect
# the judge to be honest about what it can verify.)
l1_result = await run_l1_check(
source_chunks=[""] * len(translated_chunks_for_l1),
translated_chunks=translated_chunks_for_l1,
target_lang=target_lang,
l0_failed_indices=l0_failed_indices,
job_id=job_id,
file_extension=file_extension,
max_samples=getattr(config, "QUALITY_L1_SAMPLE_SIZE", 5),
min_chunks=getattr(config, "QUALITY_L1_MIN_CHUNKS", 10),
log_only=getattr(config, "QUALITY_L1_LOG_ONLY", True),
)
except Exception as l1_err:
# L1 must NEVER break a job. Log and continue.
logger.warning(
f"Job {job_id}: quality L1 layer failed: {l1_err}"
)
if user_id:
# Determine cost factor based on selected provider and model
cost_factor = 1