feat(quality): A3 — L1 LLM judge via API (5 chunks, 0.0003 USD/job)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s
L1 quality layer — uses a cheap LLM via the OpenAI-compatible API to
validate translation quality. Designed to be the SECOND line of defense
after L0 (script detection, length, pattern).
Architecture:
- sampler.py — picks 5 representative chunks per job (longest first,
skips L0-failed indices, skips too-short or identical pairs)
- llm_judge.py — OpenAI-compatible client, binary verdict per chunk
(accurate / fluent / correct_language / no_leaks), JSON output,
hard timeout, defensive (never raises), cost estimation built in
- pipeline.py — defensive wrapper that integrates both, never breaks
a translation job, always logs a structured event
Integration:
- 5 feature flags in config.py (QUALITY_L1_ENABLED, _LOG_ONLY, etc.)
- QUALITY_L1_LOG_ONLY=true by default: log-only mode, verdict NEVER
blocks or retries a job
- Reuses the chunks extracted by L0 (no double work)
- Passes the set of L0-failed indices so L1 doesn't re-judge them
- Wrapped in try/except so a misconfigured L1 NEVER breaks a job
Default config: deepseek-chat via DeepSeek API
- Cost: ~0.0003 USD per job (5 chunks)
- Speed: typically 1-2s per call, hard ceiling at 8s
- Easy to swap: just set L1_JUDGE_BASE_URL and L1_JUDGE_MODEL
LLM judge is intentionally a SEPARATE model from the translator
(self-evaluation bias mitigation — Meta/Stanford papers 2024-2025).
Tests:
test_sampler.py — 9 tests covering the sampling strategy
test_llm_judge.py — 22 tests covering init, parsing, mocked API,
cost estimation, env factory
test_l1_pipeline.py — 6 tests covering the wrapper
Total new: 37 tests, all pass
Grand total quality+format: 264 tests passing (0 regression)
All 36 new tests + 111 L0 tests + 117 existing translator tests = 264
Phase 1 (observation) for 2 weeks. Then QUALITY_L1_LOG_ONLY=false
to enable auto-retry via the fallback chain.
This commit is contained in:
@@ -1360,28 +1360,70 @@ async def _run_translation_job(
|
||||
# L0 quality checks. NEVER blocks the job, NEVER modifies the file.
|
||||
# Enabled by feature flag QUALITY_L0_ENABLED (default: false).
|
||||
# ------------------------------------------------------------------
|
||||
quality_samples: list = [] # captured here for L1 to reuse
|
||||
l0_failed_indices: set = set() # captured for L1 sampling
|
||||
if getattr(config, "QUALITY_L0_ENABLED", False):
|
||||
try:
|
||||
from services.quality import run_l0_check, extract_sample
|
||||
samples = extract_sample(
|
||||
quality_samples = extract_sample(
|
||||
output_path,
|
||||
file_extension,
|
||||
max_samples=getattr(config, "QUALITY_L0_SAMPLE_SIZE", 20),
|
||||
)
|
||||
translated_chunks = [s["translated"] for s in samples]
|
||||
run_l0_check(
|
||||
translated_chunks = [s["translated"] for s in quality_samples]
|
||||
l0_result = run_l0_check(
|
||||
source_chunks=[""] * len(translated_chunks), # L0 doesn't need source
|
||||
translated_chunks=translated_chunks,
|
||||
target_lang=target_lang,
|
||||
job_id=job_id,
|
||||
file_extension=file_extension,
|
||||
)
|
||||
# Build the set of indices that L0 flagged as bad
|
||||
if not l0_result.passed and l0_result.samples:
|
||||
l0_failed_indices = {s["index"] for s in l0_result.samples}
|
||||
except Exception as l0_err:
|
||||
# Quality L0 must NEVER break a job. Log and continue.
|
||||
logger.warning(
|
||||
f"Job {job_id}: quality L0 layer failed: {l0_err}"
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Quality L1 layer (Track A3 — API-based LLM judge, observability first)
|
||||
# Sends a small sample of chunks to a cheap LLM (deepseek-chat by
|
||||
# default) and logs a binary verdict (pass/fail). Log-only by
|
||||
# default — the verdict NEVER blocks a job, NEVER triggers a retry.
|
||||
# After 2 weeks of observation, set QUALITY_L1_LOG_ONLY=false to
|
||||
# enable auto-retry.
|
||||
# Cost: ~$0.0003/job (deepseek) or ~$0.001/job (gpt-4o-mini).
|
||||
# ------------------------------------------------------------------
|
||||
if getattr(config, "QUALITY_L1_ENABLED", False) and quality_samples:
|
||||
try:
|
||||
from services.quality import run_l1_check
|
||||
translated_chunks_for_l1 = [s["translated"] for s in quality_samples]
|
||||
# The samples are extracted from the OUTPUT, not the source —
|
||||
# we don't have the source here. L1 still works because the
|
||||
# judge can spot language confusion, gibberish, repetition,
|
||||
# and prompt leaks without the source. (Source IS used for
|
||||
# the "accurate" check, so the verdict on that dimension is
|
||||
# conservative when no source is available — we expect
|
||||
# the judge to be honest about what it can verify.)
|
||||
l1_result = await run_l1_check(
|
||||
source_chunks=[""] * len(translated_chunks_for_l1),
|
||||
translated_chunks=translated_chunks_for_l1,
|
||||
target_lang=target_lang,
|
||||
l0_failed_indices=l0_failed_indices,
|
||||
job_id=job_id,
|
||||
file_extension=file_extension,
|
||||
max_samples=getattr(config, "QUALITY_L1_SAMPLE_SIZE", 5),
|
||||
min_chunks=getattr(config, "QUALITY_L1_MIN_CHUNKS", 10),
|
||||
log_only=getattr(config, "QUALITY_L1_LOG_ONLY", True),
|
||||
)
|
||||
except Exception as l1_err:
|
||||
# L1 must NEVER break a job. Log and continue.
|
||||
logger.warning(
|
||||
f"Job {job_id}: quality L1 layer failed: {l1_err}"
|
||||
)
|
||||
|
||||
if user_id:
|
||||
# Determine cost factor based on selected provider and model
|
||||
cost_factor = 1
|
||||
|
||||
Reference in New Issue
Block a user