sepehr
4d466699fd
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m26s
L1 quality layer — uses a cheap LLM via the OpenAI-compatible API to
validate translation quality. Designed to be the SECOND line of defense
after L0 (script detection, length, pattern).
Architecture:
- sampler.py — picks 5 representative chunks per job (longest first,
skips L0-failed indices, skips too-short or identical pairs)
- llm_judge.py — OpenAI-compatible client, binary verdict per chunk
(accurate / fluent / correct_language / no_leaks), JSON output,
hard timeout, defensive (never raises), cost estimation built in
- pipeline.py — defensive wrapper that integrates both, never breaks
a translation job, always logs a structured event
Integration:
- 5 feature flags in config.py (QUALITY_L1_ENABLED, _LOG_ONLY, etc.)
- QUALITY_L1_LOG_ONLY=true by default: log-only mode, verdict NEVER
blocks or retries a job
- Reuses the chunks extracted by L0 (no double work)
- Passes the set of L0-failed indices so L1 doesn't re-judge them
- Wrapped in try/except so a misconfigured L1 NEVER breaks a job
Default config: deepseek-chat via DeepSeek API
- Cost: ~0.0003 USD per job (5 chunks)
- Speed: typically 1-2s per call, hard ceiling at 8s
- Easy to swap: just set L1_JUDGE_BASE_URL and L1_JUDGE_MODEL
LLM judge is intentionally a SEPARATE model from the translator
(self-evaluation bias mitigation — Meta/Stanford papers 2024-2025).
Tests:
test_sampler.py — 9 tests covering the sampling strategy
test_llm_judge.py — 22 tests covering init, parsing, mocked API,
cost estimation, env factory
test_l1_pipeline.py — 6 tests covering the wrapper
Total new: 37 tests, all pass
Grand total quality+format: 264 tests passing (0 regression)
All 36 new tests + 111 L0 tests + 117 existing translator tests = 264
Phase 1 (observation) for 2 weeks. Then QUALITY_L1_LOG_ONLY=false
to enable auto-retry via the fallback chain.
56 lines
1.6 KiB
Python
56 lines
1.6 KiB
Python
"""
|
|
Quality check layer for translations.
|
|
|
|
Tracks:
|
|
A1 — L0 backend (pure Python, no API calls). Always available.
|
|
A2 — L0 frontend JavaScript mirror. Browser/Node compatible.
|
|
A3 — L1 LLM judge (API-based, opt-in). Cheap model, sampled.
|
|
|
|
Designed to be ADDITIVE: existing translation flow is untouched.
|
|
|
|
Public API:
|
|
L0:
|
|
QualityCheckResult — per-chunk result dataclass
|
|
DocumentQualityResult — aggregated result dataclass
|
|
evaluate_chunk(...) — score a single (source, translation) pair
|
|
evaluate_document(...) — score a list of pairs and aggregate
|
|
run_l0_check(...) — defensive wrapper used by the route
|
|
extract_sample(...) — extract text from a finished file
|
|
L1:
|
|
L1Result — verdict dataclass
|
|
LLMJudge — judge client
|
|
run_l1_check(...) — async wrapper used by the route
|
|
sample_chunks_for_l1(...) — pick representative chunks
|
|
Helpers:
|
|
get_script, isArabicScriptLang
|
|
"""
|
|
|
|
from .script_detector import (
|
|
QualityCheckResult,
|
|
DocumentQualityResult,
|
|
evaluate_chunk,
|
|
evaluate_document,
|
|
detect_arabic_variant,
|
|
)
|
|
from .pipeline import run_l0_check, run_l1_check
|
|
from .file_extractor import extract_sample
|
|
from .sampler import sample_chunks_for_l1
|
|
from .llm_judge import L1Result, L1ChunkVerdict, LLMJudge
|
|
|
|
__all__ = [
|
|
# L0
|
|
"QualityCheckResult",
|
|
"DocumentQualityResult",
|
|
"evaluate_chunk",
|
|
"evaluate_document",
|
|
"detect_arabic_variant",
|
|
"run_l0_check",
|
|
"extract_sample",
|
|
# L1
|
|
"L1Result",
|
|
"L1ChunkVerdict",
|
|
"LLMJudge",
|
|
"run_l1_check",
|
|
"sample_chunks_for_l1",
|
|
]
|