L1 quality layer — uses a cheap LLM via the OpenAI-compatible API to
validate translation quality. Designed to be the SECOND line of defense
after L0 (script detection, length, pattern).
Architecture:
- sampler.py — picks 5 representative chunks per job (longest first,
skips L0-failed indices, skips too-short or identical pairs)
- llm_judge.py — OpenAI-compatible client, binary verdict per chunk
(accurate / fluent / correct_language / no_leaks), JSON output,
hard timeout, defensive (never raises), cost estimation built in
- pipeline.py — defensive wrapper that integrates both, never breaks
a translation job, always logs a structured event
Integration:
- 5 feature flags in config.py (QUALITY_L1_ENABLED, _LOG_ONLY, etc.)
- QUALITY_L1_LOG_ONLY=true by default: log-only mode, verdict NEVER
blocks or retries a job
- Reuses the chunks extracted by L0 (no double work)
- Passes the set of L0-failed indices so L1 doesn't re-judge them
- Wrapped in try/except so a misconfigured L1 NEVER breaks a job
Default config: deepseek-chat via DeepSeek API
- Cost: ~0.0003 USD per job (5 chunks)
- Speed: typically 1-2s per call, hard ceiling at 8s
- Easy to swap: just set L1_JUDGE_BASE_URL and L1_JUDGE_MODEL
LLM judge is intentionally a SEPARATE model from the translator
(self-evaluation bias mitigation — Meta/Stanford papers 2024-2025).
Tests:
test_sampler.py — 9 tests covering the sampling strategy
test_llm_judge.py — 22 tests covering init, parsing, mocked API,
cost estimation, env factory
test_l1_pipeline.py — 6 tests covering the wrapper
Total new: 37 tests, all pass
Grand total quality+format: 264 tests passing (0 regression)
All 36 new tests + 111 L0 tests + 117 existing translator tests = 264
Phase 1 (observation) for 2 weeks. Then QUALITY_L1_LOG_ONLY=false
to enable auto-retry via the fallback chain.
L0 quality detection layer to catch translation failures BEFORE they
reach users. Pure Python/TypeScript, zero new dependencies, no API calls.
Backend (Python — services/quality/):
- Script detection: 145 langs mapped to 23 scripts (Latin, Cyrillic,
Greek, Arabic, Hebrew, CJK, Hangul, Kana, Devanagari, Bengali, etc.)
- Language confusion detection (e.g. Arabic text for French target)
- Arabic-script variant discrimination (Persian/Urdu/Pashto/Kurdish
confusion — e.g. Persian text returned when Arabic was requested)
- Length sanity check (with numeric/short-source exemptions)
- Prompt leak detection (Translation: / Voici la traduction: / 翻译:)
- Repetition hallucination detection (token + character level)
- File text extraction for .docx/.xlsx/.pptx/.pdf (no translator
changes needed)
- Defensive pipeline that never raises (L0 must NEVER break a job)
Frontend (TypeScript — wordly.art---traduction-de-documents/src/utils/):
- Exact 1:1 mirror of the Python module
- Zero dependencies, works in browser AND Node.js
- Native Unicode regex (\\p{L}/u) and codePoint iteration
- 63 tests using Node's built-in test runner
Integration:
- Feature-flagged: QUALITY_L0_ENABLED=false (default)
- Observation only: logs structured events, never modifies files
- try/except wrapped: impossible to break a translation job
- Lazy imports: only loaded when flag is on
- Zero impact on existing tests / behavior
Tests:
- 111 Python tests covering all paths (config, script, length, leak,
pipeline, file_extractor) — 100% pass
- 63 TypeScript tests (Node --test) — 100% pass
- 174/174 total tests for the L0 layer
Bug fixes in script mapping:
- yi (Yiddish) -> hebrew (was incorrectly mapped to arabic)
- dv (Maldivian) -> thaana (was incorrectly mapped to arabic)
- ja (Japanese) -> hiragana_katakana (distinguishes from Chinese CJK)
Phase 1 (backend) + Phase 2 (frontend) of Track A complete.
Next: Track B1 (Word/Excel format preservation quick wins).
Closes Track A phase 1+2 of the dev plan.
