sepehr
f403b2851d
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 3m5s
L0 quality detection layer to catch translation failures BEFORE they
reach users. Pure Python/TypeScript, zero new dependencies, no API calls.
Backend (Python — services/quality/):
- Script detection: 145 langs mapped to 23 scripts (Latin, Cyrillic,
Greek, Arabic, Hebrew, CJK, Hangul, Kana, Devanagari, Bengali, etc.)
- Language confusion detection (e.g. Arabic text for French target)
- Arabic-script variant discrimination (Persian/Urdu/Pashto/Kurdish
confusion — e.g. Persian text returned when Arabic was requested)
- Length sanity check (with numeric/short-source exemptions)
- Prompt leak detection (Translation: / Voici la traduction: / 翻译:)
- Repetition hallucination detection (token + character level)
- File text extraction for .docx/.xlsx/.pptx/.pdf (no translator
changes needed)
- Defensive pipeline that never raises (L0 must NEVER break a job)
Frontend (TypeScript — wordly.art---traduction-de-documents/src/utils/):
- Exact 1:1 mirror of the Python module
- Zero dependencies, works in browser AND Node.js
- Native Unicode regex (\\p{L}/u) and codePoint iteration
- 63 tests using Node's built-in test runner
Integration:
- Feature-flagged: QUALITY_L0_ENABLED=false (default)
- Observation only: logs structured events, never modifies files
- try/except wrapped: impossible to break a translation job
- Lazy imports: only loaded when flag is on
- Zero impact on existing tests / behavior
Tests:
- 111 Python tests covering all paths (config, script, length, leak,
pipeline, file_extractor) — 100% pass
- 63 TypeScript tests (Node --test) — 100% pass
- 174/174 total tests for the L0 layer
Bug fixes in script mapping:
- yi (Yiddish) -> hebrew (was incorrectly mapped to arabic)
- dv (Maldivian) -> thaana (was incorrectly mapped to arabic)
- ja (Japanese) -> hiragana_katakana (distinguishes from Chinese CJK)
Phase 1 (backend) + Phase 2 (frontend) of Track A complete.
Next: Track B1 (Word/Excel format preservation quick wins).
Closes Track A phase 1+2 of the dev plan.
144 lines
5.4 KiB
Python
144 lines
5.4 KiB
Python
"""
|
|
Tests for services/quality/config.py
|
|
Covers the language → script mapping and discriminating characters.
|
|
"""
|
|
import pytest
|
|
|
|
from services.quality import config as qconfig
|
|
|
|
|
|
class TestGetScript:
|
|
def test_cyrillic_languages(self):
|
|
assert qconfig.get_script("ru") == "cyrillic"
|
|
assert qconfig.get_script("uk") == "cyrillic"
|
|
assert qconfig.get_script("be") == "cyrillic"
|
|
assert qconfig.get_script("bg") == "cyrillic"
|
|
assert qconfig.get_script("sr") == "cyrillic"
|
|
assert qconfig.get_script("kk") == "cyrillic"
|
|
|
|
def test_latin_languages(self):
|
|
assert qconfig.get_script("en") == "latin"
|
|
assert qconfig.get_script("fr") == "latin"
|
|
assert qconfig.get_script("de") == "latin"
|
|
assert qconfig.get_script("es") == "latin"
|
|
assert qconfig.get_script("vi") == "latin"
|
|
assert qconfig.get_script("tr") == "latin"
|
|
|
|
def test_cjk_languages(self):
|
|
assert qconfig.get_script("zh") == "cjk"
|
|
assert qconfig.get_script("zh-cn") == "cjk"
|
|
assert qconfig.get_script("zh-tw") == "cjk"
|
|
|
|
def test_japanese_uses_kana(self):
|
|
# ja is mapped to hiragana_katakana specifically so it can be
|
|
# distinguished from Chinese CJK.
|
|
assert qconfig.get_script("ja") == "hiragana_katakana"
|
|
|
|
def test_korean_uses_hangul(self):
|
|
assert qconfig.get_script("ko") == "hangul"
|
|
|
|
def test_arabic_script_languages(self):
|
|
assert qconfig.get_script("ar") == "arabic"
|
|
assert qconfig.get_script("fa") == "arabic"
|
|
assert qconfig.get_script("ur") == "arabic"
|
|
assert qconfig.get_script("ps") == "arabic"
|
|
assert qconfig.get_script("ku") == "arabic"
|
|
|
|
def test_hebrew_and_yiddish(self):
|
|
assert qconfig.get_script("he") == "hebrew"
|
|
# Yiddish uses Hebrew script, not Arabic
|
|
assert qconfig.get_script("yi") == "hebrew"
|
|
|
|
def test_thaana(self):
|
|
# Maldivian (Dhivehi) uses Thaana script, not Arabic
|
|
assert qconfig.get_script("dv") == "thaana"
|
|
|
|
def test_indian_scripts(self):
|
|
assert qconfig.get_script("hi") == "devanagari"
|
|
assert qconfig.get_script("bn") == "bengali"
|
|
assert qconfig.get_script("ta") == "tamil"
|
|
assert qconfig.get_script("te") == "telugu"
|
|
assert qconfig.get_script("gu") == "gujarati"
|
|
assert qconfig.get_script("pa") == "gurmukhi"
|
|
assert qconfig.get_script("ml") == "malayalam"
|
|
assert qconfig.get_script("kn") == "kannada"
|
|
assert qconfig.get_script("si") == "sinhala"
|
|
|
|
def test_se_asian_scripts(self):
|
|
assert qconfig.get_script("th") == "thai"
|
|
assert qconfig.get_script("lo") == "lao"
|
|
assert qconfig.get_script("my") == "burmese"
|
|
assert qconfig.get_script("km") == "khmer"
|
|
|
|
def test_other_scripts(self):
|
|
assert qconfig.get_script("el") == "greek"
|
|
assert qconfig.get_script("ka") == "georgian"
|
|
assert qconfig.get_script("hy") == "armenian"
|
|
assert qconfig.get_script("am") == "ethiopic"
|
|
assert qconfig.get_script("bo") == "tibetan"
|
|
|
|
def test_unknown_falls_back_to_latin(self):
|
|
assert qconfig.get_script("xx") == "latin"
|
|
assert qconfig.get_script("") == "latin"
|
|
assert qconfig.get_script(None) == "latin"
|
|
|
|
def test_case_insensitive(self):
|
|
assert qconfig.get_script("FR") == "latin"
|
|
assert qconfig.get_script("ZH-CN") == "cjk"
|
|
assert qconfig.get_script("Ru") == "cyrillic"
|
|
|
|
|
|
class TestIsArabicScriptLang:
|
|
def test_true_for_arabic_script_langs(self):
|
|
assert qconfig.is_arabic_script_lang("ar") is True
|
|
assert qconfig.is_arabic_script_lang("fa") is True
|
|
assert qconfig.is_arabic_script_lang("ur") is True
|
|
assert qconfig.is_arabic_script_lang("ckb") is True
|
|
|
|
def test_false_for_non_arabic(self):
|
|
assert qconfig.is_arabic_script_lang("fr") is False
|
|
assert qconfig.is_arabic_script_lang("he") is False
|
|
assert qconfig.is_arabic_script_lang("hi") is False
|
|
assert qconfig.is_arabic_script_lang("en") is False
|
|
|
|
def test_false_for_unknown(self):
|
|
assert qconfig.is_arabic_script_lang("xx") is False
|
|
assert qconfig.is_arabic_script_lang("") is False
|
|
assert qconfig.is_arabic_script_lang(None) is False
|
|
|
|
|
|
class TestDiscriminatingChars:
|
|
def test_persian_chars(self):
|
|
chars = qconfig.get_discriminating_chars("fa")
|
|
assert "پ" in chars # peh
|
|
assert "چ" in chars # tcheh
|
|
assert "ژ" in chars # zheh
|
|
assert "گ" in chars # guaf
|
|
|
|
def test_urdu_chars(self):
|
|
chars = qconfig.get_discriminating_chars("ur")
|
|
assert "ٹ" in chars # tteh
|
|
assert "ڈ" in chars # ddal
|
|
|
|
def test_unknown_lang_empty(self):
|
|
assert qconfig.get_discriminating_chars("fr") == frozenset()
|
|
assert qconfig.get_discriminating_chars("") == frozenset()
|
|
assert qconfig.get_discriminating_chars(None) == frozenset()
|
|
|
|
|
|
class TestGetRanges:
|
|
def test_latin_returns_empty(self):
|
|
# Latin is the fallback — no ranges, means "anything matches".
|
|
assert qconfig.get_ranges("latin") == []
|
|
|
|
def test_cyrillic_ranges(self):
|
|
ranges = qconfig.get_ranges("cyrillic")
|
|
assert any(start == 0x0400 for start, _ in ranges)
|
|
|
|
def test_cjk_ranges(self):
|
|
ranges = qconfig.get_ranges("cjk")
|
|
assert any(start == 0x4E00 for start, _ in ranges)
|
|
|
|
def test_unknown_script_returns_empty(self):
|
|
assert qconfig.get_ranges("mystery_script") == []
|