diff --git a/_bmad-output/implementation-artifacts/deferred-work.md b/_bmad-output/implementation-artifacts/deferred-work.md new file mode 100644 index 0000000..75f59bf --- /dev/null +++ b/_bmad-output/implementation-artifacts/deferred-work.md @@ -0,0 +1,33 @@ +# Deferred Work + +Entrées ajoutées lors du cadrage de `spec-rendu-fidele-langues-rtl.md` (2026-09-01) : +améliorations de qualité identifiées pendant l'audit du pipeline de traduction, +volontairement hors du périmètre de la spec RTL pour garder un objectif unique. + +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Fusionner les runs adjacents de même style dans PowerPoint (et les notes Word traduites run par run) pour traduire des phrases complètes au lieu de fragments. + evidence: Word fusionne déjà les runs par signature rPr (word_translator.py l.1368-1475) ; PowerPoint traduit chaque run isolément (pptx_translator.py l.810-834) et les footnotes Word aussi (l.836-846, 909-919, 961-971) — une phrase coupée en plusieurs runs produit des traductions incohérentes. Interdit par la spec RTL (« Never : pas de refonte de la collecte des runs »). +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Rendre les contrôles qualité actionnables : retenter automatiquement les lots dont le script livré ne correspond pas à la langue cible (ex. arabe livré au lieu du persan). + evidence: services/quality/ détecte déjà le mauvais script et la confusion arabe/persan (script_detector.py l.89-171, l.236-256) mais tous les contrôles sont en log-only (routes/translate_routes.py l.1644-1779) — aucune traduction n'est jamais corrigée ni retentée. +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Étendre le mode batch JSON (~15 textes par requête) aux providers DeepSeek et MiniMax, comme le fait déjà le provider OpenAI. + evidence: openai_provider.py l.511-625 batch via JSON numéroté ; deepseek_provider.py et minimax_provider.py traduisent texte par texte (base.py l.60-74) — coût et latence plus élevés pour ces moteurs. +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Conserver le formatage intra-bloc des PDF (un mot en gras au milieu d'une phrase applique aujourd'hui le gras à tout le bloc). + evidence: pdf_translator.py _extract_text_blocks l.541-734 calcule des flags gras/italique globaux par bloc et la réécriture utilise une police unique par bloc (l.905-922). + +Entrées ajoutées après la revue croisée de `spec-rendu-fidele-langues-rtl.md` (2026-09-01) : + +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Utiliser insert_htmlbox (PyMuPDF) pour le texte RTL des PDF afin de garder un texte copiable et cherchable (ordre logique), au lieu du façonnage en formes de présentation. + evidence: arabic_reshaper + python-bidi écrivent les lettres arabes en formes de présentation (U+FB50–U+FEFF) en ordre visuel : un copier-coller depuis le PDF donne un texte inversé et la recherche ne fonctionne plus ; PyMuPDF ≥ 1.22 (déjà dépendance) propose insert_htmlbox qui gère nativement la direction bidi. +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Passer en droite-à-gauche les textes des graphiques et SmartArt Word (parties chart réinjectées après traduction). + evidence: les textes de graphiques/diagrammes sont traduits et réinjectés (word_translator.py l.1030-1186, l.1229-1356) mais aucune marque de direction n'est posée sur ces parties — étiquettes arabes affichées gauche-à-droite dans un document RTL. Exclu volontairement par la spec (« Never : pas d'inversion des graphiques »). +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Résoudre l'alignement hérité des masques/dispositions PowerPoint pour ne pas forcer à droite un titre centré par le modèle (sans algn explicite). + evidence: _set_pptx_paragraph_rtl pose algn="r" quand algn est absent — or l'alignement réel peut venir du masque ; détecter l'héritage exige de remonter au layout/master (chantier de résolution de placeholder dédié). +- source_spec: spec-rendu-fidele-langues-rtl.md + summary: Harmoniser RTL_LOCALES du frontend (frontend/src/lib/i18n.tsx, liste ["ar","fa"]) avec la liste serveur à onze langues. + evidence: la liste d'affichage RTL de l'interface est indépendante de core/languages.RTL_LANGUAGES ; toute nouvelle langue d'interface RTL (hébreu, ourdou) devra être ajoutée aux deux endroits sans qu'aucun contrôle ne le signale. diff --git a/_bmad-output/implementation-artifacts/spec-rendu-fidele-langues-rtl.md b/_bmad-output/implementation-artifacts/spec-rendu-fidele-langues-rtl.md new file mode 100644 index 0000000..08deb70 --- /dev/null +++ b/_bmad-output/implementation-artifacts/spec-rendu-fidele-langues-rtl.md @@ -0,0 +1,147 @@ +--- +title: 'Rendu fidèle des traductions pour les langues RTL (persan, arabe, hébreu…)' +type: 'feature' +created: '2026-09-01' +status: 'done' +review_loop_iteration: 0 +baseline_commit: fddd7b74282f6c1e51d20a70b1ca1e5cb89d5331 +context: [] +--- + + + +## Intent + +**Problem:** Un document traduit vers une langue de droite à gauche sort dégradé : Excel ignore totalement le sens, PowerPoint force l'alignement à droite même sur les titres centrés et oublie les notes du présentateur, les tableaux Word ne sont pas inversés (ni notes de bas de page, ni zones de texte), et le PDF n'applique ni le façonnage des lettres arabes ni de police adaptée (lettres déconnectées, ordre inversé, carrés). Pour un lecteur persanophone ou arabophone, le résultat est parfois illisible. + +**Approach:** Source de vérité unique pour le RTL dans `core/languages.py`, puis corrections ciblées dans les 4 traducteurs : direction et alignement corrects sans écraser les alignements explicites, inversion des tableaux, couverture des zones oubliées (notes, zones de texte), et pour le PDF façonnage bidi (`arabic-reshaper` + `python-bidi`) avec police couvrant l'écriture arabe. + +## Boundaries & Constraints + +**Always:** Comportement inchangé pour toute cible non-RTL ; alignements explicites (centré, justifié) jamais écrasés ; aucun texte re-traduit ni altéré (seuls direction/alignement/indices de police sont ajoutés) ; `RTL_LANGUAGES` n'existe plus qu'une fois ; si `arabic-reshaper`/`python-bidi` manquent ou échouent, la traduction réussit quand même (texte tel quel + log d'avertissement). + +**Ask First:** Ajouter une dépendance autre que `arabic-reshaper` et `python-bidi` ; modifier le `Dockerfile` (le paquet `fonts-noto` de Debian devrait déjà fournir Noto Naskh Arabic — vérifier dans l'image avant tout changement). + +**Never:** Pas de refonte de la collecte des runs (fusion des runs PowerPoint / notes Word = chantier différé) ; pas de retentative automatique sur les contrôles qualité (différé) ; pas de modification des providers, de l'interface ni du frontend ; pas d'inversion des images ou graphiques vectoriels. + +## I/O & Edge-Case Matrix + +| Scenario | Input / State | Expected Output / Behavior | Error Handling | +|----------|--------------|---------------------------|----------------| +| Word cible RTL | docx : corps, tableau, note de bas de page, zone de texte | `w:bidi` sur tous les paragraphes, `w:bidiVisual` sur les tableaux, `w:sectPr/w:bidi` | N/A | +| PPTX titre centré, cible ar | `algn="ctr"` existant | `rtl="1"` posé, `algn` reste `"ctr"` ; si `algn` absent ou `"l"` → `algn="r"` | N/A | +| PPTX avec notes | diapositive avec notes du présentateur | notes traduites ET passées en RTL | N/A | +| Excel cible he | xlsx multi-feuilles | `sheet_view.rightToLeft = True` sur chaque feuille | N/A | +| PDF mode mise en page, cible fa/ar | pdf avec blocs de texte | texte façonné avant insertion, police couvrant l'arabe, alignement à droite | Police introuvable → police actuelle + warning ; bibliothèque absente → texte non façonné + warning ; jamais d'échec | +| PDF scanné / mode texte, cible RTL | OCR ou mode text_only | reportlab : police TTF enregistrée, `wordWrap="RTL"` sur le style | Pas de TTF → Helvetica + warning (dégradation documentée) | +| Cible non-RTL (fr, zh…) | tous formats | Aucun attribut RTL ajouté, sortie identique à aujourd'hui | N/A | +| Code régional | `fa-IR`, `ar-EG` | `is_rtl()` normalise le préfixe → RTL appliqué | N/A | + + + +## Code Map + +- `core/languages.py` -- `LANGUAGE_NAMES` l.12-123, `language_name()` l.126-137 : y ajouter `RTL_LANGUAGES` (frozenset) et `is_rtl(code)`. +- `translators/word_translator.py` -- copie `RTL_LANGUAGES` l.30-32 (supprimer) ; `CS_FONTS` l.45-50 (ar/he/fa/ur seulement) ; `_set_paragraph_rtl` l.126-153 (modèle : `w:jc` touché seulement sans alignement explicite) ; `_apply_rtl_to_document` l.156-182 (corps + tableaux + en-têtes/pieds ; manque `w:bidiVisual`, zones de texte, notes) ; déclencheur l.422-424 ; notes/comments réécrits en ZIP post-save l.811-995 (parts sans RTL). +- `translators/pptx_translator.py` -- copie l.28-30 (supprimer) ; `_set_pptx_paragraph_rtl` l.57-71 (`algn="r"` inconditionnel — bug) ; `_apply_rtl_to_presentation` l.74-78 (pas de `has_notes_slide`) ; `_apply_ea_font_hints` l.114-158 (modèle exact pour ``) ; notes collectées l.313-319 ; déclencheur l.399-401. +- `translators/excel_translator.py` -- `translate_file` l.141-175 ; aucun support RTL (0 occurrence) ; sauvegarde en fin de méthode. +- `translators/pdf_translator.py` -- copie l.63-64 (supprimer) ; `_FONT_SEARCH_PATHS` l.115-134 (aucune police arabe) ; `_get_font_path` l.187-196 (première trouvée, insensible au script) ; alignement RTL l.903 ; `insert_textbox` l.1017-1049 (pas de façonnage) ; `_generate_clean_pdf` l.1434-1516 (reportlab, aucune police enregistrée). +- `requirements.txt` + `pyproject.toml` (dependencies l.7+) -- dépendances dupliquées : mettre les deux à jour. +- `docker/backend/Dockerfile` l.30-40 -- `fonts-noto` installé (vérifier le chemin de Noto Naskh Arabic dans l'image avant tout ajout). +- `tests/test_translators/` -- modèles : `test_word_translator.py`, `test_pptx_translator.py`, `test_excel_translator.py`, `test_b3_pdf_fixes.py`. +- `services/quality/script_detector.py` -- lecture seule (distingue déjà arabe/persan — réutilisable plus tard, hors périmètre). + +## Tasks & Acceptance + +**Execution:** +- [x] `core/languages.py` -- Ajouter `RTL_LANGUAGES` ({ar, he, fa, ur, ku, ps, ug, sd, yi, dv, ckb}) et `is_rtl(code) -> bool` (casse et préfixe régional ignorés) -- source unique, fin des 3 copies. +- [x] `translators/word_translator.py` -- Importer depuis `core.languages` ; `_apply_rtl_to_document` : `w:bidiVisual` sur chaque `w:tbl` du corps et des en-têtes/pieds (XPath `.//w:tbl`, couvre l'imbriqué), bidi/rtl sur les paragraphes des `w:txbxContent` ; réécriture ZIP (l.811-995) : `w:bidi` sur les `w:pPr` de `footnotes.xml`/`endnotes.xml`/`comments.xml` quand cible RTL ; `CS_FONTS` élargi à ps, ku, sd, ug, yi, dv, ckb (Arial). +- [x] `translators/pptx_translator.py` -- Importer ; `algn="r"` seulement si `algn` absent ou `"l"` ; `_apply_rtl_to_presentation` traite aussi `notes_text_frame` quand `has_notes_slide` ; ajouter `_apply_cs_font_hints` (modèle `_apply_ea_font_hints`) posant `` pour les cibles RTL. +- [x] `translators/excel_translator.py` -- Importer `is_rtl` ; avant sauvegarde : `ws.sheet_view.rightToLeft = True` par feuille quand cible RTL. +- [x] `translators/pdf_translator.py` -- Importer ; chemins Noto Naskh Arabic / Noto Sans Arabic (Linux) + `C:/Windows/Fonts/arial.ttf` ; `_get_font_path` prend la langue et choisit une police couvrant l'arabe pour les cibles RTL ; `_shape_rtl(text)` (import protégé, fallback texte tel quel + warning) appliquée aux blocs du mode mise en page et à `_generate_clean_pdf` ; dans ce dernier : `pdfmetrics.registerFont` de la TTF si présente + `wordWrap="RTL"` + `fontName`. +- [x] `requirements.txt` + `pyproject.toml` -- Ajouter `arabic-reshaper` et `python-bidi` (versions pinnées comme le reste). +- [x] `tests/test_translators/test_rtl_layout.py` -- Couvrir la matrice : docx (bidi corps/note, bidiVisual), pptx (algn conservé/posé, notes, ``), xlsx (`rightToLeft`), pdf (façonnage, police, fallback sans bibliothèque), `is_rtl("fa-IR")`/`is_rtl("fr")`. + +**Acceptance Criteria:** +- Given un docx avec tableau et note de bas de page, when traduction vers `fa`, then `w:bidi` sur corps/note/zone de texte et `w:bidiVisual` sur le tableau. +- Given un pptx avec un titre `algn="ctr"`, when traduction vers `ar`, then le titre garde `algn="ctr"` et reçoit `rtl="1"`. +- Given un xlsx de 3 feuilles, when traduction vers `he`, then les 3 `sheet_view.rightToLeft` sont vrais. +- Given un pdf textuel traduit vers `fa`, when mode mise en page, then texte façonné inséré et police arabe sélectionnée si présente. +- Given une traduction vers `fr`, then aucun attribut RTL ajouté (sortie inchangée). + +## Spec Change Log + +## Verification + +**Commands:** +- `python -m pytest tests/test_translators/test_rtl_layout.py -v` -- expected: tous les tests passent. +- `python -m pytest tests/test_translators/ tests/test_language_validation.py -q` -- expected: aucune régression. +- `python -c "import arabic_reshaper, bidi.algorithm; print('ok')"` -- expected: ok. + +## Suggested Review Order + +**Source de vérité des langues** + +- Liste unique partagée par les 4 traducteurs, fin des 3 copies locales + [`languages.py:142`](../../core/languages.py#L142) + +- Normalisation casse/préfixe régional (fa-IR, AR → RTL ; fr, None → non) + [`languages.py:147`](../../core/languages.py#L147) + +**Word — direction et ordre OOXML** + +- Insertion positionnée selon le schéma (bidi/jc/rtl/bidiVisual jamais en fin) + [`word_translator.py:187`](../../translators/word_translator.py#L187) + +- Paragraphe RTL : bidi + miroir d'alignement sans toucher centré/justifié + [`word_translator.py:202`](../../translators/word_translator.py#L202) + +- Tableaux inversés visuellement (w:bidiVisual) + [`word_translator.py:236`](../../translators/word_translator.py#L236) + +- Notes/commentaires : bidi + rtl sur les runs + tableaux, pendant la réécriture ZIP + [`word_translator.py:254`](../../translators/word_translator.py#L254) + +- Les 6 zones d'en-têtes/pieds (défaut, première page, pages paires) couvertes + [`word_translator.py:288`](../../translators/word_translator.py#L288) + +**PowerPoint — alignements et polices** + +- rtl=1 posé, algn forcé à droite seulement si absent ou « l » + [`pptx_translator.py:71`](../../translators/pptx_translator.py#L71) + +- Notes du présentateur incluses dans le passage RTL + [`pptx_translator.py:91`](../../translators/pptx_translator.py#L91) + +- Indice de police / inséré à sa place, rPr en tête du run, notes incluses + [`pptx_translator.py:136`](../../translators/pptx_translator.py#L136) + +**Excel — sens des feuilles** + +- rightToLeft par feuille, protégé (feuilles graphiques sans sheet_view ignorées) + [`excel_translator.py:369`](../../translators/excel_translator.py#L369) + +**PDF — façonnage et polices** + +- Façonnage bidi (lettres liées, bon ordre), repli sans échec si bibliothèques absentes + [`pdf_translator.py:69`](../../translators/pdf_translator.py#L69) + +- Police enregistrée une seule fois par chemin, nom unique par processus + [`pdf_translator.py:106`](../../translators/pdf_translator.py#L106) + +- Groupe d'écriture (arabe / hébreu / générique) avant le choix de police + [`pdf_translator.py:185`](../../translators/pdf_translator.py#L185) + +- Sélection de police sensible à l'écriture cible, cache par groupe + [`pdf_translator.py:309`](../../translators/pdf_translator.py#L309) + +- PDF recomposé : TTF enregistrée, wordWrap RTL, texte façonné + [`pdf_translator.py:1597`](../../translators/pdf_translator.py#L1597) + +**Périphérie** + +- 46 tests couvrant la matrice (docx/pptx/xlsx/pdf + endnotes/comments/chartsheet) + [`test_rtl_layout.py:1`](../../tests/test_translators/test_rtl_layout.py#L1) + +- Dépendances façonnage (requirements.txt + pyproject.toml + uv.lock alignés) + [`requirements.txt:11`](../../requirements.txt#L11) diff --git a/core/languages.py b/core/languages.py index b68be4d..4db6ac6 100644 --- a/core/languages.py +++ b/core/languages.py @@ -7,7 +7,7 @@ prompts say "Translate to Tagalog" instead of "Translate to tl" — LLMs translate noticeably better with full language names. """ -from typing import Dict +from typing import Dict, Optional LANGUAGE_NAMES: Dict[str, str] = { "af": "Afrikaans", @@ -135,3 +135,22 @@ def language_name(code: str) -> str: if name is None: name = LANGUAGE_NAMES.get(code.split("-")[0].lower(), code) return name + + +# Languages written right-to-left. Single source of truth shared by every +# translator (Word, PowerPoint, Excel, PDF) — do not duplicate this set. +RTL_LANGUAGES: frozenset = frozenset( + {"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"} +) + + +def is_rtl(code: Optional[str]) -> bool: + """True when the language is written right-to-left. + + Case and regional suffixes are ignored: "fa-IR", "AR" and "ar-EG" are + RTL, "fr" and "" are not. None is accepted (treated as not RTL). + """ + if not code: + return False + base = code.strip().replace("_", "-").split("-")[0].lower() + return base in RTL_LANGUAGES diff --git a/pyproject.toml b/pyproject.toml index af12c36..52d52ec 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,6 +9,7 @@ dependencies = [ "aiosmtplib>=3.0.0", "aiosqlite>=0.19.0", "alembic==1.13.1", + "arabic-reshaper==3.0.0", "asyncpg>=0.29.0", "bcrypt<4.1", "deep-translator==1.11.4", @@ -33,6 +34,7 @@ dependencies = [ "pymupdf>=1.24.0", "pytest>=7.0.0", "pytest-asyncio>=0.21.0", + "python-bidi==0.4.2", "python-docx==1.1.0", "python-dotenv==1.0.0", "python-magic>=0.4.27", diff --git a/requirements.txt b/requirements.txt index 58522c2..ff06c5b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,6 +8,8 @@ pdf2docx>=0.5.6 PyMuPDF>=1.24.0 lxml>=4.9.0 reportlab>=4.0.0 +arabic-reshaper==3.0.0 +python-bidi==0.4.2 deep-translator==1.11.4 python-dotenv==1.0.0 pydantic==2.5.3 diff --git a/scripts/debug_smart_fit_trace.py b/scripts/debug_smart_fit_trace.py index c2daa35..4d54a3e 100644 --- a/scripts/debug_smart_fit_trace.py +++ b/scripts/debug_smart_fit_trace.py @@ -11,7 +11,7 @@ spec.loader.exec_module(pdf_mod) original_write = pdf_mod.PDFTranslator._write_translated_block -def debug_write(self, page, block, font_path, is_rtl): +def debug_write(self, page, block, font_path, rtl_target): target_size = block['font_size'] original_rect = fitz.Rect(block['bbox']) page_rect = page.rect @@ -40,7 +40,7 @@ def debug_write(self, page, block, font_path, is_rtl): text = block['translated'][:40] is_heading_str = "HEAD" if is_heading else "BODY" print(f" [{is_heading_str}] text={text!r} | orig={original_rect.width:.0f}x{original_rect.height:.0f} | exp_v={expanded_v.width:.0f}x{expanded_v.height:.0f} | size={target_size}", flush=True) - result = original_write(self, page, block, font_path, is_rtl) + result = original_write(self, page, block, font_path, rtl_target) print(f" result: {result}", flush=True) return result diff --git a/tests/test_translators/test_b3_5_pdf_smart_fit.py b/tests/test_translators/test_b3_5_pdf_smart_fit.py index 4efde05..e76899f 100644 --- a/tests/test_translators/test_b3_5_pdf_smart_fit.py +++ b/tests/test_translators/test_b3_5_pdf_smart_fit.py @@ -99,7 +99,7 @@ class TestSmartFitTiers: translator = pdf_mod.PDFTranslator() result = translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) # Should have succeeded assert result is True @@ -122,7 +122,7 @@ class TestSmartFitTiers: block = self._make_block(font_size=12, x0=100, y0=100, x1=400, y1=120) translator = pdf_mod.PDFTranslator() result = translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) assert result is True # Should have tried multiple times @@ -137,7 +137,7 @@ class TestSmartFitTiers: block = self._make_block(font_size=20, x0=100, y0=100, x1=400, y1=120) translator = pdf_mod.PDFTranslator() result = translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) # Should have failed (returned False) because persistent overflow assert result is False @@ -159,7 +159,7 @@ class TestSmartFitTiers: block = self._make_block(font_size=22, text="X" * 1000) translator = pdf_mod.PDFTranslator() translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) # All attempted sizes should be >= 22 * 0.90 = 19.8 for s in call_sizes: @@ -180,7 +180,7 @@ class TestSmartFitTiers: block = self._make_block(font_size=12, text="Y" * 1000) translator = pdf_mod.PDFTranslator() translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) for s in call_sizes: if s is not None: @@ -201,7 +201,7 @@ class TestSmartFitTiers: with patch.object(pdf_mod, "_record_format_loss_metric", fake_record): block = self._make_block(font_size=12, text="Z" * 1000) result = translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) assert result is False # A format_loss metric should have been emitted @@ -221,7 +221,7 @@ class TestSmartFitTiers: with patch.object(pdf_mod, "_record_format_loss_metric", fake_record): block = self._make_block(font_size=12, text="Short") result = translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) assert result is True # No format_loss should have been emitted @@ -242,7 +242,7 @@ class TestSmartFitTiers: block = self._make_block(font_size=3, text="Q" * 1000) translator = pdf_mod.PDFTranslator() translator._write_translated_block( - page, block, font_path=None, is_rtl=False + page, block, font_path=None, rtl_target=False ) # Exclude the placeholder font (used in graceful failure) real_sizes = [s for s in call_sizes if s is not None and s >= 4.5] diff --git a/tests/test_translators/test_rtl_layout.py b/tests/test_translators/test_rtl_layout.py new file mode 100644 index 0000000..238f240 --- /dev/null +++ b/tests/test_translators/test_rtl_layout.py @@ -0,0 +1,1008 @@ +""" +Tests for RTL (right-to-left) layout support across the translators. + +Covers the spec matrix: +- is_rtl(): regional prefixes (fa-IR, ar-EG) and case are normalized +- Word: w:bidi on body / footnotes / text boxes, w:bidiVisual on tables, + w:bidi on sectPr, explicit alignments (center) never overwritten +- PowerPoint: rtl="1" on paragraphs, algn kept when explicit (ctr) and + only forced to "r" when absent or "l", speaker notes RTL, + hints for RTL targets +- Excel: sheet_view.rightToLeft = True on every sheet +- PDF: Arabic-script font selection per target language, bidi shaping + (arabic-reshaper + python-bidi), graceful fallback when the shaping + libraries are missing, wordWrap="RTL" + registered TTF font in the + clean-PDF mode +- Non-RTL targets (fr): no RTL attribute added anywhere +""" + +import sys +import zipfile +from pathlib import Path + +import pytest +from docx import Document +from docx.enum.text import WD_ALIGN_PARAGRAPH +from lxml import etree +from openpyxl import Workbook, load_workbook +from pptx import Presentation +from pptx.enum.text import PP_ALIGN +from pptx.util import Inches + +from core.languages import RTL_LANGUAGES, is_rtl +from services.providers.schemas import TranslationRequest, TranslationResponse +from translators.excel_translator import ExcelTranslator +from translators.pdf_translator import PDFTranslator, _shape_rtl +from translators.pptx_translator import PowerPointTranslator +from translators.word_translator import ( + CS_FONTS, + WordTranslator, + _font_hints_for_target, +) + +W_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" +A_NS = "http://schemas.openxmlformats.org/drawingml/2006/main" + + +class MockTranslationProvider: + """Mock translation provider for testing.""" + + def __init__(self, translations: dict = None): + self._translations = translations or {} + self._call_count = 0 + self._requests_received: list[TranslationRequest] = [] + + def get_name(self) -> str: + return "mock" + + def is_available(self) -> bool: + return True + + def translate_text(self, request: TranslationRequest) -> TranslationResponse: + self._call_count += 1 + self._requests_received.append(request) + + text = request.text + translated = self._translations.get(text, f"TR_{text}") + + return TranslationResponse( + translated_text=translated, + provider_name="mock", + source_language=request.source_language, + ) + + def translate_batch( + self, requests: list[TranslationRequest] + ) -> list[TranslationResponse]: + return [self.translate_text(req) for req in requests] + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _read_zip_entry(path: Path, name: str) -> bytes: + with zipfile.ZipFile(path) as zf: + return zf.read(name) + + +def _rewrite_zip_entry(path: Path, name: str, data: bytes) -> None: + with zipfile.ZipFile(path) as zin: + items = {n: zin.read(n) for n in zin.namelist()} + items[name] = data + with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zout: + for entry_name, entry_data in items.items(): + zout.writestr(entry_name, entry_data) + + +_FOOTNOTES_PART = b""" + + + + + + + + + Footnote content + +""" + +_ENDNOTES_PART = b""" + + + + + + Endnote content + + + + + Endnote cell + + +""" + +_COMMENTS_PART = b""" + + + Comment content + + + + + Comment cell + + +""" + + +def _inject_part(docx_path: Path, kind: str, xml_bytes: bytes, rel_id: str) -> None: + """Add a minimal word/.xml part (footnotes/endnotes/comments) + to a saved .docx, with its content-type override and relationship.""" + part_name = f"word/{kind}.xml" + content_type = ( + "application/vnd.openxmlformats-officedocument.wordprocessingml." + f"{kind}+xml" + ) + ct = _read_zip_entry(docx_path, "[Content_Types].xml").decode("utf-8") + if f"{kind}+xml" not in ct: + ct = ct.replace( + "", + f'', + ) + rels = _read_zip_entry(docx_path, "word/_rels/document.xml.rels").decode("utf-8") + if part_name not in rels: + rels = rels.replace( + "", + f'', + ) + _rewrite_zip_entry(docx_path, "[Content_Types].xml", ct.encode("utf-8")) + _rewrite_zip_entry(docx_path, "word/_rels/document.xml.rels", rels.encode("utf-8")) + _rewrite_zip_entry(docx_path, part_name, xml_bytes) + + +def _inject_footnotes_part(docx_path: Path) -> None: + """Add a minimal word/footnotes.xml part to a saved .docx.""" + _inject_part(docx_path, "footnotes", _FOOTNOTES_PART, "rIdRtlFootnotes") + + +def _inject_textbox(docx_path: Path) -> None: + """Add a VML text box (w:txbxContent) to the document body.""" + doc_xml = _read_zip_entry(docx_path, "word/document.xml").decode("utf-8") + textbox = ( + '' + '' + "" + "TextBox content" + "" + "" + ) + # The sectPr must stay the LAST child of w:body. + if "", textbox + "") + _rewrite_zip_entry(docx_path, "word/document.xml", doc_xml.encode("utf-8")) + + +def _word_paragraph_has_bidi(p_elem) -> bool: + pPr = p_elem.find(f"{{{W_NS}}}pPr") + return pPr is not None and pPr.find(f"{{{W_NS}}}bidi") is not None + + +def _find_pptx_paragraph(prs, needle: str): + """First paragraph (slides + notes) whose text contains needle.""" + for slide in prs.slides: + for shape in slide.shapes: + if shape.has_text_frame: + for para in shape.text_frame.paragraphs: + if needle in para.text: + return para + if slide.has_notes_slide and slide.notes_slide.notes_text_frame is not None: + for para in slide.notes_slide.notes_text_frame.paragraphs: + if needle in para.text: + return para + return None + + +def _host_rtl_font() -> str: + """An Arabic-script font actually installed on this host. + + Searches ONLY the dedicated Arabic list — not the generic fallback — + so the skipif really skips on hosts without an Arabic font. + """ + translator = PDFTranslator() + for p in translator._RTL_ARABIC_FONT_SEARCH_PATHS: + if Path(p).exists(): + return p + return None + + +# --------------------------------------------------------------------------- +# is_rtl() — single source of truth +# --------------------------------------------------------------------------- + + +class TestIsRtl: + def test_regional_prefix_normalized(self): + assert is_rtl("fa-IR") is True + assert is_rtl("ar-EG") is True + assert is_rtl("he-IL") is True + assert is_rtl("ckb") is True + + def test_case_insensitive(self): + assert is_rtl("AR") is True + assert is_rtl("Fa") is True + + def test_non_rtl_targets(self): + assert is_rtl("fr") is False + assert is_rtl("en") is False + assert is_rtl("zh-CN") is False + assert is_rtl("") is False + assert is_rtl(None) is False + + def test_full_set_is_rtl(self): + for code in RTL_LANGUAGES: + assert is_rtl(code) is True + + def test_rtl_set_exists_only_in_core(self): + """RTL_LANGUAGES must not be duplicated in the translators.""" + import translators.excel_translator as xl + import translators.pdf_translator as pdft + import translators.pptx_translator as pt + import translators.word_translator as wt + + for module in (wt, pt, pdft, xl): + assert not hasattr(module, "RTL_LANGUAGES"), module.__name__ + + +# --------------------------------------------------------------------------- +# Word — body, table, footnote, text box, section +# --------------------------------------------------------------------------- + + +class TestWordRtlLayout: + def test_fa_target_full_rtl_coverage(self, tmp_path): + """bidi on body/centered title/header, bidiVisual on the table, + bidi on the footnote and text box, bidi on sectPr.""" + provider = MockTranslationProvider( + { + "Body text": "FA_Body", + "Centered title": "FA_Title", + "Header text": "FA_Header", + "Cell": "FA_Cell", + "TextBox content": "FA_Box", + "Footnote content": "FA_Note", + } + ) + translator = WordTranslator(provider=provider) + + doc = Document() + doc.add_paragraph("Body text") + centered = doc.add_paragraph() + centered.alignment = WD_ALIGN_PARAGRAPH.CENTER + centered.add_run("Centered title") + table = doc.add_table(rows=1, cols=2) + table.cell(0, 0).text = "Cell" + doc.sections[0].header.paragraphs[0].text = "Header text" + + input_file = tmp_path / "input.docx" + doc.save(input_file) + _inject_textbox(input_file) + _inject_footnotes_part(input_file) + + output_file = tmp_path / "output.docx" + translator.translate_file(input_file, output_file, "fa") + + doc_out = Document(output_file) + + # Body paragraph: w:bidi + run-level w:rtl + body_para = doc_out.paragraphs[0] + assert _word_paragraph_has_bidi(body_para._p) + assert "FA_Body" in body_para.text + + # Centered title keeps its explicit alignment (never overwritten) + centered_out = doc_out.paragraphs[1] + assert _word_paragraph_has_bidi(centered_out._p) + assert centered_out.alignment == WD_ALIGN_PARAGRAPH.CENTER + jc = centered_out._p.find(f"{{{W_NS}}}pPr").find(f"{{{W_NS}}}jc") + assert jc.get(f"{{{W_NS}}}val") == "center" + + # OOXML schema order: w:bidi must precede spacing/ind/jc/rPr + # (out-of-order pPr children make Word flag the file for repair) + centered_pPr = centered_out._p.find(f"{{{W_NS}}}pPr") + child_tags = [child.tag for child in centered_pPr] + bidi_pos = child_tags.index(f"{{{W_NS}}}bidi") + for later in ("w:spacing", "w:ind", "w:jc", "w:rPr"): + tag = f"{{{W_NS}}}{later}" + if tag in child_tags: + assert bidi_pos < child_tags.index(tag), later + # Same for the run-level marker: w:rtl after w:rFonts + body_rPr = body_para.runs[0]._r.find(f"{{{W_NS}}}rPr") + assert body_rPr is not None + run_tags = [child.tag for child in body_rPr] + if f"{{{W_NS}}}rFonts" in run_tags: + assert run_tags.index(f"{{{W_NS}}}rFonts") < run_tags.index( + f"{{{W_NS}}}rtl" + ) + + # Table: w:bidiVisual (column order flipped visually) + tblPr = doc_out.tables[0]._tbl.find(f"{{{W_NS}}}tblPr") + assert tblPr is not None + assert tblPr.find(f"{{{W_NS}}}bidiVisual") is not None + # Schema order: w:bidiVisual after tblStyle, before tblW/tblLook + tbl_tags = [child.tag for child in tblPr] + if f"{{{W_NS}}}tblStyle" in tbl_tags: + assert tbl_tags.index(f"{{{W_NS}}}tblStyle") < tbl_tags.index( + f"{{{W_NS}}}bidiVisual" + ) + for later in ("w:tblW", "w:tblLook"): + tag = f"{{{W_NS}}}{later}" + if tag in tbl_tags: + assert tbl_tags.index(f"{{{W_NS}}}bidiVisual") < tbl_tags.index( + tag + ), later + + # Section: w:bidi (page layout direction) + assert doc_out.sections[0]._sectPr.find(f"{{{W_NS}}}bidi") is not None + + # Header paragraph: w:bidi + header_para = doc_out.sections[0].header.paragraphs[0] + assert _word_paragraph_has_bidi(header_para._p) + + # Text box: w:bidi on the paragraph inside w:txbxContent + doc_root = etree.fromstring(_read_zip_entry(output_file, "word/document.xml")) + txbx = doc_root.find(f".//{{{W_NS}}}txbxContent") + assert txbx is not None, "text box part missing from output" + box_para = txbx.find(f"{{{W_NS}}}p") + assert _word_paragraph_has_bidi(box_para) + + # Footnote: w:bidi on every paragraph of footnotes.xml, w:rtl on + # every text run, and the translated text made it through the + # ZIP write-back. + foot_root = etree.fromstring( + _read_zip_entry(output_file, "word/footnotes.xml") + ) + foot_paras = list(foot_root.iter(f"{{{W_NS}}}p")) + assert foot_paras, "footnotes.xml has no paragraphs" + for p in foot_paras: + assert _word_paragraph_has_bidi(p) + for r in foot_root.iter(f"{{{W_NS}}}r"): + if r.find(f"{{{W_NS}}}t") is None: + continue + rPr = r.find(f"{{{W_NS}}}rPr") + assert rPr is not None, "footnote run has no rPr" + assert rPr.find(f"{{{W_NS}}}rtl") is not None, "footnote run has no w:rtl" + foot_text = "".join( + t.text or "" for t in foot_root.iter(f"{{{W_NS}}}t") + ) + assert "FA_Note" in foot_text + + def test_fa_target_endnotes_and_comments_rtl(self, tmp_path): + """Endnotes and comments (separate ZIP parts) receive w:bidi on + their paragraphs, w:rtl on their runs and w:bidiVisual on their + tables when the target is RTL.""" + provider = MockTranslationProvider( + { + "Body text": "FA_Body", + "Endnote content": "FA_Endnote", + "Endnote cell": "FA_EndCell", + "Comment content": "FA_Comment", + "Comment cell": "FA_ComCell", + } + ) + translator = WordTranslator(provider=provider) + + doc = Document() + doc.add_paragraph("Body text") + input_file = tmp_path / "input.docx" + doc.save(input_file) + _inject_part(input_file, "endnotes", _ENDNOTES_PART, "rIdRtlEndnotes") + _inject_part(input_file, "comments", _COMMENTS_PART, "rIdRtlComments") + + output_file = tmp_path / "output.docx" + translator.translate_file(input_file, output_file, "fa") + + for part_name, marker in ( + ("word/endnotes.xml", "FA_Endnote"), + ("word/comments.xml", "FA_Comment"), + ): + root = etree.fromstring(_read_zip_entry(output_file, part_name)) + + # Every paragraph: w:bidi + paras = list(root.iter(f"{{{W_NS}}}p")) + assert paras, f"{part_name} has no paragraphs" + for p in paras: + assert _word_paragraph_has_bidi(p), part_name + + # Every text run: w:rPr/w:rtl + text_runs = [ + r + for r in root.iter(f"{{{W_NS}}}r") + if r.find(f"{{{W_NS}}}t") is not None + ] + assert text_runs, f"{part_name} has no text runs" + for r in text_runs: + rPr = r.find(f"{{{W_NS}}}rPr") + assert rPr is not None, f"{part_name}: run has no rPr" + assert rPr.find(f"{{{W_NS}}}rtl") is not None, part_name + + # Tables inside the part: w:bidiVisual + tables = list(root.iter(f"{{{W_NS}}}tbl")) + assert tables, f"{part_name} should contain a table" + for tbl in tables: + tblPr = tbl.find(f"{{{W_NS}}}tblPr") + assert tblPr is not None, part_name + assert tblPr.find(f"{{{W_NS}}}bidiVisual") is not None, part_name + + # The translation itself made it through the write-back + part_text = "".join( + t.text or "" for t in root.iter(f"{{{W_NS}}}t") + ) + assert marker in part_text, part_name + + def test_fr_target_no_rtl_attributes(self, tmp_path): + """A non-RTL target produces no bidi/bidiVisual/rtl markup.""" + provider = MockTranslationProvider({"Body text": "FR_Body"}) + translator = WordTranslator(provider=provider) + + doc = Document() + doc.add_paragraph("Body text") + table = doc.add_table(rows=1, cols=2) + table.cell(0, 0).text = "Cell" + + input_file = tmp_path / "input.docx" + output_file = tmp_path / "output.docx" + doc.save(input_file) + _inject_footnotes_part(input_file) + _inject_part(input_file, "endnotes", _ENDNOTES_PART, "rIdRtlEndnotes") + _inject_part(input_file, "comments", _COMMENTS_PART, "rIdRtlComments") + + translator.translate_file(input_file, output_file, "fr") + + doc_xml = _read_zip_entry(output_file, "word/document.xml").decode("utf-8") + assert "w:bidi" not in doc_xml # also covers bidiVisual + assert "w:rtl" not in doc_xml + + for part_name in ( + "word/footnotes.xml", + "word/endnotes.xml", + "word/comments.xml", + ): + part_xml = _read_zip_entry(output_file, part_name).decode("utf-8") + assert "w:bidi" not in part_xml, part_name + assert "w:rtl" not in part_xml, part_name + + +# --------------------------------------------------------------------------- +# Word — complex-script font hints (CS_FONTS coverage) +# --------------------------------------------------------------------------- + + +class TestWordCsFontHints: + @pytest.mark.parametrize("code", sorted(CS_FONTS)) + def test_every_cs_language_gets_arial_hint(self, code): + """Each complex-script target must resolve to an Arial cs hint — + pinning every CS_FONTS entry (dropping one must fail here).""" + eastasia, cs = _font_hints_for_target(code) + assert cs == "Arial", code + + def test_cs_fonts_covers_every_rtl_language(self): + """The CS hint map must cover the full RTL language set, not just + the original four entries.""" + assert set(CS_FONTS) >= set(RTL_LANGUAGES) + + +# --------------------------------------------------------------------------- +# PowerPoint — rtl attr, alignment policy, notes, cs font hint +# --------------------------------------------------------------------------- + + +class TestPptxRtlLayout: + def _build_single_text_pptx(self, tmp_path, text, alignment=None): + prs = Presentation() + slide = prs.slides.add_slide(prs.slide_layouts[0]) + textbox = slide.shapes.add_textbox( + Inches(1), Inches(1), Inches(6), Inches(1) + ) + para = textbox.text_frame.paragraphs[0] + para.text = text + if alignment is not None: + para.alignment = alignment + input_file = tmp_path / "input.pptx" + prs.save(str(input_file)) + return input_file + + def test_centered_title_keeps_alignment(self, tmp_path): + provider = MockTranslationProvider({"Centered title": "AR_Title"}) + translator = PowerPointTranslator(provider=provider) + input_file = self._build_single_text_pptx( + tmp_path, "Centered title", PP_ALIGN.CENTER + ) + output_file = tmp_path / "out.pptx" + translator.translate_file(input_file, output_file, "ar") + + prs_out = Presentation(str(output_file)) + para = _find_pptx_paragraph(prs_out, "AR_Title") + assert para is not None + pPr = para._p.find(f"{{{A_NS}}}pPr") + assert pPr is not None + assert pPr.get("rtl") == "1" + assert pPr.get("algn") == "ctr", "centered alignment must be preserved" + + def test_justified_title_keeps_alignment(self, tmp_path): + provider = MockTranslationProvider({"Justified text": "AR_Just"}) + translator = PowerPointTranslator(provider=provider) + input_file = self._build_single_text_pptx( + tmp_path, "Justified text", PP_ALIGN.JUSTIFY + ) + output_file = tmp_path / "out.pptx" + translator.translate_file(input_file, output_file, "ar") + + para = _find_pptx_paragraph(Presentation(str(output_file)), "AR_Just") + assert para is not None + pPr = para._p.find(f"{{{A_NS}}}pPr") + assert pPr is not None + assert pPr.get("rtl") == "1" + assert pPr.get("algn") == "just", "justified alignment must be preserved" + + def test_alignment_absent_becomes_right(self, tmp_path): + provider = MockTranslationProvider({"Plain text": "AR_Plain"}) + translator = PowerPointTranslator(provider=provider) + input_file = self._build_single_text_pptx(tmp_path, "Plain text") + output_file = tmp_path / "out.pptx" + translator.translate_file(input_file, output_file, "ar") + + para = _find_pptx_paragraph(Presentation(str(output_file)), "AR_Plain") + assert para is not None + pPr = para._p.find(f"{{{A_NS}}}pPr") + assert pPr is not None + assert pPr.get("rtl") == "1" + assert pPr.get("algn") == "r" + + def test_alignment_left_becomes_right(self, tmp_path): + provider = MockTranslationProvider({"Left text": "AR_Left"}) + translator = PowerPointTranslator(provider=provider) + input_file = self._build_single_text_pptx( + tmp_path, "Left text", PP_ALIGN.LEFT + ) + output_file = tmp_path / "out.pptx" + translator.translate_file(input_file, output_file, "ar") + + para = _find_pptx_paragraph(Presentation(str(output_file)), "AR_Left") + assert para is not None + pPr = para._p.find(f"{{{A_NS}}}pPr") + assert pPr.get("algn") == "r" + + def test_speaker_notes_rtl(self, tmp_path): + provider = MockTranslationProvider({"Speaker notes": "AR_Notes"}) + translator = PowerPointTranslator(provider=provider) + + prs = Presentation() + slide = prs.slides.add_slide(prs.slide_layouts[0]) + slide.notes_slide.notes_text_frame.text = "Speaker notes" + slide.shapes.add_textbox(Inches(1), Inches(1), Inches(4), Inches(1)) + + input_file = tmp_path / "input.pptx" + output_file = tmp_path / "out.pptx" + prs.save(str(input_file)) + translator.translate_file(input_file, output_file, "ar") + + prs_out = Presentation(str(output_file)) + slide_out = prs_out.slides[0] + assert slide_out.has_notes_slide + notes_para = slide_out.notes_slide.notes_text_frame.paragraphs[0] + assert "AR_Notes" in notes_para.text + pPr = notes_para._p.find(f"{{{A_NS}}}pPr") + assert pPr is not None + assert pPr.get("rtl") == "1" + # Notes runs get the complex-script hint too + for run in notes_para.runs: + rPr = run._r.find(f"{{{A_NS}}}rPr") + assert rPr is not None, "rPr missing on notes run" + cs = rPr.find(f"{{{A_NS}}}cs") + assert cs is not None, " hint missing on notes run" + assert cs.get("typeface") == "Arial" + + def test_cs_font_hint_for_rtl_target(self, tmp_path): + provider = MockTranslationProvider({"Hello": "AR_Hello"}) + translator = PowerPointTranslator(provider=provider) + input_file = self._build_single_text_pptx(tmp_path, "Hello") + output_file = tmp_path / "out.pptx" + translator.translate_file(input_file, output_file, "ar") + + para = _find_pptx_paragraph(Presentation(str(output_file)), "AR_Hello") + assert para is not None + for run in para.runs: + rPr = run._r.find(f"{{{A_NS}}}rPr") + assert rPr is not None, "rPr missing on run" + cs = rPr.find(f"{{{A_NS}}}cs") + assert cs is not None, " hint missing on run" + assert cs.get("typeface") == "Arial" + + def test_fr_target_no_rtl_attributes(self, tmp_path): + provider = MockTranslationProvider({"Hello": "FR_Hello"}) + translator = PowerPointTranslator(provider=provider) + input_file = self._build_single_text_pptx(tmp_path, "Hello") + output_file = tmp_path / "out.pptx" + translator.translate_file(input_file, output_file, "fr") + + prs_out = Presentation(str(output_file)) + para = _find_pptx_paragraph(prs_out, "FR_Hello") + assert para is not None + pPr = para._p.find(f"{{{A_NS}}}pPr") + assert pPr is None or pPr.get("rtl") is None + for run in para.runs: + rPr = run._r.find(f"{{{A_NS}}}rPr") + if rPr is not None: + assert rPr.find(f"{{{A_NS}}}cs") is None + + +# --------------------------------------------------------------------------- +# Excel — sheet view direction +# --------------------------------------------------------------------------- + + +class TestExcelRtlLayout: + def _build_workbook(self, tmp_path): + wb = Workbook() + wb.active["A1"] = "Alpha" + wb.create_sheet("Data2")["A1"] = "Beta" + wb.create_sheet("Data3")["A1"] = "Gamma" + input_file = tmp_path / "input.xlsx" + wb.save(input_file) + return input_file + + def test_he_target_all_sheets_right_to_left(self, tmp_path): + provider = MockTranslationProvider( + {"Alpha": "HE_Alpha", "Beta": "HE_Beta", "Gamma": "HE_Gamma"} + ) + translator = ExcelTranslator(provider=provider) + input_file = self._build_workbook(tmp_path) + output_file = tmp_path / "out.xlsx" + translator.translate_file(input_file, output_file, "he") + + out = load_workbook(output_file) + assert len(out.sheetnames) == 3 + for name in out.sheetnames: + assert out[name].sheet_view.rightToLeft is True, name + + def test_fr_target_sheets_unchanged(self, tmp_path): + provider = MockTranslationProvider( + {"Alpha": "FR_Alpha", "Beta": "FR_Beta", "Gamma": "FR_Gamma"} + ) + translator = ExcelTranslator(provider=provider) + input_file = self._build_workbook(tmp_path) + output_file = tmp_path / "out.xlsx" + translator.translate_file(input_file, output_file, "fr") + + out = load_workbook(output_file) + for name in out.sheetnames: + assert not out[name].sheet_view.rightToLeft, name + + def test_chartsheet_does_not_break_rtl_translation(self, tmp_path): + """A workbook containing a chartsheet (no sheet_view, no cell grid) + must still translate, flip the worksheet views, and keep the + chartsheet intact in the output.""" + from openpyxl.chart import BarChart, Reference + from openpyxl.chartsheet import Chartsheet + + provider = MockTranslationProvider({"Alpha": "HE_Alpha"}) + translator = ExcelTranslator(provider=provider) + + wb = Workbook() + wb.active["A1"] = "Alpha" + data_sheet = wb.create_sheet("Data2") + data_sheet["B2"] = 42 + chart = BarChart() + chart.add_data(Reference(data_sheet, min_col=2, min_row=2, max_row=5)) + wb.create_chartsheet("ChartSheet").add_chart(chart) + + input_file = tmp_path / "input.xlsx" + wb.save(input_file) + output_file = tmp_path / "out.xlsx" + + # Must not raise (the chartsheet has neither sheet_view nor cells) + translator.translate_file(input_file, output_file, "he") + + out = load_workbook(output_file) + chartsheets = [ + name for name in out.sheetnames if isinstance(out[name], Chartsheet) + ] + assert chartsheets, "chartsheet lost during translation" + worksheets = [ + name for name in out.sheetnames if not isinstance(out[name], Chartsheet) + ] + assert worksheets, "no worksheet found in output" + for name in worksheets: + assert out[name].sheet_view.rightToLeft is True, name + + +# --------------------------------------------------------------------------- +# PDF — shaping, font selection, clean-PDF mode +# --------------------------------------------------------------------------- + + +class TestPdfRtlShaping: + def test_shape_rtl_produces_presentation_forms(self): + shaped = _shape_rtl("سلام") + assert shaped != "سلام" + assert any(0xFB50 <= ord(c) <= 0xFEFF for c in shaped) + + def test_shape_rtl_latin_passthrough(self): + assert _shape_rtl("Hello world") == "Hello world" + + def test_shape_rtl_empty(self): + assert _shape_rtl("") == "" + + def test_shape_rtl_missing_libraries_fallback(self, monkeypatch): + """Missing arabic-reshaper / python-bidi must degrade to the raw + text (warning logged), never raise.""" + import translators.pdf_translator as pdf_mod + + # Reset the once-only warning flag so this test neither depends + # on nor leaks into the execution order of other tests. + monkeypatch.setattr(pdf_mod, "_RTL_SHAPING_MISSING_WARNED", False) + monkeypatch.setitem(sys.modules, "arabic_reshaper", None) + monkeypatch.setitem(sys.modules, "bidi", None) + monkeypatch.setitem(sys.modules, "bidi.algorithm", None) + text = "سلام دنیا" + assert _shape_rtl(text) == text + + +class _FakePage: + """Minimal fitz.Page double: records every insert_textbox call.""" + + def __init__(self): + import fitz + + self._fitz = fitz + self.rect = fitz.Rect(0, 0, 612, 792) + self.calls = [] + + def insert_textbox(self, rect, text, **kwargs): + self.calls.append({"text": text, **kwargs}) + return 0 # text fits + + +class TestPdfLayoutModeRtl: + def _block(self, translated): + return { + "bbox": (72, 72, 300, 100), + "text": "orig", + "translated": translated, + "font_size": 12.0, + "color": 0, + "is_bold": False, + "is_italic": False, + } + + def test_rtl_block_is_shaped_and_right_aligned(self): + import fitz + + translator = PDFTranslator() + page = _FakePage() + ok = translator._write_translated_block( + page, self._block("سلام دنیا"), font_path=None, rtl_target=True + ) + assert ok is True + assert page.calls, "no insert_textbox call recorded" + call = page.calls[0] + assert call["text"] != "سلام دنیا", "Arabic text must be shaped" + assert any(0xFB50 <= ord(c) <= 0xFEFF for c in call["text"]) + assert call["align"] == fitz.TEXT_ALIGN_RIGHT + + def test_ltr_block_is_not_shaped(self): + translator = PDFTranslator() + page = _FakePage() + ok = translator._write_translated_block( + page, self._block("Bonjour le monde"), font_path=None, rtl_target=False + ) + assert ok is True + assert page.calls[0]["text"] == "Bonjour le monde" + + +class TestPdfFontSelection: + def test_arabic_script_target_prefers_arabic_font(self, tmp_path, monkeypatch): + fake_arabic = tmp_path / "NotoNaskhArabic-Regular.ttf" + fake_arabic.write_bytes(b"fake") + fake_hebrew = tmp_path / "NotoSansHebrew-Regular.ttf" + fake_hebrew.write_bytes(b"fake") + fake_generic = tmp_path / "NotoSans-Regular.ttf" + fake_generic.write_bytes(b"fake") + + translator = PDFTranslator() + monkeypatch.setattr( + translator, "_RTL_ARABIC_FONT_SEARCH_PATHS", [str(fake_arabic)] + ) + monkeypatch.setattr( + translator, "_RTL_HEBREW_FONT_SEARCH_PATHS", [str(fake_hebrew)] + ) + monkeypatch.setattr(translator, "_FONT_SEARCH_PATHS", [str(fake_generic)]) + assert translator._get_font_path("fa") == str(fake_arabic) + assert translator._get_font_path("ar-EG") == str(fake_arabic) + assert translator._get_font_path("ckb") == str(fake_arabic) + + def test_hebrew_target_prefers_hebrew_font(self, tmp_path, monkeypatch): + """A Hebrew target must NOT pick the Arabic font (tofu) — it + searches the Hebrew list first.""" + fake_arabic = tmp_path / "NotoNaskhArabic-Regular.ttf" + fake_arabic.write_bytes(b"fake") + fake_hebrew = tmp_path / "NotoSansHebrew-Regular.ttf" + fake_hebrew.write_bytes(b"fake") + fake_generic = tmp_path / "NotoSans-Regular.ttf" + fake_generic.write_bytes(b"fake") + + translator = PDFTranslator() + monkeypatch.setattr( + translator, "_RTL_ARABIC_FONT_SEARCH_PATHS", [str(fake_arabic)] + ) + monkeypatch.setattr( + translator, "_RTL_HEBREW_FONT_SEARCH_PATHS", [str(fake_hebrew)] + ) + monkeypatch.setattr(translator, "_FONT_SEARCH_PATHS", [str(fake_generic)]) + assert translator._get_font_path("he") == str(fake_hebrew) + assert translator._get_font_path("yi") == str(fake_hebrew) + + def test_hebrew_target_falls_back_to_generic_font(self, tmp_path, monkeypatch): + fake_generic = tmp_path / "NotoSans-Regular.ttf" + fake_generic.write_bytes(b"fake") + + translator = PDFTranslator() + monkeypatch.setattr(translator, "_RTL_ARABIC_FONT_SEARCH_PATHS", []) + monkeypatch.setattr(translator, "_RTL_HEBREW_FONT_SEARCH_PATHS", []) + monkeypatch.setattr(translator, "_FONT_SEARCH_PATHS", [str(fake_generic)]) + # Documented degradation: generic font + warning, never a failure. + assert translator._get_font_path("he") == str(fake_generic) + + def test_arabic_target_falls_back_to_generic_font(self, tmp_path, monkeypatch): + fake_generic = tmp_path / "NotoSans-Regular.ttf" + fake_generic.write_bytes(b"fake") + + translator = PDFTranslator() + monkeypatch.setattr(translator, "_RTL_ARABIC_FONT_SEARCH_PATHS", []) + monkeypatch.setattr(translator, "_RTL_HEBREW_FONT_SEARCH_PATHS", []) + monkeypatch.setattr(translator, "_FONT_SEARCH_PATHS", [str(fake_generic)]) + assert translator._get_font_path("ar") == str(fake_generic) + + def test_thana_target_uses_generic_list_directly(self, tmp_path, monkeypatch): + """dv (Thaana) has no dedicated font list — generic list directly, + never the Arabic/Hebrew ones.""" + fake_arabic = tmp_path / "NotoNaskhArabic-Regular.ttf" + fake_arabic.write_bytes(b"fake") + fake_hebrew = tmp_path / "NotoSansHebrew-Regular.ttf" + fake_hebrew.write_bytes(b"fake") + fake_generic = tmp_path / "NotoSans-Regular.ttf" + fake_generic.write_bytes(b"fake") + + translator = PDFTranslator() + monkeypatch.setattr( + translator, "_RTL_ARABIC_FONT_SEARCH_PATHS", [str(fake_arabic)] + ) + monkeypatch.setattr( + translator, "_RTL_HEBREW_FONT_SEARCH_PATHS", [str(fake_hebrew)] + ) + monkeypatch.setattr(translator, "_FONT_SEARCH_PATHS", [str(fake_generic)]) + assert translator._get_font_path("dv") == str(fake_generic) + + def test_non_rtl_target_ignores_rtl_only_fonts(self, tmp_path, monkeypatch): + fake_arabic = tmp_path / "NotoNaskhArabic-Regular.ttf" + fake_arabic.write_bytes(b"fake") + + translator = PDFTranslator() + monkeypatch.setattr( + translator, "_RTL_ARABIC_FONT_SEARCH_PATHS", [str(fake_arabic)] + ) + monkeypatch.setattr(translator, "_FONT_SEARCH_PATHS", []) + assert translator._get_font_path("fr") is None + + +class TestPdfFontRegistration: + def test_register_rtl_font_names_unique_per_path(self, tmp_path, monkeypatch): + """Two different fonts (Arabic then Hebrew) must get distinct + reportlab names; the same path registers only once.""" + import reportlab.pdfbase.pdfmetrics as pdfmetrics_mod + import reportlab.pdfbase.ttfonts as ttf_mod + + import translators.pdf_translator as pdf_mod + + registered = [] + + class FakeTTFont: + def __init__(self, name, path): + self.fontName = name + registered.append((name, path)) + + # Only the naming/registry logic is under test here — neutralize + # reportlab's own bookkeeping. + monkeypatch.setattr(ttf_mod, "TTFont", FakeTTFont) + monkeypatch.setattr(pdfmetrics_mod, "registerFont", lambda font: None) + monkeypatch.setattr(pdf_mod, "_RTL_FONT_REGISTRATIONS", {}) + + font_a = tmp_path / "hebrew.ttf" + font_a.write_bytes(b"x") + font_b = tmp_path / "arabic.ttf" + font_b.write_bytes(b"y") + + name_a1 = pdf_mod._register_rtl_font(str(font_a)) + name_a2 = pdf_mod._register_rtl_font(str(font_a)) + name_b = pdf_mod._register_rtl_font(str(font_b)) + + assert name_a1 == name_a2, "same path must reuse its registration" + assert name_a1 != name_b, "different fonts must not share a name" + assert name_a1.startswith("WordlyRTL-") + assert len(registered) == 2, "same path must be registered once" + + +class TestPdfCleanModeRtl: + def _spy_on_body_style(self, monkeypatch): + import reportlab.lib.styles as rls + + captured = {} + original = rls.ParagraphStyle + + class SpyStyle(original): + def __init__(self, name, **kwargs): + super().__init__(name, **kwargs) + if name == "BodyText_Custom": + captured.update(kwargs) + + monkeypatch.setattr(rls, "ParagraphStyle", SpyStyle) + return captured + + def test_rtl_style_wordwrap_and_alignment(self, tmp_path, monkeypatch): + captured = self._spy_on_body_style(monkeypatch) + monkeypatch.setattr(PDFTranslator, "_RTL_ARABIC_FONT_SEARCH_PATHS", []) + monkeypatch.setattr(PDFTranslator, "_RTL_HEBREW_FONT_SEARCH_PATHS", []) + monkeypatch.setattr(PDFTranslator, "_FONT_SEARCH_PATHS", []) + + from reportlab.lib.enums import TA_RIGHT + + translator = PDFTranslator() + out = tmp_path / "clean_fa.pdf" + translator._generate_clean_pdf(["سلام دنیا"], out, "fa") + + assert out.exists() + assert captured.get("wordWrap") == "RTL" + assert captured.get("alignment") == TA_RIGHT + # No Arabic-capable font on this (fake) host → Helvetica fallback + assert captured.get("fontName") == "Helvetica" + + def test_ltr_style_unchanged(self, tmp_path, monkeypatch): + captured = self._spy_on_body_style(monkeypatch) + + from reportlab.lib.enums import TA_JUSTIFY + + translator = PDFTranslator() + out = tmp_path / "clean_fr.pdf" + translator._generate_clean_pdf(["Bonjour le monde"], out, "fr") + + assert out.exists() + assert captured.get("wordWrap") is None + assert captured.get("alignment") == TA_JUSTIFY + assert captured.get("fontName") == "Helvetica" + + @pytest.mark.skipif(_host_rtl_font() is None, reason="no Arabic-capable font on this host") + def test_rtl_real_font_registered_and_shaped(self, tmp_path): + """End-to-end: shaped Arabic rendered with an Arabic-capable TTF.""" + translator = PDFTranslator() + font_path = translator._get_font_path("fa") + assert font_path is not None + + out = tmp_path / "clean_fa_real.pdf" + translator._generate_clean_pdf(["سلام دنیا این یک آزمایش است"], out, "fa") + + import fitz + + doc = fitz.open(str(out)) + extracted = doc[0].get_text() + doc.close() + # Presentation forms in the output prove both the shaping and the + # Arabic-capable font reached the PDF (Helvetica cannot encode them). + assert any(0xFB50 <= ord(c) <= 0xFEFF for c in extracted) diff --git a/translators/excel_translator.py b/translators/excel_translator.py index 4fc7229..2804135 100644 --- a/translators/excel_translator.py +++ b/translators/excel_translator.py @@ -23,6 +23,7 @@ from openpyxl.cell.cell import Cell from openpyxl.utils import get_column_letter from services.providers.base import TranslationProvider +from core.languages import is_rtl from core.logging import get_logger @@ -354,6 +355,26 @@ class ExcelTranslator: except Exception as e: _log_error("excel_sheet_images_failed", sheet_name=sheet_name, error=str(e)) + # RTL targets: flip the sheet view so every sheet reads + # right-to-left (column A renders rightmost). Content and + # formatting are untouched — only the reading direction. + # Chartsheets have no `sheet_view` and must never break the + # save: each sheet is guarded individually. + if is_rtl(target_language): + flipped = 0 + for sheet_name in workbook.sheetnames: + ws = workbook[sheet_name] + try: + if getattr(ws, "sheet_view", None) is not None: + ws.sheet_view.rightToLeft = True + flipped += 1 + except Exception as e: + _log_error( + "excel_rtl_view_failed", sheet=sheet_name, error=str(e) + ) + if flipped: + _log_info("excel_rtl_view_applied", sheets=flipped) + try: workbook.save(output_path) except Exception as e: @@ -580,7 +601,14 @@ class ExcelTranslator: ) -> None: """Collect all translatable text from worksheet cells, cell comments, and cell hyperlinks (URLs are preserved as-is, but their display - labels are translated).""" + labels are translated). + + Chartsheets are skipped: they have no cell grid (no iter_rows) + — their text lives in the chart XML, collected separately from + the ZIP. + """ + if not hasattr(worksheet, "iter_rows"): + return for row in worksheet.iter_rows(): for cell in row: if cell.value is not None: diff --git a/translators/pdf_translator.py b/translators/pdf_translator.py index a34283f..1dd2dd3 100644 --- a/translators/pdf_translator.py +++ b/translators/pdf_translator.py @@ -33,10 +33,12 @@ Scanned PDFs: import time import shutil import subprocess +import zlib from pathlib import Path from typing import Dict, Any, Optional, Callable, List from core.logging import get_logger +from core.languages import is_rtl logger = get_logger(__name__) @@ -60,8 +62,70 @@ HEADING_MIN_SCALE = 0.90 # Maximum font shrink for body text BODY_MIN_SCALE = 0.75 -# RTL language codes -RTL_LANGUAGES = frozenset({"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"}) +# Warn at most once when the RTL shaping libraries are missing. +_RTL_SHAPING_MISSING_WARNED = False + + +def _shape_rtl(text: str) -> str: + """Shape Arabic-script text for PDF engines that do no bidi of their own. + + arabic-reshaper joins the letters into their contextual forms and + python-bidi reorders the string for visual (left-to-right) rendering. + Both libraries are optional on purpose: when they are missing or + fail, the text is returned unchanged with a warning — the + translation itself must always succeed, only the rendering degrades. + """ + global _RTL_SHAPING_MISSING_WARNED + if not text: + return text + try: + import arabic_reshaper + from bidi.algorithm import get_display + except Exception: + if not _RTL_SHAPING_MISSING_WARNED: + logger.warning( + "rtl_shaping_libs_missing", + hint="pip install arabic-reshaper python-bidi for correct RTL rendering", + ) + _RTL_SHAPING_MISSING_WARNED = True + return text + try: + return get_display(arabic_reshaper.reshape(text)) + except Exception as e: + logger.warning("rtl_shaping_failed", error=str(e)[:200]) + return text + + +# Font path → reportlab font name, for RTL fonts registered in this +# process. A fixed name would make two consecutive translations with +# different fonts (e.g. Arabic then Hebrew) overwrite each other's +# registration. +_RTL_FONT_REGISTRATIONS: Dict[str, str] = {} + + +def _register_rtl_font(font_path: str) -> str: + """Register a TTF under a stable per-path name, once per process. + + Returns the reportlab font name, or "Helvetica" when registration + fails (warning logged — a font problem must never fail the job). + """ + registered = _RTL_FONT_REGISTRATIONS.get(font_path) + if registered: + return registered + try: + from reportlab.pdfbase import pdfmetrics + from reportlab.pdfbase.ttfonts import TTFont + + digest = format(zlib.crc32(str(font_path).encode("utf-8")) & 0xFFFFFFFF, "08x") + name = f"WordlyRTL-{digest}" + pdfmetrics.registerFont(TTFont(name, font_path)) + _RTL_FONT_REGISTRATIONS[font_path] = name + return name + except Exception as e: + logger.warning( + "pdf_rtl_font_register_failed", error=str(e)[:200], font=font_path + ) + return "Helvetica" def _record_format_loss_metric(element_type: str, count: int = 1) -> None: @@ -109,6 +173,29 @@ def _libreoffice_available() -> bool: _libreoffice_available._cache = None # type: ignore[attr-defined] +# Script groups among the RTL languages — the preferred font depends on +# the writing system, not just the direction: a Hebrew target would +# render as tofu with an Arabic-only font. +_ARABIC_SCRIPT_LANGUAGES = frozenset( + {"ar", "fa", "ur", "ps", "ku", "sd", "ug", "ckb"} +) +_HEBREW_SCRIPT_LANGUAGES = frozenset({"he", "yi"}) + + +def _rtl_script(code: str) -> str: + """Script group of an RTL language code: "arabic", "hebrew" or "" + + ("" meaning: use the generic Unicode font list — e.g. Thaana "dv", + for which no dedicated font is searched). + """ + base = (code or "").strip().replace("_", "-").split("-")[0].lower() + if base in _ARABIC_SCRIPT_LANGUAGES: + return "arabic" + if base in _HEBREW_SCRIPT_LANGUAGES: + return "hebrew" + return "" + + class PDFTranslator: """Translates PDF files with layout preservation using PyMuPDF.""" @@ -133,9 +220,37 @@ class PDFTranslator: "/System/Library/Fonts/Helvetica.ttc", ] + # Arabic-script-capable fonts (ar, fa, ur, ps, ku, sd, ug, ckb), + # searched FIRST for those targets. Debian's fonts-noto package + # ships Noto Naskh Arabic and Noto Sans Arabic under + # /usr/share/fonts/truetype/noto/; on Windows, Arial covers the + # whole Arabic script. + _RTL_ARABIC_FONT_SEARCH_PATHS = [ + "/usr/share/fonts/truetype/noto/NotoNaskhArabic-Regular.ttf", + "/usr/share/fonts/truetype/noto/NotoSansArabic-Regular.ttf", + "/usr/share/fonts/opentype/noto/NotoNaskhArabic-Regular.ttf", + "/usr/share/fonts/opentype/noto/NotoSansArabic-Regular.ttf", + "/app/fonts/NotoNaskhArabic-Regular.ttf", + "/app/fonts/NotoSansArabic-Regular.ttf", + "C:/Windows/Fonts/arial.ttf", + ] + + # Hebrew-script-capable fonts (he, yi), searched FIRST for those + # targets. Same Debian package provides Noto Sans Hebrew; Arial + # covers Hebrew on Windows. + _RTL_HEBREW_FONT_SEARCH_PATHS = [ + "/usr/share/fonts/truetype/noto/NotoSansHebrew-Regular.ttf", + "/usr/share/fonts/opentype/noto/NotoSansHebrew-Regular.ttf", + "/app/fonts/NotoSansHebrew-Regular.ttf", + "C:/Windows/Fonts/arial.ttf", + ] + def __init__(self, provider=None): self._provider = provider self._font_path: Optional[str] = None + # script group ("arabic"/"hebrew"/"") → resolved font path, so a + # single instance can serve Arabic and Hebrew jobs in a row. + self._rtl_font_paths: Dict[str, str] = {} self._translation_stats = {"attempted": 0, "changed": 0} self._custom_prompt: Optional[str] = None # OCR overrides (admin settings); None → fall back to config.MISTRAL_* @@ -184,14 +299,56 @@ class PDFTranslator: recorder = getattr(self, "_segment_recorder", None) return recorder.get_pairs() if recorder is not None else [] - def _get_font_path(self) -> Optional[str]: - """Resolve a Unicode-capable TTF/OTF font file.""" - if self._font_path is not None: - return self._font_path - for p in self._FONT_SEARCH_PATHS: + @staticmethod + def _first_existing_font(paths) -> Optional[str]: + for p in paths: if Path(p).exists(): - self._font_path = p return p + return None + + def _get_font_path(self, target_language: str = "") -> Optional[str]: + """Resolve a Unicode-capable TTF/OTF font file. + + RTL targets first search a font covering THEIR script — Noto + Naskh/Sans Arabic for the Arabic script (ar, fa, ur…), Noto Sans + Hebrew for Hebrew (he, yi); other RTL scripts (Thaana "dv") use + the generic Unicode list directly. A missing script font falls + back to the generic list with a warning — a missing font + degrades the rendering but never fails the job. Non-RTL targets + use the generic list only (unchanged behavior). + """ + if not is_rtl(target_language): + if self._font_path: + return self._font_path + found = self._first_existing_font(self._FONT_SEARCH_PATHS) + if not found: + logger.warning("no_unicode_font_found") + return None + self._font_path = found + return found + + script = _rtl_script(target_language) + cached = self._rtl_font_paths.get(script) + if cached: + return cached + + if script == "arabic": + preferred = self._RTL_ARABIC_FONT_SEARCH_PATHS + elif script == "hebrew": + preferred = self._RTL_HEBREW_FONT_SEARCH_PATHS + else: + preferred = [] + + found = self._first_existing_font(preferred + self._FONT_SEARCH_PATHS) + if found: + if preferred and found not in preferred: + logger.warning( + "rtl_script_font_not_found_current_font_used", + script=script, + font=found, + ) + self._rtl_font_paths[script] = found + return found logger.warning("no_unicode_font_found") return None @@ -259,7 +416,7 @@ class PDFTranslator: doc.close() raise RuntimeError("PDF has no pages.") - font_path = self._get_font_path() + font_path = self._get_font_path(target_language) logger.info( "pdf_layout_start", pages=total_pages, @@ -311,7 +468,7 @@ class PDFTranslator: """Core PyMuPDF in-place processing — one page at a time.""" import fitz - is_rtl = target_language.lower() in RTL_LANGUAGES + rtl_target = is_rtl(target_language) total_blocks = 0 translated_blocks = 0 @@ -514,7 +671,7 @@ class PDFTranslator: for block in blocks: if block.get("translated"): self._write_translated_block( - page, block, font_path, is_rtl + page, block, font_path, rtl_target ) if progress_callback: @@ -871,7 +1028,7 @@ class PDFTranslator: page, block: Dict, font_path: Optional[str], - is_rtl: bool, + rtl_target: bool, ) -> bool: """Write translated text into the block's bounding box. @@ -897,6 +1054,12 @@ class PDFTranslator: original_rect = fitz.Rect(block["bbox"]) translated = block["translated"] + if rtl_target: + # Shape Arabic-script text (contextual letter forms + visual + # order) before insertion — insert_textbox does no bidi + # processing of its own. Missing/failed libs degrade to the + # raw text (warning logged inside _shape_rtl). + translated = _shape_rtl(translated) target_size = block["font_size"] color = self._int_to_rgb(block["color"]) @@ -1443,8 +1606,20 @@ class PDFTranslator: from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak from reportlab.lib.styles import getSampleStyleSheet - is_rtl = target_language.lower() in RTL_LANGUAGES - alignment = TA_RIGHT if is_rtl else TA_JUSTIFY + rtl_target = is_rtl(target_language) + alignment = TA_RIGHT if rtl_target else TA_JUSTIFY + + # RTL targets need a TTF font covering the target script (the + # built-in Helvetica has no Arabic/Hebrew glyphs). Registration + # failures degrade to Helvetica with a warning — never a failed + # job. + font_name = "Helvetica" + if rtl_target: + rtl_font = self._get_font_path(target_language) + if rtl_font: + font_name = _register_rtl_font(rtl_font) + else: + logger.warning("pdf_rtl_font_missing_helvetica_fallback") styles = getSampleStyleSheet() @@ -1464,6 +1639,8 @@ class PDFTranslator: leading=16, spaceAfter=6, alignment=alignment, + fontName=font_name, + wordWrap="RTL" if rtl_target else None, textColor=colors.HexColor("#1a1a1a"), ) @@ -1490,6 +1667,11 @@ class PDFTranslator: elements.append(Spacer(1, 4)) continue + if rtl_target: + # Shape Arabic-script text before rendering — + # reportlab does no bidi processing of its own. + para_text = _shape_rtl(para_text) + safe = ( para_text .replace("&", "&") diff --git a/translators/pptx_translator.py b/translators/pptx_translator.py index 72e9dca..e2e2bcd 100644 --- a/translators/pptx_translator.py +++ b/translators/pptx_translator.py @@ -20,16 +20,30 @@ from pptx.shapes.group import GroupShape from pptx.enum.shapes import MSO_SHAPE_TYPE from services.providers.base import TranslationProvider +from core.languages import is_rtl # DrawingML namespace used by pptx XML _NS_A = "http://schemas.openxmlformats.org/drawingml/2006/main" -# Languages written right-to-left -RTL_LANGUAGES: frozenset = frozenset( - {"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"} +# a:rPr children that must come AFTER // per the +# DrawingML CT_TextCharacterProperties sequence. Inserting / +# before them keeps the XML schema-valid (PowerPoint repairs otherwise). +_A_RPR_HINT_SUCCESSORS = tuple( + f"{{{_NS_A}}}{tag}" + for tag in ("sym", "hlinkClick", "hlinkMouseOver", "rtl", "extLst") ) +def _insert_a_rpr_child(rPr, child) -> None: + """Insert child into a:rPr at its schema position (before sym, + hlink*, rtl and extLst — or appended at the end when none exist).""" + for existing in rPr: + if existing.tag in _A_RPR_HINT_SUCCESSORS: + existing.addprevious(child) + return + rPr.append(child) + + from core.logging import get_logger logger = get_logger(__name__) @@ -58,8 +72,10 @@ def _set_pptx_paragraph_rtl(paragraph) -> None: """ Enable RTL mode on a PowerPoint paragraph. - Sets rtl="1" and algn="r" on the element, which controls - both text direction and horizontal alignment in DrawingML. + Sets rtl="1" on the element, which controls the paragraph + text direction. The horizontal alignment (algn) is only forced to + "r" when the paragraph has no explicit alignment or is aligned + "l" (left) — a centered or justified title keeps its alignment. """ p_elem = paragraph._p tag_pPr = f"{{{_NS_A}}}pPr" @@ -68,14 +84,20 @@ def _set_pptx_paragraph_rtl(paragraph) -> None: pPr = etree.Element(tag_pPr) p_elem.insert(0, pPr) pPr.set("rtl", "1") - pPr.set("algn", "r") + if pPr.get("algn") in (None, "", "l"): + pPr.set("algn", "r") def _apply_rtl_to_presentation(presentation: Presentation) -> None: - """Apply RTL direction to every paragraph in all slides.""" + """Apply RTL direction to every paragraph in all slides and notes.""" for slide in presentation.slides: for shape in slide.shapes: _apply_rtl_to_shape(shape) + # Speaker notes are translated like any other text — they read + # right-to-left too when the target language does. + if slide.has_notes_slide and slide.notes_slide.notes_text_frame is not None: + for paragraph in slide.notes_slide.notes_text_frame.paragraphs: + _set_pptx_paragraph_rtl(paragraph) def _apply_rtl_to_shape(shape) -> None: @@ -111,11 +133,10 @@ def _ea_typeface_for_target(target_language: str): return _EA_TYPEFACES.get(code) or _EA_TYPEFACES.get(base) -def _apply_ea_font_hints(presentation: Presentation, target_language: str) -> None: - """Set the (east-asian) typeface on every run for CJK targets. - - Blanket application is safe — the hint only affects CJK glyphs. - """ +def _apply_run_typeface_hint( + presentation: Presentation, tag: str, typeface: str, log_event: str +) -> None: + """Set the typeface hint on every run of every slide shape.""" def _hint_shape(shape) -> int: hinted = 0 @@ -133,29 +154,59 @@ def _apply_ea_font_hints(presentation: Presentation, target_language: str) -> No def _hint_text_frame(text_frame) -> int: hinted = 0 tag_rPr = f"{{{_NS_A}}}rPr" - tag_ea = f"{{{_NS_A}}}ea" + tag_hint = f"{{{_NS_A}}}{tag}" for paragraph in text_frame.paragraphs: for run in paragraph.runs: rPr = run._r.find(tag_rPr) if rPr is None: - rPr = etree.SubElement(run._r, tag_rPr) - ea = rPr.find(tag_ea) - if ea is None: - ea = etree.SubElement(rPr, tag_ea) - if not ea.get("typeface"): - ea.set("typeface", typeface) + # a:rPr must be the FIRST child of a:r (before a:t). + rPr = etree.Element(tag_rPr) + run._r.insert(0, rPr) + hint = rPr.find(tag_hint) + if hint is None: + hint = etree.Element(tag_hint) + _insert_a_rpr_child(rPr, hint) + if not hint.get("typeface"): + hint.set("typeface", typeface) hinted += 1 return hinted - typeface = _ea_typeface_for_target(target_language) - if not typeface: - return total = 0 for slide in presentation.slides: for shape in slide.shapes: total += _hint_shape(shape) + # Speaker notes get the RTL direction — they need the typeface + # hint too. + if slide.has_notes_slide and slide.notes_slide.notes_text_frame is not None: + total += _hint_text_frame(slide.notes_slide.notes_text_frame) if total: - _log_info("pptx_ea_font_hints_applied", runs=total, typeface=typeface) + _log_info(log_event, runs=total, typeface=typeface) + + +def _apply_ea_font_hints(presentation: Presentation, target_language: str) -> None: + """Set the (east-asian) typeface on every run for CJK targets. + + Blanket application is safe — the hint only affects CJK glyphs. + """ + typeface = _ea_typeface_for_target(target_language) + if not typeface: + return + _apply_run_typeface_hint( + presentation, "ea", typeface, "pptx_ea_font_hints_applied" + ) + + +def _apply_cs_font_hints(presentation: Presentation, target_language: str) -> None: + """Set the (complex-script) typeface on every run for RTL targets. + + Blanket application is safe — the hint only affects Arabic-script + glyphs, and Arial covers the whole Arabic script on every platform. + """ + if not is_rtl(target_language): + return + _apply_run_typeface_hint( + presentation, "cs", "Arial", "pptx_cs_font_hints_applied" + ) class PptxProcessorError(Exception): @@ -397,13 +448,17 @@ class PowerPointTranslator: ) # Apply RTL layout when the target language is written right-to-left. - if target_language.lower() in RTL_LANGUAGES: + if is_rtl(target_language): _apply_rtl_to_presentation(presentation) # CJK font hint so the target script renders with a proper # typeface instead of shape-dependent fallbacks. _apply_ea_font_hints(presentation, target_language) + # Arabic-script font hint for RTL targets (same idea as the + # CJK hint, via the typeface). + _apply_cs_font_hints(presentation, target_language) + if translate_images: try: self._translate_images(presentation, target_language) diff --git a/translators/word_translator.py b/translators/word_translator.py index 0b46ff3..7827710 100644 --- a/translators/word_translator.py +++ b/translators/word_translator.py @@ -25,11 +25,7 @@ from docx.section import Section from lxml import etree from services.providers.base import TranslationProvider - -# Languages written right-to-left -RTL_LANGUAGES: frozenset = frozenset( - {"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"} -) +from core.languages import is_rtl # East-Asian / complex-script font hints: when the target language uses # glyphs a Latin theme font lacks, Word falls back to a substitute — @@ -47,6 +43,13 @@ CS_FONTS: dict = { "he": "Arial", "fa": "Arial", "ur": "Arial", + "ps": "Arial", + "ku": "Arial", + "sd": "Arial", + "ug": "Arial", + "yi": "Arial", + "dv": "Arial", + "ckb": "Arial", } @@ -123,6 +126,79 @@ def _log_error(event: str, **kwargs): logger.error(msg) +# --- OOXML schema order --------------------------------------------------- +# w:pPr, w:rPr and w:tblPr require their children in a FIXED order. +# Appending at the end produces out-of-order XML that Word "repairs" on +# open (with a repair prompt in some versions). Each RTL element is +# therefore inserted just before its first successor element, per the +# CT_PPr / CT_RPr / CT_TblPr sequences of the schema. + +# Elements that must come AFTER w:bidi inside w:pPr. +_PPR_BIDI_SUCCESSORS = ( + qn("w:adjustRightInd"), + qn("w:snapToGrid"), + qn("w:spacing"), + qn("w:ind"), + qn("w:contextualSpacing"), + qn("w:mirrorIndents"), + qn("w:suppressOverlap"), + qn("w:jc"), + qn("w:textDirection"), + qn("w:textAlignment"), + qn("w:textboxTightWrap"), + qn("w:outlineLvl"), + qn("w:divId"), + qn("w:cnfStyle"), + qn("w:rPr"), + qn("w:sectPr"), + qn("w:pPrChange"), +) +# Elements that must come AFTER w:jc inside w:pPr (w:jc itself excluded). +_PPR_JC_SUCCESSORS = _PPR_BIDI_SUCCESSORS[_PPR_BIDI_SUCCESSORS.index(qn("w:jc")) + 1 :] +# Elements that must come AFTER w:rtl inside w:rPr. +_RPR_RTL_SUCCESSORS = ( + qn("w:cs"), + qn("w:em"), + qn("w:lang"), + qn("w:eastAsianLayout"), + qn("w:specVanish"), + qn("w:oMath"), + qn("w:rPrChange"), +) +# Elements that must come AFTER w:bidiVisual inside w:tblPr. +_TBLPR_BIDIVISUAL_SUCCESSORS = ( + qn("w:tblStyleRowBandSize"), + qn("w:tblStyleColBandSize"), + qn("w:tblW"), + qn("w:jc"), + qn("w:tblCellSpacing"), + qn("w:tblInd"), + qn("w:tblBorders"), + qn("w:shd"), + qn("w:tblLayout"), + qn("w:tblCellMar"), + qn("w:tblLook"), + qn("w:tblCaption"), + qn("w:tblDescription"), + qn("w:tblPrChange"), +) + + +def _insert_ordered(parent, child, successors) -> None: + """Insert child into parent at its schema position. + + The child is placed just before the first existing element whose tag + is listed in ``successors`` (the elements that must come after it in + the OOXML sequence); appended at the end when no successor exists. + Works on python-docx oxml elements and plain lxml elements alike. + """ + for existing in parent: + if existing.tag in successors: + existing.addprevious(child) + return + parent.append(child) + + def _set_paragraph_rtl(paragraph: Paragraph) -> None: """ Enable RTL mode on a paragraph and all its runs. @@ -133,37 +209,97 @@ def _set_paragraph_rtl(paragraph: Paragraph) -> None: paragraph has no explicit alignment — centered/justified titles must not be forced right-aligned. - w:rPr/w:rtl → run-level RTL marker for each run + + Every element is inserted at its schema position inside w:pPr/w:rPr + (OOXML requires a fixed child order; appended-at-the-end elements + make Word flag the file for repair). """ pPr = paragraph._p.get_or_add_pPr() if pPr.find(qn("w:bidi")) is None: - pPr.append(OxmlElement("w:bidi")) + _insert_ordered(pPr, OxmlElement("w:bidi"), _PPR_BIDI_SUCCESSORS) jc = pPr.find(qn("w:jc")) explicit_alignment = jc is not None and jc.get(qn("w:val")) not in (None, "", "left") if not explicit_alignment: if jc is None: jc = OxmlElement("w:jc") - pPr.append(jc) + _insert_ordered(pPr, jc, _PPR_JC_SUCCESSORS) jc.set(qn("w:val"), "right") for run in paragraph.runs: rPr = run._r.get_or_add_rPr() if rPr.find(qn("w:rtl")) is None: - rPr.append(OxmlElement("w:rtl")) + _insert_ordered(rPr, OxmlElement("w:rtl"), _RPR_RTL_SUCCESSORS) + + +def _set_table_rtl(tbl_element) -> None: + """ + Mark a table as visually right-to-left. + + w:bidiVisual inside w:tblPr flips the column order at render time: + the first column displays rightmost. Works on python-docx CT_Tbl + elements and on plain lxml elements (notes/comments trees) alike. + """ + tblPr = tbl_element.find(qn("w:tblPr")) + if tblPr is None: + tblPr = OxmlElement("w:tblPr") + tbl_element.insert(0, tblPr) + if tblPr.find(qn("w:bidiVisual")) is None: + _insert_ordered( + tblPr, OxmlElement("w:bidiVisual"), _TBLPR_BIDIVISUAL_SUCCESSORS + ) + + +def _apply_bidi_to_part_tree(root) -> int: + """ + Switch every paragraph, run and table of a footnotes/endnotes/comments + tree to RTL. + + These parts are rewritten as raw XML after the document save (they + live outside python-docx's object model), so the RTL direction is + injected into the same in-memory tree just before write-back: + w:bidi on every paragraph, w:rtl on every run, w:bidiVisual on the + tables these parts may contain. + + Returns the number of paragraphs switched to RTL. + """ + switched = 0 + for p in root.iter(qn("w:p")): + pPr = p.find(qn("w:pPr")) + if pPr is None: + pPr = OxmlElement("w:pPr") + p.insert(0, pPr) + if pPr.find(qn("w:bidi")) is None: + _insert_ordered(pPr, OxmlElement("w:bidi"), _PPR_BIDI_SUCCESSORS) + switched += 1 + for r in p.iter(qn("w:r")): + rPr = r.find(qn("w:rPr")) + if rPr is None: + rPr = OxmlElement("w:rPr") + r.insert(0, rPr) + if rPr.find(qn("w:rtl")) is None: + _insert_ordered(rPr, OxmlElement("w:rtl"), _RPR_RTL_SUCCESSORS) + for tbl in root.iter(qn("w:tbl")): + _set_table_rtl(tbl) + return switched def _apply_rtl_to_document(document: Document) -> None: """Apply RTL direction to every paragraph and section in the document.""" - # Body paragraphs - for para in document.paragraphs: - _set_paragraph_rtl(para) - # Body tables - for table in document.tables: - for row in table.rows: - for cell in row.cells: - for para in cell.paragraphs: - _set_paragraph_rtl(para) + body = document.element.body + + # Every paragraph of the body tree in ONE pass: top-level paragraphs, + # table cells (nested tables included), text boxes (w:txbxContent) + # and SDT content all receive w:bidi + run-level w:rtl. + for p in body.iter(qn("w:p")): + _set_paragraph_rtl(Paragraph(p, document)) + + # Tables: w:bidiVisual flips the visual column order. iter() covers + # nested tables too. + for tbl in body.iter(qn("w:tbl")): + _set_table_rtl(tbl) + # Headers, footers, and section-level RTL (page layout direction) for section in document.sections: # Set the section (page) direction to RTL so Word renders margins, @@ -172,14 +308,37 @@ def _apply_rtl_to_document(document: Document) -> None: if sectPr.find(qn("w:bidi")) is None: sectPr.append(OxmlElement("w:bidi")) - for hf in (section.header, section.footer): - for para in hf.paragraphs: - _set_paragraph_rtl(para) - for table in hf.tables: - for row in table.rows: - for cell in row.cells: - for para in cell.paragraphs: - _set_paragraph_rtl(para) + # Same six header/footer zones the text collection translates + # (default, first page, even pages) — a translated first-page + # header must not stay left-to-right. Some may be missing + # depending on the python-docx version: getattr defensively. + headers_footers = ( + section.header, + section.footer, + getattr(section, "first_page_header", None), + getattr(section, "first_page_footer", None), + getattr(section, "even_page_header", None), + getattr(section, "even_page_footer", None), + ) + for hf in headers_footers: + if hf is None: + continue + # Touch paragraphs first so a linked/missing definition is + # created exactly like the paragraph walk below would. + paras = hf.paragraphs + root = getattr(hf, "_element", None) + if root is None and paras: + root = paras[0]._p.getroottree().getroot() + if root is not None: + # Full story tree: paragraphs, tables (bidiVisual) and + # text boxes inside the header/footer. + for p in root.iter(qn("w:p")): + _set_paragraph_rtl(Paragraph(p, document)) + for tbl in root.iter(qn("w:tbl")): + _set_table_rtl(tbl) + else: + for para in paras: + _set_paragraph_rtl(para) class WordProcessorError(Exception): @@ -324,7 +483,9 @@ class WordTranslator: # that python-docx doesn't manage (footnotes, endnotes). post_save_callbacks: List[Callable[[Path], None]] = [] - self._collect_from_body(document, text_elements, post_save_callbacks) + self._collect_from_body( + document, text_elements, post_save_callbacks, rtl=is_rtl(target_language) + ) # Collect chart text from ZIP (chart titles, axis labels, series names) self._collect_charts_from_zip(input_path, text_elements, chart_translations) @@ -420,7 +581,7 @@ class WordTranslator: ) # Apply RTL layout when the target language is written right-to-left. - if target_language.lower() in RTL_LANGUAGES: + if is_rtl(target_language): _apply_rtl_to_document(document) # CJK / Arabic-script font hints so Word renders the target @@ -677,6 +838,7 @@ class WordTranslator: def _collect_from_body( self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]], post_save_callbacks: List[Callable[[Path], None]] = None, + rtl: bool = False, ) -> None: """Collect all text elements from document body. @@ -711,9 +873,9 @@ class WordTranslator: # Pass 3: footnotes, endnotes and comments (live in separate parts) if post_save_callbacks is None: post_save_callbacks = [] - self._collect_from_footnotes(document, text_elements, post_save_callbacks) - self._collect_from_endnotes(document, text_elements, post_save_callbacks) - self._collect_from_comments(document, text_elements, post_save_callbacks) + self._collect_from_footnotes(document, text_elements, post_save_callbacks, rtl=rtl) + self._collect_from_endnotes(document, text_elements, post_save_callbacks, rtl=rtl) + self._collect_from_comments(document, text_elements, post_save_callbacks, rtl=rtl) total = len(text_elements) - count_before _log_info( @@ -811,6 +973,7 @@ class WordTranslator: def _collect_from_footnotes( self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]], post_save_callbacks: List[Callable[[Path], None]] = None, + rtl: bool = False, ) -> None: """Collect text from footnotes. @@ -856,6 +1019,8 @@ class WordTranslator: # setters hold references to t_elems inside this tree. def write_footnotes_back(output_path: Path) -> None: try: + if rtl: + _apply_bidi_to_part_tree(footnotes_xml) new_blob = etree.tostring( footnotes_xml, xml_declaration=True, @@ -894,6 +1059,7 @@ class WordTranslator: def _collect_from_endnotes( self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]], post_save_callbacks: List[Callable[[Path], None]] = None, + rtl: bool = False, ) -> None: """Collect text from endnotes (python-docx 1.x compatible). @@ -921,6 +1087,8 @@ class WordTranslator: if text_elements and post_save_callbacks is not None: def write_endnotes_back(output_path: Path) -> None: try: + if rtl: + _apply_bidi_to_part_tree(endnotes_xml) new_blob = etree.tostring( endnotes_xml, xml_declaration=True, @@ -944,6 +1112,7 @@ class WordTranslator: def _collect_from_comments( self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]], post_save_callbacks: List[Callable[[Path], None]] = None, + rtl: bool = False, ) -> None: """Collect text from comments/balloons (word/comments.xml part). @@ -974,6 +1143,8 @@ class WordTranslator: if collected and post_save_callbacks is not None: def write_comments_back(output_path: Path) -> None: try: + if rtl: + _apply_bidi_to_part_tree(comments_xml) new_blob = etree.tostring( comments_xml, xml_declaration=True,