feat(translation): quality pipeline overhaul + new features (audit 2026-08-29)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s

Translation quality & format preservation:
- Word: merge adjacent same-format runs into one unit (sentence-level
  coherence like inline-tag handling); translate comments/balloons;
  dedupe textbox collection (was translated twice); RTL no longer
  overrides center/justify alignment; CJK/Arabic font hints (eastAsia/cs)
- PPTX: chart translations now actually reach the output file
  (ChartPart.blob is read-only — rewrite chart XML in the saved ZIP);
  CJK typeface hints (a:ea)
- Excel: sheet renames no longer break references — rewrite cell
  formulas (3D/quoted), defined names, data validations, cond. formats
- PDF: bold/italic honored (hebo/heit/hebi); table cells never merge;
  unchanged blocks left untouched (typography preserved, fixes duplicate
  hyperlinks); attempted/changed stats + route gate now cover PDF;
  CJK font paths; scanned PDFs via Mistral OCR (detection + admin settings)

Features:
- formality param (formal/informal) + automatic regional-variant prompts
- output_mode=bilingual docx (source above translation)
- per-user translation memory on Redis (falls back to LRU), context-hashed
- QA report + 0-100 confidence score in job status; L0 on by default
- OpenAI-compatible providers: whole chunk in ONE numbered-JSON request
  (~15x fewer calls) with per-item fallback; base prompt always present
  (custom prompt no longer replaces translation instructions)

Infra & marketing alignment:
- plan-based engine gating + vision gating (closes paid-engine leak);
  /providers/available filtered per plan; 107 languages exposed
- zh-CN/zh-TW validation fixed; libmagic disabled on Windows (native crash)
- admin: Mistral OCR settings + engine status dashboard; httpx<0.28 pin
  (TestClient breakage); Prometheus test fixture fixed
- marketing docs aligned with code (PDF+OCR, retention, engines, pricing)
- security: .env.ionos/.env.production/provider_settings.json removed

Tests: 1173 passed / 0 failed (6 network tests deselected: free Google
endpoint temporarily blocked from this machine)
This commit is contained in:
2026-08-29 18:38:09 +02:00
parent 992f13d53c
commit 526c87348f
87 changed files with 6996 additions and 1024 deletions

90
translators/bilingual.py Normal file
View File

@@ -0,0 +1,90 @@
"""
Bilingual output: interleave source paragraphs with their translation.
Given the ORIGINAL document and the TRANSLATED document (same structure —
the pipeline only rewrites run texts), produce a copy of the translated
document where each translated body paragraph is preceded by its source
paragraph in gray italic. Tables, headers and footers keep the translated
version only (duplicating them would double the layout).
"""
from pathlib import Path
from typing import Optional
from docx import Document
from docx.text.paragraph import Paragraph
from docx.oxml import OxmlElement
from docx.oxml.ns import qn
from docx.shared import Pt, RGBColor
from core.logging import get_logger
logger = get_logger(__name__)
def make_bilingual_docx(
source_path: Path, translated_path: Path, output_path: Path
) -> Optional[Path]:
"""Create a bilingual .docx (source paragraph above its translation).
Returns the output path, or None when the pairing failed (structure
mismatch) — callers fall back to the translated-only file.
"""
source_path = Path(source_path)
translated_path = Path(translated_path)
output_path = Path(output_path)
try:
src = Document(str(source_path))
tr = Document(str(translated_path))
except Exception as e:
logger.warning("bilingual_open_failed", error=str(e))
return None
src_children = list(src.element.body)
tr_children = list(tr.element.body)
# The pipeline preserves structure exactly; a drift beyond a small
# tolerance means pairing by index is unsafe → bail out.
if abs(len(src_children) - len(tr_children)) > 0:
logger.warning(
"bilingual_structure_mismatch",
source_elements=len(src_children),
translated_elements=len(tr_children),
)
if len(src_children) != len(tr_children):
return None
inserted = 0
for src_el, tr_el in zip(src_children, tr_children):
if tr_el.tag != qn("w:p") or src_el.tag != qn("w:p"):
continue
src_text = "".join(
t.text or "" for t in src_el.iter(qn("w:t"))
).strip()
if not src_text:
continue
# Insert a NEW paragraph directly above the translated one, inside
# the translated document (keeps styles/sections untouched).
new_p = OxmlElement("w:p")
tr_el.addprevious(new_p)
para = Paragraph(new_p, tr)
run = para.add_run(src_text)
run.font.size = Pt(9)
run.font.italic = True
run.font.color.rgb = RGBColor(0x80, 0x80, 0x80)
inserted += 1
if inserted == 0:
logger.info("bilingual_nothing_inserted")
return None
try:
tr.save(str(output_path))
except Exception as e:
logger.warning("bilingual_save_failed", error=str(e))
return None
logger.info("bilingual_docx_created", paragraphs=inserted)
return output_path