feat(translation): quality pipeline overhaul + new features (audit 2026-08-29)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
Translation quality & format preservation: - Word: merge adjacent same-format runs into one unit (sentence-level coherence like inline-tag handling); translate comments/balloons; dedupe textbox collection (was translated twice); RTL no longer overrides center/justify alignment; CJK/Arabic font hints (eastAsia/cs) - PPTX: chart translations now actually reach the output file (ChartPart.blob is read-only — rewrite chart XML in the saved ZIP); CJK typeface hints (a:ea) - Excel: sheet renames no longer break references — rewrite cell formulas (3D/quoted), defined names, data validations, cond. formats - PDF: bold/italic honored (hebo/heit/hebi); table cells never merge; unchanged blocks left untouched (typography preserved, fixes duplicate hyperlinks); attempted/changed stats + route gate now cover PDF; CJK font paths; scanned PDFs via Mistral OCR (detection + admin settings) Features: - formality param (formal/informal) + automatic regional-variant prompts - output_mode=bilingual docx (source above translation) - per-user translation memory on Redis (falls back to LRU), context-hashed - QA report + 0-100 confidence score in job status; L0 on by default - OpenAI-compatible providers: whole chunk in ONE numbered-JSON request (~15x fewer calls) with per-item fallback; base prompt always present (custom prompt no longer replaces translation instructions) Infra & marketing alignment: - plan-based engine gating + vision gating (closes paid-engine leak); /providers/available filtered per plan; 107 languages exposed - zh-CN/zh-TW validation fixed; libmagic disabled on Windows (native crash) - admin: Mistral OCR settings + engine status dashboard; httpx<0.28 pin (TestClient breakage); Prometheus test fixture fixed - marketing docs aligned with code (PDF+OCR, retention, engines, pricing) - security: .env.ionos/.env.production/provider_settings.json removed Tests: 1173 passed / 0 failed (6 network tests deselected: free Google endpoint temporarily blocked from this machine)
This commit is contained in:
90
translators/bilingual.py
Normal file
90
translators/bilingual.py
Normal file
@@ -0,0 +1,90 @@
|
||||
"""
|
||||
Bilingual output: interleave source paragraphs with their translation.
|
||||
|
||||
Given the ORIGINAL document and the TRANSLATED document (same structure —
|
||||
the pipeline only rewrites run texts), produce a copy of the translated
|
||||
document where each translated body paragraph is preceded by its source
|
||||
paragraph in gray italic. Tables, headers and footers keep the translated
|
||||
version only (duplicating them would double the layout).
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from docx import Document
|
||||
from docx.text.paragraph import Paragraph
|
||||
from docx.oxml import OxmlElement
|
||||
from docx.oxml.ns import qn
|
||||
from docx.shared import Pt, RGBColor
|
||||
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def make_bilingual_docx(
|
||||
source_path: Path, translated_path: Path, output_path: Path
|
||||
) -> Optional[Path]:
|
||||
"""Create a bilingual .docx (source paragraph above its translation).
|
||||
|
||||
Returns the output path, or None when the pairing failed (structure
|
||||
mismatch) — callers fall back to the translated-only file.
|
||||
"""
|
||||
source_path = Path(source_path)
|
||||
translated_path = Path(translated_path)
|
||||
output_path = Path(output_path)
|
||||
|
||||
try:
|
||||
src = Document(str(source_path))
|
||||
tr = Document(str(translated_path))
|
||||
except Exception as e:
|
||||
logger.warning("bilingual_open_failed", error=str(e))
|
||||
return None
|
||||
|
||||
src_children = list(src.element.body)
|
||||
tr_children = list(tr.element.body)
|
||||
|
||||
# The pipeline preserves structure exactly; a drift beyond a small
|
||||
# tolerance means pairing by index is unsafe → bail out.
|
||||
if abs(len(src_children) - len(tr_children)) > 0:
|
||||
logger.warning(
|
||||
"bilingual_structure_mismatch",
|
||||
source_elements=len(src_children),
|
||||
translated_elements=len(tr_children),
|
||||
)
|
||||
if len(src_children) != len(tr_children):
|
||||
return None
|
||||
|
||||
inserted = 0
|
||||
for src_el, tr_el in zip(src_children, tr_children):
|
||||
if tr_el.tag != qn("w:p") or src_el.tag != qn("w:p"):
|
||||
continue
|
||||
src_text = "".join(
|
||||
t.text or "" for t in src_el.iter(qn("w:t"))
|
||||
).strip()
|
||||
if not src_text:
|
||||
continue
|
||||
|
||||
# Insert a NEW paragraph directly above the translated one, inside
|
||||
# the translated document (keeps styles/sections untouched).
|
||||
new_p = OxmlElement("w:p")
|
||||
tr_el.addprevious(new_p)
|
||||
para = Paragraph(new_p, tr)
|
||||
run = para.add_run(src_text)
|
||||
run.font.size = Pt(9)
|
||||
run.font.italic = True
|
||||
run.font.color.rgb = RGBColor(0x80, 0x80, 0x80)
|
||||
inserted += 1
|
||||
|
||||
if inserted == 0:
|
||||
logger.info("bilingual_nothing_inserted")
|
||||
return None
|
||||
|
||||
try:
|
||||
tr.save(str(output_path))
|
||||
except Exception as e:
|
||||
logger.warning("bilingual_save_failed", error=str(e))
|
||||
return None
|
||||
|
||||
logger.info("bilingual_docx_created", paragraphs=inserted)
|
||||
return output_path
|
||||
Reference in New Issue
Block a user