feat(translation): quality pipeline overhaul + new features (audit 2026-08-29)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
Translation quality & format preservation: - Word: merge adjacent same-format runs into one unit (sentence-level coherence like inline-tag handling); translate comments/balloons; dedupe textbox collection (was translated twice); RTL no longer overrides center/justify alignment; CJK/Arabic font hints (eastAsia/cs) - PPTX: chart translations now actually reach the output file (ChartPart.blob is read-only — rewrite chart XML in the saved ZIP); CJK typeface hints (a:ea) - Excel: sheet renames no longer break references — rewrite cell formulas (3D/quoted), defined names, data validations, cond. formats - PDF: bold/italic honored (hebo/heit/hebi); table cells never merge; unchanged blocks left untouched (typography preserved, fixes duplicate hyperlinks); attempted/changed stats + route gate now cover PDF; CJK font paths; scanned PDFs via Mistral OCR (detection + admin settings) Features: - formality param (formal/informal) + automatic regional-variant prompts - output_mode=bilingual docx (source above translation) - per-user translation memory on Redis (falls back to LRU), context-hashed - QA report + 0-100 confidence score in job status; L0 on by default - OpenAI-compatible providers: whole chunk in ONE numbered-JSON request (~15x fewer calls) with per-item fallback; base prompt always present (custom prompt no longer replaces translation instructions) Infra & marketing alignment: - plan-based engine gating + vision gating (closes paid-engine leak); /providers/available filtered per plan; 107 languages exposed - zh-CN/zh-TW validation fixed; libmagic disabled on Windows (native crash) - admin: Mistral OCR settings + engine status dashboard; httpx<0.28 pin (TestClient breakage); Prometheus test fixture fixed - marketing docs aligned with code (PDF+OCR, retention, engines, pricing) - security: .env.ionos/.env.production/provider_settings.json removed Tests: 1173 passed / 0 failed (6 network tests deselected: free Google endpoint temporarily blocked from this machine)
This commit is contained in:
20
config.py
20
config.py
@@ -70,8 +70,8 @@ class Config:
|
||||
|
||||
# ============== Quality Layer (L0) ==============
|
||||
# Track A1 of the dev plan — observability only, no behavior change.
|
||||
# Set to "true" to enable. Default: false (opt-in).
|
||||
QUALITY_L0_ENABLED = os.getenv("QUALITY_L0_ENABLED", "false").lower() == "true"
|
||||
# Enabled by default since 2026-08-29: log-only, never blocks a job.
|
||||
QUALITY_L0_ENABLED = os.getenv("QUALITY_L0_ENABLED", "true").lower() == "true"
|
||||
# Number of text samples to extract from the output file for L0 analysis.
|
||||
# Keep small to avoid overhead. 20 is enough to catch language confusion.
|
||||
QUALITY_L0_SAMPLE_SIZE = int(os.getenv("QUALITY_L0_SAMPLE_SIZE", "20"))
|
||||
@@ -117,6 +117,22 @@ class Config:
|
||||
# Set to false to use the legacy aggressive-shrink strategy (NOT recommended).
|
||||
PDF_SMART_FIT_ENABLED = os.getenv("PDF_SMART_FIT_ENABLED", "true").lower() == "true"
|
||||
|
||||
# ============== Scanned PDF OCR (Mistral) ==============
|
||||
# Image-only PDFs have no extractable text layer. When a PDF looks
|
||||
# scanned, it is routed to the Mistral OCR API to recover the text
|
||||
# before translation (output: clean re-typeset PDF).
|
||||
# Pricing reference: ~$1 / 1000 pages — set MISTRAL_OCR_ENABLED=false
|
||||
# to disable and reject scanned PDFs with an explicit error instead.
|
||||
MISTRAL_API_KEY = os.getenv("MISTRAL_API_KEY", "").strip()
|
||||
MISTRAL_OCR_MODEL = os.getenv("MISTRAL_OCR_MODEL", "mistral-ocr-latest")
|
||||
MISTRAL_OCR_TIMEOUT = int(os.getenv("MISTRAL_OCR_TIMEOUT", "180"))
|
||||
MISTRAL_OCR_ENABLED = os.getenv("MISTRAL_OCR_ENABLED", "true").lower() == "true"
|
||||
# A page with fewer extractable characters than this is text-poor; it
|
||||
# counts as a scan page only when raster images also cover most of it.
|
||||
SCANNED_PDF_MIN_CHARS_PER_PAGE = int(
|
||||
os.getenv("SCANNED_PDF_MIN_CHARS_PER_PAGE", "100")
|
||||
)
|
||||
|
||||
|
||||
# ============== API Configuration ==============
|
||||
API_TITLE = "Document Translation API"
|
||||
|
||||
Reference in New Issue
Block a user