feat(translation): quality pipeline overhaul + new features (audit 2026-08-29)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
Translation quality & format preservation: - Word: merge adjacent same-format runs into one unit (sentence-level coherence like inline-tag handling); translate comments/balloons; dedupe textbox collection (was translated twice); RTL no longer overrides center/justify alignment; CJK/Arabic font hints (eastAsia/cs) - PPTX: chart translations now actually reach the output file (ChartPart.blob is read-only — rewrite chart XML in the saved ZIP); CJK typeface hints (a:ea) - Excel: sheet renames no longer break references — rewrite cell formulas (3D/quoted), defined names, data validations, cond. formats - PDF: bold/italic honored (hebo/heit/hebi); table cells never merge; unchanged blocks left untouched (typography preserved, fixes duplicate hyperlinks); attempted/changed stats + route gate now cover PDF; CJK font paths; scanned PDFs via Mistral OCR (detection + admin settings) Features: - formality param (formal/informal) + automatic regional-variant prompts - output_mode=bilingual docx (source above translation) - per-user translation memory on Redis (falls back to LRU), context-hashed - QA report + 0-100 confidence score in job status; L0 on by default - OpenAI-compatible providers: whole chunk in ONE numbered-JSON request (~15x fewer calls) with per-item fallback; base prompt always present (custom prompt no longer replaces translation instructions) Infra & marketing alignment: - plan-based engine gating + vision gating (closes paid-engine leak); /providers/available filtered per plan; 107 languages exposed - zh-CN/zh-TW validation fixed; libmagic disabled on Windows (native crash) - admin: Mistral OCR settings + engine status dashboard; httpx<0.28 pin (TestClient breakage); Prometheus test fixture fixed - marketing docs aligned with code (PDF+OCR, retention, engines, pricing) - security: .env.ionos/.env.production/provider_settings.json removed Tests: 1173 passed / 0 failed (6 network tests deselected: free Google endpoint temporarily blocked from this machine)
This commit is contained in:
@@ -38,6 +38,9 @@ class FileCleanupManager:
|
||||
self.max_file_age_seconds = max_file_age_minutes * 60
|
||||
self.cleanup_interval = cleanup_interval_minutes * 60
|
||||
self.max_total_size_bytes = int(max_total_size_gb * 1024 * 1024 * 1024)
|
||||
# Untracked (orphan) files must be at least this old before deletion:
|
||||
# protects in-flight files whose Redis metadata is missing or delayed.
|
||||
self.orphan_grace_seconds = 900
|
||||
|
||||
self._running = False
|
||||
self._task: Optional[asyncio.Task] = None
|
||||
@@ -191,10 +194,13 @@ class FileCleanupManager:
|
||||
data = await redis_client.get(key)
|
||||
if data:
|
||||
metadata = json.loads(data)
|
||||
if "file_path" in metadata:
|
||||
# Normalize path to absolute string for comparison
|
||||
path_str = str(Path(metadata["file_path"]).absolute())
|
||||
tracked_paths.add(path_str)
|
||||
# Metadata uses "input_path" (and possibly other *_path keys);
|
||||
# collect every path field so tracked files are never
|
||||
# misclassified as orphans.
|
||||
for path_key in ("input_path", "file_path", "output_path"):
|
||||
if path_key in metadata:
|
||||
path_str = str(Path(metadata[path_key]).absolute())
|
||||
tracked_paths.add(path_str)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch tracked paths from Redis: {e}")
|
||||
redis_available = False
|
||||
@@ -234,8 +240,11 @@ class FileCleanupManager:
|
||||
reason = ""
|
||||
|
||||
if is_orphan:
|
||||
should_delete = True
|
||||
reason = "orphan"
|
||||
# Never delete a young file as orphan: it may belong to
|
||||
# a job whose Redis metadata is not yet visible.
|
||||
if file_age > self.orphan_grace_seconds:
|
||||
should_delete = True
|
||||
reason = "orphan"
|
||||
elif file_age > self.max_file_age_seconds:
|
||||
should_delete = True
|
||||
reason = "expired"
|
||||
|
||||
Reference in New Issue
Block a user