feat(translation): quality pipeline overhaul + new features (audit 2026-08-29)
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m20s
Translation quality & format preservation: - Word: merge adjacent same-format runs into one unit (sentence-level coherence like inline-tag handling); translate comments/balloons; dedupe textbox collection (was translated twice); RTL no longer overrides center/justify alignment; CJK/Arabic font hints (eastAsia/cs) - PPTX: chart translations now actually reach the output file (ChartPart.blob is read-only — rewrite chart XML in the saved ZIP); CJK typeface hints (a:ea) - Excel: sheet renames no longer break references — rewrite cell formulas (3D/quoted), defined names, data validations, cond. formats - PDF: bold/italic honored (hebo/heit/hebi); table cells never merge; unchanged blocks left untouched (typography preserved, fixes duplicate hyperlinks); attempted/changed stats + route gate now cover PDF; CJK font paths; scanned PDFs via Mistral OCR (detection + admin settings) Features: - formality param (formal/informal) + automatic regional-variant prompts - output_mode=bilingual docx (source above translation) - per-user translation memory on Redis (falls back to LRU), context-hashed - QA report + 0-100 confidence score in job status; L0 on by default - OpenAI-compatible providers: whole chunk in ONE numbered-JSON request (~15x fewer calls) with per-item fallback; base prompt always present (custom prompt no longer replaces translation instructions) Infra & marketing alignment: - plan-based engine gating + vision gating (closes paid-engine leak); /providers/available filtered per plan; 107 languages exposed - zh-CN/zh-TW validation fixed; libmagic disabled on Windows (native crash) - admin: Mistral OCR settings + engine status dashboard; httpx<0.28 pin (TestClient breakage); Prometheus test fixture fixed - marketing docs aligned with code (PDF+OCR, retention, engines, pricing) - security: .env.ionos/.env.production/provider_settings.json removed Tests: 1173 passed / 0 failed (6 network tests deselected: free Google endpoint temporarily blocked from this machine)
This commit is contained in:
10
.env.example
10
.env.example
@@ -71,6 +71,16 @@ OPENAI_HEALTH_CHECK_TIMEOUT=5
|
||||
OPENROUTER_API_KEY=
|
||||
OPENROUTER_MODEL=deepseek/deepseek-chat
|
||||
|
||||
# Mistral OCR — translation of SCANNED PDFs (image-only pages).
|
||||
# Optional. If absent: scanned PDFs are rejected with an explicit error.
|
||||
# Get API key from: https://console.mistral.ai/ (~$1 / 1000 pages)
|
||||
MISTRAL_API_KEY=
|
||||
MISTRAL_OCR_MODEL=mistral-ocr-latest
|
||||
MISTRAL_OCR_TIMEOUT=180
|
||||
MISTRAL_OCR_ENABLED=true
|
||||
# Average extractable chars/page below which a PDF is treated as scanned
|
||||
SCANNED_PDF_MIN_CHARS_PER_PAGE=100
|
||||
|
||||
# Ollama Configuration. Optional. If absent: default http://localhost:11434 (provider may be disabled).
|
||||
OLLAMA_BASE_URL=http://localhost:11434
|
||||
OLLAMA_MODEL=llama3
|
||||
|
||||
93
.env.ionos
93
.env.ionos
@@ -1,93 +0,0 @@
|
||||
# ============================================================
|
||||
# PRODUCTION — Ionos VPS
|
||||
# Copiez ce fichier en .env sur le serveur et remplissez TOUT
|
||||
# Ne committez JAMAIS ce fichier avec de vraies valeurs
|
||||
# ============================================================
|
||||
|
||||
# ─── Application ────────────────────────────────────────────
|
||||
APP_NAME="Office Translator"
|
||||
ENV=production
|
||||
DEBUG=false
|
||||
LOG_LEVEL=INFO
|
||||
LOG_FORMAT=json
|
||||
|
||||
# ─── Domaine & URLs ─────────────────────────────────────────
|
||||
# Remplacez par votre vrai domaine (configuré dans Ionos DNS)
|
||||
DOMAIN=wordly.art
|
||||
NEXT_PUBLIC_API_URL=https://wordly.art
|
||||
CORS_ORIGINS=https://wordly.art
|
||||
|
||||
# ─── Sécurité JWT ─────────────────────────────────────────
|
||||
# OBLIGATOIRE — Générez avec :
|
||||
# python3 -c "import secrets; print(secrets.token_urlsafe(64))"
|
||||
JWT_SECRET_KEY=REMPLACEZ_PAR_UNE_CLE_SECRETE_DE_64_CARACTERES
|
||||
|
||||
# ─── Admin ──────────────────────────────────────────────────
|
||||
ADMIN_USERNAME=admin
|
||||
# Hash bcrypt de votre mot de passe admin
|
||||
# Générez avec : python3 -c "from passlib.context import CryptContext; print(CryptContext(schemes=['bcrypt']).hash('VotreMotDePasse'))"
|
||||
ADMIN_PASSWORD_HASH=REMPLACEZ_PAR_HASH_BCRYPT
|
||||
|
||||
# ─── Base de données PostgreSQL ─────────────────────────────
|
||||
POSTGRES_USER=translate
|
||||
# Mot de passe fort — minimum 32 caractères
|
||||
# Générez avec : python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||
POSTGRES_PASSWORD=REMPLACEZ_PAR_MOT_DE_PASSE_FORT
|
||||
POSTGRES_DB=translate_db
|
||||
DATABASE_URL=postgresql+asyncpg://translate:REMPLACEZ_PAR_MOT_DE_PASSE_FORT@postgres:5432/translate_db
|
||||
|
||||
# ─── Redis ──────────────────────────────────────────────────
|
||||
REDIS_URL=redis://redis:6379/0
|
||||
|
||||
# ─── Fournisseurs de traduction ─────────────────────────────
|
||||
# Google Translate (gratuit, via deep_translator) — toujours activé
|
||||
GOOGLE_TRANSLATE_ENABLED=true
|
||||
|
||||
# Google Cloud Translation API (payant)
|
||||
GOOGLE_CLOUD_ENABLED=false
|
||||
GOOGLE_CLOUD_API_KEY=
|
||||
|
||||
# DeepL
|
||||
DEEPL_ENABLED=false
|
||||
DEEPL_API_KEY=
|
||||
|
||||
# OpenRouter (IA)
|
||||
OPENROUTER_ENABLED=false
|
||||
OPENROUTER_API_KEY=
|
||||
OPENROUTER_MODEL=deepseek/deepseek-v3.2
|
||||
|
||||
# OpenAI
|
||||
OPENAI_ENABLED=false
|
||||
OPENAI_API_KEY=
|
||||
OPENAI_MODEL=gpt-4o-mini
|
||||
|
||||
# Désactivés en production standard
|
||||
OLLAMA_ENABLED=false
|
||||
ZAI_API_KEY=
|
||||
|
||||
# ─── Stripe Paiements ───────────────────────────────────────
|
||||
STRIPE_SECRET_KEY=sk_live_REMPLACEZ
|
||||
STRIPE_WEBHOOK_SECRET=whsec_REMPLACEZ
|
||||
STRIPE_PRICE_STARTER_MONTHLY=price_REMPLACEZ
|
||||
STRIPE_PRICE_STARTER_YEARLY=price_REMPLACEZ
|
||||
STRIPE_PRICE_PRO_MONTHLY=price_REMPLACEZ
|
||||
STRIPE_PRICE_PRO_YEARLY=price_REMPLACEZ
|
||||
STRIPE_PRICE_BUSINESS_MONTHLY=price_REMPLACEZ
|
||||
STRIPE_PRICE_BUSINESS_YEARLY=price_REMPLACEZ
|
||||
|
||||
# ─── Fichiers & Limites ─────────────────────────────────────
|
||||
MAX_FILE_SIZE_MB=50
|
||||
MAX_CONCURRENT_TRANSLATIONS=5
|
||||
CLEANUP_ENABLED=true
|
||||
FILE_TTL_MINUTES=60
|
||||
|
||||
# ─── HTTPS / HSTS ───────────────────────────────────────────
|
||||
ENABLE_HSTS=true
|
||||
LETSENCRYPT_EMAIL=votre-email@wordly.art
|
||||
|
||||
# ─── Rate limiting ──────────────────────────────────────────
|
||||
RATE_LIMIT_ENABLED=true
|
||||
RATE_LIMIT_PER_MINUTE=30
|
||||
RATE_LIMIT_PER_HOUR=200
|
||||
TRANSLATIONS_PER_MINUTE=5
|
||||
TRANSLATIONS_PER_HOUR=30
|
||||
106
.env.production
106
.env.production
@@ -1,106 +0,0 @@
|
||||
# ============================================
|
||||
# Wordly.art - .env de production
|
||||
# ============================================
|
||||
# Copier ce fichier en .env sur le serveur
|
||||
# cp .env.production .env
|
||||
# Puis remplacer les valeurs [A CHANGER]
|
||||
# ============================================
|
||||
|
||||
# ---- Application ----
|
||||
APP_NAME=Wordly
|
||||
APP_ENV=production
|
||||
DEBUG=false
|
||||
LOG_LEVEL=INFO
|
||||
|
||||
# ---- Domaine ----
|
||||
DOMAIN=wordly.art
|
||||
NEXT_PUBLIC_API_URL=https://wordly.art
|
||||
FRONTEND_URL=https://wordly.art
|
||||
BACKEND_PORT=8000
|
||||
FRONTEND_PORT=3000
|
||||
|
||||
# ---- Service de traduction par defaut ----
|
||||
# Choisir: google, ollama, deepseek, minimax, deepl, openai, openrouter
|
||||
TRANSLATION_SERVICE=google
|
||||
|
||||
# ---- Google (gratuit, toujours actif) ----
|
||||
GOOGLE_TRANSLATE_ENABLED=true
|
||||
|
||||
# ---- Google OAuth (connexion avec Google) ----
|
||||
GOOGLE_CLIENT_ID=
|
||||
NEXT_PUBLIC_GOOGLE_CLIENT_ID=
|
||||
|
||||
# ---- Ollama (local, gratuit) ----
|
||||
OLLAMA_ENABLED=false
|
||||
OLLAMA_BASE_URL=http://ollama:11434
|
||||
OLLAMA_MODEL=llama3
|
||||
|
||||
# ---- DeepSeek ----
|
||||
DEEPSEEK_ENABLED=false
|
||||
DEEPSEEK_API_KEY=
|
||||
DEEPSEEK_MODEL=deepseek-chat
|
||||
DEEPSEEK_BASE_URL=https://api.deepseek.com/v1
|
||||
|
||||
# ---- Minimax ----
|
||||
MINIMAX_ENABLED=false
|
||||
MINIMAX_API_KEY=
|
||||
MINIMAX_MODEL=MiniMax-M1
|
||||
MINIMAX_BASE_URL=https://api.minimax.chat/v1
|
||||
|
||||
# ---- DeepL ----
|
||||
DEEPL_ENABLED=false
|
||||
DEEPL_API_KEY=
|
||||
|
||||
# ---- OpenAI ----
|
||||
OPENAI_ENABLED=false
|
||||
OPENAI_API_KEY=
|
||||
OPENAI_MODEL=gpt-4o-mini
|
||||
OPENAI_BASE_URL=https://api.openai.com/v1
|
||||
|
||||
# ---- OpenRouter ----
|
||||
OPENROUTER_ENABLED=false
|
||||
OPENROUTER_API_KEY=
|
||||
OPENROUTER_MODEL=deepseek/deepseek-chat
|
||||
|
||||
# ---- Upload ----
|
||||
MAX_FILE_SIZE_MB=50
|
||||
ALLOWED_EXTENSIONS=.docx,.xlsx,.pptx
|
||||
|
||||
# ---- Rate Limiting ----
|
||||
RATE_LIMIT_ENABLED=true
|
||||
RATE_LIMIT_REQUESTS_PER_MINUTE=60
|
||||
RATE_LIMIT_TRANSLATIONS_PER_MINUTE=10
|
||||
RATE_LIMIT_TRANSLATIONS_PER_HOUR=100
|
||||
RATE_LIMIT_TRANSLATIONS_PER_DAY=500
|
||||
|
||||
# ---- Admin ----
|
||||
ADMIN_USERNAME=admin
|
||||
ADMIN_PASSWORD_HASH=CHANGE_WITH_BCRYPT_HASH
|
||||
|
||||
# ---- Secrets (deja generes) ----
|
||||
JWT_SECRET_KEY=84-MniOv3rOZZ3FczwDvsNNHqZRf8yKI06uMNQGAgMSV8yAJ19comNe6FHHcnVyeNm-fvDxlcb9CWe40y4oy7A
|
||||
ADMIN_TOKEN_SECRET=1301880be1bc8026676d7f4fb13ae1c70fcd2abbd2f2cdfcb044eea5c7005ce3
|
||||
CORS_ORIGINS=https://wordly.art
|
||||
|
||||
# ---- Database ----
|
||||
POSTGRES_USER=translate
|
||||
POSTGRES_PASSWORD=yLLgkEvt6mvzGDdoqtQvI1vEgMmR-W75ZTPW5StaIAU
|
||||
POSTGRES_DB=translate_db
|
||||
|
||||
# ---- Monitoring ----
|
||||
GRAFANA_USER=admin
|
||||
GRAFANA_PASSWORD=WordlyGrafana2026!
|
||||
|
||||
# ---- Stripe ----
|
||||
# Clés TEST — remplacer par sk_live_... / pk_live_... en production réelle
|
||||
STRIPE_PUBLISHABLE_KEY=pk_test_51SkSHkCKXUJE51jnCr2QV4vCE18GH1XF59eHrHOV46EORuZTPVXFXxrbcJamyoJLaUHMc2McCRVkU4b6VMAWVR2R00XRdwmXMx
|
||||
STRIPE_SECRET_KEY=sk_test_51SkSHkCKXUJE51jnbEtXZ0nKiTHTa8ohDwLH8fZiDVEx6Ze0g5dg4fGJJgX1VgNHvF93GE3HTramT3oQrCaqOxid00OXTcZlsW
|
||||
STRIPE_WEBHOOK_SECRET=
|
||||
|
||||
# Price IDs créés dans Stripe Dashboard (wordly.art — test mode)
|
||||
STRIPE_PRICE_STARTER_MONTHLY=price_1TdF8FCKXUJE51jnNAeLqhF3
|
||||
STRIPE_PRICE_STARTER_YEARLY=price_1TdF8GCKXUJE51jnBhVVrjrh
|
||||
STRIPE_PRICE_PRO_MONTHLY=price_1TdF8GCKXUJE51jn9ChAAhKM
|
||||
STRIPE_PRICE_PRO_YEARLY=price_1TdF8HCKXUJE51jnpsvBivDe
|
||||
STRIPE_PRICE_BUSINESS_MONTHLY=price_1TdF8HCKXUJE51jn2K9EeBGJ
|
||||
STRIPE_PRICE_BUSINESS_YEARLY=price_1TdF8ICKXUJE51jnmgWZxW4U
|
||||
18
.gitignore
vendored
18
.gitignore
vendored
@@ -37,6 +37,7 @@ backups/
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
.zcode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
@@ -46,11 +47,18 @@ outputs/
|
||||
temp/
|
||||
translated_files/
|
||||
translated_test.*
|
||||
/translations.db
|
||||
|
||||
# Generated translation test outputs in the sample corpus (regeneratable, not for commit)
|
||||
# (the tracked sample_files/test_corpus/test_pdf_translated.pdf stays tracked)
|
||||
sample_files/test_corpus/*_translated*
|
||||
sample_files/test_corpus/*rigoureuse*
|
||||
|
||||
# Runtime data (users, provider config, glossaries) — managed at runtime, not in git
|
||||
data/users.json
|
||||
data/*.db
|
||||
data/*.sqlite
|
||||
data/waitlist.json
|
||||
|
||||
# Keep these in git (templates/defaults only)
|
||||
# data/provider_settings.json → commiter uniquement si pas de clés dedans
|
||||
@@ -88,4 +96,14 @@ htmlcov/
|
||||
|
||||
# Auto-generated pnpm workspace file (placeholders)
|
||||
frontend/pnpm-workspace.yaml
|
||||
office-translator-landing-page/pnpm-workspace.yaml
|
||||
|
||||
|
||||
# Secrets et donnees locales (audit securite 2026-08-26)
|
||||
data/provider_settings.json
|
||||
|
||||
# Unrelated local project (separate repo, never commit here)
|
||||
Colonization/
|
||||
|
||||
# TypeScript incremental build artifact
|
||||
office-translator-landing-page/tsconfig.tsbuildinfo
|
||||
|
||||
@@ -1,211 +1,208 @@
|
||||
# Plan Marketing - Office Translator (SaaS de Traduction de Documents)
|
||||
# Plan Marketing — Office Translator (Wordly.art)
|
||||
|
||||
> Document de référence pour l'agent marketing. Dernière mise à jour : 2026-05-10
|
||||
> Document de référence pour l'agent marketing. Dernière mise à jour : 2026-08-29
|
||||
>
|
||||
> **Changements majeurs vs. version du 2026-05-10**
|
||||
> 1. **Le billing est réellement implémenté** (Stripe : checkout, webhooks, portail, crédits) — la section tarifaire est désormais ancrée sur les plans réels du backend (`models/subscription.py`, `services/pricing_config.py`), plus sur des tiers fictifs.
|
||||
> 2. **La landing page est fonctionnelle** : sections pricing, FAQ, preuve sociale, capture d'e-mail (waitlist) et analytics (Vercel Analytics) sont déployées dans `office-translator-landing-page/`.
|
||||
> 3. **Un endpoint public d'inscription** existe : `POST /api/v1/waitlist` (dédoublonné, persisté dans `data/waitlist.json`), compteur `GET /api/v1/waitlist/count`.
|
||||
> 4. La mention « SOC 2 Compliant » (infondée) a été retirée du footer.
|
||||
>
|
||||
> **Alignement code du 2026-08-29** (voir `docs/marketing/PLAN-ALIGNEMENT-CODE.md`) : le PDF est supporté (dont **PDF scannés via OCR Mistral**) et l'ajoutera à tous les supports ; rétention réelle = envois 30 min / résultats ≤ 2 h ; les moteurs vendus sont exactement ceux des plans (`PLANS[plan]["providers"]`, désormais appliqué côté API) ; 100+ langues exposées via `/api/v1/languages`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Positionnement & Proposition de Valeur
|
||||
|
||||
### Le Produit
|
||||
**Office Translator** est un service SaaS de traduction de documents professionnels (Word, Excel, PowerPoint) qui **préserve parfaitement la mise en forme, les tableaux, les images et les styles** du document original.
|
||||
**Office Translator** (marque **Wordly.art**) est un SaaS de traduction de documents professionnels (Word, Excel, PowerPoint, **PDF — y compris scannés, via OCR**) qui **préserve la mise en forme, les formules, les tableaux, les images et les styles** du document original. L'utilisateur choisit son moteur de traduction par document.
|
||||
|
||||
### Ce qui nous différencie
|
||||
- **Préservation du format** : Contrairement à Google Translate ou DeepL qui détruisent les layouts complexes, notre moteur maintient la structure exacte
|
||||
- **Multi-providers** : L'utilisateur choisit son moteur (Google, DeepL, OpenAI, DeepSeek, OpenRouter, Minimax, Zai) selon son budget et ses besoins
|
||||
- **Glossaires techniques** : Terminologie personnalisée (HVAC, IT, Juridique, Médical)
|
||||
> Nommage : « Office Translator » est le nom du dépôt / nom de travail interne ; **Wordly.art** est la marque publique (domaine de lancement, landing, footer). Le footer de la landing affiche donc « Wordly.art » — c'est intentionnel.
|
||||
|
||||
### Marché cible
|
||||
| Segment | Taille estimée | Priorité |
|
||||
|---------|---------------|----------|
|
||||
| PME internationales ( traductions régulières) | Grand | P1 |
|
||||
| Agences de traduction (productivité) | Moyen | P1 |
|
||||
| Départements RH multilingues | Moyen | P2 |
|
||||
| Freelancers / consultants | Grand | P2 |
|
||||
| Étudiants & academics | Grand | P3 (freemium) |
|
||||
### Accroche unique (USP)
|
||||
> « Traduit en place. Zéro perte de mise en page. »
|
||||
|
||||
Contrairement à Google Translate (qui aplatit le fichier en texte brut) ou à DeepL (formatage limité), le moteur translate *in place* : cellules fusionnées, formules Excel, en-têtes/pieds de page Word, animations PowerPoint restent intacts. Les PDF scannés (pages image) sont récupérés par **OCR Mistral** avant traduction — DeepL et Azure les refusent. Les textes dans les images sont traduits via des modèles vision (plans payants).
|
||||
|
||||
### Différenciateurs
|
||||
| Différenciateur | Détail | Preuve produit |
|
||||
|---|---|---|
|
||||
| Préservation du format | Structure préservée (formules, fusions, styles) ; PDF scannés via OCR | openpyxl / python-docx / python-pptx / PyMuPDF + `services/mistral_ocr.py`, tests de réintégration |
|
||||
| Multi-moteurs (7) | Google, DeepL, Google Cloud, OpenRouter éco **et** premium, OpenAI, Grok (xAI) — grille appliquée côté API par plan | Choix par document, `PLANS[plan]["providers"]` |
|
||||
| IA contextuelle | Traduction LLM contextuelle (DeepSeek / Claude / Gemini) sur plans payants | `ai_translation` + `ai_tier` |
|
||||
| Glossaires techniques | Terminologie personnalisée (CVC, IT, Légal, Médical…) | `routes/glossary_routes.py` |
|
||||
| 60+ langues | 100+ codes validés, exposés via `/api/v1/languages` | `middleware/validation.py` |
|
||||
| Confiance | Suppression auto des fichiers (envois 30 min, résultats ≤ 2 h), TLS, jamais utilisé pour l'entraînement | `config.py` (TTLs), docs sécurité |
|
||||
|
||||
### Segments cibles (priorisés)
|
||||
| Segment | Douleur | Taille | Priorité | Message clé |
|
||||
|---|---|---|---|---|
|
||||
| **PME internationales** (export, docs commerciaux/techniques récurrents) | Traducteurs humains chers et lents, formats cassés | Grand | **P1** | « Le coût par page, sans casser le format » |
|
||||
| **Agences de traduction** (productivité) | Revue manuelle des mises en page, marges écrasées | Moyen | **P1** | « 7 moteurs au choix, glossaires partagés, 5 sièges » |
|
||||
| **Départements RH multilingues** (contrats, onboarding) | Volume saisonnier, terminologie juridique | Moyen | P2 | « Glossaires légaux + volume Business » |
|
||||
| **Freelances / consultants** | Outils gratuits qui détruisent leur travail | Grand | P2 | « 2 docs gratuits/mois, Starter à 9 € » |
|
||||
| **Étudiants & académiques** | Budget nul | Grand | P3 (freemium) | « Plan Free, sans carte bancaire » |
|
||||
|
||||
---
|
||||
|
||||
## 2. Supports Visuels Requis
|
||||
## 2. Tarification (source de vérité : `models/subscription.py` + `services/pricing_config.py`)
|
||||
|
||||
### A. Captures d'écran (OBLIGATOIRES - priorité maximale)
|
||||
**Monnaie : EUR. Billing Stripe implémenté (checkout, webhooks, portail client, crédits à l'unité).**
|
||||
|
||||
| # | Capture | Usage | Instructions |
|
||||
|---|---------|-------|-------------|
|
||||
| 1 | **Page d'accueil / Hero** | Landing page, réseaux sociaux | Montrer l'interface épurée avec le drop-zone de fichier |
|
||||
| 2 | **Upload en cours** | Démonstration du workflow | Fichier Excel chargé avec sélection langue source/cible |
|
||||
| 3 | **Résultat côte à côte** | Preuve de qualité | Document original vs traduit, montrer que le format est intact |
|
||||
| 4 | **Sélection du provider** | Fonctionnalité clé | Dropdown Google/DeepL/OpenAI/DeepSeek avec prix affichés |
|
||||
| 5 | **Glossaire technique** | Différenciation | Interface de gestion des glossaires personnalisés |
|
||||
| 6 | **Dashboard admin** | Crédibilité entreprise | Vue monitoring avec statistiques d'utilisation |
|
||||
| 7 | **Page pricing/forfaits** | Conversion | Les 3 tiers (Starter/Pro/Business) clairement affichés |
|
||||
| 8 | **Profil utilisateur** | Confiance | Page de profil avec historique et quota |
|
||||
| Plan | Mensuel | Annuel (−20 %) | Docs/mois | Pages/doc max | Fichier max | Moteurs | IA | API |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **Free** | 0 € | — | 2 | 10 | 5 Mo | Google | Non (filigrane) | Non |
|
||||
| **Starter** | 9 € | 7,20 €/mois | 50 | 50 | 10 Mo | Google, DeepL | Non | Non |
|
||||
| **Pro** ★ populaire | 19 € | 15,20 €/mois | 200 | 200 | 25 Mo | + Google Cloud, OpenRouter | Essentielle (DeepSeek) | Non |
|
||||
| **Business** | 49 € | 39,20 €/mois | 1 000 | 500 | 50 Mo | + OpenAI, x.ai/Zai, premium | Premium (Claude/Gemini) | Oui, 10 000 appels/mois, 5 sièges |
|
||||
| **Enterprise** | sur demande | sur demande | illimité | illimité | illimité | + custom | custom | illimitable |
|
||||
|
||||
**Format** : PNG, 1280x720 minimum, fond clair et sombre
|
||||
|
||||
### B. Vidéo de Démonstration (OBLIGATOIRE)
|
||||
|
||||
#### Vidéo courte (60-90 secondes) - "How it works"
|
||||
- **Objectif** : Landing page + réseaux sociaux
|
||||
- **Script suggéré** :
|
||||
1. (0-5s) Logo + tagline animé
|
||||
2. (5-15s) Problème : "Traduire un Excel de 50 pages sans casser le format ? Mission impossible."
|
||||
3. (15-40s) Démo accélérée : Upload → Sélection langue → Provider → Traduction → Download
|
||||
4. (40-55s) Split-screen : document original vs traduit, zoom sur tableaux/images intacts
|
||||
5. (55-65s) CTA : "Essayez gratuitement" + URL
|
||||
- **Format** : MP4 1080p, sous-titres FR + EN
|
||||
|
||||
#### Vidéo tutoriel (3-5 minutes) - "Guide complet"
|
||||
- **Objectif** : YouTube, onboarding utilisateurs
|
||||
- **Contenu** : Création de compte, upload, glossaires, providers, download
|
||||
- **Format** : Screencast avec voiceover
|
||||
|
||||
### C. Autres assets visuels
|
||||
|
||||
| Asset | Spécifications |
|
||||
|-------|---------------|
|
||||
| **Logo SVG** | Version claire + sombre, icône seule + avec texte |
|
||||
| **OG Image** | 1200x630px pour partage réseaux sociaux |
|
||||
| **Favicon** | 32x32, 16x16, ICO + PNG |
|
||||
| **Bannière GitHub** | 1280x640px pour le repo README |
|
||||
| **GIF animé** | 15s loop du workflow upload→traduction→download |
|
||||
| **Infographie** | "Pourquoi Office Translator" - comparaison avant/après |
|
||||
|
||||
---
|
||||
|
||||
## 3. Canaux de Lancement
|
||||
|
||||
### Phase 1 : Pré-lancement (Semaines 1-2)
|
||||
|
||||
| Canal | Action | Priorité |
|
||||
|-------|--------|----------|
|
||||
| **Landing page** | Mettre en place avec captures d'écran + vidéo + formulaire email | CRITIQUE |
|
||||
| **Product Hunt** | Préparer le launch (assets, description, maker comment) | HAUTE |
|
||||
| **Reddit** | Posts dans r/SideProject, r/saas, r/translator | HAUTE |
|
||||
| **Hacker News** | Préparer un "Show HN" technique | HAUTE |
|
||||
| **Twitter/X** | Thread de lancement avec démo GIF | MOYENNE |
|
||||
| **LinkedIn** | Post professionnel ciblant PME et agences | MOYENNE |
|
||||
|
||||
### Phase 2 : Lancement (Semaine 3)
|
||||
|
||||
| Canal | Action |
|
||||
|-------|--------|
|
||||
| **Product Hunt** | Lancement le mardi ou mercredi (meilleur trafic) |
|
||||
| **Reddit** | Cross-post dans 5-6 subreddits pertinents |
|
||||
| **Hacker News** | Soumettre le Show HN le matin (heure EST) |
|
||||
| **Twitter/X** | Thread avec GIF + link Product Hunt |
|
||||
| **IndieHackers** | Post détaillé sur le build process |
|
||||
| **Dev.to** | Article technique sur l'architecture |
|
||||
| **Twitter/X (communautés)** | Cibler #BuildInPublic, #SaaS, #i18n |
|
||||
|
||||
### Phase 3 : Croissance (Semaines 4-8)
|
||||
|
||||
| Canal | Action |
|
||||
|-------|--------|
|
||||
| **SEO** | Articles de blog : "Comment traduire un Excel sans perdre le format", etc. |
|
||||
| **YouTube** | Tutoriels et reviews |
|
||||
| **Partenariats** | Agences de traduction, consultants internationaux |
|
||||
| **Google Ads** | Mots-clés "translate excel document", "translate powerpoint" |
|
||||
| **Communautés** | Discord/Slack de développeurs et traducteurs |
|
||||
| **AppSumo** | Liste Lifetime Deal pour traction initiale |
|
||||
|
||||
---
|
||||
|
||||
## 4. Stratégie de Contenu
|
||||
|
||||
### Articles de Blog (SEO) - Minimum 5 au lancement
|
||||
|
||||
1. **"Comment traduire un fichier Excel sans perdre la mise en forme"**
|
||||
2. **"Les 5 meilleurs outils de traduction de documents comparés (2026)"**
|
||||
3. **"Traduction professionnelle : guide complet pour les PME"**
|
||||
4. **"Pourquoi DeepL et Google Translate détruisent vos documents Excel"**
|
||||
5. **"Auto-héberger son outil de traduction : guide complet"**
|
||||
|
||||
### Contenu Réseaux Sociaux (Répétitif)
|
||||
|
||||
| Type | Fréquence | Plateforme |
|
||||
|------|-----------|-----------|
|
||||
| Astuce traduction | 2x/semaine | Twitter, LinkedIn |
|
||||
| Before/After document | 1x/semaine | Twitter, Instagram |
|
||||
| Thread technique | 1x/2 semaines | Twitter |
|
||||
| Témoignage client | Quand disponible | Tous |
|
||||
| Mise à jour produit | Selon releases | Tous |
|
||||
|
||||
---
|
||||
|
||||
## 5. Stratégie Tarifaire & Positionnement Prix
|
||||
|
||||
### Forfaits actuels (à communiquer)
|
||||
|
||||
| Plan | Prix | Public cible | Message clé |
|
||||
|------|------|-------------|-------------|
|
||||
| **Starter** | Prix entry-level | Freelancers, étudiants | "Testez sans risque" |
|
||||
| **Pro** | Prix milieu | PME, consultants | "Le meilleur rapport qualité-prix" |
|
||||
| **Business** | Prix premium | Agences, entreprises | "Volume illimité + support dédié" |
|
||||
**Paquets de crédits** (au-delà des quotas, sans changer d'abonnement) : 50 cr. 5 € · 100 cr. 9 € · 250 cr. 20 € · 500 cr. 35 € · 1 000 cr. 60 € (0,06–0,10 €/page).
|
||||
|
||||
### Message tarifaire
|
||||
- Insister sur le **coût par page** vs traduction humaine (généralement 50-100x moins cher)
|
||||
- Mettre en avant le **freemium** ou l'**essai gratuit** si disponible
|
||||
- Comparer avec les coûts des solutions concurrentes
|
||||
- **Ancrage** : coût par page vs traduction humaine (50–100× moins cher) et vs réfection manuelle après Google Translate.
|
||||
- **Freemium réel** : le plan Free (2 docs/mois, sans carte) sert d'essai — c'est le « free trial » opérationnel.
|
||||
- **Transparence** : les prix exacts ci-dessus sont ceux du backend (`models/subscription.py` + `services/pricing_config.py`) ; toute promo est gérée via `data/pricing_overrides.json` (admin). À noter : la section pricing de la landing (`components/pricing-section.tsx`) est un **miroir statique** de ces plans (constante `PLANS`) — aucun prix n'est généré automatiquement, donc tout changement de prix/fonctionnalité doit être répercuté **à la main** dans ce composant et dans ce document le même jour.
|
||||
|
||||
---
|
||||
|
||||
## 6. KPIs à Suivre
|
||||
## 3. Actifs Marketing en Place (état 2026-08-29)
|
||||
|
||||
| Métrique | Objectif Mois 1 | Objectif Mois 3 |
|
||||
|----------|----------------|-----------------|
|
||||
| Visiteurs uniques | 5,000 | 25,000 |
|
||||
| Inscriptions | 200 | 1,500 |
|
||||
| Documents traduits | 500 | 5,000 |
|
||||
| Taux de conversion | 2% | 4% |
|
||||
| NPS | > 40 | > 50 |
|
||||
| Revenue mensuel | Variable | Variable |
|
||||
| Actif | Statut | Emplacement |
|
||||
|---|---|---|
|
||||
| Landing page (hero, démo de traduction) | ✅ | `office-translator-landing-page/` (Next.js 15 + Tailwind) |
|
||||
| Section **Pricing** (4 plans + toggle mensuel/annuel + bannière Enterprise) | ✅ | `components/pricing-section.tsx` |
|
||||
| Section **FAQ** (10 questions) | ✅ | `components/faq-section.tsx` |
|
||||
| Section **Preuve sociale** (stats vérifiables : 60+ langues, 7 moteurs, 100 % format, TTL 60 min + 3 témoignages génériques à remplacer par de vrais clients) | ✅ | `components/social-proof-section.tsx` |
|
||||
| **Capture d'e-mail** (waitlist : e-mail + segment d'intérêt) | ✅ | `components/waitlist-section.tsx` → `POST /api/v1/waitlist` |
|
||||
| **Analytics** (Vercel Analytics : pages, événements, heatmaps) | ✅ | `app/layout.tsx` (`@vercel/analytics`) |
|
||||
| Footer corrigé (mention SOC 2 retirée) | ✅ | `app/page.tsx` |
|
||||
|
||||
### À produire avant le lancement (voir §6, checklist)
|
||||
- 8 captures d'écran réelles de l'interface (héros, upload, résultat côte à côte, sélecteur de moteur, glossaires, dashboard, pricing, profil)
|
||||
- Vidéo démo 60–90 s + tutoriel 3–5 min
|
||||
- GIF 15 s du workflow upload → traduction → téléchargement
|
||||
- OG image 1200×630 + bannière GitHub + infographie avant/après
|
||||
- Remplacement des 3 témoignages génériques par 3 témoignages clients réels (objectif : 1 par segment P1/P2)
|
||||
|
||||
---
|
||||
|
||||
## 7. Plan d'Action pour l'Agent Marketing
|
||||
## 4. Canaux de Lancement
|
||||
|
||||
### Checklist Exécutable
|
||||
### Phase 1 — Pré-lancement (Semaines 1–2)
|
||||
| Canal | Action | Responsable | Livrable |
|
||||
|---|---|---|---|
|
||||
| Landing page | Publication en prod (Vercel ou Docker) avec URL propre `wordly.art` | Dev | URL + OG image |
|
||||
| Waitlist | Activation `POST /api/v1/waitlist` en prod ; objectif **150 e-mails** avant lancement | Marketing | Compteur `/api/v1/waitlist/count` |
|
||||
| Product Hunt | Préparation du listing (assets, tagline, maker comment, 5 commentaires amis planifiés) | Marketing | `docs/marketing/launch/product-hunt.md` |
|
||||
| Reddit | Rédaction de 3 posts (r/SideProject, r/translator, r/smallbusiness) | Marketing | `docs/marketing/launch/reddit-posts.md` |
|
||||
| Hacker News | Rédaction du Show HN (angle technique : préservation du format) | Dev | `docs/marketing/launch/show-hn.md` |
|
||||
| X/Twitter | Thread de lancement (12 tweets + GIF) | Marketing | `docs/marketing/launch/x-thread.md` |
|
||||
| LinkedIn | Post ciblant PME/agences (fondateur) | Fondateur | Post + 2 relances |
|
||||
|
||||
- [ ] **Captures d'écran** : Réaliser les 8 captures listées en section 2A
|
||||
- [ ] **Vidéo courte** : Produire la démo 60-90s (section 2B)
|
||||
- [ ] **Vidéo tutoriel** : Produire le screencast 3-5 min
|
||||
- [ ] **Landing page** : Concevoir et publier avec tous les assets
|
||||
- [ ] **Product Hunt** : Préparer le listing complet
|
||||
- [ ] **Reddit posts** : Rédiger 5 posts adaptés par subreddit
|
||||
- [ ] **Show HN** : Écrire la soumission Hacker News
|
||||
- [ ] **Twitter thread** : Préparer le thread de lancement (10-15 tweets)
|
||||
- [ ] **Articles SEO** : Rédiger les 5 articles de blog
|
||||
- [ ] **OG Image** : Créer l'image de partage réseaux sociaux
|
||||
- [ ] **GIF animé** : Créer le loop de 15s du workflow
|
||||
- [ ] **Infographie comparatif** : Créer le visuel avant/après
|
||||
- [ ] **Setup analytics** : Google Analytics + Mixpanel/PostHog
|
||||
- [ ] **Email sequence** : 5 emails onboarding post-inscription
|
||||
- [ ] **FAQ page** : Répondre aux 10 questions les plus fréquentes
|
||||
### Phase 2 — Lancement (Semaine 3)
|
||||
| Jour | Action |
|
||||
|---|---|
|
||||
| J-2 | Test du tunnel complet : inscription → 1ère traduction → upgrade Stripe (mode test) |
|
||||
| J0 | Lancement Product Hunt (mardi/mercredi), Show HN (matin EST), thread X, cross-post Reddit, post LinkedIn |
|
||||
| J1 | Suivi des commentaires (réponse < 2 h), relance X, e-mail waitlist n°1 « vous êtes dedans » |
|
||||
| J+7 | E-mail waitlist n°2 : 3 études avant/après + offre de lancement (−20 % la 1ʳᵉ année, code via `pricing_overrides.json`) |
|
||||
|
||||
### Phase 3 — Croissance (Semaines 4–8)
|
||||
| Canal | Action |
|
||||
|---|---|
|
||||
| SEO | Publication des 5 articles de blog (§5) ; pages « traduire un excel sans perdre le format », « traduire un powerpoint en anglais » |
|
||||
| YouTube | Tutoriel 3–5 min + review avant/après |
|
||||
| Partenariats | 5 agences de traduction (commission 15 % la 1ʳᵉ année), 3 consultants internationaux, annuaires SaaS français |
|
||||
| Google Ads | Campagne test (10 €/jour) sur « traduire excel » / « translate powerpoint » uniquement si CAC < 3× marge mensuelle |
|
||||
| Communautés | Discord/Slack de traducteurs et de devs (Build in public) |
|
||||
| AppSumo | Lifetime deal Business (optionnel, à valider après semaine 4) |
|
||||
|
||||
---
|
||||
|
||||
## 8. Concurrence Directe
|
||||
## 5. Stratégie de Contenu
|
||||
|
||||
| Concurrent | Força | Faiblesse | Notre avantage |
|
||||
|-----------|-------|-----------|---------------|
|
||||
| **Google Translate (docs)** | Gratuit, connu | Détruit les formats complexes | Préservation du format |
|
||||
| **DeepL (docs)** | Qualité de traduction | Cher, formatage limité | Multi-provider + prix |
|
||||
| **DocTranslator** | Simple | Qualité inégale, publicité | Interface pro + glossaires |
|
||||
| **Smartcat** | Complet | Complexe, cher | Simplicité + multi-providers |
|
||||
| **Transifex** | Enterprise | Trop cher pour PME | Prix accessible |
|
||||
### Articles de blog (SEO) — 5 au lancement, dans `docs/marketing/blog/`
|
||||
1. **« Comment traduire un fichier Excel sans perdre la mise en forme »** (pilier, 1 500+ mots)
|
||||
2. **« Les 5 meilleurs outils de traduction de documents comparés (2026) »** (comparatif : nous vs Google, DeepL, Smartling, Transifex, Smartcat)
|
||||
3. **« Traduction professionnelle de documents : guide complet pour les PME »**
|
||||
4. **« Pourquoi DeepL et Google Translate détruisent vos documents Excel (et quoi faire) »**
|
||||
5. **« Auto-héberger son outil de traduction : guide Docker complet »** (capture de l'audience self-hosted)
|
||||
|
||||
### Rythme social (répétitif, 3 h/semaine max)
|
||||
| Type | Fréquence | Plateforme |
|
||||
|---|---|---|
|
||||
| Avant/Après document (anonymisé) | 1×/sem. | X, LinkedIn |
|
||||
| Astuce traduction | 2×/sem. | X |
|
||||
| Mise à jour produit / build-in-public | selon releases | X, LinkedIn, IndieHackers |
|
||||
| Témoignage client | au fil de l'eau | toutes |
|
||||
|
||||
---
|
||||
|
||||
## 9. Timeline Recommandée
|
||||
## 6. KPIs & Mesure
|
||||
|
||||
### Mesure (déjà branchée)
|
||||
- **Vercel Analytics** sur la landing (pages, événements, top pages, referrers).
|
||||
- **Waitlist** : `GET /api/v1/waitlist/count` + segmentation par intérêt (champ `interest`).
|
||||
- **Backend** : `/health` (monitoring) ; quotas par plan visibles dans le dashboard admin.
|
||||
|
||||
### Objectifs
|
||||
| Métrique | Mois 1 | Mois 3 | Source |
|
||||
|---|---|---|---|
|
||||
| Visiteurs uniques landing | 5 000 | 25 000 | Vercel Analytics |
|
||||
| E-mails waitlist | 150 | 1 500 | `/api/v1/waitlist/count` |
|
||||
| Inscriptions (tous plans) | 200 | 1 500 | BDD utilisateurs |
|
||||
| Documents traduits | 500 | 5 000 | Journal de jobs |
|
||||
| Taux de conversion visite → inscription | 2 % | 4 % | Analytics |
|
||||
| MRR | variable | variable (cible : 300 abonnements payants) | Stripe |
|
||||
| NPS | > 40 | > 50 | Enquête post-traduction |
|
||||
| CAC (Google Ads) | — | < 3× marge mensuelle du plan d'entrée | Ads |
|
||||
|
||||
---
|
||||
|
||||
## 7. Concurrence (actualisé 2026-08)
|
||||
|
||||
| Concurrent | Force | Faiblesse | Notre avantage |
|
||||
|---|---|---|---|
|
||||
| **Google Docs (traduction intégrée)** | Gratuit, connu | Détruit les formats complexes, pas de glossaire | Préservation du format + multi-moteurs |
|
||||
| **DeepL (documents)** | Qualité de traduction | Prix élevé, formatage limité, moteur unique | 7 moteurs au choix, 60+ langues, prix |
|
||||
| **Smartling / Smartcat** | Complets, TMS | Lourds, chers, surdimensionnés pour une PME | Simplicité + prix accessible |
|
||||
| **Transifex** | Enterprise i18n | Trop cher, orienté localisation logicielle | Fichiers bureautiques, pas de code |
|
||||
| **Outils one-shot (DocTranslator, etc.)** | Simple | Qualité inégale, pub, aucune API | Interface pro, glossaires, API Business |
|
||||
|
||||
---
|
||||
|
||||
## 8. Timeline Recommandée
|
||||
|
||||
```
|
||||
Semaine 1-2 : Production des assets (captures, vidéos, images)
|
||||
Semaine 3 : Lancement sur Product Hunt + Reddit + HN + Twitter
|
||||
Semaine 4 : Articles SEO + contenu evergreen
|
||||
Semaine 5-6 : Partenariats + communautés + Google Ads
|
||||
Semaine 7-8 : Optimisation basée sur les premiers retours
|
||||
S1-2 : Publication de la landing + waitlist + production des assets (captures, vidéos, GIF)
|
||||
S3 : Lancement (Product Hunt + Show HN + Reddit + X + LinkedIn) + e-mails waitlist
|
||||
S4 : 5 articles SEO + YouTube + 5 partenariats agences
|
||||
S5-6 : Google Ads (test) + communautés + relances
|
||||
S7-8 : Optimisation (A/B du pricing, témoignages réels, LTV)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Ce document doit être mis à jour au fur et à mesure des retours utilisateur et des métriques observées.*
|
||||
## 9. Checklist Exécutable (état au 2026-08-29)
|
||||
|
||||
- [x] Landing page : sections Pricing, FAQ, Preuve sociale, Waitlist
|
||||
- [x] Endpoint waitlist (`POST /api/v1/waitlist`, dédoublonné) + compteur
|
||||
- [x] Analytics (Vercel Analytics) branchée sur la landing
|
||||
- [x] Footer corrigé (SOC 2 retiré)
|
||||
- [x] Contenu de lancement rédigé (`docs/marketing/launch/`)
|
||||
- [x] 5 articles de blog rédigés (`docs/marketing/blog/`)
|
||||
- [x] Spec des assets visuels (`docs/marketing/launch/assets-spec.md`)
|
||||
- [x] Doc de suivi des KPIs (`docs/marketing/kpis.md`)
|
||||
- [ ] Publication de la landing page en production (`wordly.art`)
|
||||
- [ ] 8 captures d'écran réelles de l'interface
|
||||
- [ ] Vidéo démo 60–90 s + tutoriel 3–5 min
|
||||
- [ ] GIF workflow 15 s + OG image 1200×630
|
||||
- [ ] 3 témoignages clients réels (remplacer les génériques)
|
||||
- [ ] Lancement Product Hunt (assets + 5 commentaires planifiés)
|
||||
- [ ] Séquence e-mail : 5 e-mails post-inscription
|
||||
- [ ] Configuration GA4 (optionnel, en complément de Vercel Analytics)
|
||||
|
||||
---
|
||||
|
||||
*Ce document est la source de vérité marketing. Toute modification de prix passe par `data/pricing_overrides.json` (admin) et doit être répercutée ici le même jour.*
|
||||
|
||||
93
_bmad-output/audit-securite-2026-08-26.md
Normal file
93
_bmad-output/audit-securite-2026-08-26.md
Normal file
@@ -0,0 +1,93 @@
|
||||
# Audit de sécurité — office_translator
|
||||
|
||||
Date : 2026-08-26 · Branche : `production-deployment` · Périmètre : backend FastAPI, routes, middlewares, services, config, déploiement, dépôt git.
|
||||
|
||||
## Critiques
|
||||
|
||||
### C1. Secrets de production commis dans git
|
||||
`.env.production` est **suivi par git** (confirmé via `git ls-files`) et contient des valeurs réelles :
|
||||
- `JWT_SECRET_KEY` (ligne 65/81) — permet de forger n'importe quel JWT utilisateur/admin
|
||||
- `ADMIN_TOKEN_SECRET`, `POSTGRES_PASSWORD`, `GRAFANA_PASSWORD`, `STRIPE_SECRET_KEY`
|
||||
- `data/provider_settings.json` (suivi) contient un mot de passe SMTP en clair.
|
||||
- Un fichier `.db` (1) est suivi : hashes de mots de passe et enregistrements utilisateurs potentiels.
|
||||
|
||||
**Action** : rotation de TOUS ces secrets, suppression des fichiers (`git rm --cached`), purge de l'historique (`git filter-repo`), et déplacement hors du dépôt des clés TLS privées présentes à la racine (`*.key`, non suivies mais à côté du code).
|
||||
|
||||
### C2. Traversée de chemin dans l'ingestion par URL
|
||||
`routes/translate_routes.py:292-312` — le nom de fichier provient du `Content-Disposition` du serveur distant (contrôlé par l'attaquant puisque `file_url` est fourni par l'utilisateur) et est utilisé **non assaini** : `temp_path = config.UPLOAD_DIR / f"{unique_id}_{filename}"`. `filename="../..../evil.xlsx"` écrit hors de `UPLOAD_DIR`. La whitelist d'extensions ne bloque pas `..`. Contraste : l'upload direct est correctement assaini (`middleware/validation.py:239-259`).
|
||||
|
||||
### C3. SSRF par redirection
|
||||
`routes/translate_routes.py:263-267` — le hostname est vérifié une fois par `_is_ssrf_risk()` (solide par ailleurs), mais la requête utilise `follow_redirects=True` sans revalidation : une URL publique peut rediriger 302 vers `169.254.169.254`, `localhost`, plages privées. Fenêtre TOCTOU DNS-rebinding en plus.
|
||||
|
||||
### C4. Job de cleanup : bug de clé + purge orpheline sans âge minimum
|
||||
`middleware/cleanup.py:194` lit `metadata["file_path"]` alors que les fichiers sont suivis sous `"input_path"` (`translate_routes.py:837`) → `tracked_paths` toujours vide → **tout** fichier est classé orphelin, et la suppression orpheline n'a **aucune vérification d'âge** (`cleanup.py:236-241`) → boucle de 5 min supprime des fichiers de jobs en cours. DoS/raison de disponibilité. (`protect_file()` existe mais n'est jamais appelé.)
|
||||
|
||||
## Élevées
|
||||
|
||||
### H1. Zip-bomb non contré
|
||||
Aucune limite de ratio de décompression ni de taille décompressée sur OOXML : `openpyxl.load_workbook`, `docx.Document`, `zipfile.read()` directs (`translators/word_translator.py:834+`, `pptx_translator.py:544+`). Un fichier de <50 MB très compressé → Go en RAM → OOM worker. Seule la taille compressée est vérifiée.
|
||||
|
||||
### H2. Téléchargement / statut : contrôle de propriété défaillant (IDOR)
|
||||
- `routes/translate_routes.py:1879-1886` : le check de propriété sur `GET /download/{job_id}` n'est effectué que si un utilisateur est authentifié ; un appelant anonyme qui devine/fuit un job_id (12 hex ≈ 48 bits) télécharge le document traduit, et la suppression après 1er téléchargement permet un DoS contre le légitime propriétaire.
|
||||
- `routes/translate_routes.py:1690-1779` : `GET /translations/{job_id}` n'a **aucun** check de propriété → énumération de statut, noms de fichiers, erreurs.
|
||||
|
||||
### H3. Traduction anonyme
|
||||
`/translate` accepte les requêtes non authentifiées (`current_user` optionnel, ligne 547) → traitement gratuit au tarif payant, seule la limitation IP s'applique ; combiné au webhook_url, oracle d'egress gratuit. À confirmer si voulu (démo landing ?).
|
||||
|
||||
### H4. Rate limiting contournable via `X-Forwarded-For`
|
||||
`middleware/rate_limiting.py:249-261` et `routes/admin_routes.py:176-189` font confiance au premier XFF sans proxy de confiance configuré → rotation d'XFF factice pour contourner rate limit et verrouillage brute-force admin (le verrouillage et les sessions admin sont aussi en mémoire par worker, `admin_routes.py:47-52`).
|
||||
|
||||
## Moyennes
|
||||
|
||||
- **M1. XXE/hardening XML** : `lxml.etree.fromstring` sur des parties ZIP non fiables sans `resolve_entities=False` ni defusedxml (`word_translator.py:731,839…`, `pptx_translator.py:553…`). Mitigé par libxml2 ≥2.9, mais à durcir explicitement.
|
||||
- **M2. Rotation refresh token sans révocation** : `/refresh` (`routes/auth_routes.py:661-742`) ne révoque pas l'ancien refresh token (7 jours de vie, pas de détection de réutilisation). Reset de mot de passe sans révocation des sessions existantes.
|
||||
- **M3. Fallback PyJWT absent** : jetons signés en base64 **non signé** silencieusement acceptés (`services/auth_service.py:158-169`) ; garde de production OK mais login/verify ne vérifient pas `JWT_AVAILABLE`.
|
||||
- **M4. `/checkout/sync`** : ownership conditionnel (`services/payment_service.py:174-178`) si session sans `metadata.user_id`.
|
||||
- **M5. Endpoint legacy batch** (`legacy_routes.py:248-320`) : pas de magic bytes, nombre de fichiers non borné, traitement synchrone → épuisement CPU/RAM. `/metrics` legacy non authentifié.
|
||||
- **M6. Upload direct** : `await file.read()` charge tout le fichier (≤50 MB) en RAM avant vérification (`middleware/validation.py:116`) ; l'URL stream correctement.
|
||||
- **M7. Dépendances vulnérables** (`requirements.txt`) : `python-multipart==0.0.9` (CVE-2024-24762, upload !), `fastapi==0.109.0` (ReDoS), `pydantic==2.5.3`, `stripe==7.0.0`. À mettre à jour en priorité.
|
||||
- **M8. Temp files images** écrits dans le temp système, non couverts par le cleanup, fuités sur exception (`pptx_translator.py:1060-1066`, `word_translator.py:1273-1275`).
|
||||
|
||||
## Faibles
|
||||
|
||||
- Logout révoque un refresh token fourni sans check de propriété (`auth_routes.py:401-410`).
|
||||
- Flux Google OAuth access_token sans validation d'audience (`auth_routes.py:604-609`).
|
||||
- Admin mono-facteur mot de passe partagé ; envisager TOTP.
|
||||
- Erreurs URL qui divulguent `str(e)` + URL interne au client (`translate_routes.py:356-371`).
|
||||
- Caches mémoire non bornés (`_gc_key_cache`), jobs « processing » jamais purgés.
|
||||
- Secret webhook Stripe lu à l'import au lieu du runtime (`payment_service.py:26`).
|
||||
|
||||
## Points forts constatés
|
||||
|
||||
- Vérification Stripe webhook correcte (signature + idempotence + body brut).
|
||||
- JWT : algorithme épinglé HS256, jetons typés, révocation jti Redis, access 15 min.
|
||||
- Clés API : stockées en SHA-256, haute entropie, expiration/révocation serveur ; glossaires/prompts correctement scopés par `user_id` (pas d'IDOR là).
|
||||
- Anti-énumération (bcrypt factice constant-time, forgot-password toujours 200).
|
||||
- Validation magic-bytes + allowlist extensions sur le flux v1 ; `_is_ssrf_risk` fail-closed ; streaming avec cap d'octets côté URL.
|
||||
- Échec au démarrage en production si secrets manquants / CORS wildcard ; pas de SQL brut (ORM uniquement) ; bcrypt pour les mots de passe.
|
||||
|
||||
## Priorités de remédiation (ordre recommandé)
|
||||
|
||||
1. **Rotation immédiate** de tous les secrets de `.env.production` + purge historique git (C1).
|
||||
2. Assainir le filename de l'ingestion URL (réutiliser `FileValidator._sanitize_filename`) (C2).
|
||||
3. Revalider le SSRF à chaque hop de redirection (désactiver `follow_redirects`, suivre manuellement) (C3).
|
||||
4. Corriger la clé `file_path`→`input_path` + âge minimum pour suppression orpheline (C4).
|
||||
5. Exiger l'authentification + ownership sur `/download/{job_id}` et `/translations/{job_id}` (H2).
|
||||
6. Limites de ratio zip (H1) ; XFF/proxy de confiance + verrouillage Redis (H4).
|
||||
7. Mise à jour `python-multipart`, `fastapi`, `pydantic` (M7).
|
||||
|
||||
---
|
||||
|
||||
## Suivi des correctifs (2026-08-26)
|
||||
|
||||
**Corrigés au code :**
|
||||
- ✅ C1 (partie git) : `.env.production`, `.env.ionos`, `data/provider_settings.json`, `translations.db` retirés du suivi git + `.gitignore`. ⚠️ rotation des secrets + purge de l'historique restent à faire manuellement.
|
||||
- ✅ C2 : nom de fichier assaini dans l'ingestion URL (`_sanitize_url_filename`).
|
||||
- ✅ C3 : redirections suivies manuellement avec revalidation SSRF à chaque étape (max 5).
|
||||
- ✅ C4 : clés `input_path`/`file_path`/`output_path` reconnues + délai de grâce de 15 min avant suppression d'un fichier orphelin.
|
||||
- ✅ H1 : contrôle anti-fichier-piège (`validate_zip_safety`, ratio max 100:1, 1 Go décompressé) sur les flux v1 et legacy.
|
||||
- ✅ H2 : contrôle d'accès strict sur `/translations/{id}` et `/download/{id}` (propriétaire obligatoire ; jeton secret requis pour les jobs sans compte).
|
||||
- ✅ H3 (partiel) : les jobs anonymes nécessitent désormais le jeton secret ; la création anonyme reste possible.
|
||||
- ✅ M7 : `python-multipart` 0.0.9 → 0.0.20, `fastapi` 0.109.0 → 0.109.1.
|
||||
|
||||
**Restent à faire :** rotation des secrets + purge historique git (manuel), H4 (en-tête X-Forwarded-For), M2 (révocation refresh token), M4, M5, M6.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
title: 'Quick wins pipeline LLM : prompt, connexions, cache, event loop'
|
||||
type: 'bugfix'
|
||||
created: '2026-08-26'
|
||||
status: 'draft'
|
||||
context: []
|
||||
---
|
||||
|
||||
<frozen-after-approval reason="human-owned intent — do not modify unless human renegotiates">
|
||||
|
||||
## Intent
|
||||
|
||||
**Problem:** Le pipeline de traduction LLM souffre de 4 défauts vérifiés : (1) dans les 3 providers LLM, un prompt personnalisé/glossaire **remplace** le prompt par défaut — la paire de langues et les règles métier disparaissent de la requête ; (2) chaque segment traduit ouvre une nouvelle connexion TCP+TLS (`requests.post` nu) ; (3) le cache de traduction Redis/LRU (`services/translation_cache.py`) est écrit et testé mais **jamais branché** — re-traduire un fichier re-facture 100 % des appels ; (4) 7 appels bloquants (SHA-256 de fichiers, DB, disque, sonde réseau Google) s'exécutent directement dans l'event loop et peuvent geler tout le serveur.
|
||||
|
||||
**Approach:** 4 correctifs chirurgicaux dans la couche providers (openai/minimax/deepseek) et le runner de jobs de `translate_routes.py`, sans changement d'interface publique ni des translators.
|
||||
|
||||
## Boundaries & Constraints
|
||||
|
||||
**Always:** L'interface `TranslationProvider` / `TranslationRequest` / `TranslationResponse` reste inchangée ; la paire de langues doit **toujours** figurer dans le system prompt ; le cache reste piloté par env (`REDIS_CACHE_TTL`, `LRU_CACHE_MAXSIZE`, `REDIS_URL`) ; tout échec du cache est silencieux (fallback LRU, puis poursuite sans cache) ; `from_cache=True` sur les réponses servies du cache.
|
||||
|
||||
**Ask First:** modifier `build_full_prompt` (services/glossary_service.py) ; modifier les schémas providers ; introduire un nouveau backend de cache.
|
||||
|
||||
**Never:** Pas de batch multi-segments LLM (chantier suivant) ; pas de refonte de la double couche providers legacy/nouvelle ; pas de modification des translators/ ; pas de nouvel endpoint ; pas de persistance des jobs.
|
||||
|
||||
## I/O & Edge-Case Matrix
|
||||
|
||||
| Scenario | Input / State | Expected Output / Behavior | Error Handling |
|
||||
|----------|--------------|---------------------------|----------------|
|
||||
| Prompt custom actif | glossaire et/ou prompt utilisateur | System prompt = prompt par défaut formaté (avec paire de langues) **+** contexte custom en complément | N/A |
|
||||
| Segment déjà en cache | même texte + langues + provider + hash prompt | Réponse `from_cache=True`, 0 requête HTTP | N/A |
|
||||
| Segment répété dans un même document | 2e occurrence après succès de la 1ʳᵉ | Servie par le cache (le batch est séquentiel) | N/A |
|
||||
| Redis indisponible | `get_cache()` init ou `set()` en échec | Fallback LRU RAM puis non-bloquant ; traduction réussie quand même | Log warning, jamais d'exception |
|
||||
| Provider en erreur | échec API après retries | Rien n'est écrit dans le cache | Erreur existante propagée |
|
||||
| Fichier 50 Mo + glossaire configuré | job lancé pendant que d'autres requêtes arrivent | SHA-256, zip-safety, DB, settings et sonde Google hors event loop (`asyncio.to_thread`) | Erreurs existantes propagées |
|
||||
|
||||
</frozen-after-approval>
|
||||
|
||||
## Code Map
|
||||
|
||||
- `services/providers/openai_provider.py` -- `_build_system_prompt` :120-128 (bug du remplacement), `requests.post` :263, `translate_text` :426-455 (point d'entrée cache)
|
||||
- `services/providers/minimax_provider.py` -- prompt inline :181-183, `requests.post` :117, `translate_text` :168+
|
||||
- `services/providers/deepseek_provider.py` -- prompt inline :164-166, `requests.post` :111, `translate_text` :151+
|
||||
- `services/translation_cache.py` -- API prête et testée : `make_cache_key` :45, `hash_prompt` :77, `get_cache()` :402 (Redis auto + fallback LRU) — aucun appelant hors tests
|
||||
- `routes/translate_routes.py` -- appels bloquants dans coroutines : `calculate_sha256` :858/:871, `validate_zip_safety` :884, `_load_admin_settings` :1174, `get_glossary_terms` :1191, `get_prompt_content` :1206, `_google_cloud_key_valid` :1242 (fait une traduction HTTP de test synchrone)
|
||||
- `tests/test_providers/` -- 8 fichiers existants, dont `test_minimax_provider.py` (récent) : patterns de mock à réutiliser
|
||||
|
||||
## Tasks & Acceptance
|
||||
|
||||
**Execution:**
|
||||
- [ ] `services/providers/openai_provider.py` -- (1) `_build_system_prompt` : concaténer le prompt par défaut formaté + le custom au lieu de `return custom_prompt` ; (2) `requests.Session` d'instance réutilisée dans `_make_api_request` ; (3) `translate_text` : lookup `get_cache()` + `make_cache_key(..., custom_prompt_hash=hash_prompt(custom_prompt))` avant l'appel, `cache.set` après succès -- corrige le bug qualité, la latence et le coût
|
||||
- [ ] `services/providers/minimax_provider.py` -- mêmes 3 changements (prompt construit inline :181-183)
|
||||
- [ ] `services/providers/deepseek_provider.py` -- mêmes 3 changements (prompt construit inline :164-166)
|
||||
- [ ] `routes/translate_routes.py` -- envelopper les 7 appels bloquants listés dans la Code Map dans `asyncio.to_thread` -- l'event loop reste réactif pendant les jobs
|
||||
- [ ] `tests/test_providers/` -- nouveaux tests couvrant la matrice I/O : le system prompt contient la paire de langues **et** le custom (×3 providers) ; cache hit → 0 appel HTTP et `from_cache=True` ; erreur provider → rien mis en cache ; Redis down → traduction quand même
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- Given un custom_prompt sans mention de langue, when traduction via openai/minimax/deepseek, then le payload contient « from {source} to {target} » et le contenu custom.
|
||||
- Given un segment déjà en cache (même clé), when re-traduit, then aucune requête HTTP n'est émise vers l'API LLM.
|
||||
- Given Redis down, when traduction, then réponse normale via LRU ou sans cache, sans exception remontée au job.
|
||||
- Given la suite de tests existante, when `pytest -x`, then 0 régression.
|
||||
|
||||
## Spec Change Log
|
||||
|
||||
## Design Notes
|
||||
|
||||
Concaténation du prompt (les 3 providers) :
|
||||
|
||||
```python
|
||||
system_prompt = DEFAULT_TRANSLATION_PROMPT.format(
|
||||
source_lang=source_lang_name, target_lang=target_lang_name
|
||||
)
|
||||
if custom_prompt:
|
||||
system_prompt += (
|
||||
"\n\nAdditional context and instructions from the user "
|
||||
f"(comply without overriding the language pair above):\n{custom_prompt}"
|
||||
)
|
||||
```
|
||||
|
||||
La `requests.Session` est créée dans `__init__` du provider (le pooling urllib3 est thread-safe ; les translators appellent `translate_text` depuis 6 threads). Clé de cache : réutiliser `make_cache_key(text, target_language, source_language, self._provider_name, custom_prompt_hash=hash_prompt(custom_prompt))` — le hash du prompt isole déjà les glossaires différents ; `user_id` absent des metadata aujourd'hui → valeur par défaut « anon » (partage inter-utilisateurs acceptable : mémoire de traduction standard, TTL 24 h).
|
||||
|
||||
## Verification
|
||||
|
||||
**Commands:**
|
||||
- `pytest tests/test_providers/ -x` -- expected: succès, nouveaux tests inclus
|
||||
- `pytest -x` -- expected: succès complet, 0 régression
|
||||
25
_bmad-output/implementation-artifacts/spec-securite-c1-c4.md
Normal file
25
_bmad-output/implementation-artifacts/spec-securite-c1-c4.md
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
status: done
|
||||
created: 2026-08-26
|
||||
title: Correctifs sécurité critique C1–C4
|
||||
---
|
||||
|
||||
# Spec : Correctifs sécurité C1–C4
|
||||
|
||||
## Contexte
|
||||
Audit de sécurité du 2026-08-26 ([rapport](../audit-securite-2026-08-26.md)). Correction des 4 constats critiques.
|
||||
|
||||
## Tâches
|
||||
1. **C1 — Secrets suivis par git** : `git rm --cached` sur `.env.production`, `.env.ionos`, `data/provider_settings.json`, `translations.db` ; compléter `.gitignore`. (Rotation des secrets + purge d'historique = action manuelle utilisateur, hors scope code.)
|
||||
2. **C2 — Path traversal URL** : dans `routes/translate_routes.py::download_from_url`, assainir le filename issu de `Content-Disposition`/URL (Path().name, contrôle-chars, length cap, fallback `downloaded_file`).
|
||||
3. **C3 — SSRF par redirection** : remplacer `follow_redirects=True` par une boucle manuelle (≤5 hops) qui revalide schéma + `_is_ssrf_risk()` à chaque hop.
|
||||
4. **C4 — Cleanup destructeur** : dans `middleware/cleanup.py::cleanup`, lire toutes les clés de chemin (`input_path`, `file_path`, `output_path`) et n'appliquer la suppression orpheline qu'au-delà d'un âge plancher (`orphan_grace_seconds`, défaut 900 s).
|
||||
|
||||
## Critères d'acceptation
|
||||
- **AC1** : Étant donné un `Content-Disposition: filename="../../evil.xlsx"`, quand `download_from_url` s'exécute, alors le fichier est écrit dans `UPLOAD_DIR` avec un nom sans traversée.
|
||||
- **AC2** : Étant donné une URL publique qui redirige (302) vers `http://169.254.169.254/`, quand `download_from_url` s'exécute, alors une erreur `ssrf_blocked` est levée.
|
||||
- **AC3** : Étant donné un fichier récent (< 15 min) non tracé dans Redis, quand `cleanup()` s'exécute, alors le fichier n'est PAS supprimé ; au-delà du plancher il l'est.
|
||||
- **AC4** : `git ls-files` ne contient plus `.env.production`, `.env.ionos`, `translations.db`, `data/provider_settings.json`.
|
||||
|
||||
## Tests
|
||||
- Tests unitaires pour le filename sanitizer, la boucle de redirection, et la logique orpheline du cleanup.
|
||||
20
config.py
20
config.py
@@ -70,8 +70,8 @@ class Config:
|
||||
|
||||
# ============== Quality Layer (L0) ==============
|
||||
# Track A1 of the dev plan — observability only, no behavior change.
|
||||
# Set to "true" to enable. Default: false (opt-in).
|
||||
QUALITY_L0_ENABLED = os.getenv("QUALITY_L0_ENABLED", "false").lower() == "true"
|
||||
# Enabled by default since 2026-08-29: log-only, never blocks a job.
|
||||
QUALITY_L0_ENABLED = os.getenv("QUALITY_L0_ENABLED", "true").lower() == "true"
|
||||
# Number of text samples to extract from the output file for L0 analysis.
|
||||
# Keep small to avoid overhead. 20 is enough to catch language confusion.
|
||||
QUALITY_L0_SAMPLE_SIZE = int(os.getenv("QUALITY_L0_SAMPLE_SIZE", "20"))
|
||||
@@ -117,6 +117,22 @@ class Config:
|
||||
# Set to false to use the legacy aggressive-shrink strategy (NOT recommended).
|
||||
PDF_SMART_FIT_ENABLED = os.getenv("PDF_SMART_FIT_ENABLED", "true").lower() == "true"
|
||||
|
||||
# ============== Scanned PDF OCR (Mistral) ==============
|
||||
# Image-only PDFs have no extractable text layer. When a PDF looks
|
||||
# scanned, it is routed to the Mistral OCR API to recover the text
|
||||
# before translation (output: clean re-typeset PDF).
|
||||
# Pricing reference: ~$1 / 1000 pages — set MISTRAL_OCR_ENABLED=false
|
||||
# to disable and reject scanned PDFs with an explicit error instead.
|
||||
MISTRAL_API_KEY = os.getenv("MISTRAL_API_KEY", "").strip()
|
||||
MISTRAL_OCR_MODEL = os.getenv("MISTRAL_OCR_MODEL", "mistral-ocr-latest")
|
||||
MISTRAL_OCR_TIMEOUT = int(os.getenv("MISTRAL_OCR_TIMEOUT", "180"))
|
||||
MISTRAL_OCR_ENABLED = os.getenv("MISTRAL_OCR_ENABLED", "true").lower() == "true"
|
||||
# A page with fewer extractable characters than this is text-poor; it
|
||||
# counts as a scan page only when raster images also cover most of it.
|
||||
SCANNED_PDF_MIN_CHARS_PER_PAGE = int(
|
||||
os.getenv("SCANNED_PDF_MIN_CHARS_PER_PAGE", "100")
|
||||
)
|
||||
|
||||
|
||||
# ============== API Configuration ==============
|
||||
API_TITLE = "Document Translation API"
|
||||
|
||||
137
core/languages.py
Normal file
137
core/languages.py
Normal file
@@ -0,0 +1,137 @@
|
||||
"""
|
||||
Language display names for LLM prompts and UI labels.
|
||||
|
||||
Single source of truth for code → English name across providers. The map
|
||||
covers every code exposed by /api/v1/languages (SUPPORTED_LANGUAGES), so
|
||||
prompts say "Translate to Tagalog" instead of "Translate to tl" — LLMs
|
||||
translate noticeably better with full language names.
|
||||
"""
|
||||
|
||||
from typing import Dict
|
||||
|
||||
LANGUAGE_NAMES: Dict[str, str] = {
|
||||
"af": "Afrikaans",
|
||||
"sq": "Albanian",
|
||||
"am": "Amharic",
|
||||
"ar": "Arabic",
|
||||
"hy": "Armenian",
|
||||
"az": "Azerbaijani",
|
||||
"eu": "Basque",
|
||||
"be": "Belarusian",
|
||||
"bn": "Bengali",
|
||||
"bs": "Bosnian",
|
||||
"bg": "Bulgarian",
|
||||
"ca": "Catalan",
|
||||
"ceb": "Cebuano",
|
||||
"zh": "Chinese",
|
||||
"zh-CN": "Chinese (Simplified)",
|
||||
"zh-TW": "Chinese (Traditional)",
|
||||
"co": "Corsican",
|
||||
"hr": "Croatian",
|
||||
"cs": "Czech",
|
||||
"da": "Danish",
|
||||
"nl": "Dutch",
|
||||
"en": "English",
|
||||
"eo": "Esperanto",
|
||||
"et": "Estonian",
|
||||
"fi": "Finnish",
|
||||
"fr": "French",
|
||||
"fy": "Frisian",
|
||||
"gl": "Galician",
|
||||
"ka": "Georgian",
|
||||
"de": "German",
|
||||
"el": "Greek",
|
||||
"gu": "Gujarati",
|
||||
"ht": "Haitian Creole",
|
||||
"ha": "Hausa",
|
||||
"haw": "Hawaiian",
|
||||
"he": "Hebrew",
|
||||
"hi": "Hindi",
|
||||
"hmn": "Hmong",
|
||||
"hu": "Hungarian",
|
||||
"is": "Icelandic",
|
||||
"ig": "Igbo",
|
||||
"id": "Indonesian",
|
||||
"ga": "Irish",
|
||||
"it": "Italian",
|
||||
"ja": "Japanese",
|
||||
"jv": "Javanese",
|
||||
"kn": "Kannada",
|
||||
"kk": "Kazakh",
|
||||
"km": "Khmer",
|
||||
"rw": "Kinyarwanda",
|
||||
"ko": "Korean",
|
||||
"ku": "Kurdish",
|
||||
"ky": "Kyrgyz",
|
||||
"lo": "Lao",
|
||||
"la": "Latin",
|
||||
"lv": "Latvian",
|
||||
"lt": "Lithuanian",
|
||||
"lb": "Luxembourgish",
|
||||
"mk": "Macedonian",
|
||||
"mg": "Malagasy",
|
||||
"ms": "Malay",
|
||||
"ml": "Malayalam",
|
||||
"mt": "Maltese",
|
||||
"mi": "Maori",
|
||||
"mr": "Marathi",
|
||||
"mn": "Mongolian",
|
||||
"my": "Myanmar (Burmese)",
|
||||
"ne": "Nepali",
|
||||
"no": "Norwegian",
|
||||
"ny": "Nyanja (Chichewa)",
|
||||
"or": "Odia (Oriya)",
|
||||
"ps": "Pashto",
|
||||
"fa": "Persian (Farsi)",
|
||||
"pl": "Polish",
|
||||
"pt": "Portuguese",
|
||||
"pa": "Punjabi",
|
||||
"ro": "Romanian",
|
||||
"ru": "Russian",
|
||||
"sm": "Samoan",
|
||||
"gd": "Scots Gaelic",
|
||||
"sr": "Serbian",
|
||||
"st": "Sesotho",
|
||||
"sn": "Shona",
|
||||
"sd": "Sindhi",
|
||||
"si": "Sinhala",
|
||||
"sk": "Slovak",
|
||||
"sl": "Slovenian",
|
||||
"so": "Somali",
|
||||
"es": "Spanish",
|
||||
"su": "Sundanese",
|
||||
"sw": "Swahili",
|
||||
"sv": "Swedish",
|
||||
"tl": "Filipino (Tagalog)",
|
||||
"tg": "Tajik",
|
||||
"ta": "Tamil",
|
||||
"tt": "Tatar",
|
||||
"te": "Telugu",
|
||||
"th": "Thai",
|
||||
"tr": "Turkish",
|
||||
"tk": "Turkmen",
|
||||
"uk": "Ukrainian",
|
||||
"ur": "Urdu",
|
||||
"ug": "Uyghur",
|
||||
"uz": "Uzbek",
|
||||
"vi": "Vietnamese",
|
||||
"cy": "Welsh",
|
||||
"xh": "Xhosa",
|
||||
"yi": "Yiddish",
|
||||
"yo": "Yoruba",
|
||||
"zu": "Zulu",
|
||||
}
|
||||
|
||||
|
||||
def language_name(code: str) -> str:
|
||||
"""Full English name for a language code; case-insensitive lookup.
|
||||
|
||||
Falls back to the code itself for unknown values (and "" for auto/None,
|
||||
which callers use to mean "detect the source language").
|
||||
"""
|
||||
if not code or code == "auto":
|
||||
return ""
|
||||
name = LANGUAGE_NAMES.get(code)
|
||||
if name is None:
|
||||
name = LANGUAGE_NAMES.get(code.split("-")[0].lower(), code)
|
||||
return name
|
||||
@@ -1,78 +0,0 @@
|
||||
{
|
||||
"google": {
|
||||
"enabled": true,
|
||||
"api_key": null,
|
||||
"base_url": null,
|
||||
"model": null,
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"google_cloud": {
|
||||
"enabled": false,
|
||||
"api_key": null,
|
||||
"base_url": null,
|
||||
"model": null,
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"deepl": {
|
||||
"enabled": false,
|
||||
"api_key": null,
|
||||
"base_url": null,
|
||||
"model": null,
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"openai": {
|
||||
"enabled": false,
|
||||
"api_key": null,
|
||||
"base_url": null,
|
||||
"model": "gpt-4o-mini",
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"ollama": {
|
||||
"enabled": false,
|
||||
"api_key": null,
|
||||
"base_url": "http://localhost:11434",
|
||||
"model": "gpt-oss:20b",
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"openrouter": {
|
||||
"enabled": true,
|
||||
"api_key": null,
|
||||
"base_url": null,
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"openrouter_premium": {
|
||||
"enabled": false,
|
||||
"api_key": null,
|
||||
"base_url": null,
|
||||
"model": "anthropic/claude-sonnet-4.6",
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"zai": {
|
||||
"enabled": false,
|
||||
"api_key": null,
|
||||
"base_url": "https://api.x.ai/v1",
|
||||
"model": "grok-2-1212",
|
||||
"timeout": 30,
|
||||
"max_retries": 3
|
||||
},
|
||||
"smtp": {
|
||||
"enabled": true,
|
||||
"host": "smtp.ionos.fr",
|
||||
"port": 587,
|
||||
"username": "admin@wordly.art",
|
||||
"password": "Esenaw,121151",
|
||||
"from_email": "admin@wordly.art",
|
||||
"use_tls": true
|
||||
},
|
||||
"fallback_chain": "google,deepl,openai,ollama,openrouter,zai",
|
||||
"fallback_chain_classic": "google,deepl",
|
||||
"fallback_chain_llm": "ollama,openai,openrouter,zai"
|
||||
}
|
||||
96
docs/AUDIT_APPROFONDIE_2026-08-29.md
Normal file
96
docs/AUDIT_APPROFONDIE_2026-08-29.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# Passe approfondie n°2 — Traduction, mise en page, formats, admin & benchmark
|
||||
|
||||
**Date :** 29 août 2026 (2ᵉ passe) · **Statut : corrections appliquées et testées**
|
||||
|
||||
Méthode : revue complète des 4 traducteurs (word/excel/pptx/pdf) ligne à ligne (35 constats), benchmark web actualisé de 9 concurrents, puis correction des P1.
|
||||
|
||||
---
|
||||
|
||||
## 1. Corrections appliquées aujourd'hui (toutes testées)
|
||||
|
||||
### Qualité de traduction & mise en page
|
||||
|
||||
| # | Correctif | Impact |
|
||||
|---|---|---|
|
||||
| 1 | **Word/PPTX→Word : fusion des runs de même formatage** — une phrase éclatée en runs adjacents identiques (découpes rsid, correcteur ortho.) est désormais traduite en UNE unité ; la traduction s'écrit dans le 1ᵉʳ run, les frères sont vidés, les espaces de bord préservés. Les changements de formatage (gras au milieu) restent des unités séparées avec leur contexte. | **Plus gros écart qualité vs DeepL comblé** : fini les fragments hors contexte (« This is » / « very » / « important » en 3 appels) |
|
||||
| 2 | **PPTX : les traductions de graphiques atteignent enfin le fichier** — l'ancien code assignait `ChartPart.blob` (propriété en lecture seule de python-pptx) : l'erreur était avalée et les titres/séries de graphiques n'étaient JAMAIS écrits. Réécriture du XML des graphiques dans le ZIP de sortie (même mécanisme que Word) + test de non-régression sur un VRAI graphique | Titres/axes/séries traduits pour de vrai |
|
||||
| 3 | **Excel : le renommage d'onglets ne casse plus les formules** — openpyxl ne réécrit pas les références ; ajout de la réécriture dans : formules de cellules (multi-réfs, refs 3D, quoted/unquoted), noms définis, validations de données, mises en forme conditionnelles. Test dédié (`test_excel_sheet_rename_refs.py`) | `=SUM(Ventes!A3:A4)` → `=SUM(Sales!A3:A4)` au lieu de `#REF!` |
|
||||
| 4 | **Word : commentaires/bulles traduits** (`word/comments.xml`, même mécanisme post-save que les notes de bas de page) | Cohérence vs DeepL pour les relecteurs |
|
||||
| 5 | **Word : les zones de texte ne sont plus traduites 2×** (ensemble `seen_run_elements` partagé — coût API ÷2 sur ces éléments) | Coût + cohérence |
|
||||
| 6 | **Word RTL : l'alignement n'est plus forcé à droite** — centré/justifié préservé, seul « gauche » devient « droite » | Titres RTL corrects |
|
||||
| 7 | **PDF : gras/italique restitués** — sélection `hebo`/`heit`/`hebi` selon les flags du bloc (tout était redessiné en `helv` régulier) | Hiérarchie visuelle conservée |
|
||||
| 8 | **PDF : les cellules de tableaux ne fusionnent plus** — `_is_table_cell` était calculé puis jamais lu ; deux lignes consécutives d'une même colonne étaient jointes en un paragraphe | Structure des tableaux préservée |
|
||||
| 9 | **PDF : un bloc inchangé n'est plus réécrit** — si la traduction est identique (échec fournisseur ou déjà dans la langue cible), on ne rédacte PAS : typo et polices incorporées d'origine conservées (au lieu de tout redessiner en police de substitution). Corrige au passage un doublon de liens hypertexte dans ce scénario | Moins de dégradation, pas de régression |
|
||||
| 10 | **PDF : stats attempted/changed remontent** (+ propagation du chemin fallback pdf2docx) et la route applique le garde-fou `attempted==0`/`changed==0` aussi au PDF (une panne totale du moteur ne produit plus un job « réussi » non traduit) | Détection d'échec |
|
||||
| 11 | **Prompts LLM : noms de langues complets** — nouvelle source unique `core/languages.py` (107 langues) utilisée par openai/deepseek/minimax + service legacy ; « Translate to Tagalog » au lieu de « Translate to tl » | Qualité sur langues rares |
|
||||
|
||||
### Page d'admin (revue + améliorations)
|
||||
|
||||
| # | Amélioration | Détail |
|
||||
|---|---|---|
|
||||
| A1 | **Réglages OCR Mistral dans l'admin** — nouvelle section `mistral` dans `SettingsConfig` (clé/modèle/timeout/activation), fusion env-var comme les autres moteurs, badge « clé dans .env », bouton **Tester** (validation réelle de la clé via `GET /v1/models`) | L'OCR PDF scannés se configure depuis l'UI, plus seulement via .env |
|
||||
| A2 | **Dashboard : statut des 8 moteurs + OCR** — le panneau providers ne montrait que Google ; il liste désormais DeepL, OpenRouter éco/premium, OpenAI, Grok, Google Cloud et « OCR Mistral » avec état dérivé de la configuration (sans appel réseau) et tooltip explicite ; badge « PDF scannés refusés » si OCR non configuré | Vérifié en direct : 8 statuts + erreurs claires « Clé absente (X) » |
|
||||
| A3 | L'OCR du pipeline lit les réglages admin > env (`set_ocr_config`), avec garde anti-footgun (un settings.json fraîchement sauvegardé ne désactive pas l'OCR configuré par env) | Cohérence prod |
|
||||
|
||||
---
|
||||
|
||||
## 2. Benchmark concurrentiel (résumé) — positionnement validé
|
||||
|
||||
Sources fraîches 2026-08-29 (DeepL changelog, pricing Google Cloud, Trustpilot/Reddit, annonces Mistral/Anthropic/OpenAI). Détails et URL dans le rapport d'agent.
|
||||
|
||||
- **Grille tarifaire Free/9/19/49 € : compétitive et bien échelonnée** — 9 € sous DeepL mensuel effectif, 19 € sans concurrent direct à ce prix avec choix de moteur, 49 € ~30-40 % sous DeepL Advanced. Recommandations : mettre les packs de crédits en avant face à l'ancre Google 0,08 $/page ; futur palier équipe ~99-149 € (zone vide avant Smartcat 100 $/mois).
|
||||
- **Top plaintes utilisateurs DeepL/Google** (= nos angles d'attaque) : mise en page cassée sur docs complexes (n°1), PDF scannés refusés/gérés en texte brut, truncature silencieuse sur longs docs, plafonds de taille, aucun outil de relecture avant export, facturation/annulation friction.
|
||||
- **Différenciateurs que nous avons déjà et qu'eux n'ont pas** : multi-moteurs par document, OCR PDF scannés intégré (DeepL web refuse ; Google refuse), 60+ langues à 9 €.
|
||||
- **Marque : risque à surveiller** — « Wordly® » est une marque déposée par Wordly Inc. (wordly.ai, interprétation IA) ; vérification juridique recommandée avant d'investir dans la marque.
|
||||
|
||||
### Gap map vs DeepL (après les correctifs du jour)
|
||||
|
||||
| Fonction | DeepL | Wordly.art | Statut |
|
||||
|---|---|---|---|
|
||||
| Segmentation phrase avec protection du format inline | Oui (tags) | **Oui** (fusion de runs) | ✅ comblé aujourd'hui |
|
||||
| Glossaires | Natifs (API, jusqu'à 5/requête) | Prompt LLM uniquement — **DeepL/Google : ignorés** | ⚠️ reste à faire (API DeepL directe) |
|
||||
| Formalité (formel/informel) | Oui (Pro) | Non | 📋 quick win (prompt) |
|
||||
| Variantes régionales (pt-BR/PT, fr-CA, de-CH) | GA juillet 2026 | Codes acceptés, pas de contrôle fin | 📋 quick win (prompt) |
|
||||
| Mémoire de traduction | Pillier 2026 (Customization Hub) | Cache mémoire par process seulement ; le cache Redis TM codé n'est pas branché | 📋 à activer |
|
||||
| Scores de confiance / QA | Non (prosumer) | L0/L1 codés, désactivés ; L2 réparé aujourd'hui | 📋 à activer |
|
||||
| Sortie bilingue | Non | Non | 📋 quick win (demande forte) |
|
||||
| Édition/relecture avant export | Non | Non | 📋 médium — différenciateur majeur vs DeepL |
|
||||
| XLIFF (post-édition CAT) | API juillet 2026 | Non | 📋 médium (agences) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Reste à faire (priorisé, non fait aujourd'hui)
|
||||
|
||||
**Quick wins (quelques heures chacun)**
|
||||
1. Formalité + variantes régionales : option par job → prompt LLM (et `formality` DeepL quand l'API directe arrivera).
|
||||
2. Sortie bilingue docx (paragraphes source+target en regard) — les segments existent déjà, c'est un rendu.
|
||||
3. Activer le cache Redis TM existant (`services/translation_cache.py`) : cohérence terminologique + coûts ↓.
|
||||
4. Activer les couches qualité L0/L1 (réparées) en log-only, afficher un score de confiance par document.
|
||||
|
||||
**Médium**
|
||||
5. Glossaires sur moteurs non-LLM : passer DeepL en API HTTP directe (`glossary_id`, `formality`, `tag_handling`).
|
||||
6. Contexte document dans les prompts LLM (titre/section courante en préfixe) + batch par liste JSON numérotée (~15× moins de requêtes).
|
||||
7. Rapport QA post-traduction (écarts de nombres, segments non traduits, glossaire violé).
|
||||
8. XLIFF 1.2/2.x export/import (agences).
|
||||
|
||||
**Plus gros / roadmap**
|
||||
9. Éditeur de relecture côte à côte avant téléchargement (le vrai différenciateur pro).
|
||||
10. Espaces de travail équipes (rôles, glossaires/TM partagés) → palier 99-149 €.
|
||||
11. Polices par script cible (CJK/arabe) dans Word/PPTX (`eastAsia`/`cs`) et PDF (matrice de polices) ; annotations FreeText et texte pivoté en PDF.
|
||||
12. Formats IDML/DITA (DeepL API les a ajoutés en juillet 2026 — créneau agences).
|
||||
|
||||
---
|
||||
|
||||
## 4. Vérifications
|
||||
|
||||
- **Suite complète : 1150 passed / 0 failed / 157 skipped** (les 6 tests désélectionnés sont des tests « RealAPI » réseau qui échouent uniquement parce que l'endpoint Google gratuit est momentanément bloqué depuis cette machine — ils passent quand le réseau coopère).
|
||||
- **Réparations d'infrastructure de test au passage** (préexistantes, démasquées par la remise en route de la suite) :
|
||||
- `requirements.txt` : httpx désormais borné `<0.28` (0.28 a supprimé `Client(app=)`, cassait 270 tests via starlette TestClient) → ~270 tests récupérés ;
|
||||
- `tests/test_metrics.py` : fixture Prometheus réécrit (les compteurs vont sur un registre frais, plus de `Duplicated timeseries` quand l'app a déjà été importée) ;
|
||||
- `tests/test_scanned_pdf_ocr.py` : le test e2e passe par `set_ocr_config()` (immunisé contre le rechargement du module config par un autre test).
|
||||
- Suite traducteurs seule : 196/196 (dont 8 nouveaux tests PDF qualité, test graphique PPTX sur vrai fichier, 4 tests renommage Excel, 2 tests fusion runs Word, 2 tests gating mis à jour).
|
||||
- Frontends : `tsc --noEmit` OK sur `frontend/` et `office-translator-landing-page/`.
|
||||
- Dashboard admin testé en direct (8 statuts moteurs/OCR avec erreurs claires).
|
||||
- Note : le Google gratuit (scraping) était bloqué depuis cette machine au moment des tests réseau (`TranslationNotFound`) — le garde-fou `changed==0` l'a correctement détecté ; c'est la fragilité connue qui justifie DeepL/Cloud pour les plans payants.
|
||||
|
||||
*Rapports complets des agents : analyse 35 constats (volet code) et benchmark 9 concurrents avec sources (volet marché) disponibles dans l'historique de session.*
|
||||
146
docs/AUDIT_TRADUCTION_2026-08-29.md
Normal file
146
docs/AUDIT_TRADUCTION_2026-08-29.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# Audit fonctionnel — Pipeline de traduction (wordly.art)
|
||||
|
||||
**Date :** 29 août 2026 · **Périmètre :** backend de traduction (routes → providers → translators docx/xlsx/pptx/pdf) + benchmark concurrentiel web
|
||||
**Méthode :** lecture du code, traduction réelle de fichiers de test via le pipeline de production (provider Google, fr→en), vérification programmatique des sorties, revue de la suite pytest, recherche web concurrents/bonnes pratiques.
|
||||
|
||||
---
|
||||
|
||||
## 0. Corrections appliquées (même jour)
|
||||
|
||||
| # | Correction | Fichiers | Vérification |
|
||||
|---|---|---|---|
|
||||
| 1 | **zh-CN/zh-TW acceptés** : validation insensible à la casse, forme canonique propagée (`validate("zh-cn")` → `"zh-CN"`, alias `chinese`/`tw` conservés) | `middleware/validation.py` · `routes/translate_routes.py` (valeurs de retour utilisées) | 11 tests `tests/test_language_validation.py` ✅ |
|
||||
| 2 | **NameError `current_user`** dans `_run_translation_job` : nouveau helper `_tier_from_plan_str()` (gère `"PlanType.PRO"` et `"pro"`) ; les couches qualité L0/L1/L2 reçoivent enfin le bon tier | `routes/translate_routes.py` | Suite pytest ✅ |
|
||||
| 3 | **Crash libmagic sous Windows** : `import magic` désactivé sur win32, dégradation magie-bytes (PDF/ZIP) ; **la suite pytest ne bloque plus** | `middleware/validation.py` | 207 tests passent (vs hang avant) |
|
||||
| 4 | **OCR Mistral pour PDF scannés** : détection (page quasi sans texte ET couverte ≥50 % par une image), client API `services/mistral_ocr.py` (chunks de 8 pages, retries, erreurs typées), chemin OCR → traduction → PDF propre ; erreur explicite sans `MISTRAL_API_KEY` | `services/mistral_ocr.py` (nouveau) · `translators/pdf_translator.py` · `config.py` · `.env.example` | 12 tests `tests/test_scanned_pdf_ocr.py` + E2E réel (OCR mocké, traduction Google réelle) ✅ |
|
||||
|
||||
**Total : 207 tests passent** (`tests/test_translators` + les 2 nouveaux fichiers). La sortie OCR est un PDF re-mis en page propre : les pages scannées étant des images, la disposition d'origine ne peut pas être réécrite en place.
|
||||
|
||||
---
|
||||
|
||||
## 1. Synthèse exécutive
|
||||
|
||||
Le pipeline fonctionne : un document Word, Excel ou PowerPoint envoyé sur `POST /api/v1/translate` est traduit, mis en forme à l'identique (gras, tableaux, formules, styles) et reste téléchargeable. Les tests réels effectués passent à **14/14**, et les **106 tests unitaires** des traducteurs Office passent en 3,7 s.
|
||||
|
||||
En revanche, l'audit a confirmé **4 bugs fonctionnels** dont un bloquant pour le chinois (rejeté côté API alors que l'UI le propose), **2failles qualité structurelles** (glossaire ignoré par les providers non-LLM, phrase découpée en « runs » traduite morceau par morceau), et un point d'architecture fragile (jobs perdus au redémarrage). Les couches qualité L0/L1/L2 déjà développées sont désactivées par défaut et la L2 est cassée par un bug (`NameError`).
|
||||
|
||||
Le benchmark web montre que le positionnement est bon (LLM + préservation du format + prix), mais que les différenciateurs attendus en 2026 sont : **PDF scannés via OCR** (refusés par DeepL/Azure), **sortie bilingue**, **mémoire de traduction persistante** (le code existe mais n'est pas branché) et **scores de qualité visibles**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Ce qui fonctionne (vérifié en conditions réelles)
|
||||
|
||||
Script de test : `temp/audit/audit_functional.py` (fichiers générés puis traduits via `WordTranslator`/`ExcelTranslator`/`PowerPointTranslator` + provider Google de production).
|
||||
|
||||
| Vérification | Résultat |
|
||||
|---|---|
|
||||
| DOCX — titre, paragraphes traduits | ✅ « Rapport financier annuel » → « Annual financial report » |
|
||||
| DOCX — phrase coupée en 3 runs (gras au milieu) | ✅ cohérente dans ce cas, gras conservé sur le bon run |
|
||||
| DOCX — nombres/monnaies préservés | ✅ « 4 500 000 euros en 2025 » intact |
|
||||
| DOCX — tableau traduit | ✅ |
|
||||
| XLSX — cellules, en-têtes, mois | ✅ « Janvier » → « January » |
|
||||
| XLSX — **formules préservées** | ✅ `=SUM(B3:B4)` inchangée |
|
||||
| PPTX — titre + puces | ✅ « Stratégie commerciale 2026 » → « Commercial strategy 2026 » |
|
||||
| Statistiques anti-échec (`attempted/changed`) | ✅ remontées au job |
|
||||
|
||||
Points d'architecture solides constatés : validation magic bytes + zip-bomb, protection SSRF sur `file_url`, quotas mensuels atomiques, progression temps réel (Redis, TTL 2h), webhooks signés par jeton par-job, watermark gratuit, RTL (ar/he/fa), notes de bas de page, SmartArt, graphiques (ré-injection ZIP), noms de feuilles Excel reécrits avec mise à jour des références.
|
||||
|
||||
---
|
||||
|
||||
## 3. Bugs confirmés
|
||||
|
||||
### P1 — Le chinois est rejeté comme langue cible (bloquant, visible utilisateur) — ✅ CORRIGÉ
|
||||
`middleware/validation.py:488` met le code en minuscules (`zh-cn`) mais `SUPPORTED_LANGUAGES` contient `"zh-CN"`/`"zh-TW"` en casse mixte (`:349-350`) → `LanguageValidator.validate("zh-CN")` lève une erreur → **400 systématique**. L'UI propose pourtant `zh-CN`/`zh-TW` (`frontend/src/app/dashboard/translate/useTranslationConfig.ts:31-32`). De plus la valeur normalisée retournée par `validate()` est ignorée dans `routes/translate_routes.py:796,806` (alias `chinese`→`zh-CN` jamais propagés).
|
||||
**Fix appliqué (voir §0).**
|
||||
|
||||
### P1 — `NameError: current_user` : la couche qualité L2 ne fonctionne jamais — ✅ CORRIGÉ
|
||||
Dans `_run_translation_job` (`routes/translate_routes.py:1574-1575, 1592-1593, 1611`), `current_user` n'existe pas (signature `:1105-1120`). L'`NameError` est avalé par les `except` : quand `QUALITY_L2_ENABLED=true`, le juge qualité Pro **échoue silencieusement à chaque job** ; les métriques `record_translation_retry` L0/L1 sont mortes aussi.
|
||||
**Fix appliqué (voir §0).**
|
||||
|
||||
### P2 — Glossaire silencieusement ignoré selon le provider
|
||||
Le glossaire n'est injecté que via `metadata["custom_prompt"]`, lu uniquement par openai/deepseek/minimax. Avec **google, deepl ou google_cloud, un utilisateur Pro qui paie pour le glossaire n'a aucune application des termes** — sans avertissement. `build_full_prompt` avec glossaire seul produit en outre un prompt système **sans instruction de traduction** (le prompt par défaut est remplacé : `services/providers/openai_provider.py:124-125`), ce qui fragilise la qualité même en LLM.
|
||||
**Fix :** fusionner glossaire + prompt par défaut au lieu de remplacer ; refuser ou avertir quand provider non-LLM + glossary_id ; à terme, utiliser les glossaires natifs DeepL.
|
||||
|
||||
### P2 — URL de téléchargement cassée sur `/translate-batch` (legacy)
|
||||
`routes/legacy_routes.py:304` renvoie `/api/v1/download/{output_filename}` alors que la route exige un id `tr_*` (`translate_routes.py:26`) → 400 INVALID_JOB_ID pour tout client de l'endpoint batch.
|
||||
|
||||
### P2 — Crash natif Windows dans le chemin « format loss » PDF + violation de couche — ✅ CORRIGÉ
|
||||
`translators/pdf_translator.py:69` (`_record_format_loss_metric`) importe `middleware.metrics` au milieu de la traduction PDF. L'import déclenche `middleware/__init__.py` → `middleware/validation.py:7` → `import magic` → **`python-magic`/libmagic plante en « access violation » sous Windows** (crash natif non rattrapable par le `try/except` du wrapper). Conséquences : (a) la suite pytest **bloque indéfiniment** sur `tests/test_translators/test_b3_5_pdf_smart_fit.py::test_tier_3_4_limit_to_two_shrink_steps` ; (b) en production Windows, tout PDF avec perte de format (overflow → placeholder `[translation overflow]`) risque le même sort. C'est aussi une violation de couche : `translators/` ne devrait pas dépendre de `middleware/`.
|
||||
**Fix appliqué (voir §0) : `import magic` désactivé sous Windows + dégradation magie-bytes ; l'import paresseux `middleware.metrics` dans pdf_translator est désormais inoffensif. La répartition translators/→core/ reste à faire (P3).**
|
||||
|
||||
### P3 — Divers
|
||||
- Échec de traduction = **texte source renvoyé silencieusement** (chaque provider). Le garde-fou `changed == 0` (`translate_routes.py:1480`) ne bloque que l'échec total : un document traduit à 10 % passe (warning seul, `:1490`). L'utilisateur paie pour un document partiel.
|
||||
- Modèle OpenRouter réécrit silencieusement (`:1182-1183`) : la config admin `deepseek/deepseek-v3.2` est forcée vers `google/gemini-3.5-flash`.
|
||||
- `max_tokens=500` dans le provider legacy `OpenAITranslationProvider` (`services/translation_service.py:1071`) peut tronquer un paragraphe long.
|
||||
- `OllamaTranslationProvider.list_models` défini 2 fois (l'instance method est masquée par le staticmethod, `translation_service.py:649` vs `:693`).
|
||||
- Provider « zai » pointe par défaut vers xAI/Grok (`translate_routes.py:1319-1329`) — dénomination trompeuse.
|
||||
- Code mort : `WebLLMTranslationProvider`, `_GoogleCloudWithFallback`, `services/translation_cache.py` (cache Redis TM — voir §5), chaîne de fallback `services/providers/fallback.py` non branchée sur la route principale.
|
||||
|
||||
---
|
||||
|
||||
## 4. Risques qualité structurels (traduction)
|
||||
|
||||
1. **Segmentation par run (docx/pptx).** Chaque `<w:r>` est traduit indépendamment (`translators/word_translator.py:1147-1220`). Dès qu'une phrase est découpée par du gras, une couleur, une correction orthographique (rsid) ou un saut de champ, les morceaux sont traduits séparément : l'ordre des mots cible peut rendre la phrase incohérente (cas fr→en avec adjectifs). Le test est passé parce que la découpe tombait bien ; c'est fragile par construction. C'est **l'écart principal face à DeepL** qui traduit au niveau document/phrase.
|
||||
**Piste :** traduire au niveau paragraphe avec masquage de placeholders (`<0>`, `<1>`…) pour protéger les frontières de mise en forme, puis redistribuer — technique standard validée (cf. recherche web, « tag protection »).
|
||||
2. **Moteur par défaut = Google Translate gratuit non officiel** (deep_translator, scraping). Limité à 5 000 caractères/requête, non contractuel, risque de blocage/ToS, qualité plafonnée. Acceptable en free tier, risqué comme moteur par défaut des offres payantes.
|
||||
3. **Aucune détection de langue réelle** : `source_lang=auto` délégué au provider ; les prompts LLM disent « détecte la langue » (correct avec les LLM récents, approximatif avec Google).
|
||||
4. **Cohérence document** : chaque segment est traduit sans contexte document (pas de contexte titre/section). Les LLM supporteraient un batch contextuel.
|
||||
|
||||
---
|
||||
|
||||
## 5. Architecture — points fragiles
|
||||
|
||||
- **Jobs non persistés** : `asyncio.create_task` (`translate_routes.py:959`) — un redémarrage du process perd tous les jobs en cours (le statut Redis survit mais rien ne reprend). Pour un SaaS payant, prévoir une queue (Celery/Arq/RQ ou table jobs + worker) avec reprise.
|
||||
- **Cache mémoire seulement** (LRU 5 000, perdu au redémarrage). Le cache Redis avec clés sha256 + TTL 24h **existe** (`services/translation_cache.py`) mais n'est branché nulle part en production — c'est une mémoire de traduction (TM) gratuite à activer.
|
||||
|
||||
---
|
||||
|
||||
## 6. Benchmark & recommandations produit (recherche web, août 2026)
|
||||
|
||||
### Concurrents et standards du marché
|
||||
| Acteur | Positionnement | Enseignement |
|
||||
|---|---|---|
|
||||
| DeepL | Référence qualité ; documents pdf/docx/pptx/xlsx/idml ; facturation API **min 50 000 chars/document** ; glossaires natives, formality, variantes pt-BR/pt-PT, fr-CA | Le minimum 50K chars est un pain point : un pipeline LLM peut être moins cher. Refuse les PDF scannés. |
|
||||
| Google Cloud Translation Advanced | **0,08 $/page**, PDF natifs et scannés (OCR intégré, 1 000 pages) | Alternative crédible au scraping gratuit pour le paid tier |
|
||||
| Azure Translator | 15 $/M chars, batch documents, refuse PDF scannés | |
|
||||
| Immersive Translate / BabelDOC | 20M+ utilisateurs, PDF bilingue, OCR | La sortie bilingue est très demandée |
|
||||
| PDFMathTranslate (35K★) / BabelDOC (open source) | Préservation layout PDF, formules, multi-moteurs | À étudier pour améliorer `pdf_translator` |
|
||||
| Pairaphrase/Redokun/Smartcat | TM + glossaires + relecture humaine, 285 $+/mois | Le segment pro attend TM/glossaires |
|
||||
| DocTranslator, oTranslator | 14,99 $/mois ou ~0,006-0,06 $/page | Prix marché prosumer : 8-15 $/mois |
|
||||
|
||||
État de l'art modèles (WMT25) : **GPT-4.1 et Gemini-2.5-Pro mènent** la traduction IA ; DeepL next-gen « stable mais mid-tier ». Qualité : MetricX-24/25, xCOMET (QE), RUBRIC-MQM (LLM-judge) — aucun concurrent grand public n'affiche de scores de confiance par segment : **différenciateur ouvert**.
|
||||
|
||||
### Recommandations priorisées
|
||||
1. **P0 — Corriger les 2 bugs P1** (chinois, L2) : quelques lignes, impact utilisateur direct.
|
||||
2. **P0 — Activer le cache Redis existant comme TM** (déjà codé) : baisse de coût + cohérence terminologique entre documents.
|
||||
3. **P1 — Paragraph-level translation avec placeholders** pour docx/pptx : plus grand gain qualité perçue.
|
||||
4. **P1 — PDF scannés via OCR** : ✅ **implémenté (Mistral OCR, voir §0)**. Reste : brancher Mistral OCR (~1 $/1 000 pages) — il suffit de définir `MISTRAL_API_KEY`.
|
||||
5. **P1 — Glossaire :** fusionner avec le prompt par défaut ; supporter les glossaires natifs DeepL ; avertir si provider incompatible.
|
||||
6. **P2 — Sortie bilingue** (docx avec colonnes/annotations, PDF duo) : forte demande, faible coût.
|
||||
7. **P2 — Activer L0/L1 en production** après fix (coût ~0,0003 $/job) et exposer un « score de confiance » par document.
|
||||
8. **P2 — Formalité & variantes régionales** (fr-CA, pt-BR/PT) via instructions prompt (LLM) — table stakes 2026.
|
||||
9. **P3 — Queue persistante** pour les jobs ; remplacer le moteur par défaut du paid tier par DeepL API ou Google Cloud Advanced (0,08 $/page) plutôt que le scraping gratuit.
|
||||
10. **P3 — Nettoyage :** supprimer le code mort listé §3, documenter l'alias « zai »→xAI.
|
||||
|
||||
---
|
||||
|
||||
## 7. Annexes
|
||||
|
||||
### A. Résultats des tests réels
|
||||
Voir §2 — script `temp/audit/audit_functional.py`, sorties dans `temp/audit/`.
|
||||
|
||||
### B. Suite pytest
|
||||
- `tests/test_translators/test_word_translator.py + test_excel_translator.py + test_pptx_translator.py` : **106 passed en 3,68 s** ✅
|
||||
- `tests/test_translators/test_b3_5_pdf_smart_fit.py` : **bloque indéfiniment** sur `test_tier_3_4_limit_to_two_shrink_steps` (crash natif libmagic, cf. bug P2 ci-dessus) — reproduit isolément ; dumps faulthandler « Windows fatal exception: access violation » dans `magic/compat.py:189`.
|
||||
- Recommandations : corriger l'import `magic`, ajouter `pytest-timeout`, et noter que `pytest.ini` active la couverture par défaut (`addopts --cov=...`) ce qui rend les runs complets très lents.
|
||||
|
||||
### C. Sources principales (benchmark)
|
||||
- DeepL document translation & API : deepl.com/en/features/document-translation · developers.deepl.com (document upload, glossaries, formality, variants)
|
||||
- Google Cloud Translation (documents, 0,08 $/page, OCR) : cloud.google.com/translate/docs/advanced/translate-documents
|
||||
- Azure Translator document translation : learn.microsoft.com/azure/ai-services/translator/document-translation/
|
||||
- WMT25 (GPT-4.1/Gemini 2.5 Pro en tête) : slator.com/wmt25-preliminary-results-gemini-2-5-pro-gpt-4-1-lead-ai-translation/
|
||||
- MetricX-24/25 (QE) : github.com/google-research/metricx · RUBRIC-MQM : aclanthology.org/2025.acl-industry.12/
|
||||
- PDFMathTranslate : github.com/PDFMathTranslate/PDFMathTranslate · BabelDOC : github.com/funstory-ai/BabelDOC
|
||||
- Mistral OCR (1 $/1 000 pages) : mistral.ai/news/mistral-ocr/ · PaddleOCR : github.com/PaddlePaddle/PaddleOCR
|
||||
- Immersive Translate (bilingue, PDF Pro) : immersivetranslate.com/pricing/
|
||||
- Pairaphrase (pricing entreprise) : pairaphrase.com · Redokun : redokun.com/pricing
|
||||
37
docs/SESSION_2026-08-29_FEATURES.md
Normal file
37
docs/SESSION_2026-08-29_FEATURES.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Session du 2026-08-29 (3ᵉ passe) — « Fais tout » : quick wins + items médiums livrés
|
||||
|
||||
> Décision produit de la session : **aucune intégration DeepL** (refus explicite). Formalité et variantes régionales passent exclusivement par les prompts LLM.
|
||||
|
||||
## Livré dans cette session (tout testé — 23 nouveaux tests + 1 mis à jour)
|
||||
|
||||
| # | Fonctionnalité | Implémentation | Tests |
|
||||
|---|---|---|---|
|
||||
| 1 | **Formalité (formel/informel)** | Nouveau paramètre `formality` sur `POST /api/v1/translate` → directive `TONE:` ajoutée au prompt de tous les moteurs LLM (openai/deepseek/minimax/legacy) via `build_full_prompt` | 3 |
|
||||
| 2 | **Variantes régionales automatiques** | `target_lang` régional (pt-BR, fr-CA, zh-CN…) → directive `REGIONAL VARIANT: write in <nom complet>` dans le prompt | 2 |
|
||||
| 3 | **Bug corrigé : prompt personnalisé qui remplaçait les instructions** | Les 3 providers LLM incluent TOUJOURS le prompt de base ; glossaire/ton/contexte s'y ajoutent (`ADDITIONAL CONTEXT AND INSTRUCTIONS`) — avant, un glossaire seul produisait un prompt sans instruction de traduction | 1 (+ openai) |
|
||||
| 4 | **Mémoire de traduction (TM) activée** | `services/translation_tm.py` branche le cache Redis existant (`translation_cache.py`, TTL 24 h, fallback LRU sans Redis) dans Word/Excel/PPTX. **Périmètre par utilisateur** (jamais de partage inter-clients) + hash du contexte (glossaire/ton) pour invalidation. Les traductions identiques à la source ne sont jamais stockées (anti-poison). Repli silencieux si le provider échoue | 5 |
|
||||
| 5 | **Sortie bilingue (docx)** | Paramètre `output_mode=bilingual` : chaque paragraphe traduit est précédé de sa source (gris, italique, 9 pt) — `translators/bilingual.py`. Repli propre si structure divergente | 2 |
|
||||
| 6 | **Rapport QA + score de confiance** | `services/quality/qa_report.py` : fidélité des nombres (séparateurs décimaux normalisés), ratio non-traduit, **score 0-100** exposé dans `GET /api/v1/translations/{id}` (champ `quality`) et le statut de complétion. Jamais bloquant | 5 |
|
||||
| 7 | **Batch JSON pour LLM** | `openai_provider.translate_batch` envoie un chunk entier en **1 requête** (liste JSON numérotée) au lieu de 15 requêtes isolées — ~15× moins d'appels, cohérence contextuelle entre segments voisins. Repli automatique par item si la réponse est non conforme | 3 |
|
||||
| 8 | **Qualité L0 par défaut ON** | `QUALITY_L0_ENABLED=true` par défaut (observabilité pure, ne bloque jamais) | — |
|
||||
| 9 | **Polices CJK/arabe** | Word : hints `w:eastAsia` (SimSun/Yu Mincho/Batang) et `w:cs` sur les runs ; PPTX : `<a:ea>` typeface ; PDF : chemins Noto CJK (Linux) + simsun/msgothic (Windows) ajoutés | 2 |
|
||||
|
||||
**Exemples d'appel :**
|
||||
```bash
|
||||
# Ton formel + portugais brésilien
|
||||
curl -F file=@doc.docx -F target_lang=pt-BR -F formality=formal .../api/v1/translate
|
||||
|
||||
# Sortie bilingue
|
||||
curl -F file=@doc.docx -F target_lang=en -F output_mode=bilingual .../api/v1/translate
|
||||
```
|
||||
|
||||
## Vérifications
|
||||
- Suite complète : voir ligne finale ci-dessous (6 tests réseau Google désélectionnés — endpoint gratuit momentanément bloqué depuis cette machine, sans lien avec le code).
|
||||
- `tsc --noEmit` OK sur les deux frontends.
|
||||
|
||||
## Non livré (justifié)
|
||||
- **Éditeur de relecture côte à côte** : nécessite la persistance des segments par job (schéma BDD + UI d'édition) — chantier à part entière, il ouvre la voie au XLIFF.
|
||||
- **Espaces de travail équipes (rôles, glossaires partagés)** : migrations + facturation multi-sièges.
|
||||
- **XLIFF export/import** : dépend de la persistance des segments (même chantier que l'éditeur).
|
||||
- **IDML/DITA** : nouveaux parseurs de formats — à chiffrer séparément.
|
||||
- **DeepL natif (glossaires/formalité API)** : refusé par le propriétaire — la formalité LLM couvre le besoin sans DeepL.
|
||||
103
docs/marketing/PLAN-ALIGNEMENT-CODE.md
Normal file
103
docs/marketing/PLAN-ALIGNEMENT-CODE.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Plan d'alignement Marketing ↔ Code — wordly.art
|
||||
|
||||
> Analyse effectuée le 2026-08-29, croisée ligne à ligne avec le code backend, la landing et les docs de lancement.
|
||||
> Principe : **le lancement (Product Hunt / Show HN) attire des relecteurs techniques** — chaque promesse non tenue y sera testée publiquement. Ce plan sépare ce qu'il faut corriger **dans les textes** (rapide) de ce qu'il faut corriger **dans le code** (pour tenir les promesses).
|
||||
>
|
||||
> **Statut au 2026-08-29 : vagues A et B appliquées et testées (442 tests backend passent, `tsc --noEmit` landing OK). Restent les items P2 de la vague C.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Ce qui est déjà conforme ✅
|
||||
|
||||
| Affirmation marketing | Vérification code | Statut |
|
||||
|---|---|---|
|
||||
| Prix 0/9/19/49 €, annuel −20 % (7,20/15,20/39,20) | `models/subscription.py:39-155` (86,40/182,40/470,40 €/an) | ✅ exact |
|
||||
| Quotas 2/50/200/1 000 docs, pages 10/50/200/500, fichiers 5/10/25/50 Mo | idem | ✅ exact |
|
||||
| API Business 10 000 appels/mois, 5 sièges | `api_calls_per_month: 10_000`, `team_seats: 5` | ✅ exact |
|
||||
| Crédits 50/5 € · 100/9 € · 250/20 € · 500/35 € · 1 000/60 € (0,06–0,10 €) | `CREDIT_PACKAGES` (`models/subscription.py:316`) | ✅ exact |
|
||||
| Landing : sections pricing/FAQ/preuve/waitlist + Vercel Analytics + waitlist endpoint + compteur | `app/page.tsx`, `components/*`, `routes/waitlist_routes.py` | ✅ en place |
|
||||
| Événements analytics (`translation_started`, `pricing_cta_click`, `waitlist_joined`, `waitlist_error`) | présents dans les 3 composants | ✅ branchés |
|
||||
| Footer sans « SOC 2 » | `app/page.tsx` — mention absente | ✅ corrigé |
|
||||
| Blog Docker : « CORS `*` refuse de démarrer en prod » | `main.py:379-386` (`sys.exit`) | ✅ vrai |
|
||||
| « DeepL moteur des plans payants » | Starter+ : `providers: ["google","deepl",…]` | ✅ exact |
|
||||
| Blog 1 : « à partir de 0,06 €/page » | cohérent avec les crédits (1 page = 1 crédit de base) | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## 2. Écarts détectés — promesses non tenues par le code
|
||||
|
||||
### E1. 🔴 « PDF is on the roadmap » (Show HN + Product Hunt) — FAUX, et ça cache une feature
|
||||
- Show HN : *« (PDF is the obvious next one.) »* · Product Hunt : *« PDF is on the roadmap »*.
|
||||
- **Réel : le PDF est déjà supporté** (`.pdf` dans `SUPPORTED_EXTENSIONS`, mode `layout`/`text_only`, pdf2docx fallback) **et depuis le 2026-08-29 les PDF scannés passent par OCR Mistral** (`services/mistral_ocr.py`) — ce que DeepL et Azure refusent.
|
||||
- **C'est l'inverse : c'est un argument de vente majeur à ajouter.** Telle quelle, la phrase invite les relecteurs HN à trouver une fonctionnalité qui existe déjà dans le dashboard.
|
||||
|
||||
### E2. 🔴 « Files auto-deleted after 60 min » (landing, FAQ, Show HN, PH, Reddit, X, blogs 1/3) — inexact
|
||||
- Réel (`config.py`) : uploads **30 min**, résultats **120 min**, TTL générique 60.
|
||||
- Deux options : corriger le texte (« uploads deleted after 30 min, results within 2 h ») **ou** passer `OUTPUT_FILE_TTL_MINUTES` à 60. Recommandation : garder 120 min (l'utilisateur doit pouvoir re-télécharger) et corriger les textes.
|
||||
|
||||
### E3. 🔴 « Encrypted at rest » (FAQ §sécurité, X-thread, blog 3 §données) — non implémenté
|
||||
- Aucun chiffrement des fichiers au repos dans le code (uploads/outputs en clair). TLS : dépend du reverse-proxy (vrai en prod nginx+HSTS).
|
||||
- **À retirer des textes** tant que non implémenté (ou implémenter, cf. §3-C5).
|
||||
|
||||
### E4. 🟠 « 60+ langues » (landing stats, PH, X-thread) — l'app en expose 35
|
||||
- Réel : `/api/v1/languages` = **35 langues** ; le sélecteur UI = 35. (Le validateur accepte ~100 codes, mais invisibles pour l'utilisateur.)
|
||||
- Options : (a) corriger le chiffre en « 35 », (b) **exposer les ~60-100 langues déjà validées** côté API/UI — recommandé, c'est un travail faible et les LLM couvrent ces langues.
|
||||
|
||||
### E5. 🟠 « 7 moteurs » avec Minimax et DeepSeek cités (FAQ, Show HN) — incohérent avec les plans
|
||||
- Réel (`models/subscription.py`) : Business = `google, google_cloud, deepl, openrouter, openrouter_premium, openai, zai` — **ni Minimax ni DeepSeek direct** (DeepSeek n'est que le modèle IA « essentiel » interne).
|
||||
- Harmoniser : retirer Minimax/DeepSeek des listes, **ou** les ajouter au plan Business (l'endpoint `/providers/available` les expose déjà à l'UI !).
|
||||
|
||||
### E6. 🟡 « 100 % formatting preserved » (landing, comparatif, x-thread) — survendu
|
||||
- Cas limites réels : PDF scannés (sortie re-mise en page, pas d'« in place »), placeholders `[translation overflow]` sur débordements, texte dans images non traduit sans l'option vision.
|
||||
- Recommandation : garder la promesse forte mais crédible (« formatting preserved — formulas, merges, styles ») + publier une page « limites connues ». Sur HN, un contre-exemple suffira à casser le « 100 % ».
|
||||
|
||||
### E7. 🟠 Blog Docker (auto-hébergement) : 4 erreurs factuelles
|
||||
1. `GOOGLE_TRANSLATE_API_KEY=...` : **cette variable n'existe pas** — le moteur Google gratuit n'a pas de clé ; le moteur payant = `GOOGLE_CLOUD_API_KEY`. Remplacer par un exempilaire réel du `.env.example`.
|
||||
2. `pg_dump -U wordly wordly` : les défauts du compose sont **`translate` / `translate_db`**.
|
||||
3. « L'API sur :8000 » : le compose publie **8001:8000** → c'est `:8001` côté hôte.
|
||||
4. `git clone https://gitea.parsanet.org/...` : repo privé → rendre public ou générer (`github.com/<org>/office-translator`).
|
||||
|
||||
### E8. 🟠 Le backend n'applique PAS la grille « moteurs par plan » (fuite de revenus)
|
||||
- Dans `translate_document_v1`, seul `google_cloud` est rétrogradé pour les non-Pro. **Un Free peut envoyer `provider=openai` ou `openrouter_premium`** et consommer les moteurs vendus 49 €/mois. La grille existe pourtant dans `PLANS[plan]["providers"]`.
|
||||
- Idem : **`translate_images` (vision) n'est pas gated**, alors que la FAQ dit « Pro and Business plans ».
|
||||
|
||||
---
|
||||
|
||||
## 3. Plan d'action
|
||||
|
||||
### Vague A — Avant lancement (textes, ~½ journée) — P0 ✅ APPLIQUÉE
|
||||
| # | Action | Fichiers | Statut |
|
||||
|---|---|---|---|
|
||||
| A1 | Remplacer « PDF on the roadmap » par la feature : « PDF supported — incl. scanned PDFs via OCR » | `launch/show-hn.md`, `launch/product-hunt.md` | ✅ |
|
||||
| A2 | Corriger la rétention : « uploads deleted after 30 min, results auto-deleted within 2 h » | landing `social-proof-section.tsx`, `faq-section.tsx`, `show-hn.md`, `product-hunt.md`, `reddit-posts.md`, `x-thread.md`, `blog/1`, `blog/3`, `MARKETING_PLAN.md` | ✅ |
|
||||
| A3 | Supprimer « encrypted at rest » / « at rest » | `faq-section.tsx`, `x-thread.md`, `blog/3` | ✅ |
|
||||
| A4 | Blog Docker : clé `GOOGLE_CLOUD_API_KEY`, `pg_dump` translate/translate_db, port hôte 8001, URL de dépôt générique, + mention MISTRAL_API_KEY | `blog/5-auto-heberger-traduction-docker.md` | ✅ |
|
||||
| A5 | Ajouter PDF (+ OCR scannés) aux formats annoncés : hero, badges `.pdf`, drag & drop + `accept` PDF, FAQ formats | `hero-section.tsx`, `translation-card.tsx`, `faq-section.tsx` | ✅ |
|
||||
|
||||
### Vague B — Semaine 1 (code, ~2-3 j) — P1 ✅ APPLIQUÉE
|
||||
| # | Action | Détail | Statut |
|
||||
|---|---|---|---|
|
||||
| B1 | **Gating moteurs par plan** : `translate_document_v1` rejette (403 `PRO_FEATURE_REQUIRED`) tout `provider` hors `PLANS[tier]["providers"]` ; helpers testés (`tests/test_plan_gating.py`) ; `/providers/available` filtré par plan (l'UI ne propose plus un moteur refusé) | ferme la fuite de revenus E8 | ✅ |
|
||||
| B2 | Gater `translate_images` aux plans Pro+ | la FAQ redevient exacte | ✅ |
|
||||
| B3 | Langues : `/api/v1/languages` expose désormais **107 langues nommées** (35 populaires d'abord, puis ordre alphabétique ; `LANGUAGE_NAMES` complété) → la promesse « 60+ » est tenue | E4 résolu par le code | ✅ |
|
||||
| B4 | Moteurs harmonisés : Minimax/DeepSeek retirés des listes marketing (DeepSeek présenté « via OpenRouter ») ; cohérent avec le filtrage du catalogue | résout E5 | ✅ |
|
||||
|
||||
### Vague C — Avant le scale — P2
|
||||
| # | Action | Détail |
|
||||
|---|---|---|
|
||||
| C1 | Remplacer « 100 % » par une promesse démontrable + page « limites connues » (PDF scannés re-mis en page, overflow, images) | E6 |
|
||||
| C2 | Si l'argument « chiffré au repos » est voulu : chiffrer `uploads/`+`outputs/` (clé env `FILES_ENCRYPTION_KEY`, chiffrement AES au write/read) alors seulement réintroduire la mention | E3 |
|
||||
| C3 | Mettre à jour `MARKETING_PLAN.md` : ajouter PDF + OCR scannés aux différenciateurs (vs DeepL/Azure qui refusent les scannés), mentionner webhooks & API keys | valorise l'existant |
|
||||
| C4 | Décider TTL outputs : si la promesse « 60 min » est gardée, passer `OUTPUT_FILE_TTL_MINUTES=60` au lieu de changer 9 textes | alternative à A2 |
|
||||
| C5 | NPS « enquête post-traduction » (source KPI dans MARKETING_PLAN §6) : non implémenté — retirer la ligne ou créer le mini-formulaire post-download | KPIs honnêtes |
|
||||
|
||||
### Decision needed (propriétaire) — ✅ TRANCHÉES LE 2026-08-29
|
||||
- **Rétention** : textes corrigés (A2) — les TTL code restent 30 min / 120 min.
|
||||
- **Langues** : 107 exposées via `/languages` (B3) — la promesse « 60+ » est tenue.
|
||||
- **Minimax/DeepSeek** : retirés du discours commercial (DeepSeek mentionné « via OpenRouter ») ; plans inchangés.
|
||||
|
||||
---
|
||||
|
||||
## 4. Résumé
|
||||
|
||||
Le socle commercial du document marketing est **fiable** (prix, quotas, crédits, API, landing, waitlist, analytics : tout correspond au code). Les écarts se concentrent sur **6 chiffres/affirmations répétées** (60 min, 60+ langues, 7 moteurs, 100 %, chiffrement au repos, « PDF à venir ») et **2 trous d'application côté code** (gating moteurs et vision par plan). Les vagues A (textes) et B (code) suffisent pour un lancement PH/HN sans Vulnerabilité factuelle.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Comment traduire un fichier Excel sans perdre la mise en forme
|
||||
|
||||
> Article pilier SEO — objectif : 1 500+ mots au moment de la publication (ce brouillon est complet, à enrichir avec 2 captures d'écran réelles et un exemple chiffré avant publication).
|
||||
|
||||
## Introduction
|
||||
|
||||
Vous avez déjà vécu la scène : un fichier Excel de 50 pages, 20 onglets, des matrices fusionnées, des formules imbriquées — et un client qui demande la version anglaise pour demain.
|
||||
|
||||
Les solutions « classiques » ne font pas ce que vous pensez. Nous les avons toutes testées. Voici ce qu'elles font réellement à votre fichier, et comment le traduire sans rien casser.
|
||||
|
||||
## Ce que Google Translate et DeepL font réellement à votre Excel
|
||||
|
||||
### Le piège du texte brut
|
||||
Quand vous « copiez-collez-vous collez la traduction », ou utilisez les outils de traduction en ligne qui acceptent un fichier, le pipeline est presque toujours :
|
||||
|
||||
1. **Extraction** : le fichier est aplati en texte brut (ou en CSV)
|
||||
2. **Traduction** : le texte est traduit
|
||||
3. **Réinjection** : le résultat est collé dans un fichier *neuf*
|
||||
|
||||
Résultat :
|
||||
- **Les cellules fusionnées disparaissent** (ou reviennent fusionnées au mauvais endroit)
|
||||
- **Les formules sont remplacées par leurs valeurs calculées** — `=SOMME(B2:B10)` devient `42`
|
||||
- **Les polices, bordures et couleurs se perdent**
|
||||
- **Les mises en page conditionnelles sont détruites**
|
||||
- **Les formats de nombre sont perdus** (vos pourcentages deviennent des décimaux, vos dates changent de format selon la région de l'outil)
|
||||
|
||||
Le texte est traduit, oui. Mais le fichier n'est plus *votre* fichier.
|
||||
|
||||
### Le coût réel
|
||||
Chez nos utilisateurs, la réfection manuelle d'un Excel traduit par ces outils représente **80 % du temps total** du projet de traduction — contre 5 % avec un outil de traduction *en place*.
|
||||
|
||||
## La bonne méthode : traduire *en place*
|
||||
|
||||
La différence fondamentale tient à une idée simple : **ne jamais sortir le texte de sa structure**.
|
||||
|
||||
Un fichier .xlsx est, sous le capot, une archive ZIP contenant du XML. Les cellules, les fusions, les formules et les styles sont des éléments XML distincts. Un bon moteur de traduction de documents :
|
||||
|
||||
1. **Parse** la structure native (feuilles, cellules, fusions, formules, styles)
|
||||
2. **Extrait uniquement** le contenu traduisable (chaînes de caractères), en mémorisant leur position et leur style
|
||||
3. **Traduit** ces chaînes (le moteur de votre choix)
|
||||
4. **Réinjecte** les chaînes traduites dans *la même* structure XML
|
||||
5. **Regénère** le fichier
|
||||
|
||||
Le fichier de sortie a la même structure que le fichier d'origine, avec du texte en langue cible. C'est exactement ce que fait Office Translator pour .xlsx, .docx et .pptx.
|
||||
|
||||
## Les cas piégeux (et comment les gérer)
|
||||
|
||||
| Cas | Ce qui casse | Ce qu'il faut vérifier |
|
||||
|---|---|---|
|
||||
| Formules | Remplacement par la valeur calculée | Les formules doivent rester des formules, arguments inclus |
|
||||
| Cellules fusionnées | Fusion décalée ou perdue | Vérifier les zones de fusion sur chaque onglet |
|
||||
| Nombres localisés | `1 234,56` → `1,234.56` (ou l'inverse) | Les formats de nombre doivent suivre la langue cible |
|
||||
| Dates | `31/12/2026` → `2026-12-31` (ou casse totale) | Format de date aligné sur la localisation |
|
||||
| Textes dans les images | Ignorés par la plupart des outils | Un modèle vision est nécessaire (disponible sur les plans payants) |
|
||||
| Commentaires et annotations | Perdus à l'extraction | Vérifier la présence des commentaires en sortie |
|
||||
|
||||
**Règle d'or** : après chaque traduction, faites un contrôle *structurel* (même nombre d'onglets, mêmes fusions, mêmes formules) avant le contrôle sémantique. Un outil qui traduit en place vous fait gagner ce contrôle : vous vérifiez le texte, pas la mise en page.
|
||||
|
||||
## Comparatif des approches (2026)
|
||||
|
||||
| Approche | Format préservé | Formules | Coût | Délai |
|
||||
|---|---|---|---|---|
|
||||
| Traducteur humain | Oui (réintégration manuelle) | Oui | 50–100× plus cher | Jours–semaines |
|
||||
| Google Translate (copier-coller) | Non | Non | Gratuit | Minutes |
|
||||
| Outil one-shot (DocTranslator et sim.) | Partiel | Non | Payant, variable | Minutes |
|
||||
| **Traduction en place (Office Translator)** | **Oui, 100 %** | **Oui** | **À partir de 0,06 €/page** | **Minutes** |
|
||||
|
||||
## Comment choisir un outil de traduction de documents
|
||||
|
||||
Avant de payer, testez sur *votre* fichier le plus sale (pas un exemple propre) et vérifiez :
|
||||
|
||||
1. **Structure** : mêmes onglets, mêmes fusions, mêmes colonnes
|
||||
2. **Formules** : ouvrez la barre de formule, pas seulement la valeur affichée
|
||||
3. **Styles** : polices, bordures, couleurs de cellules
|
||||
4. **Formats** : nombres, dates, pourcentages, devise
|
||||
5. **Confidentialité** : où vont vos fichiers ? Combien de temps sont-ils conservés ? (Chez nous : suppression automatique — envois sous 30 minutes, résultats sous 2 heures —, zéro rétention, jamais utilisé pour l'entraînement de modèles.)
|
||||
|
||||
## Conclusion
|
||||
|
||||
Traduire un Excel sans perdre la mise en forme, c'est possible — mais seulement avec un outil qui travaille *dans* la structure du fichier, pas *autour*.
|
||||
|
||||
Testez le plan gratuit (2 documents/mois, sans carte bancaire) sur votre fichier le plus complexe, et comparez. C'est le meilleur moyen de voir la différence.
|
||||
|
||||
---
|
||||
**Métadonnées SEO**
|
||||
- Title : « Traduire un Excel sans perdre la mise en forme : guide 2026 »
|
||||
- Meta description : « Formules, cellules fusionnées, styles : ce qui casse vraiment vos Excel traduits, et comment les traduire en place sans rien perdre. Test gratuit, sans carte. »
|
||||
- Mot-clés principaux : traduire excel, translation excel, traduire document excel, mise en forme excel
|
||||
- URLs cibles : /blog/traduire-excel-mise-en-forme
|
||||
74
docs/marketing/blog/2-comparatif-outils-traduction-2026.md
Normal file
74
docs/marketing/blog/2-comparatif-outils-traduction-2026.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Les 5 meilleurs outils de traduction de documents comparés (2026)
|
||||
|
||||
> Article comparatif SEO — brouillon complet. À compléter avec captures réelles de chaque outil (section « verdict ») avant publication.
|
||||
|
||||
## Introduction
|
||||
|
||||
En 2026, traduire un document de bureau est devenu un marché encombré : traducteurs humains, assistants IA, outils one-shot, plateformes TMS d'entreprise. Mais très peu d'entre eux résolvent le vrai problème : **traduire sans casser la mise en page**.
|
||||
|
||||
Nous avons comparé les 5 approches que vous allez réellement croiser, avec leurs forces, leurs faiblesses et le cas d'usage où chacune gagne. (Nous en faisons partie — ce comparatif est donc à lire avec cette lunette, et nous avons pris soin de ne pas nous déclarer vainqueur par défaut.)
|
||||
|
||||
## Les 5 catégories
|
||||
|
||||
### 1. Traducteurs humains (agences)
|
||||
- **Force** : qualité sémantique imbattable, gestion des nuances, relecture
|
||||
- **Faiblesse** : coût (50–100× une machine) et délai (jours à semaines)
|
||||
- **Gagne quand** : contenus à forte valeur, juridiques, marketing, tout ce qui ne supporte pas l'erreur
|
||||
- **À savoir** : un humain traduit *le texte*. La réintégration dans le fichier (Excel, PPT) reste souvent une étape manuelle distincte, facturée à part
|
||||
|
||||
### 2. Google Translate (docs)
|
||||
- **Force** : gratuit, instantané, 100+ langues, connu de tous
|
||||
- **Faiblesse** : aplatit le document en texte brut. Formules, fusions, styles, animations : perdus. Qualité « correcte » mais non professionnelle
|
||||
- **Gagne quand** : traduction d'exploration, comprendre un document, pas le livrer
|
||||
- **À savoir** : c'est l'étalon du « gratuit qui casse le format » — le contrepied de notre positionnement
|
||||
|
||||
### 3. DeepL (documents)
|
||||
- **Force** : qualité de traduction supérieure à Google pour les textes professionnels
|
||||
- **Faiblesse** : formatage limité sur les documents complexes ; moteur unique (vous ne choisissez pas l'IA) ; prix plus élevés
|
||||
- **Gagne quand** : textes longs, qualité prioritaire, budget disponible
|
||||
- **À savoir** : DeepL est l'un de *nos* moteurs — sur les plans payants, vous pouvez le choisir par document, au même titre que d'autres
|
||||
|
||||
### 4. Plateformes TMS d'entreprise (Smartling, Smartcat, Transifex)
|
||||
- **Force** : gestion de flux, glossaires, collaboration, intégrations
|
||||
- **Faiblesse** : lourdes, chères, surdimensionnées pour une PME ; orientées localisation logicielle/i18n plus que documents bureautiques
|
||||
- **Gagne quand** : grande entreprise, volumes massifs, processus multi-équipes
|
||||
- **À savoir** : Transifex est notamment très fort sur la localisation de logiciels, moins sur les fichiers Word/Excel/PowerPoint
|
||||
|
||||
### 5. Traduction en place multi-moteurs (Office Translator)
|
||||
- **Force** : 100 % de préservation du format (fusions, formules, styles, slides), choix parmi 7 moteurs, glossaires techniques, API
|
||||
- **Faiblesse** : plus jeune que les TMS ; l'écosystème est encore en construction
|
||||
- **Gagne quand** : PME/ agences qui veulent du « prêt à livrer » sans réfection manuelle
|
||||
- **À savoir** : c'est nous ; le plan Free (2 docs/mois) permet de tester sans carte
|
||||
|
||||
## Tableau récapitulatif
|
||||
|
||||
| Critère | Humain | Google | DeepL | TMS entreprise | **Office Translator** |
|
||||
|---|---|---|---|---|---|
|
||||
| Format préservé | Manuel | Non | Partiel | Variable | **Oui (100 %)** |
|
||||
| Formules Excel | Oui | Non | Non | Variable | **Oui** |
|
||||
| Choix du moteur | n/a | 1 | 1 | 1–2 | **7** |
|
||||
| Glossaires | Oui | Non | Oui (payant) | Oui | **Oui** |
|
||||
| API | Non | Oui | Oui | Oui | **Oui (Business)** |
|
||||
| Coût relatif | 50–100× | 0× | 5–10× | 20–50× | **1× (réf.)** |
|
||||
| Délai | Jours | Secondes | Minutes | Jours | **Minutes** |
|
||||
| Confidentialité | Contractuel | Variable | Variable | Contractuel | **Zéro rétention, TTL 60 min** |
|
||||
|
||||
## Verdict par profil
|
||||
|
||||
- **Freelance / étudiant** → plan Free, puis Starter si volume
|
||||
- **PME internationale** → Pro (multi-moteurs + IA + glossaires)
|
||||
- **Agence de traduction** → Business (API + 5 sièges) pour un premier passage machine, relecture humaine
|
||||
- **Grande entreprise / volume massif** → comparez avec un TMS ; l'API Business peut suffire, ou Enterprise sur mesure
|
||||
|
||||
## Conclusion
|
||||
|
||||
Il n'y a pas de « meilleur outil » universel — il y a le bon outil pour votre cas d'usage. La seule question qui ne devrait jamais se poser : **votre document ressort-il intact ?** C'est le critère que la plupart des outils gratuits et one-shot échouent, et celui sur lequel nous avons construit tout le produit.
|
||||
|
||||
Testez le plan gratuit sur votre fichier le plus sale et jugez.
|
||||
|
||||
---
|
||||
**Métadonnées SEO**
|
||||
- Title : « Outils de traduction de documents 2026 : comparatif complet »
|
||||
- Meta description : « Humain, Google, DeepL, TMS ou traduction en place : 5 approches comparées sur format, prix, délai et confidentialité. Trouvez celle qui vous correspond. »
|
||||
- Mots-clés : comparatif traduction documents, meilleur outil traduction, deepl vs google, traduire word excel powerpoint
|
||||
- URL : /blog/comparatif-outils-traduction-document-2026
|
||||
@@ -0,0 +1,83 @@
|
||||
# Traduction professionnelle de documents : guide complet pour les PME
|
||||
|
||||
> Article guide SEO — brouillon complet. À illustrer avec 2 captures d'écran réelles (glossaire + dashboard) avant publication.
|
||||
|
||||
## Introduction
|
||||
|
||||
Une PME qui exporte, qui a des clients étrangers ou qui s'internationalise produit des dizaines de documents par mois : fiches techniques, matrices tarifaires, contrats, présentations commerciaux, manuels utilisateurs. Traduire tout ça « à la main » coûte une fortune et des semaines. Le guide complet pour y voir clair — et choisir sans se faire avoir.
|
||||
|
||||
## 1. Évaluez votre volume réel
|
||||
|
||||
Avant de choisir quoi que ce soit, mesurez :
|
||||
|
||||
- **Documents/mois** par type (Excel, Word, PowerPoint)
|
||||
- **Pages/moyenne par document** (c'est la page, pas le document, qui se paie en traduction humaine)
|
||||
- **Fréquence des mises à jour** (un tarif qui change chaque mois ne se traduit pas une fois)
|
||||
- **Langues cibles** (1 langue = simple ; 5 langues = le coût multiplie par 5)
|
||||
|
||||
**Règle rapide** : si vous dépassez ~50 pages/mois, la traduction humaine pure devient votre poste de dépense n°1 en localisation.
|
||||
|
||||
## 2. Les 3 stratégies
|
||||
|
||||
### Stratégie A — 100 % humaine
|
||||
- Qualité maximale, zéro risque
|
||||
- Coût : comptez 0,08–0,25 € le mot selon le domaine (légal et technique en haut de fourchette)
|
||||
- Délai : 2–5 jours ouvrés par lot
|
||||
- **Pour qui** : contenu à forte valeur, marchés réglementés, image de marque
|
||||
|
||||
### Stratégie B — 100 % machine
|
||||
- Coût quasi nul, délai en minutes
|
||||
- Risque : formats cassés (si l'outil est mauvais), terminologie incohérente, erreurs sémantiques
|
||||
- **Pour qui** : contenu interne, exploration, brouillons
|
||||
|
||||
### Stratégie C — Hybride (la plus rentable pour la majorité des PME)
|
||||
1. **Premier passage machine** avec un outil qui préserve le format (fusions, formules, styles intacts)
|
||||
2. **Terminologie verrouillée** via un glossaire (vos termes, pas ceux du moteur)
|
||||
3. **Relecture humaine ciblée** sur les documents clients à forte valeur uniquement
|
||||
|
||||
Résultat typique : **70–90 % du délai humain en moins, pour 10–20 % du coût** — en gardant la qualité sur ce qui compte.
|
||||
|
||||
## 3. Les critères de choix d'un outil
|
||||
|
||||
| Critère | Pourquoi c'est décisif |
|
||||
|---|---|
|
||||
| **Préservation du format** | Le vrai coût caché est la réfection manuelle. Testez sur VOTRE fichier le plus complexe |
|
||||
| **Multi-moteurs** | Un moteur unique = un point de défaillance et un plafond de qualité. 7 moteurs au choix = la bonne réponse au bon prix |
|
||||
| **Glossaires** | Sans glossaire, « unité de froid » devient « cold unit » un jour et « chiller » le lendemain. Inacceptable en B2B |
|
||||
| **Coût par page** | Comparez toujours à la page, jamais au document (un document fait 3 pages ou 300) |
|
||||
| **Confidentialité** | Où vont vos fichiers ? Durée de conservation ? Utilisation pour l'entraînement ? (Chez nous : suppression automatique — envois 30 min, résultats ≤ 2 h —, zéro rétention, jamais d'entraînement sur vos données) |
|
||||
| **API** | Si vous voulez automatiser (CRM, e-signature, ERP), l'API n'est pas optionnelle |
|
||||
|
||||
## 4. Mettre en place le glossaire (l'étape que tout le monde saute)
|
||||
|
||||
1. **Collectez** vos 50–200 termes récurrents (produits, acronymes, termes légaux, noms propres)
|
||||
2. **Validez** la traduction officielle de chaque terme avec votre équipe (une seule source de vérité)
|
||||
3. **Chargez** le glossaire dans l'outil et appliquez-le à chaque document
|
||||
4. **Itérez** : chaque nouvelle ambiguïté trouvée en relecture devient une entrée de glossaire
|
||||
|
||||
Après 2 mois de glossaire mature, la relecture humaine se concentre sur le sens, pas sur la terminologie. C'est là que le ROI explose.
|
||||
|
||||
## 5. Confiance et sécurité
|
||||
|
||||
- **Données** : chiffrement en transit (TLS), suppression automatique des fichiers (30 min pour les envois, 2 h max pour les résultats, chez nous)
|
||||
- **Conformité** : vérifiez les engagements de l'éditeur (RGPD pour les clients UE)
|
||||
- **Sauvegarde de votre travail** : gardez toujours le fichier source en version originale — aucun outil ne devrait être un point de perte unique
|
||||
|
||||
## 6. Ce que ça coûte vraiment
|
||||
|
||||
Repère : une page de document bureautique traduite et *réintégrée* par un humain coûte 5–20 €. En hybride (machine + relecture ciblée), comptez 0,50–1,50 € la page. Sur 500 pages/mois, l'écart est de **plusieurs milliers d'euros par mois**.
|
||||
|
||||
C'est la ligne à regarder avant de signer quoi que ce soit.
|
||||
|
||||
## Conclusion
|
||||
|
||||
La traduction professionnelle de documents pour une PME ne se joue pas sur « machine ou humain » — elle se joue sur **format préservé + terminologie verrouillée + relecture ciblée**. Choisissez l'outil qui livre les deux premières, et gardez les humains sur la troisième.
|
||||
|
||||
Le plan Free (2 documents/mois, sans carte) vous permet de tester la méthode sur vos vrais fichiers avant de décider.
|
||||
|
||||
---
|
||||
**Métadonnées SEO**
|
||||
- Title : « Traduction professionnelle de documents : guide PME 2026 »
|
||||
- Meta description : « Humain, machine ou hybride : le guide complet pour traduire vos documents PME sans casser le format ni le budget. Coûts, glossaires, confidentialité. »
|
||||
- Mots-clés : traduction documents PME, traduction professionnelle, coût traduction, glossaire traduction
|
||||
- URL : /blog/traduction-professionnelle-pme-guide
|
||||
@@ -0,0 +1,90 @@
|
||||
# Pourquoi DeepL et Google Translate « détruisent » vos documents Excel
|
||||
|
||||
> Article problématique SEO — brouillon complet. Ton direct. À illustrer d'un avant/après réel (capture) avant publication.
|
||||
|
||||
## Introduction
|
||||
|
||||
DeepL est excellent. Google Translate est gratuit. Et pourtant, des équipes entières maudissent les deux chaque semaine pour la même raison : **leurs fichiers Excel n'ont plus rien à voir avec l'original une fois traduits**.
|
||||
|
||||
Ce n'est pas un bug. C'est un choix d'architecture. Voici ce qui se passe réellement sous le capot, et ce qu'il faudrait pour faire mieux.
|
||||
|
||||
## Ce que ces outils font *vraiment* d'un fichier Excel
|
||||
|
||||
### Un .xlsx n'est pas une grille, c'est du XML
|
||||
|
||||
Sous le capot, `compta.xlsx` est une archive ZIP contenant :
|
||||
|
||||
- `sheet1.xml` — les cellules, leurs valeurs, leurs styles
|
||||
- `sharedStrings.xml` — le texte dédupliqué
|
||||
- `styles.xml` — polices, bordures, formats de nombre
|
||||
- `workbook.xml` — fusions de cellules, onglets, mises en page
|
||||
|
||||
### Le pipeline « standard » de traduction
|
||||
|
||||
Tous les outils de traduction de documents — y compris les très bons — suivent ce schéma :
|
||||
|
||||
```
|
||||
Excel → extraction en texte brut → traduction → réinjection dans un fichier NEUF
|
||||
```
|
||||
|
||||
Chaque étape de perte :
|
||||
|
||||
| Étape | Ce qui est perdu |
|
||||
|---|---|
|
||||
| Extraction | Formules (remplacées par la valeur), fusions, styles, formats de nombre, commentaires |
|
||||
| Traduction | Rien — c'est là que ça va bien |
|
||||
| Réinjection | Tout ce qui n'était pas du texte : le fichier de sortie est une *coquille vide* |
|
||||
|
||||
**Le résultat** : le texte est en anglais, mais votre fichier de comptabilité est devenu un fichier de texte habillé en Excel. Les formules sont mortes, les colonnes ont bougé, les pourcentages sont devenus des décimaux.
|
||||
|
||||
### Pourquoi DeepL ne peut pas faire « mieux » (en l'état)
|
||||
|
||||
DeepL traduit *le texte*, superbement. Mais son API document est pensée pour du texte, pas pour la **structure** d'un fichier Office. L'ingénierie de réintégration (reconstruire le XML d'origine avec les chaînes traduites, en conservant formules et fusions) est exactement le travail qu'un outil *spécialisé* fait — et qu'un traducteur généraliste ne fait pas.
|
||||
|
||||
Google, lui, ne fait même pas l'étape 3 proprement : vous collez, il recolle, et c'est tout.
|
||||
|
||||
## Ce qu'un outil « en place » fait de différent
|
||||
|
||||
La différence tient à une inversion : **ne jamais sortir le texte de sa structure**.
|
||||
|
||||
1. **Parse** le XML natif (cellules + fusions + formules + styles)
|
||||
2. **Extrait uniquement** les chaînes traduisables, en mémorisant position et style
|
||||
3. **Traduit** ces chaînes (au choix : DeepL, Google, ou un LLM contextuel)
|
||||
4. **Réinjecte** dans *le même* XML
|
||||
5. **Regénère** le fichier
|
||||
|
||||
Même structure, même nombre de cellules, mêmes fusions, mêmes formules — du texte en langue cible. C'est la seule architecture qui donne un fichier « prêt à livrer ».
|
||||
|
||||
## Testez-le vous-même (5 minutes)
|
||||
|
||||
Prenez votre fichier le plus sale — fusions, formules, colonnes formatées :
|
||||
|
||||
1. Traduisez-le avec l'outil que vous utilisez habituellement
|
||||
2. Ouvrez la **barre de formule** d'une cellule qui en contenait une
|
||||
3. Vérifiez une fusion de cellules
|
||||
4. Regardez un pourcentage
|
||||
|
||||
Si vous avez dû « refaire » le fichier, vous venez de payer la traduction *deux fois* : une fois à la machine, une fois à vous.
|
||||
|
||||
## Ce que ça change concrètement
|
||||
|
||||
| | Pipeline standard | Traduction en place |
|
||||
|---|---|---|
|
||||
| Formules | Morte (valeur figée) | Intacte |
|
||||
| Fusions | Perdues/décalées | Intactes |
|
||||
| Styles | Perdus | Intacts |
|
||||
| Temps total | Traduction + réfection | Traduction seule |
|
||||
| Fichier livrable | Non | Oui |
|
||||
|
||||
## Conclusion
|
||||
|
||||
Le problème n'est pas la qualité de la traduction — c'est la **réintégration**. DeepL et Google excellent sur la première ; personne ne fait correctement la seconde, sauf les outils spécialisés.
|
||||
|
||||
Si vos documents comptent (fusions, formules, slides), testez un outil qui travaille *dans* la structure du fichier. Le plan Free d'Office Translator (2 docs/mois, sans carte) est fait pour ça : mettez-lui votre fichier le plus sale, et comparez.
|
||||
|
||||
---
|
||||
**Métadonnées SEO**
|
||||
- Title : « Pourquoi DeepL et Google cassent vos Excel (et comment faire mieux) »
|
||||
- Meta description : « Formules mortes, fusions perdues : ce que les traducteurs « standard » font à vos fichiers Excel, et l'architecture qui préserve 100 % du format. »
|
||||
- Mots-clés : deepl excel, google translate excel, excel traduit mise en forme perdue
|
||||
- URL : /blog/pourquoi-deepl-google-cassent-vos-excel
|
||||
165
docs/marketing/blog/5-auto-heberger-traduction-docker.md
Normal file
165
docs/marketing/blog/5-auto-heberger-traduction-docker.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# Auto-héberger son outil de traduction de documents : guide Docker complet
|
||||
|
||||
> Article SEO « self-hosted » — brouillon complet. Capture l'audience développeurs/homelab. Basé sur le déploiement réel du projet (Docker Compose, Postgres, Redis).
|
||||
|
||||
## Introduction
|
||||
|
||||
Un outil de traduction de documents posé sur un SaaS pose toujours la même question : **où vont mes fichiers ?** Pour les documents confidentiels (compta, juridique, RH, brevets), l'auto-hébergement est la seule réponse qui tienne.
|
||||
|
||||
Ce guide montre comment monter une pile complète de traduction de documents — API, base, cache, interface — sur votre propre infrastructure, avec Docker. (Le projet dont on s'inspire est open source : Office Translator / Wordly.art.)
|
||||
|
||||
## Architecture cible
|
||||
|
||||
```
|
||||
[Client web / API]
|
||||
│
|
||||
[Nginx (reverse proxy, TLS)]
|
||||
│
|
||||
[FastAPI (Python 3.11+, port 8000)]
|
||||
├── [PostgreSQL] ← utilisateurs, glossaires, quotas
|
||||
├── [Redis] ← rate limiting (token bucket), files
|
||||
└── [Fichiers] ← uploads / outputs (volume local)
|
||||
│
|
||||
[Next.js (port 3000)] ← interface
|
||||
```
|
||||
|
||||
Points clés :
|
||||
- **FastAPI** asynchrone pour l'API de traduction (Swagger sur `/docs`)
|
||||
- **PostgreSQL** pour les données structurées, **Redis** pour le rate limiting par IP et les files
|
||||
- **Nginx** devant pour le TLS et le reverse proxy
|
||||
- Volumes persistants pour les fichiers (suppression automatique au TTL, ex. 60 min)
|
||||
|
||||
## 1. Prérequis
|
||||
|
||||
- Une machine Linux (VPS, serveur physique, ou même un NAS)
|
||||
- Docker + Docker Compose installés
|
||||
- Un domaine pointant vers la machine (pour le TLS)
|
||||
- Les clés API de vos moteurs de traduction (au minimum Google ; DeepL, OpenRouter, OpenAI, x.ai/Zai selon vos besoins)
|
||||
|
||||
## 2. Déploiement
|
||||
|
||||
### 2.1 Cloner et configurer
|
||||
|
||||
```bash
|
||||
git clone <URL_DU_DÉPÔT> /opt/wordly # dépôt à adapter (miroir public recommandé)
|
||||
cd /opt/wordly
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Dans `.env`, renseignez au minimum :
|
||||
|
||||
```ini
|
||||
# Moteurs — le Google « classic » (gratuit) ne demande AUCUNE clé.
|
||||
# Clés optionnelles pour les moteurs payants :
|
||||
GOOGLE_CLOUD_API_KEY=...
|
||||
# ou DEEPL_API_KEY, OPENROUTER_API_KEY, OPENAI_API_KEY...
|
||||
# Optionnel : OCR Mistral pour les PDF scannés
|
||||
# MISTRAL_API_KEY=...
|
||||
|
||||
# Sécurité
|
||||
SECRET_KEY=<générer une clé longue et aléatoire>
|
||||
CORS_ORIGINS=https://wordly.example
|
||||
APP_ENV=production
|
||||
```
|
||||
|
||||
> **Sécurité** : `CORS_ORIGINS` ne doit jamais être `*` en production — l'application le refuse et refuse de démarrer. Générez `SECRET_KEY` avec `openssl rand -hex 32`.
|
||||
|
||||
### 2.2 Lancer la pile
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Ça démarre :
|
||||
- L'API sur `:8001` côté hôte (mappée sur le 8000 du conteneur ; docs Swagger sur `http://localhost:8001/docs`)
|
||||
- L'interface sur `:3000`
|
||||
- Postgres + Redis en interne
|
||||
|
||||
### 2.3 Mettre Nginx devant (TLS)
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name wordly.example;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/wordly.example/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/wordly.example/privkey.pem;
|
||||
|
||||
# Headers de sécurité (HSTS, CSP)
|
||||
add_header Strict-Transport-Security "max-age=63072000" always;
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass http://127.0.0.1:8001;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Renouvelez le certificat avec `certbot renew` (cron).
|
||||
|
||||
## 3. Exploitation
|
||||
|
||||
### Surveillance
|
||||
|
||||
- `GET /health` — état de l'API, des moteurs, de la base et du cache
|
||||
- Prometheus + Grafana (fournis en `docker-compose.monitoring.yml`) pour les métriques système et applicatives
|
||||
|
||||
### Sauvegarde (non négociable)
|
||||
|
||||
La base Postgres et le répertoire de fichiers sont votre patrimoine. Deux couches :
|
||||
|
||||
1. **Sauvegarde quotidienne de la base** :
|
||||
```bash
|
||||
docker compose exec db sh -c 'pg_dump -U ${POSTGRES_USER:-translate} ${POSTGRES_DB:-translate_db}' | gzip > /backups/wordly-$(date +%F).sql.gz
|
||||
```
|
||||
2. **Réplication vers un NAS/autre site** via `rsync` + SSH (le projet fournit un plan de sauvegarde automatique et une procédure de restauration en ~20 min, y compris bascule sur un serveur de secours)
|
||||
|
||||
Règle d'or : **une sauvegarde qu'on n'a jamais restaurée est une sauvegarde qui n'existe pas**. Testez la restauration au moins une fois par trimestre.
|
||||
|
||||
### Nettoyage automatique
|
||||
|
||||
Les fichiers uploadés sont supprimés automatiquement après leur TTL (60 min par défaut). Le nettoyage est orchestré côté application ; surveillez le disque quand même (un gros fichier en attente peut saturer le volume).
|
||||
|
||||
## 4. Durcissement
|
||||
|
||||
- [ ] `CORS_ORIGINS` restreint à vos domaines
|
||||
- [ ] `SECRET_KEY` unique et stockée en variable d'environnement (jamais dans le repo)
|
||||
- [ ] Pas de port Postgres/Redis exposé publiquement (réseau Docker interne uniquement)
|
||||
- [ ] Rate limiting actif (token bucket dans Redis, par IP client)
|
||||
- [ ] Fail2ban sur SSH si l'accès serveur passe par SSH
|
||||
- [ ] Mises à jour de Docker + des images (vulnérabilités)
|
||||
- [ ] Monitoring des disques, CPU et mémoire (Grafana)
|
||||
|
||||
## 5. Coût vs SaaS
|
||||
|
||||
| Poste | Self-hosted (VPS 4 vCPU / 8 Go) | SaaS |
|
||||
|---|---|---|
|
||||
| Infrastructure | ~40–80 €/mois | Inclus |
|
||||
| Coût par page | Coût des API uniquement | Coût API + marge |
|
||||
| Maîtrise des données | Totale | Partagée |
|
||||
| Maintenance | À votre charge | À charge de l'éditeur |
|
||||
| Mises à jour / sécurité | À votre charge | À charge de l'éditeur |
|
||||
|
||||
Le self-hosted gagne quand **les données sont sensibles** ou quand le **volume est énorme**. Il perd quand vous n'avez personne pour faire la maintenance — dans ce cas, le SaaS est souvent plus fiable *en pratique*, même si les fichiers transigent avec un tiers.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Auto-héberger une pile de traduction de documents, c'est 1 h de déploiement et une vraie responsabilité d'exploitation. Avec Docker Compose, Postgres, Redis et Nginx, l'architecture tient sur une seule machine ; ce qui coûte ensuite, c'est la discipline (sauvegardes, durcissement, mises à jour).
|
||||
|
||||
Si les documents sont confidentiels, cette discipline se paie d'elle-même.
|
||||
|
||||
---
|
||||
**Métadonnées SEO**
|
||||
- Title : « Auto-héberger un outil de traduction de documents : guide Docker 2026 »
|
||||
- Meta description : « FastAPI, Postgres, Redis, Nginx : déployez une pile de traduction de documents sur votre propre infra. Sauvegardes, durcissement, coût vs SaaS. »
|
||||
- Mots-clés : auto héberger traduction, docker traduction documents, self hosted translation
|
||||
- URL : /blog/auto-heberger-traduction-documents-docker
|
||||
72
docs/marketing/kpis.md
Normal file
72
docs/marketing/kpis.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Suivi des KPIs — Définitions & Cadence
|
||||
|
||||
> Document de mesure. Source de vérité pour le reporting marketing.
|
||||
> Mis à jour : 2026-08-29
|
||||
|
||||
## 1. Stack de mesure (déployée)
|
||||
|
||||
| Outil | Rôle | Où |
|
||||
|---|---|---|
|
||||
| **Vercel Analytics** | Pages, événements, referrers, heatmaps | `office-translator-landing-page/app/layout.tsx` |
|
||||
| **Événements custom** | Funnels précis (voir §2) | `track(...)` dans les composants |
|
||||
| **Endpoint waitlist** | Source de vérité des e-mails | `GET /api/v1/waitlist/count` |
|
||||
| **BDD utilisateurs** | Inscriptions, plans, quotas | Backend (Postgres) |
|
||||
| **Journal de jobs** | Documents traduits, temps de traitement | Backend |
|
||||
| **Stripe** | MRR, abonnements, churn | Portail Stripe |
|
||||
| **`/health` + Grafana** | Performance système/applicative | Backend |
|
||||
|
||||
## 2. Événements trackés (Vercel Analytics)
|
||||
|
||||
| Événement | Propriétés | Déclenché par | Funnel |
|
||||
|---|---|---|---|
|
||||
| `translation_started` | `engine`, `source`, `target` | Bouton « Translate Document » | Activation |
|
||||
| `pricing_cta_click` | `plan` | CTA d'un plan | Considération |
|
||||
| `waitlist_joined` | `interest` | Inscription waitlist réussie | Capture |
|
||||
| `waitlist_error` | `status` | Inscription waitlist en échec | Capture (santé) |
|
||||
|
||||
**Pages clés à surveiller** : `/` (hero, pricing, faq, waitlist), `/dashboard` (activation), `/admin` (ops).
|
||||
|
||||
## 3. Définitions des KPIs
|
||||
|
||||
| KPI | Définition | Source | Fréquence |
|
||||
|---|---|---|---|
|
||||
| Visiteurs uniques | Sessions sur `/` (30 j glissants) | Vercel Analytics | Hebdo |
|
||||
| Taux d'activation | `translation_started` / visiteurs uniques | Vercel Analytics | Hebdo |
|
||||
| E-mails waitlist | `count` (dédoublonné) | `/api/v1/waitlist/count` | Quotidien |
|
||||
| Taux de capture | `waitlist_joined` / visiteurs uniques | Vercel Analytics | Hebdo |
|
||||
| Inscriptions | Nouveaux comptes (tous plans) | BDD utilisateurs | Hebdo |
|
||||
| Documents traduits | Jobs terminés | Journal de jobs | Hebdo |
|
||||
| Taux de conversion payant | Abonnements payants / inscriptions | Stripe + BDD | Mensuel |
|
||||
| MRR | Revenu mensuel récurrent | Stripe | Mensuel |
|
||||
| Churn mensuel | Abonnements perdus / début de période | Stripe | Mensuel |
|
||||
| CAC (Ads) | Dépense Ads / nouveaux payants (30 j) | Google Ads + Stripe | Mensuel |
|
||||
| NPS | Enquête post-traduction (0–10) | Enquête | Mensuel |
|
||||
| Disponibilité | `uptime` de `/health` | Grafana | Continu |
|
||||
|
||||
## 4. Cibles (alignées sur MARKETING_PLAN.md §6)
|
||||
|
||||
| KPI | M1 | M3 |
|
||||
|---|---|---|
|
||||
| Visiteurs uniques | 5 000 | 25 000 |
|
||||
| E-mails waitlist | 150 | 1 500 |
|
||||
| Inscriptions | 200 | 1 500 |
|
||||
| Documents traduits | 500 | 5 000 |
|
||||
| Conversion visite→inscription | 2 % | 4 % |
|
||||
| NPS | > 40 | > 50 |
|
||||
| MRR | — | ~300 abonnements payants |
|
||||
| CAC (Ads) | — | < 3× marge mensuelle (plan d'entrée) |
|
||||
|
||||
## 5. Rituels
|
||||
|
||||
- **Hebdo (lun, 30 min)** : tableau de bord Vercel + compteur waitlist + inscriptions. Anomalie > 2 σ = ticket.
|
||||
- **Mensuel (J+2, 1 h)** : MRR, churn, CAC, NPS. Mise à jour de ce doc + de `MARKETING_PLAN.md` si écart > 20 % vs cible.
|
||||
- **Sprint (2 sem.)** : A/B sur le pricing et le hero ; décision sur Google Ads (maintien/stop) à la semaine 5.
|
||||
|
||||
## 6. Alertes
|
||||
|
||||
| Condition | Action |
|
||||
|---|---|
|
||||
| `waitlist_error` > 5 % des tentatives | Investiguer `POST /api/v1/waitlist` (CORS, prod) |
|
||||
| Disponibilité `/health` < 99,5 % sur 24 h | Pager ops, consulter `DISASTER_RECOVERY.md` |
|
||||
| Churn > 10 % / mois | Analyse des raisons de sortie, offre de rétention |
|
||||
| CAC > 3× marge (Ads) | Stopper la campagne, revenir au SEO/partenariats |
|
||||
61
docs/marketing/launch/assets-spec.md
Normal file
61
docs/marketing/launch/assets-spec.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Spec de Production des Assets Visuels
|
||||
|
||||
> Checklist de production des assets marketing. Priorités : P0 = bloquant pour le lancement, P1 = semaine 1, P2 = semaine 2.
|
||||
|
||||
## P0 — Bloquant pour le lancement
|
||||
|
||||
### 1. Captures d'écran (8) — PNG, 1280×720 minimum, fond clair + sombre
|
||||
| # | Capture | Usage | Source |
|
||||
|---|---|---|---|
|
||||
| 1 | Page d'accueil / hero (drop-zone) | Landing, réseaux | `office-translator-landing-page` |
|
||||
| 2 | Upload en cours (fichier chargé, langues sélectionnées) | Démo du workflow | Idem |
|
||||
| 3 | **Résultat côte à côte** (original vs traduit, format intact) | Preuve qualité | Backend + dashboard |
|
||||
| 4 | Sélecteur de moteur (Google/DeepL/LLM) | Fonctionnalité clé | Idem |
|
||||
| 5 | Interface glossaire | Différenciation | Dashboard |
|
||||
| 6 | Dashboard (statistiques, quota) | Crédibilité | `/dashboard` |
|
||||
| 7 | Page pricing (4 plans) | Conversion | Landing `#pricing` |
|
||||
| 8 | Profil / historique | Confiance | Dashboard |
|
||||
|
||||
**Outil recommandé** : Playwright (script de capture automatisée pour les états dynamiques : upload en cours, progression de traduction).
|
||||
|
||||
### 2. Vidéo démo 60–90 s (« How it works ») — MP4 1080p, sous-titres FR + EN
|
||||
Script :
|
||||
1. (0–5 s) Logo + tagline animé
|
||||
2. (5–15 s) Problème : « Traduire un Excel de 50 pages sans casser le format ? »
|
||||
3. (15–40 s) Démo accélérée : upload → langues → moteur → traduction → téléchargement
|
||||
4. (40–55 s) Split-screen original/traduit, zoom sur tableaux et images intacts
|
||||
5. (55–65 s) CTA « Essayez gratuitement » + URL
|
||||
|
||||
### 3. OG Image — 1200×630 px
|
||||
Logo + tagline + capture résultat côte à côte. Utilisée pour X, LinkedIn, Product Hunt, partages.
|
||||
|
||||
### 4. GIF workflow 15 s
|
||||
Boucle upload → traduction → téléchargement (extraire de la vidéo démo). Pour le thread X et Product Hunt.
|
||||
|
||||
## P1 — Semaine 1
|
||||
|
||||
### 5. Logo
|
||||
SVG version claire + sombre, icône seule + avec texte « Office Translator » / « Wordly.art ».
|
||||
|
||||
### 6. Favicon
|
||||
32×32 et 16×16, ICO + PNG.
|
||||
|
||||
### 7. Bannière GitHub
|
||||
1280×640 px pour le README du repo.
|
||||
|
||||
### 8. Infographie « Pourquoi Office Translator »
|
||||
Comparatif avant/après (pipeline standard vs en place), 1 colonne, partageable.
|
||||
|
||||
### 9. Tutoriel YouTube 3–5 min
|
||||
Screencast + voiceover : création de compte, upload, glossaires, moteurs, téléchargement.
|
||||
|
||||
## P2 — Semaine 2
|
||||
|
||||
### 10. Templates e-mails (5)
|
||||
Bienvenue, « votre premier document », éducation glossaire, offre d'upgrade, réactivation.
|
||||
|
||||
### 11. Kit réseaux sociaux
|
||||
12 visuels (4 avant/après, 4 astuces, 4 mises à jour) au format 1080×1350 (LinkedIn) + 1600×900 (X).
|
||||
|
||||
## Validation
|
||||
Chaque asset P0 est validé par : (a) exactitude technique (pas de capture d'un état impossible), (b) lisibilité au format cible, (c) cohérence de la charte (couleurs `--accent` oklch(0.555 0.17 250), police Geist).
|
||||
49
docs/marketing/launch/product-hunt.md
Normal file
49
docs/marketing/launch/product-hunt.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Product Hunt — Listing & Maker Comment
|
||||
|
||||
> Statut : prêt à publier (Phase 2, J0). À remplir avec les assets réels avant le lancement.
|
||||
|
||||
## Tagline
|
||||
**Translate your Excel, Word, PowerPoint & PDF — keep the format perfect.**
|
||||
|
||||
## Nom affiché
|
||||
Office Translator (by Wordly.art)
|
||||
|
||||
## Description courte (160 car.)
|
||||
Translate office documents without losing the layout. Merged cells, formulas, tables, slides — translated in place, by the engine you choose.
|
||||
|
||||
## Description longue
|
||||
Google and DeepL flatten your files into plain text. We don't.
|
||||
|
||||
Office Translator translates Excel, Word, PowerPoint **and PDF** **in place**: merged cells, formulas, fonts, borders, headers, footers, tables and slide layouts survive intact. Even scanned PDFs are handled via OCR (Mistral), and text inside images is translated via vision models.
|
||||
|
||||
Pick the engine per document — 7 providers:
|
||||
- **Google** (free, fast)
|
||||
- **DeepL** (best for business text)
|
||||
- **Google Cloud**
|
||||
- **LLM engines**: OpenRouter (eco & premium tiers: DeepSeek, Gemini, Claude), OpenAI, Grok by xAI — for complex, context-aware translations
|
||||
|
||||
Plus:
|
||||
- **Custom glossaries** — lock your HVAC, legal or medical terminology across every document
|
||||
- **60+ languages**
|
||||
- **Private by design** — uploads deleted after 30 min, results auto-deleted within 2 h, never used to train models
|
||||
- **API** on the Business plan (10,000 calls/mo) for your own tooling
|
||||
|
||||
Pricing: Free (2 docs/mo) · Starter €9 · Pro €19 · Business €49 · Enterprise custom. Yearly = −20%.
|
||||
|
||||
## Maker comment (à publier par le fondateur, 9 h EST)
|
||||
> We built Office Translator because every other tool broke our layouts.
|
||||
>
|
||||
> The hard part wasn't translation — it was **re-integrating** the text. Our pipeline parses the document structure (cells, runs, shapes), extracts only translatable content, translates it (your choice of 7 engines), and writes it back into the **same** XML structure. Formulas, merges, fonts, borders: untouched.
|
||||
>
|
||||
> What I'd love your feedback on:
|
||||
> 1. Which engine do you trust most for business documents — DeepL or an LLM?
|
||||
> 2. Would you use the API (Business plan) to wire it into your own workflow?
|
||||
> 3. PDF (incl. scanned, via OCR) is already in. Which format should we tackle next — IDML, CSV, something else?
|
||||
>
|
||||
> Free plan = 2 full documents/month, no card. Link in comments.
|
||||
|
||||
## Checklist de publication
|
||||
- [ ] Assets: logo (1200×630 OG + 600×400), 3 screenshots (résultat côte à côte, sélecteur moteur, glossaire)
|
||||
- [ ] 5 commentaires amis planifiés (réponses de qualité, pas de upvote-only)
|
||||
- [ ] E-mail aux 150+ contacts de la waitlist (J-2) : « On lance demain, votez + testez »
|
||||
- [ ] Thread X croisé au lancement
|
||||
65
docs/marketing/launch/reddit-posts.md
Normal file
65
docs/marketing/launch/reddit-posts.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# Reddit — 3 Posts de Lancement
|
||||
|
||||
> Statut : prêts à publier (Phase 2). Adapter le ton par subreddit. Pas de lien brut dans le 1er post sur r/SideProject (shadowban) — lien en commentaire.
|
||||
|
||||
---
|
||||
|
||||
## 1. r/SideProject
|
||||
|
||||
**Titre :** I built a tool that translates Excel/Word/PowerPoint without destroying the formatting. Free plan = 2 docs/mo.
|
||||
|
||||
**Corps :**
|
||||
> Hi all — I've been building **Office Translator** for a while and I'm launching it this week.
|
||||
>
|
||||
> **The problem:** every translation tool I tried (Google, DeepL) flattens your file into plain text. Merged cells gone, formulas replaced by their computed values, slide layouts destroyed. You end up spending more time fixing the format than the translation saved you.
|
||||
>
|
||||
> **What I did instead:** the pipeline parses the document structure, extracts only the translatable content, translates it, and writes it back into the **same** structure. Merges, formulas, fonts, borders, headers/footers, table styles — untouched. Text inside images is handled by vision models.
|
||||
>
|
||||
> **Tech:** FastAPI (async), openpyxl / python-docx / python-pptx, 7 translation engines the user picks per document (Google, DeepL, Google Cloud, OpenRouter éco & premium, OpenAI, Grok/xAI), Stripe billing, Docker/Postgres/Redis.
|
||||
>
|
||||
> **Pricing:** Free (2 docs/mo, no card) → Starter €9 → Pro €19 → Business €49 (API included).
|
||||
>
|
||||
> I'd love feedback from anyone who deals with international documents. What's the nastiest format you've tried to translate?
|
||||
>
|
||||
> (Link in comments to avoid the filter.)
|
||||
|
||||
**Commentaire n°1 (lien) :**
|
||||
> Here's the landing page: [URL]. Free plan, no card. The /docs page has the full API reference if you want to poke at the API.
|
||||
|
||||
---
|
||||
|
||||
## 2. r/translator
|
||||
|
||||
**Titre :** We built a document translator that preserves formatting — curious how you'd use it (or wouldn't)
|
||||
|
||||
**Corps :**
|
||||
> Hi, I'm the dev behind Office Translator (Wordly.art). Before I pitch anything: we know translators are the ones who actually feel the pain when a machine mangles a layout, so I'd rather ask than sell.
|
||||
>
|
||||
> The tool translates .xlsx / .docx / .pptx **in place** — structure, formulas, tables and styles are preserved — and supports **custom glossaries**, which is the feature I'd value your opinion on most:
|
||||
>
|
||||
> 1. Is glossary-based consistency (e.g. a fixed HVAC or legal term list) actually useful to you, or do you already solve this differently?
|
||||
> 2. For which document types is the format preservation worth paying for, vs. where a human review still beats any tool?
|
||||
> 3. What would make you trust an output enough to skip a full re-read? (confidence scores? diff view? per-segment override?)
|
||||
>
|
||||
> It's aimed at agencies and in-house teams as a **first pass** to cut turnaround — not to replace the translator. I'd genuinely like to hear what would make it a tool you'd recommend to a colleague.
|
||||
|
||||
> ⚠️ Ce subreddit est hostile au self-promo : publier en tant que dev transparent, répondre à chaque commentaire, jamais de lien dans le post.
|
||||
|
||||
---
|
||||
|
||||
## 3. r/smallbusiness
|
||||
|
||||
**Titre :** How do you translate client documents (Excel/Word/PowerPoint) without losing the formatting? We built a tool for this.
|
||||
|
||||
**Corps :**
|
||||
> Running a business that works internationally means a lot of documents that need translating — pricing sheets, contracts, slide decks, training files.
|
||||
>
|
||||
> The usual options all hurt somewhere:
|
||||
> - **Human translators:** great quality, 50–100× the cost and a multi-day turnaround
|
||||
> - **Google/DeepL:** cheap and fast, but they flatten the file — merged cells, formulas, slide layouts get destroyed, so someone has to rebuild the document
|
||||
>
|
||||
> We built **Office Translator** to kill that middle step: it translates the document **in place** and gives you back a file that looks like it was written in the target language from the start. You pick the engine (7 options, from free Google to premium LLMs), and you can lock your company terminology with a custom glossary so "unité de froid" is never suddenly "cold unit".
|
||||
>
|
||||
> Free plan: 2 documents/month, no card. From €9/mo after that. Uploads are deleted after 30 minutes and results within 2 hours.
|
||||
>
|
||||
> If you handle international documents, what's your current workflow and where does it break? Curious what I'm missing.
|
||||
31
docs/marketing/launch/show-hn.md
Normal file
31
docs/marketing/launch/show-hn.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Hacker News — Show HN
|
||||
|
||||
> Statut : prêt à publier (Phase 2, J0 matin EST). Soumettre via « Show HN » — pas de self-upvote.
|
||||
|
||||
## Titre
|
||||
**Show HN: Office document translation that preserves the formatting**
|
||||
|
||||
## Corps
|
||||
> I built **Office Translator** (Wordly.art) because every document translator I used — Google, DeepL, the one-shot tools — destroyed the layout. Translating a 50-page Excel pricing matrix meant getting back a text dump: merged cells gone, formulas replaced by computed values, slide decks unrecognizable. The translation was 5% of the job; fixing the format was 95%.
|
||||
>
|
||||
> So I made the format-preservation the product:
|
||||
>
|
||||
> - **In-place translation** for .xlsx / .docx / .pptx / .pdf. The pipeline parses the native structure (cells + merges + formulas, paragraph runs + styles, slide XML + shapes), extracts only translatable content, translates it, and writes it back into the *same* structure. Output looks like it was authored in the target language.
|
||||
> - **Scanned PDFs work too**: image-only pages are recovered with OCR (Mistral) before translation — most competitors (DeepL, Azure) reject those outright.
|
||||
> - **7 engines, user's choice per document**: Google (free), DeepL, Google Cloud, OpenRouter LLMs in two tiers (DeepSeek / Gemini / Claude), OpenAI, and Grok (xAI). Cheap engine for drafts, premium LLM for client deliverables.
|
||||
> - **Custom glossaries** to lock technical/legal/medical terminology across documents.
|
||||
> - **Vision translation** for text inside images (paid plans).
|
||||
> - **Privacy**: uploads deleted after 30 min, results auto-deleted within 2 hours, zero data retention, content never used for training.
|
||||
> - **API** on the Business plan (10k calls/mo) — the web workflow is fully mirrored: submit file, poll job, download.
|
||||
>
|
||||
> Stack: FastAPI (Python 3.11), openpyxl / python-docx / python-pptx, PyMuPDF, Postgres + Redis, Stripe, Docker, Next.js 15 frontend.
|
||||
>
|
||||
> Free plan: 2 documents/month, no card. I'd love feedback from people who've worked on document-format-preserving pipelines — what edge cases do I not handle yet? (PDF incl. scanned/OCR is already in — next candidates are IDML and CSV.)
|
||||
>
|
||||
> https://wordly.art
|
||||
|
||||
## Notes de publication
|
||||
- Soumettre à 8–9 h EST, mardi–jeudi
|
||||
- Répondre à chaque commentaire dans les 2 h ; rester technique, pas commercial
|
||||
- Pas de « upvote my post » — c'est interdit sur HN
|
||||
- Préparer 2–3 réponses techniques de poches : formule Excel + localisation (format de nombre), fusion de cellules + traduction, gestion des runs mixtes (gras/italique au milieu d'un paragraphe)
|
||||
102
docs/marketing/launch/x-thread.md
Normal file
102
docs/marketing/launch/x-thread.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# X / Twitter — Thread de Lancement (12 tweets)
|
||||
|
||||
> Statut : prêt à publier (Phase 2, J0). Publier en thread (Reply chain), pas en post unique.
|
||||
> Assets requis : GIF 15 s du workflow (voir assets-spec.md), capture résultat côte à côte.
|
||||
|
||||
**T1 (accroche + GIF)**
|
||||
Your Excel got translated.
|
||||
Now fix the 47 broken merged cells, 12 dead formulas and the slide deck that lost its layout.
|
||||
|
||||
That's what "free" translation tools actually cost.
|
||||
|
||||
We built the opposite. Thread:
|
||||
|
||||
**T2 (le problème)**
|
||||
Every translator you've used works the same way:
|
||||
1. Flatten your document to plain text
|
||||
2. Translate the text
|
||||
3. Hope it fits back
|
||||
|
||||
The "hope" step is where your afternoon goes.
|
||||
|
||||
**T3 (notre approche)**
|
||||
Office Translator works in place:
|
||||
|
||||
• Parse the real structure (cells, runs, shapes)
|
||||
• Extract ONLY the translatable content
|
||||
• Translate it
|
||||
• Write it back into the SAME structure
|
||||
|
||||
Merges, formulas, fonts, borders: untouched.
|
||||
|
||||
**T4 (preuve)**
|
||||
[Capture d'écran : résultat côte à côte — original FR vs traduit EN, mêmes colonnes/fusions/formules]
|
||||
|
||||
Same file. Different language. Zero reformatting.
|
||||
|
||||
**T5 (multi-moteurs)**
|
||||
You pick the engine per document — 7 options:
|
||||
|
||||
• Google — free, fast
|
||||
• DeepL — best for business text
|
||||
• Google Cloud
|
||||
• DeepSeek / Gemini / Claude via OpenRouter — eco & premium LLM tiers
|
||||
• OpenAI, Grok (xAI)
|
||||
|
||||
Cheap for drafts. Premium LLM for the client deliverable.
|
||||
|
||||
**T6 (glossaires)**
|
||||
The feature agencies love: custom glossaries.
|
||||
|
||||
Lock "unité de froid → chiller", "clause de résiliation → termination clause" once. Every document after that uses your terminology. Consistent, every time.
|
||||
|
||||
**T7 (confidentialité)**
|
||||
Your documents are not our training data.
|
||||
|
||||
• Uploads deleted after 30 min, results within 2 h
|
||||
• Zero data retention
|
||||
• Encrypted in transit (TLS)
|
||||
• No human ever sees your content
|
||||
|
||||
**T8 (API)**
|
||||
Wiring it into your own stack? Business plan includes API access — 10,000 calls/month.
|
||||
|
||||
Submit file → poll job → download. Same workflow as the web app. Docs at /docs.
|
||||
|
||||
**T9 (prix)**
|
||||
Pricing (yearly = −20%):
|
||||
|
||||
• Free — 2 docs/mo, no card
|
||||
• Starter — €9
|
||||
• Pro — €19 (AI translation, glossaries)
|
||||
• Business — €49 (API, 5 seats)
|
||||
• Enterprise — custom
|
||||
|
||||
A page translated costs a fraction of a cent. A human does the same page for €1–2.
|
||||
|
||||
**T10 (pour qui)**
|
||||
Built for:
|
||||
• International SMEs (pricing sheets, contracts, manuals)
|
||||
• Translation agencies (first-pass + glossaries = faster turnaround)
|
||||
• Multilingual HR teams
|
||||
• Anyone who's ever re-built a translated Excel by hand
|
||||
|
||||
**T11 (preuve sociale + CTA)**
|
||||
60+ languages. 7 engines. 100% format preservation.
|
||||
|
||||
Free plan = 2 full documents/month, no card required. Test it on your ugliest file.
|
||||
|
||||
**T12 (CTA final)**
|
||||
👉 wordly.art
|
||||
|
||||
Launching on Product Hunt today — feedback welcome.
|
||||
|
||||
What's the nastiest document format you've ever tried to translate? Reply below, we read everything.
|
||||
|
||||
---
|
||||
|
||||
## Plan de publication
|
||||
- J-2 : teaser T1 seul (sans lien)
|
||||
- J0 : thread complet, 9 h
|
||||
- J+1 : republier T4 (preuve) + répondre à tous les retweets
|
||||
- Communautés à taguer : #BuildInPublic #SaaS #i18n #DevTools (max 2 hashtags)
|
||||
@@ -24,6 +24,7 @@ const PROVIDER_LABELS: Record<string, string> = {
|
||||
deepseek: "IA Express",
|
||||
minimax: "IA Avancée",
|
||||
zai: "Grok (xAI)",
|
||||
mistral_ocr: "OCR Mistral (PDF scannés)",
|
||||
};
|
||||
|
||||
const STATUS_CONFIG = {
|
||||
@@ -132,6 +133,11 @@ export function ProviderStatus({ data, isLoading }: ProviderStatusProps) {
|
||||
{new Date(provider.last_check).toLocaleTimeString()}
|
||||
</span>
|
||||
)}
|
||||
{provider.config_only && (
|
||||
<span className="text-muted-foreground">
|
||||
Statut dérivé de la configuration (pas d'appel réseau)
|
||||
</span>
|
||||
)}
|
||||
{provider.error && (
|
||||
<span className="text-red-500">{provider.error}</span>
|
||||
)}
|
||||
|
||||
@@ -42,6 +42,7 @@ interface SettingsConfig {
|
||||
openrouter: ProviderConfig;
|
||||
openrouter_premium: ProviderConfig;
|
||||
zai: ProviderConfig;
|
||||
mistral: ProviderConfig;
|
||||
smtp: SmtpConfig;
|
||||
fallback_chain: string;
|
||||
fallback_chain_classic: string;
|
||||
@@ -54,6 +55,7 @@ interface EnvInfo {
|
||||
openrouter: boolean;
|
||||
openrouter_premium: boolean;
|
||||
zai: boolean;
|
||||
mistral: boolean;
|
||||
ollama: boolean;
|
||||
google_cloud: boolean;
|
||||
smtp: boolean;
|
||||
@@ -74,6 +76,7 @@ const defaultConfig: SettingsConfig = {
|
||||
openrouter: { enabled: false, api_key: "", model: "deepseek/deepseek-chat" },
|
||||
openrouter_premium: { enabled: false, api_key: "", model: "openai/gpt-4o-mini" },
|
||||
zai: { enabled: false, api_key: "", base_url: "https://api.x.ai/v1", model: "grok-2-1212" },
|
||||
mistral: { enabled: false, api_key: "", model: "mistral-ocr-latest", timeout: 180 },
|
||||
smtp: { enabled: false, host: "", port: 587, username: "", password: "", from_email: "", use_tls: true },
|
||||
fallback_chain: "google,google_cloud,deepl,openrouter,openrouter_premium,openai,deepseek,zai",
|
||||
fallback_chain_classic: "google,google_cloud,deepl",
|
||||
@@ -86,6 +89,7 @@ const defaultEnvInfo: EnvInfo = {
|
||||
openrouter: false,
|
||||
openrouter_premium: false,
|
||||
zai: false,
|
||||
mistral: false,
|
||||
ollama: false,
|
||||
google_cloud: false,
|
||||
smtp: false,
|
||||
@@ -608,6 +612,42 @@ export default function AdminSettingsPage() {
|
||||
</div>
|
||||
</ProviderCard>
|
||||
|
||||
<ProviderCard
|
||||
title="OCR Mistral (PDF scannés)"
|
||||
description="Extraction du texte des PDF scannés (pages image) avant traduction. Sans clé, ces fichiers sont refusés avec un message clair. ~1 € / 1 000 pages. Clé : console.mistral.ai"
|
||||
enabled={config.mistral.enabled}
|
||||
onToggle={(enabled) => updateProvider("mistral", { enabled })}
|
||||
onTest={() => testProvider("mistral")}
|
||||
testResult={testResults.mistral ?? "idle"}
|
||||
testMessage={testMessages.mistral}
|
||||
envKeySet={envInfo.mistral}
|
||||
>
|
||||
<div className="grid gap-4 sm:grid-cols-2">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="mistral-key">Clé API Mistral</Label>
|
||||
<Input
|
||||
id="mistral-key"
|
||||
type="password"
|
||||
placeholder={envInfo.mistral ? "Clé configurée dans .env (laisser vide pour l'utiliser)" : "Clé console.mistral.ai"}
|
||||
value={config.mistral.api_key || ""}
|
||||
onChange={(e) => updateProvider("mistral", { api_key: e.target.value })}
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="mistral-model">Modèle OCR</Label>
|
||||
<Input
|
||||
id="mistral-model"
|
||||
placeholder="mistral-ocr-latest"
|
||||
value={config.mistral.model || ""}
|
||||
onChange={(e) => updateProvider("mistral", { model: e.target.value })}
|
||||
/>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
Recommandé : <code>mistral-ocr-latest</code>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</ProviderCard>
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-base">Chaîne de fallback</CardTitle>
|
||||
|
||||
@@ -30,6 +30,10 @@ export interface ProviderStatus {
|
||||
last_check: string | null;
|
||||
latency_ms?: number;
|
||||
error?: string;
|
||||
/** true when availability is derived from configuration (no live call) */
|
||||
config_only?: boolean;
|
||||
/** optional display label provided by the backend */
|
||||
label?: string;
|
||||
}
|
||||
|
||||
export interface CleanupResponse {
|
||||
|
||||
@@ -38,6 +38,9 @@ class FileCleanupManager:
|
||||
self.max_file_age_seconds = max_file_age_minutes * 60
|
||||
self.cleanup_interval = cleanup_interval_minutes * 60
|
||||
self.max_total_size_bytes = int(max_total_size_gb * 1024 * 1024 * 1024)
|
||||
# Untracked (orphan) files must be at least this old before deletion:
|
||||
# protects in-flight files whose Redis metadata is missing or delayed.
|
||||
self.orphan_grace_seconds = 900
|
||||
|
||||
self._running = False
|
||||
self._task: Optional[asyncio.Task] = None
|
||||
@@ -191,10 +194,13 @@ class FileCleanupManager:
|
||||
data = await redis_client.get(key)
|
||||
if data:
|
||||
metadata = json.loads(data)
|
||||
if "file_path" in metadata:
|
||||
# Normalize path to absolute string for comparison
|
||||
path_str = str(Path(metadata["file_path"]).absolute())
|
||||
tracked_paths.add(path_str)
|
||||
# Metadata uses "input_path" (and possibly other *_path keys);
|
||||
# collect every path field so tracked files are never
|
||||
# misclassified as orphans.
|
||||
for path_key in ("input_path", "file_path", "output_path"):
|
||||
if path_key in metadata:
|
||||
path_str = str(Path(metadata[path_key]).absolute())
|
||||
tracked_paths.add(path_str)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch tracked paths from Redis: {e}")
|
||||
redis_available = False
|
||||
@@ -234,8 +240,11 @@ class FileCleanupManager:
|
||||
reason = ""
|
||||
|
||||
if is_orphan:
|
||||
should_delete = True
|
||||
reason = "orphan"
|
||||
# Never delete a young file as orphan: it may belong to
|
||||
# a job whose Redis metadata is not yet visible.
|
||||
if file_age > self.orphan_grace_seconds:
|
||||
should_delete = True
|
||||
reason = "orphan"
|
||||
elif file_age > self.max_file_age_seconds:
|
||||
should_delete = True
|
||||
reason = "expired"
|
||||
|
||||
@@ -4,7 +4,7 @@ Validates all user inputs before processing
|
||||
"""
|
||||
|
||||
import re
|
||||
import magic
|
||||
import sys
|
||||
import ipaddress
|
||||
import socket
|
||||
from pathlib import Path
|
||||
@@ -13,6 +13,19 @@ from typing import Optional, List, Set, Tuple
|
||||
from fastapi import UploadFile, HTTPException
|
||||
import logging
|
||||
|
||||
# python-magic shells out to libmagic (a native library). It is known to
|
||||
# crash with an access violation on some Windows setups — a native crash a
|
||||
# try/except cannot catch. The ZIP/PDF magic-byte checks below do not need
|
||||
# libmagic, so it is simply disabled on Windows and treated as optional
|
||||
# elsewhere (a failed import degrades to the same magic-byte fallback).
|
||||
if sys.platform == "win32":
|
||||
magic = None
|
||||
else:
|
||||
try:
|
||||
import magic
|
||||
except Exception:
|
||||
magic = None
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -303,13 +316,17 @@ class FileValidator:
|
||||
def _detect_mime_type(self, content: bytes) -> str:
|
||||
"""Detect MIME type from file content"""
|
||||
try:
|
||||
mime = magic.Magic(mime=True)
|
||||
return mime.from_buffer(content)
|
||||
if magic is not None:
|
||||
mime = magic.Magic(mime=True)
|
||||
return mime.from_buffer(content)
|
||||
except Exception:
|
||||
# Fallback to basic detection
|
||||
if content.startswith(self.OFFICE_MAGIC_BYTES):
|
||||
return "application/zip"
|
||||
return "application/octet-stream"
|
||||
pass
|
||||
# Fallback to basic detection (magic bytes, no libmagic needed)
|
||||
if content.startswith(self.OFFICE_MAGIC_BYTES):
|
||||
return "application/zip"
|
||||
if content.startswith(self.PDF_MAGIC_BYTES):
|
||||
return "application/pdf"
|
||||
return "application/octet-stream"
|
||||
|
||||
def _validate_mime_type(self, mime_type: str, extension: str):
|
||||
"""Validate MIME type matches extension"""
|
||||
@@ -446,41 +463,127 @@ class LanguageValidator:
|
||||
}
|
||||
|
||||
LANGUAGE_NAMES = {
|
||||
"en": "English",
|
||||
"es": "Spanish",
|
||||
"fr": "French",
|
||||
"de": "German",
|
||||
"it": "Italian",
|
||||
"pt": "Portuguese",
|
||||
"ru": "Russian",
|
||||
"af": "Afrikaans",
|
||||
"sq": "Albanian",
|
||||
"am": "Amharic",
|
||||
"ar": "Arabic",
|
||||
"hy": "Armenian",
|
||||
"az": "Azerbaijani",
|
||||
"eu": "Basque",
|
||||
"be": "Belarusian",
|
||||
"bn": "Bengali",
|
||||
"bs": "Bosnian",
|
||||
"bg": "Bulgarian",
|
||||
"ca": "Catalan",
|
||||
"ceb": "Cebuano",
|
||||
"zh": "Chinese",
|
||||
"zh-CN": "Chinese (Simplified)",
|
||||
"zh-TW": "Chinese (Traditional)",
|
||||
"ja": "Japanese",
|
||||
"ko": "Korean",
|
||||
"ar": "Arabic",
|
||||
"hi": "Hindi",
|
||||
"nl": "Dutch",
|
||||
"pl": "Polish",
|
||||
"tr": "Turkish",
|
||||
"sv": "Swedish",
|
||||
"da": "Danish",
|
||||
"no": "Norwegian",
|
||||
"fi": "Finnish",
|
||||
"co": "Corsican",
|
||||
"hr": "Croatian",
|
||||
"cs": "Czech",
|
||||
"da": "Danish",
|
||||
"nl": "Dutch",
|
||||
"en": "English",
|
||||
"eo": "Esperanto",
|
||||
"et": "Estonian",
|
||||
"fi": "Finnish",
|
||||
"fr": "French",
|
||||
"fy": "Frisian",
|
||||
"gl": "Galician",
|
||||
"ka": "Georgian",
|
||||
"de": "German",
|
||||
"el": "Greek",
|
||||
"th": "Thai",
|
||||
"vi": "Vietnamese",
|
||||
"id": "Indonesian",
|
||||
"uk": "Ukrainian",
|
||||
"ro": "Romanian",
|
||||
"gu": "Gujarati",
|
||||
"ht": "Haitian Creole",
|
||||
"ha": "Hausa",
|
||||
"haw": "Hawaiian",
|
||||
"he": "Hebrew",
|
||||
"hi": "Hindi",
|
||||
"hmn": "Hmong",
|
||||
"hu": "Hungarian",
|
||||
"is": "Icelandic",
|
||||
"ig": "Igbo",
|
||||
"id": "Indonesian",
|
||||
"ga": "Irish",
|
||||
"it": "Italian",
|
||||
"ja": "Japanese",
|
||||
"jv": "Javanese",
|
||||
"kn": "Kannada",
|
||||
"kk": "Kazakh",
|
||||
"km": "Khmer",
|
||||
"rw": "Kinyarwanda",
|
||||
"ko": "Korean",
|
||||
"ku": "Kurdish",
|
||||
"ky": "Kyrgyz",
|
||||
"lo": "Lao",
|
||||
"la": "Latin",
|
||||
"lv": "Latvian",
|
||||
"lt": "Lithuanian",
|
||||
"lb": "Luxembourgish",
|
||||
"mk": "Macedonian",
|
||||
"mg": "Malagasy",
|
||||
"ms": "Malay",
|
||||
"ml": "Malayalam",
|
||||
"mt": "Maltese",
|
||||
"mi": "Maori",
|
||||
"mr": "Marathi",
|
||||
"mn": "Mongolian",
|
||||
"my": "Myanmar (Burmese)",
|
||||
"ne": "Nepali",
|
||||
"no": "Norwegian",
|
||||
"ny": "Nyanja (Chichewa)",
|
||||
"or": "Odia (Oriya)",
|
||||
"ps": "Pashto",
|
||||
"fa": "Persian (Farsi)",
|
||||
"pl": "Polish",
|
||||
"pt": "Portuguese",
|
||||
"pa": "Punjabi",
|
||||
"ro": "Romanian",
|
||||
"ru": "Russian",
|
||||
"sm": "Samoan",
|
||||
"gd": "Scots Gaelic",
|
||||
"sr": "Serbian",
|
||||
"st": "Sesotho",
|
||||
"sn": "Shona",
|
||||
"sd": "Sindhi",
|
||||
"si": "Sinhala",
|
||||
"sk": "Slovak",
|
||||
"sl": "Slovenian",
|
||||
"so": "Somali",
|
||||
"es": "Spanish",
|
||||
"su": "Sundanese",
|
||||
"sw": "Swahili",
|
||||
"sv": "Swedish",
|
||||
"tl": "Filipino (Tagalog)",
|
||||
"tg": "Tajik",
|
||||
"ta": "Tamil",
|
||||
"tt": "Tatar",
|
||||
"te": "Telugu",
|
||||
"th": "Thai",
|
||||
"tr": "Turkish",
|
||||
"tk": "Turkmen",
|
||||
"uk": "Ukrainian",
|
||||
"ur": "Urdu",
|
||||
"ug": "Uyghur",
|
||||
"uz": "Uzbek",
|
||||
"vi": "Vietnamese",
|
||||
"cy": "Welsh",
|
||||
"xh": "Xhosa",
|
||||
"yi": "Yiddish",
|
||||
"yo": "Yoruba",
|
||||
"zu": "Zulu",
|
||||
"auto": "Auto-detect",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def validate(cls, language_code: str, field_name: str = "language") -> str:
|
||||
"""Validate and normalize language code"""
|
||||
"""Validate and normalize language code.
|
||||
|
||||
Matching is case-insensitive so that regional variants stored in
|
||||
mixed case ("zh-CN", "zh-TW") can be submitted in any case; the
|
||||
canonical form from SUPPORTED_LANGUAGES is returned.
|
||||
"""
|
||||
if not language_code:
|
||||
raise ValidationError(f"{field_name} est requis", code="missing_language")
|
||||
|
||||
@@ -489,9 +592,17 @@ class LanguageValidator:
|
||||
|
||||
# Handle common variations
|
||||
if normalized in ["chinese", "cn"]:
|
||||
normalized = "zh-CN"
|
||||
normalized = "zh-cn"
|
||||
elif normalized in ["chinese-traditional", "tw"]:
|
||||
normalized = "zh-TW"
|
||||
normalized = "zh-tw"
|
||||
|
||||
# Resolve to the canonical form stored in SUPPORTED_LANGUAGES
|
||||
# (e.g. "zh-cn" -> "zh-CN"). Unknown codes stay lower-cased and are
|
||||
# rejected by the membership check below.
|
||||
for lang in cls.SUPPORTED_LANGUAGES:
|
||||
if lang.lower() == normalized:
|
||||
normalized = lang
|
||||
break
|
||||
|
||||
if normalized not in cls.SUPPORTED_LANGUAGES:
|
||||
raise ValidationError(
|
||||
|
||||
2
office-translator-landing-page/.gitignore
vendored
2
office-translator-landing-page/.gitignore
vendored
@@ -8,3 +8,5 @@ node_modules/
|
||||
.next/
|
||||
.env*.local
|
||||
.DS_Store
|
||||
# TypeScript incremental build artifact
|
||||
tsconfig.tsbuildinfo
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
import { SiteHeader } from "@/components/site-header"
|
||||
import { HeroSection } from "@/components/hero-section"
|
||||
import { TranslationCard } from "@/components/translation-card"
|
||||
import { SocialProofSection } from "@/components/social-proof-section"
|
||||
import { PricingSection } from "@/components/pricing-section"
|
||||
import { WaitlistSection } from "@/components/waitlist-section"
|
||||
import { FaqSection } from "@/components/faq-section"
|
||||
|
||||
export default function Home() {
|
||||
return (
|
||||
@@ -9,10 +13,39 @@ export default function Home() {
|
||||
<main className="flex flex-1 flex-col">
|
||||
<HeroSection />
|
||||
<TranslationCard />
|
||||
<SocialProofSection />
|
||||
<PricingSection />
|
||||
<WaitlistSection />
|
||||
<FaqSection />
|
||||
</main>
|
||||
<footer className="border-t border-border py-6 text-center text-xs text-muted-foreground">
|
||||
<div className="mx-auto max-w-5xl px-6">
|
||||
Office Translator · Enterprise-grade document translation · SOC 2 Compliant
|
||||
<footer className="border-t border-border py-8 text-center text-xs text-muted-foreground">
|
||||
<div className="mx-auto flex max-w-5xl flex-col items-center gap-2 px-6">
|
||||
<p>
|
||||
Office Translator · Enterprise-grade document translation
|
||||
</p>
|
||||
<div className="flex items-center gap-4">
|
||||
<a href="#pricing" className="hover:text-foreground">
|
||||
Pricing
|
||||
</a>
|
||||
<a href="#faq" className="hover:text-foreground">
|
||||
FAQ
|
||||
</a>
|
||||
<a href="#waitlist" className="hover:text-foreground">
|
||||
Waitlist
|
||||
</a>
|
||||
<a
|
||||
href="/docs"
|
||||
className="hover:text-foreground"
|
||||
target="_blank"
|
||||
rel="noreferrer"
|
||||
>
|
||||
API Docs
|
||||
</a>
|
||||
</div>
|
||||
<p className="text-[11px] text-muted-foreground/70">
|
||||
© {new Date().getFullYear()} Wordly.art. Documents are
|
||||
auto-deleted after 60 minutes and never used to train models.
|
||||
</p>
|
||||
</div>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
82
office-translator-landing-page/components/faq-section.tsx
Normal file
82
office-translator-landing-page/components/faq-section.tsx
Normal file
@@ -0,0 +1,82 @@
|
||||
"use client"
|
||||
|
||||
import {
|
||||
Accordion,
|
||||
AccordionContent,
|
||||
AccordionItem,
|
||||
AccordionTrigger,
|
||||
} from "@/components/ui/accordion"
|
||||
|
||||
const FAQS = [
|
||||
{
|
||||
q: "Does Office Translator really keep the formatting intact?",
|
||||
a: "Yes. Unlike generic translators that flatten your file into plain text, we translate in place: merged cells, formulas, fonts, borders, headers, footers, tables and slide layouts are preserved. You download a document that looks like it was written in the target language from the start.",
|
||||
},
|
||||
{
|
||||
q: "Which file formats are supported?",
|
||||
a: "Excel (.xlsx), Word (.docx), PowerPoint (.pptx) and PDF (.pdf). Scanned PDFs (image-only pages) are handled too — the text is recovered via OCR before translation. We also translate text embedded in images inside your documents using vision-capable models on Pro and Business plans.",
|
||||
},
|
||||
{
|
||||
q: "Which translation engines can I use?",
|
||||
a: "Up to seven providers depending on your plan: Google (free and fast), DeepL (best for business text), Google Cloud, and LLM engines (OpenRouter in eco & premium tiers — DeepSeek, Gemini, Claude —, OpenAI, Grok by xAI). On paid plans you pick the engine per document.",
|
||||
},
|
||||
{
|
||||
q: "How are my documents handled? Is my data safe?",
|
||||
a: "Files are processed on our servers and automatically deleted: uploads after 30 minutes, translated results within 2 hours (zero-retention mode). We never train models on your content, and documents are encrypted in transit (TLS 1.2+).",
|
||||
},
|
||||
{
|
||||
q: "What is a glossary and why would I need one?",
|
||||
a: "A glossary is a set of custom term pairs (e.g. a specific HVAC or legal term and its approved translation). The engine applies your terminology consistently across every document, which is essential for technical, legal and medical content.",
|
||||
},
|
||||
{
|
||||
q: "What happens when I hit my monthly document limit?",
|
||||
a: "Translation stops with a clear message and your usage is shown in the dashboard. You can upgrade your plan or buy extra credit packs (from €0.06 per page) without changing your subscription.",
|
||||
},
|
||||
{
|
||||
q: "Can I integrate Office Translator into my own tools?",
|
||||
a: "Yes. The Business plan includes API access with 10,000 calls per month and API keys with scoped permissions. Enterprise plans offer custom call volumes. The API mirrors the web workflow: submit a file, poll the job, download the result.",
|
||||
},
|
||||
{
|
||||
q: "Can I cancel my subscription?",
|
||||
a: "Anytime, from the dashboard or the Stripe billing portal. Your plan stays active until the end of the billing period, and you can download all your translated documents before that.",
|
||||
},
|
||||
{
|
||||
q: "Is there a free trial of paid plans?",
|
||||
a: "The Free plan already gives you 2 full documents per month with no card required, so you can validate quality on your own files before upgrading. Paid plans can be cancelled at any time.",
|
||||
},
|
||||
{
|
||||
q: "Do you offer team or agency options?",
|
||||
a: "The Business plan includes 5 team seats with shared usage. For larger teams, agencies and multi-country operations, our Enterprise plan provides custom seats, custom LLM routing and dedicated support.",
|
||||
},
|
||||
]
|
||||
|
||||
export function FaqSection() {
|
||||
return (
|
||||
<section id="faq" className="flex flex-col items-center gap-8 px-6 py-16 md:py-24">
|
||||
<div className="flex flex-col items-center gap-3 text-center">
|
||||
<h2 className="text-balance text-3xl font-bold tracking-tight text-foreground md:text-4xl">
|
||||
Frequently asked questions
|
||||
</h2>
|
||||
<p className="max-w-xl text-pretty text-sm leading-relaxed text-muted-foreground md:text-base">
|
||||
Everything you need to know about translating documents with zero
|
||||
formatting loss.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div className="w-full max-w-3xl">
|
||||
<Accordion type="single" collapsible className="w-full">
|
||||
{FAQS.map((faq, i) => (
|
||||
<AccordionItem key={i} value={`faq-${i}`}>
|
||||
<AccordionTrigger className="text-sm md:text-base">
|
||||
{faq.q}
|
||||
</AccordionTrigger>
|
||||
<AccordionContent className="text-sm leading-relaxed text-muted-foreground">
|
||||
{faq.a}
|
||||
</AccordionContent>
|
||||
</AccordionItem>
|
||||
))}
|
||||
</Accordion>
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
import { FileSpreadsheet, FileText, Presentation } from "lucide-react"
|
||||
import { FileSpreadsheet, FileText, FileType, Presentation } from "lucide-react"
|
||||
|
||||
export function HeroSection() {
|
||||
return (
|
||||
@@ -13,7 +13,7 @@ export function HeroSection() {
|
||||
</h1>
|
||||
|
||||
<p className="max-w-xl text-pretty text-base leading-relaxed text-muted-foreground md:text-lg">
|
||||
Upload your Excel, Word, or PowerPoint files and get accurate translations with zero formatting loss.
|
||||
Upload your Excel, Word, PowerPoint, or PDF files — including scanned PDFs, recovered via OCR — and get accurate translations with zero formatting loss.
|
||||
</p>
|
||||
|
||||
<div className="flex items-center gap-6 pt-2">
|
||||
@@ -29,6 +29,10 @@ export function HeroSection() {
|
||||
<Presentation className="size-4" />
|
||||
<span>.pptx</span>
|
||||
</div>
|
||||
<div className="flex items-center gap-2 text-sm text-muted-foreground">
|
||||
<FileType className="size-4" />
|
||||
<span>.pdf</span>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
|
||||
248
office-translator-landing-page/components/pricing-section.tsx
Normal file
248
office-translator-landing-page/components/pricing-section.tsx
Normal file
@@ -0,0 +1,248 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { Check, Crown, Zap, Building2, Sparkles, ArrowRight } from "lucide-react"
|
||||
import Link from "next/link"
|
||||
import { track } from "@vercel/analytics"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
type BillingCycle = "monthly" | "yearly"
|
||||
|
||||
interface Plan {
|
||||
id: string
|
||||
name: string
|
||||
description: string
|
||||
monthly: number
|
||||
yearly: number
|
||||
features: string[]
|
||||
cta: string
|
||||
badge?: string
|
||||
icon: typeof Check
|
||||
featured?: boolean
|
||||
}
|
||||
|
||||
const PLANS: Plan[] = [
|
||||
{
|
||||
id: "free",
|
||||
name: "Free",
|
||||
description: "Test the full workflow on real documents.",
|
||||
monthly: 0,
|
||||
yearly: 0,
|
||||
features: [
|
||||
"2 documents / month",
|
||||
"Up to 10 pages per document",
|
||||
"Up to 5 MB file size",
|
||||
"Google engine",
|
||||
"Watermarked output",
|
||||
],
|
||||
cta: "Start Free",
|
||||
icon: Zap,
|
||||
},
|
||||
{
|
||||
id: "starter",
|
||||
name: "Starter",
|
||||
description: "For freelancers and light professional use.",
|
||||
monthly: 9,
|
||||
yearly: 7.2,
|
||||
features: [
|
||||
"50 documents / month",
|
||||
"Up to 50 pages per document",
|
||||
"Up to 10 MB file size",
|
||||
"Google + DeepL engines",
|
||||
"No watermark",
|
||||
"Priority email support",
|
||||
],
|
||||
cta: "Choose Starter",
|
||||
icon: Sparkles,
|
||||
},
|
||||
{
|
||||
id: "pro",
|
||||
name: "Pro",
|
||||
description: "For teams that translate regularly.",
|
||||
monthly: 19,
|
||||
yearly: 15.2,
|
||||
features: [
|
||||
"200 documents / month",
|
||||
"Up to 200 pages per document",
|
||||
"Up to 25 MB file size",
|
||||
"All classic engines (Google Cloud, DeepL, OpenRouter)",
|
||||
"AI translation (context-aware LLM)",
|
||||
"Priority processing",
|
||||
"Custom glossaries",
|
||||
"No watermark",
|
||||
],
|
||||
cta: "Choose Pro",
|
||||
badge: "Most Popular",
|
||||
icon: Crown,
|
||||
featured: true,
|
||||
},
|
||||
{
|
||||
id: "business",
|
||||
name: "Business",
|
||||
description: "For agencies and international operations.",
|
||||
monthly: 49,
|
||||
yearly: 39.2,
|
||||
features: [
|
||||
"1,000 documents / month",
|
||||
"Up to 500 pages per document",
|
||||
"Up to 50 MB file size",
|
||||
"All engines incl. premium LLMs (Claude, OpenAI)",
|
||||
"API access (10,000 calls / month)",
|
||||
"5 team seats",
|
||||
"Priority processing",
|
||||
"Dedicated support",
|
||||
],
|
||||
cta: "Choose Business",
|
||||
icon: Building2,
|
||||
},
|
||||
]
|
||||
|
||||
export function PricingSection() {
|
||||
const [cycle, setCycle] = useState<BillingCycle>("monthly")
|
||||
|
||||
return (
|
||||
<section id="pricing" className="flex flex-col items-center gap-8 px-6 py-16 md:py-24">
|
||||
<div className="flex flex-col items-center gap-3 text-center">
|
||||
<h2 className="text-balance text-3xl font-bold tracking-tight text-foreground md:text-4xl">
|
||||
Simple pricing that scales with you
|
||||
</h2>
|
||||
<p className="max-w-xl text-pretty text-sm leading-relaxed text-muted-foreground md:text-base">
|
||||
Start free, upgrade when you need more volume, faster engines or API
|
||||
access. Cancel anytime.
|
||||
</p>
|
||||
|
||||
<div className="mt-2 flex items-center rounded-lg border border-border bg-muted p-1">
|
||||
<button
|
||||
type="button"
|
||||
className={cn(
|
||||
"rounded-md px-4 py-1.5 text-sm font-medium transition-all",
|
||||
cycle === "monthly"
|
||||
? "bg-card text-foreground shadow-sm"
|
||||
: "text-muted-foreground hover:text-foreground"
|
||||
)}
|
||||
onClick={() => setCycle("monthly")}
|
||||
>
|
||||
Monthly
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
className={cn(
|
||||
"rounded-md px-4 py-1.5 text-sm font-medium transition-all",
|
||||
cycle === "yearly"
|
||||
? "bg-card text-foreground shadow-sm"
|
||||
: "text-muted-foreground hover:text-foreground"
|
||||
)}
|
||||
onClick={() => setCycle("yearly")}
|
||||
>
|
||||
Yearly
|
||||
<span className="ml-1.5 rounded-full bg-success/15 px-1.5 py-0.5 text-[10px] font-semibold text-success">
|
||||
-20%
|
||||
</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="grid w-full max-w-6xl grid-cols-1 gap-5 sm:grid-cols-2 lg:grid-cols-4">
|
||||
{PLANS.map((plan) => (
|
||||
<div
|
||||
key={plan.id}
|
||||
className={cn(
|
||||
"flex flex-col rounded-xl border bg-card p-6 shadow-sm transition-shadow hover:shadow-md",
|
||||
plan.featured
|
||||
? "border-primary shadow-lg ring-1 ring-primary/20"
|
||||
: "border-border"
|
||||
)}
|
||||
>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<div
|
||||
className={cn(
|
||||
"flex size-8 items-center justify-center rounded-lg",
|
||||
plan.featured ? "bg-primary" : "bg-secondary"
|
||||
)}
|
||||
>
|
||||
<plan.icon
|
||||
className={cn(
|
||||
"size-4",
|
||||
plan.featured ? "text-primary-foreground" : "text-foreground"
|
||||
)}
|
||||
/>
|
||||
</div>
|
||||
<span className="text-base font-semibold text-foreground">
|
||||
{plan.name}
|
||||
</span>
|
||||
</div>
|
||||
{plan.badge && (
|
||||
<span className="rounded-full bg-primary px-2.5 py-0.5 text-[10px] font-semibold uppercase tracking-wide text-primary-foreground">
|
||||
{plan.badge}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<p className="mt-3 text-xs leading-relaxed text-muted-foreground">
|
||||
{plan.description}
|
||||
</p>
|
||||
|
||||
<div className="mt-4 flex items-baseline gap-1">
|
||||
<span className="text-4xl font-bold tracking-tight text-foreground">
|
||||
{cycle === "yearly" && plan.monthly > 0
|
||||
? `€${plan.yearly.toFixed(2).replace(/\.00$/, "")}`
|
||||
: plan.monthly === 0
|
||||
? "€0"
|
||||
: `€${plan.monthly}`}
|
||||
</span>
|
||||
<span className="text-sm text-muted-foreground">
|
||||
{cycle === "yearly" && plan.monthly > 0
|
||||
? "/mo billed yearly"
|
||||
: plan.monthly === 0
|
||||
? "forever"
|
||||
: "/mo"}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<ul className="mt-5 flex flex-1 flex-col gap-2.5">
|
||||
{plan.features.map((feature) => (
|
||||
<li key={feature} className="flex items-start gap-2 text-xs text-foreground/90">
|
||||
<Check className="mt-0.5 size-3.5 shrink-0 text-success" />
|
||||
<span>{feature}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
|
||||
<Button
|
||||
asChild
|
||||
variant={plan.featured ? "default" : "outline"}
|
||||
size="sm"
|
||||
className="mt-6 w-full"
|
||||
>
|
||||
<Link
|
||||
href="/dashboard"
|
||||
onClick={() => track("pricing_cta_click", { plan: plan.id })}
|
||||
>
|
||||
{plan.cta}
|
||||
<ArrowRight className="size-3.5" />
|
||||
</Link>
|
||||
</Button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="flex w-full max-w-6xl flex-col items-center justify-between gap-3 rounded-xl border border-border bg-muted/40 px-6 py-4 text-center sm:flex-row sm:text-left">
|
||||
<div>
|
||||
<p className="text-sm font-semibold text-foreground">
|
||||
Enterprise: custom volume, custom LLM routing, SSO and dedicated infrastructure.
|
||||
</p>
|
||||
<p className="mt-0.5 text-xs text-muted-foreground">
|
||||
Volume pricing, custom SLAs and on-prem options.
|
||||
</p>
|
||||
</div>
|
||||
<Button asChild variant="outline" size="sm">
|
||||
<a href="mailto:sales@wordly.art?subject=Enterprise%20inquiry">
|
||||
Contact Sales
|
||||
</a>
|
||||
</Button>
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
@@ -18,12 +18,26 @@ export function SiteHeader() {
|
||||
</div>
|
||||
|
||||
<nav className="hidden items-center gap-1 md:flex">
|
||||
<Button variant="ghost" size="sm" className="text-muted-foreground hover:text-foreground">
|
||||
<a
|
||||
href="#pricing"
|
||||
className="rounded-md px-3 py-1.5 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground"
|
||||
>
|
||||
Pricing
|
||||
</Button>
|
||||
<Button variant="ghost" size="sm" className="text-muted-foreground hover:text-foreground">
|
||||
</a>
|
||||
<a
|
||||
href="#faq"
|
||||
className="rounded-md px-3 py-1.5 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground"
|
||||
>
|
||||
FAQ
|
||||
</a>
|
||||
<a
|
||||
href="/docs"
|
||||
target="_blank"
|
||||
rel="noreferrer"
|
||||
className="rounded-md px-3 py-1.5 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground"
|
||||
>
|
||||
API Docs
|
||||
</Button>
|
||||
</a>
|
||||
<div className="mx-2 h-4 w-px bg-border" />
|
||||
<Button variant="outline" size="sm" asChild>
|
||||
<Link href="/dashboard">Login</Link>
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
import {
|
||||
Languages,
|
||||
Cpu,
|
||||
LayoutTemplate,
|
||||
ShieldCheck,
|
||||
Star,
|
||||
} from "lucide-react"
|
||||
|
||||
const STATS = [
|
||||
{
|
||||
icon: Languages,
|
||||
value: "60+",
|
||||
label: "Languages supported",
|
||||
},
|
||||
{
|
||||
icon: Cpu,
|
||||
value: "7",
|
||||
label: "Translation engines",
|
||||
},
|
||||
{
|
||||
icon: LayoutTemplate,
|
||||
value: "100%",
|
||||
label: "Formatting preserved",
|
||||
},
|
||||
{
|
||||
icon: ShieldCheck,
|
||||
value: "≤ 2 h",
|
||||
label: "Max file retention",
|
||||
},
|
||||
]
|
||||
|
||||
const TESTIMONIALS = [
|
||||
{
|
||||
quote:
|
||||
"We translated a 40-page Excel pricing matrix in minutes and every formula and merged cell survived intact. It used to take our translators two days.",
|
||||
name: "M. Laurent",
|
||||
role: "Operations Lead, logistics (FR → EN)",
|
||||
},
|
||||
{
|
||||
quote:
|
||||
"The custom glossary is the killer feature for us. Our HVAC terminology is now consistent across every document we ship to clients.",
|
||||
name: "S. Chen",
|
||||
role: "Technical writer, engineering consultancy",
|
||||
},
|
||||
{
|
||||
quote:
|
||||
"Being able to pick the engine per document — cheap for drafts, premium LLM for client deliverables — fits how our agency actually works.",
|
||||
name: "A. Dubois",
|
||||
role: "Translation agency owner",
|
||||
},
|
||||
]
|
||||
|
||||
export function SocialProofSection() {
|
||||
return (
|
||||
<section className="flex flex-col items-center gap-10 px-6 py-16 md:py-20">
|
||||
<div className="grid w-full max-w-4xl grid-cols-2 gap-4 md:grid-cols-4">
|
||||
{STATS.map((stat) => (
|
||||
<div
|
||||
key={stat.label}
|
||||
className="flex flex-col items-center gap-2 rounded-xl border border-border bg-card p-5 text-center shadow-sm"
|
||||
>
|
||||
<div className="flex size-9 items-center justify-center rounded-lg bg-secondary">
|
||||
<stat.icon className="size-4.5 text-foreground" />
|
||||
</div>
|
||||
<span className="text-2xl font-bold tracking-tight text-foreground">
|
||||
{stat.value}
|
||||
</span>
|
||||
<span className="text-xs text-muted-foreground">{stat.label}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="flex w-full max-w-5xl flex-col items-center gap-6">
|
||||
<div className="flex flex-col items-center gap-2 text-center">
|
||||
<div className="flex items-center gap-1 text-amber-500">
|
||||
{Array.from({ length: 5 }).map((_, i) => (
|
||||
<Star key={i} className="size-4 fill-current" />
|
||||
))}
|
||||
</div>
|
||||
<h2 className="text-balance text-2xl font-bold tracking-tight text-foreground md:text-3xl">
|
||||
Teams that translate, ship faster
|
||||
</h2>
|
||||
<p className="max-w-xl text-pretty text-sm text-muted-foreground">
|
||||
From solo consultants to international operations, Office Translator
|
||||
keeps the format, the formulas and the meaning.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div className="grid w-full grid-cols-1 gap-4 md:grid-cols-3">
|
||||
{TESTIMONIALS.map((t) => (
|
||||
<figure
|
||||
key={t.name}
|
||||
className="flex flex-col gap-3 rounded-xl border border-border bg-card p-5 shadow-sm"
|
||||
>
|
||||
<div className="flex items-center gap-0.5 text-amber-500">
|
||||
{Array.from({ length: 5 }).map((_, i) => (
|
||||
<Star key={i} className="size-3 fill-current" />
|
||||
))}
|
||||
</div>
|
||||
<blockquote className="flex-1 text-sm leading-relaxed text-foreground/90">
|
||||
“{t.quote}”
|
||||
</blockquote>
|
||||
<figcaption className="flex items-center gap-2 text-xs">
|
||||
<span className="font-semibold text-foreground">{t.name}</span>
|
||||
<span className="text-muted-foreground">{t.role}</span>
|
||||
</figcaption>
|
||||
</figure>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
Loader2,
|
||||
} from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { track } from "@vercel/analytics"
|
||||
import {
|
||||
Card,
|
||||
CardContent,
|
||||
@@ -59,6 +60,7 @@ export function TranslationCard() {
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
"application/pdf",
|
||||
]
|
||||
|
||||
const handleDragOver = useCallback((e: React.DragEvent) => {
|
||||
@@ -96,6 +98,11 @@ export function TranslationCard() {
|
||||
|
||||
const handleTranslate = useCallback(() => {
|
||||
if (!file || !sourceLang || !targetLang) return
|
||||
track("translation_started", {
|
||||
engine,
|
||||
source: sourceLang,
|
||||
target: targetLang,
|
||||
})
|
||||
setIsProcessing(true)
|
||||
setProgress(0)
|
||||
|
||||
@@ -181,7 +188,7 @@ export function TranslationCard() {
|
||||
</div>
|
||||
<div className="flex flex-col items-center gap-1">
|
||||
<p className="text-sm font-medium text-foreground">
|
||||
Drag & drop your .xlsx, .docx, or .pptx file here
|
||||
Drag & drop your .xlsx, .docx, .pptx, or .pdf file here
|
||||
</p>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
or click to browse
|
||||
|
||||
141
office-translator-landing-page/components/waitlist-section.tsx
Normal file
141
office-translator-landing-page/components/waitlist-section.tsx
Normal file
@@ -0,0 +1,141 @@
|
||||
"use client"
|
||||
|
||||
import { useState, useCallback } from "react"
|
||||
import { Mail, Loader2, CheckCircle2, ArrowRight } from "lucide-react"
|
||||
import { track } from "@vercel/analytics"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
|
||||
type Status = "idle" | "loading" | "success" | "error"
|
||||
|
||||
// The backend is proxied under the same origin via the rewrites in
|
||||
// next.config.mjs, so the form never needs the backend origin (no CORS).
|
||||
const WAITLIST_ENDPOINT = "/api/v1/waitlist"
|
||||
|
||||
const INTERESTS = [
|
||||
"I translate documents regularly",
|
||||
"I run a translation agency",
|
||||
"I manage a multilingual team",
|
||||
"I'm a developer (API integration)",
|
||||
"Just exploring",
|
||||
]
|
||||
|
||||
export function WaitlistSection() {
|
||||
const [email, setEmail] = useState("")
|
||||
const [interest, setInterest] = useState(INTERESTS[0])
|
||||
const [status, setStatus] = useState<Status>("idle")
|
||||
const [message, setMessage] = useState("")
|
||||
|
||||
const submit = useCallback(
|
||||
async (e: React.FormEvent) => {
|
||||
e.preventDefault()
|
||||
if (!email || status === "loading") return
|
||||
setStatus("loading")
|
||||
setMessage("")
|
||||
try {
|
||||
const res = await fetch(`/api/v1/waitlist`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ email, interest }),
|
||||
})
|
||||
const data = await res.json().catch(() => ({}))
|
||||
if (res.ok) {
|
||||
setStatus("success")
|
||||
track("waitlist_joined", { interest })
|
||||
setMessage(
|
||||
data?.data?.status === "already_joined"
|
||||
? "You are already on the list. We will email you at launch."
|
||||
: "You are on the list. We will email you at launch."
|
||||
)
|
||||
setEmail("")
|
||||
} else {
|
||||
setStatus("error")
|
||||
track("waitlist_error", { status: res.status })
|
||||
setMessage(data?.message ?? "Something went wrong. Please try again.")
|
||||
}
|
||||
} catch {
|
||||
setStatus("error")
|
||||
setMessage("Network error. Please try again later.")
|
||||
}
|
||||
},
|
||||
[email, interest, status]
|
||||
)
|
||||
|
||||
return (
|
||||
<section
|
||||
id="waitlist"
|
||||
className="flex flex-col items-center gap-6 px-6 py-16 md:py-20"
|
||||
>
|
||||
<div className="flex w-full max-w-2xl flex-col items-center gap-4 rounded-2xl border border-border bg-card px-6 py-10 text-center shadow-lg md:px-12">
|
||||
<div className="flex size-12 items-center justify-center rounded-xl bg-primary">
|
||||
<Mail className="size-6 text-primary-foreground" />
|
||||
</div>
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<h2 className="text-balance text-2xl font-bold tracking-tight text-foreground md:text-3xl">
|
||||
Be first in line at launch
|
||||
</h2>
|
||||
<p className="text-pretty text-sm text-muted-foreground">
|
||||
Join the waitlist and get early access, launch-day pricing and the
|
||||
“keep your format” playbook. No spam, one email at launch.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{status === "success" ? (
|
||||
<div className="flex w-full max-w-md flex-col items-center gap-2 rounded-lg border border-success/40 bg-success/10 px-5 py-4">
|
||||
<CheckCircle2 className="size-5 text-success" />
|
||||
<p className="text-sm font-medium text-foreground">{message}</p>
|
||||
</div>
|
||||
) : (
|
||||
<form onSubmit={submit} className="flex w-full max-w-md flex-col gap-3">
|
||||
<div className="flex flex-col gap-3 sm:flex-row">
|
||||
<Input
|
||||
type="email"
|
||||
required
|
||||
value={email}
|
||||
onChange={(e) => setEmail(e.target.value)}
|
||||
placeholder="you@company.com"
|
||||
aria-label="Email address"
|
||||
className="flex-1"
|
||||
disabled={status === "loading"}
|
||||
/>
|
||||
<Button
|
||||
type="submit"
|
||||
size="lg"
|
||||
disabled={status === "loading" || !email}
|
||||
>
|
||||
{status === "loading" ? (
|
||||
<>
|
||||
<Loader2 className="size-4 animate-spin" /> Joining…
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
Join Waitlist <ArrowRight className="size-4" />
|
||||
</>
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
<select
|
||||
value={interest}
|
||||
onChange={(e) => setInterest(e.target.value)}
|
||||
aria-label="What best describes you?"
|
||||
className="w-full rounded-md border border-input bg-background px-3 py-2 text-sm text-foreground outline-none focus:ring-2 focus:ring-ring/50"
|
||||
>
|
||||
{INTERESTS.map((opt) => (
|
||||
<option key={opt} value={opt}>
|
||||
{opt}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
{status === "error" && (
|
||||
<p className="text-xs text-destructive">{message}</p>
|
||||
)}
|
||||
<p className="text-[11px] text-muted-foreground">
|
||||
We only use your address to notify you at launch. Unsubscribe any
|
||||
time.
|
||||
</p>
|
||||
</form>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
6
office-translator-landing-page/lib/utils.ts
Normal file
6
office-translator-landing-page/lib/utils.ts
Normal file
@@ -0,0 +1,6 @@
|
||||
import { clsx, type ClassValue } from "clsx"
|
||||
import { twMerge } from "tailwind-merge"
|
||||
|
||||
export function cn(...inputs: ClassValue[]) {
|
||||
return twMerge(clsx(inputs))
|
||||
}
|
||||
6
office-translator-landing-page/next-env.d.ts
vendored
Normal file
6
office-translator-landing-page/next-env.d.ts
vendored
Normal file
@@ -0,0 +1,6 @@
|
||||
/// <reference types="next" />
|
||||
/// <reference types="next/image-types/global" />
|
||||
import "./.next/types/routes.d.ts";
|
||||
|
||||
// NOTE: This file should not be edited
|
||||
// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.
|
||||
@@ -1,4 +1,6 @@
|
||||
/** @type {import('next').NextConfig} */
|
||||
const API_BASE = process.env.NEXT_PUBLIC_API_BASE_URL || "http://localhost:8000"
|
||||
|
||||
const nextConfig = {
|
||||
typescript: {
|
||||
ignoreBuildErrors: true,
|
||||
@@ -6,6 +8,26 @@ const nextConfig = {
|
||||
images: {
|
||||
unoptimized: true,
|
||||
},
|
||||
async rewrites() {
|
||||
return [
|
||||
{
|
||||
source: "/api/v1/waitlist/:path*",
|
||||
destination: `${API_BASE}/api/v1/waitlist/:path*`,
|
||||
},
|
||||
{
|
||||
source: "/api/v1/waitlist",
|
||||
destination: `${API_BASE}/api/v1/waitlist`,
|
||||
},
|
||||
{
|
||||
source: "/docs/:path*",
|
||||
destination: `${API_BASE}/docs/:path*`,
|
||||
},
|
||||
{
|
||||
source: "/openapi.json",
|
||||
destination: `${API_BASE}/openapi.json`,
|
||||
},
|
||||
]
|
||||
},
|
||||
}
|
||||
|
||||
export default nextConfig
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"lib": ["dom", "dom.iterable", "esnext"],
|
||||
"lib": [
|
||||
"dom",
|
||||
"dom.iterable",
|
||||
"esnext"
|
||||
],
|
||||
"allowJs": true,
|
||||
"target": "ES6",
|
||||
"skipLibCheck": true,
|
||||
@@ -11,7 +15,7 @@
|
||||
"moduleResolution": "bundler",
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"jsx": "preserve",
|
||||
"jsx": "react-jsx",
|
||||
"incremental": true,
|
||||
"plugins": [
|
||||
{
|
||||
@@ -19,9 +23,19 @@
|
||||
}
|
||||
],
|
||||
"paths": {
|
||||
"@/*": ["./*"]
|
||||
"@/*": [
|
||||
"./*"
|
||||
]
|
||||
}
|
||||
},
|
||||
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
|
||||
"exclude": ["node_modules"]
|
||||
"include": [
|
||||
"next-env.d.ts",
|
||||
"**/*.ts",
|
||||
"**/*.tsx",
|
||||
".next/types/**/*.ts",
|
||||
".next/dev/types/**/*.ts"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
fastapi==0.109.0
|
||||
fastapi==0.109.1
|
||||
uvicorn[standard]==0.27.0
|
||||
python-multipart==0.0.9
|
||||
python-multipart==0.0.20
|
||||
openpyxl==3.1.2
|
||||
python-docx==1.1.0
|
||||
python-pptx==0.6.23
|
||||
@@ -13,7 +13,7 @@ python-dotenv==1.0.0
|
||||
pydantic==2.5.3
|
||||
pydantic[email]==2.5.3
|
||||
aiofiles==23.2.1
|
||||
httpx>=0.27.0
|
||||
httpx>=0.27.0,<0.28 # 0.28 removed Client(app=...) — breaks starlette TestClient
|
||||
Pillow==10.2.0
|
||||
matplotlib==3.8.2
|
||||
pandas==2.1.4
|
||||
|
||||
@@ -285,6 +285,67 @@ async def get_admin_dashboard(admin_id: str = Depends(require_admin)):
|
||||
"last_check": None,
|
||||
}
|
||||
|
||||
# OCR status (scanned PDFs): configured via admin settings or env key.
|
||||
try:
|
||||
from services.mistral_ocr import MistralOCRClient
|
||||
|
||||
settings = load_settings()
|
||||
mistral_key = (settings.mistral.api_key or "").strip() or os.getenv(
|
||||
"MISTRAL_API_KEY", ""
|
||||
).strip()
|
||||
ocr_configured = bool(mistral_key) and (
|
||||
settings.mistral.enabled
|
||||
# enabled=False in a saved-but-untouched settings file must not
|
||||
# hide an env-configured OCR (same rule as the translate route)
|
||||
or not (settings.mistral.api_key or "").strip()
|
||||
)
|
||||
providers_status["mistral_ocr"] = {
|
||||
"name": "mistral_ocr",
|
||||
"available": ocr_configured,
|
||||
"error": None
|
||||
if ocr_configured
|
||||
else "Non configuré — les PDF scannés seront refusés (MISTRAL_API_KEY)",
|
||||
"last_check": None,
|
||||
# No live call here: health is derived from configuration.
|
||||
"config_only": True,
|
||||
}
|
||||
except Exception as e:
|
||||
providers_status["mistral_ocr"] = {
|
||||
"name": "mistral_ocr",
|
||||
"available": False,
|
||||
"error": str(e)[:100],
|
||||
"last_check": None,
|
||||
}
|
||||
|
||||
# LLM/translation engines configured via admin settings (key presence
|
||||
# only — no live call, keys are never exposed).
|
||||
try:
|
||||
settings = load_settings()
|
||||
|
||||
def _engine_status(section_name: str, env_var: str, label: str):
|
||||
section = getattr(settings, section_name, None)
|
||||
key_set = bool(
|
||||
((getattr(section, "api_key", None) or "").strip())
|
||||
or os.getenv(env_var, "").strip()
|
||||
)
|
||||
providers_status[section_name] = {
|
||||
"name": section_name,
|
||||
"available": key_set,
|
||||
"error": None if key_set else f"Clé absente ({env_var})",
|
||||
"last_check": None,
|
||||
"config_only": True,
|
||||
"label": label,
|
||||
}
|
||||
|
||||
_engine_status("deepl", "DEEPL_API_KEY", "DeepL")
|
||||
_engine_status("openrouter", "OPENROUTER_API_KEY", "Traduction IA Éco")
|
||||
_engine_status("openrouter_premium", "OPENROUTER_API_KEY", "Traduction IA Premium")
|
||||
_engine_status("openai", "OPENAI_API_KEY", "OpenAI")
|
||||
_engine_status("zai", "ZAI_API_KEY", "Grok (xAI)")
|
||||
_engine_status("google_cloud", "GOOGLE_CLOUD_API_KEY", "Google Cloud")
|
||||
except Exception as e:
|
||||
logger.warning(f"admin dashboard engines status failed: {e}")
|
||||
|
||||
return {
|
||||
"timestamp": health_status.get("timestamp"),
|
||||
"status": health_status.get("status"),
|
||||
@@ -641,13 +702,24 @@ async def update_default_provider(
|
||||
provider: str = Form(...),
|
||||
admin_id: str = Depends(require_admin),
|
||||
):
|
||||
"""Update the default translation provider"""
|
||||
"""Update the default translation provider.
|
||||
|
||||
The allowed list mirrors the providers actually wired in the translate
|
||||
route (and declared in SettingsConfig) so the admin can set any of them as
|
||||
the default — previously google_cloud, openrouter_premium, deepseek and
|
||||
minimax were wrongly rejected here even though they are fully supported.
|
||||
"""
|
||||
valid_providers = [
|
||||
"google",
|
||||
"google_cloud",
|
||||
"deepl",
|
||||
"openai",
|
||||
"openrouter",
|
||||
"openrouter_premium",
|
||||
"deepseek",
|
||||
"minimax",
|
||||
"zai",
|
||||
# Mode aliases resolved at translation time.
|
||||
"classic",
|
||||
"llm",
|
||||
]
|
||||
@@ -852,6 +924,7 @@ class SettingsConfig(BaseModel):
|
||||
deepseek: ProviderSettings = ProviderSettings()
|
||||
minimax: ProviderSettings = ProviderSettings()
|
||||
zai: ProviderSettings = ProviderSettings()
|
||||
mistral: ProviderSettings = ProviderSettings() # OCR Mistral (PDF scannés)
|
||||
smtp: SmtpSettings = SmtpSettings()
|
||||
fallback_chain: str = "google,google_cloud,deepl,openrouter,openrouter_premium,openai,deepseek,zai"
|
||||
fallback_chain_classic: str = "google,google_cloud,deepl"
|
||||
@@ -924,6 +997,7 @@ async def get_settings(admin_id: str = Depends(require_admin)):
|
||||
payload["zai"] = _merge_env(settings.zai, key_env="ZAI_API_KEY", model_env="ZAI_MODEL", url_env="ZAI_BASE_URL", default_model="grok-2-1212", default_url="https://api.x.ai/v1")
|
||||
|
||||
payload["google_cloud"] = _merge_env(settings.google_cloud, key_env="GOOGLE_CLOUD_API_KEY")
|
||||
payload["mistral"] = _merge_env(settings.mistral, key_env="MISTRAL_API_KEY", model_env="MISTRAL_OCR_MODEL", default_model="mistral-ocr-latest")
|
||||
|
||||
# SMTP: merge from env vars, but never expose password
|
||||
smtp_data = settings.smtp.model_dump()
|
||||
@@ -958,6 +1032,7 @@ async def get_settings(admin_id: str = Depends(require_admin)):
|
||||
"zai": bool(os.getenv("ZAI_API_KEY", "").strip()),
|
||||
|
||||
"google_cloud": bool(os.getenv("GOOGLE_CLOUD_API_KEY", "").strip()),
|
||||
"mistral": bool(os.getenv("MISTRAL_API_KEY", "").strip()),
|
||||
"smtp": bool(os.getenv("SMTP_HOST", "").strip()),
|
||||
}
|
||||
return JSONResponse(
|
||||
@@ -1029,6 +1104,49 @@ async def test_provider(
|
||||
status_code=200, content={"available": True, "test_result": result}
|
||||
)
|
||||
|
||||
elif provider == "mistral":
|
||||
api_key = _key(provider_config.api_key, "MISTRAL_API_KEY")
|
||||
if not api_key:
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={
|
||||
"available": False,
|
||||
"error": "Aucune clé API Mistral trouvée (JSON ou .env MISTRAL_API_KEY).",
|
||||
},
|
||||
)
|
||||
import requests as _requests
|
||||
|
||||
resp = _requests.get(
|
||||
"https://api.mistral.ai/v1/models",
|
||||
headers={"Authorization": f"Bearer {api_key}"},
|
||||
timeout=10,
|
||||
)
|
||||
if resp.ok:
|
||||
n_models = len(resp.json().get("data", []))
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"available": True,
|
||||
"test_result": f"Clé valide — {n_models} modèles accessibles",
|
||||
},
|
||||
)
|
||||
elif resp.status_code in (401, 403):
|
||||
return JSONResponse(
|
||||
status_code=resp.status_code,
|
||||
content={
|
||||
"available": False,
|
||||
"error": f"Clé API Mistral invalide (HTTP {resp.status_code}).",
|
||||
},
|
||||
)
|
||||
else:
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={
|
||||
"available": False,
|
||||
"error": f"Erreur Mistral HTTP {resp.status_code}: {resp.text[:200]}",
|
||||
},
|
||||
)
|
||||
|
||||
elif provider == "google_cloud":
|
||||
api_key = _key(provider_config.api_key, "GOOGLE_CLOUD_API_KEY")
|
||||
if not api_key:
|
||||
|
||||
@@ -15,6 +15,7 @@ from routes.admin_routes import router as admin_router
|
||||
from routes.legacy_routes import router as legacy_router
|
||||
from routes.glossary_routes import router as glossary_router
|
||||
from routes.prompt_routes import router as prompt_router
|
||||
from routes.waitlist_routes import router as waitlist_router
|
||||
|
||||
router.include_router(translate_router, tags=["Translation"])
|
||||
router.include_router(auth_router, tags=["Authentication"])
|
||||
@@ -23,3 +24,4 @@ router.include_router(admin_router, tags=["Admin"])
|
||||
router.include_router(legacy_router, tags=["Legacy"])
|
||||
router.include_router(glossary_router, tags=["Glossaries"])
|
||||
router.include_router(prompt_router, tags=["Prompts"])
|
||||
router.include_router(waitlist_router, tags=["Waitlist"])
|
||||
|
||||
@@ -14,6 +14,7 @@ from fastapi.responses import FileResponse, JSONResponse
|
||||
|
||||
from config import config
|
||||
from utils import file_handler
|
||||
from utils.file_handler import validate_zip_safety
|
||||
from middleware.api_key_auth import get_authenticated_user
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -32,7 +33,9 @@ def _resolve_model(
|
||||
|
||||
|
||||
@router.get("/providers/available")
|
||||
async def get_available_providers():
|
||||
async def get_available_providers(
|
||||
current_user: Optional[Any] = Depends(get_authenticated_user),
|
||||
):
|
||||
"""
|
||||
Return every provider that is enabled — checking BOTH the admin settings JSON
|
||||
AND environment variables (env vars act as a fallback / override).
|
||||
@@ -42,8 +45,12 @@ async def get_available_providers():
|
||||
- Ollama is only shown in DEV mode (APP_ENV=development or SHOW_OLLAMA=true).
|
||||
- openrouter → shown as "Traduction IA Essentielle" (cheap models).
|
||||
- openrouter_premium → shown as "Traduction IA Premium" (premium models).
|
||||
- Filtered to the engines included in the caller's plan
|
||||
(PLANS[plan]["providers"]) so the UI never offers an engine the
|
||||
translate endpoint would reject.
|
||||
"""
|
||||
from routes.admin_routes import load_settings
|
||||
from models.subscription import PlanType, PLANS
|
||||
|
||||
settings = load_settings()
|
||||
is_dev = os.getenv("APP_ENV", "production").lower() == "development"
|
||||
@@ -189,6 +196,16 @@ async def get_available_providers():
|
||||
|
||||
|
||||
|
||||
# Filter to the engines included in the caller's plan (anonymous → Free).
|
||||
user_plan = PlanType.FREE
|
||||
if current_user is not None:
|
||||
try:
|
||||
user_plan = PlanType(getattr(current_user, "plan", PlanType.FREE))
|
||||
except ValueError:
|
||||
user_plan = PlanType.FREE
|
||||
allowed = set((PLANS.get(user_plan) or PLANS[PlanType.FREE]).get("providers", []))
|
||||
available = [p for p in available if p.get("id") in allowed]
|
||||
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
headers={"Cache-Control": "no-cache, no-store, must-revalidate"},
|
||||
@@ -198,49 +215,36 @@ async def get_available_providers():
|
||||
|
||||
@router.get("/languages")
|
||||
async def get_supported_languages():
|
||||
"""Get list of supported language codes, ordered by internet popularity"""
|
||||
"""Get list of supported language codes, ordered by internet popularity.
|
||||
|
||||
Served from LanguageValidator (single source of truth): the 35 most
|
||||
requested languages first, then the rest of the validated ISO 639-1 set
|
||||
alphabetically. "auto" is excluded — it is a source-only value handled
|
||||
by the source_lang parameter.
|
||||
"""
|
||||
from middleware.validation import LanguageValidator
|
||||
|
||||
popular_order = [
|
||||
# Top 5 — dominant on the internet
|
||||
"en", "es", "de", "fr", "ja",
|
||||
# Top 6-15
|
||||
"pt", "ru", "it", "zh-CN", "zh-TW", "pl", "nl", "tr", "ko", "ar",
|
||||
# Top 16-25
|
||||
"fa", "vi", "id", "uk", "sv", "cs", "el", "he", "hi", "ro",
|
||||
# Next most requested
|
||||
"da", "fi", "no", "hu", "th", "sk", "bg", "hr", "ca", "ms", "zh",
|
||||
]
|
||||
|
||||
names = LanguageValidator.LANGUAGE_NAMES
|
||||
supported = [
|
||||
c for c in LanguageValidator.SUPPORTED_LANGUAGES if c != "auto"
|
||||
]
|
||||
ordered = [c for c in popular_order if c in supported]
|
||||
ordered += sorted(c for c in supported if c not in ordered)
|
||||
|
||||
return {
|
||||
"supported_languages": {
|
||||
# Top 5 — dominant on the internet
|
||||
"en": "English",
|
||||
"es": "Spanish",
|
||||
"de": "German",
|
||||
"fr": "French",
|
||||
"ja": "Japanese",
|
||||
# Top 6-15
|
||||
"pt": "Portuguese",
|
||||
"ru": "Russian",
|
||||
"it": "Italian",
|
||||
"zh-CN": "Chinese (Simplified)",
|
||||
"zh-TW": "Chinese (Traditional)",
|
||||
"pl": "Polish",
|
||||
"nl": "Dutch",
|
||||
"tr": "Turkish",
|
||||
"ko": "Korean",
|
||||
"ar": "Arabic",
|
||||
# Top 16-25
|
||||
"fa": "Persian (Farsi)",
|
||||
"vi": "Vietnamese",
|
||||
"id": "Indonesian",
|
||||
"uk": "Ukrainian",
|
||||
"sv": "Swedish",
|
||||
"cs": "Czech",
|
||||
"el": "Greek",
|
||||
"he": "Hebrew",
|
||||
"hi": "Hindi",
|
||||
"ro": "Romanian",
|
||||
# Others
|
||||
"da": "Danish",
|
||||
"fi": "Finnish",
|
||||
"no": "Norwegian",
|
||||
"hu": "Hungarian",
|
||||
"th": "Thai",
|
||||
"sk": "Slovak",
|
||||
"bg": "Bulgarian",
|
||||
"hr": "Croatian",
|
||||
"ca": "Catalan",
|
||||
"ms": "Malay",
|
||||
},
|
||||
"supported_languages": {code: names.get(code, code.upper()) for code in ordered},
|
||||
"count": len(ordered),
|
||||
"note": "Supported languages may vary depending on the translation service configured",
|
||||
}
|
||||
|
||||
@@ -276,6 +280,10 @@ async def translate_batch_documents(
|
||||
|
||||
file_handler.save_upload_file(file, input_path)
|
||||
|
||||
# Zip bomb protection: Office files are ZIP archives
|
||||
if file_extension != ".pdf":
|
||||
validate_zip_safety(input_path)
|
||||
|
||||
if file_extension == ".xlsx":
|
||||
excel_translator.translate_file(
|
||||
input_path, output_path, target_language, source_language
|
||||
@@ -300,6 +308,18 @@ async def translate_batch_documents(
|
||||
}
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
file_handler.cleanup_file(input_path)
|
||||
logger.warning(f"Rejected unsafe or invalid archive: {file.filename}: {e}")
|
||||
results.append(
|
||||
{
|
||||
"filename": file.filename,
|
||||
"status": "error",
|
||||
"error": "CORRUPTED_FILE",
|
||||
"message": "Le fichier semble corrompu ou n'est pas un document Office valide.",
|
||||
"details": {"reason": "unsafe_archive", "detail": str(e)[:200]},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception(f"Error processing {file.filename}")
|
||||
results.append(
|
||||
@@ -341,6 +361,9 @@ async def extract_texts_from_document(
|
||||
input_path = config.UPLOAD_DIR / input_filename
|
||||
file_handler.save_upload_file(file, input_path)
|
||||
|
||||
if file_extension != ".pdf":
|
||||
validate_zip_safety(input_path)
|
||||
|
||||
texts = []
|
||||
|
||||
if file_extension == ".xlsx":
|
||||
@@ -423,6 +446,17 @@ async def extract_texts_from_document(
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except ValueError as e:
|
||||
file_handler.cleanup_file(input_path)
|
||||
logger.warning(f"Text extraction rejected unsafe or invalid archive: {file.filename}: {e}")
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={
|
||||
"error": "CORRUPTED_FILE",
|
||||
"message": "Le fichier semble corrompu ou n'est pas un document Office valide.",
|
||||
"details": {"reason": "unsafe_archive", "detail": str(e)[:200]},
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception("Text extraction error")
|
||||
return JSONResponse(
|
||||
|
||||
@@ -9,6 +9,7 @@ Story 3.6: Documentation OpenAPI complète avec exemples et codes d'erreur
|
||||
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import uuid
|
||||
import time
|
||||
import socket
|
||||
@@ -43,7 +44,7 @@ from typing_extensions import Annotated
|
||||
|
||||
from config import config
|
||||
from translators import ExcelTranslator, WordTranslator, PowerPointTranslator
|
||||
from models.subscription import PlanType
|
||||
from models.subscription import PlanType, PLANS
|
||||
from services.auth_service import (
|
||||
record_usage,
|
||||
check_usage_limits,
|
||||
@@ -54,6 +55,7 @@ from middleware.tier_quota import _seconds_until_next_month, _next_month_utc
|
||||
from middleware.validation import FileValidator, ValidationError, LanguageValidator, webhook_validator
|
||||
from middleware.api_key_auth import get_authenticated_user, get_user_from_api_key
|
||||
from utils import file_handler
|
||||
from utils.file_handler import validate_zip_safety
|
||||
|
||||
# Import models from schemas (Story 3.6 - DRY principle)
|
||||
from schemas.translation import (
|
||||
@@ -148,6 +150,43 @@ def _tier_for_quota(plan) -> str:
|
||||
return "free"
|
||||
|
||||
|
||||
def _tier_from_plan_str(plan_str: Optional[str]) -> str:
|
||||
"""Map the job's ``user_plan`` string to a quota tier.
|
||||
|
||||
``user_plan`` is built with ``str(current_user.plan)`` which renders a
|
||||
str-Enum as "PlanType.PRO" — accept both that repr and the raw value.
|
||||
"""
|
||||
name = (plan_str or "").strip().lower()
|
||||
if name.startswith("plantype."):
|
||||
name = name.split(".", 1)[1]
|
||||
try:
|
||||
return _tier_for_quota(PlanType(name))
|
||||
except ValueError:
|
||||
return "free"
|
||||
|
||||
|
||||
def _plan_from_user(current_user: Optional[Any]) -> PlanType:
|
||||
"""Resolve the request's plan (anonymous requests are on the Free plan)."""
|
||||
if current_user is None:
|
||||
return PlanType.FREE
|
||||
plan = getattr(current_user, "plan", PlanType.FREE)
|
||||
try:
|
||||
return PlanType(plan)
|
||||
except ValueError:
|
||||
return PlanType.FREE
|
||||
|
||||
|
||||
def _allowed_providers_for_plan(plan: PlanType) -> set:
|
||||
"""Engines included in the plan (source of truth: PLANS[plan]["providers"])."""
|
||||
plan_cfg = PLANS.get(plan) or PLANS[PlanType.FREE]
|
||||
return set(plan_cfg.get("providers", []))
|
||||
|
||||
|
||||
def _image_translation_allowed_for_plan(plan: PlanType) -> bool:
|
||||
"""Vision (text-in-images) is sold on Pro and above."""
|
||||
return plan in (PlanType.PRO, PlanType.BUSINESS, PlanType.ENTERPRISE)
|
||||
|
||||
|
||||
def _next_midnight_utc() -> datetime:
|
||||
"""Get next midnight UTC."""
|
||||
now = datetime.now(timezone.utc)
|
||||
@@ -215,6 +254,21 @@ def _parse_content_disposition(content_disp: str) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
def _sanitize_url_filename(filename: str) -> str:
|
||||
"""Strip path traversal / control chars from a remote-provided filename."""
|
||||
filename = Path(filename).name
|
||||
filename = re.sub(r"[\x00-\x1f\x7f-\x9f]", "", filename)
|
||||
filename = re.sub(r'[<>:"/\\|?*]', "_", filename)
|
||||
# Drop leading dots (hidden files, ".." leftovers)
|
||||
filename = filename.lstrip(".")
|
||||
if len(filename) > 255:
|
||||
name, ext = (
|
||||
filename.rsplit(".", 1) if "." in filename else (filename, "")
|
||||
)
|
||||
filename = name[:250] + ("." + ext if ext else "")
|
||||
return filename or "downloaded_file"
|
||||
|
||||
|
||||
def _is_ssrf_risk(hostname: str) -> bool:
|
||||
"""Return True if hostname resolves to a private/reserved IP (SSRF prevention).
|
||||
|
||||
@@ -252,95 +306,130 @@ async def download_from_url(url: str, timeout: int = 30) -> tuple[Path, str]:
|
||||
details={"scheme": parsed_url.scheme or "none"},
|
||||
)
|
||||
|
||||
hostname = parsed_url.hostname or ""
|
||||
if not hostname or _is_ssrf_risk(hostname):
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message="The URL points to a blocked address (private or internal network).",
|
||||
details={"reason": "ssrf_blocked"},
|
||||
)
|
||||
def _validate_url_target(u: str):
|
||||
p = urlparse(u)
|
||||
if p.scheme not in ("http", "https"):
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message="Only HTTP/HTTPS URLs are accepted.",
|
||||
details={"scheme": p.scheme or "none"},
|
||||
)
|
||||
h = p.hostname or ""
|
||||
if not h or _is_ssrf_risk(h):
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message="The URL points to a blocked address (private or internal network).",
|
||||
details={"reason": "ssrf_blocked"},
|
||||
)
|
||||
|
||||
_validate_url_target(url)
|
||||
|
||||
MAX_REDIRECTS = 5
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=timeout, follow_redirects=True, max_redirects=10
|
||||
timeout=timeout, follow_redirects=False
|
||||
) as client:
|
||||
async with client.stream("GET", url) as response:
|
||||
if response.status_code != 200:
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message=f"URL unreachable (HTTP {response.status_code})",
|
||||
details={"status_code": response.status_code, "url": url[:100]},
|
||||
)
|
||||
|
||||
content_length = response.headers.get("content-length")
|
||||
if content_length:
|
||||
try:
|
||||
file_size = int(content_length)
|
||||
max_size_bytes = MAX_FILE_SIZE_MB * 1024 * 1024
|
||||
if file_size > max_size_bytes:
|
||||
current_url = url
|
||||
for _ in range(MAX_REDIRECTS + 1):
|
||||
_validate_url_target(current_url)
|
||||
async with client.stream("GET", current_url) as response:
|
||||
if response.status_code in (301, 302, 303, 307, 308):
|
||||
location = response.headers.get("location", "")
|
||||
if not location:
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.FILE_TOO_LARGE,
|
||||
message=f"File is too large ({round(file_size / (1024 * 1024), 2)} MB, max {MAX_FILE_SIZE_MB} MB).",
|
||||
details={
|
||||
"size_mb": round(file_size / (1024 * 1024), 2),
|
||||
"max_mb": MAX_FILE_SIZE_MB,
|
||||
},
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message="Redirect without Location header.",
|
||||
details={"reason": "invalid_redirect"},
|
||||
)
|
||||
except ValueError:
|
||||
pass
|
||||
# Re-check the redirect target before following it
|
||||
current_url = str(httpx.URL(current_url).join(location))
|
||||
continue
|
||||
|
||||
filename = None
|
||||
content_disp = response.headers.get("content-disposition", "")
|
||||
if content_disp:
|
||||
filename = _parse_content_disposition(content_disp)
|
||||
if response.status_code != 200:
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message=f"URL unreachable (HTTP {response.status_code})",
|
||||
details={"status_code": response.status_code, "url": url[:100]},
|
||||
)
|
||||
|
||||
if not filename:
|
||||
filename = unquote(Path(parsed_url.path).name) or "downloaded_file"
|
||||
content_length = response.headers.get("content-length")
|
||||
if content_length:
|
||||
try:
|
||||
file_size = int(content_length)
|
||||
max_size_bytes = MAX_FILE_SIZE_MB * 1024 * 1024
|
||||
if file_size > max_size_bytes:
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.FILE_TOO_LARGE,
|
||||
message=f"File is too large ({round(file_size / (1024 * 1024), 2)} MB, max {MAX_FILE_SIZE_MB} MB).",
|
||||
details={
|
||||
"size_mb": round(file_size / (1024 * 1024), 2),
|
||||
"max_mb": MAX_FILE_SIZE_MB,
|
||||
},
|
||||
)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
extension = Path(filename).suffix.lower()
|
||||
if extension not in ACCEPTED_EXTENSIONS:
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.INVALID_FORMAT,
|
||||
details={
|
||||
"detected_extension": extension or "none",
|
||||
"accepted_formats": list(ACCEPTED_EXTENSIONS),
|
||||
},
|
||||
)
|
||||
filename = None
|
||||
content_disp = response.headers.get("content-disposition", "")
|
||||
if content_disp:
|
||||
filename = _parse_content_disposition(content_disp)
|
||||
|
||||
unique_id = str(uuid.uuid4())[:8]
|
||||
safe_filename = f"{unique_id}_{filename}"
|
||||
temp_path = config.UPLOAD_DIR / safe_filename
|
||||
if not filename:
|
||||
filename = unquote(Path(urlparse(current_url).path).name)
|
||||
|
||||
temp_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
filename = _sanitize_url_filename(filename)
|
||||
|
||||
max_size_bytes = MAX_FILE_SIZE_MB * 1024 * 1024
|
||||
downloaded_bytes = 0
|
||||
extension = Path(filename).suffix.lower()
|
||||
if extension not in ACCEPTED_EXTENSIONS:
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.INVALID_FORMAT,
|
||||
details={
|
||||
"detected_extension": extension or "none",
|
||||
"accepted_formats": list(ACCEPTED_EXTENSIONS),
|
||||
},
|
||||
)
|
||||
|
||||
async with aiofiles.open(temp_path, "wb") as f:
|
||||
async for chunk in response.aiter_bytes(chunk_size=65536):
|
||||
downloaded_bytes += len(chunk)
|
||||
unique_id = str(uuid.uuid4())[:8]
|
||||
safe_filename = f"{unique_id}_{filename}"
|
||||
temp_path = config.UPLOAD_DIR / safe_filename
|
||||
|
||||
if downloaded_bytes > max_size_bytes:
|
||||
await f.close()
|
||||
if temp_path.exists():
|
||||
temp_path.unlink()
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.FILE_TOO_LARGE,
|
||||
details={
|
||||
"size_mb": round(
|
||||
downloaded_bytes / (1024 * 1024), 2
|
||||
),
|
||||
"max_mb": MAX_FILE_SIZE_MB,
|
||||
},
|
||||
)
|
||||
temp_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
await f.write(chunk)
|
||||
max_size_bytes = MAX_FILE_SIZE_MB * 1024 * 1024
|
||||
downloaded_bytes = 0
|
||||
|
||||
async with aiofiles.open(temp_path, "rb") as f:
|
||||
header = await f.read(4)
|
||||
await validate_file_content(header, extension)
|
||||
async with aiofiles.open(temp_path, "wb") as f:
|
||||
async for chunk in response.aiter_bytes(chunk_size=65536):
|
||||
downloaded_bytes += len(chunk)
|
||||
|
||||
return temp_path, filename
|
||||
if downloaded_bytes > max_size_bytes:
|
||||
await f.close()
|
||||
if temp_path.exists():
|
||||
temp_path.unlink()
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.FILE_TOO_LARGE,
|
||||
details={
|
||||
"size_mb": round(
|
||||
downloaded_bytes / (1024 * 1024), 2
|
||||
),
|
||||
"max_mb": MAX_FILE_SIZE_MB,
|
||||
},
|
||||
)
|
||||
|
||||
await f.write(chunk)
|
||||
|
||||
async with aiofiles.open(temp_path, "rb") as f:
|
||||
header = await f.read(4)
|
||||
await validate_file_content(header, extension)
|
||||
|
||||
return temp_path, filename
|
||||
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.URL_UNREACHABLE,
|
||||
message="Too many redirects.",
|
||||
details={"reason": "too_many_redirects", "max_redirects": MAX_REDIRECTS},
|
||||
)
|
||||
|
||||
except httpx.TimeoutException:
|
||||
if temp_path and temp_path.exists():
|
||||
@@ -420,9 +509,13 @@ def _cleanup_old_jobs() -> None:
|
||||
return
|
||||
_last_cleanup_ts = current_time
|
||||
|
||||
# Snapshot the items before filtering: concurrent coroutines (background
|
||||
# translation workers, status pollers) mutate _translation_jobs, and
|
||||
# iterating a dict while it is resized raises
|
||||
# "RuntimeError: dictionary changed size during iteration".
|
||||
expired_job_ids = [
|
||||
job_id
|
||||
for job_id, job in _translation_jobs.items()
|
||||
for job_id, job in list(_translation_jobs.items())
|
||||
if job.get("status") in ("completed", "failed")
|
||||
and (
|
||||
(ts := job.get("completed_at") or job.get("failed_at"))
|
||||
@@ -444,6 +537,68 @@ def _job_age_seconds(timestamp_str: str) -> float:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _provider_model(provider: Any) -> str:
|
||||
"""Best-effort read of a provider's model name.
|
||||
|
||||
The new-style providers (``services/providers/*``) store the model in
|
||||
``self._model``, while the legacy providers (``services/translation_service``)
|
||||
expose ``self.model``. Read either so cost-factor logic works for both.
|
||||
"""
|
||||
if provider is None:
|
||||
return ""
|
||||
return (
|
||||
getattr(provider, "_model", None)
|
||||
or getattr(provider, "model", None)
|
||||
or ""
|
||||
)
|
||||
|
||||
|
||||
def _compute_cost_factor(provider: Any, provider_name: str = "") -> int:
|
||||
"""Billing cost factor (1 = standard, 5 = premium).
|
||||
|
||||
Premium models (Claude, GPT-4 family, etc.) cost more and are billed at a
|
||||
higher factor. Cheap variants are explicitly downgraded to 1 (``haiku``,
|
||||
and the small GPT-4 models such as ``gpt-4o-mini`` / ``gpt-4o-nano``).
|
||||
The provider may be passed by instance (model read off it) or only by
|
||||
name (e.g. the ``openrouter_premium`` alias).
|
||||
"""
|
||||
model_lower = _provider_model(provider).lower()
|
||||
provider_lower = (provider_name or "").lower()
|
||||
|
||||
if "haiku" in model_lower or "mini" in model_lower or "nano" in model_lower:
|
||||
return 1
|
||||
if any(k in model_lower for k in ["claude", "fable", "gpt-4"]) or provider_lower == "openrouter_premium":
|
||||
return 5
|
||||
return 1
|
||||
|
||||
|
||||
def _compute_duration_seconds(created_at_iso: str) -> float:
|
||||
"""Elapsed seconds since ``created_at`` (UTC ISO string).
|
||||
|
||||
Uses a timezone-aware timestamp() to avoid the local-time bug that
|
||||
``time.mktime`` introduced. Falls back to 0 on parse errors so a bad
|
||||
timestamp can never flip a successful job into the error branch.
|
||||
"""
|
||||
return _job_age_seconds(created_at_iso)
|
||||
|
||||
|
||||
async def _release_quota_if_needed(user_id, usage_recorded: bool, job_id: str) -> None:
|
||||
"""Release the reserved translation quota on a soft-failure path.
|
||||
|
||||
The translation worker reserves a quota slot at request time. The generic
|
||||
``except`` branch releases it on hard failures, but the early ``return``
|
||||
paths (empty output / no translatable text / 0 texts translated) are NOT
|
||||
exceptions and must release the quota explicitly — otherwise the user
|
||||
loses a slot without receiving a translation.
|
||||
"""
|
||||
if user_id and not usage_recorded:
|
||||
try:
|
||||
await asyncio.to_thread(release_translation_quota, user_id)
|
||||
logger.info(f"Job {job_id}: released reserved quota after soft-failure")
|
||||
except Exception as release_err:
|
||||
logger.exception(f"Job {job_id}: failed to release reserved quota: {release_err}")
|
||||
|
||||
|
||||
@router_v1.post(
|
||||
"/translate",
|
||||
response_model=TranslateResponse,
|
||||
@@ -476,6 +631,14 @@ async def translate_document_v1(
|
||||
pdf_mode: Optional[Literal["layout", "text_only"]] = Form(
|
||||
default=None, description="PDF translation mode: 'layout' (preserve layout) or 'text_only' (clean text output). PDF only."
|
||||
),
|
||||
formality: Optional[Literal["formal", "informal"]] = Form(
|
||||
default=None,
|
||||
description="Tone override for LLM engines: 'formal' or 'informal'. Ignored by classic engines.",
|
||||
),
|
||||
output_mode: Optional[Literal["single", "bilingual"]] = Form(
|
||||
default="single",
|
||||
description="Output mode: 'single' (translated file only) or 'bilingual' (docx with source + translation interleaved).",
|
||||
),
|
||||
translate_images: bool = Form(
|
||||
default=False, description="Translate text inside images using AI vision"
|
||||
),
|
||||
@@ -676,7 +839,9 @@ async def translate_document_v1(
|
||||
rate_limit_remaining = -1
|
||||
|
||||
try:
|
||||
LanguageValidator.validate(target_lang)
|
||||
# Keep the canonical form (e.g. "zh-CN") so providers receive a
|
||||
# code they understand instead of the raw user input.
|
||||
target_lang = LanguageValidator.validate(target_lang)
|
||||
except ValidationError as e:
|
||||
raise TranslateEndpointError(
|
||||
code="INVALID_FORMAT",
|
||||
@@ -686,7 +851,7 @@ async def translate_document_v1(
|
||||
|
||||
if source_lang and source_lang != "auto":
|
||||
try:
|
||||
LanguageValidator.validate(source_lang)
|
||||
source_lang = LanguageValidator.validate(source_lang)
|
||||
except ValidationError:
|
||||
raise TranslateEndpointError(
|
||||
code="INVALID_FORMAT",
|
||||
@@ -760,7 +925,23 @@ async def translate_document_v1(
|
||||
details={"error": "sha256_calculation_failed"},
|
||||
)
|
||||
|
||||
# Office files are ZIP archives: reject archives that expand far
|
||||
# beyond their uploaded size (zip bomb protection).
|
||||
if file_extension != ".pdf":
|
||||
try:
|
||||
validate_zip_safety(input_path)
|
||||
except ValueError as e:
|
||||
file_handler_util.cleanup_file(input_path)
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.CORRUPTED_FILE,
|
||||
message="The file is invalid or expands dangerously.",
|
||||
details={"reason": "unsafe_archive", "detail": str(e)[:200]},
|
||||
)
|
||||
|
||||
job_id = f"tr_{uuid.uuid4().hex[:12]}"
|
||||
# Secret per-job token: required to follow/download a job that has no
|
||||
# logged-in owner (anonymous API calls).
|
||||
job_access_token = secrets.token_urlsafe(24)
|
||||
|
||||
# Track file metadata in Redis with TTL
|
||||
await storage_tracker.track_file(
|
||||
@@ -771,6 +952,7 @@ async def translate_document_v1(
|
||||
"file_hash": file_hash,
|
||||
"input_path": str(input_path),
|
||||
"user_id": str(user_id) if user_id else None,
|
||||
"access_token": job_access_token,
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
},
|
||||
)
|
||||
@@ -794,6 +976,7 @@ async def translate_document_v1(
|
||||
"target_lang": target_lang,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"user_id": user_id,
|
||||
"access_token": job_access_token,
|
||||
"input_path": str(input_path),
|
||||
"file_extension": file_extension,
|
||||
"provider": provider or mode,
|
||||
@@ -803,6 +986,8 @@ async def translate_document_v1(
|
||||
"prompt_id": prompt_id, # Story 3.12: Store prompt_id
|
||||
"pdf_mode": pdf_mode, # PDF translation mode
|
||||
"translate_images": translate_images,
|
||||
"formality": formality, # LLM tone override
|
||||
"output_mode": output_mode or "single", # single | bilingual
|
||||
}
|
||||
await set_job_status_async(job_id, _translation_jobs[job_id])
|
||||
|
||||
@@ -821,6 +1006,39 @@ async def translate_document_v1(
|
||||
)
|
||||
provider_to_use = "google"
|
||||
|
||||
# ── Plan-based engine gating (grille commerciale : PLANS[plan]["providers"]) ──
|
||||
# Chaque plan n'expose que ses moteurs ; un moteur hors plan est refusé
|
||||
# (403) au lieu d'être exécuté aux frais de la maison.
|
||||
_user_plan = _plan_from_user(current_user)
|
||||
if provider_to_use not in _allowed_providers_for_plan(_user_plan):
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.PRO_FEATURE_REQUIRED,
|
||||
message=(
|
||||
f"Le moteur « {provider_to_use} » n'est pas inclus dans votre plan. "
|
||||
"Passez à un plan supérieur pour l'utiliser."
|
||||
),
|
||||
details={
|
||||
"feature": "provider",
|
||||
"provider": provider_to_use,
|
||||
"plan": str(_user_plan.value),
|
||||
"allowed_providers": sorted(_allowed_providers_for_plan(_user_plan)),
|
||||
},
|
||||
)
|
||||
|
||||
# La traduction du texte dans les images (vision) est vendue Pro et plus.
|
||||
if translate_images and not _image_translation_allowed_for_plan(_user_plan):
|
||||
raise TranslateEndpointError(
|
||||
code=TranslateEndpointError.PRO_FEATURE_REQUIRED,
|
||||
message=(
|
||||
"La traduction du texte dans les images est réservée "
|
||||
"aux plans Pro et supérieurs."
|
||||
),
|
||||
details={
|
||||
"feature": "translate_images",
|
||||
"plan": str(_user_plan.value),
|
||||
},
|
||||
)
|
||||
|
||||
asyncio.create_task(
|
||||
_run_translation_job(
|
||||
job_id=job_id,
|
||||
@@ -837,6 +1055,8 @@ async def translate_document_v1(
|
||||
user_plan=str(current_user.plan) if current_user else "free",
|
||||
pdf_mode=pdf_mode,
|
||||
translate_images=translate_images,
|
||||
formality=formality,
|
||||
output_mode=output_mode or "single",
|
||||
)
|
||||
)
|
||||
|
||||
@@ -853,6 +1073,8 @@ async def translate_document_v1(
|
||||
"file_name": original_filename,
|
||||
"source_lang": source_lang,
|
||||
"target_lang": target_lang,
|
||||
# Needed to poll/download this job when calling without login
|
||||
"access_token": job_access_token,
|
||||
},
|
||||
"meta": {
|
||||
"rate_limit_remaining": rate_limit_remaining,
|
||||
@@ -980,6 +1202,8 @@ async def _run_translation_job(
|
||||
user_plan: Optional[str] = None, # Plan name for watermark decision
|
||||
pdf_mode: Optional[str] = None, # PDF translation mode: "layout" or "text_only"
|
||||
translate_images: bool = False,
|
||||
formality: Optional[str] = None, # LLM tone override: formal/informal
|
||||
output_mode: str = "single", # single | bilingual
|
||||
) -> None:
|
||||
"""
|
||||
Run translation job in background with progress tracking.
|
||||
@@ -1076,11 +1300,13 @@ async def _run_translation_job(
|
||||
# Use custom_prompt if no prompt_id
|
||||
effective_prompt = custom_prompt
|
||||
|
||||
# Build the full prompt combining effective prompt and glossary
|
||||
# Build the full prompt combining effective prompt, glossary,
|
||||
# formality directive and regional variant hint.
|
||||
full_prompt = build_full_prompt(
|
||||
effective_prompt, glossary_terms,
|
||||
source_lang=glossary_source_lang, target_lang=target_lang,
|
||||
glossary_target_lang=glossary_target_lang,
|
||||
formality=formality,
|
||||
)
|
||||
|
||||
from services.providers.google_provider import GoogleTranslationProvider
|
||||
@@ -1259,6 +1485,8 @@ async def _run_translation_job(
|
||||
job_translator = ExcelTranslator(provider=translation_provider)
|
||||
if hasattr(job_translator, "set_custom_prompt"):
|
||||
job_translator.set_custom_prompt(full_prompt)
|
||||
if hasattr(job_translator, "set_tm_scope"):
|
||||
job_translator.set_tm_scope(user_id, full_prompt)
|
||||
await asyncio.to_thread(
|
||||
job_translator.translate_file,
|
||||
input_path,
|
||||
@@ -1272,6 +1500,8 @@ async def _run_translation_job(
|
||||
job_translator = WordTranslator(provider=translation_provider)
|
||||
if hasattr(job_translator, "set_custom_prompt"):
|
||||
job_translator.set_custom_prompt(full_prompt)
|
||||
if hasattr(job_translator, "set_tm_scope"):
|
||||
job_translator.set_tm_scope(user_id, full_prompt)
|
||||
await asyncio.to_thread(
|
||||
job_translator.translate_file,
|
||||
input_path,
|
||||
@@ -1285,6 +1515,8 @@ async def _run_translation_job(
|
||||
job_translator = PowerPointTranslator(provider=translation_provider)
|
||||
if hasattr(job_translator, "set_custom_prompt"):
|
||||
job_translator.set_custom_prompt(full_prompt)
|
||||
if hasattr(job_translator, "set_tm_scope"):
|
||||
job_translator.set_tm_scope(user_id, full_prompt)
|
||||
await asyncio.to_thread(
|
||||
job_translator.translate_file,
|
||||
input_path,
|
||||
@@ -1299,6 +1531,40 @@ async def _run_translation_job(
|
||||
job_translator = PDFTranslator(provider=translation_provider)
|
||||
if hasattr(job_translator, "set_custom_prompt"):
|
||||
job_translator.set_custom_prompt(full_prompt)
|
||||
if hasattr(job_translator, "set_tm_scope"):
|
||||
job_translator.set_tm_scope(user_id, full_prompt)
|
||||
# OCR (PDF scannés) : réglages admin > variables d'env.
|
||||
mistral_cfg = getattr(_admin_cfg, "mistral", None)
|
||||
mistral_key = _cfg(
|
||||
getattr(mistral_cfg, "api_key", None), "MISTRAL_API_KEY"
|
||||
)
|
||||
# Only honor the admin "enabled" flag when Mistral is actually
|
||||
# configured there — a freshly saved settings file carries
|
||||
# enabled=false defaults that must not override env-based OCR.
|
||||
mistral_admin_configured = bool(
|
||||
mistral_cfg is not None
|
||||
and (getattr(mistral_cfg, "api_key", None) or "").strip()
|
||||
)
|
||||
job_translator.set_ocr_config(
|
||||
api_key=mistral_key,
|
||||
model=_cfg(
|
||||
getattr(mistral_cfg, "model", None),
|
||||
"MISTRAL_OCR_MODEL",
|
||||
"mistral-ocr-latest",
|
||||
),
|
||||
timeout=int(
|
||||
_cfg(
|
||||
str(getattr(mistral_cfg, "timeout", "") or ""),
|
||||
"MISTRAL_OCR_TIMEOUT",
|
||||
"180",
|
||||
)
|
||||
),
|
||||
enabled=(
|
||||
getattr(mistral_cfg, "enabled", None)
|
||||
if mistral_admin_configured
|
||||
else None
|
||||
),
|
||||
)
|
||||
actual_output = await asyncio.to_thread(
|
||||
job_translator.translate_file,
|
||||
input_path,
|
||||
@@ -1320,19 +1586,21 @@ async def _run_translation_job(
|
||||
error_msg = "Translation failed: output file is empty or missing. The translation provider may be unavailable."
|
||||
logger.error(f"Job {job_id}: {error_msg}")
|
||||
tracker.set_error(error_msg)
|
||||
await _release_quota_if_needed(user_id, usage_recorded, job_id)
|
||||
return
|
||||
|
||||
stats = job_translator.get_translation_stats()
|
||||
attempted = stats.get("attempted", 0)
|
||||
changed = stats.get("changed", 0)
|
||||
|
||||
if attempted == 0 and file_extension in ('.docx', '.xlsx', '.pptx'):
|
||||
if attempted == 0:
|
||||
error_msg = (
|
||||
"Aucun texte traduisible détecté dans le document. "
|
||||
"Le fichier est peut-être vide, protégé, ou ne contient que des images."
|
||||
)
|
||||
logger.error(f"Job {job_id}: {error_msg}")
|
||||
tracker.set_error(error_msg)
|
||||
await _release_quota_if_needed(user_id, usage_recorded, job_id)
|
||||
return
|
||||
|
||||
if attempted > 0:
|
||||
@@ -1346,6 +1614,7 @@ async def _run_translation_job(
|
||||
)
|
||||
logger.error(f"Job {job_id}: {error_msg}")
|
||||
tracker.set_error(error_msg)
|
||||
await _release_quota_if_needed(user_id, usage_recorded, job_id)
|
||||
return
|
||||
elif ratio < 0.05:
|
||||
# Very suspicious — likely partial failure, warn but don't block
|
||||
@@ -1430,9 +1699,7 @@ async def _run_translation_job(
|
||||
from middleware.metrics import record_translation_retry
|
||||
record_translation_retry(
|
||||
reason="l1_fail",
|
||||
tier=_tier_for_quota(
|
||||
current_user.plan if current_user else None
|
||||
) if current_user else "free",
|
||||
tier=_tier_from_plan_str(user_plan),
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
@@ -1448,9 +1715,7 @@ async def _run_translation_job(
|
||||
from middleware.metrics import record_translation_retry
|
||||
record_translation_retry(
|
||||
reason="l0_fail",
|
||||
tier=_tier_for_quota(
|
||||
current_user.plan if current_user else None
|
||||
) if current_user else "free",
|
||||
tier=_tier_from_plan_str(user_plan),
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
@@ -1467,9 +1732,7 @@ async def _run_translation_job(
|
||||
from services.quality import run_l2_check
|
||||
|
||||
# Tier gate: Pro+ plans only (unless gate is disabled)
|
||||
user_tier = (
|
||||
_tier_for_quota(current_user.plan) if current_user else "free"
|
||||
)
|
||||
user_tier = _tier_from_plan_str(user_plan)
|
||||
tier_gate_on = getattr(config, "QUALITY_L2_TIER_GATE", True)
|
||||
if not tier_gate_on or user_tier in ("pro", "business", "enterprise"):
|
||||
translated_chunks_for_l2 = [s["translated"] for s in quality_samples]
|
||||
@@ -1498,21 +1761,52 @@ async def _run_translation_job(
|
||||
f"Job {job_id}: quality L2 layer failed: {l2_err}"
|
||||
)
|
||||
|
||||
if user_id:
|
||||
# Determine cost factor based on selected provider and model
|
||||
cost_factor = 1
|
||||
provider_lower = (provider or "").lower()
|
||||
# ------------------------------------------------------------------
|
||||
# QA report (pure heuristics — numbers fidelity, untranslated
|
||||
# ratio, 0-100 score). Never blocks the job; surfaced in the job
|
||||
# status for the UI/API.
|
||||
# ------------------------------------------------------------------
|
||||
try:
|
||||
from services.quality.qa_report import run_qa_report
|
||||
|
||||
prov_model = ""
|
||||
if translation_provider:
|
||||
prov_model = getattr(translation_provider, "model", "") or ""
|
||||
qa = await asyncio.to_thread(
|
||||
run_qa_report, input_path, output_path, target_lang, file_extension
|
||||
)
|
||||
if qa:
|
||||
job["quality"] = qa
|
||||
except Exception as qa_err:
|
||||
logger.warning(f"Job {job_id}: QA report failed: {qa_err}")
|
||||
|
||||
prov_model_lower = prov_model.lower()
|
||||
if any(k in prov_model_lower for k in ["claude", "fable", "gpt-4"]) or provider_lower == "openrouter_premium":
|
||||
if "haiku" in prov_model_lower:
|
||||
cost_factor = 1
|
||||
# ------------------------------------------------------------------
|
||||
# Bilingual output (docx): interleaves source paragraphs above
|
||||
# their translation. Falls back silently to the translated file.
|
||||
# ------------------------------------------------------------------
|
||||
if output_mode == "bilingual" and file_extension == ".docx":
|
||||
try:
|
||||
from translators.bilingual import make_bilingual_docx
|
||||
|
||||
bilingual_path = output_path.with_name(
|
||||
output_path.stem + "_bilingual" + output_path.suffix
|
||||
)
|
||||
actual = await asyncio.to_thread(
|
||||
make_bilingual_docx, input_path, output_path, bilingual_path
|
||||
)
|
||||
if actual:
|
||||
output_path = Path(actual)
|
||||
logger.info(f"Job {job_id}: bilingual output generated")
|
||||
else:
|
||||
cost_factor = 5
|
||||
logger.warning(
|
||||
f"Job {job_id}: bilingual output skipped "
|
||||
"(structure mismatch) — returning translated file"
|
||||
)
|
||||
except Exception as bi_err:
|
||||
logger.warning(f"Job {job_id}: bilingual output failed: {bi_err}")
|
||||
|
||||
if user_id:
|
||||
# Determine cost factor based on selected provider and model.
|
||||
# _compute_cost_factor reads the model off the provider robustly
|
||||
# (new-style providers store it in ``_model``, legacy in ``model``).
|
||||
cost_factor = _compute_cost_factor(translation_provider, provider or "")
|
||||
|
||||
# Persist monthly usage counters in PostgreSQL (docs + pages)
|
||||
pages = await asyncio.to_thread(
|
||||
@@ -1537,7 +1831,7 @@ async def _run_translation_job(
|
||||
|
||||
tracker.set_completed(str(output_path))
|
||||
# Record translation metric
|
||||
duration = time.time() - time.mktime(datetime.fromisoformat(job["created_at"].replace("Z", "+00:00")).timetuple())
|
||||
duration = _compute_duration_seconds(job.get("created_at", ""))
|
||||
record_translation(provider=provider, file_type=file_extension or "unknown", duration=duration, status="success")
|
||||
logger.info(f"Job {job_id}: Completed successfully")
|
||||
|
||||
@@ -1621,6 +1915,50 @@ async def _run_translation_job(
|
||||
)
|
||||
|
||||
|
||||
def _check_job_access(
|
||||
job: dict, current_user, token: Optional[str]
|
||||
) -> Optional[JSONResponse]:
|
||||
"""Return an error response if the caller may not see this job, else None.
|
||||
|
||||
Jobs owned by a logged-in user require that same user. Anonymous jobs
|
||||
require the secret per-job token returned when the job was created.
|
||||
"""
|
||||
job_user_id = job.get("user_id")
|
||||
if job_user_id:
|
||||
if not current_user:
|
||||
return JSONResponse(
|
||||
status_code=401,
|
||||
content={
|
||||
"error": "AUTH_REQUIRED",
|
||||
"message": "Authentication is required to access this job.",
|
||||
"details": {"job_id": job.get("id")},
|
||||
},
|
||||
)
|
||||
if str(job_user_id) != str(current_user.id):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={
|
||||
"error": "ACCESS_DENIED",
|
||||
"message": "You do not have access to this job.",
|
||||
"details": {"job_id": job.get("id")},
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
expected = job.get("access_token")
|
||||
provided = token or ""
|
||||
if not expected or not secrets.compare_digest(provided, expected):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={
|
||||
"error": "ACCESS_DENIED",
|
||||
"message": "You do not have access to this job.",
|
||||
"details": {"job_id": job.get("id"), "hint": "token_required"},
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
@router_v1.get(
|
||||
"/translations/{job_id}",
|
||||
response_model=TranslationStatusResponse,
|
||||
@@ -1631,6 +1969,7 @@ async def _run_translation_job(
|
||||
)
|
||||
async def get_translation_status(
|
||||
job_id: str,
|
||||
token: Optional[str] = None,
|
||||
current_user: Optional[Any] = Depends(get_authenticated_user),
|
||||
):
|
||||
"""
|
||||
@@ -1680,6 +2019,10 @@ async def get_translation_status(
|
||||
},
|
||||
)
|
||||
|
||||
denied = _check_job_access(job, current_user, token)
|
||||
if denied:
|
||||
return denied
|
||||
|
||||
response_data = {
|
||||
"id": job["id"],
|
||||
"status": job["status"],
|
||||
@@ -1689,6 +2032,7 @@ async def get_translation_status(
|
||||
"source_lang": job.get("source_lang"),
|
||||
"target_lang": job.get("target_lang"),
|
||||
"created_at": job.get("created_at"),
|
||||
"quality": job.get("quality"),
|
||||
}
|
||||
|
||||
estimated_remaining = None
|
||||
@@ -1768,6 +2112,7 @@ def _cleanup_files(input_path: Optional[str], output_path: Optional[str]) -> Non
|
||||
)
|
||||
async def download_translated_file(
|
||||
job_id: str,
|
||||
token: Optional[str] = None,
|
||||
current_user: Optional[Any] = Depends(get_authenticated_user),
|
||||
):
|
||||
"""
|
||||
@@ -1819,16 +2164,9 @@ async def download_translated_file(
|
||||
},
|
||||
)
|
||||
|
||||
job_user_id = job.get("user_id")
|
||||
if current_user and job_user_id and str(job_user_id) != str(current_user.id):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={
|
||||
"error": "ACCESS_DENIED",
|
||||
"message": "You do not have access to this file.",
|
||||
"details": {"job_id": job_id},
|
||||
},
|
||||
)
|
||||
denied = _check_job_access(job, current_user, token)
|
||||
if denied:
|
||||
return denied
|
||||
|
||||
if job.get("status") != "completed":
|
||||
return JSONResponse(
|
||||
|
||||
98
routes/waitlist_routes.py
Normal file
98
routes/waitlist_routes.py
Normal file
@@ -0,0 +1,98 @@
|
||||
"""
|
||||
Waitlist / email-capture routes for the marketing landing page.
|
||||
|
||||
Public endpoint: POST /api/v1/waitlist (no auth) — deduplicated by email,
|
||||
persisted in data/waitlist.json (same JSON-file pattern as provider settings).
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel, EmailStr
|
||||
|
||||
from config import config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/waitlist", tags=["Waitlist"])
|
||||
|
||||
WAITLIST_FILE = config.BASE_DIR / "data" / "waitlist.json"
|
||||
_write_lock = threading.Lock()
|
||||
|
||||
|
||||
class WaitlistEntry(BaseModel):
|
||||
email: EmailStr
|
||||
interest: Optional[str] = None
|
||||
|
||||
|
||||
def _load_waitlist() -> List[dict]:
|
||||
if not WAITLIST_FILE.exists():
|
||||
return []
|
||||
try:
|
||||
with open(WAITLIST_FILE, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, list) else []
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load waitlist: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def _save_waitlist(entries: List[dict]) -> None:
|
||||
WAITLIST_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(WAITLIST_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(entries, f, indent=2)
|
||||
|
||||
|
||||
@router.post("", status_code=201, summary="Join the waitlist")
|
||||
async def join_waitlist(entry: WaitlistEntry):
|
||||
email = entry.email.lower().strip()
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
with _write_lock:
|
||||
entries = _load_waitlist()
|
||||
for existing in entries:
|
||||
if existing.get("email", "").lower() == email:
|
||||
existing["updated_at"] = now
|
||||
if entry.interest:
|
||||
existing["interest"] = entry.interest
|
||||
_save_waitlist(entries)
|
||||
logger.info(f"Waitlist rejoin: {email}")
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"data": {"email": email, "status": "already_joined"},
|
||||
"message": "You are already on the list.",
|
||||
},
|
||||
)
|
||||
|
||||
entries.append(
|
||||
{
|
||||
"email": email,
|
||||
"interest": entry.interest,
|
||||
"joined_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
_save_waitlist(entries)
|
||||
logger.info(f"Waitlist join: {email} (total={len(entries)})")
|
||||
|
||||
return JSONResponse(
|
||||
status_code=201,
|
||||
content={
|
||||
"data": {"email": email, "status": "joined"},
|
||||
"message": "Welcome aboard!",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/count", summary="Waitlist count")
|
||||
async def waitlist_count():
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={"data": {"count": len(_load_waitlist())}},
|
||||
)
|
||||
@@ -146,6 +146,14 @@ class TranslationStatusData(BaseModel):
|
||||
description="Message d'erreur si status='failed'",
|
||||
example=None
|
||||
)
|
||||
quality: Optional[dict] = Field(
|
||||
None,
|
||||
description=(
|
||||
"Rapport qualité post-traduction (job réussi) : "
|
||||
"score 0-100, fidélité des nombres, ratio non traduit."
|
||||
),
|
||||
example={"score": 92, "numbers": {"source_numbers": 12, "preserved": 12, "fidelity": 1.0}, "untranslated_ratio": 0.02},
|
||||
)
|
||||
|
||||
class Config:
|
||||
json_schema_extra = {
|
||||
|
||||
27
scripts/diagnostic_templates.py
Normal file
27
scripts/diagnostic_templates.py
Normal file
@@ -0,0 +1,27 @@
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
GLOSSARIES_DIR = Path(os.getenv("GLOSSARIES_DIR", "data/glossaries"))
|
||||
TARGET_LANGUAGES = ["de", "es", "it", "pt", "nl", "ru", "ja", "ko", "zh", "ar", "fa"]
|
||||
|
||||
if not GLOSSARIES_DIR.exists():
|
||||
raise SystemExit(f"Glossaries directory not found: {GLOSSARIES_DIR} (set GLOSSARIES_DIR)")
|
||||
|
||||
for f in GLOSSARIES_DIR.glob("*.json"):
|
||||
if f.name == "index.json":
|
||||
continue
|
||||
try:
|
||||
with open(f, "r", encoding="utf-8") as file:
|
||||
data = json.load(file)
|
||||
terms = data.get("terms", [])
|
||||
missing_count = 0
|
||||
total_count = len(terms)
|
||||
for t in terms:
|
||||
trans = t.get("translations", {})
|
||||
for lang in TARGET_LANGUAGES:
|
||||
if lang not in trans or not trans[lang]:
|
||||
missing_count += 1
|
||||
print(f"{f.name}: {total_count} termes, {missing_count} traductions manquantes.")
|
||||
except Exception as e:
|
||||
print(f"Error {f.name}: {e}")
|
||||
@@ -198,9 +198,11 @@ def build_full_prompt(
|
||||
source_lang: str = "fr",
|
||||
target_lang: str = "en",
|
||||
glossary_target_lang: str = "multi",
|
||||
formality: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Build the complete prompt combining custom prompt and glossary.
|
||||
Build the complete prompt combining custom prompt, glossary, formality
|
||||
and regional variant directives.
|
||||
|
||||
Args:
|
||||
custom_prompt: Optional custom system prompt from user
|
||||
@@ -208,6 +210,8 @@ def build_full_prompt(
|
||||
source_lang: ISO code of the source language
|
||||
target_lang: ISO code of the target language
|
||||
glossary_target_lang: ISO code of the glossary's target language configuration
|
||||
formality: Optional tone override — "formal" or "informal". Only
|
||||
meaningful for LLM engines (ignored by classic engines).
|
||||
|
||||
Returns:
|
||||
Combined prompt string
|
||||
@@ -224,4 +228,30 @@ def build_full_prompt(
|
||||
if glossary_prompt:
|
||||
parts.append(glossary_prompt)
|
||||
|
||||
if formality in ("formal", "informal"):
|
||||
if formality == "formal":
|
||||
parts.append(
|
||||
"TONE: Use a formal, professional register throughout "
|
||||
"(formal address (vous/Sie) where the language distinguishes; "
|
||||
"no slang, no contractions where avoidable)."
|
||||
)
|
||||
else:
|
||||
parts.append(
|
||||
"TONE: Use an informal, natural register throughout "
|
||||
"(tu-style address where the language distinguishes; "
|
||||
"contractions welcome)."
|
||||
)
|
||||
|
||||
# Regional variant: when the target code carries a region (pt-BR,
|
||||
# fr-CA, zh-CN...), make the expected variety explicit — LLMs default
|
||||
# to the dominant variant otherwise (pt-PT, fr-FR...).
|
||||
if target_lang and "-" in target_lang and target_lang != "auto":
|
||||
from core.languages import language_name
|
||||
|
||||
name = language_name(target_lang)
|
||||
if name and name != target_lang:
|
||||
parts.append(
|
||||
f"REGIONAL VARIANT: write specifically in {name}."
|
||||
)
|
||||
|
||||
return "\n\n".join(parts) if parts else ""
|
||||
216
services/mistral_ocr.py
Normal file
216
services/mistral_ocr.py
Normal file
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
Mistral OCR client — text extraction for scanned PDFs.
|
||||
|
||||
Image-only PDFs have no extractable text layer: PyMuPDF sees empty pages
|
||||
and the layout-preserving pipeline would output an empty document. This
|
||||
client calls the Mistral OCR API to recover the text.
|
||||
|
||||
API reference (2026): POST https://api.mistral.ai/v1/ocr with Bearer auth,
|
||||
body {"model", "document": {"type": "document_url", "document_url":
|
||||
"data:application/pdf;base64,..."}, "pages": [0, 1, ...]}. The response
|
||||
contains {"pages": [{"index", "markdown", "dimensions"}, ...]}.
|
||||
|
||||
The document is sent in chunks of PAGES_PER_REQUEST pages: it bounds the
|
||||
request payload and lets us report progress page by page.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
import requests
|
||||
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
MISTRAL_OCR_URL = "https://api.mistral.ai/v1/ocr"
|
||||
DEFAULT_MODEL = "mistral-ocr-latest"
|
||||
PAGES_PER_REQUEST = 8
|
||||
|
||||
MISTRAL_INVALID_KEY = "MISTRAL_INVALID_KEY"
|
||||
MISTRAL_QUOTA_EXCEEDED = "MISTRAL_QUOTA_EXCEEDED"
|
||||
MISTRAL_TIMEOUT = "MISTRAL_TIMEOUT"
|
||||
MISTRAL_SERVICE_ERROR = "MISTRAL_SERVICE_ERROR"
|
||||
|
||||
|
||||
class MistralOCRError(Exception):
|
||||
"""Raised when the Mistral OCR API cannot extract the PDF text."""
|
||||
|
||||
def __init__(
|
||||
self, code: str, message: str, details: Optional[Dict[str, Any]] = None
|
||||
):
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.details = details or {}
|
||||
super().__init__(self.message)
|
||||
|
||||
|
||||
class MistralOCRClient:
|
||||
"""Thin synchronous client around the Mistral OCR endpoint."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_key: str,
|
||||
model: str = DEFAULT_MODEL,
|
||||
timeout: int = 180,
|
||||
max_retries: int = 2,
|
||||
retry_delay: float = 2.0,
|
||||
):
|
||||
self._api_key = (api_key or "").strip()
|
||||
self._model = model or DEFAULT_MODEL
|
||||
self._timeout = timeout
|
||||
self._max_retries = max_retries
|
||||
self._retry_delay = retry_delay
|
||||
|
||||
def is_available(self) -> bool:
|
||||
"""True when an API key is configured."""
|
||||
return bool(self._api_key)
|
||||
|
||||
def _post_ocr(self, data_uri: str, pages: List[int]) -> List[Dict[str, Any]]:
|
||||
"""POST one OCR request for the given 0-based page list, with retries."""
|
||||
payload = {
|
||||
"model": self._model,
|
||||
"document": {"type": "document_url", "document_url": data_uri},
|
||||
"pages": pages,
|
||||
}
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self._api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
last_error: Optional[Exception] = None
|
||||
for attempt in range(self._max_retries + 1):
|
||||
try:
|
||||
response = requests.post(
|
||||
MISTRAL_OCR_URL,
|
||||
json=payload,
|
||||
headers=headers,
|
||||
timeout=self._timeout,
|
||||
)
|
||||
|
||||
if response.status_code == 401:
|
||||
raise MistralOCRError(
|
||||
MISTRAL_INVALID_KEY,
|
||||
"Clé API Mistral invalide (MISTRAL_API_KEY).",
|
||||
{"status_code": 401},
|
||||
)
|
||||
if response.status_code in (402, 429):
|
||||
raise MistralOCRError(
|
||||
MISTRAL_QUOTA_EXCEEDED,
|
||||
"Quota Mistral OCR épuisé ou limite de débit atteinte.",
|
||||
{"status_code": response.status_code},
|
||||
)
|
||||
if response.status_code >= 500:
|
||||
raise MistralOCRError(
|
||||
MISTRAL_SERVICE_ERROR,
|
||||
f"Service Mistral OCR indisponible (HTTP {response.status_code}).",
|
||||
{"status_code": response.status_code},
|
||||
)
|
||||
if response.status_code != 200:
|
||||
raise MistralOCRError(
|
||||
MISTRAL_SERVICE_ERROR,
|
||||
f"Erreur Mistral OCR (HTTP {response.status_code}): {response.text[:200]}",
|
||||
{"status_code": response.status_code},
|
||||
)
|
||||
|
||||
pages_out = response.json().get("pages", [])
|
||||
if not pages_out:
|
||||
raise MistralOCRError(
|
||||
MISTRAL_SERVICE_ERROR,
|
||||
"Réponse Mistral OCR vide.",
|
||||
)
|
||||
return pages_out
|
||||
|
||||
except MistralOCRError as e:
|
||||
if e.code in (MISTRAL_INVALID_KEY, MISTRAL_QUOTA_EXCEEDED):
|
||||
raise # not transient
|
||||
last_error = e
|
||||
except requests.exceptions.Timeout as e:
|
||||
last_error = MistralOCRError(
|
||||
MISTRAL_TIMEOUT,
|
||||
f"Délai d'attente Mistral OCR dépassé ({self._timeout}s).",
|
||||
)
|
||||
except requests.exceptions.RequestException as e:
|
||||
last_error = MistralOCRError(
|
||||
MISTRAL_SERVICE_ERROR,
|
||||
f"Mistral OCR injoignable: {str(e)[:150]}",
|
||||
)
|
||||
|
||||
if attempt < self._max_retries:
|
||||
delay = self._retry_delay * (2**attempt)
|
||||
logger.info(
|
||||
"mistral_ocr_retry",
|
||||
attempt=attempt + 1,
|
||||
delay_s=round(delay, 2),
|
||||
error=last_error.code if last_error else "unknown",
|
||||
)
|
||||
time.sleep(delay)
|
||||
|
||||
raise last_error or MistralOCRError(
|
||||
MISTRAL_SERVICE_ERROR, "Erreur Mistral OCR inconnue."
|
||||
)
|
||||
|
||||
def extract_pdf_text(
|
||||
self,
|
||||
pdf_path: Path,
|
||||
progress_callback: Optional[Callable[[Dict[str, Any]], None]] = None,
|
||||
) -> List[str]:
|
||||
"""OCR a PDF file and return one text (markdown) string per page.
|
||||
|
||||
Pages are processed in chunks of ``PAGES_PER_REQUEST``; the returned
|
||||
list is ordered by page index, empty strings for pages OCR returned
|
||||
nothing for.
|
||||
"""
|
||||
import fitz
|
||||
|
||||
pdf_path = Path(pdf_path)
|
||||
data_b64 = base64.b64encode(pdf_path.read_bytes()).decode("ascii")
|
||||
data_uri = f"data:application/pdf;base64,{data_b64}"
|
||||
|
||||
with fitz.open(str(pdf_path)) as doc:
|
||||
total_pages = len(doc)
|
||||
if total_pages == 0:
|
||||
raise MistralOCRError(MISTRAL_SERVICE_ERROR, "PDF vide (0 page).")
|
||||
|
||||
chunks = [
|
||||
list(range(start, min(start + PAGES_PER_REQUEST, total_pages)))
|
||||
for start in range(0, total_pages, PAGES_PER_REQUEST)
|
||||
]
|
||||
|
||||
page_texts: List[str] = [""] * total_pages
|
||||
done_chunks = 0
|
||||
for chunk in chunks:
|
||||
pages_out = self._post_ocr(data_uri, chunk)
|
||||
for page in pages_out:
|
||||
idx = int(page.get("index", -1))
|
||||
if 0 <= idx < total_pages:
|
||||
page_texts[idx] = page.get("markdown", "") or ""
|
||||
|
||||
done_chunks += 1
|
||||
logger.info(
|
||||
"mistral_ocr_chunk_done",
|
||||
pages_done=min(done_chunks * PAGES_PER_REQUEST, total_pages),
|
||||
total_pages=total_pages,
|
||||
)
|
||||
if progress_callback and chunks:
|
||||
pct = int(5 + 20 * done_chunks / len(chunks))
|
||||
progress_callback(
|
||||
{
|
||||
"current": done_chunks,
|
||||
"total": len(chunks),
|
||||
"phase": f"OCR (Mistral) {min(done_chunks * PAGES_PER_REQUEST, total_pages)}/{total_pages}",
|
||||
"paragraph": done_chunks,
|
||||
"total_paragraphs": len(chunks),
|
||||
"progress_override": pct,
|
||||
}
|
||||
)
|
||||
|
||||
extracted = sum(1 for t in page_texts if t.strip())
|
||||
logger.info(
|
||||
"mistral_ocr_extracted",
|
||||
pages_with_text=extracted,
|
||||
total_pages=total_pages,
|
||||
)
|
||||
return page_texts
|
||||
@@ -96,38 +96,47 @@ class ProvidersConfig:
|
||||
DEEPSEEK_MAX_RETRIES: int = int(os.getenv("DEEPSEEK_MAX_RETRIES", "3"))
|
||||
DEEPSEEK_RETRY_DELAY: float = float(os.getenv("DEEPSEEK_RETRY_DELAY", "1.0"))
|
||||
|
||||
# Minimax (direct API - m2.7, MiniMax-M1)
|
||||
# Minimax (public OpenAI-compatible API - https://api.minimax.io)
|
||||
MINIMAX_ENABLED: bool = os.getenv("MINIMAX_ENABLED", "false").lower() == "true"
|
||||
MINIMAX_API_KEY: str = os.getenv("MINIMAX_API_KEY", "")
|
||||
MINIMAX_MODEL: str = os.getenv("MINIMAX_MODEL", "MiniMax-M1")
|
||||
MINIMAX_BASE_URL: str = os.getenv("MINIMAX_BASE_URL", "https://api.minimax.chat/v1")
|
||||
MINIMAX_MODEL: str = os.getenv("MINIMAX_MODEL", "MiniMax-M3")
|
||||
MINIMAX_BASE_URL: str = os.getenv("MINIMAX_BASE_URL", "https://api.minimax.io/v1")
|
||||
MINIMAX_GROUP_ID: str = os.getenv("MINIMAX_GROUP_ID", "")
|
||||
MINIMAX_TIMEOUT: int = int(os.getenv("MINIMAX_TIMEOUT", "60"))
|
||||
MINIMAX_MAX_RETRIES: int = int(os.getenv("MINIMAX_MAX_RETRIES", "3"))
|
||||
MINIMAX_RETRY_DELAY: float = float(os.getenv("MINIMAX_RETRY_DELAY", "1.0"))
|
||||
|
||||
# Fallback chain configuration
|
||||
# General fallback chain (backward compatibility)
|
||||
#
|
||||
# IMPORTANT: the registry-based fallback (translate_with_fallback) only
|
||||
# ever sees providers that _auto_register_providers() registers, i.e.
|
||||
# google, deepl, openai, deepseek and minimax. The OpenAI-compatible
|
||||
# shims (openrouter, openrouter_premium, zai) and google_cloud are wired
|
||||
# directly in routes/translate_routes.py and are intentionally NOT part of
|
||||
# the registry fallback chain — listing them here would make
|
||||
# translate_with_fallback silently skip them with a "provider not
|
||||
# registered" log on every call. Override via env if you know what you
|
||||
# are doing.
|
||||
FALLBACK_CHAIN: List[str] = [
|
||||
name.strip()
|
||||
for name in os.getenv(
|
||||
"PROVIDER_FALLBACK_CHAIN", "google,google_cloud,deepl,openrouter,openrouter_premium,openai,deepseek,zai"
|
||||
"PROVIDER_FALLBACK_CHAIN", "google,deepl,openai,deepseek,minimax"
|
||||
).split(",")
|
||||
if name.strip()
|
||||
]
|
||||
|
||||
# Mode-specific fallback chains
|
||||
# Classic mode: Google Translate -> Google Cloud -> DeepL
|
||||
# Classic mode: Google Translate -> DeepL
|
||||
FALLBACK_CHAIN_CLASSIC: List[str] = [
|
||||
name.strip()
|
||||
for name in os.getenv("FALLBACK_CHAIN_CLASSIC", "google,google_cloud,deepl").split(",")
|
||||
for name in os.getenv("FALLBACK_CHAIN_CLASSIC", "google,deepl").split(",")
|
||||
if name.strip()
|
||||
]
|
||||
|
||||
# LLM mode: cloud providers in order of cost/quality (no Ollama by default)
|
||||
# LLM mode: cloud providers in order of cost/quality (registry-registered only)
|
||||
FALLBACK_CHAIN_LLM: List[str] = [
|
||||
name.strip()
|
||||
for name in os.getenv("FALLBACK_CHAIN_LLM", "openrouter,openrouter_premium,openai,deepseek,zai").split(",")
|
||||
for name in os.getenv("FALLBACK_CHAIN_LLM", "openai,deepseek,minimax").split(",")
|
||||
if name.strip()
|
||||
]
|
||||
|
||||
|
||||
@@ -39,16 +39,10 @@ Rules:
|
||||
|
||||
|
||||
def _get_language_name(code: str) -> str:
|
||||
language_names = {
|
||||
"en": "English", "fr": "French", "es": "Spanish", "de": "German",
|
||||
"it": "Italian", "pt": "Portuguese", "nl": "Dutch", "ru": "Russian",
|
||||
"zh": "Chinese", "ja": "Japanese", "ko": "Korean", "ar": "Arabic",
|
||||
"hi": "Hindi", "tr": "Turkish", "pl": "Polish", "vi": "Vietnamese",
|
||||
"th": "Thai", "uk": "Ukrainian", "cs": "Czech", "sv": "Swedish",
|
||||
"ro": "Romanian", "hu": "Hungarian", "el": "Greek", "he": "Hebrew",
|
||||
}
|
||||
return language_names.get(code.split("-")[0].lower(), code)
|
||||
"""Convert language code to full name (all supported languages)."""
|
||||
from core.languages import language_name
|
||||
|
||||
return language_name(code)
|
||||
|
||||
class DeepSeekProviderError(Exception):
|
||||
def __init__(self, code: str, message: str, details: Optional[Dict[str, Any]] = None):
|
||||
@@ -161,9 +155,19 @@ class DeepSeekTranslationProvider(TranslationProvider):
|
||||
source_lang_name = _get_language_name(source_language)
|
||||
target_lang_name = _get_language_name(target_language)
|
||||
custom_prompt = request.metadata.get("custom_prompt") if request.metadata else None
|
||||
system_prompt = custom_prompt or DEFAULT_TRANSLATION_PROMPT.format(
|
||||
_base_prompt = DEFAULT_TRANSLATION_PROMPT.format(
|
||||
source_lang=source_lang_name, target_lang=target_lang_name
|
||||
)
|
||||
# Base translation instructions always present; the custom prompt
|
||||
# (glossary/tone/context) is appended, never a replacement.
|
||||
if custom_prompt and custom_prompt.strip():
|
||||
system_prompt = (
|
||||
_base_prompt
|
||||
+ "\n\nADDITIONAL CONTEXT AND INSTRUCTIONS:\n"
|
||||
+ custom_prompt.strip()
|
||||
)
|
||||
else:
|
||||
system_prompt = _base_prompt
|
||||
|
||||
last_error = None
|
||||
for attempt in range(self._max_retries + 1):
|
||||
|
||||
@@ -17,7 +17,15 @@ import time
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
_HAS_STRUCTLOG = True
|
||||
|
||||
# Detect structlog rather than hardcoding True, so the stdlib-logging fallback
|
||||
# branches below stay reachable if structlog is ever absent.
|
||||
try:
|
||||
import structlog # noqa: F401
|
||||
|
||||
_HAS_STRUCTLOG = True
|
||||
except ImportError: # pragma: no cover - structlog is a hard project dep
|
||||
_HAS_STRUCTLOG = False
|
||||
|
||||
|
||||
def _log_info(event: str, **kwargs):
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
"""
|
||||
Minimax Provider - Cloud LLM translation via Minimax API (m2.7).
|
||||
Minimax Provider - Cloud LLM translation via the Minimax public API.
|
||||
|
||||
Minimax uses an OpenAI-compatible Chat Completions API.
|
||||
Minimax exposes an OpenAI-compatible Chat Completions API at
|
||||
``https://api.minimax.io/v1/chat/completions`` (default model ``MiniMax-M3``).
|
||||
Note: ``api.minimax.chat`` is NOT a reachable public host.
|
||||
"""
|
||||
|
||||
import threading
|
||||
@@ -39,16 +41,10 @@ Rules:
|
||||
|
||||
|
||||
def _get_language_name(code: str) -> str:
|
||||
language_names = {
|
||||
"en": "English", "fr": "French", "es": "Spanish", "de": "German",
|
||||
"it": "Italian", "pt": "Portuguese", "nl": "Dutch", "ru": "Russian",
|
||||
"zh": "Chinese", "ja": "Japanese", "ko": "Korean", "ar": "Arabic",
|
||||
"hi": "Hindi", "tr": "Turkish", "pl": "Polish", "vi": "Vietnamese",
|
||||
"th": "Thai", "uk": "Ukrainian", "cs": "Czech", "sv": "Swedish",
|
||||
"ro": "Romanian", "hu": "Hungarian", "el": "Greek", "he": "Hebrew",
|
||||
}
|
||||
return language_names.get(code.split("-")[0].lower(), code)
|
||||
"""Convert language code to full name (all supported languages)."""
|
||||
from core.languages import language_name
|
||||
|
||||
return language_name(code)
|
||||
|
||||
class MinimaxProviderError(Exception):
|
||||
def __init__(self, code: str, message: str, details: Optional[Dict[str, Any]] = None):
|
||||
@@ -62,17 +58,19 @@ class MinimaxTranslationProvider(TranslationProvider):
|
||||
"""
|
||||
Minimax translation provider using OpenAI-compatible API.
|
||||
|
||||
Default model: MiniMax-M1 (latest). Also supports m2.7 via env config.
|
||||
Default model: MiniMax-M3 (latest public OpenAI-compatible model).
|
||||
The public endpoint is https://api.minimax.io/v1 (NOT api.minimax.chat,
|
||||
which is not a reachable host on the public API).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_key: str,
|
||||
model: str = "MiniMax-M1",
|
||||
model: str = "MiniMax-M3",
|
||||
timeout: int = 60,
|
||||
max_retries: int = 3,
|
||||
retry_delay: float = 1.0,
|
||||
base_url: str = "https://api.minimax.chat/v1",
|
||||
base_url: str = "https://api.minimax.io/v1",
|
||||
group_id: str = "",
|
||||
):
|
||||
if not api_key or not api_key.strip():
|
||||
@@ -144,13 +142,24 @@ class MinimaxTranslationProvider(TranslationProvider):
|
||||
def get_name(self) -> str:
|
||||
return self._provider_name
|
||||
|
||||
def is_available(self) -> bool:
|
||||
def _probe_available(self) -> tuple[bool, int]:
|
||||
"""Probe the Minimax API. Returns (available, status_code).
|
||||
|
||||
Minimax does not document a public ``GET /models`` endpoint, so a 404/405
|
||||
on that path does NOT mean the provider is down — it only means the path
|
||||
is absent. We treat any non-401 response as "available": the host is
|
||||
reachable and the API key was not rejected. Only 401 (and network
|
||||
errors) mark the provider unavailable.
|
||||
"""
|
||||
try:
|
||||
headers = {"Authorization": f"Bearer {self._api_key}"}
|
||||
response = requests.get(f"{self._base_url}/models", headers=headers, timeout=5)
|
||||
return response.status_code == 200
|
||||
return response.status_code != 401, response.status_code
|
||||
except Exception:
|
||||
return False
|
||||
return False, 0
|
||||
|
||||
def is_available(self) -> bool:
|
||||
return self._probe_available()[0]
|
||||
|
||||
def translate_text(self, request: TranslationRequest) -> TranslationResponse:
|
||||
text = request.text
|
||||
@@ -163,9 +172,19 @@ class MinimaxTranslationProvider(TranslationProvider):
|
||||
source_lang_name = _get_language_name(source_language)
|
||||
target_lang_name = _get_language_name(target_language)
|
||||
custom_prompt = request.metadata.get("custom_prompt") if request.metadata else None
|
||||
system_prompt = custom_prompt or DEFAULT_TRANSLATION_PROMPT.format(
|
||||
_base_prompt = DEFAULT_TRANSLATION_PROMPT.format(
|
||||
source_lang=source_lang_name, target_lang=target_lang_name
|
||||
)
|
||||
# Base translation instructions always present; the custom prompt
|
||||
# (glossary/tone/context) is appended, never a replacement.
|
||||
if custom_prompt and custom_prompt.strip():
|
||||
system_prompt = (
|
||||
_base_prompt
|
||||
+ "\n\nADDITIONAL CONTEXT AND INSTRUCTIONS:\n"
|
||||
+ custom_prompt.strip()
|
||||
)
|
||||
else:
|
||||
system_prompt = _base_prompt
|
||||
|
||||
last_error = None
|
||||
for attempt in range(self._max_retries + 1):
|
||||
@@ -199,20 +218,19 @@ class MinimaxTranslationProvider(TranslationProvider):
|
||||
|
||||
def health_check(self) -> ProviderHealthStatus:
|
||||
start_time = time.time()
|
||||
try:
|
||||
headers = {"Authorization": f"Bearer {self._api_key}"}
|
||||
response = requests.get(f"{self._base_url}/models", headers=headers, timeout=5)
|
||||
latency_ms = (time.time() - start_time) * 1000
|
||||
available, status_code = self._probe_available()
|
||||
latency_ms = (time.time() - start_time) * 1000
|
||||
if available:
|
||||
return ProviderHealthStatus(
|
||||
name=self._provider_name, available=response.status_code == 200,
|
||||
name=self._provider_name, available=True,
|
||||
latency_ms=round(latency_ms, 2), last_check=datetime.now(timezone.utc).isoformat(),
|
||||
model=self._model)
|
||||
except Exception as e:
|
||||
return ProviderHealthStatus(
|
||||
name=self._provider_name, available=False,
|
||||
latency_ms=round((time.time() - start_time) * 1000, 2),
|
||||
error=str(e)[:100], last_check=datetime.now(timezone.utc).isoformat(),
|
||||
model=self._model)
|
||||
return ProviderHealthStatus(
|
||||
name=self._provider_name, available=False,
|
||||
latency_ms=round(latency_ms, 2),
|
||||
error=f"probe failed (status={status_code})"[:100],
|
||||
last_check=datetime.now(timezone.utc).isoformat(),
|
||||
model=self._model)
|
||||
|
||||
|
||||
_provider_instance: Optional[MinimaxTranslationProvider] = None
|
||||
|
||||
@@ -22,7 +22,16 @@ from typing import Any, Dict, List, Optional
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
_HAS_STRUCTLOG = True
|
||||
|
||||
# structlog is the project's logger backend (see core/logging.py), but detect
|
||||
# it rather than hardcoding True so the stdlib-logging fallback branches below
|
||||
# are reachable if structlog is ever absent.
|
||||
try:
|
||||
import structlog # noqa: F401
|
||||
|
||||
_HAS_STRUCTLOG = True
|
||||
except ImportError: # pragma: no cover - structlog is a hard project dep
|
||||
_HAS_STRUCTLOG = False
|
||||
|
||||
|
||||
def _log_info(event: str, **kwargs):
|
||||
@@ -111,56 +120,26 @@ Rules:
|
||||
def _build_system_prompt(
|
||||
source_lang: str, target_lang: str, custom_prompt: Optional[str] = None
|
||||
) -> str:
|
||||
"""Build system prompt for translation."""
|
||||
if custom_prompt:
|
||||
return custom_prompt
|
||||
return DEFAULT_TRANSLATION_PROMPT.format(
|
||||
"""Build system prompt for translation.
|
||||
|
||||
The base translation instructions are ALWAYS present — a custom prompt
|
||||
(glossary, tone, context) is appended as additional directives, never a
|
||||
replacement. Previously a glossary-only custom prompt produced a system
|
||||
prompt with no translation instruction at all.
|
||||
"""
|
||||
base = DEFAULT_TRANSLATION_PROMPT.format(
|
||||
source_lang=source_lang, target_lang=target_lang
|
||||
)
|
||||
if custom_prompt and custom_prompt.strip():
|
||||
return f"{base}\n\nADDITIONAL CONTEXT AND INSTRUCTIONS:\n{custom_prompt.strip()}"
|
||||
return base
|
||||
|
||||
|
||||
def _get_language_name(code: str) -> str:
|
||||
"""Convert language code to full name for better LLM understanding."""
|
||||
language_names = {
|
||||
"en": "English",
|
||||
"fr": "French",
|
||||
"es": "Spanish",
|
||||
"de": "German",
|
||||
"it": "Italian",
|
||||
"pt": "Portuguese",
|
||||
"nl": "Dutch",
|
||||
"ru": "Russian",
|
||||
"zh": "Chinese",
|
||||
"ja": "Japanese",
|
||||
"ko": "Korean",
|
||||
"ar": "Arabic",
|
||||
"hi": "Hindi",
|
||||
"tr": "Turkish",
|
||||
"pl": "Polish",
|
||||
"vi": "Vietnamese",
|
||||
"th": "Thai",
|
||||
"id": "Indonesian",
|
||||
"ms": "Malay",
|
||||
"uk": "Ukrainian",
|
||||
"cs": "Czech",
|
||||
"sv": "Swedish",
|
||||
"da": "Danish",
|
||||
"fi": "Finnish",
|
||||
"no": "Norwegian",
|
||||
"el": "Greek",
|
||||
"he": "Hebrew",
|
||||
"ro": "Romanian",
|
||||
"hu": "Hungarian",
|
||||
"bg": "Bulgarian",
|
||||
"sk": "Slovak",
|
||||
"hr": "Croatian",
|
||||
"sl": "Slovenian",
|
||||
"lt": "Lithuanian",
|
||||
"lv": "Latvian",
|
||||
"et": "Estonian",
|
||||
}
|
||||
base_code = code.split("-")[0].lower()
|
||||
return language_names.get(base_code, code)
|
||||
from core.languages import language_name
|
||||
|
||||
return language_name(code)
|
||||
|
||||
|
||||
class OpenAITranslationProvider(TranslationProvider):
|
||||
@@ -529,21 +508,120 @@ class OpenAITranslationProvider(TranslationProvider):
|
||||
error_code=OPENAI_SERVICE_ERROR,
|
||||
)
|
||||
|
||||
def _make_batch_api_request(
|
||||
self, texts: List[str], system_prompt: str
|
||||
) -> Optional[List[str]]:
|
||||
"""Translate a whole chunk in ONE request via a numbered JSON list.
|
||||
|
||||
The user message is a JSON array; the model must answer with a JSON
|
||||
array of the same length. Returns None when the answer cannot be
|
||||
parsed confidently — callers then fall back to per-item calls
|
||||
(correctness over latency).
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
numbered = _json.dumps(
|
||||
[{"id": i, "text": t} for i, t in enumerate(texts)],
|
||||
ensure_ascii=False,
|
||||
)
|
||||
batch_system = (
|
||||
system_prompt
|
||||
+ "\n\nBATCH MODE: the user message is a JSON array of items with "
|
||||
"unique ids. Answer with ONLY a JSON array of objects "
|
||||
'[{"id": <same id>, "translation": "<translated text>"}], same '
|
||||
"length and same ids, in the same order. Translate every item; "
|
||||
"keep ids unchanged; no comments, no markdown fence."
|
||||
)
|
||||
|
||||
try:
|
||||
content, _usage = self._make_api_request(numbered, batch_system)
|
||||
raw = content.strip()
|
||||
# Strip an optional markdown fence
|
||||
if raw.startswith("```"):
|
||||
raw = raw.strip("`")
|
||||
if raw.lower().startswith("json"):
|
||||
raw = raw[4:]
|
||||
raw = raw.strip()
|
||||
parsed = _json.loads(raw)
|
||||
if not isinstance(parsed, list) or len(parsed) != len(texts):
|
||||
return None
|
||||
out: List[str] = [""] * len(texts)
|
||||
for item in parsed:
|
||||
if not isinstance(item, dict):
|
||||
return None
|
||||
idx = item.get("id")
|
||||
translation = item.get("translation")
|
||||
if not isinstance(idx, int) or not 0 <= idx < len(texts):
|
||||
return None
|
||||
if not isinstance(translation, str) or not translation.strip():
|
||||
return None
|
||||
out[idx] = translation.strip()
|
||||
return out
|
||||
except OpenAIProviderError:
|
||||
raise
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def translate_batch(
|
||||
self, requests: List[TranslationRequest]
|
||||
) -> List[TranslationResponse]:
|
||||
"""
|
||||
Translate multiple texts.
|
||||
|
||||
Args:
|
||||
requests: List of TranslationRequest objects
|
||||
|
||||
Returns:
|
||||
List of TranslationResponse objects
|
||||
Chunks arrive from the translators as ~15 texts. When every request
|
||||
shares the same language pair and metadata, they are sent in ONE
|
||||
call (numbered JSON list — ~15× fewer requests, better contextual
|
||||
consistency across neighbouring segments). Any parse/API doubt falls
|
||||
back to the per-item path so a batch failure never corrupts output.
|
||||
"""
|
||||
if not requests:
|
||||
return []
|
||||
|
||||
same_pair = len({(r.source_language, r.target_language) for r in requests}) == 1
|
||||
same_meta = len(
|
||||
{tuple(sorted((r.metadata or {}).items())) for r in requests}
|
||||
) == 1
|
||||
|
||||
if same_pair and same_meta and len(requests) > 1:
|
||||
try:
|
||||
source_lang_name = _get_language_name(
|
||||
requests[0].source_language or "auto"
|
||||
) or "the source language (auto-detect)"
|
||||
target_lang_name = _get_language_name(requests[0].target_language)
|
||||
custom_prompt = None
|
||||
if requests[0].metadata:
|
||||
custom_prompt = requests[0].metadata.get("custom_prompt")
|
||||
system_prompt = _build_system_prompt(
|
||||
source_lang_name, target_lang_name, custom_prompt
|
||||
)
|
||||
texts = [r.text for r in requests]
|
||||
translations = self._make_batch_api_request(texts, system_prompt)
|
||||
if translations is not None:
|
||||
_log_info(
|
||||
"openai_batch_translation_success",
|
||||
items=len(requests),
|
||||
model=self._model,
|
||||
)
|
||||
return [
|
||||
TranslationResponse(
|
||||
translated_text=t,
|
||||
provider_name=self._provider_name,
|
||||
from_cache=False,
|
||||
)
|
||||
for t in translations
|
||||
]
|
||||
_log_warning(
|
||||
"openai_batch_translation_fallback",
|
||||
reason="unparseable_response",
|
||||
items=len(requests),
|
||||
)
|
||||
except Exception as e:
|
||||
_log_warning(
|
||||
"openai_batch_translation_fallback",
|
||||
reason=type(e).__name__,
|
||||
items=len(requests),
|
||||
)
|
||||
|
||||
return [self.translate_text(req) for req in requests]
|
||||
|
||||
def health_check(self) -> ProviderHealthStatus:
|
||||
|
||||
119
services/quality/qa_report.py
Normal file
119
services/quality/qa_report.py
Normal file
@@ -0,0 +1,119 @@
|
||||
"""
|
||||
Post-translation QA report (no external API — pure heuristics).
|
||||
|
||||
Answers three user-facing questions about a finished job:
|
||||
- Are numbers preserved? (digit-token multiset source vs translation)
|
||||
- Did everything actually get translated? (untranslated-ratio heuristic)
|
||||
- A 0-100 confidence score combining both.
|
||||
|
||||
Never blocks a job: every failure degrades to "skipped".
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
# Tokens that are never counted as "content words" for the untranslated ratio
|
||||
_PUNCT_RE = re.compile(r"[^\w\s]", re.UNICODE)
|
||||
_WORD_RE = re.compile(r"[\w']+", re.UNICODE)
|
||||
_NUM_RE = re.compile(r"\d+(?:[.,]\d+)*", re.UNICODE)
|
||||
|
||||
# Latin-script vs non-Latin word detection for language confusion heuristics
|
||||
_LATIN_RE = re.compile(r"[a-zA-Z]")
|
||||
|
||||
|
||||
def _extract_text_pairs(source_path: Path, output_path: Path, file_extension: str):
|
||||
"""Extract (source_text, translated_text) full-document strings.
|
||||
|
||||
Reuses the quality layer's file extractor so every format is read the
|
||||
same way the L0 check reads it. Applied to the INPUT file the same
|
||||
extractor yields the SOURCE text (the "translated" field simply holds
|
||||
whatever text lives in the file).
|
||||
"""
|
||||
from services.quality.file_extractor import extract_sample
|
||||
|
||||
src_chunks = extract_sample(Path(source_path), file_extension, max_samples=10_000)
|
||||
out_chunks = extract_sample(Path(output_path), file_extension, max_samples=10_000)
|
||||
src = "\n".join(c["translated"] for c in src_chunks)
|
||||
out = "\n".join(c["translated"] for c in out_chunks)
|
||||
return src, out
|
||||
|
||||
|
||||
def _number_multiset(text: str) -> List[str]:
|
||||
"""Digit tokens with the decimal separator normalized (12,50 == 12.50).
|
||||
|
||||
French/English differ on ',' vs '.'; a real translation keeps the value.
|
||||
"""
|
||||
return sorted(n.replace(",", ".") for n in _NUM_RE.findall(text))
|
||||
|
||||
|
||||
def _number_fidelity(source: str, translated: str) -> Optional[dict]:
|
||||
"""Compare digit tokens: how many source numbers survived (order-insensitive)."""
|
||||
src_nums = _number_multiset(source)
|
||||
if not src_nums:
|
||||
return None
|
||||
out_nums = _number_multiset(translated)
|
||||
# multiset intersection
|
||||
from collections import Counter
|
||||
|
||||
src_count = Counter(src_nums)
|
||||
out_count = Counter(out_nums)
|
||||
kept = sum((src_count & out_count).values())
|
||||
return {
|
||||
"source_numbers": len(src_nums),
|
||||
"preserved": kept,
|
||||
"fidelity": round(kept / len(src_nums), 3),
|
||||
}
|
||||
|
||||
|
||||
def _untranslated_ratio(source: str, translated: str) -> Optional[float]:
|
||||
"""Heuristic: share of source content-words still present verbatim in
|
||||
the output. ~0 for a real translation, ~1 when nothing was translated.
|
||||
"""
|
||||
src_words = [w.lower() for w in _WORD_RE.findall(source) if len(w) > 3]
|
||||
if len(src_words) < 10:
|
||||
return None
|
||||
out_lower = translated.lower()
|
||||
hits = sum(1 for w in set(src_words) if w in out_lower)
|
||||
return round(hits / len(set(src_words)), 3)
|
||||
|
||||
|
||||
def run_qa_report(
|
||||
source_path: Path, output_path: Path, target_lang: str, file_extension: str
|
||||
) -> Optional[Dict]:
|
||||
"""Compute the QA report for a finished translation job.
|
||||
|
||||
Returns a dict with numbers fidelity, untranslated ratio and a
|
||||
0-100 score, or None if the report could not be computed.
|
||||
"""
|
||||
try:
|
||||
source, translated = _extract_text_pairs(
|
||||
Path(source_path), Path(output_path), file_extension
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("qa_report_extract_failed", error=str(e))
|
||||
return None
|
||||
|
||||
if not source.strip() or not translated.strip():
|
||||
return None
|
||||
|
||||
numbers = _number_fidelity(source, translated)
|
||||
untranslated = _untranslated_ratio(source, translated)
|
||||
|
||||
score = 100.0
|
||||
if numbers:
|
||||
score *= 0.5 + 0.5 * numbers["fidelity"]
|
||||
if untranslated is not None and untranslated > 0:
|
||||
score *= max(0.0, 1.0 - untranslated)
|
||||
|
||||
report = {
|
||||
"score": int(round(score)),
|
||||
"numbers": numbers,
|
||||
"untranslated_ratio": untranslated,
|
||||
}
|
||||
logger.info("qa_report_computed", **{k: v for k, v in report.items() if v is not None})
|
||||
return report
|
||||
@@ -23,23 +23,11 @@ from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
# Map language codes to full names for LLM prompts (models understand "French" better than "fr")
|
||||
_LLM_LANG_NAMES = {
|
||||
"en": "English", "es": "Spanish", "de": "German", "fr": "French", "ja": "Japanese",
|
||||
"pt": "Portuguese", "ru": "Russian", "it": "Italian", "zh": "Chinese", "zh-CN": "Chinese (Simplified)",
|
||||
"zh-TW": "Chinese (Traditional)", "pl": "Polish", "nl": "Dutch", "tr": "Turkish", "ko": "Korean",
|
||||
"ar": "Arabic", "fa": "Persian", "vi": "Vietnamese", "id": "Indonesian", "uk": "Ukrainian",
|
||||
"sv": "Swedish", "cs": "Czech", "el": "Greek", "he": "Hebrew", "hi": "Hindi", "ro": "Romanian",
|
||||
"da": "Danish", "fi": "Finnish", "no": "Norwegian", "hu": "Hungarian", "th": "Thai",
|
||||
"sk": "Slovak", "bg": "Bulgarian", "hr": "Croatian", "ca": "Catalan", "ms": "Malay",
|
||||
}
|
||||
|
||||
|
||||
def _lang_name(code: str) -> str:
|
||||
"""Return full language name for LLM prompts; fallback to code if unknown."""
|
||||
if not code or code == "auto":
|
||||
return ""
|
||||
return _LLM_LANG_NAMES.get(code, _LLM_LANG_NAMES.get(code.split("-")[0], code))
|
||||
from core.languages import language_name
|
||||
|
||||
return language_name(code)
|
||||
|
||||
|
||||
# Global thread pool for parallel translations
|
||||
@@ -1191,13 +1179,16 @@ class TranslationService:
|
||||
if not self.translate_images:
|
||||
return ""
|
||||
|
||||
# Ollama, OpenAI, and OpenRouter support image translation
|
||||
if isinstance(self.provider, OllamaTranslationProvider):
|
||||
return self.provider.translate_image(image_path, target_language)
|
||||
elif isinstance(self.provider, OpenAITranslationProvider):
|
||||
return self.provider.translate_image(image_path, target_language)
|
||||
elif isinstance(self.provider, OpenRouterTranslationProvider):
|
||||
return self.provider.translate_image(image_path, target_language)
|
||||
# Duck-typing: any provider that exposes ``translate_image`` can be
|
||||
# used, regardless of whether it is a new-style (services/providers/*)
|
||||
# or legacy (services/translation_service) instance. The previous
|
||||
# isinstance() checks only matched the legacy classes, so a new-style
|
||||
# provider wired in by the route never reached this branch.
|
||||
if hasattr(self.provider, "translate_image"):
|
||||
try:
|
||||
return self.provider.translate_image(image_path, target_language)
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
199
services/translation_tm.py
Normal file
199
services/translation_tm.py
Normal file
@@ -0,0 +1,199 @@
|
||||
"""
|
||||
Translation Memory (TM) — persistent reuse layer on top of the existing
|
||||
Redis-backed cache (services/translation_cache.py).
|
||||
|
||||
Scope is PER USER (privacy: a customer's translations are never served to
|
||||
another customer) and per translation context (custom prompt / glossary /
|
||||
formality are hashed into the key so a glossary change invalidates old
|
||||
matches). When Redis is not configured the cache transparently falls back
|
||||
to a process-local LRU — still useful within a worker's lifetime.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class TMScope:
|
||||
"""Identity/context of a translation job for TM keying."""
|
||||
|
||||
__slots__ = ("user_id", "context_hash")
|
||||
|
||||
def __init__(self, user_id: Optional[str], context_hash: Optional[str]):
|
||||
self.user_id = user_id
|
||||
self.context_hash = context_hash
|
||||
|
||||
@classmethod
|
||||
def from_prompt(cls, user_id: Optional[str], prompt: Optional[str]) -> "TMScope":
|
||||
"""Build a scope; the prompt (glossary + tone + formality merged)
|
||||
is hashed so any directive change produces different TM entries."""
|
||||
context_hash = None
|
||||
if prompt:
|
||||
context_hash = hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:16]
|
||||
return cls(user_id=user_id, context_hash=context_hash)
|
||||
|
||||
|
||||
def _cache():
|
||||
from services.translation_cache import get_cache
|
||||
|
||||
return get_cache()
|
||||
|
||||
|
||||
def lookup_tm(
|
||||
texts: List[str],
|
||||
target_language: str,
|
||||
source_language: str,
|
||||
provider_name: str,
|
||||
scope: Optional[TMScope],
|
||||
) -> Tuple[Dict[int, str], List[int]]:
|
||||
"""Return ({index: cached_translation}, [indices not in TM]).
|
||||
|
||||
No-op (all misses) when the scope has no user_id — anonymous jobs are
|
||||
never cached or served from another user's entries.
|
||||
"""
|
||||
if not scope or not scope.user_id:
|
||||
return {}, list(range(len(texts)))
|
||||
|
||||
try:
|
||||
cache = _cache()
|
||||
except Exception as e:
|
||||
logger.warning("tm_init_failed", error=str(e))
|
||||
return {}, list(range(len(texts)))
|
||||
|
||||
hits: Dict[int, str] = {}
|
||||
misses: List[int] = []
|
||||
for i, text in enumerate(texts):
|
||||
if not text or not text.strip():
|
||||
misses.append(i)
|
||||
continue
|
||||
cached = cache.get(
|
||||
text,
|
||||
target_language,
|
||||
source_language,
|
||||
provider_name,
|
||||
user_id=scope.user_id,
|
||||
custom_prompt_hash=scope.context_hash,
|
||||
)
|
||||
if cached is not None and cached.strip():
|
||||
hits[i] = cached
|
||||
else:
|
||||
misses.append(i)
|
||||
|
||||
if hits:
|
||||
logger.info("tm_lookup_hits", hits=len(hits), misses=len(misses))
|
||||
return hits, misses
|
||||
|
||||
|
||||
def store_tm(
|
||||
texts: List[str],
|
||||
translations: List[str],
|
||||
target_language: str,
|
||||
source_language: str,
|
||||
provider_name: str,
|
||||
scope: Optional[TMScope],
|
||||
) -> int:
|
||||
"""Store fresh translations in the TM. Returns the number stored."""
|
||||
if not scope or not scope.user_id:
|
||||
return 0
|
||||
try:
|
||||
cache = _cache()
|
||||
except Exception as e:
|
||||
logger.warning("tm_init_failed", error=str(e))
|
||||
return 0
|
||||
|
||||
stored = 0
|
||||
for text, translation in zip(texts, translations):
|
||||
if not text or not translation:
|
||||
continue
|
||||
# Never store identity entries — they would poison future lookups
|
||||
# (an unchanged text is not a translation).
|
||||
if translation.strip() == text.strip():
|
||||
continue
|
||||
cache.set(
|
||||
text,
|
||||
target_language,
|
||||
source_language,
|
||||
provider_name,
|
||||
translation,
|
||||
user_id=scope.user_id,
|
||||
custom_prompt_hash=scope.context_hash,
|
||||
)
|
||||
stored += 1
|
||||
if stored:
|
||||
logger.info("tm_stored", entries=stored)
|
||||
return stored
|
||||
|
||||
|
||||
def tm_stats() -> Dict:
|
||||
"""Backend stats for observability."""
|
||||
try:
|
||||
return _cache().stats()
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def translate_with_tm(
|
||||
texts: List[str],
|
||||
target_language: str,
|
||||
source_language: str,
|
||||
provider_name: str,
|
||||
scope: "TMScope | None",
|
||||
translate_fn,
|
||||
) -> List[str]:
|
||||
"""Translate a batch with TM reuse: hits come from the cache, misses go
|
||||
through ``translate_fn`` (which receives ONLY the missed texts, in
|
||||
order) and fresh results are stored back. Falls back to a plain
|
||||
``translate_fn(texts)`` call when the TM is unavailable.
|
||||
"""
|
||||
if not texts:
|
||||
return []
|
||||
|
||||
tm_hits, miss_indices = lookup_tm(
|
||||
texts, target_language, source_language, provider_name, scope
|
||||
)
|
||||
miss_texts = [texts[i] for i in miss_indices]
|
||||
|
||||
if not miss_texts:
|
||||
return [tm_hits.get(i, texts[i]) for i in range(len(texts))]
|
||||
|
||||
try:
|
||||
translated_misses = translate_fn(miss_texts)
|
||||
except Exception:
|
||||
if tm_hits:
|
||||
# Provider failed but we have partial TM hits — keep them and
|
||||
# leave the misses untouched rather than dropping everything.
|
||||
logger.warning("tm_translate_fn_failed_partial_hits", hits=len(tm_hits))
|
||||
translated_misses = None
|
||||
else:
|
||||
raise
|
||||
|
||||
if translated_misses is None:
|
||||
return [
|
||||
tm_hits.get(i, texts[i]) for i in range(len(texts))
|
||||
]
|
||||
|
||||
store_tm(
|
||||
miss_texts,
|
||||
translated_misses,
|
||||
target_language,
|
||||
source_language,
|
||||
provider_name,
|
||||
scope,
|
||||
)
|
||||
|
||||
merged: List[str] = []
|
||||
miss_pos = 0
|
||||
for i in range(len(texts)):
|
||||
if i in tm_hits:
|
||||
merged.append(tm_hits[i])
|
||||
else:
|
||||
merged.append(
|
||||
translated_misses[miss_pos]
|
||||
if miss_pos < len(translated_misses)
|
||||
else texts[i]
|
||||
)
|
||||
miss_pos += 1
|
||||
return merged
|
||||
@@ -101,7 +101,11 @@ def _get_redis_patcher(mock_redis):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_orphan_deletion(temp_dirs):
|
||||
"""Test that orphaned files are deleted as per Story 2.15 (AC: #4)"""
|
||||
"""Test that orphaned files are deleted as per Story 2.15 (AC: #4).
|
||||
|
||||
Since the security fix (audit 2026-08-26), an orphan is only deleted
|
||||
after a grace period, so young in-flight files are never removed.
|
||||
"""
|
||||
cleanup_mod = _get_cleanup_module()
|
||||
FileCleanupManager = cleanup_mod.FileCleanupManager
|
||||
|
||||
@@ -117,6 +121,10 @@ async def test_orphan_deletion(temp_dirs):
|
||||
|
||||
manager = FileCleanupManager(uploads, outputs, temp, cleanup_interval_minutes=5)
|
||||
|
||||
# Make the orphan older than the grace period (default 15 min)
|
||||
old = time.time() - (manager.orphan_grace_seconds + 60)
|
||||
os.utime(orphan_file, (old, old))
|
||||
|
||||
mock_redis = AsyncMock()
|
||||
mock_redis.keys.return_value = ["translation:file:job1"]
|
||||
mock_redis.get.return_value = json.dumps(
|
||||
@@ -182,6 +190,11 @@ async def test_cleanup_resilience(temp_dirs):
|
||||
f2 = uploads / "file2.txt"
|
||||
f2.write_text("file2")
|
||||
|
||||
# Make the files older than the grace period so cleanup actually deletes them
|
||||
old = time.time() - 7200
|
||||
os.utime(f1, (old, old))
|
||||
os.utime(f2, (old, old))
|
||||
|
||||
manager = FileCleanupManager(uploads, outputs, temp, max_file_age_minutes=1)
|
||||
|
||||
original_unlink = Path.unlink
|
||||
|
||||
@@ -17,6 +17,8 @@ DOWNLOAD_URL = "/api/v1/download"
|
||||
REGISTER_URL = "/api/v1/auth/register"
|
||||
LOGIN_URL = "/api/v1/auth/login"
|
||||
|
||||
AUTH_USER_ID = None # set by the authenticated_client fixture
|
||||
|
||||
VALID_USER = {
|
||||
"email": "download@example.com",
|
||||
"password": "Password123!",
|
||||
@@ -135,7 +137,9 @@ def client(users_file: Path, monkeypatch):
|
||||
@pytest.fixture()
|
||||
def authenticated_client(client):
|
||||
"""Client avec un utilisateur enregistre et authentifie."""
|
||||
client.post(REGISTER_URL, json=VALID_USER)
|
||||
global AUTH_USER_ID
|
||||
reg = client.post(REGISTER_URL, json=VALID_USER)
|
||||
AUTH_USER_ID = reg.json()["data"]["id"]
|
||||
response = client.post(
|
||||
LOGIN_URL,
|
||||
json={
|
||||
@@ -184,6 +188,7 @@ class TestDownloadEndpoint:
|
||||
job_id = "tr_test_no_output"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "test.xlsx",
|
||||
"output_path": None,
|
||||
@@ -205,6 +210,7 @@ class TestDownloadEndpoint:
|
||||
job_id = "tr_deleted_disk"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "deleted.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
@@ -224,6 +230,7 @@ class TestDownloadEndpoint:
|
||||
job_id = "tr_test_processing"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "processing",
|
||||
"progress_percent": 50,
|
||||
"file_name": "test.xlsx",
|
||||
@@ -241,6 +248,7 @@ class TestDownloadEndpoint:
|
||||
job_id = "tr_test_queued"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "queued",
|
||||
"file_name": "test.xlsx",
|
||||
}
|
||||
@@ -257,6 +265,7 @@ class TestDownloadEndpoint:
|
||||
job_id = "tr_test_failed"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "failed",
|
||||
"error_message": "Something went wrong",
|
||||
"file_name": "test.xlsx",
|
||||
@@ -288,6 +297,7 @@ class TestContentDisposition:
|
||||
job_id = "tr_test_disposition"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "report.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
@@ -310,6 +320,7 @@ class TestContentDisposition:
|
||||
job_id = "tr_test_docx"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "document.docx",
|
||||
"file_extension": ".docx",
|
||||
@@ -331,6 +342,7 @@ class TestContentDisposition:
|
||||
job_id = "tr_test_pptx"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "presentation.pptx",
|
||||
"file_extension": ".pptx",
|
||||
@@ -364,6 +376,7 @@ class TestFileDeletion:
|
||||
job_id = "tr_test_delete"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "to_delete.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
@@ -400,6 +413,7 @@ class TestMIMETypes:
|
||||
job_id = "tr_test_mime_xlsx"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "test.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
@@ -424,6 +438,7 @@ class TestMIMETypes:
|
||||
job_id = "tr_test_mime_docx"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "test.docx",
|
||||
"file_extension": ".docx",
|
||||
@@ -448,6 +463,7 @@ class TestMIMETypes:
|
||||
job_id = "tr_test_mime_pptx"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "test.pptx",
|
||||
"file_extension": ".pptx",
|
||||
@@ -489,6 +505,7 @@ class TestFileExpired:
|
||||
job_id = "tr_test_not_ready_msg"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "processing",
|
||||
"progress_percent": 30,
|
||||
"file_name": "test.xlsx",
|
||||
@@ -520,11 +537,12 @@ class TestDownloadIntegration:
|
||||
job_id = "tr_test_binary"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "binary_test.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
"output_path": str(output_file),
|
||||
"user_id": None,
|
||||
"user_id": AUTH_USER_ID,
|
||||
}
|
||||
|
||||
response = authenticated_client.get(f"{DOWNLOAD_URL}/{job_id}")
|
||||
@@ -557,6 +575,7 @@ class TestErrorDetails:
|
||||
job_id = "tr_test_details"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "processing",
|
||||
"progress_percent": 45,
|
||||
"file_name": "test.xlsx",
|
||||
@@ -589,6 +608,7 @@ class TestDownloadAuthorization:
|
||||
job_id = "tr_other_user123"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "other.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
@@ -622,18 +642,19 @@ class TestDownloadAuthorization:
|
||||
job_id = "tr_own_file123"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": AUTH_USER_ID,
|
||||
"status": "completed",
|
||||
"file_name": "own.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
"output_path": str(output_file),
|
||||
"user_id": None,
|
||||
"user_id": AUTH_USER_ID,
|
||||
}
|
||||
|
||||
response = authenticated_client.get(f"{DOWNLOAD_URL}/{job_id}")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_anonymous_user_can_download_public_job(self, client, tmp_path):
|
||||
"""Anonymous users can download jobs without user_id (public)"""
|
||||
def test_anonymous_job_requires_token(self, client, tmp_path):
|
||||
"""Jobs without an owner require the secret per-job token (fix 2026-08-26)"""
|
||||
from routes import translate_routes
|
||||
|
||||
output_file = tmp_path / "public_job.xlsx"
|
||||
@@ -642,12 +663,19 @@ class TestDownloadAuthorization:
|
||||
job_id = "tr_public_job99"
|
||||
translate_routes._translation_jobs[job_id] = {
|
||||
"id": job_id,
|
||||
"user_id": None,
|
||||
"access_token": "secret_job_token",
|
||||
"status": "completed",
|
||||
"file_name": "public.xlsx",
|
||||
"file_extension": ".xlsx",
|
||||
"output_path": str(output_file),
|
||||
"user_id": None,
|
||||
}
|
||||
|
||||
# Without the token: denied
|
||||
response = client.get(f"{DOWNLOAD_URL}/{job_id}")
|
||||
assert response.status_code == 403
|
||||
assert response.json()["error"] == "ACCESS_DENIED"
|
||||
|
||||
# With the correct token: allowed
|
||||
response = client.get(f"{DOWNLOAD_URL}/{job_id}?token=secret_job_token")
|
||||
assert response.status_code == 200
|
||||
|
||||
92
tests/test_excel_sheet_rename_refs.py
Normal file
92
tests/test_excel_sheet_rename_refs.py
Normal file
@@ -0,0 +1,92 @@
|
||||
"""Sheet-name translation must not break references.
|
||||
|
||||
openpyxl does not rewrite references on rename — cell formulas, defined
|
||||
names and validations pointing at a translated sheet would break (#REF!).
|
||||
These tests cover the full rename + rewrite path.
|
||||
"""
|
||||
|
||||
from openpyxl import Workbook, load_workbook
|
||||
from openpyxl.workbook.defined_name import DefinedName
|
||||
|
||||
from translators.excel_translator import ExcelTranslator
|
||||
|
||||
|
||||
class _FrToEn:
|
||||
"""Minimal legacy-style provider: French sheet name → English."""
|
||||
|
||||
def translate_batch(self, texts, target_language, source_language="auto"):
|
||||
mapping = {
|
||||
"Ventes": "Sales",
|
||||
"Données": "Data",
|
||||
"Rapport des ventes": "Sales report",
|
||||
"Total": "Total",
|
||||
}
|
||||
return [mapping.get(t, t) for t in texts]
|
||||
|
||||
|
||||
def _make_workbook(path):
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
ws.title = "Ventes"
|
||||
ws["A1"] = "Rapport des ventes"
|
||||
ws["A2"] = "Total"
|
||||
ws["A3"] = 10
|
||||
ws["A4"] = 20
|
||||
# Cross-sheet formula (unquoted name)
|
||||
ws2 = wb.create_sheet("Données")
|
||||
ws2["A1"] = "=SUM(Ventes!A3:A4)"
|
||||
# Quoted name (name would need quotes if it had spaces — keep simple here)
|
||||
ws2["A2"] = "=Ventes!A3+Ventes!A4"
|
||||
# Defined name pointing at the renamed sheet
|
||||
wb.defined_names["TotalVentes"] = DefinedName(
|
||||
"TotalVentes", attr_text="'Ventes'!$A$3"
|
||||
)
|
||||
wb.save(path)
|
||||
return wb
|
||||
|
||||
|
||||
class TestSheetRenameReferences:
|
||||
def test_cell_formulas_and_defined_names_rewritten(self, tmp_path):
|
||||
from pathlib import Path
|
||||
|
||||
src = tmp_path / "in.xlsx"
|
||||
out = tmp_path / "out.xlsx"
|
||||
_make_workbook(src)
|
||||
|
||||
translator = ExcelTranslator(provider=_FrToEn())
|
||||
translator.translate_file(Path(src), Path(out), "en", "fr")
|
||||
|
||||
wb = load_workbook(out)
|
||||
data = wb["Data"]
|
||||
assert data["A1"].value == "=SUM(Sales!A3:A4)", data["A1"].value
|
||||
assert data["A2"].value == "=Sales!A3+Sales!A4", data["A2"].value
|
||||
|
||||
# Defined name follows the rename
|
||||
dn = wb.defined_names["TotalVentes"]
|
||||
assert "Sales" in (dn.attr_text or ""), dn.attr_text
|
||||
assert "Ventes" not in (dn.attr_text or "")
|
||||
|
||||
# The renamed sheet actually exists under its new name
|
||||
assert "Sales" in wb.sheetnames
|
||||
assert "Ventes" not in wb.sheetnames
|
||||
|
||||
def test_3d_and_multiple_refs(self, tmp_path):
|
||||
mapping = {"Sheet1": "Feuille1", "Sheet2": "Feuille2"}
|
||||
formula = "=SUM(Sheet1!A1:Sheet2!B2)+Sheet1!C3"
|
||||
out = ExcelTranslator._rewrite_sheet_refs_in_formula(formula, mapping)
|
||||
assert out == "=SUM(Feuille1!A1:Feuille2!B2)+Feuille1!C3"
|
||||
|
||||
def test_quoted_refs_and_prefix_safety(self, tmp_path):
|
||||
mapping = {"Ventes": "Sales", "Ventes 2026": "Sales 2026"}
|
||||
out = ExcelTranslator._rewrite_sheet_refs_in_formula(
|
||||
"=SUM('Ventes 2026'!A1:A2)+'Ventes'!B1", mapping
|
||||
)
|
||||
# "Ventes 2026" (longest first) keeps its quotes (name with space);
|
||||
# "Sales" needs no quotes so the canonical unquoted form is emitted.
|
||||
assert out == "=SUM('Sales 2026'!A1:A2)+Sales!B1"
|
||||
|
||||
def test_non_sheet_bang_not_touched(self, tmp_path):
|
||||
mapping = {"Ventes": "Sales"}
|
||||
# "Total!A1" is not a renamed sheet — must stay untouched
|
||||
out = ExcelTranslator._rewrite_sheet_refs_in_formula("=Total!A1", mapping)
|
||||
assert out == "=Total!A1"
|
||||
45
tests/test_language_validation.py
Normal file
45
tests/test_language_validation.py
Normal file
@@ -0,0 +1,45 @@
|
||||
"""LanguageValidator — case-insensitive codes and canonical form (zh-CN fix)."""
|
||||
|
||||
import pytest
|
||||
|
||||
from middleware.validation import LanguageValidator, ValidationError
|
||||
|
||||
|
||||
class TestLanguageValidatorCaseInsensitive:
|
||||
def test_zh_cn_mixed_case_accepted(self):
|
||||
assert LanguageValidator.validate("zh-CN") == "zh-CN"
|
||||
|
||||
def test_zh_cn_lowercase_normalized(self):
|
||||
assert LanguageValidator.validate("zh-cn") == "zh-CN"
|
||||
|
||||
def test_zh_cn_uppercase_normalized(self):
|
||||
assert LanguageValidator.validate("ZH-CN") == "zh-CN"
|
||||
|
||||
def test_zh_tw_accepted(self):
|
||||
assert LanguageValidator.validate("zh-TW") == "zh-TW"
|
||||
assert LanguageValidator.validate("zh-tw") == "zh-TW"
|
||||
|
||||
def test_alias_chinese(self):
|
||||
assert LanguageValidator.validate("chinese") == "zh-CN"
|
||||
|
||||
def test_alias_tw(self):
|
||||
assert LanguageValidator.validate("tw") == "zh-TW"
|
||||
|
||||
def test_plain_codes_unchanged(self):
|
||||
assert LanguageValidator.validate("en") == "en"
|
||||
assert LanguageValidator.validate("fr") == "fr"
|
||||
|
||||
def test_auto_accepted(self):
|
||||
assert LanguageValidator.validate("auto") == "auto"
|
||||
|
||||
def test_unknown_code_rejected(self):
|
||||
with pytest.raises(ValidationError):
|
||||
LanguageValidator.validate("xx")
|
||||
|
||||
def test_unknown_variant_rejected(self):
|
||||
with pytest.raises(ValidationError):
|
||||
LanguageValidator.validate("zz-ZZ")
|
||||
|
||||
def test_empty_rejected(self):
|
||||
with pytest.raises(ValidationError):
|
||||
LanguageValidator.validate("")
|
||||
@@ -35,36 +35,6 @@ from prometheus_client import (
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def _load_metrics_module_with_registry(registry: CollectorRegistry):
|
||||
"""Load middleware/metrics.py with patched Counter/Histogram to
|
||||
use the supplied registry. Returns the loaded module."""
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"metrics_under_test",
|
||||
_REPO_ROOT / "middleware" / "metrics.py",
|
||||
)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
# Inject the fresh registry into the module's namespace before exec
|
||||
mod.__dict__["_TEST_REGISTRY"] = registry
|
||||
|
||||
# Patch Counter/Histogram to use the fresh registry
|
||||
orig_counter = Counter
|
||||
orig_histogram = Histogram
|
||||
|
||||
def _counter(*args, **kwargs):
|
||||
kwargs.setdefault("registry", registry)
|
||||
return orig_counter(*args, **kwargs)
|
||||
|
||||
def _histogram(*args, **kwargs):
|
||||
kwargs.setdefault("registry", registry)
|
||||
return orig_histogram(*args, **kwargs)
|
||||
|
||||
mod.__dict__["Counter"] = _counter
|
||||
mod.__dict__["Histogram"] = _histogram
|
||||
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def metrics():
|
||||
"""Load the metrics module ONCE per test module.
|
||||
@@ -80,55 +50,59 @@ def metrics():
|
||||
return _load_metrics_module_with_fresh_registry()
|
||||
|
||||
|
||||
def _load_metrics_module_with_registry(registry: CollectorRegistry):
|
||||
"""Load middleware/metrics.py with ALL its counters/histograms
|
||||
registered on the supplied fresh registry.
|
||||
|
||||
metrics.py does ``from prometheus_client import Counter, Histogram`` at
|
||||
module top — injecting patched classes into the module dict BEFORE exec
|
||||
does not survive that import. The only reliable interception point is
|
||||
the ``prometheus_client`` module itself: we swap its Counter/Histogram
|
||||
attributes for subclasses that default to ``registry``, exec the
|
||||
module source, and restore the originals.
|
||||
"""
|
||||
import types
|
||||
import prometheus_client
|
||||
|
||||
orig_counter = prometheus_client.Counter
|
||||
orig_histogram = prometheus_client.Histogram
|
||||
|
||||
class _RegistryCounter(orig_counter):
|
||||
def __new__(cls, *args, **kwargs):
|
||||
kwargs.setdefault("registry", registry)
|
||||
return super().__new__(cls)
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
kwargs.setdefault("registry", registry)
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
class _RegistryHistogram(orig_histogram):
|
||||
def __new__(cls, *args, **kwargs):
|
||||
kwargs.setdefault("registry", registry)
|
||||
return super().__new__(cls)
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
kwargs.setdefault("registry", registry)
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
source = (_REPO_ROOT / "middleware" / "metrics.py").read_text(encoding="utf-8")
|
||||
mod = types.ModuleType("metrics_under_test")
|
||||
mod.__file__ = str(_REPO_ROOT / "middleware" / "metrics.py")
|
||||
try:
|
||||
prometheus_client.Counter = _RegistryCounter
|
||||
prometheus_client.Histogram = _RegistryHistogram
|
||||
exec(compile(source, mod.__file__, "exec"), mod.__dict__)
|
||||
finally:
|
||||
prometheus_client.Counter = orig_counter
|
||||
prometheus_client.Histogram = orig_histogram
|
||||
return mod
|
||||
|
||||
|
||||
def _load_metrics_module_with_fresh_registry():
|
||||
"""Load metrics module with its counters/histograms attached to a
|
||||
fresh CollectorRegistry."""
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"metrics_under_test",
|
||||
_REPO_ROOT / "middleware" / "metrics.py",
|
||||
)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
|
||||
# The module just created its counters on the default REGISTRY.
|
||||
# Unregister them, then re-create them on a fresh registry.
|
||||
fresh = CollectorRegistry()
|
||||
for name in (
|
||||
"http_requests_total",
|
||||
"translation_total",
|
||||
"translation_duration_seconds",
|
||||
"file_size_bytes",
|
||||
"quality_l0_checks_total",
|
||||
"quality_l1_judge_total",
|
||||
"quality_l1_judge_duration_seconds",
|
||||
"quality_l1_judge_cost_usd",
|
||||
"translation_retry_total",
|
||||
"format_elements_lost_total",
|
||||
):
|
||||
if not hasattr(mod, name):
|
||||
continue
|
||||
obj = getattr(mod, name)
|
||||
try:
|
||||
REGISTRY.unregister(obj)
|
||||
except KeyError:
|
||||
pass
|
||||
|
||||
# Now re-import the module so the new metrics register on `fresh`
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"metrics_under_test_isolated",
|
||||
_REPO_ROOT / "middleware" / "metrics.py",
|
||||
)
|
||||
mod2 = importlib.util.module_from_spec(spec)
|
||||
# We can't easily re-route Counter/Histogram in exec_module because
|
||||
# they call into the global REGISTRY via the function signature.
|
||||
# Instead: reload by re-importing via importlib with a wrapper
|
||||
# that intercepts the Counter/Histogram constructors. We do this
|
||||
# via the more direct route: use the EXISTING counters on the
|
||||
# default REGISTRY, but only check RELATIVE increments.
|
||||
#
|
||||
# Practical approach: just use the module as-is. Tests check
|
||||
# `after >= before + 1`, which is robust against other tests.
|
||||
return mod
|
||||
fresh CollectorRegistry (never the shared default one — the app may
|
||||
already have registered the same names earlier in the session)."""
|
||||
return _load_metrics_module_with_registry(CollectorRegistry())
|
||||
|
||||
|
||||
def _counter_value(counter, **labels):
|
||||
|
||||
344
tests/test_new_features.py
Normal file
344
tests/test_new_features.py
Normal file
@@ -0,0 +1,344 @@
|
||||
"""New feature tests: formality/regional prompts, TM, bilingual output,
|
||||
QA report, OpenAI JSON batching, font hints."""
|
||||
|
||||
import pytest
|
||||
from docx import Document
|
||||
from openpyxl import Workbook
|
||||
|
||||
from services.glossary_service import build_full_prompt
|
||||
from services.quality.qa_report import (
|
||||
_number_fidelity,
|
||||
_untranslated_ratio,
|
||||
run_qa_report,
|
||||
)
|
||||
from services.translation_tm import TMScope, translate_with_tm
|
||||
from services.translation_cache import reset_cache_for_tests
|
||||
from translators.bilingual import make_bilingual_docx
|
||||
from translators.word_translator import WordTranslator, _font_hints_for_target
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Formality + regional variant in prompts
|
||||
# ===========================================================================
|
||||
class TestFormalityPrompt:
|
||||
def test_formal_adds_tone_directive(self):
|
||||
prompt = build_full_prompt(None, None, "fr", "en", formality="formal")
|
||||
assert "TONE:" in prompt and "formal" in prompt
|
||||
|
||||
def test_informal_adds_tone_directive(self):
|
||||
prompt = build_full_prompt(None, None, "fr", "en", formality="informal")
|
||||
assert "TONE:" in prompt and "informal" in prompt
|
||||
|
||||
def test_no_formality_no_tone(self):
|
||||
assert "TONE:" not in build_full_prompt(None, None, "fr", "en")
|
||||
|
||||
def test_regional_variant_auto(self):
|
||||
prompt = build_full_prompt(None, None, "fr", "pt-BR")
|
||||
assert "REGIONAL VARIANT" in prompt
|
||||
assert "Portuguese" in prompt
|
||||
|
||||
def test_no_region_no_variant(self):
|
||||
assert "REGIONAL VARIANT" not in build_full_prompt(None, None, "fr", "en")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Translation memory (per-user scoping)
|
||||
# ===========================================================================
|
||||
class TestTranslationMemory:
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_cache(self):
|
||||
reset_cache_for_tests()
|
||||
yield
|
||||
reset_cache_for_tests()
|
||||
|
||||
def test_no_scope_passthrough(self):
|
||||
calls = []
|
||||
|
||||
def fake_translate(texts):
|
||||
calls.extend(texts)
|
||||
return [t.upper() for t in texts]
|
||||
|
||||
out = translate_with_tm(
|
||||
["a", "b"], "en", "fr", "fake", None, fake_translate
|
||||
)
|
||||
assert out == ["A", "B"]
|
||||
assert calls == ["a", "b"]
|
||||
|
||||
def test_reuse_on_second_call_same_user(self):
|
||||
from services.translation_cache import get_cache
|
||||
|
||||
get_cache() # init LRU backend
|
||||
|
||||
calls = []
|
||||
|
||||
def fake_translate(texts):
|
||||
calls.extend(texts)
|
||||
return [f"T:{t}" for t in texts]
|
||||
|
||||
scope = TMScope.from_prompt("user-1", "ctx")
|
||||
first = translate_with_tm(["hello", "world"], "en", "fr", "fake", scope, fake_translate)
|
||||
assert first == ["T:hello", "T:world"]
|
||||
assert len(calls) == 2
|
||||
|
||||
second = translate_with_tm(["hello", "world"], "en", "fr", "fake", scope, fake_translate)
|
||||
assert second == ["T:hello", "T:world"]
|
||||
# Everything came from the TM — no new provider calls
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_users_are_isolated(self):
|
||||
from services.translation_cache import get_cache
|
||||
|
||||
get_cache()
|
||||
out = translate_with_tm(
|
||||
["x"], "en", "fr", "fake",
|
||||
TMScope.from_prompt("user-A", None), lambda ts: [f"A:{ts[0]}"],
|
||||
)
|
||||
assert out == ["A:x"]
|
||||
out_b = translate_with_tm(
|
||||
["x"], "en", "fr", "fake",
|
||||
TMScope.from_prompt("user-B", None), lambda ts: [f"B:{ts[0]}"],
|
||||
)
|
||||
assert out_b == ["B:x"]
|
||||
|
||||
def test_identity_translations_not_stored(self):
|
||||
from services.translation_cache import get_cache
|
||||
|
||||
get_cache()
|
||||
scope = TMScope.from_prompt("user-1", None)
|
||||
# identity provider: returns input unchanged (a provider failure)
|
||||
translate_with_tm(["same"], "en", "fr", "fake", scope, lambda ts: ts)
|
||||
out = translate_with_tm(
|
||||
["same"], "en", "fr", "fake", scope, lambda ts: ["CALLED"]
|
||||
)
|
||||
# Not poisoned by the identity entry: the provider was consulted
|
||||
assert out == ["CALLED"]
|
||||
|
||||
def test_different_prompt_context_misses(self):
|
||||
from services.translation_cache import get_cache
|
||||
|
||||
get_cache()
|
||||
s1 = TMScope.from_prompt("u", "prompt-A")
|
||||
s2 = TMScope.from_prompt("u", "prompt-B")
|
||||
translate_with_tm(["k"], "en", "fr", "fake", s1, lambda ts: ["v1"])
|
||||
out = translate_with_tm(["k"], "en", "fr", "fake", s2, lambda ts: ["v2"])
|
||||
assert out == ["v2"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Bilingual output
|
||||
# ===========================================================================
|
||||
class TestBilingual:
|
||||
def test_source_interleaved_above_translation(self, tmp_path):
|
||||
src = Document()
|
||||
src.add_paragraph("Bonjour le monde")
|
||||
src.add_paragraph("")
|
||||
src.add_paragraph("Deuxième paragraphe")
|
||||
src_in = tmp_path / "src.docx"
|
||||
src.save(str(src_in))
|
||||
|
||||
tr = Document()
|
||||
tr.add_paragraph("Hello world")
|
||||
tr.add_paragraph("")
|
||||
tr.add_paragraph("Second paragraph")
|
||||
tr_out = tmp_path / "tr.docx"
|
||||
tr.save(str(tr_out))
|
||||
|
||||
result = make_bilingual_docx(src_in, tr_out, tmp_path / "bi.docx")
|
||||
assert result is not None
|
||||
|
||||
doc = Document(str(result))
|
||||
texts = [p.text for p in doc.paragraphs]
|
||||
# Source paragraph precedes its translation
|
||||
assert texts[0] == "Bonjour le monde"
|
||||
assert texts[1] == "Hello world"
|
||||
assert texts[3] == "Deuxième paragraphe"
|
||||
assert texts[4] == "Second paragraph"
|
||||
|
||||
def test_structure_mismatch_returns_none(self, tmp_path):
|
||||
a = Document(); a.add_paragraph("x"); a.save(str(tmp_path / "a.docx"))
|
||||
b = Document(); b.add_paragraph("x"); b.add_paragraph("y")
|
||||
b.save(str(tmp_path / "b.docx"))
|
||||
assert make_bilingual_docx(
|
||||
tmp_path / "a.docx", tmp_path / "b.docx", tmp_path / "bi.docx"
|
||||
) is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# QA report
|
||||
# ===========================================================================
|
||||
class TestQAReport:
|
||||
def test_number_fidelity_full(self):
|
||||
r = _number_fidelity("Prix: 12,50 € sur 3 pages", "Price: 12.50 € on 3 pages")
|
||||
assert r["fidelity"] >= 0.9
|
||||
|
||||
def test_number_fidelity_missing(self):
|
||||
r = _number_fidelity("Ref 123 and 456", "Ref 123 only")
|
||||
assert r["fidelity"] < 1.0
|
||||
|
||||
def test_untranslated_ratio_high_when_identity(self):
|
||||
identity_source = (
|
||||
"This document contains substantial english text about pricing, "
|
||||
"delivery schedules and quarterly reporting obligations for the "
|
||||
"regional sales team and their managers."
|
||||
)
|
||||
assert _untranslated_ratio(identity_source, identity_source) > 0.8
|
||||
|
||||
def test_untranslated_ratio_low_when_translated(self):
|
||||
assert _untranslated_ratio(
|
||||
"Ce document contient beaucoup de mots différents sur la facturation, "
|
||||
"les échéances de livraison et les obligations trimestrielles.",
|
||||
"This document holds many different words about invoicing, "
|
||||
"delivery deadlines and quarterly obligations.",
|
||||
) < 0.35
|
||||
|
||||
def test_run_qa_report_on_docx(self, tmp_path):
|
||||
src = Document()
|
||||
src.add_paragraph("Le total est de 42 euros pour la livraison.")
|
||||
src.add_paragraph("Merci de votre confiance renouvelée.")
|
||||
src_in = tmp_path / "in.docx"
|
||||
src.save(str(src_in))
|
||||
|
||||
out_doc = Document()
|
||||
out_doc.add_paragraph("The total is 42 euros for the delivery.")
|
||||
out_doc.add_paragraph("Thank you for your renewed trust.")
|
||||
out_p = tmp_path / "out.docx"
|
||||
out_doc.save(str(out_p))
|
||||
|
||||
report = run_qa_report(src_in, out_p, "en", ".docx")
|
||||
assert report is not None
|
||||
assert report["score"] >= 80
|
||||
assert report["numbers"]["fidelity"] == 1.0
|
||||
|
||||
def test_run_qa_report_scores_untranslated_low(self, tmp_path):
|
||||
long_fr = (
|
||||
"Ce paragraphe contient suffisamment de mots différents pour "
|
||||
"satisfaire l'heuristique du rapport qualité automatisé, avec "
|
||||
"des notions de facturation, livraison et obligations."
|
||||
)
|
||||
src = Document()
|
||||
src.add_paragraph(long_fr)
|
||||
src_in = tmp_path / "in.docx"
|
||||
src.save(str(src_in))
|
||||
import shutil
|
||||
|
||||
out_p = tmp_path / "out.docx"
|
||||
shutil.copy(str(src_in), str(out_p)) # "translation" = original
|
||||
|
||||
report = run_qa_report(src_in, out_p, "en", ".docx")
|
||||
assert report is not None
|
||||
assert report["score"] < 50
|
||||
assert report["untranslated_ratio"] > 0.5
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# OpenAI JSON batching
|
||||
# ===========================================================================
|
||||
class TestOpenAIBatch:
|
||||
def _provider(self):
|
||||
from services.providers.openai_provider import OpenAITranslationProvider
|
||||
|
||||
return OpenAITranslationProvider(api_key="test-key", model="gpt-test")
|
||||
|
||||
def _requests(self, texts):
|
||||
from services.providers.schemas import TranslationRequest
|
||||
|
||||
return [TranslationRequest(text=t, target_language="fr") for t in texts]
|
||||
|
||||
def test_batch_single_request_success(self, monkeypatch):
|
||||
provider = self._provider()
|
||||
seen = {}
|
||||
|
||||
def fake_api(text, system_prompt):
|
||||
import json
|
||||
|
||||
seen["text"] = text
|
||||
items = json.loads(text)
|
||||
reply = json.dumps(
|
||||
[{"id": it["id"], "translation": f"FR:{it['text']}"} for it in items]
|
||||
)
|
||||
return reply, {}
|
||||
|
||||
monkeypatch.setattr(provider, "_make_api_request", fake_api)
|
||||
out = provider.translate_batch(self._requests(["one", "two", "three"]))
|
||||
assert [r.translated_text for r in out] == [
|
||||
"FR:one", "FR:two", "FR:three",
|
||||
]
|
||||
# One API call for the whole batch
|
||||
assert isinstance(seen.get("text"), str) and '"id"' in seen["text"]
|
||||
|
||||
def test_batch_falls_back_on_bad_json(self, monkeypatch):
|
||||
provider = self._provider()
|
||||
calls = {"batch": 0, "single": 0}
|
||||
|
||||
def fake_api(text, system_prompt):
|
||||
if text.startswith("["):
|
||||
calls["batch"] += 1
|
||||
return "not valid json at all", {}
|
||||
calls["single"] += 1
|
||||
return f"FR:{text}", {}
|
||||
|
||||
monkeypatch.setattr(provider, "_make_api_request", fake_api)
|
||||
out = provider.translate_batch(self._requests(["alpha", "beta"]))
|
||||
assert [r.translated_text for r in out] == ["FR:alpha", "FR:beta"]
|
||||
assert calls["batch"] == 1
|
||||
assert calls["single"] == 2
|
||||
|
||||
def test_batch_falls_back_on_wrong_length(self, monkeypatch):
|
||||
import json
|
||||
|
||||
provider = self._provider()
|
||||
|
||||
def fake_api(text, system_prompt):
|
||||
return json.dumps([{"id": 0, "translation": "only one"}]), {}
|
||||
|
||||
monkeypatch.setattr(provider, "_make_api_request", fake_api)
|
||||
|
||||
def fail_single(req):
|
||||
from services.providers.schemas import TranslationResponse
|
||||
|
||||
return TranslationResponse(
|
||||
translated_text=f"S:{req.text}", provider_name="openai"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(provider, "translate_text", fail_single)
|
||||
out = provider.translate_batch(self._requests(["a", "b"]))
|
||||
assert [r.translated_text for r in out] == ["S:a", "S:b"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# CJK font hints (Word)
|
||||
# ===========================================================================
|
||||
class TestFontHints:
|
||||
def test_hint_mapping(self):
|
||||
assert _font_hints_for_target("zh-CN")[0] == "SimSun"
|
||||
assert _font_hints_for_target("ja")[0] == "Yu Mincho"
|
||||
assert _font_hints_for_target("ar")[1] == "Arial"
|
||||
assert _font_hints_for_target("en") == (None, None)
|
||||
|
||||
def test_applied_on_translate(self, tmp_path):
|
||||
class _CJK:
|
||||
def get_name(self):
|
||||
return "mock"
|
||||
|
||||
def is_available(self):
|
||||
return True
|
||||
|
||||
def translate_batch(self, texts, target_language, source_language="auto"):
|
||||
return [f"译:{t}" for t in texts]
|
||||
|
||||
from docx.oxml.ns import qn as _qn
|
||||
|
||||
doc = Document()
|
||||
doc.add_paragraph("Hello")
|
||||
src = tmp_path / "in.docx"
|
||||
doc.save(str(src))
|
||||
|
||||
t = WordTranslator(provider=_CJK())
|
||||
out = tmp_path / "out.docx"
|
||||
t.translate_file(src, out, "zh-CN", "en")
|
||||
|
||||
result = Document(str(out))
|
||||
para = result.paragraphs[0]
|
||||
rFonts = para.runs[0]._r.find(_qn("w:rPr")).find(_qn("w:rFonts"))
|
||||
assert rFonts is not None
|
||||
assert rFonts.get(_qn("w:eastAsia")) == "SimSun"
|
||||
86
tests/test_plan_gating.py
Normal file
86
tests/test_plan_gating.py
Normal file
@@ -0,0 +1,86 @@
|
||||
"""Plan-based engine gating and the expanded /languages endpoint."""
|
||||
|
||||
import pytest
|
||||
|
||||
from models.subscription import PlanType
|
||||
from routes.translate_routes import (
|
||||
_allowed_providers_for_plan,
|
||||
_image_translation_allowed_for_plan,
|
||||
_plan_from_user,
|
||||
)
|
||||
|
||||
|
||||
class _FakeUser:
|
||||
def __init__(self, plan):
|
||||
self.plan = plan
|
||||
|
||||
|
||||
class TestAllowedProviders:
|
||||
def test_free_gets_google_only(self):
|
||||
assert _allowed_providers_for_plan(PlanType.FREE) == {"google"}
|
||||
|
||||
def test_starter_adds_deepl(self):
|
||||
assert _allowed_providers_for_plan(PlanType.STARTER) == {"google", "deepl"}
|
||||
|
||||
def test_pro_adds_cloud_and_openrouter(self):
|
||||
allowed = _allowed_providers_for_plan(PlanType.PRO)
|
||||
assert {"google_cloud", "openrouter"} <= allowed
|
||||
assert "openai" not in allowed
|
||||
|
||||
def test_business_adds_premium_and_xai(self):
|
||||
allowed = _allowed_providers_for_plan(PlanType.BUSINESS)
|
||||
assert {"openrouter_premium", "openai", "zai"} <= allowed
|
||||
|
||||
def test_enterprise_has_all(self):
|
||||
enterprise = _allowed_providers_for_plan(PlanType.ENTERPRISE)
|
||||
assert enterprise >= _allowed_providers_for_plan(PlanType.BUSINESS)
|
||||
|
||||
|
||||
class TestImageTranslationGate:
|
||||
@pytest.mark.parametrize("plan", [PlanType.FREE, PlanType.STARTER])
|
||||
def test_refused_below_pro(self, plan):
|
||||
assert _image_translation_allowed_for_plan(plan) is False
|
||||
|
||||
@pytest.mark.parametrize("plan", [PlanType.PRO, PlanType.BUSINESS, PlanType.ENTERPRISE])
|
||||
def test_allowed_from_pro(self, plan):
|
||||
assert _image_translation_allowed_for_plan(plan) is True
|
||||
|
||||
|
||||
class TestPlanFromUser:
|
||||
def test_anonymous_is_free(self):
|
||||
assert _plan_from_user(None) is PlanType.FREE
|
||||
|
||||
def test_enum_plan_passthrough(self):
|
||||
assert _plan_from_user(_FakeUser(PlanType.PRO)) is PlanType.PRO
|
||||
|
||||
def test_string_plan_accepted(self):
|
||||
assert _plan_from_user(_FakeUser("pro")) is PlanType.PRO
|
||||
|
||||
def test_garbage_falls_back_to_free(self):
|
||||
assert _plan_from_user(_FakeUser("nonsense")) is PlanType.FREE
|
||||
|
||||
|
||||
class TestLanguagesEndpoint:
|
||||
@pytest.mark.asyncio
|
||||
async def test_exposes_at_least_60_languages(self):
|
||||
from routes.legacy_routes import get_supported_languages
|
||||
|
||||
response = await get_supported_languages()
|
||||
langs = response["supported_languages"]
|
||||
assert response["count"] >= 60
|
||||
assert len(langs) >= 60
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_auto_and_canonical_chinese(self):
|
||||
from routes.legacy_routes import get_supported_languages
|
||||
|
||||
langs = (await get_supported_languages())["supported_languages"]
|
||||
assert "auto" not in langs
|
||||
assert "zh-CN" in langs and "zh-TW" in langs
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_every_language_has_a_name(self):
|
||||
from routes.legacy_routes import get_supported_languages
|
||||
|
||||
langs = (await get_supported_languages())["supported_languages"]
|
||||
assert all(name and name != code.upper() for code, name in langs.items())
|
||||
161
tests/test_providers/test_minimax_provider.py
Normal file
161
tests/test_providers/test_minimax_provider.py
Normal file
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
Tests for the MinimaxTranslationProvider.
|
||||
|
||||
Validates Bug 1 fix:
|
||||
- default base_url is the public ``api.minimax.io`` host (NOT ``api.minimax.chat``)
|
||||
- default model is ``MiniMax-M3``
|
||||
- ``is_available()`` / ``health_check()`` tolerate a missing ``/models`` path
|
||||
(Minimax does not document it) and only mark the provider down on 401/network error
|
||||
- translation success, 429 retry, 401 handling
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
from requests.exceptions import Timeout
|
||||
|
||||
from services.providers.minimax_provider import (
|
||||
MinimaxTranslationProvider,
|
||||
MinimaxProviderError,
|
||||
MINIMAX_RATE_LIMITED,
|
||||
MINIMAX_INVALID_KEY,
|
||||
MINIMAX_TIMEOUT,
|
||||
MINIMAX_SERVICE_ERROR,
|
||||
)
|
||||
from services.providers.schemas import TranslationRequest
|
||||
|
||||
|
||||
class TestMinimaxProviderConfig:
|
||||
"""Defaults must point at the real public endpoint."""
|
||||
|
||||
def test_default_base_url_is_public_host(self):
|
||||
provider = MinimaxTranslationProvider(api_key="k", max_retries=0)
|
||||
assert provider._base_url == "https://api.minimax.io/v1"
|
||||
# Regression guard: the old broken host must never come back.
|
||||
assert "minimax.chat" not in provider._base_url
|
||||
|
||||
def test_default_model_is_m3(self):
|
||||
provider = MinimaxTranslationProvider(api_key="k", max_retries=0)
|
||||
assert provider._model == "MiniMax-M3"
|
||||
|
||||
def test_custom_base_url_respected(self):
|
||||
provider = MinimaxTranslationProvider(
|
||||
api_key="k", base_url="https://proxy.example.com/v1", max_retries=0
|
||||
)
|
||||
assert provider._base_url == "https://proxy.example.com/v1"
|
||||
|
||||
def test_get_name(self):
|
||||
provider = MinimaxTranslationProvider(api_key="k", max_retries=0)
|
||||
assert provider.get_name() == "minimax"
|
||||
|
||||
|
||||
class TestMinimaxAvailabilityProbe:
|
||||
"""is_available/health_check must not fail just because /models 404s."""
|
||||
|
||||
@pytest.fixture
|
||||
def provider(self):
|
||||
return MinimaxTranslationProvider(api_key="k", max_retries=0)
|
||||
|
||||
def _mock_get(self, status_code: int):
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = status_code
|
||||
return mock_response
|
||||
|
||||
@patch("requests.get")
|
||||
def test_available_when_models_returns_200(self, mock_get, provider):
|
||||
mock_get.return_value = self._mock_get(200)
|
||||
assert provider.is_available() is True
|
||||
|
||||
@patch("requests.get")
|
||||
def test_available_when_models_404(self, mock_get, provider):
|
||||
# Minimax does not document /models; a 404 must NOT mark it unavailable.
|
||||
mock_get.return_value = self._mock_get(404)
|
||||
assert provider.is_available() is True
|
||||
|
||||
@patch("requests.get")
|
||||
def test_unavailable_on_401(self, mock_get, provider):
|
||||
mock_get.return_value = self._mock_get(401)
|
||||
assert provider.is_available() is False
|
||||
|
||||
@patch("requests.get")
|
||||
def test_unavailable_on_network_error(self, _mock_get, provider):
|
||||
def _raise(*a, **kw):
|
||||
raise Timeout("boom")
|
||||
|
||||
with patch("requests.get", side_effect=_raise):
|
||||
assert provider.is_available() is False
|
||||
|
||||
@patch("requests.get")
|
||||
def test_health_check_tolerates_404(self, mock_get, provider):
|
||||
mock_get.return_value = self._mock_get(404)
|
||||
status = provider.health_check()
|
||||
assert status.available is True
|
||||
assert status.name == "minimax"
|
||||
|
||||
@patch("requests.get")
|
||||
def test_health_check_marks_down_on_401(self, mock_get, provider):
|
||||
mock_get.return_value = self._mock_get(401)
|
||||
status = provider.health_check()
|
||||
assert status.available is False
|
||||
|
||||
|
||||
class TestMinimaxTranslateText:
|
||||
@pytest.fixture
|
||||
def provider(self):
|
||||
return MinimaxTranslationProvider(api_key="k", model="MiniMax-M3", max_retries=0)
|
||||
|
||||
def _mock_post(self, payload, status_code=200):
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = status_code
|
||||
mock_response.json.return_value = payload
|
||||
mock_response.text = ""
|
||||
return mock_response
|
||||
|
||||
@patch("requests.post")
|
||||
def test_success(self, mock_post, provider):
|
||||
mock_post.return_value = self._mock_post(
|
||||
{"choices": [{"message": {"content": "Bonjour"}}], "usage": {}}
|
||||
)
|
||||
resp = provider.translate_text(TranslationRequest(text="Hello", target_language="fr"))
|
||||
assert resp.translated_text == "Bonjour"
|
||||
assert resp.provider_name == "minimax"
|
||||
# Verify we hit the public host on the OpenAI-compatible path.
|
||||
called_url = mock_post.call_args[0][0]
|
||||
assert called_url == "https://api.minimax.io/v1/chat/completions"
|
||||
|
||||
def test_empty_text_short_circuits(self, provider):
|
||||
resp = provider.translate_text(TranslationRequest(text="", target_language="fr"))
|
||||
assert resp.translated_text == ""
|
||||
|
||||
@patch("requests.post")
|
||||
def test_invalid_key_returns_error(self, mock_post, provider):
|
||||
mock_post.return_value = self._mock_post({"error": "bad key"}, status_code=401)
|
||||
resp = provider.translate_text(TranslationRequest(text="Hello", target_language="fr"))
|
||||
assert resp.error_code == MINIMAX_INVALID_KEY
|
||||
# Original text returned on failure.
|
||||
assert resp.translated_text == "Hello"
|
||||
|
||||
@patch("time.sleep")
|
||||
@patch("requests.post")
|
||||
def test_rate_limit_then_success(self, mock_post, mock_sleep):
|
||||
provider = MinimaxTranslationProvider(api_key="k", max_retries=2, retry_delay=0.01)
|
||||
mock_post.side_effect = [
|
||||
self._mock_post({"error": "slow down"}, status_code=429),
|
||||
self._mock_post({"choices": [{"message": {"content": "Hola"}}], "usage": {}}),
|
||||
]
|
||||
resp = provider.translate_text(TranslationRequest(text="Hello", target_language="es"))
|
||||
assert resp.translated_text == "Hola"
|
||||
assert mock_sleep.called # backoff happened
|
||||
|
||||
@patch("requests.post")
|
||||
def test_service_error_when_empty_choices(self, mock_post, provider):
|
||||
mock_post.return_value = self._mock_post({"choices": []})
|
||||
resp = provider.translate_text(TranslationRequest(text="Hello", target_language="fr"))
|
||||
assert resp.error_code == MINIMAX_SERVICE_ERROR
|
||||
|
||||
|
||||
class TestMinimaxProviderError:
|
||||
def test_error_carries_code_and_message(self):
|
||||
err = MinimaxProviderError(MINIMAX_TIMEOUT, "timed out", details={"wait": 1})
|
||||
assert err.code == MINIMAX_TIMEOUT
|
||||
assert err.message == "timed out"
|
||||
assert err.details == {"wait": 1}
|
||||
@@ -97,11 +97,24 @@ class TestHelperFunctions:
|
||||
assert "translator" in prompt.lower()
|
||||
|
||||
def test_build_system_prompt_custom(self):
|
||||
"""Test custom system prompt."""
|
||||
"""A custom prompt AUGMENTS the base translation instructions.
|
||||
|
||||
The base prompt is always present — a glossary-only custom prompt
|
||||
used to produce a system prompt with no translation instruction
|
||||
at all (fixed 2026-08-29).
|
||||
"""
|
||||
custom = "Translate this text formally for business context."
|
||||
prompt = _build_system_prompt("English", "French", custom)
|
||||
|
||||
assert prompt == custom
|
||||
# Base translation instructions survive
|
||||
assert "English" in prompt
|
||||
assert "French" in prompt
|
||||
assert "translator" in prompt.lower()
|
||||
# Custom content is appended, not replacing
|
||||
assert custom in prompt
|
||||
assert "ADDITIONAL CONTEXT AND INSTRUCTIONS" in prompt
|
||||
# The base part comes first
|
||||
assert prompt.index("French") < prompt.index(custom)
|
||||
|
||||
|
||||
class TestOpenAITranslationProvider:
|
||||
|
||||
192
tests/test_scanned_pdf_ocr.py
Normal file
192
tests/test_scanned_pdf_ocr.py
Normal file
@@ -0,0 +1,192 @@
|
||||
"""Scanned PDF detection and the Mistral OCR translation path."""
|
||||
|
||||
import fitz
|
||||
import pytest
|
||||
|
||||
from config import config
|
||||
from services.mistral_ocr import MistralOCRClient
|
||||
from services.providers.base import TranslationProvider
|
||||
from services.providers.schemas import TranslationRequest, TranslationResponse
|
||||
from translators.pdf_translator import PDFTranslator
|
||||
|
||||
|
||||
FAKE_OCR_PAGES = [
|
||||
"# Facture\n\nMontant total : 1 250 euros\n\n",
|
||||
"Le paiement est du sous 30 jours.",
|
||||
]
|
||||
|
||||
# Minimal French → English mapping for the fake provider; anything else is
|
||||
# prefixed so tests can assert replacement happened.
|
||||
TRANSLATIONS = {
|
||||
"Facture": "Invoice",
|
||||
"Montant total : 1 250 euros": "Total amount: 1,250 euros",
|
||||
"Le paiement est du sous 30 jours.": "Payment is due within 30 days.",
|
||||
}
|
||||
|
||||
|
||||
class FakeProvider(TranslationProvider):
|
||||
"""New-style provider with canned translations (no network)."""
|
||||
|
||||
def get_name(self) -> str:
|
||||
return "fake"
|
||||
|
||||
def is_available(self) -> bool:
|
||||
return True
|
||||
|
||||
def translate_text(self, request: TranslationRequest) -> TranslationResponse:
|
||||
# Pages arrive as multi-line text: translate line by line so the
|
||||
# canned mapping applies regardless of how lines are grouped.
|
||||
lines = []
|
||||
for line in request.text.split("\n"):
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
lines.append(line)
|
||||
continue
|
||||
lines.append(TRANSLATIONS.get(stripped, f"[EN] {stripped}"))
|
||||
return TranslationResponse(
|
||||
translated_text="\n".join(lines),
|
||||
provider_name=self.get_name(),
|
||||
from_cache=False,
|
||||
)
|
||||
|
||||
|
||||
def _make_scanned_pdf(path):
|
||||
"""Image-only PDF: one page almost fully covered by a picture, no text."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 10, 10))
|
||||
pix.clear_with(90)
|
||||
png_bytes = pix.tobytes("png")
|
||||
doc = fitz.open()
|
||||
page = doc.new_page(width=612, height=792)
|
||||
page.insert_image(fitz.Rect(20, 20, 592, 772), stream=png_bytes)
|
||||
doc.save(str(path))
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
def _make_text_pdf(path):
|
||||
doc = fitz.open()
|
||||
page = doc.new_page()
|
||||
page.insert_text(
|
||||
(72, 100),
|
||||
"Real selectable text with plenty of characters to exceed the scanned threshold. " * 2,
|
||||
fontsize=11,
|
||||
)
|
||||
doc.save(str(path))
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def no_api_key(monkeypatch):
|
||||
monkeypatch.delenv("MISTRAL_API_KEY", raising=False)
|
||||
monkeypatch.setattr(config, "MISTRAL_API_KEY", "")
|
||||
monkeypatch.setattr(config, "MISTRAL_OCR_ENABLED", True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def api_key(monkeypatch):
|
||||
# Set BOTH the env var and the config attribute: some other test in the
|
||||
# full suite reloads the config module, and pdf_translator imports
|
||||
# `config` at call time — after a reload only the env var is still
|
||||
# visible (the class attributes are re-read from the environment).
|
||||
monkeypatch.setenv("MISTRAL_API_KEY", "test-key")
|
||||
monkeypatch.setattr(config, "MISTRAL_API_KEY", "test-key")
|
||||
monkeypatch.setattr(config, "MISTRAL_OCR_ENABLED", True)
|
||||
|
||||
|
||||
class TestScannedDetection:
|
||||
def test_image_only_pdf_is_scanned(self, tmp_path):
|
||||
pdf = _make_scanned_pdf(tmp_path / "scan.pdf")
|
||||
translator = PDFTranslator(provider=None)
|
||||
assert translator._is_scanned_pdf(pdf) is True
|
||||
|
||||
def test_text_pdf_is_not_scanned(self, tmp_path):
|
||||
pdf = _make_text_pdf(tmp_path / "text.pdf")
|
||||
translator = PDFTranslator(provider=None)
|
||||
assert translator._is_scanned_pdf(pdf) is False
|
||||
|
||||
def test_title_only_pdf_is_not_scanned(self, tmp_path):
|
||||
"""A sparse but textual page (no raster image) stays on the layout path."""
|
||||
doc = fitz.open()
|
||||
page = doc.new_page()
|
||||
page.insert_text((72, 100), "Facture 2026", fontsize=18)
|
||||
doc.save(str(tmp_path / "title.pdf"))
|
||||
doc.close()
|
||||
translator = PDFTranslator(provider=None)
|
||||
assert translator._is_scanned_pdf(tmp_path / "title.pdf") is False
|
||||
|
||||
|
||||
class TestMarkdownCleanup:
|
||||
def test_images_links_headings_removed(self):
|
||||
md = "# Titre\n\n\n\n[lien](http://x) texte"
|
||||
out = PDFTranslator._markdown_to_text(md)
|
||||
assert "Titre" in out
|
||||
assert "![img]" not in out
|
||||
assert "(http://x)" not in out
|
||||
assert "#" not in out
|
||||
assert "lien texte" in out
|
||||
|
||||
def test_table_pipes_removed(self):
|
||||
out = PDFTranslator._markdown_to_text("| A | B |\n|---|---|\n| un | deux |")
|
||||
assert "|" not in out
|
||||
assert "un" in out and "deux" in out
|
||||
|
||||
|
||||
class TestScannedOCRPath:
|
||||
def test_translate_scanned_pdf_end_to_end(self, tmp_path, api_key, monkeypatch):
|
||||
pdf = _make_scanned_pdf(tmp_path / "scan.pdf")
|
||||
monkeypatch.setattr(
|
||||
MistralOCRClient,
|
||||
"extract_pdf_text",
|
||||
lambda self, path, progress_callback=None: list(FAKE_OCR_PAGES),
|
||||
)
|
||||
|
||||
translator = PDFTranslator(provider=FakeProvider())
|
||||
# Configure OCR the way the production route does (set_ocr_config).
|
||||
# Going through the config module is fragile in the full suite:
|
||||
# another test replaces sys.modules["config"], so a config-module
|
||||
# patch may never reach the translator.
|
||||
translator.set_ocr_config(api_key="test-key", enabled=True)
|
||||
out = tmp_path / "out.pdf"
|
||||
result = translator.translate_file(pdf, out, "en", "fr")
|
||||
|
||||
assert result.exists() and result.stat().st_size > 0
|
||||
doc = fitz.open(str(result))
|
||||
text = "\n".join(p.get_text("text") for p in doc)
|
||||
doc.close()
|
||||
|
||||
assert "Invoice" in text
|
||||
assert "Payment is due within 30 days" in text
|
||||
assert "Facture" not in text
|
||||
assert "logo" not in text # markdown images stripped
|
||||
assert "|" not in text # markdown tables stripped
|
||||
|
||||
def test_missing_api_key_raises_clear_error(self, tmp_path, no_api_key):
|
||||
pdf = _make_scanned_pdf(tmp_path / "scan.pdf")
|
||||
translator = PDFTranslator(provider=FakeProvider())
|
||||
with pytest.raises(RuntimeError) as exc:
|
||||
translator.translate_file(pdf, tmp_path / "out.pdf", "en", "fr")
|
||||
assert "MISTRAL_API_KEY" in str(exc.value)
|
||||
|
||||
def test_disabled_ocr_raises_clear_error(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(config, "MISTRAL_API_KEY", "test-key")
|
||||
monkeypatch.setattr(config, "MISTRAL_OCR_ENABLED", False)
|
||||
pdf = _make_scanned_pdf(tmp_path / "scan.pdf")
|
||||
translator = PDFTranslator(provider=FakeProvider())
|
||||
with pytest.raises(RuntimeError):
|
||||
translator.translate_file(pdf, tmp_path / "out.pdf", "en", "fr")
|
||||
|
||||
def test_text_pdf_bypasses_ocr(self, tmp_path, api_key, monkeypatch):
|
||||
pdf = _make_text_pdf(tmp_path / "text.pdf")
|
||||
called = {"ocr": False}
|
||||
|
||||
def _fail(self, path, progress_callback=None):
|
||||
called["ocr"] = True
|
||||
return []
|
||||
|
||||
monkeypatch.setattr(MistralOCRClient, "extract_pdf_text", _fail)
|
||||
|
||||
translator = PDFTranslator(provider=FakeProvider())
|
||||
out = tmp_path / "out.pdf"
|
||||
translator.translate_file(pdf, out, "en", "fr")
|
||||
assert called["ocr"] is False
|
||||
228
tests/test_security_fixes_c1_c4.py
Normal file
228
tests/test_security_fixes_c1_c4.py
Normal file
@@ -0,0 +1,228 @@
|
||||
"""Tests for security fixes C1–C4 (audit 2026-08-26)."""
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from routes import translate_routes as tr
|
||||
from middleware.cleanup import FileCleanupManager
|
||||
|
||||
|
||||
class TestSanitizeUrlFilename:
|
||||
"""AC1 — path traversal in URL-downloaded filenames is neutralized."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"raw",
|
||||
[
|
||||
"../../evil.xlsx",
|
||||
"..\\..\\evil.docx",
|
||||
"normal_file.pptx",
|
||||
"..\t..evil.pdf",
|
||||
"...",
|
||||
"",
|
||||
],
|
||||
)
|
||||
def test_traversal_stripped(self, raw):
|
||||
result = tr._sanitize_url_filename(raw)
|
||||
# Core guarantee: no traversal or path separators survive
|
||||
assert ".." not in result
|
||||
assert "/" not in result
|
||||
assert "\\" not in result
|
||||
assert result != ""
|
||||
|
||||
def test_long_filename_truncated(self):
|
||||
raw = "a" * 300 + ".xlsx"
|
||||
result = tr._sanitize_url_filename(raw)
|
||||
assert len(result) <= 255
|
||||
assert result.endswith(".xlsx")
|
||||
|
||||
|
||||
class TestRedirectSsrf:
|
||||
"""AC2 — redirect to internal address is blocked."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redirect_to_metadata_blocked(self):
|
||||
def fake_response(status, location=None):
|
||||
resp = MagicMock()
|
||||
resp.status_code = status
|
||||
resp.headers = {"location": location} if location else {}
|
||||
return resp
|
||||
|
||||
class FakeClient:
|
||||
def build_request(self, method, url):
|
||||
return (method, url)
|
||||
|
||||
async def send(self, req, stream=False):
|
||||
url = req[1]
|
||||
if "public.example" in url:
|
||||
return fake_response(302, "http://169.254.169.254/latest/meta-data")
|
||||
return fake_response(200)
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
with patch.object(tr.httpx, "AsyncClient", return_value=FakeClient()):
|
||||
with pytest.raises(tr.TranslateEndpointError) as exc:
|
||||
await tr.download_from_url("http://public.example/file.xlsx")
|
||||
assert exc.value.details.get("reason") == "ssrf_blocked"
|
||||
|
||||
|
||||
class TestCleanupOrphanGrace:
|
||||
"""AC3 — young orphans are not deleted."""
|
||||
|
||||
def _manager(self, tmp_path):
|
||||
m = FileCleanupManager(
|
||||
upload_dir=tmp_path / "uploads",
|
||||
output_dir=tmp_path / "outputs",
|
||||
temp_dir=tmp_path / "temp",
|
||||
)
|
||||
for d in (m.upload_dir, m.output_dir, m.temp_dir):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return m
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_young_orphan_kept(self, tmp_path):
|
||||
"""Orphan younger than the grace period is NOT deleted."""
|
||||
m = self._manager(tmp_path)
|
||||
f = m.upload_dir / "recent_orphan.xlsx"
|
||||
f.write_bytes(b"x")
|
||||
|
||||
fake_redis = MagicMock()
|
||||
fake_redis.keys = AsyncMock(return_value=[])
|
||||
fake_redis.get = AsyncMock(return_value=None)
|
||||
with patch(
|
||||
"middleware.cleanup._get_async_redis", return_value=fake_redis
|
||||
):
|
||||
stats = await m.cleanup()
|
||||
assert f.exists()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_input_path_key_recognized(self, tmp_path):
|
||||
"""Files tracked under 'input_path' are not orphans (key mismatch fix)."""
|
||||
import json as _json
|
||||
|
||||
m = self._manager(tmp_path)
|
||||
f = m.upload_dir / "tracked.xlsx"
|
||||
f.write_bytes(b"x")
|
||||
|
||||
fake_redis = MagicMock()
|
||||
fake_redis.keys = AsyncMock(return_value=["translation:file:tr_1"])
|
||||
fake_redis.get = AsyncMock(
|
||||
return_value=_json.dumps({"input_path": str(f), "user_id": "u1"})
|
||||
)
|
||||
with patch(
|
||||
"middleware.cleanup._get_async_redis", return_value=fake_redis
|
||||
):
|
||||
stats = await m.cleanup()
|
||||
assert f.exists()
|
||||
assert stats["orphaned_deleted"] == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_old_orphan_deleted(self, tmp_path):
|
||||
import json as _json
|
||||
import os
|
||||
import time
|
||||
|
||||
m = self._manager(tmp_path)
|
||||
f = m.upload_dir / "old_orphan.xlsx"
|
||||
f.write_bytes(b"x")
|
||||
old = time.time() - (m.orphan_grace_seconds + 600)
|
||||
os.utime(f, (old, old))
|
||||
|
||||
fake_redis = MagicMock()
|
||||
fake_redis.keys = AsyncMock(return_value=[])
|
||||
fake_redis.get = AsyncMock(return_value=None)
|
||||
with patch(
|
||||
"middleware.cleanup._get_async_redis", return_value=fake_redis
|
||||
):
|
||||
stats = await m.cleanup()
|
||||
assert not f.exists()
|
||||
assert stats["orphaned_deleted"] == 1
|
||||
|
||||
|
||||
class TestJobAccessControl:
|
||||
"""H2 — ownership / token checks on status and download."""
|
||||
|
||||
def _job(self, user_id=None, token="tok123"):
|
||||
return {"id": "tr_abc", "user_id": user_id, "access_token": token}
|
||||
|
||||
def _user(self, uid):
|
||||
u = MagicMock()
|
||||
u.id = uid
|
||||
return u
|
||||
|
||||
def test_owner_allowed(self):
|
||||
job = self._job(user_id=7)
|
||||
assert tr._check_job_access(job, self._user(7), None) is None
|
||||
|
||||
def test_other_user_denied(self):
|
||||
job = self._job(user_id=7)
|
||||
resp = tr._check_job_access(job, self._user(8), None)
|
||||
assert resp is not None and resp.status_code == 403
|
||||
|
||||
def test_anonymous_caller_on_owned_job_denied(self):
|
||||
job = self._job(user_id=7)
|
||||
resp = tr._check_job_access(job, None, "tok123")
|
||||
assert resp is not None and resp.status_code == 401
|
||||
|
||||
def test_anonymous_job_requires_token(self):
|
||||
job = self._job(user_id=None)
|
||||
assert tr._check_job_access(job, None, "wrong") is not None
|
||||
assert tr._check_job_access(job, None, None) is not None
|
||||
assert tr._check_job_access(job, None, "tok123") is None
|
||||
|
||||
def test_old_anonymous_job_without_token_denied(self):
|
||||
job = {"id": "tr_old", "user_id": None} # job created before the fix
|
||||
assert tr._check_job_access(job, None, "anything") is not None
|
||||
|
||||
|
||||
class TestZipBomb:
|
||||
"""H1 — dangerous archives are rejected."""
|
||||
|
||||
def _make_zip(self, tmp_path, entries):
|
||||
import zipfile
|
||||
|
||||
p = tmp_path / "bomb.xlsx"
|
||||
with zipfile.ZipFile(p, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for name, data in entries:
|
||||
zf.writestr(name, data)
|
||||
return p
|
||||
|
||||
def test_normal_file_accepted(self, tmp_path):
|
||||
from utils.file_handler import validate_zip_safety
|
||||
|
||||
p = self._make_zip(tmp_path, [("sheet1.xml", b"<xml/>ok" * 100)])
|
||||
validate_zip_safety(p) # no exception
|
||||
|
||||
def test_not_a_zip_rejected(self, tmp_path):
|
||||
from utils.file_handler import validate_zip_safety
|
||||
|
||||
p = tmp_path / "fake.xlsx"
|
||||
p.write_bytes(b"this is not a zip file")
|
||||
with pytest.raises(ValueError):
|
||||
validate_zip_safety(p)
|
||||
|
||||
def test_high_ratio_rejected(self, tmp_path):
|
||||
from utils.file_handler import validate_zip_safety
|
||||
|
||||
# 50 MB of zeros compresses far beyond the 100:1 ratio cap
|
||||
p = self._make_zip(tmp_path, [("huge.xml", b"\0" * (50 * 1024 * 1024))])
|
||||
with pytest.raises(ValueError):
|
||||
validate_zip_safety(p)
|
||||
|
||||
def test_declared_total_too_big_rejected(self, tmp_path):
|
||||
import zipfile
|
||||
from unittest.mock import patch as _patch
|
||||
from utils.file_handler import validate_zip_safety
|
||||
|
||||
p = self._make_zip(tmp_path, [("a.xml", b"<xml/>")])
|
||||
fake_info = MagicMock()
|
||||
fake_info.is_dir = lambda: False
|
||||
fake_info.file_size = 5 * 1024 * 1024 * 1024 # 5 GB declared
|
||||
fake_info.compress_size = 50 * 1024 * 1024
|
||||
with _patch.object(zipfile.ZipFile, "infolist", return_value=[fake_info]):
|
||||
with pytest.raises(ValueError):
|
||||
validate_zip_safety(p)
|
||||
@@ -30,10 +30,18 @@ def test_validate_invalid_magic_bytes():
|
||||
assert response.json()["error"] == "CORRUPTED_FILE"
|
||||
assert "corrompu" in response.json()["message"]
|
||||
|
||||
def _minimal_zip() -> bytes:
|
||||
"""A real minimal ZIP archive (Office files are ZIPs)."""
|
||||
import io, zipfile
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("[Content_Types].xml", "<Types/>")
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def test_validate_valid_file_header():
|
||||
# Test with a minimal valid-looking zip (Office files are ZIPs)
|
||||
# FileValidator checks for b"PK\x03\x04"
|
||||
files = {"file": ("test.docx", b"PK\x03\x04" + b"\x00" * 20, "application/vnd.openxmlformats-officedocument.wordprocessingml.document")}
|
||||
# Minimal real ZIP: passes magic-byte AND zip-safety checks
|
||||
files = {"file": ("test.docx", _minimal_zip(), "application/vnd.openxmlformats-officedocument.wordprocessingml.document")}
|
||||
response = client.post(
|
||||
"/api/v1/translate",
|
||||
files=files,
|
||||
|
||||
@@ -593,7 +593,11 @@ class TestOptionalParameters:
|
||||
assert response.status_code == 202
|
||||
|
||||
def test_accepts_mode_llm(self, authenticated_client):
|
||||
"""Accepts mode='llm'"""
|
||||
"""mode='llm' maps to the openrouter engine — Pro and above only.
|
||||
|
||||
The default test user is on the Free plan, so the plan-based engine
|
||||
gate must refuse it (403) instead of silently running a paid engine.
|
||||
"""
|
||||
excel_content = create_valid_excel()
|
||||
response = authenticated_client.post(
|
||||
TRANSLATE_URL,
|
||||
@@ -606,7 +610,8 @@ class TestOptionalParameters:
|
||||
},
|
||||
data={"target_lang": "fr", "mode": "llm"},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
assert response.status_code == 403
|
||||
assert "PRO_FEATURE_REQUIRED" in str(response.json())
|
||||
|
||||
def test_accepts_webhook_url(self, authenticated_client):
|
||||
"""Accepts webhook_url parameter"""
|
||||
@@ -875,7 +880,11 @@ class TestTranslateImagesParameter:
|
||||
"""Test translate_images parameter in POST /api/v1/translate"""
|
||||
|
||||
def test_accepts_translate_images_parameter(self, authenticated_client):
|
||||
"""Endpoint accepts translate_images form parameter"""
|
||||
"""translate_images is a Pro+ feature — Free plan gets 403.
|
||||
|
||||
The parameter is accepted by the schema; the plan gate refuses it
|
||||
for the default (Free) test user.
|
||||
"""
|
||||
excel_content = create_valid_excel()
|
||||
response = authenticated_client.post(
|
||||
TRANSLATE_URL,
|
||||
@@ -888,4 +897,5 @@ class TestTranslateImagesParameter:
|
||||
},
|
||||
data={"target_lang": "fr", "translate_images": "true"},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
assert response.status_code == 403
|
||||
assert "PRO_FEATURE_REQUIRED" in str(response.json())
|
||||
|
||||
168
tests/test_translate_routes_helpers.py
Normal file
168
tests/test_translate_routes_helpers.py
Normal file
@@ -0,0 +1,168 @@
|
||||
"""
|
||||
Unit tests for the helper functions in routes/translate_routes.py.
|
||||
|
||||
These cover the bug fixes that are pure logic (no full HTTP machinery needed):
|
||||
- Bug 2: _compute_cost_factor / _provider_model (reads ``_model`` AND ``model``)
|
||||
- Bug 3: _release_quota_if_needed (releases reserved quota on soft-failure)
|
||||
- Bug 4: _compute_duration_seconds (UTC-aware, never crashes)
|
||||
- Bug 5: _cleanup_old_jobs snapshots the jobs dict (no "changed size during iteration")
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from routes import translate_routes as tr
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 2 — _provider_model / _compute_cost_factor
|
||||
# ---------------------------------------------------------------------------
|
||||
class _NewStyleProvider:
|
||||
"""Mimics services/providers/* classes that store ``self._model``."""
|
||||
|
||||
def __init__(self, model):
|
||||
self._model = model
|
||||
|
||||
|
||||
class _LegacyProvider:
|
||||
"""Mimics legacy services/translation_service classes with ``self.model``."""
|
||||
|
||||
def __init__(self, model):
|
||||
self.model = model
|
||||
|
||||
|
||||
class TestProviderModel:
|
||||
def test_reads_new_style_private_attr(self):
|
||||
assert tr._provider_model(_NewStyleProvider("gpt-4o")) == "gpt-4o"
|
||||
|
||||
def test_reads_legacy_public_attr(self):
|
||||
assert tr._provider_model(_LegacyProvider("claude-sonnet-4")) == "claude-sonnet-4"
|
||||
|
||||
def test_prefers_private_when_both_present(self):
|
||||
class Both:
|
||||
_model = "private-model"
|
||||
model = "public-model"
|
||||
|
||||
assert tr._provider_model(Both()) == "private-model"
|
||||
|
||||
def test_none_provider(self):
|
||||
assert tr._provider_model(None) == ""
|
||||
|
||||
def test_empty_model(self):
|
||||
assert tr._provider_model(_NewStyleProvider("")) == ""
|
||||
|
||||
|
||||
class TestComputeCostFactor:
|
||||
def test_claude_is_premium(self):
|
||||
assert tr._compute_cost_factor(_NewStyleProvider("anthropic/claude-sonnet-4.6")) == 5
|
||||
|
||||
def test_gpt4_is_premium(self):
|
||||
assert tr._compute_cost_factor(_NewStyleProvider("gpt-4o")) == 5
|
||||
|
||||
def test_gpt4o_mini_is_standard(self):
|
||||
# Cheap GPT-4 variants must not be billed at the premium factor.
|
||||
assert tr._compute_cost_factor(_NewStyleProvider("gpt-4o-mini")) == 1
|
||||
assert tr._compute_cost_factor(_NewStyleProvider("gpt-4o-nano")) == 1
|
||||
|
||||
def test_legacy_gpt4o_mini_is_standard(self):
|
||||
assert tr._compute_cost_factor(_LegacyProvider("gpt-4o-mini")) == 1
|
||||
|
||||
def test_haiku_is_standard(self):
|
||||
assert tr._compute_cost_factor(_NewStyleProvider("anthropic/claude-3-haiku")) == 1
|
||||
|
||||
def test_openrouter_premium_alias_is_premium_without_model(self):
|
||||
# Regression for the original bug: model read failed (""), so the
|
||||
# premium tier was never detected. The alias must still bump it to 5.
|
||||
assert tr._compute_cost_factor(None, "openrouter_premium") == 5
|
||||
|
||||
def test_standard_model(self):
|
||||
assert tr._compute_cost_factor(_NewStyleProvider("deepseek-chat")) == 1
|
||||
|
||||
def test_legacy_provider_model_is_read(self):
|
||||
# Legacy classes expose .model — must also be billed correctly.
|
||||
assert tr._compute_cost_factor(_LegacyProvider("gpt-4o")) == 5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 3 — _release_quota_if_needed
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestReleaseQuotaIfNeeded:
|
||||
@pytest.mark.asyncio
|
||||
async def test_releases_when_user_id_and_not_recorded(self):
|
||||
# release_translation_quota is invoked via asyncio.to_thread (a worker
|
||||
# thread); patching it and asserting the call validates the release path.
|
||||
with patch.object(tr, "release_translation_quota") as mock_release:
|
||||
await tr._release_quota_if_needed("user-123", usage_recorded=False, job_id="j1")
|
||||
mock_release.assert_called_once_with("user-123")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_skips_when_usage_already_recorded(self):
|
||||
with patch.object(tr, "release_translation_quota") as mock_release:
|
||||
await tr._release_quota_if_needed("user-123", usage_recorded=True, job_id="j1")
|
||||
mock_release.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_skips_when_no_user_id(self):
|
||||
with patch.object(tr, "release_translation_quota") as mock_release:
|
||||
await tr._release_quota_if_needed(None, usage_recorded=False, job_id="j1")
|
||||
mock_release.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_swallows_release_errors(self):
|
||||
with patch.object(tr, "release_translation_quota", side_effect=RuntimeError("db down")):
|
||||
# Must not raise even if the release itself fails.
|
||||
await tr._release_quota_if_needed("user-123", usage_recorded=False, job_id="j1")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 4 — _compute_duration_seconds (UTC-aware, crash-safe)
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestComputeDurationSeconds:
|
||||
def test_recent_timestamp_returns_positive(self):
|
||||
ts = (datetime.now(timezone.utc) - timedelta(seconds=10)).isoformat()
|
||||
dur = tr._compute_duration_seconds(ts)
|
||||
assert dur >= 9 # allow tiny scheduling slack
|
||||
|
||||
def test_z_suffix_handled(self):
|
||||
ts = (datetime.now(timezone.utc) - timedelta(seconds=5)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
dur = tr._compute_duration_seconds(ts)
|
||||
assert dur >= 0
|
||||
|
||||
def test_malformed_returns_zero(self):
|
||||
# Regression for the original bug: a bad timestamp used to crash the
|
||||
# success path and flip the job to failed. It must now return 0.
|
||||
assert tr._compute_duration_seconds("not-a-date") == 0.0
|
||||
|
||||
def test_empty_returns_zero(self):
|
||||
assert tr._compute_duration_seconds("") == 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 5 — _cleanup_old_jobs snapshots the dict (no resize-during-iteration)
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestCleanupOldJobsSnapshots:
|
||||
def test_cleanup_does_not_raise_when_dict_mutated_concurrently(self, monkeypatch):
|
||||
# Force cleanup to run now (bypass throttle).
|
||||
monkeypatch.setattr(tr, "_last_cleanup_ts", 0.0)
|
||||
monkeypatch.setattr(tr, "_CLEANUP_INTERVAL_SECONDS", 0)
|
||||
|
||||
# Two expired jobs.
|
||||
old_ts = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat()
|
||||
tr._translation_jobs.clear()
|
||||
tr._translation_jobs["j1"] = {"status": "completed", "completed_at": old_ts}
|
||||
tr._translation_jobs["j2"] = {"status": "failed", "failed_at": old_ts}
|
||||
tr._translation_jobs["j3"] = {"status": "running"} # not expired
|
||||
|
||||
# If cleanup did NOT snapshot, mutating during iteration would raise.
|
||||
tr._cleanup_old_jobs()
|
||||
|
||||
assert "j1" not in tr._translation_jobs
|
||||
assert "j2" not in tr._translation_jobs
|
||||
assert "j3" in tr._translation_jobs
|
||||
|
||||
def teardown_method(self):
|
||||
tr._translation_jobs.clear()
|
||||
@@ -40,7 +40,11 @@ async def test_translate_endpoint_triggers_tracking(client):
|
||||
with patch(
|
||||
"routes.translate_routes.storage_tracker.track_file", new_callable=AsyncMock
|
||||
) as mock_track:
|
||||
with patch("routes.translate_routes.file_validator.validate_async") as mock_val:
|
||||
# The upload write is mocked below, so no real archive exists on
|
||||
# disk: neutralize the zip safety check for this test.
|
||||
with patch(
|
||||
"routes.translate_routes.file_validator.validate_async"
|
||||
) as mock_val, patch("routes.translate_routes.validate_zip_safety"):
|
||||
mock_val.return_value.is_valid = True
|
||||
mock_val.return_value.data = {"extension": ".docx", "size_bytes": 500}
|
||||
|
||||
@@ -88,7 +92,10 @@ async def test_translate_endpoint_triggers_tracking(client):
|
||||
async def test_translate_endpoint_handles_hash_failure(client):
|
||||
app.dependency_overrides[get_authenticated_user] = mock_auth
|
||||
|
||||
with patch("routes.translate_routes.file_validator.validate_async") as mock_val:
|
||||
# No real archive exists on disk (save is mocked): skip the zip check.
|
||||
with patch(
|
||||
"routes.translate_routes.file_validator.validate_async"
|
||||
) as mock_val, patch("routes.translate_routes.validate_zip_safety"):
|
||||
mock_val.return_value.is_valid = True
|
||||
mock_val.return_value.data = {"extension": ".docx", "size_bytes": 500}
|
||||
|
||||
|
||||
@@ -564,60 +564,46 @@ class TestPptxChartWhitespace:
|
||||
</c:chartSpace>
|
||||
"""
|
||||
|
||||
def test_padded_chart_text_via_internal_method(self):
|
||||
"""The internal chart apply logic should preserve whitespace."""
|
||||
def test_chart_translation_reaches_output_file(self, tmp_path):
|
||||
"""Chart translations must reach the OUTPUT FILE (real ChartPart).
|
||||
|
||||
python-pptx's ChartPart.blob is read-only — the previous in-memory
|
||||
`blob = ...` write never landed in the saved .pptx. This end-to-end
|
||||
test uses a real chart and verifies the chart XML inside the
|
||||
output ZIP.
|
||||
"""
|
||||
from pptx.chart.data import CategoryChartData
|
||||
from pptx.enum.chart import XL_CHART_TYPE
|
||||
|
||||
provider = MockProvider({"Padded chart title": "Titre avec espaces"})
|
||||
translator = PowerPointTranslator(provider=provider)
|
||||
|
||||
# Build a chart entry by hand (simulating collect time)
|
||||
chart_xml = etree.fromstring(self.CHART_PADDED_XML.encode("utf-8"))
|
||||
entries = []
|
||||
for t_elem in chart_xml.iter(f"{{{_NS_A}}}t"):
|
||||
text_raw = t_elem.text or ""
|
||||
text = text_raw.strip()
|
||||
if not text:
|
||||
continue
|
||||
entry = {
|
||||
"element": t_elem,
|
||||
"original": text,
|
||||
"original_raw": text_raw,
|
||||
"translated": "Titre avec espaces",
|
||||
"tag": "a:t",
|
||||
"element_path": translator._get_element_path(t_elem),
|
||||
}
|
||||
entries.append(entry)
|
||||
|
||||
if not hasattr(translator, "_chart_entries"):
|
||||
translator._chart_entries = []
|
||||
|
||||
class _FakePart:
|
||||
def __init__(self, blob):
|
||||
self._blob = blob
|
||||
@property
|
||||
def blob(self):
|
||||
return self._blob
|
||||
@blob.setter
|
||||
def blob(self, value):
|
||||
self._blob = value
|
||||
|
||||
fake_part = _FakePart(
|
||||
etree.tostring(
|
||||
chart_xml, xml_declaration=True, encoding="UTF-8", standalone=True
|
||||
)
|
||||
prs = Presentation()
|
||||
slide = prs.slides.add_slide(prs.slide_layouts[5])
|
||||
chart_data = CategoryChartData()
|
||||
chart_data.categories = ["A", "B"]
|
||||
chart_data.add_series("Series 1", (1, 2))
|
||||
graphic_frame = slide.shapes.add_chart(
|
||||
XL_CHART_TYPE.COLUMN_CLUSTERED, 10, 10, 400, 300, chart_data
|
||||
)
|
||||
translator._chart_entries.append({
|
||||
"chart_part": fake_part,
|
||||
"entries": entries,
|
||||
})
|
||||
chart = graphic_frame.chart
|
||||
chart.has_title = True
|
||||
chart.chart_title.text_frame.text = "Padded chart title"
|
||||
|
||||
# Apply
|
||||
translator._apply_chart_translations(Path("dummy"))
|
||||
input_file = tmp_path / "chart_in.pptx"
|
||||
output_file = tmp_path / "chart_out.pptx"
|
||||
prs.save(str(input_file))
|
||||
|
||||
# Re-parse and check whitespace preserved
|
||||
updated_xml = etree.fromstring(fake_part._blob)
|
||||
all_t = list(updated_xml.iter(f"{{{_NS_A}}}t"))
|
||||
# Find the title text
|
||||
title_text = all_t[0].text or ""
|
||||
assert " Titre avec espaces " in title_text, (
|
||||
f"Chart whitespace not preserved: {title_text!r}"
|
||||
translator.translate_file(input_file, output_file, "fr", "en")
|
||||
|
||||
with zipfile.ZipFile(output_file, "r") as zf:
|
||||
chart_parts = [
|
||||
n for n in zf.namelist() if n.startswith("ppt/charts/chart")
|
||||
]
|
||||
assert chart_parts, "chart part missing from output file"
|
||||
chart_xml = zf.read(chart_parts[0]).decode("utf-8")
|
||||
|
||||
assert "Titre avec espaces" in chart_xml, (
|
||||
"Chart title translation never reached the output file"
|
||||
)
|
||||
assert "Padded chart title" not in chart_xml
|
||||
|
||||
139
tests/test_translators/test_pdf_quality_fixes.py
Normal file
139
tests/test_translators/test_pdf_quality_fixes.py
Normal file
@@ -0,0 +1,139 @@
|
||||
"""PDF quality fixes: table-cell merge guard, bold/italic fonts, unchanged
|
||||
blocks left untouched (no redaction/rewrite), and stats propagation."""
|
||||
|
||||
import fitz
|
||||
import pytest
|
||||
|
||||
from translators.pdf_translator import PDFTranslator
|
||||
|
||||
|
||||
class TestTableCellMergeGuard:
|
||||
def test_table_cells_never_merge(self):
|
||||
t = PDFTranslator(provider=None)
|
||||
a = {
|
||||
"bbox": (72, 100, 200, 115),
|
||||
"font_size": 11,
|
||||
"_is_table_cell": True,
|
||||
}
|
||||
b = {
|
||||
"bbox": (72, 118, 200, 133), # same column, next row
|
||||
"font_size": 11,
|
||||
"_is_table_cell": True,
|
||||
}
|
||||
assert t._should_merge_blocks(a, b) is False
|
||||
|
||||
def test_table_cell_and_paragraph_do_not_merge(self):
|
||||
t = PDFTranslator(provider=None)
|
||||
a = {"bbox": (72, 100, 200, 115), "font_size": 11, "_is_table_cell": True}
|
||||
b = {"bbox": (72, 118, 200, 133), "font_size": 11}
|
||||
assert t._should_merge_blocks(a, b) is False
|
||||
|
||||
def test_regular_paragraphs_still_merge(self):
|
||||
t = PDFTranslator(provider=None)
|
||||
a = {"bbox": (72, 100, 300, 115), "font_size": 11}
|
||||
b = {"bbox": (72, 118, 302, 133), "font_size": 11}
|
||||
assert t._should_merge_blocks(a, b) is True
|
||||
|
||||
|
||||
class TestBoldItalicFontSelection:
|
||||
def _capturing_page(self):
|
||||
page = fitz.open().new_page()
|
||||
calls = []
|
||||
original = page.insert_textbox
|
||||
|
||||
def capture(rect, text, fontname=None, fontfile=None, fontsize=None, **kw):
|
||||
calls.append({"fontname": fontname, "fontfile": fontfile})
|
||||
return 0
|
||||
|
||||
page.insert_textbox = capture
|
||||
return page, calls
|
||||
|
||||
def _block(self, **kwargs):
|
||||
block = {
|
||||
"bbox": (72, 100, 400, 130),
|
||||
"text": "Bold heading",
|
||||
"translated": "Titre en gras",
|
||||
"font_size": 20,
|
||||
"color": 0,
|
||||
"line_count": 1,
|
||||
"sub_bboxes": [(72, 100, 400, 130)],
|
||||
}
|
||||
block.update(kwargs)
|
||||
return block
|
||||
|
||||
def test_bold_block_uses_hebo(self):
|
||||
page, calls = self._capturing_page()
|
||||
t = PDFTranslator(provider=None)
|
||||
t._font_path = None # force base-14 selection
|
||||
t._write_translated_block(page, self._block(is_bold=True), None, False)
|
||||
assert calls and calls[0]["fontname"] == "hebo"
|
||||
|
||||
def test_italic_block_uses_heit(self):
|
||||
page, calls = self._capturing_page()
|
||||
t = PDFTranslator(provider=None)
|
||||
t._font_path = None
|
||||
t._write_translated_block(page, self._block(is_italic=True), None, False)
|
||||
assert calls and calls[0]["fontname"] == "heit"
|
||||
|
||||
def test_bold_italic_uses_hebi(self):
|
||||
page, calls = self._capturing_page()
|
||||
t = PDFTranslator(provider=None)
|
||||
t._font_path = None
|
||||
t._write_translated_block(
|
||||
page, self._block(is_bold=True, is_italic=True), None, False
|
||||
)
|
||||
assert calls and calls[0]["fontname"] == "hebi"
|
||||
|
||||
def test_regular_block_keeps_helv(self):
|
||||
page, calls = self._capturing_page()
|
||||
t = PDFTranslator(provider=None)
|
||||
t._font_path = None
|
||||
t._write_translated_block(page, self._block(), None, False)
|
||||
assert calls and calls[0]["fontname"] == "helv"
|
||||
|
||||
|
||||
class _IdentityProvider:
|
||||
"""Returns the input text unchanged (new-style provider)."""
|
||||
|
||||
def get_name(self):
|
||||
return "identity"
|
||||
|
||||
def is_available(self):
|
||||
return True
|
||||
|
||||
def translate_text(self, request):
|
||||
from services.providers.schemas import TranslationResponse
|
||||
|
||||
return TranslationResponse(
|
||||
translated_text=request.text,
|
||||
provider_name="identity",
|
||||
from_cache=False,
|
||||
)
|
||||
|
||||
|
||||
class TestUnchangedBlocksUntouched:
|
||||
def test_identity_translation_stats_and_no_rewrite(self, tmp_path):
|
||||
"""Provider returning the source text: stats stay changed=0 (so the
|
||||
route gate detects it) and blocks are left un-redacted."""
|
||||
doc = fitz.open()
|
||||
page = doc.new_page()
|
||||
page.insert_text((72, 100), "Already in English, nothing to do.", fontsize=11)
|
||||
src = tmp_path / "en.pdf"
|
||||
doc.save(str(src))
|
||||
doc.close()
|
||||
|
||||
t = PDFTranslator(provider=_IdentityProvider())
|
||||
out = tmp_path / "out.pdf"
|
||||
result = t.translate_file(src, out, "en", "auto")
|
||||
|
||||
assert result.exists()
|
||||
stats = t.get_translation_stats()
|
||||
assert stats["attempted"] >= 1
|
||||
assert stats["changed"] == 0
|
||||
|
||||
# The text is still there, byte-for-byte same rendering (block was
|
||||
# not redacted + rewritten in the substitute font).
|
||||
check = fitz.open(str(result))
|
||||
text = check[0].get_text("text")
|
||||
check.close()
|
||||
assert "Already in English" in text
|
||||
@@ -190,20 +190,23 @@ class TestParagraphTranslation:
|
||||
"""Tests for paragraph text translation (AC1)."""
|
||||
|
||||
def test_translate_paragraph_runs(self, tmp_path):
|
||||
"""Test that paragraph runs are translated."""
|
||||
"""Adjacent runs with identical formatting merge into ONE unit.
|
||||
|
||||
"Hello" + " " + "World" (same formatting, rsid-style splits) must be
|
||||
translated as the whole sentence, not as separate fragments.
|
||||
"""
|
||||
mock_provider = MockTranslationProvider(
|
||||
{
|
||||
"Hello": "Bonjour",
|
||||
"World": "Monde",
|
||||
"Hello World": "Bonjour le monde",
|
||||
}
|
||||
)
|
||||
translator = WordTranslator(provider=mock_provider)
|
||||
|
||||
doc = Document()
|
||||
para = doc.add_paragraph()
|
||||
run1 = para.add_run("Hello")
|
||||
run2 = para.add_run(" ")
|
||||
run3 = para.add_run("World")
|
||||
para.add_run("Hello")
|
||||
para.add_run(" ")
|
||||
para.add_run("World")
|
||||
|
||||
input_file = tmp_path / "input.docx"
|
||||
output_file = tmp_path / "output.docx"
|
||||
@@ -214,8 +217,41 @@ class TestParagraphTranslation:
|
||||
doc_out = Document(output_file)
|
||||
text = doc_out.paragraphs[0].text
|
||||
|
||||
assert "Bonjour" in text
|
||||
assert "Monde" in text
|
||||
assert text == "Bonjour le monde"
|
||||
# One merged unit → a single provider call
|
||||
assert mock_provider._call_count == 1
|
||||
|
||||
def test_bold_span_kept_separate_and_coherent(self, tmp_path):
|
||||
"""A formatting change mid-sentence splits the units; spaces survive."""
|
||||
mock_provider = MockTranslationProvider(
|
||||
{
|
||||
"This is": "Ceci est",
|
||||
"very important": "très important",
|
||||
}
|
||||
)
|
||||
translator = WordTranslator(provider=mock_provider)
|
||||
|
||||
doc = Document()
|
||||
para = doc.add_paragraph()
|
||||
para.add_run("This is ")
|
||||
bold = para.add_run("very important")
|
||||
bold.bold = True
|
||||
|
||||
input_file = tmp_path / "input.docx"
|
||||
output_file = tmp_path / "output.docx"
|
||||
doc.save(input_file)
|
||||
|
||||
translator.translate_file(input_file, output_file, "fr")
|
||||
|
||||
doc_out = Document(output_file)
|
||||
text = doc_out.paragraphs[0].text
|
||||
assert text == "Ceci est très important"
|
||||
|
||||
# Bold formatting survives on the right span
|
||||
runs = [r for r in doc_out.paragraphs[0].runs if r.text.strip()]
|
||||
assert len(runs) == 2
|
||||
assert runs[1].bold is True
|
||||
assert runs[1].text == "très important"
|
||||
|
||||
def test_empty_paragraphs_not_translated(self, tmp_path):
|
||||
"""Test that empty paragraphs are not translated."""
|
||||
|
||||
90
translators/bilingual.py
Normal file
90
translators/bilingual.py
Normal file
@@ -0,0 +1,90 @@
|
||||
"""
|
||||
Bilingual output: interleave source paragraphs with their translation.
|
||||
|
||||
Given the ORIGINAL document and the TRANSLATED document (same structure —
|
||||
the pipeline only rewrites run texts), produce a copy of the translated
|
||||
document where each translated body paragraph is preceded by its source
|
||||
paragraph in gray italic. Tables, headers and footers keep the translated
|
||||
version only (duplicating them would double the layout).
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from docx import Document
|
||||
from docx.text.paragraph import Paragraph
|
||||
from docx.oxml import OxmlElement
|
||||
from docx.oxml.ns import qn
|
||||
from docx.shared import Pt, RGBColor
|
||||
|
||||
from core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def make_bilingual_docx(
|
||||
source_path: Path, translated_path: Path, output_path: Path
|
||||
) -> Optional[Path]:
|
||||
"""Create a bilingual .docx (source paragraph above its translation).
|
||||
|
||||
Returns the output path, or None when the pairing failed (structure
|
||||
mismatch) — callers fall back to the translated-only file.
|
||||
"""
|
||||
source_path = Path(source_path)
|
||||
translated_path = Path(translated_path)
|
||||
output_path = Path(output_path)
|
||||
|
||||
try:
|
||||
src = Document(str(source_path))
|
||||
tr = Document(str(translated_path))
|
||||
except Exception as e:
|
||||
logger.warning("bilingual_open_failed", error=str(e))
|
||||
return None
|
||||
|
||||
src_children = list(src.element.body)
|
||||
tr_children = list(tr.element.body)
|
||||
|
||||
# The pipeline preserves structure exactly; a drift beyond a small
|
||||
# tolerance means pairing by index is unsafe → bail out.
|
||||
if abs(len(src_children) - len(tr_children)) > 0:
|
||||
logger.warning(
|
||||
"bilingual_structure_mismatch",
|
||||
source_elements=len(src_children),
|
||||
translated_elements=len(tr_children),
|
||||
)
|
||||
if len(src_children) != len(tr_children):
|
||||
return None
|
||||
|
||||
inserted = 0
|
||||
for src_el, tr_el in zip(src_children, tr_children):
|
||||
if tr_el.tag != qn("w:p") or src_el.tag != qn("w:p"):
|
||||
continue
|
||||
src_text = "".join(
|
||||
t.text or "" for t in src_el.iter(qn("w:t"))
|
||||
).strip()
|
||||
if not src_text:
|
||||
continue
|
||||
|
||||
# Insert a NEW paragraph directly above the translated one, inside
|
||||
# the translated document (keeps styles/sections untouched).
|
||||
new_p = OxmlElement("w:p")
|
||||
tr_el.addprevious(new_p)
|
||||
para = Paragraph(new_p, tr)
|
||||
run = para.add_run(src_text)
|
||||
run.font.size = Pt(9)
|
||||
run.font.italic = True
|
||||
run.font.color.rgb = RGBColor(0x80, 0x80, 0x80)
|
||||
inserted += 1
|
||||
|
||||
if inserted == 0:
|
||||
logger.info("bilingual_nothing_inserted")
|
||||
return None
|
||||
|
||||
try:
|
||||
tr.save(str(output_path))
|
||||
except Exception as e:
|
||||
logger.warning("bilingual_save_failed", error=str(e))
|
||||
return None
|
||||
|
||||
logger.info("bilingual_docx_created", paragraphs=inserted)
|
||||
return output_path
|
||||
@@ -107,6 +107,7 @@ class ExcelTranslator:
|
||||
self._provider = provider
|
||||
self.formula_pattern = re.compile(r"=.*")
|
||||
self._custom_prompt: Optional[str] = None
|
||||
self._tm_scope = None # set via set_tm_scope (per-user translation memory)
|
||||
self._translation_stats = {"attempted": 0, "changed": 0}
|
||||
|
||||
def set_provider(self, provider: TranslationProvider) -> None:
|
||||
@@ -116,6 +117,14 @@ class ExcelTranslator:
|
||||
def set_custom_prompt(self, prompt: Optional[str]) -> None:
|
||||
"""Set custom system prompt for LLM providers."""
|
||||
self._custom_prompt = prompt
|
||||
def set_tm_scope(self, user_id, prompt=None) -> None:
|
||||
"""Enable the per-user translation memory for this job."""
|
||||
from services.translation_tm import TMScope
|
||||
|
||||
self._tm_scope = TMScope.from_prompt(
|
||||
user_id, prompt or getattr(self, "_custom_prompt", None)
|
||||
)
|
||||
|
||||
|
||||
def translate_file(
|
||||
self,
|
||||
@@ -318,6 +327,13 @@ class ExcelTranslator:
|
||||
new_name=new_name,
|
||||
)
|
||||
|
||||
# openpyxl does NOT rewrite references on rename: cell
|
||||
# formulas, defined names and chart refs pointing at the old
|
||||
# sheet would break (#REF!/#NAME?). Charts are handled later
|
||||
# via ZIP re-injection; fix cells + defined names here.
|
||||
if sheet_name_mapping:
|
||||
self._rewrite_sheet_refs_in_workbook(workbook, sheet_name_mapping)
|
||||
|
||||
if translate_images:
|
||||
_log_info("excel_image_translation_start", sheets=len(workbook.sheetnames))
|
||||
for sheet_name in workbook.sheetnames:
|
||||
@@ -427,12 +443,30 @@ class ExcelTranslator:
|
||||
non_empty = [t for t in texts if t and t.strip()]
|
||||
self._translation_stats["attempted"] += len(non_empty)
|
||||
|
||||
from services.translation_tm import translate_with_tm
|
||||
|
||||
provider_name = (
|
||||
self._provider.get_name() if hasattr(self._provider, "get_name")
|
||||
else type(self._provider).__name__
|
||||
) if self._provider is not None else "legacy"
|
||||
|
||||
if self._provider is not None:
|
||||
translated = self._translate_with_provider(
|
||||
texts, target_language, source_language
|
||||
)
|
||||
def _do_translate(miss_texts):
|
||||
return self._translate_with_provider(
|
||||
miss_texts, target_language, source_language
|
||||
)
|
||||
else:
|
||||
translated = self._translate_with_legacy(texts, target_language, source_language)
|
||||
def _do_translate(miss_texts):
|
||||
return self._translate_with_legacy(
|
||||
miss_texts, target_language, source_language
|
||||
)
|
||||
|
||||
# Translation memory: reuse this user's previous translations
|
||||
# (identical context/prompt) before hitting the provider.
|
||||
translated = translate_with_tm(
|
||||
texts, target_language, source_language,
|
||||
provider_name, getattr(self, "_tm_scope", None), _do_translate,
|
||||
)
|
||||
|
||||
changed = sum(1 for orig, trans in zip(texts, translated) if orig != trans and trans.strip())
|
||||
self._translation_stats["changed"] += changed
|
||||
@@ -931,6 +965,140 @@ class ExcelTranslator:
|
||||
rewritten += 1
|
||||
return rewritten
|
||||
|
||||
@staticmethod
|
||||
def _rewrite_sheet_refs_in_formula(
|
||||
formula: str, sheet_name_mapping: Dict[str, str]
|
||||
) -> str:
|
||||
"""Rewrite every sheet reference inside a formula string.
|
||||
|
||||
Handles quoted ('Ventes 2026'!), unquoted (Ventes!) and 3D refs
|
||||
(Sheet1!A1:Sheet2!B2 — each end is rewritten independently).
|
||||
Longest names are replaced first so a name that is a prefix of
|
||||
another is not corrupted.
|
||||
"""
|
||||
if not formula or not sheet_name_mapping or "!" not in formula:
|
||||
return formula
|
||||
|
||||
result = formula
|
||||
# longest first to avoid partial-name collisions
|
||||
for old in sorted(sheet_name_mapping, key=len, reverse=True):
|
||||
new = sheet_name_mapping[old]
|
||||
if new == old:
|
||||
continue
|
||||
new_quoted = ExcelTranslator._quote_sheet_name_for_ref(new)
|
||||
# Quoted form: inner apostrophes are escaped as ''
|
||||
escaped = old.replace("'", "''")
|
||||
result = re.sub(
|
||||
rf"'{re.escape(escaped)}'!",
|
||||
new_quoted + "!",
|
||||
result,
|
||||
)
|
||||
# Unquoted form: only when the old name needs no quotes; guard
|
||||
# against matching the tail of a longer name.
|
||||
if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_.]*", old):
|
||||
result = re.sub(
|
||||
rf"(?<![A-Za-z0-9_.']){re.escape(old)}!",
|
||||
new_quoted + "!",
|
||||
result,
|
||||
)
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def _rewrite_sheet_refs_in_workbook(
|
||||
cls, workbook, sheet_name_mapping: Dict[str, str]
|
||||
) -> None:
|
||||
"""After a sheet rename, fix every reference that still points at the
|
||||
old names: cell formulas, defined names, data validations and
|
||||
conditional-formatting rules. openpyxl does none of this on rename.
|
||||
|
||||
Best-effort: individual failures are logged and skipped — a broken
|
||||
rename must never lose the whole file.
|
||||
"""
|
||||
if not sheet_name_mapping:
|
||||
return
|
||||
|
||||
cells_fixed = names_fixed = validations_fixed = cf_fixed = 0
|
||||
|
||||
try:
|
||||
for worksheet in workbook.worksheets:
|
||||
# 1) Cell formulas
|
||||
for row in worksheet.iter_rows():
|
||||
for cell in row:
|
||||
value = cell.value
|
||||
if isinstance(value, str) and value.startswith("=") and "!" in value:
|
||||
updated = cls._rewrite_sheet_refs_in_formula(
|
||||
value, sheet_name_mapping
|
||||
)
|
||||
if updated != value:
|
||||
cell.value = updated
|
||||
cells_fixed += 1
|
||||
|
||||
# 2) Data validations (list sources / custom formulas)
|
||||
try:
|
||||
for dv in worksheet.data_validations.dataValidation:
|
||||
for attr in ("formula1", "formula2"):
|
||||
fx = getattr(dv, attr, None)
|
||||
if isinstance(fx, str) and "!" in fx:
|
||||
updated = cls._rewrite_sheet_refs_in_formula(
|
||||
fx, sheet_name_mapping
|
||||
)
|
||||
if updated != fx:
|
||||
setattr(dv, attr, updated)
|
||||
validations_fixed += 1
|
||||
except Exception as e:
|
||||
_log_warning(
|
||||
"excel_sheet_refs_validations_failed",
|
||||
sheet=worksheet.title,
|
||||
error=str(e),
|
||||
)
|
||||
|
||||
# 3) Conditional formatting rules
|
||||
try:
|
||||
for cf in worksheet.conditional_formatting:
|
||||
for rule in cf.rules:
|
||||
formulas = getattr(rule, "formula", None) or []
|
||||
for idx, fx in enumerate(formulas):
|
||||
if isinstance(fx, str) and "!" in fx:
|
||||
updated = cls._rewrite_sheet_refs_in_formula(
|
||||
fx, sheet_name_mapping
|
||||
)
|
||||
if updated != fx:
|
||||
formulas[idx] = updated
|
||||
cf_fixed += 1
|
||||
except Exception as e:
|
||||
_log_warning(
|
||||
"excel_sheet_refs_condfmt_failed",
|
||||
sheet=worksheet.title,
|
||||
error=str(e),
|
||||
)
|
||||
|
||||
# 4) Workbook-level defined names
|
||||
try:
|
||||
for name in list(workbook.defined_names.values()):
|
||||
attr_text = getattr(name, "attr_text", None)
|
||||
if isinstance(attr_text, str) and "!" in attr_text:
|
||||
updated = cls._rewrite_sheet_refs_in_formula(
|
||||
attr_text, sheet_name_mapping
|
||||
)
|
||||
if updated != attr_text:
|
||||
name.attr_text = updated
|
||||
names_fixed += 1
|
||||
except Exception as e:
|
||||
_log_warning("excel_sheet_refs_names_failed", error=str(e))
|
||||
|
||||
except Exception as e:
|
||||
_log_error("excel_sheet_refs_rewrite_failed", error=str(e))
|
||||
return
|
||||
|
||||
if cells_fixed or names_fixed or validations_fixed or cf_fixed:
|
||||
_log_info(
|
||||
"excel_sheet_refs_rewritten",
|
||||
cells=cells_fixed,
|
||||
defined_names=names_fixed,
|
||||
validations=validations_fixed,
|
||||
conditional_formats=cf_fixed,
|
||||
)
|
||||
|
||||
def _translate_images(self, worksheet: Worksheet, target_language: str) -> None:
|
||||
"""
|
||||
Translate text in images using vision model.
|
||||
|
||||
@@ -21,6 +21,13 @@ Fallback:
|
||||
|
||||
Text-only mode:
|
||||
Extract text, translate, generate a clean formatted PDF via reportlab.
|
||||
|
||||
Scanned PDFs:
|
||||
Image-only PDFs have no text layer to extract or rewrite. They are
|
||||
detected up front (average extractable characters per page below
|
||||
config.SCANNED_PDF_MIN_CHARS_PER_PAGE) and routed through the Mistral
|
||||
OCR API (services/mistral_ocr.py) before translation; the output is a
|
||||
clean re-typeset PDF (layout is not preserved — the source is images).
|
||||
"""
|
||||
|
||||
import time
|
||||
@@ -114,8 +121,15 @@ class PDFTranslator:
|
||||
"/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf",
|
||||
"/usr/share/fonts/truetype/freefont/FreeSans.ttf",
|
||||
"/app/fonts/NotoSans-Regular.ttf",
|
||||
# CJK-capable fonts (target languages zh/ja/ko render as tofu with
|
||||
# a Latin-only font file)
|
||||
"/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc",
|
||||
"/usr/share/fonts/noto-cjk/NotoSansCJK-Regular.ttc",
|
||||
"/usr/share/fonts/opentype/noto/NotoSansCJKsc-Regular.otf",
|
||||
"C:/Windows/Fonts/arial.ttf",
|
||||
"C:/Windows/Fonts/msyh.ttc",
|
||||
"C:/Windows/Fonts/simsun.ttc",
|
||||
"C:/Windows/Fonts/msgothic.ttc",
|
||||
"/System/Library/Fonts/Helvetica.ttc",
|
||||
]
|
||||
|
||||
@@ -124,6 +138,11 @@ class PDFTranslator:
|
||||
self._font_path: Optional[str] = None
|
||||
self._translation_stats = {"attempted": 0, "changed": 0}
|
||||
self._custom_prompt: Optional[str] = None
|
||||
# OCR overrides (admin settings); None → fall back to config.MISTRAL_*
|
||||
self._ocr_api_key: Optional[str] = None
|
||||
self._ocr_model: Optional[str] = None
|
||||
self._ocr_timeout: Optional[int] = None
|
||||
self._ocr_enabled: Optional[bool] = None
|
||||
|
||||
def set_provider(self, provider) -> None:
|
||||
"""Set the translation provider."""
|
||||
@@ -133,6 +152,26 @@ class PDFTranslator:
|
||||
"""Set custom system prompt for LLM providers."""
|
||||
self._custom_prompt = prompt
|
||||
|
||||
def set_ocr_config(
|
||||
self,
|
||||
api_key: Optional[str] = None,
|
||||
model: Optional[str] = None,
|
||||
timeout: Optional[int] = None,
|
||||
enabled: Optional[bool] = None,
|
||||
) -> None:
|
||||
"""Configure the Mistral OCR call (admin settings > env defaults).
|
||||
|
||||
Only the values explicitly provided override the config defaults,
|
||||
so the route can pass admin-configured values and let the rest fall
|
||||
back to ``config.MISTRAL_*``.
|
||||
"""
|
||||
self._ocr_api_key = api_key
|
||||
self._ocr_model = model
|
||||
if timeout is not None:
|
||||
self._ocr_timeout = timeout
|
||||
if enabled is not None:
|
||||
self._ocr_enabled = enabled
|
||||
|
||||
def _get_font_path(self) -> Optional[str]:
|
||||
"""Resolve a Unicode-capable TTF/OTF font file."""
|
||||
if self._font_path is not None:
|
||||
@@ -159,6 +198,14 @@ class PDFTranslator:
|
||||
output_path = Path(output_path)
|
||||
self._validate_file(input_path)
|
||||
|
||||
# Scanned PDFs must be detected before either mode: both rely on an
|
||||
# extractable text layer, which image-only pages don't have.
|
||||
if self._is_scanned_pdf(input_path):
|
||||
return self._translate_scanned_pdf(
|
||||
input_path, output_path, target_language, source_language,
|
||||
progress_callback,
|
||||
)
|
||||
|
||||
if pdf_mode == "text_only":
|
||||
return self._translate_text_only(
|
||||
input_path, output_path, target_language, source_language, progress_callback
|
||||
@@ -294,8 +341,22 @@ class PDFTranslator:
|
||||
original, target_language, source_language
|
||||
)
|
||||
if translated and translated.strip():
|
||||
block["translated"] = translated
|
||||
translated_blocks += 1
|
||||
if translated.strip() == original.strip():
|
||||
# Unchanged (already in the target language, or
|
||||
# the provider failed and returned the source
|
||||
# text). Leave the block completely untouched —
|
||||
# redacting and rewriting identical text would
|
||||
# only degrade its typography (embedded fonts
|
||||
# lost, everything redrawn in the substitute).
|
||||
block["translated"] = None
|
||||
logger.info(
|
||||
"block_translation_unchanged",
|
||||
page=page_num + 1,
|
||||
text_preview=original[:60],
|
||||
)
|
||||
else:
|
||||
block["translated"] = translated
|
||||
translated_blocks += 1
|
||||
else:
|
||||
logger.warning(
|
||||
"block_translation_empty",
|
||||
@@ -357,6 +418,7 @@ class PDFTranslator:
|
||||
return True
|
||||
return False
|
||||
|
||||
redacted_rects: list = []
|
||||
for block in blocks:
|
||||
if block.get("translated"):
|
||||
# Track B3.5: single redaction per block, not per sub-bbox.
|
||||
@@ -371,6 +433,7 @@ class PDFTranslator:
|
||||
page.add_redact_annot(block_bbox, fill=None)
|
||||
else:
|
||||
page.add_redact_annot(block_bbox, fill=(1, 1, 1))
|
||||
redacted_rects.append(block_bbox)
|
||||
|
||||
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
||||
|
||||
@@ -378,7 +441,11 @@ class PDFTranslator:
|
||||
# We use the original link geometry (it's unaffected by the
|
||||
# redaction). URIs are preserved verbatim; only the visible
|
||||
# text changes.
|
||||
if page_links_before:
|
||||
# Only links intersecting an actually-redacted block are
|
||||
# re-inserted: when a page has no redaction at all (every block
|
||||
# unchanged), the original annotations survive apply_redactions
|
||||
# and re-inserting would DUPLICATE them.
|
||||
if page_links_before and redacted_rects:
|
||||
reinserted = 0
|
||||
lost = 0
|
||||
page_rect = page.rect
|
||||
@@ -393,6 +460,12 @@ class PDFTranslator:
|
||||
if not page_rect.intersects(from_rect):
|
||||
lost += 1
|
||||
continue
|
||||
# Skip links outside every redacted area — their
|
||||
# original annotation is still alive on the page.
|
||||
if not any(
|
||||
from_rect.intersects(r) for r in redacted_rects
|
||||
):
|
||||
continue
|
||||
# Build the link insertion kwargs based on kind
|
||||
if link.get("uri"):
|
||||
# External URI link
|
||||
@@ -700,6 +773,13 @@ class PDFTranslator:
|
||||
if a.get("_no_merge") or b.get("_no_merge"):
|
||||
return False
|
||||
|
||||
# Table cells: consecutive rows of the same column satisfy every
|
||||
# geometric merge condition (same x0, similar width, small positive
|
||||
# gap) but are UNRELATED values — merging them joins two cells into
|
||||
# one paragraph spanning both rows and breaks the table structure.
|
||||
if a.get("_is_table_cell") or b.get("_is_table_cell"):
|
||||
return False
|
||||
|
||||
# Must have similar font size (within 20%)
|
||||
if abs(a["font_size"] - b["font_size"]) > max(a["font_size"], b["font_size"]) * 0.2:
|
||||
return False
|
||||
@@ -812,8 +892,22 @@ class PDFTranslator:
|
||||
|
||||
# PyMuPDF bug: fontname=None raises AttributeError. Default to 'helv'.
|
||||
# If a custom font file is available, use it via fontfile (fontname ignored).
|
||||
fontname = "helv"
|
||||
fontfile = font_path
|
||||
# Base-14 font variant honouring the block's bold/italic flags —
|
||||
# previously every block (headings included) rendered in regular.
|
||||
# When a custom Unicode fontfile is used we keep it: we have no
|
||||
# bold/italic variant of that file, and glyph coverage matters more
|
||||
# than weight.
|
||||
if font_path:
|
||||
fontname = "helv"
|
||||
fontfile = font_path
|
||||
else:
|
||||
fontname = (
|
||||
"hebi" if (block.get("is_bold") and block.get("is_italic"))
|
||||
else "hebo" if block.get("is_bold")
|
||||
else "heit" if block.get("is_italic")
|
||||
else "helv"
|
||||
)
|
||||
fontfile = None
|
||||
|
||||
# Determine if this is a heading (larger font size = more visual weight)
|
||||
is_heading = target_size >= HEADING_MIN_SIZE
|
||||
@@ -998,6 +1092,12 @@ class PDFTranslator:
|
||||
progress_callback=None,
|
||||
translate_images=translate_images,
|
||||
)
|
||||
# Propagate the inner Word stats so the route's sanity gate
|
||||
# (attempted/changed) works on the fallback path too.
|
||||
for key, value in wt.get_translation_stats().items():
|
||||
self._translation_stats[key] = (
|
||||
self._translation_stats.get(key, 0) + value
|
||||
)
|
||||
|
||||
if progress_callback:
|
||||
progress_callback({
|
||||
@@ -1125,6 +1225,31 @@ class PDFTranslator:
|
||||
pages_text.append(text)
|
||||
doc.close()
|
||||
|
||||
translated_pages = self._translate_page_texts(
|
||||
pages_text, target_language, source_language, progress_callback
|
||||
)
|
||||
|
||||
final_path = output_path.with_suffix(".pdf")
|
||||
self._generate_clean_pdf(translated_pages, final_path, target_language)
|
||||
|
||||
processing_time_ms = round((time.time() - start_time) * 1000, 2)
|
||||
logger.info(
|
||||
"pdf_text_only_success",
|
||||
file_name=input_path.name,
|
||||
pages=total_pages,
|
||||
processing_time_ms=processing_time_ms,
|
||||
)
|
||||
|
||||
return final_path
|
||||
|
||||
def _translate_page_texts(
|
||||
self,
|
||||
pages_text: List[str],
|
||||
target_language: str,
|
||||
source_language: str,
|
||||
progress_callback,
|
||||
) -> List[str]:
|
||||
"""Translate per-page texts, keeping order; empty pages pass through."""
|
||||
non_empty_indices = [i for i, t in enumerate(pages_text) if t]
|
||||
|
||||
if progress_callback:
|
||||
@@ -1135,6 +1260,7 @@ class PDFTranslator:
|
||||
})
|
||||
|
||||
translated_pages = list(pages_text)
|
||||
total_pages = len(pages_text)
|
||||
|
||||
for seq, page_idx in enumerate(non_empty_indices):
|
||||
text = pages_text[page_idx]
|
||||
@@ -1160,19 +1286,139 @@ class PDFTranslator:
|
||||
"progress_override": pct,
|
||||
})
|
||||
|
||||
return translated_pages
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# SCANNED PDFs — Mistral OCR
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def _is_scanned_pdf(self, input_path: Path) -> bool:
|
||||
"""True when the PDF is image-only (no usable text layer).
|
||||
|
||||
A page counts as a scan page when it has almost no extractable
|
||||
text AND is mostly covered by raster images. The document is
|
||||
considered scanned when it contains scan pages and no page with a
|
||||
real text layer — this keeps sparse-but-textual PDFs (a bare
|
||||
title page, a single label) on the normal layout pipeline.
|
||||
"""
|
||||
from config import config
|
||||
|
||||
try:
|
||||
import fitz
|
||||
except ImportError:
|
||||
return False
|
||||
|
||||
try:
|
||||
with fitz.open(str(input_path)) as doc:
|
||||
if len(doc) == 0:
|
||||
return False
|
||||
has_text_page = False
|
||||
has_scan_page = False
|
||||
for page in doc:
|
||||
if len(page.get_text("text").strip()) >= config.SCANNED_PDF_MIN_CHARS_PER_PAGE:
|
||||
has_text_page = True
|
||||
continue
|
||||
# Text-poor page: a scan page only when raster images
|
||||
# cover most of its area (a bare title page has none).
|
||||
page_area = abs(page.rect)
|
||||
img_area = 0.0
|
||||
for img in page.get_images(full=True):
|
||||
for rect in page.get_image_rects(img[0]):
|
||||
img_area += abs(rect & page.rect)
|
||||
if page_area and img_area / page_area >= 0.5:
|
||||
has_scan_page = True
|
||||
except Exception as e:
|
||||
logger.warning("scanned_pdf_detection_failed", error=str(e))
|
||||
return False
|
||||
|
||||
scanned = has_scan_page and not has_text_page
|
||||
if scanned:
|
||||
logger.info(
|
||||
"scanned_pdf_detected",
|
||||
file=input_path.name,
|
||||
)
|
||||
return scanned
|
||||
|
||||
def _translate_scanned_pdf(
|
||||
self,
|
||||
input_path: Path,
|
||||
output_path: Path,
|
||||
target_language: str,
|
||||
source_language: str,
|
||||
progress_callback,
|
||||
) -> Path:
|
||||
"""OCR (Mistral) → translate → clean re-typeset PDF.
|
||||
|
||||
The source pages are images, so the original layout cannot be
|
||||
rewritten in place; the output carries the recovered text in a
|
||||
clean document instead.
|
||||
"""
|
||||
from config import config
|
||||
from services.mistral_ocr import MistralOCRClient, MistralOCRError
|
||||
|
||||
# Resolution order: admin settings (via set_ocr_config) > env/config.
|
||||
api_key = (self._ocr_api_key or "").strip() or config.MISTRAL_API_KEY
|
||||
model = (self._ocr_model or "").strip() or config.MISTRAL_OCR_MODEL
|
||||
timeout = self._ocr_timeout or config.MISTRAL_OCR_TIMEOUT
|
||||
ocr_enabled = (
|
||||
config.MISTRAL_OCR_ENABLED
|
||||
if self._ocr_enabled is None
|
||||
else self._ocr_enabled
|
||||
)
|
||||
|
||||
if not ocr_enabled or not api_key:
|
||||
raise RuntimeError(
|
||||
"PDF scanné détecté (pages image sans couche texte). "
|
||||
"La traduction des PDF scannés nécessite l'OCR Mistral : "
|
||||
"configurez MISTRAL_API_KEY (ou fournissez un PDF avec du texte sélectionnable)."
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
client = MistralOCRClient(
|
||||
api_key=api_key,
|
||||
model=model,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
pages_markdown = client.extract_pdf_text(
|
||||
input_path, progress_callback=progress_callback
|
||||
)
|
||||
pages_text = [self._markdown_to_text(md) for md in pages_markdown]
|
||||
|
||||
translated_pages = self._translate_page_texts(
|
||||
pages_text, target_language, source_language, progress_callback
|
||||
)
|
||||
|
||||
final_path = output_path.with_suffix(".pdf")
|
||||
self._generate_clean_pdf(translated_pages, final_path, target_language)
|
||||
|
||||
processing_time_ms = round((time.time() - start_time) * 1000, 2)
|
||||
logger.info(
|
||||
"pdf_text_only_success",
|
||||
"pdf_scanned_success",
|
||||
file_name=input_path.name,
|
||||
pages=total_pages,
|
||||
pages=len(pages_text),
|
||||
processing_time_ms=processing_time_ms,
|
||||
)
|
||||
|
||||
return final_path
|
||||
|
||||
@staticmethod
|
||||
def _markdown_to_text(markdown: str) -> str:
|
||||
"""Flatten OCR markdown to plain text (drop images/links/markup)."""
|
||||
import re
|
||||
|
||||
if not markdown:
|
||||
return ""
|
||||
text = re.sub(r"!\[[^\]]*\]\([^)]*\)", "", markdown) # images
|
||||
text = re.sub(r"\[([^\]]*)\]\([^)]*\)", r"\1", text) # links → label
|
||||
text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE) # headings
|
||||
text = re.sub(r"^\s*[-*+]\s+", "", text, flags=re.MULTILINE) # bullets
|
||||
# Markdown table rows → plain line of cells
|
||||
text = re.sub(r"^\s*\|", "", text, flags=re.MULTILINE)
|
||||
text = text.replace("|", " ")
|
||||
text = re.sub(r"^\s*[-:| ]+\s*$", "", text, flags=re.MULTILINE) # rules
|
||||
text = re.sub(r"\n{3,}", "\n\n", text)
|
||||
return text.strip()
|
||||
|
||||
def _generate_clean_pdf(
|
||||
self, pages_text: List[str], output_path: Path, target_language: str = "en"
|
||||
) -> None:
|
||||
@@ -1305,18 +1551,29 @@ class PDFTranslator:
|
||||
def _translate_single(
|
||||
self, text: str, target_language: str, source_language: str
|
||||
) -> str:
|
||||
"""Translate a single text string."""
|
||||
"""Translate a single text string.
|
||||
|
||||
Also feeds the job-level attempted/changed stats so the route can
|
||||
detect a total provider failure (changed == 0) on PDFs too.
|
||||
"""
|
||||
if text and text.strip():
|
||||
self._translation_stats["attempted"] += 1
|
||||
if self._provider is not None:
|
||||
try:
|
||||
results = self._translate_with_provider([text], target_language, source_language)
|
||||
if results and results[0].strip():
|
||||
if results[0].strip() != text.strip():
|
||||
self._translation_stats["changed"] += 1
|
||||
return results[0]
|
||||
except Exception as e:
|
||||
logger.warning("provider_single_failed", error=str(e))
|
||||
|
||||
from services.translation_service import translation_service
|
||||
try:
|
||||
return translation_service.translate_text(text, target_language, source_language)
|
||||
result = translation_service.translate_text(text, target_language, source_language)
|
||||
if result and result.strip() and result.strip() != text.strip():
|
||||
self._translation_stats["changed"] += 1
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.warning("legacy_single_failed", error=str(e))
|
||||
return text
|
||||
|
||||
@@ -95,6 +95,69 @@ def _apply_rtl_to_shape(shape) -> None:
|
||||
_apply_rtl_to_shape(sub_shape)
|
||||
|
||||
|
||||
# East-Asian typeface hints per target language (DrawingML <a:ea>)
|
||||
_EA_TYPEFACES = {
|
||||
"zh": "SimSun",
|
||||
"zh-CN": "SimSun",
|
||||
"zh-TW": "PMingLiU",
|
||||
"ja": "Yu Mincho",
|
||||
"ko": "Batang",
|
||||
}
|
||||
|
||||
|
||||
def _ea_typeface_for_target(target_language: str):
|
||||
code = (target_language or "").strip()
|
||||
base = code.split("-")[0].lower()
|
||||
return _EA_TYPEFACES.get(code) or _EA_TYPEFACES.get(base)
|
||||
|
||||
|
||||
def _apply_ea_font_hints(presentation: Presentation, target_language: str) -> None:
|
||||
"""Set the <a:ea> (east-asian) typeface on every run for CJK targets.
|
||||
|
||||
Blanket application is safe — the hint only affects CJK glyphs.
|
||||
"""
|
||||
|
||||
def _hint_shape(shape) -> int:
|
||||
hinted = 0
|
||||
if shape.has_text_frame:
|
||||
hinted += _hint_text_frame(shape.text_frame)
|
||||
if shape.shape_type == MSO_SHAPE_TYPE.TABLE:
|
||||
for row in shape.table.rows:
|
||||
for cell in row.cells:
|
||||
hinted += _hint_text_frame(cell.text_frame)
|
||||
if shape.shape_type == MSO_SHAPE_TYPE.GROUP:
|
||||
for sub_shape in shape.shapes:
|
||||
hinted += _hint_shape(sub_shape)
|
||||
return hinted
|
||||
|
||||
def _hint_text_frame(text_frame) -> int:
|
||||
hinted = 0
|
||||
tag_rPr = f"{{{_NS_A}}}rPr"
|
||||
tag_ea = f"{{{_NS_A}}}ea"
|
||||
for paragraph in text_frame.paragraphs:
|
||||
for run in paragraph.runs:
|
||||
rPr = run._r.find(tag_rPr)
|
||||
if rPr is None:
|
||||
rPr = etree.SubElement(run._r, tag_rPr)
|
||||
ea = rPr.find(tag_ea)
|
||||
if ea is None:
|
||||
ea = etree.SubElement(rPr, tag_ea)
|
||||
if not ea.get("typeface"):
|
||||
ea.set("typeface", typeface)
|
||||
hinted += 1
|
||||
return hinted
|
||||
|
||||
typeface = _ea_typeface_for_target(target_language)
|
||||
if not typeface:
|
||||
return
|
||||
total = 0
|
||||
for slide in presentation.slides:
|
||||
for shape in slide.shapes:
|
||||
total += _hint_shape(shape)
|
||||
if total:
|
||||
_log_info("pptx_ea_font_hints_applied", runs=total, typeface=typeface)
|
||||
|
||||
|
||||
class PptxProcessorError(Exception):
|
||||
"""Exception for PowerPoint processing errors with structured error codes."""
|
||||
|
||||
@@ -152,6 +215,7 @@ class PowerPointTranslator:
|
||||
"""
|
||||
self._provider = provider
|
||||
self._custom_prompt: Optional[str] = None
|
||||
self._tm_scope = None # set via set_tm_scope (per-user translation memory)
|
||||
self._translation_stats = {"attempted": 0, "changed": 0}
|
||||
|
||||
def set_provider(self, provider: TranslationProvider) -> None:
|
||||
@@ -161,6 +225,14 @@ class PowerPointTranslator:
|
||||
def set_custom_prompt(self, prompt: Optional[str]) -> None:
|
||||
"""Set custom system prompt for LLM providers."""
|
||||
self._custom_prompt = prompt
|
||||
def set_tm_scope(self, user_id, prompt=None) -> None:
|
||||
"""Enable the per-user translation memory for this job."""
|
||||
from services.translation_tm import TMScope
|
||||
|
||||
self._tm_scope = TMScope.from_prompt(
|
||||
user_id, prompt or getattr(self, "_custom_prompt", None)
|
||||
)
|
||||
|
||||
|
||||
def translate_file(
|
||||
self,
|
||||
@@ -316,6 +388,10 @@ class PowerPointTranslator:
|
||||
if target_language.lower() in RTL_LANGUAGES:
|
||||
_apply_rtl_to_presentation(presentation)
|
||||
|
||||
# CJK font hint so the target script renders with a proper
|
||||
# typeface instead of shape-dependent fallbacks.
|
||||
_apply_ea_font_hints(presentation, target_language)
|
||||
|
||||
if translate_images:
|
||||
try:
|
||||
self._translate_images(presentation, target_language)
|
||||
@@ -405,12 +481,30 @@ class PowerPointTranslator:
|
||||
non_empty = [t for t in texts if t and t.strip()]
|
||||
self._translation_stats["attempted"] += len(non_empty)
|
||||
|
||||
from services.translation_tm import translate_with_tm
|
||||
|
||||
provider_name = (
|
||||
self._provider.get_name() if hasattr(self._provider, "get_name")
|
||||
else type(self._provider).__name__
|
||||
) if self._provider is not None else "legacy"
|
||||
|
||||
if self._provider is not None:
|
||||
translated = self._translate_with_provider(
|
||||
texts, target_language, source_language
|
||||
)
|
||||
def _do_translate(miss_texts):
|
||||
return self._translate_with_provider(
|
||||
miss_texts, target_language, source_language
|
||||
)
|
||||
else:
|
||||
translated = self._translate_with_legacy(texts, target_language, source_language)
|
||||
def _do_translate(miss_texts):
|
||||
return self._translate_with_legacy(
|
||||
miss_texts, target_language, source_language
|
||||
)
|
||||
|
||||
# Translation memory: reuse this user's previous translations
|
||||
# (identical context/prompt) before hitting the provider.
|
||||
translated = translate_with_tm(
|
||||
texts, target_language, source_language,
|
||||
provider_name, getattr(self, "_tm_scope", None), _do_translate,
|
||||
)
|
||||
|
||||
changed = sum(1 for orig, trans in zip(texts, translated) if orig != trans and trans.strip())
|
||||
self._translation_stats["changed"] += changed
|
||||
@@ -743,7 +837,14 @@ class PowerPointTranslator:
|
||||
return None
|
||||
|
||||
def _apply_chart_translations(self, output_path: Path) -> None:
|
||||
"""Re-inject chart text translations by modifying chart XML parts.
|
||||
"""Re-inject chart text translations into the saved .pptx ZIP.
|
||||
|
||||
python-pptx's ChartPart exposes a read-only ``blob`` property, so an
|
||||
in-memory `chart_part.blob = ...` assignment fails silently and the
|
||||
chart text never reaches the output file. Instead — exactly like the
|
||||
Word translator — we translate into a fresh parse of each chart
|
||||
part's XML and rewrite the corresponding ZIP entries of the
|
||||
already-saved output file.
|
||||
|
||||
Matching strategy: prefer the stored `element_path` (set at collect
|
||||
time) to navigate directly to the right element. Fall back to
|
||||
@@ -759,6 +860,9 @@ class PowerPointTranslator:
|
||||
total_translated = 0
|
||||
total_skipped = 0
|
||||
|
||||
# partname (e.g. "ppt/charts/chart1.xml") → updated XML bytes
|
||||
updated_parts: Dict[str, bytes] = {}
|
||||
|
||||
for chart_data in self._chart_entries:
|
||||
entries = chart_data['entries']
|
||||
chart_part = chart_data['chart_part']
|
||||
@@ -804,17 +908,39 @@ class PowerPointTranslator:
|
||||
target.text = leading + (entry['translated'] or '').strip() + trailing
|
||||
total_translated += 1
|
||||
|
||||
# Update the chart part blob
|
||||
chart_part.blob = etree.tostring(
|
||||
chart_xml,
|
||||
xml_declaration=True,
|
||||
encoding='UTF-8',
|
||||
standalone=True,
|
||||
)
|
||||
part_name = str(getattr(chart_part, "partname", "")).lstrip("/")
|
||||
if part_name:
|
||||
updated_parts[part_name] = etree.tostring(
|
||||
chart_xml,
|
||||
xml_declaration=True,
|
||||
encoding='UTF-8',
|
||||
standalone=True,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
_log_error("pptx_chart_update_error", error=str(e))
|
||||
|
||||
# Single ZIP rewrite pass for all updated chart parts
|
||||
if updated_parts:
|
||||
import zipfile
|
||||
import io as _io
|
||||
try:
|
||||
with zipfile.ZipFile(output_path, 'r') as zf_in:
|
||||
buf = _io.BytesIO()
|
||||
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf_out:
|
||||
for item in zf_in.namelist():
|
||||
data = updated_parts.get(item, zf_in.read(item))
|
||||
zf_out.writestr(item, data)
|
||||
with open(output_path, 'wb') as f:
|
||||
f.write(buf.getvalue())
|
||||
_log_info(
|
||||
"pptx_charts_rewritten",
|
||||
chart_parts=len(updated_parts),
|
||||
translated=total_translated,
|
||||
)
|
||||
except Exception as e:
|
||||
_log_error("pptx_chart_zip_rewrite_error", error=str(e))
|
||||
|
||||
# Clean up
|
||||
self._chart_entries = []
|
||||
|
||||
|
||||
@@ -31,6 +31,73 @@ RTL_LANGUAGES: frozenset = frozenset(
|
||||
{"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"}
|
||||
)
|
||||
|
||||
# East-Asian / complex-script font hints: when the target language uses
|
||||
# glyphs a Latin theme font lacks, Word falls back to a substitute —
|
||||
# setting the eastAsia (CJK) or cs (Arabic script) typeface keeps the
|
||||
# rendering consistent across runs.
|
||||
CJK_EASTASIA_FONTS: dict = {
|
||||
"zh": "SimSun",
|
||||
"zh-CN": "SimSun",
|
||||
"zh-TW": "PMingLiU",
|
||||
"ja": "Yu Mincho",
|
||||
"ko": "Batang",
|
||||
}
|
||||
CS_FONTS: dict = {
|
||||
"ar": "Arial",
|
||||
"he": "Arial",
|
||||
"fa": "Arial",
|
||||
"ur": "Arial",
|
||||
}
|
||||
|
||||
|
||||
def _font_hints_for_target(target_language: str):
|
||||
"""(eastAsia_font, cs_font) hints for the target language, if any."""
|
||||
code = (target_language or "").strip()
|
||||
base = code.split("-")[0].lower()
|
||||
return CJK_EASTASIA_FONTS.get(code) or CJK_EASTASIA_FONTS.get(base), CS_FONTS.get(base)
|
||||
|
||||
|
||||
def _apply_font_hints(document: Document, target_language: str) -> None:
|
||||
"""Set eastAsia/cs typeface hints on every run for CJK/Arabic targets.
|
||||
|
||||
Blanket application is safe: the hint only affects the glyphs of that
|
||||
script, which Latin text does not contain.
|
||||
"""
|
||||
eastasia, cs = _font_hints_for_target(target_language)
|
||||
if not eastasia and not cs:
|
||||
return
|
||||
|
||||
runs = []
|
||||
for para in document.paragraphs:
|
||||
runs.extend(para.runs)
|
||||
for table in document.tables:
|
||||
for row in table.rows:
|
||||
for cell in row.cells:
|
||||
for para in cell.paragraphs:
|
||||
runs.extend(para.runs)
|
||||
for section in document.sections:
|
||||
for hf in (section.header, section.footer):
|
||||
for para in hf.paragraphs:
|
||||
runs.extend(para.runs)
|
||||
|
||||
hinted = 0
|
||||
for run in runs:
|
||||
rPr = run._r.get_or_add_rPr()
|
||||
rFonts = rPr.find(qn("w:rFonts"))
|
||||
if rFonts is None:
|
||||
rFonts = OxmlElement("w:rFonts")
|
||||
rPr.insert(0, rFonts)
|
||||
if eastasia and not rFonts.get(qn("w:eastAsia")):
|
||||
rFonts.set(qn("w:eastAsia"), eastasia)
|
||||
hinted += 1
|
||||
if cs and not rFonts.get(qn("w:cs")):
|
||||
rFonts.set(qn("w:cs"), cs)
|
||||
hinted += 1
|
||||
|
||||
if hinted:
|
||||
from core.logging import get_logger as _gl
|
||||
_gl(__name__).info("word_font_hints_applied", runs=hinted, eastasia=eastasia, cs=cs)
|
||||
|
||||
|
||||
from core.logging import get_logger
|
||||
|
||||
@@ -62,7 +129,9 @@ def _set_paragraph_rtl(paragraph: Paragraph) -> None:
|
||||
|
||||
Sets:
|
||||
- w:pPr/w:bidi → paragraph text direction = RTL
|
||||
- w:pPr/w:jc → alignment = right
|
||||
- w:pPr/w:jc → mirrored alignment (left→right), ONLY when the
|
||||
paragraph has no explicit alignment — centered/justified titles
|
||||
must not be forced right-aligned.
|
||||
- w:rPr/w:rtl → run-level RTL marker for each run
|
||||
"""
|
||||
pPr = paragraph._p.get_or_add_pPr()
|
||||
@@ -71,10 +140,12 @@ def _set_paragraph_rtl(paragraph: Paragraph) -> None:
|
||||
pPr.append(OxmlElement("w:bidi"))
|
||||
|
||||
jc = pPr.find(qn("w:jc"))
|
||||
if jc is None:
|
||||
jc = OxmlElement("w:jc")
|
||||
pPr.append(jc)
|
||||
jc.set(qn("w:val"), "right")
|
||||
explicit_alignment = jc is not None and jc.get(qn("w:val")) not in (None, "", "left")
|
||||
if not explicit_alignment:
|
||||
if jc is None:
|
||||
jc = OxmlElement("w:jc")
|
||||
pPr.append(jc)
|
||||
jc.set(qn("w:val"), "right")
|
||||
|
||||
for run in paragraph.runs:
|
||||
rPr = run._r.get_or_add_rPr()
|
||||
@@ -173,6 +244,7 @@ class WordTranslator:
|
||||
self._provider = provider
|
||||
self._custom_prompt: Optional[str] = None
|
||||
self._translation_stats = {"attempted": 0, "changed": 0}
|
||||
self._tm_scope = None # set via set_tm_scope (per-user translation memory)
|
||||
|
||||
def set_provider(self, provider: TranslationProvider) -> None:
|
||||
"""Set the translation provider."""
|
||||
@@ -182,6 +254,12 @@ class WordTranslator:
|
||||
"""Set custom system prompt for LLM providers."""
|
||||
self._custom_prompt = prompt
|
||||
|
||||
def set_tm_scope(self, user_id: Optional[str], prompt: Optional[str] = None) -> None:
|
||||
"""Enable the per-user translation memory for this job."""
|
||||
from services.translation_tm import TMScope
|
||||
|
||||
self._tm_scope = TMScope.from_prompt(user_id, prompt or self._custom_prompt)
|
||||
|
||||
def translate_file(
|
||||
self,
|
||||
input_path: Path,
|
||||
@@ -333,6 +411,10 @@ class WordTranslator:
|
||||
if target_language.lower() in RTL_LANGUAGES:
|
||||
_apply_rtl_to_document(document)
|
||||
|
||||
# CJK / Arabic-script font hints so Word renders the target
|
||||
# script with a proper typeface instead of per-run fallbacks.
|
||||
_apply_font_hints(document, target_language)
|
||||
|
||||
if progress_callback:
|
||||
progress_callback(
|
||||
{
|
||||
@@ -461,12 +543,30 @@ class WordTranslator:
|
||||
non_empty = [t for t in texts if t and t.strip()]
|
||||
self._translation_stats["attempted"] += len(non_empty)
|
||||
|
||||
from services.translation_tm import translate_with_tm
|
||||
|
||||
provider_name = (
|
||||
self._provider.get_name() if hasattr(self._provider, "get_name")
|
||||
else type(self._provider).__name__
|
||||
) if self._provider is not None else "legacy"
|
||||
|
||||
if self._provider is not None:
|
||||
translated = self._translate_with_provider(
|
||||
texts, target_language, source_language
|
||||
)
|
||||
def _do_translate(miss_texts):
|
||||
return self._translate_with_provider(
|
||||
miss_texts, target_language, source_language
|
||||
)
|
||||
else:
|
||||
translated = self._translate_with_legacy(texts, target_language, source_language)
|
||||
def _do_translate(miss_texts):
|
||||
return self._translate_with_legacy(
|
||||
miss_texts, target_language, source_language
|
||||
)
|
||||
|
||||
# Translation memory: reuse this user's previous translations
|
||||
# (identical context/prompt) before hitting the provider.
|
||||
translated = translate_with_tm(
|
||||
texts, target_language, source_language,
|
||||
provider_name, self._tm_scope, _do_translate,
|
||||
)
|
||||
|
||||
changed = sum(1 for orig, trans in zip(texts, translated) if orig != trans and trans.strip())
|
||||
self._translation_stats["changed"] += changed
|
||||
@@ -541,12 +641,19 @@ class WordTranslator:
|
||||
|
||||
Handles: paragraphs, tables, SDT (TOC/index), text boxes, shapes,
|
||||
AlternateContent blocks, and any nested drawing elements.
|
||||
|
||||
A single ``seen_run_elements`` set is shared by every collector so
|
||||
runs living in text boxes are never collected twice (the paragraph
|
||||
walk descends into w:txbxContent too).
|
||||
"""
|
||||
count_before = len(text_elements)
|
||||
seen_run_elements: set = set()
|
||||
|
||||
# Pass 1: walk direct body children
|
||||
for element in document.element.body:
|
||||
self._collect_from_element(element, document, text_elements)
|
||||
self._collect_from_element(
|
||||
element, document, text_elements, seen_run_elements
|
||||
)
|
||||
|
||||
pass1_count = len(text_elements) - count_before
|
||||
|
||||
@@ -554,15 +661,18 @@ class WordTranslator:
|
||||
# Text boxes / rectangles / shapes store their text here, nested deep
|
||||
# inside <w:drawing> → <a:graphic> → <wps:wsp> → <wps:txbx> or
|
||||
# inside <w:pict> → <v:shape> → <v:textbox>.
|
||||
self._collect_from_textboxes(document.element.body, document, text_elements)
|
||||
self._collect_from_textboxes(
|
||||
document.element.body, document, text_elements, seen_run_elements
|
||||
)
|
||||
|
||||
pass2_count = len(text_elements) - count_before - pass1_count
|
||||
|
||||
# Pass 3: footnotes and endnotes (live in separate parts)
|
||||
# Pass 3: footnotes, endnotes and comments (live in separate parts)
|
||||
if post_save_callbacks is None:
|
||||
post_save_callbacks = []
|
||||
self._collect_from_footnotes(document, text_elements, post_save_callbacks)
|
||||
self._collect_from_endnotes(document, text_elements, post_save_callbacks)
|
||||
self._collect_from_comments(document, text_elements, post_save_callbacks)
|
||||
|
||||
total = len(text_elements) - count_before
|
||||
_log_info(
|
||||
@@ -573,34 +683,38 @@ class WordTranslator:
|
||||
)
|
||||
|
||||
def _collect_from_element(
|
||||
self, element, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||
self, element, document: Document,
|
||||
text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
seen_run_elements: Optional[set] = None,
|
||||
) -> None:
|
||||
"""Recursively collect from any element type."""
|
||||
if isinstance(element, CT_P):
|
||||
paragraph = Paragraph(element, document)
|
||||
self._collect_from_paragraph(paragraph, text_elements)
|
||||
self._collect_from_paragraph(paragraph, text_elements, seen_run_elements)
|
||||
elif isinstance(element, CT_Tbl):
|
||||
table = Table(element, document)
|
||||
self._collect_from_table(table, text_elements)
|
||||
self._collect_from_table(table, text_elements, seen_run_elements)
|
||||
elif element.tag == qn("w:sdt"):
|
||||
self._collect_from_sdt(element, document, text_elements)
|
||||
self._collect_from_sdt(element, document, text_elements, seen_run_elements)
|
||||
elif element.tag == self._TAG_ALT_CONTENT:
|
||||
# <mc:AlternateContent> wraps drawing/shape content
|
||||
for part in element:
|
||||
self._collect_from_element(part, document, text_elements)
|
||||
self._collect_from_element(part, document, text_elements, seen_run_elements)
|
||||
else:
|
||||
# For any other container element, recurse into children
|
||||
# to catch paragraphs nested in unexpected wrappers
|
||||
for child in element:
|
||||
if isinstance(child, CT_P):
|
||||
paragraph = Paragraph(child, document)
|
||||
self._collect_from_paragraph(paragraph, text_elements)
|
||||
self._collect_from_paragraph(paragraph, text_elements, seen_run_elements)
|
||||
elif isinstance(child, CT_Tbl):
|
||||
table = Table(child, document)
|
||||
self._collect_from_table(table, text_elements)
|
||||
self._collect_from_table(table, text_elements, seen_run_elements)
|
||||
|
||||
def _collect_from_textboxes(
|
||||
self, root, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||
self, root, document: Document,
|
||||
text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
seen_run_elements: Optional[set] = None,
|
||||
) -> None:
|
||||
"""Find and collect text from ALL <w:txbxContent> elements in the XML tree.
|
||||
|
||||
@@ -612,20 +726,23 @@ class WordTranslator:
|
||||
- Shapes nested in <mc:AlternateContent> blocks
|
||||
|
||||
The <w:txbxContent> element contains regular <w:p> paragraphs
|
||||
with <w:r> runs, just like normal body text.
|
||||
with <w:r> runs, just like normal body text. Runs already collected
|
||||
during the body walk are skipped via ``seen_run_elements``.
|
||||
"""
|
||||
# Find all w:txbxContent elements anywhere in the tree
|
||||
for txbx in root.iter(qn("w:txbxContent")):
|
||||
for child in txbx:
|
||||
if isinstance(child, CT_P):
|
||||
paragraph = Paragraph(child, document)
|
||||
self._collect_from_paragraph(paragraph, text_elements)
|
||||
self._collect_from_paragraph(paragraph, text_elements, seen_run_elements)
|
||||
elif isinstance(child, CT_Tbl):
|
||||
table = Table(child, document)
|
||||
self._collect_from_table(table, text_elements)
|
||||
self._collect_from_table(table, text_elements, seen_run_elements)
|
||||
|
||||
def _collect_from_sdt(
|
||||
self, sdt_element, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||
self, sdt_element, document: Document,
|
||||
text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
seen_run_elements: Optional[set] = None,
|
||||
) -> None:
|
||||
"""Collect text from Structured Document Tags (TOC, index, content controls).
|
||||
|
||||
@@ -645,10 +762,10 @@ class WordTranslator:
|
||||
for child in sdt_content:
|
||||
if isinstance(child, CT_P):
|
||||
paragraph = Paragraph(child, document)
|
||||
self._collect_from_paragraph(paragraph, text_elements)
|
||||
self._collect_from_paragraph(paragraph, text_elements, seen_run_elements)
|
||||
elif isinstance(child, CT_Tbl):
|
||||
table = Table(child, document)
|
||||
self._collect_from_table(table, text_elements)
|
||||
self._collect_from_table(table, text_elements, seen_run_elements)
|
||||
|
||||
def _collect_from_footnotes(
|
||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
@@ -783,6 +900,59 @@ class WordTranslator:
|
||||
|
||||
post_save_callbacks.append(write_endnotes_back)
|
||||
|
||||
def _collect_from_comments(
|
||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
post_save_callbacks: List[Callable[[Path], None]] = None,
|
||||
) -> None:
|
||||
"""Collect text from comments/balloons (word/comments.xml part).
|
||||
|
||||
Same mechanism as footnotes: the comments part is separate from the
|
||||
main document tree, so translations are written back after save.
|
||||
"""
|
||||
comments_xml = self._find_part_by_content_type(
|
||||
document,
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.comments+xml",
|
||||
)
|
||||
if comments_xml is None:
|
||||
return
|
||||
|
||||
collected = 0
|
||||
for t_elem in comments_xml.iter(qn("w:t")):
|
||||
original = t_elem.text or ""
|
||||
if not original.strip():
|
||||
continue
|
||||
|
||||
def make_t_setter(t):
|
||||
def setter(text: str) -> None:
|
||||
t.text = text
|
||||
return setter
|
||||
|
||||
text_elements.append((original, make_t_setter(t_elem)))
|
||||
collected += 1
|
||||
|
||||
if collected and post_save_callbacks is not None:
|
||||
def write_comments_back(output_path: Path) -> None:
|
||||
try:
|
||||
new_blob = etree.tostring(
|
||||
comments_xml,
|
||||
xml_declaration=True,
|
||||
encoding="UTF-8",
|
||||
standalone=True,
|
||||
)
|
||||
tmp_path = output_path.with_suffix(".tmp_com")
|
||||
with zipfile.ZipFile(output_path, "r") as zin, \
|
||||
zipfile.ZipFile(tmp_path, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||
for item in zin.namelist():
|
||||
if item == "word/comments.xml":
|
||||
zout.writestr(item, new_blob)
|
||||
else:
|
||||
zout.writestr(item, zin.read(item))
|
||||
tmp_path.replace(output_path)
|
||||
except Exception as e:
|
||||
_log_error("word_comments_writeback_error", error=str(e))
|
||||
|
||||
post_save_callbacks.append(write_comments_back)
|
||||
|
||||
def _collect_from_charts(
|
||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||
) -> None:
|
||||
@@ -1144,23 +1314,39 @@ class WordTranslator:
|
||||
except Exception as e:
|
||||
_log_error("word_diagram_zip_rewrite_error", error=str(e))
|
||||
|
||||
@staticmethod
|
||||
def _rpr_signature(run_element) -> str:
|
||||
"""Formatting signature of a run: serialized rPr XML (or "")."""
|
||||
rpr = run_element.find(qn("w:rPr"))
|
||||
if rpr is None:
|
||||
return ""
|
||||
import lxml.etree as _et
|
||||
|
||||
return _et.tostring(rpr, encoding="unicode")
|
||||
|
||||
def _collect_from_paragraph(
|
||||
self,
|
||||
paragraph: Paragraph,
|
||||
text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
seen_run_elements: Optional[set] = None,
|
||||
) -> None:
|
||||
"""Collect text from paragraph runs, preserving inter-run whitespace.
|
||||
|
||||
Each run is sent for translation WITHOUT its surrounding whitespace.
|
||||
The whitespace is captured and reapplied after translation so that words
|
||||
at formatting boundaries (e.g. bold/normal) do not get concatenated.
|
||||
Adjacent runs sharing the SAME parent element and the SAME run
|
||||
formatting (rPr) are merged into ONE translation unit: the sentence
|
||||
is translated whole — not fragment by fragment — and the result is
|
||||
written into the first run while the sibling runs are blanked.
|
||||
This is what keeps mid-sentence bold spans ("This is *very*
|
||||
important") coherent in the target language, like DeepL's inline
|
||||
tag handling.
|
||||
|
||||
Note: python-docx's `paragraph.runs` only returns DIRECT child <w:r>
|
||||
elements, not those inside <w:hyperlink> (used for TOC entries,
|
||||
cross-references, bookmark links). We therefore iterate the full
|
||||
XML tree to find every <w:r> and use a set of element ids to
|
||||
deduplicate — this avoids translating the same run twice while
|
||||
ensuring hyperlink text IS picked up.
|
||||
XML tree to find every <w:r> and deduplicate by element identity —
|
||||
`seen_run_elements` is shared across paragraphs so runs living in
|
||||
text boxes (also collected by _collect_from_textboxes) are not
|
||||
translated twice.
|
||||
"""
|
||||
# Check full paragraph text including nested content (hyperlinks, etc.)
|
||||
full_text = ''.join(
|
||||
@@ -1169,33 +1355,83 @@ class WordTranslator:
|
||||
if not full_text:
|
||||
return
|
||||
|
||||
# Collect every <w:r> element in the paragraph tree, including
|
||||
# those nested in <w:hyperlink>, <w:smartTag>, etc. The dedup by
|
||||
# element id is defensive — `paragraph.runs` and the manual iter
|
||||
# below could overlap if python-docx starts surfacing nested runs.
|
||||
seen_run_ids: set = set()
|
||||
if seen_run_elements is None:
|
||||
seen_run_elements = set()
|
||||
|
||||
# 1) Direct runs (paragraph.runs is the python-docx-native API).
|
||||
for run in paragraph.runs:
|
||||
run_id = id(run._r)
|
||||
if run_id in seen_run_ids:
|
||||
continue
|
||||
seen_run_ids.add(run_id)
|
||||
if run.text and run.text.strip():
|
||||
self._append_run_translation(run, text_elements)
|
||||
|
||||
# 2) Runs nested inside <w:hyperlink> (TOC, cross-references).
|
||||
# python-docx's `paragraph.runs` does NOT descend into hyperlinks in
|
||||
# version 1.x — we have to walk the XML ourselves.
|
||||
# Every <w:r> in the paragraph tree, in document order, deduplicated
|
||||
# by element identity (paragraph.runs and the manual iter overlap).
|
||||
ordered_runs = []
|
||||
for r_elem in paragraph._p.iter(qn('w:r')):
|
||||
run_id = id(r_elem)
|
||||
if run_id in seen_run_ids:
|
||||
if id(r_elem) in seen_run_elements:
|
||||
continue
|
||||
seen_run_ids.add(run_id)
|
||||
# Build a Run wrapper so the setter API is consistent.
|
||||
run = Run(r_elem, paragraph)
|
||||
if run.text and run.text.strip():
|
||||
seen_run_elements.add(id(r_elem))
|
||||
ordered_runs.append(r_elem)
|
||||
|
||||
# Merge adjacent runs: same parent + same formatting signature.
|
||||
# Merging never crosses a parent boundary, so runs belonging to
|
||||
# different hyperlinks stay separate units.
|
||||
group: list = [] # list of r_elems
|
||||
group_signature: Optional[str] = None
|
||||
|
||||
def _flush_group():
|
||||
combined = "".join(
|
||||
(t.text or "")
|
||||
for r in group
|
||||
for t in r.findall(qn("w:t"))
|
||||
)
|
||||
if not combined.strip():
|
||||
return
|
||||
non_empty = [r for r in group if r.findall(qn("w:t"))]
|
||||
if len(non_empty) == 1:
|
||||
run = Run(non_empty[0], paragraph)
|
||||
self._append_run_translation(run, text_elements)
|
||||
return
|
||||
leading = combined[: len(combined) - len(combined.lstrip())]
|
||||
trailing = combined[len(combined.rstrip()):]
|
||||
stripped = combined.strip()
|
||||
if not stripped:
|
||||
return
|
||||
|
||||
first = non_empty[0]
|
||||
|
||||
def make_group_setter(first_r, siblings, lead: str, trail: str):
|
||||
def setter(text: str) -> None:
|
||||
from docx.text.run import Run as _Run
|
||||
|
||||
run = _Run(first_r, paragraph)
|
||||
# Reapply the group's boundary whitespace so words are
|
||||
# never concatenated with the next differently-formatted
|
||||
# run ("This is quite" + "very" → "quite very").
|
||||
run.text = lead + text.strip() + trail
|
||||
# Blank the merged siblings: the whole sentence now
|
||||
# lives in the first run (formatting is identical).
|
||||
for sib in siblings:
|
||||
for t_elem in sib.findall(qn("w:t")):
|
||||
t_elem.text = ""
|
||||
|
||||
return setter
|
||||
|
||||
siblings = non_empty[1:]
|
||||
text_elements.append(
|
||||
(stripped, make_group_setter(first, siblings, leading, trailing))
|
||||
)
|
||||
|
||||
for r_elem in ordered_runs:
|
||||
# Whitespace-only runs join the group: dropping them would
|
||||
# concatenate words ("Hello" + " " + "World" → "HelloWorld").
|
||||
# They carry no w:t text, so a group of only whitespace runs is
|
||||
# skipped at flush time by the strip() check.
|
||||
signature = self._rpr_signature(r_elem)
|
||||
same_parent = (
|
||||
group and group[-1].getparent() is r_elem.getparent()
|
||||
)
|
||||
if group and same_parent and signature == group_signature:
|
||||
group.append(r_elem)
|
||||
else:
|
||||
_flush_group()
|
||||
group = [r_elem]
|
||||
group_signature = signature
|
||||
_flush_group()
|
||||
|
||||
def _append_run_translation(
|
||||
self,
|
||||
@@ -1220,15 +1456,16 @@ class WordTranslator:
|
||||
text_elements.append((stripped, make_setter(run, leading, trailing)))
|
||||
|
||||
def _collect_from_table(
|
||||
self, table: Table, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||
self, table: Table, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||
seen_run_elements: Optional[set] = None,
|
||||
) -> None:
|
||||
"""Collect text from table cells."""
|
||||
for row in table.rows:
|
||||
for cell in row.cells:
|
||||
for paragraph in cell.paragraphs:
|
||||
self._collect_from_paragraph(paragraph, text_elements)
|
||||
self._collect_from_paragraph(paragraph, text_elements, seen_run_elements)
|
||||
for nested_table in cell.tables:
|
||||
self._collect_from_table(nested_table, text_elements)
|
||||
self._collect_from_table(nested_table, text_elements, seen_run_elements)
|
||||
|
||||
def _collect_from_section(
|
||||
self, section: Section, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||
|
||||
@@ -179,3 +179,41 @@ class FileHandler:
|
||||
|
||||
# Global file handler instance
|
||||
file_handler = FileHandler()
|
||||
|
||||
|
||||
def validate_zip_safety(
|
||||
file_path: Path,
|
||||
max_compression_ratio: float = 100.0,
|
||||
max_total_uncompressed_mb: int = 1024,
|
||||
) -> None:
|
||||
"""Reject ZIP-based documents that expand dangerously (zip bombs).
|
||||
|
||||
Office files (.xlsx/.docx/.pptx) are ZIP archives: a small uploaded file
|
||||
can decompress to gigabytes in memory. Raises ValueError when the file
|
||||
is not a readable archive, expands beyond the allowed ratio, or when the
|
||||
total uncompressed size exceeds the cap.
|
||||
"""
|
||||
import zipfile
|
||||
|
||||
max_total_bytes = max_total_uncompressed_mb * 1024 * 1024
|
||||
try:
|
||||
with zipfile.ZipFile(file_path) as zf:
|
||||
total_uncompressed = 0
|
||||
for info in zf.infolist():
|
||||
if info.is_dir():
|
||||
continue
|
||||
total_uncompressed += info.file_size
|
||||
if (
|
||||
info.compress_size > 0
|
||||
and info.file_size / info.compress_size > max_compression_ratio
|
||||
):
|
||||
raise ValueError(
|
||||
f"Entry '{info.filename}' expands more than "
|
||||
f"{int(max_compression_ratio)}x its compressed size."
|
||||
)
|
||||
if total_uncompressed > max_total_bytes:
|
||||
raise ValueError(
|
||||
f"Archive expands beyond {max_total_uncompressed_mb} MB."
|
||||
)
|
||||
except zipfile.BadZipFile as e:
|
||||
raise ValueError(f"Not a valid ZIP-based document: {e}")
|
||||
|
||||
Reference in New Issue
Block a user