Compare commits
32 Commits
96bac2e792
...
production
| Author | SHA1 | Date | |
|---|---|---|---|
| 8f96ddfe71 | |||
| 9b15b7c9fa | |||
| 2da2c4765c | |||
| 4aebb49c7b | |||
| 4255a1a0c5 | |||
| 3ae28dd3cb | |||
| e706cef5d6 | |||
| 12cd0c6893 | |||
| b706cbf802 | |||
| dd1e005c70 | |||
| 04a9328860 | |||
| d40d7f3e86 | |||
| c794eff823 | |||
| 8d0fc818ef | |||
| 13d2f83081 | |||
| 4d466699fd | |||
| 5ae1587428 | |||
| f403b2851d | |||
| ebb2537fda | |||
| 36aeac2c5e | |||
| de9407f974 | |||
| 5e3fb0098b | |||
| a57b8a8e4d | |||
| dde80f6bc3 | |||
| 7398cae359 | |||
| 030950c962 | |||
| 5d6afd2dad | |||
| 07c50151cb | |||
| 299173fff7 | |||
| 9b354befe3 | |||
| 4a8f33d36f | |||
| 489df66c0f |
142
.agents/skills/find-skills/SKILL.md
Normal file
142
.agents/skills/find-skills/SKILL.md
Normal file
@@ -0,0 +1,142 @@
|
|||||||
|
---
|
||||||
|
name: find-skills
|
||||||
|
description: Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Find Skills
|
||||||
|
|
||||||
|
This skill helps you discover and install skills from the open agent skills ecosystem.
|
||||||
|
|
||||||
|
## When to Use This Skill
|
||||||
|
|
||||||
|
Use this skill when the user:
|
||||||
|
|
||||||
|
- Asks "how do I do X" where X might be a common task with an existing skill
|
||||||
|
- Says "find a skill for X" or "is there a skill for X"
|
||||||
|
- Asks "can you do X" where X is a specialized capability
|
||||||
|
- Expresses interest in extending agent capabilities
|
||||||
|
- Wants to search for tools, templates, or workflows
|
||||||
|
- Mentions they wish they had help with a specific domain (design, testing, deployment, etc.)
|
||||||
|
|
||||||
|
## What is the Skills CLI?
|
||||||
|
|
||||||
|
The Skills CLI (`npx skills`) is the package manager for the open agent skills ecosystem. Skills are modular packages that extend agent capabilities with specialized knowledge, workflows, and tools.
|
||||||
|
|
||||||
|
**Key commands:**
|
||||||
|
|
||||||
|
- `npx skills find [query] [--owner <owner>]` - Search for skills interactively or by keyword, optionally scoped to a GitHub owner
|
||||||
|
- `npx skills add <package>` - Install a skill from GitHub or other sources
|
||||||
|
- `npx skills check` - Check for skill updates
|
||||||
|
- `npx skills update` - Update all installed skills
|
||||||
|
|
||||||
|
**Browse skills at:** https://skills.sh/
|
||||||
|
|
||||||
|
## How to Help Users Find Skills
|
||||||
|
|
||||||
|
### Step 1: Understand What They Need
|
||||||
|
|
||||||
|
When a user asks for help with something, identify:
|
||||||
|
|
||||||
|
1. The domain (e.g., React, testing, design, deployment)
|
||||||
|
2. The specific task (e.g., writing tests, creating animations, reviewing PRs)
|
||||||
|
3. Whether this is a common enough task that a skill likely exists
|
||||||
|
|
||||||
|
### Step 2: Check the Leaderboard First
|
||||||
|
|
||||||
|
Before running a CLI search, check the [skills.sh leaderboard](https://skills.sh/) to see if a well-known skill already exists for the domain. The leaderboard ranks skills by total installs, surfacing the most popular and battle-tested options.
|
||||||
|
|
||||||
|
For example, top skills for web development include:
|
||||||
|
- `vercel-labs/agent-skills` — React, Next.js, web design (100K+ installs each)
|
||||||
|
- `anthropics/skills` — Frontend design, document processing (100K+ installs)
|
||||||
|
|
||||||
|
### Step 3: Search for Skills
|
||||||
|
|
||||||
|
If the leaderboard doesn't cover the user's need, run the find command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx skills find [query] [--owner <owner>]
|
||||||
|
```
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
- User asks "how do I make my React app faster?" → `npx skills find react performance`
|
||||||
|
- User asks "can you help me with PR reviews?" → `npx skills find pr review`
|
||||||
|
- User asks "I need to create a changelog" → `npx skills find changelog`
|
||||||
|
|
||||||
|
### Step 4: Verify Quality Before Recommending
|
||||||
|
|
||||||
|
**Do not recommend a skill based solely on search results.** Always verify:
|
||||||
|
|
||||||
|
1. **Install count** — Prefer skills with 1K+ installs. Be cautious with anything under 100.
|
||||||
|
2. **Source reputation** — Official sources (`vercel-labs`, `anthropics`, `microsoft`) are more trustworthy than unknown authors.
|
||||||
|
3. **GitHub stars** — Check the source repository. A skill from a repo with <100 stars should be treated with skepticism.
|
||||||
|
|
||||||
|
### Step 5: Present Options to the User
|
||||||
|
|
||||||
|
When you find relevant skills, present them to the user with:
|
||||||
|
|
||||||
|
1. The skill name and what it does
|
||||||
|
2. The install count and source
|
||||||
|
3. The install command they can run
|
||||||
|
4. A link to learn more at skills.sh
|
||||||
|
|
||||||
|
Example response:
|
||||||
|
|
||||||
|
```
|
||||||
|
I found a skill that might help! The "react-best-practices" skill provides
|
||||||
|
React and Next.js performance optimization guidelines from Vercel Engineering.
|
||||||
|
(185K installs)
|
||||||
|
|
||||||
|
To install it:
|
||||||
|
npx skills add vercel-labs/agent-skills@react-best-practices
|
||||||
|
|
||||||
|
Learn more: https://skills.sh/vercel-labs/agent-skills/react-best-practices
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 6: Offer to Install
|
||||||
|
|
||||||
|
If the user wants to proceed, you can install the skill for them:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx skills add <owner/repo@skill> -g -y
|
||||||
|
```
|
||||||
|
|
||||||
|
The `-g` flag installs globally (user-level) and `-y` skips confirmation prompts.
|
||||||
|
|
||||||
|
## Common Skill Categories
|
||||||
|
|
||||||
|
When searching, consider these common categories:
|
||||||
|
|
||||||
|
| Category | Example Queries |
|
||||||
|
| --------------- | ---------------------------------------- |
|
||||||
|
| Web Development | react, nextjs, typescript, css, tailwind |
|
||||||
|
| Testing | testing, jest, playwright, e2e |
|
||||||
|
| DevOps | deploy, docker, kubernetes, ci-cd |
|
||||||
|
| Documentation | docs, readme, changelog, api-docs |
|
||||||
|
| Code Quality | review, lint, refactor, best-practices |
|
||||||
|
| Design | ui, ux, design-system, accessibility |
|
||||||
|
| Productivity | workflow, automation, git |
|
||||||
|
|
||||||
|
## Tips for Effective Searches
|
||||||
|
|
||||||
|
1. **Use specific keywords**: "react testing" is better than just "testing"
|
||||||
|
2. **Try alternative terms**: If "deploy" doesn't work, try "deployment" or "ci-cd"
|
||||||
|
3. **Check popular sources**: Many skills come from `vercel-labs/agent-skills` or `ComposioHQ/awesome-claude-skills`
|
||||||
|
|
||||||
|
## When No Skills Are Found
|
||||||
|
|
||||||
|
If no relevant skills exist:
|
||||||
|
|
||||||
|
1. Acknowledge that no existing skill was found
|
||||||
|
2. Offer to help with the task directly using your general capabilities
|
||||||
|
3. Suggest the user could create their own skill with `npx skills init`
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```
|
||||||
|
I searched for skills related to "xyz" but didn't find any matches.
|
||||||
|
I can still help you with this task directly! Would you like me to proceed?
|
||||||
|
|
||||||
|
If this is something you do often, you could create your own skill:
|
||||||
|
npx skills init my-xyz-skill
|
||||||
|
```
|
||||||
39
.agents/skills/writing-guidelines/SKILL.md
Normal file
39
.agents/skills/writing-guidelines/SKILL.md
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
---
|
||||||
|
name: writing-guidelines
|
||||||
|
description: Review docs/prose for Writing Guidelines compliance. Use when asked to "review my docs", "check writing style", "audit prose", "review docs voice and tone", or "check this page against the writing handbook".
|
||||||
|
metadata:
|
||||||
|
author: vercel
|
||||||
|
version: "1.0.0"
|
||||||
|
argument-hint: <file-or-pattern>
|
||||||
|
---
|
||||||
|
|
||||||
|
# Writing Guidelines
|
||||||
|
|
||||||
|
Review files for compliance with Writing Guidelines.
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
1. Fetch the latest guidelines from the source URL below
|
||||||
|
2. Read the specified files (or prompt user for files/pattern)
|
||||||
|
3. Check against all rules in the fetched guidelines
|
||||||
|
4. Output findings in the terse `file:line` format
|
||||||
|
|
||||||
|
## Guidelines Source
|
||||||
|
|
||||||
|
Fetch fresh guidelines before each review:
|
||||||
|
|
||||||
|
```
|
||||||
|
https://raw.githubusercontent.com/vercel-labs/writing-guidelines/main/command.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Use WebFetch to retrieve the latest rules. The fetched content contains all the rules and output format instructions.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
When a user provides a file or pattern argument:
|
||||||
|
1. Fetch guidelines from the source URL above
|
||||||
|
2. Read the specified files
|
||||||
|
3. Apply all rules from the fetched guidelines
|
||||||
|
4. Output findings using the format specified in the guidelines
|
||||||
|
|
||||||
|
If no files specified, ask the user which files to review.
|
||||||
32
.env.example
32
.env.example
@@ -96,6 +96,38 @@ TRANSLATIONS_PER_MINUTE=10
|
|||||||
TRANSLATIONS_PER_HOUR=50
|
TRANSLATIONS_PER_HOUR=50
|
||||||
MAX_CONCURRENT_TRANSLATIONS=5
|
MAX_CONCURRENT_TRANSLATIONS=5
|
||||||
|
|
||||||
|
# ============== Quality Layer (L0) ==============
|
||||||
|
# Track A1 of the dev plan — observability only, no behavior change.
|
||||||
|
# When enabled, the L0 layer logs the script detection result for each
|
||||||
|
# translation job but does NOT modify the file or the job status.
|
||||||
|
# Default: false (opt-in). Set to "true" to enable.
|
||||||
|
QUALITY_L0_ENABLED=false
|
||||||
|
QUALITY_L0_SAMPLE_SIZE=20
|
||||||
|
|
||||||
|
# ============== Quality Layer (L1) ==============
|
||||||
|
# Track A3 of the dev plan — API-based LLM judge.
|
||||||
|
# Sends 5 sampled chunks per job to a cheap LLM (deepseek-chat by default)
|
||||||
|
# and logs the verdict (pass/fail) but does NOT modify the file or job
|
||||||
|
# status. Log-only by default for the first 2 weeks of observation.
|
||||||
|
# After that, set QUALITY_L1_LOG_ONLY=false to enable auto-retry.
|
||||||
|
#
|
||||||
|
# Cost: ~$0.0003 per job with deepseek-chat. ~$0.001 with gpt-4o-mini.
|
||||||
|
QUALITY_L1_ENABLED=false
|
||||||
|
QUALITY_L1_LOG_ONLY=true
|
||||||
|
QUALITY_L1_SAMPLE_SIZE=5
|
||||||
|
QUALITY_L1_MIN_CHUNKS=10
|
||||||
|
QUALITY_L1_TIMEOUT_SEC=8.0
|
||||||
|
|
||||||
|
# L1 judge configuration (any OpenAI-compatible endpoint).
|
||||||
|
# DeepSeek is the default (cheapest, ~$0.14/M input tokens).
|
||||||
|
L1_JUDGE_API_KEY=
|
||||||
|
L1_JUDGE_BASE_URL=https://api.deepseek.com/v1
|
||||||
|
L1_JUDGE_MODEL=deepseek-chat
|
||||||
|
# L1_JUDGE_MODEL=gpt-4o-mini
|
||||||
|
# L1_JUDGE_BASE_URL=https://api.openai.com/v1
|
||||||
|
# L1_JUDGE_MODEL=google/gemini-2.5-flash-lite
|
||||||
|
# L1_JUDGE_BASE_URL=https://openrouter.ai/api/v1
|
||||||
|
|
||||||
# ============== Cleanup Service ==============
|
# ============== Cleanup Service ==============
|
||||||
# Enable automatic file cleanup
|
# Enable automatic file cleanup
|
||||||
CLEANUP_ENABLED=true
|
CLEANUP_ENABLED=true
|
||||||
|
|||||||
206
_bmad-output/innovation-strategy-2026-06-13.md
Normal file
206
_bmad-output/innovation-strategy-2026-06-13.md
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
# Innovation Strategy: office_translator (Wordly.art)
|
||||||
|
|
||||||
|
**Date:** 2026-06-13
|
||||||
|
**Strategist:** Sepehr
|
||||||
|
**Strategic Focus:** Identifier comment transformer la préservation parfaite du format Office (aujourd'hui un avantage technique) en un positionnement de marché défendable et un modèle d'affaires scalable, face à des acteurs comme DeepL (qui copie déjà ce créneau) et Google (qui peut le copier à tout moment).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Strategic Context
|
||||||
|
|
||||||
|
### Current Situation
|
||||||
|
|
||||||
|
**Produit technique mature, pré-lancement commercial.**
|
||||||
|
|
||||||
|
- Stack production-ready : FastAPI + Next.js 15 + PostgreSQL + Redis + Docker, déployé en self-hosted
|
||||||
|
- Différenciateur technique central et vérifiable : **préservation de la mise en forme complexe Excel/Word/PowerPoint** (formules, fusions, styles, images, headers/footers, animations) — c'est le *seul* vrai fossé concurrentiel selon ton positionnement
|
||||||
|
- Architecture multi-providers déjà en place (Google, DeepL, OpenAI, Ollama, DeepSeek, OpenRouter, LibreTranslate, WebLLM en navigateur) — flexibilité réelle mais risque de dilution du message
|
||||||
|
- Capacités secondaires existantes : self-hosting, WebLLM zero-data, vision pour images dans Excel, glossaires techniques
|
||||||
|
- Infrastructure monétisation prête (Stripe) mais **pas encore de clients payants actifs** documentés
|
||||||
|
- Tarification définie : Free / Starter $9 / Pro $29 / Business $79
|
||||||
|
- Marketing plan rédigé mais assets visuels (captures, vidéo démo) non produits
|
||||||
|
- Faiblesses techniques identifiées (credentials en dur, stockage JSON, CORS permissif, pas de Celery)
|
||||||
|
|
||||||
|
### Strategic Challenge
|
||||||
|
|
||||||
|
**Le défi stratégique central : la préservation du format est un avantage technique reproductible.**
|
||||||
|
|
||||||
|
DeepL a déjà empiété sur ce territoire et Google peut le cloner en quelques trimestres. Ton avantage actuel est *technique* (comment tu traduis), pas *positionnel* (où tu te places dans l'esprit du client). Trois sous-défis critiques :
|
||||||
|
|
||||||
|
1. **Quel segment précis est prêt à payer significativement plus pour la préservation du format ?** (Pas "tout le monde qui traduit des documents" — c'est trop large et DeepL/Google les servent déjà.)
|
||||||
|
2. **Quel modèle d'affaires verrouille cet avantage** au point que même un clone gratuit de Google ne puisse pas te déloger ? (Self-hosting enterprise ? Vertical spécialisé ? Données/apprentissage ? Communauté ?)
|
||||||
|
3. **Quel "job to be done" non-traduction ton outil accomplit-il déjà** et que tu n'exploites pas encore commercialement ? (Compliance ? Archivage multilingue ? Workflow documentaire intégré ?)
|
||||||
|
|
||||||
|
Sans réponse tranchée à ces trois questions, tu risques d'être un excellent produit technique avec une proposition de valeur diluée.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 MARKET ANALYSIS
|
||||||
|
|
||||||
|
### Market Landscape
|
||||||
|
|
||||||
|
{{market_landscape}}
|
||||||
|
|
||||||
|
### Competitive Dynamics
|
||||||
|
|
||||||
|
{{competitive_dynamics}}
|
||||||
|
|
||||||
|
### Market Opportunities
|
||||||
|
|
||||||
|
{{market_opportunities}}
|
||||||
|
|
||||||
|
### Critical Insights
|
||||||
|
|
||||||
|
{{market_insights}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 💼 BUSINESS MODEL ANALYSIS
|
||||||
|
|
||||||
|
### Current Business Model
|
||||||
|
|
||||||
|
{{current_business_model}}
|
||||||
|
|
||||||
|
### Value Proposition Assessment
|
||||||
|
|
||||||
|
{{value_proposition}}
|
||||||
|
|
||||||
|
### Revenue and Cost Structure
|
||||||
|
|
||||||
|
{{revenue_cost_structure}}
|
||||||
|
|
||||||
|
### Business Model Weaknesses
|
||||||
|
|
||||||
|
{{model_weaknesses}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚡ DISRUPTION OPPORTUNITIES
|
||||||
|
|
||||||
|
### Disruption Vectors
|
||||||
|
|
||||||
|
{{disruption_vectors}}
|
||||||
|
|
||||||
|
### Unmet Customer Jobs
|
||||||
|
|
||||||
|
{{unmet_jobs}}
|
||||||
|
|
||||||
|
### Technology Enablers
|
||||||
|
|
||||||
|
{{technology_enablers}}
|
||||||
|
|
||||||
|
### Strategic White Space
|
||||||
|
|
||||||
|
{{strategic_whitespace}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 INNOVATION OPPORTUNITIES
|
||||||
|
|
||||||
|
### Innovation Initiatives
|
||||||
|
|
||||||
|
{{innovation_initiatives}}
|
||||||
|
|
||||||
|
### Business Model Innovation
|
||||||
|
|
||||||
|
{{business_model_innovation}}
|
||||||
|
|
||||||
|
### Value Chain Opportunities
|
||||||
|
|
||||||
|
{{value_chain_opportunities}}
|
||||||
|
|
||||||
|
### Partnership and Ecosystem Plays
|
||||||
|
|
||||||
|
{{partnership_opportunities}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎲 STRATEGIC OPTIONS
|
||||||
|
|
||||||
|
### Option A: {{option_a_name}}
|
||||||
|
|
||||||
|
{{option_a_description}}
|
||||||
|
|
||||||
|
**Pros:** {{option_a_pros}}
|
||||||
|
|
||||||
|
**Cons:** {{option_a_cons}}
|
||||||
|
|
||||||
|
### Option B: {{option_b_name}}
|
||||||
|
|
||||||
|
{{option_b_description}}
|
||||||
|
|
||||||
|
**Pros:** {{option_b_pros}}
|
||||||
|
|
||||||
|
**Cons:** {{option_b_cons}}
|
||||||
|
|
||||||
|
### Option C: {{option_c_name}}
|
||||||
|
|
||||||
|
{{option_c_description}}
|
||||||
|
|
||||||
|
**Pros:** {{option_c_pros}}
|
||||||
|
|
||||||
|
**Cons:** {{option_c_cons}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏆 RECOMMENDED STRATEGY
|
||||||
|
|
||||||
|
### Strategic Direction
|
||||||
|
|
||||||
|
{{recommended_strategy}}
|
||||||
|
|
||||||
|
### Key Hypotheses to Validate
|
||||||
|
|
||||||
|
{{key_hypotheses}}
|
||||||
|
|
||||||
|
### Critical Success Factors
|
||||||
|
|
||||||
|
{{success_factors}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📋 EXECUTION ROADMAP
|
||||||
|
|
||||||
|
### Phase 1: Immediate Impact
|
||||||
|
|
||||||
|
{{phase_1}}
|
||||||
|
|
||||||
|
### Phase 2: Foundation Building
|
||||||
|
|
||||||
|
{{phase_2}}
|
||||||
|
|
||||||
|
### Phase 3: Scale & Optimization
|
||||||
|
|
||||||
|
{{phase_3}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📈 SUCCESS METRICS
|
||||||
|
|
||||||
|
### Leading Indicators
|
||||||
|
|
||||||
|
{{leading_indicators}}
|
||||||
|
|
||||||
|
### Lagging Indicators
|
||||||
|
|
||||||
|
{{lagging_indicators}}
|
||||||
|
|
||||||
|
### Decision Gates
|
||||||
|
|
||||||
|
{{decision_gates}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ RISKS AND MITIGATION
|
||||||
|
|
||||||
|
### Key Risks
|
||||||
|
|
||||||
|
{{key_risks}}
|
||||||
|
|
||||||
|
### Mitigation Strategies
|
||||||
|
|
||||||
|
{{risk_mitigation}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Generated using BMAD Creative Intelligence Suite - Innovation Strategy Workflow_
|
||||||
50
config.py
50
config.py
@@ -68,6 +68,56 @@ class Config:
|
|||||||
)
|
)
|
||||||
MAX_MEMORY_PERCENT = float(os.getenv("MAX_MEMORY_PERCENT", "80"))
|
MAX_MEMORY_PERCENT = float(os.getenv("MAX_MEMORY_PERCENT", "80"))
|
||||||
|
|
||||||
|
# ============== Quality Layer (L0) ==============
|
||||||
|
# Track A1 of the dev plan — observability only, no behavior change.
|
||||||
|
# Set to "true" to enable. Default: false (opt-in).
|
||||||
|
QUALITY_L0_ENABLED = os.getenv("QUALITY_L0_ENABLED", "false").lower() == "true"
|
||||||
|
# Number of text samples to extract from the output file for L0 analysis.
|
||||||
|
# Keep small to avoid overhead. 20 is enough to catch language confusion.
|
||||||
|
QUALITY_L0_SAMPLE_SIZE = int(os.getenv("QUALITY_L0_SAMPLE_SIZE", "20"))
|
||||||
|
|
||||||
|
|
||||||
|
# ============== Quality Layer (L1) ==============
|
||||||
|
# Track A3 of the dev plan — API-based LLM judge.
|
||||||
|
# Observability first; the verdict is logged but never used to retry
|
||||||
|
# or block a job (QUALITY_L1_LOG_ONLY=true). After 2 weeks of
|
||||||
|
# monitoring, set QUALITY_L1_LOG_ONLY=false to enable auto-retry.
|
||||||
|
QUALITY_L1_ENABLED = os.getenv("QUALITY_L1_ENABLED", "false").lower() == "true"
|
||||||
|
QUALITY_L1_LOG_ONLY = os.getenv("QUALITY_L1_LOG_ONLY", "true").lower() == "true"
|
||||||
|
# Number of chunks sampled per job. 5 is the sweet spot (cost vs coverage).
|
||||||
|
QUALITY_L1_SAMPLE_SIZE = int(os.getenv("QUALITY_L1_SAMPLE_SIZE", "5"))
|
||||||
|
# Skip the check if the document has fewer than this many chunks.
|
||||||
|
QUALITY_L1_MIN_CHUNKS = int(os.getenv("QUALITY_L1_MIN_CHUNKS", "10"))
|
||||||
|
# Hard ceiling on the L1 call (seconds). Anything longer is a skip.
|
||||||
|
QUALITY_L1_TIMEOUT_SEC = float(os.getenv("QUALITY_L1_TIMEOUT_SEC", "8.0"))
|
||||||
|
|
||||||
|
# ============== Quality Layer (L2 — Pro tier) ==============
|
||||||
|
# Track A4 of the dev plan — STRONGER LLM judge (8 dimensions).
|
||||||
|
# Gated to Pro+ plans in the route. Default off everywhere.
|
||||||
|
# Cost: ~$0.005–$0.02/job (gpt-4o) or ~$0.001/job (gpt-4o-mini).
|
||||||
|
# Set QUALITY_L2_TIER_GATE=false to allow L2 for free tier too.
|
||||||
|
QUALITY_L2_ENABLED = os.getenv("QUALITY_L2_ENABLED", "false").lower() == "true"
|
||||||
|
QUALITY_L2_LOG_ONLY = os.getenv("QUALITY_L2_LOG_ONLY", "true").lower() == "true"
|
||||||
|
QUALITY_L2_SAMPLE_SIZE = int(os.getenv("QUALITY_L2_SAMPLE_SIZE", "15"))
|
||||||
|
QUALITY_L2_MIN_CHUNKS = int(os.getenv("QUALITY_L2_MIN_CHUNKS", "20"))
|
||||||
|
QUALITY_L2_TIMEOUT_SEC = float(os.getenv("QUALITY_L2_TIMEOUT_SEC", "20.0"))
|
||||||
|
# When true, only Pro+ plans can use L2. Otherwise, all plans can.
|
||||||
|
QUALITY_L2_TIER_GATE = os.getenv("QUALITY_L2_TIER_GATE", "true").lower() == "true"
|
||||||
|
|
||||||
|
# ============== PDF Smart-Fit (Track B3.5) ==============
|
||||||
|
# When true, the PDF translator uses a smart overflow strategy:
|
||||||
|
# 1. Try original bbox at original size
|
||||||
|
# 2. Expand bbox vertically (3x original height)
|
||||||
|
# 3. Shrink font ONCE (0.93x) with expanded bbox
|
||||||
|
# 4. Shrink font AGAIN (0.87x cumulative) with expanded bbox
|
||||||
|
# 5. For headings (font >= 14pt): never below 90% of original
|
||||||
|
# 6. For body: never below 75% of original
|
||||||
|
# 7. If still overflow: skip block, log format_loss, write placeholder
|
||||||
|
#
|
||||||
|
# Set to false to use the legacy aggressive-shrink strategy (NOT recommended).
|
||||||
|
PDF_SMART_FIT_ENABLED = os.getenv("PDF_SMART_FIT_ENABLED", "true").lower() == "true"
|
||||||
|
|
||||||
|
|
||||||
# ============== API Configuration ==============
|
# ============== API Configuration ==============
|
||||||
API_TITLE = "Document Translation API"
|
API_TITLE = "Document Translation API"
|
||||||
API_VERSION = "1.0.0"
|
API_VERSION = "1.0.0"
|
||||||
|
|||||||
@@ -35,6 +35,8 @@ interface PricingData {
|
|||||||
}
|
}
|
||||||
interface StripeStatus {
|
interface StripeStatus {
|
||||||
configured: boolean;
|
configured: boolean;
|
||||||
|
mode: "live" | "test" | "unknown";
|
||||||
|
is_test_mode_in_production: boolean;
|
||||||
has_secret_key: boolean;
|
has_secret_key: boolean;
|
||||||
has_publishable_key: boolean;
|
has_publishable_key: boolean;
|
||||||
has_webhook_secret: boolean;
|
has_webhook_secret: boolean;
|
||||||
@@ -103,6 +105,9 @@ export default function AdminPricingPage() {
|
|||||||
const [saving, setSaving] = useState(false);
|
const [saving, setSaving] = useState(false);
|
||||||
const [setupLoading, setSetupLoading] = useState(false);
|
const [setupLoading, setSetupLoading] = useState(false);
|
||||||
const [setupResult, setSetupResult] = useState<any>(null);
|
const [setupResult, setSetupResult] = useState<any>(null);
|
||||||
|
const [webhookLoading, setWebhookLoading] = useState(false);
|
||||||
|
const [webhookResult, setWebhookResult] = useState<any>(null);
|
||||||
|
const [envMode, setEnvMode] = useState<string>("development");
|
||||||
const [toast, setToast] = useState<{ type: "ok" | "err"; text: string } | null>(null);
|
const [toast, setToast] = useState<{ type: "ok" | "err"; text: string } | null>(null);
|
||||||
|
|
||||||
const getToken = () => useTranslationStore.getState().settings.adminToken ?? "";
|
const getToken = () => useTranslationStore.getState().settings.adminToken ?? "";
|
||||||
@@ -130,6 +135,7 @@ export default function AdminPricingPage() {
|
|||||||
setPricing(d);
|
setPricing(d);
|
||||||
}
|
}
|
||||||
if (j.stripe) setStripeStatus(j.stripe);
|
if (j.stripe) setStripeStatus(j.stripe);
|
||||||
|
if (j.meta?.env) setEnvMode(j.meta.env);
|
||||||
} else if (res.status === 401) {
|
} else if (res.status === 401) {
|
||||||
showToast("err", "Session admin expirée. Reconnectez-vous.");
|
showToast("err", "Session admin expirée. Reconnectez-vous.");
|
||||||
} else {
|
} else {
|
||||||
@@ -263,6 +269,33 @@ export default function AdminPricingPage() {
|
|||||||
showToast("err", j.errors?.[0]?.error || j.message || "Échec de la configuration Stripe.");
|
showToast("err", j.errors?.[0]?.error || j.message || "Échec de la configuration Stripe.");
|
||||||
}
|
}
|
||||||
loadPricing();
|
loadPricing();
|
||||||
|
} else if (res.status === 409 && j.error === "TEST_MODE_IN_PRODUCTION") {
|
||||||
|
// Give the user a way to override
|
||||||
|
const ok = window.confirm(
|
||||||
|
`${j.message}\n\nForcer la création en mode test ? (utile pour staging, à éviter en prod)`,
|
||||||
|
);
|
||||||
|
if (!ok) {
|
||||||
|
setSetupResult(j);
|
||||||
|
setSetupLoading(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const retry = await fetch(`${API_BASE}/api/v1/admin/pricing/setup-stripe`, {
|
||||||
|
method: "POST",
|
||||||
|
cache: "no-store",
|
||||||
|
headers: {
|
||||||
|
Authorization: `Bearer ${token}`,
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ force: true }),
|
||||||
|
});
|
||||||
|
const rj = await retry.json();
|
||||||
|
setSetupResult(rj);
|
||||||
|
if (retry.ok) {
|
||||||
|
showToast("ok", "✅ Produits créés (forcé).");
|
||||||
|
loadPricing();
|
||||||
|
} else {
|
||||||
|
showToast("err", rj.message || "Échec forcé.");
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
showToast("err", j.message || "Erreur lors de la configuration Stripe.");
|
showToast("err", j.message || "Erreur lors de la configuration Stripe.");
|
||||||
setSetupResult(j);
|
setSetupResult(j);
|
||||||
@@ -274,6 +307,69 @@ export default function AdminPricingPage() {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const autoSetupWebhook = async () => {
|
||||||
|
const token = useTranslationStore.getState().settings.adminToken;
|
||||||
|
if (!token) {
|
||||||
|
showToast("err", "Session admin absente. Reconnectez-vous.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setWebhookLoading(true);
|
||||||
|
setWebhookResult(null);
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${API_BASE}/api/v1/admin/pricing/setup-webhook`, {
|
||||||
|
method: "POST",
|
||||||
|
cache: "no-store",
|
||||||
|
headers: {
|
||||||
|
Authorization: `Bearer ${token}`,
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const j = await res.json();
|
||||||
|
if (res.ok) {
|
||||||
|
setWebhookResult(j);
|
||||||
|
if (j.data?.secret_saved) {
|
||||||
|
showToast("ok", "✅ Webhook Stripe créé/mis à jour et signing secret enregistré.");
|
||||||
|
} else {
|
||||||
|
showToast("ok", "✅ Webhook Stripe mis à jour. (Secret inchangé — il existait déjà.)");
|
||||||
|
}
|
||||||
|
loadPricing();
|
||||||
|
} else if (res.status === 409 && j.error === "TEST_MODE_IN_PRODUCTION") {
|
||||||
|
const ok = window.confirm(
|
||||||
|
`${j.message}\n\nForcer quand même ?`,
|
||||||
|
);
|
||||||
|
if (!ok) {
|
||||||
|
setWebhookResult(j);
|
||||||
|
setWebhookLoading(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const retry = await fetch(`${API_BASE}/api/v1/admin/pricing/setup-webhook`, {
|
||||||
|
method: "POST",
|
||||||
|
cache: "no-store",
|
||||||
|
headers: {
|
||||||
|
Authorization: `Bearer ${token}`,
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ force: true }),
|
||||||
|
});
|
||||||
|
const rj = await retry.json();
|
||||||
|
setWebhookResult(rj);
|
||||||
|
if (retry.ok) {
|
||||||
|
showToast("ok", "✅ Webhook créé (forcé).");
|
||||||
|
loadPricing();
|
||||||
|
} else {
|
||||||
|
showToast("err", rj.message || "Échec forcé.");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
showToast("err", j.message || "Erreur lors de la création du webhook.");
|
||||||
|
setWebhookResult(j);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
showToast("err", "Erreur réseau.");
|
||||||
|
} finally {
|
||||||
|
setWebhookLoading(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
const updatePlan = (planId: keyof PricingData, field: keyof PlanConfig, value: string | number) => {
|
const updatePlan = (planId: keyof PricingData, field: keyof PlanConfig, value: string | number) => {
|
||||||
setPricing((prev) => {
|
setPricing((prev) => {
|
||||||
const next = { ...prev[planId], [field]: value } as PlanConfig;
|
const next = { ...prev[planId], [field]: value } as PlanConfig;
|
||||||
@@ -328,9 +424,48 @@ export default function AdminPricingPage() {
|
|||||||
<CardTitle className="text-base flex items-center gap-2">
|
<CardTitle className="text-base flex items-center gap-2">
|
||||||
<Sparkles className="w-4 h-4 text-violet-400" />
|
<Sparkles className="w-4 h-4 text-violet-400" />
|
||||||
Statut Stripe
|
Statut Stripe
|
||||||
|
{stripeStatus?.mode === "live" && (
|
||||||
|
<Badge className="ml-2 bg-emerald-600/20 text-emerald-300 border-emerald-500/40 text-[10px]">
|
||||||
|
● LIVE
|
||||||
|
</Badge>
|
||||||
|
)}
|
||||||
|
{stripeStatus?.mode === "test" && (
|
||||||
|
<Badge className="ml-2 bg-amber-600/20 text-amber-300 border-amber-500/40 text-[10px]">
|
||||||
|
● TEST
|
||||||
|
</Badge>
|
||||||
|
)}
|
||||||
|
{stripeStatus?.mode === "unknown" && (
|
||||||
|
<Badge variant="outline" className="ml-2 text-muted-foreground text-[10px]">
|
||||||
|
non configuré
|
||||||
|
</Badge>
|
||||||
|
)}
|
||||||
|
<span className="ml-auto text-xs text-muted-foreground font-mono">
|
||||||
|
ENV={envMode}
|
||||||
|
</span>
|
||||||
</CardTitle>
|
</CardTitle>
|
||||||
</CardHeader>
|
</CardHeader>
|
||||||
<CardContent className="space-y-4">
|
<CardContent className="space-y-4">
|
||||||
|
{/* BLOCKING warning if test mode in production */}
|
||||||
|
{stripeStatus?.is_test_mode_in_production && (
|
||||||
|
<div className="flex items-start gap-3 p-4 rounded-xl border-2 border-red-500/50 bg-red-900/20 text-sm">
|
||||||
|
<AlertTriangle className="w-5 h-5 text-red-400 flex-shrink-0 mt-0.5" />
|
||||||
|
<div className="flex-1">
|
||||||
|
<p className="text-red-300 font-bold">
|
||||||
|
⚠️ CLÉ STRIPE TEST EN PRODUCTION
|
||||||
|
</p>
|
||||||
|
<p className="text-red-400/90 text-xs mt-1 leading-relaxed">
|
||||||
|
<code className="bg-red-950/50 px-1.5 py-0.5 rounded text-red-200">ENV=production</code>{" "}
|
||||||
|
mais{" "}
|
||||||
|
<code className="bg-red-950/50 px-1.5 py-0.5 rounded text-red-200">STRIPE_SECRET_KEY</code>{" "}
|
||||||
|
commence par <code className="bg-red-950/50 px-1.5 py-0.5 rounded text-red-200">sk_test_…</code>.
|
||||||
|
Les paiements en prod ne fonctionneront PAS — soit tu remplaces par{" "}
|
||||||
|
<code className="bg-red-950/50 px-1.5 py-0.5 rounded text-red-200">sk_live_…</code> dans
|
||||||
|
.env.ionos, soit tu remets <code className="bg-red-950/50 px-1.5 py-0.5 rounded text-red-200">ENV=staging</code>.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
<div className="grid grid-cols-3 gap-3">
|
<div className="grid grid-cols-3 gap-3">
|
||||||
{[
|
{[
|
||||||
{ label: "Clé secrète", ok: stripeStatus?.has_secret_key },
|
{ label: "Clé secrète", ok: stripeStatus?.has_secret_key },
|
||||||
@@ -350,10 +485,10 @@ export default function AdminPricingPage() {
|
|||||||
|
|
||||||
<Separator />
|
<Separator />
|
||||||
|
|
||||||
{/* Auto-setup button */}
|
{/* Auto-setup button (products + prices) */}
|
||||||
<div className="flex items-start gap-4">
|
<div className="flex items-start gap-4">
|
||||||
<div className="flex-1">
|
<div className="flex-1">
|
||||||
<p className="text-sm font-medium text-foreground">Configuration automatique Stripe</p>
|
<p className="text-sm font-medium text-foreground">1. Produits & prix</p>
|
||||||
<p className="text-xs text-muted-foreground mt-1">
|
<p className="text-xs text-muted-foreground mt-1">
|
||||||
Crée automatiquement les 3 produits (Starter, Pro, Business) et les 6 prix (mensuel + annuel) dans votre compte Stripe, puis sauvegarde les Price IDs dans le .env.
|
Crée automatiquement les 3 produits (Starter, Pro, Business) et les 6 prix (mensuel + annuel) dans votre compte Stripe, puis sauvegarde les Price IDs dans le .env.
|
||||||
</p>
|
</p>
|
||||||
@@ -387,6 +522,45 @@ export default function AdminPricingPage() {
|
|||||||
|
|
||||||
<Separator />
|
<Separator />
|
||||||
|
|
||||||
|
{/* Auto-setup webhook */}
|
||||||
|
<div className="flex items-start gap-4">
|
||||||
|
<div className="flex-1">
|
||||||
|
<p className="text-sm font-medium text-foreground">2. Webhook Stripe</p>
|
||||||
|
<p className="text-xs text-muted-foreground mt-1">
|
||||||
|
Crée ou met à jour l'endpoint webhook dans Stripe pointant sur ce backend, et enregistre automatiquement le <code className="text-foreground/80">whsec_…</code> dans le .env.
|
||||||
|
L'URL est déduite de la requête courante (header Host + X-Forwarded-Proto).
|
||||||
|
</p>
|
||||||
|
{webhookResult?.data && (
|
||||||
|
<div className="mt-2 p-3 rounded-lg bg-secondary/40 text-xs space-y-1">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<CheckCircle className="w-3 h-3 text-emerald-400" />
|
||||||
|
<span>ID: <code>{webhookResult.data.id}</code></span>
|
||||||
|
</div>
|
||||||
|
<div className="flex items-center gap-2 text-muted-foreground">
|
||||||
|
<ExternalLink className="w-3 h-3" />
|
||||||
|
<span className="break-all">{webhookResult.data.url}</span>
|
||||||
|
</div>
|
||||||
|
<div className="text-muted-foreground">
|
||||||
|
Events: <code>{(webhookResult.data.enabled_events || []).length}</code> abonné(s)
|
||||||
|
{webhookResult.data.created ? " — endpoint créé" : " — endpoint mis à jour"}
|
||||||
|
{webhookResult.data.secret_saved ? " — secret enregistré" : " — secret déjà connu"}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<Button
|
||||||
|
onClick={autoSetupWebhook}
|
||||||
|
disabled={webhookLoading || !stripeStatus?.has_secret_key}
|
||||||
|
variant="outline"
|
||||||
|
className="gap-2 flex-shrink-0 border-violet-500/40 text-violet-300 hover:bg-violet-500/10"
|
||||||
|
>
|
||||||
|
{webhookLoading ? <Loader2 className="w-4 h-4 animate-spin" /> : <RefreshCw className="w-4 h-4" />}
|
||||||
|
{webhookLoading ? "Configuration…" : "Setup webhook auto"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<Separator />
|
||||||
|
|
||||||
{/* Manual Stripe keys */}
|
{/* Manual Stripe keys */}
|
||||||
<div>
|
<div>
|
||||||
<p className="text-sm font-medium text-foreground mb-3">Clés Stripe (laisser vide pour garder les valeurs actuelles)</p>
|
<p className="text-sm font-medium text-foreground mb-3">Clés Stripe (laisser vide pour garder les valeurs actuelles)</p>
|
||||||
|
|||||||
@@ -68,8 +68,13 @@ export function DashboardSidebar() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Momento promo section */}
|
{/* Memento promo section */}
|
||||||
<div className="px-6 py-4">
|
<a
|
||||||
|
href={t('memento.url', { defaultValue: 'https://memento-note.com/' })}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="block px-6 py-4"
|
||||||
|
>
|
||||||
<div className="bg-brand-muted/50 dark:bg-[#1f1f1f]/50 rounded-3xl p-6 border border-black/5 dark:border-white/5 group cursor-pointer hover:bg-brand-dark dark:hover:bg-brand-accent transition-all duration-500">
|
<div className="bg-brand-muted/50 dark:bg-[#1f1f1f]/50 rounded-3xl p-6 border border-black/5 dark:border-white/5 group cursor-pointer hover:bg-brand-dark dark:hover:bg-brand-accent transition-all duration-500">
|
||||||
<div className="flex items-center gap-3 mb-4">
|
<div className="flex items-center gap-3 mb-4">
|
||||||
<div className="w-8 h-8 bg-brand-dark group-hover:bg-brand-accent rounded-lg flex items-center justify-center text-white text-xs font-black shadow-sm transition-all group-hover:rotate-12">M</div>
|
<div className="w-8 h-8 bg-brand-dark group-hover:bg-brand-accent rounded-lg flex items-center justify-center text-white text-xs font-black shadow-sm transition-all group-hover:rotate-12">M</div>
|
||||||
@@ -82,7 +87,7 @@ export function DashboardSidebar() {
|
|||||||
{t('memento.ctaFree')}
|
{t('memento.ctaFree')}
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</a>
|
||||||
|
|
||||||
{/* User section */}
|
{/* User section */}
|
||||||
<div className="border-t border-black/5 dark:border-white/5">
|
<div className="border-t border-black/5 dark:border-white/5">
|
||||||
|
|||||||
@@ -41,9 +41,11 @@ function getDisplaySource(
|
|||||||
glossarySourceLang: string
|
glossarySourceLang: string
|
||||||
): string {
|
): string {
|
||||||
if (!lang || lang === 'multi') return '';
|
if (!lang || lang === 'multi') return '';
|
||||||
if (lang === glossarySourceLang) return term.source;
|
const normalizedLang = lang.toLowerCase();
|
||||||
|
const normalizedSourceLang = glossarySourceLang.toLowerCase();
|
||||||
|
if (normalizedLang === normalizedSourceLang) return term.source;
|
||||||
const translations = term.translations || {};
|
const translations = term.translations || {};
|
||||||
return translations[lang] || '';
|
return translations[normalizedLang] || '';
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Target term in the chosen language.
|
/** Target term in the chosen language.
|
||||||
@@ -54,9 +56,11 @@ function getDisplayTarget(
|
|||||||
glossaryTargetLang: string
|
glossaryTargetLang: string
|
||||||
): string {
|
): string {
|
||||||
if (!lang) return '';
|
if (!lang) return '';
|
||||||
if (lang === 'multi' || lang === glossaryTargetLang) return term.target;
|
const normalizedLang = lang.toLowerCase();
|
||||||
|
const normalizedTargetLang = glossaryTargetLang.toLowerCase();
|
||||||
|
if (normalizedLang === 'multi' || normalizedLang === normalizedTargetLang) return term.target;
|
||||||
const translations = term.translations || {};
|
const translations = term.translations || {};
|
||||||
return translations[lang] || '';
|
return translations[normalizedLang] || '';
|
||||||
}
|
}
|
||||||
|
|
||||||
export default function GlossaryDetailPage() {
|
export default function GlossaryDetailPage() {
|
||||||
@@ -149,10 +153,12 @@ export default function GlossaryDetailPage() {
|
|||||||
if (i !== index) return t;
|
if (i !== index) return t;
|
||||||
const translations = { ...(t.translations || {}) } as Record<string, string>;
|
const translations = { ...(t.translations || {}) } as Record<string, string>;
|
||||||
|
|
||||||
const editLang = field === 'source' ? sourceLanguage : targetLanguage;
|
const editLang = (field === 'source' ? sourceLanguage : targetLanguage).toLowerCase();
|
||||||
|
const srcLang = glossary.source_language.toLowerCase();
|
||||||
|
const tgtLang = glossary.target_language.toLowerCase();
|
||||||
|
|
||||||
if (field === 'source') {
|
if (field === 'source') {
|
||||||
if (editLang === glossary.source_language) {
|
if (editLang === srcLang) {
|
||||||
return { ...t, source: value };
|
return { ...t, source: value };
|
||||||
} else {
|
} else {
|
||||||
if (editLang && editLang !== 'multi') {
|
if (editLang && editLang !== 'multi') {
|
||||||
@@ -161,7 +167,7 @@ export default function GlossaryDetailPage() {
|
|||||||
return { ...t, translations };
|
return { ...t, translations };
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
if (editLang === 'multi' || editLang === glossary.target_language) {
|
if (editLang === 'multi' || editLang === tgtLang) {
|
||||||
return { ...t, target: value };
|
return { ...t, target: value };
|
||||||
} else {
|
} else {
|
||||||
if (editLang && editLang !== 'multi') {
|
if (editLang && editLang !== 'multi') {
|
||||||
@@ -211,12 +217,24 @@ export default function GlossaryDetailPage() {
|
|||||||
};
|
};
|
||||||
|
|
||||||
const handleSourceLanguageChange = (newLang: string) => {
|
const handleSourceLanguageChange = (newLang: string) => {
|
||||||
|
// For multilingual glossaries, the selector is a VIEW filter only.
|
||||||
|
// migrateTerms must not be called — it would destroy translation data.
|
||||||
|
if (!glossary || glossary.target_language === 'multi') {
|
||||||
|
setSourceLanguage(newLang);
|
||||||
|
return;
|
||||||
|
}
|
||||||
const updated = migrateTerms(terms, sourceLanguage, newLang, targetLanguage, targetLanguage);
|
const updated = migrateTerms(terms, sourceLanguage, newLang, targetLanguage, targetLanguage);
|
||||||
setSourceLanguage(newLang);
|
setSourceLanguage(newLang);
|
||||||
setTerms(updated);
|
setTerms(updated);
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleTargetLanguageChange = (newLang: string) => {
|
const handleTargetLanguageChange = (newLang: string) => {
|
||||||
|
// For multilingual glossaries, the selector is a VIEW filter only.
|
||||||
|
// migrateTerms must not be called — it would destroy translation data.
|
||||||
|
if (!glossary || glossary.target_language === 'multi') {
|
||||||
|
setTargetLanguage(newLang);
|
||||||
|
return;
|
||||||
|
}
|
||||||
const updated = migrateTerms(terms, sourceLanguage, sourceLanguage, targetLanguage, newLang);
|
const updated = migrateTerms(terms, sourceLanguage, sourceLanguage, targetLanguage, newLang);
|
||||||
setTargetLanguage(newLang);
|
setTargetLanguage(newLang);
|
||||||
setTerms(updated);
|
setTerms(updated);
|
||||||
|
|||||||
749
frontend/src/app/dashboard/glossaries/new/page.tsx
Normal file
749
frontend/src/app/dashboard/glossaries/new/page.tsx
Normal file
@@ -0,0 +1,749 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { useState, useRef } from 'react';
|
||||||
|
import {
|
||||||
|
BookText, Upload, PenLine, ArrowRight, ArrowLeft, CheckCircle2,
|
||||||
|
AlertCircle, Loader2, Scale, Cpu, TrendingUp, HeartPulse,
|
||||||
|
Megaphone, Users, FlaskConical, ShoppingCart, X, Plus
|
||||||
|
} from 'lucide-react';
|
||||||
|
import Link from 'next/link';
|
||||||
|
import { useRouter } from 'next/navigation';
|
||||||
|
import { cn } from '@/lib/utils';
|
||||||
|
import { useI18n } from '@/lib/i18n';
|
||||||
|
import { useGlossaries, useGlossaryTemplates } from '../useGlossaries';
|
||||||
|
import { SUPPORTED_LANGUAGES } from '../types';
|
||||||
|
import type { GlossaryTermInput } from '../types';
|
||||||
|
import { parseFileToTerms } from '../csvUtils';
|
||||||
|
import { useToast } from '@/components/ui/toast';
|
||||||
|
|
||||||
|
// ── Icônes par catégorie de modèle ─────────────────────────────────────────
|
||||||
|
const TEMPLATE_ICONS: Record<string, React.ComponentType<{ size?: number; className?: string }>> = {
|
||||||
|
legal: Scale,
|
||||||
|
technology: Cpu,
|
||||||
|
finance: TrendingUp,
|
||||||
|
medical: HeartPulse,
|
||||||
|
marketing: Megaphone,
|
||||||
|
hr: Users,
|
||||||
|
scientific: FlaskConical,
|
||||||
|
ecommerce: ShoppingCart,
|
||||||
|
};
|
||||||
|
|
||||||
|
// ── Types ──────────────────────────────────────────────────────────────────
|
||||||
|
type CreationMethod = 'template' | 'file' | 'manual' | null;
|
||||||
|
type Step = 1 | 2;
|
||||||
|
|
||||||
|
export default function NewGlossaryPage() {
|
||||||
|
const { t } = useI18n();
|
||||||
|
const router = useRouter();
|
||||||
|
const { toast } = useToast();
|
||||||
|
const { createGlossary, importTemplate, isCreating, isImportingTemplate } = useGlossaries();
|
||||||
|
const { templates, isLoading: isLoadingTemplates } = useGlossaryTemplates();
|
||||||
|
|
||||||
|
const [step, setStep] = useState<Step>(1);
|
||||||
|
const [method, setMethod] = useState<CreationMethod>(null);
|
||||||
|
|
||||||
|
// État pour modèle
|
||||||
|
const [selectedTemplateId, setSelectedTemplateId] = useState<string | null>(null);
|
||||||
|
const [isImportingThis, setIsImportingThis] = useState(false);
|
||||||
|
|
||||||
|
// État pour fichier
|
||||||
|
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||||
|
const [fileStatus, setFileStatus] = useState<'idle' | 'parsing' | 'success' | 'error'>('idle');
|
||||||
|
const [fileError, setFileError] = useState('');
|
||||||
|
const [parsedTerms, setParsedTerms] = useState<GlossaryTermInput[]>([]);
|
||||||
|
const [parsedFileName, setParsedFileName] = useState('');
|
||||||
|
const [isDragging, setIsDragging] = useState(false);
|
||||||
|
const [fileWizardStep, setFileWizardStep] = useState<1 | 2 | 3>(1);
|
||||||
|
const [fileSrcLang, setFileSrcLang] = useState('fr');
|
||||||
|
const [fileTgtLang, setFileTgtLang] = useState('multi');
|
||||||
|
|
||||||
|
// État pour manuel
|
||||||
|
const [manualName, setManualName] = useState('');
|
||||||
|
const [manualSrc, setManualSrc] = useState('fr');
|
||||||
|
const [manualTgt, setManualTgt] = useState('multi');
|
||||||
|
|
||||||
|
const isProcessing = isCreating || isImportingTemplate || isImportingThis;
|
||||||
|
|
||||||
|
// ── Handlers ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const handleSelectMethod = (m: CreationMethod) => {
|
||||||
|
setMethod(m);
|
||||||
|
setStep(2);
|
||||||
|
// reset
|
||||||
|
setSelectedTemplateId(null);
|
||||||
|
setFileStatus('idle');
|
||||||
|
setParsedTerms([]);
|
||||||
|
setParsedFileName('');
|
||||||
|
setManualName('');
|
||||||
|
setFileWizardStep(1);
|
||||||
|
setFileSrcLang('fr');
|
||||||
|
setFileTgtLang('multi');
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleBack = () => {
|
||||||
|
setStep(1);
|
||||||
|
setMethod(null);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Importer un modèle
|
||||||
|
const handleImportTemplate = async () => {
|
||||||
|
if (!selectedTemplateId) return;
|
||||||
|
const tpl = templates.find(t => t.id === selectedTemplateId);
|
||||||
|
setIsImportingThis(true);
|
||||||
|
try {
|
||||||
|
await importTemplate(selectedTemplateId, tpl?.name);
|
||||||
|
toast({ title: 'Glossaire importé !', description: `Le modèle « ${tpl?.name} » a été ajouté à vos glossaires.` });
|
||||||
|
router.push('/dashboard/glossaries');
|
||||||
|
} catch {
|
||||||
|
toast({ variant: 'destructive', title: 'Erreur', description: 'Impossible d\'importer ce modèle.' });
|
||||||
|
} finally {
|
||||||
|
setIsImportingThis(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Parser un fichier CSV
|
||||||
|
const processFile = async (file: File) => {
|
||||||
|
const ext = file.name.split('.').pop()?.toLowerCase();
|
||||||
|
if (!ext || !['csv', 'xlsx', 'xls', 'ods', 'txt', 'tsv'].includes(ext)) {
|
||||||
|
setFileStatus('error');
|
||||||
|
setFileError('Format non supporté. Utilisez CSV, Excel (.xlsx), ODS, ou TSV.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (file.size > 5 * 1024 * 1024) {
|
||||||
|
setFileStatus('error');
|
||||||
|
setFileError('Fichier trop volumineux (maximum 5 MB).');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setFileStatus('parsing');
|
||||||
|
setFileError('');
|
||||||
|
try {
|
||||||
|
const terms = await parseFileToTerms(file);
|
||||||
|
if (terms.length === 0) {
|
||||||
|
setFileStatus('error');
|
||||||
|
setFileError('Le fichier est vide ou n\'a pas pu être lu.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setFileStatus('success');
|
||||||
|
setParsedTerms(terms);
|
||||||
|
setParsedFileName(file.name.replace(/\.[^.]+$/, '').replace(/[_-]/g, ' '));
|
||||||
|
} catch {
|
||||||
|
setFileStatus('error');
|
||||||
|
setFileError('Erreur lors de la lecture du fichier.');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Télécharger exemple CSV
|
||||||
|
const handleDownloadSample = () => {
|
||||||
|
const sample = `source,target\nbénéfice,profit\nflux de trésorerie,cash flow\nbilan,balance sheet\ncompte de résultat,income statement\nrésiliation,termination`;
|
||||||
|
const blob = new Blob([sample], { type: 'text/csv;charset=utf-8;' });
|
||||||
|
const url = URL.createObjectURL(blob);
|
||||||
|
const a = document.createElement('a');
|
||||||
|
a.href = url;
|
||||||
|
a.download = 'exemple_glossaire.csv';
|
||||||
|
document.body.appendChild(a);
|
||||||
|
a.click();
|
||||||
|
document.body.removeChild(a);
|
||||||
|
URL.revokeObjectURL(url);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Créer depuis fichier
|
||||||
|
const handleCreateFromFile = async () => {
|
||||||
|
if (!parsedTerms.length) return;
|
||||||
|
try {
|
||||||
|
await createGlossary({
|
||||||
|
name: parsedFileName || 'Mon glossaire',
|
||||||
|
source_language: fileSrcLang,
|
||||||
|
target_language: fileTgtLang,
|
||||||
|
terms: parsedTerms,
|
||||||
|
});
|
||||||
|
toast({ title: 'Glossaire créé !', description: `${parsedTerms.length} termes importés depuis votre fichier.` });
|
||||||
|
router.push('/dashboard/glossaries');
|
||||||
|
} catch {
|
||||||
|
toast({ variant: 'destructive', title: 'Erreur', description: 'Impossible de créer le glossaire.' });
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Créer manuellement
|
||||||
|
const handleCreateManual = async () => {
|
||||||
|
if (!manualName.trim()) return;
|
||||||
|
try {
|
||||||
|
await createGlossary({ name: manualName.trim(), source_language: manualSrc, target_language: manualTgt, terms: [] });
|
||||||
|
toast({ title: 'Glossaire créé !', description: `« ${manualName} » est prêt. Ajoutez vos premiers termes.` });
|
||||||
|
router.push('/dashboard/glossaries');
|
||||||
|
} catch {
|
||||||
|
toast({ variant: 'destructive', title: 'Erreur', description: 'Impossible de créer le glossaire.' });
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// ── Rendu ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="max-w-4xl mx-auto w-full p-6 lg:p-8">
|
||||||
|
|
||||||
|
{/* ── Fil d'Ariane ──────────────────────────────────────────── */}
|
||||||
|
<div className="flex items-center gap-2 mb-8 text-[11px] font-medium text-[#555555] dark:text-white/40">
|
||||||
|
<Link href="/dashboard/glossaries" className="hover:text-[#1A1A1A] dark:hover:text-white transition-colors">
|
||||||
|
Mes glossaires
|
||||||
|
</Link>
|
||||||
|
<ArrowRight size={11} />
|
||||||
|
<span className="text-[#1A1A1A] dark:text-white font-bold">Nouveau glossaire</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* ── Indicateur d'étapes ────────────────────────────────────── */}
|
||||||
|
<div className="flex items-center gap-4 mb-10">
|
||||||
|
{/* Étape 1 */}
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<div className={cn(
|
||||||
|
'w-7 h-7 rounded-full flex items-center justify-center text-[11px] font-black transition-all',
|
||||||
|
step >= 1 ? 'bg-[#1A1A1A] text-white' : 'bg-[#EBEBEB] text-[#888888]'
|
||||||
|
)}>
|
||||||
|
{step > 1 ? <CheckCircle2 size={14} /> : '1'}
|
||||||
|
</div>
|
||||||
|
<span className={cn('text-xs font-bold uppercase tracking-wider', step === 1 ? 'text-[#1A1A1A] dark:text-white' : 'text-[#888888]')}>
|
||||||
|
Choisir le type
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex-1 h-px bg-[#E5E3DF] dark:bg-white/10" />
|
||||||
|
|
||||||
|
{/* Étape 2 */}
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<div className={cn(
|
||||||
|
'w-7 h-7 rounded-full flex items-center justify-center text-[11px] font-black transition-all',
|
||||||
|
step === 2 ? 'bg-[#1A1A1A] text-white' : 'bg-[#EBEBEB] text-[#888888]'
|
||||||
|
)}>
|
||||||
|
2
|
||||||
|
</div>
|
||||||
|
<span className={cn('text-xs font-bold uppercase tracking-wider', step === 2 ? 'text-[#1A1A1A] dark:text-white' : 'text-[#888888]')}>
|
||||||
|
Configurer
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* ══════════════ ÉTAPE 1 : Choisir le type ══════════════════ */}
|
||||||
|
{step === 1 && (
|
||||||
|
<div className="animate-fade-in">
|
||||||
|
<div className="mb-8">
|
||||||
|
<h1 className="text-3xl font-serif font-medium text-[#1A1A1A] dark:text-white tracking-tight mb-2">
|
||||||
|
Comment voulez-vous créer votre{' '}
|
||||||
|
<span className="italic">glossaire ?</span>
|
||||||
|
</h1>
|
||||||
|
<p className="text-[#555555] dark:text-white/50 text-sm font-light">
|
||||||
|
Choisissez la méthode qui correspond à votre situation.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="grid grid-cols-1 md:grid-cols-3 gap-5">
|
||||||
|
|
||||||
|
{/* Option 1 — Modèle professionnel */}
|
||||||
|
<button
|
||||||
|
onClick={() => handleSelectMethod('template')}
|
||||||
|
className="group text-left p-7 bg-white dark:bg-[#141414] border-2 border-[#E5E3DF] dark:border-white/10 rounded-2xl hover:border-[#C5A17A] dark:hover:border-brand-accent/50 hover:shadow-lg transition-all cursor-pointer"
|
||||||
|
>
|
||||||
|
<div className="w-12 h-12 rounded-xl bg-[#F0EDE8] dark:bg-brand-accent/10 flex items-center justify-center text-[#8B6F47] mb-5 group-hover:scale-110 transition-transform">
|
||||||
|
<BookText size={24} />
|
||||||
|
</div>
|
||||||
|
<h2 className="text-base font-serif font-bold text-[#1A1A1A] dark:text-white mb-2 tracking-tight">
|
||||||
|
Depuis un modèle
|
||||||
|
</h2>
|
||||||
|
<p className="text-[12px] text-[#555555] dark:text-white/50 font-light leading-relaxed">
|
||||||
|
Choisissez parmi nos modèles professionnels pré-remplis : RH, Médical, Juridique, Tech…
|
||||||
|
</p>
|
||||||
|
<div className="mt-5 flex items-center gap-1.5 text-[11px] font-bold text-[#8B6F47] dark:text-brand-accent uppercase tracking-wider">
|
||||||
|
Choisir un modèle <ArrowRight size={12} />
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
{/* Option 2 — Fichier CSV */}
|
||||||
|
<button
|
||||||
|
onClick={() => handleSelectMethod('file')}
|
||||||
|
className="group text-left p-7 bg-white dark:bg-[#141414] border-2 border-[#E5E3DF] dark:border-white/10 rounded-2xl hover:border-[#C5A17A] dark:hover:border-brand-accent/50 hover:shadow-lg transition-all cursor-pointer"
|
||||||
|
>
|
||||||
|
<div className="w-12 h-12 rounded-xl bg-[#F0EDE8] dark:bg-brand-accent/10 flex items-center justify-center text-[#8B6F47] mb-5 group-hover:scale-110 transition-transform">
|
||||||
|
<Upload size={24} />
|
||||||
|
</div>
|
||||||
|
<h2 className="text-base font-serif font-bold text-[#1A1A1A] dark:text-white mb-2 tracking-tight">
|
||||||
|
Importer un fichier
|
||||||
|
</h2>
|
||||||
|
<p className="text-[12px] text-[#555555] dark:text-white/50 font-light leading-relaxed">
|
||||||
|
Importez vos propres termes depuis un fichier CSV, Excel (.xlsx) ou ODS existant.
|
||||||
|
</p>
|
||||||
|
<div className="mt-5 flex items-center gap-1.5 text-[11px] font-bold text-[#8B6F47] dark:text-brand-accent uppercase tracking-wider">
|
||||||
|
Importer un fichier <ArrowRight size={12} />
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
{/* Option 3 — Manuel */}
|
||||||
|
<button
|
||||||
|
onClick={() => handleSelectMethod('manual')}
|
||||||
|
className="group text-left p-7 bg-white dark:bg-[#141414] border-2 border-[#E5E3DF] dark:border-white/10 rounded-2xl hover:border-[#C5A17A] dark:hover:border-brand-accent/50 hover:shadow-lg transition-all cursor-pointer"
|
||||||
|
>
|
||||||
|
<div className="w-12 h-12 rounded-xl bg-[#F0EDE8] dark:bg-brand-accent/10 flex items-center justify-center text-[#8B6F47] mb-5 group-hover:scale-110 transition-transform">
|
||||||
|
<PenLine size={24} />
|
||||||
|
</div>
|
||||||
|
<h2 className="text-base font-serif font-bold text-[#1A1A1A] dark:text-white mb-2 tracking-tight">
|
||||||
|
Créer manuellement
|
||||||
|
</h2>
|
||||||
|
<p className="text-[12px] text-[#555555] dark:text-white/50 font-light leading-relaxed">
|
||||||
|
Partez de zéro et ajoutez vos termes un par un directement dans l'éditeur.
|
||||||
|
</p>
|
||||||
|
<div className="mt-5 flex items-center gap-1.5 text-[11px] font-bold text-[#8B6F47] dark:text-brand-accent uppercase tracking-wider">
|
||||||
|
Créer vide <ArrowRight size={12} />
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* ══════════════ ÉTAPE 2 : Configurer ═══════════════════════ */}
|
||||||
|
{step === 2 && (
|
||||||
|
<div className="animate-fade-in">
|
||||||
|
|
||||||
|
{/* Bouton retour */}
|
||||||
|
<button
|
||||||
|
onClick={handleBack}
|
||||||
|
className="flex items-center gap-2 text-[11px] font-bold uppercase tracking-wider text-[#555555] hover:text-[#1A1A1A] dark:hover:text-white transition-colors mb-8 cursor-pointer"
|
||||||
|
>
|
||||||
|
<ArrowLeft size={14} /> Retour
|
||||||
|
</button>
|
||||||
|
|
||||||
|
{/* ── CAS A : Depuis un modèle ─── */}
|
||||||
|
{method === 'template' && (
|
||||||
|
<div>
|
||||||
|
<h1 className="text-2xl font-serif font-medium text-[#1A1A1A] dark:text-white mb-2">
|
||||||
|
Choisissez un <span className="italic">modèle professionnel</span>
|
||||||
|
</h1>
|
||||||
|
<p className="text-[#555555] dark:text-white/50 text-sm font-light mb-8">
|
||||||
|
Sélectionnez le modèle le plus adapté à vos documents. Vous pourrez modifier les termes après l'import.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{isLoadingTemplates ? (
|
||||||
|
<div className="flex items-center justify-center py-20">
|
||||||
|
<Loader2 className="size-8 animate-spin text-[#C5A17A]" />
|
||||||
|
</div>
|
||||||
|
) : templates.length === 0 ? (
|
||||||
|
<div className="p-12 text-center bg-white dark:bg-[#141414] border border-[#E5E3DF] dark:border-white/5 rounded-2xl">
|
||||||
|
<p className="text-sm text-[#555555] dark:text-white/40">
|
||||||
|
Tous les modèles disponibles ont déjà été importés.
|
||||||
|
</p>
|
||||||
|
<button onClick={handleBack} className="mt-4 text-[11px] font-bold text-[#7A5C35] hover:underline cursor-pointer">
|
||||||
|
← Choisir une autre méthode
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4 mb-8">
|
||||||
|
{templates.map((tpl) => {
|
||||||
|
const Icon = TEMPLATE_ICONS[tpl.id] || BookText;
|
||||||
|
const isSelected = selectedTemplateId === tpl.id;
|
||||||
|
return (
|
||||||
|
<button
|
||||||
|
key={tpl.id}
|
||||||
|
onClick={() => setSelectedTemplateId(isSelected ? null : tpl.id)}
|
||||||
|
className={cn(
|
||||||
|
'text-left p-5 rounded-xl border-2 transition-all cursor-pointer',
|
||||||
|
isSelected
|
||||||
|
? 'border-[#1A1A1A] dark:border-white bg-[#F5F3EF] dark:bg-white/5 shadow-md'
|
||||||
|
: 'border-[#E5E3DF] dark:border-white/10 bg-white dark:bg-[#141414] hover:border-[#C5A17A] dark:hover:border-brand-accent/40'
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
<div className="flex items-start gap-4">
|
||||||
|
<div className={cn(
|
||||||
|
'w-10 h-10 rounded-xl flex items-center justify-center shrink-0',
|
||||||
|
isSelected ? 'bg-[#1A1A1A] text-white' : 'bg-[#F0EDE8] dark:bg-brand-accent/10 text-[#8B6F47]'
|
||||||
|
)}>
|
||||||
|
<Icon size={18} />
|
||||||
|
</div>
|
||||||
|
<div className="flex-1 min-w-0">
|
||||||
|
<div className="flex items-center justify-between gap-2">
|
||||||
|
<p className="text-sm font-serif font-bold text-[#1A1A1A] dark:text-white leading-tight">
|
||||||
|
{tpl.name.split(' - ')[0]}
|
||||||
|
</p>
|
||||||
|
<span className="shrink-0 text-[9px] font-mono font-semibold text-[#666666] dark:text-white/40">
|
||||||
|
{tpl.terms_count} termes
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/40 font-light mt-1 line-clamp-2">
|
||||||
|
{tpl.description}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
{isSelected && (
|
||||||
|
<CheckCircle2 size={18} className="text-[#1A1A1A] dark:text-white shrink-0 mt-0.5" />
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Bouton d'import */}
|
||||||
|
<div className="flex items-center justify-end gap-4">
|
||||||
|
{selectedTemplateId && (
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/40 font-light">
|
||||||
|
Modèle sélectionné : <strong className="text-[#1A1A1A] dark:text-white">{templates.find(t => t.id === selectedTemplateId)?.name.split(' - ')[0]}</strong>
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
<button
|
||||||
|
onClick={handleImportTemplate}
|
||||||
|
disabled={!selectedTemplateId || isProcessing}
|
||||||
|
className="flex items-center gap-2 px-7 py-3 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all disabled:opacity-40 disabled:cursor-not-allowed cursor-pointer"
|
||||||
|
>
|
||||||
|
{isImportingThis ? <><Loader2 size={14} className="animate-spin" /> Import en cours…</> : <><CheckCircle2 size={14} /> Importer ce modèle</>}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* ── CAS B : Depuis un fichier — Wizard 3 sous-étapes ─── */}
|
||||||
|
{method === 'file' && (
|
||||||
|
<div>
|
||||||
|
<h1 className="text-2xl font-serif font-medium text-[#1A1A1A] dark:text-white mb-2">
|
||||||
|
Importez votre <span className="italic">fichier de termes</span>
|
||||||
|
</h1>
|
||||||
|
|
||||||
|
{/* Indicateur sous-étapes */}
|
||||||
|
<div className="flex items-center gap-2 mb-8">
|
||||||
|
{([1, 2, 3] as const).map((s, i) => {
|
||||||
|
const labels = ['Format', 'Fichier', 'Configurer'];
|
||||||
|
const done = fileWizardStep > s;
|
||||||
|
const active = fileWizardStep === s;
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<div key={s} className="flex items-center gap-1.5">
|
||||||
|
<div className={cn(
|
||||||
|
'w-6 h-6 rounded-full flex items-center justify-center text-[10px] font-black transition-all',
|
||||||
|
done ? 'bg-emerald-600 text-white' : active ? 'bg-[#1A1A1A] text-white' : 'bg-[#EBEBEB] dark:bg-white/10 text-[#888]'
|
||||||
|
)}>
|
||||||
|
{done ? <CheckCircle2 size={12} /> : s}
|
||||||
|
</div>
|
||||||
|
<span className={cn('text-[10px] font-bold uppercase tracking-wider hidden sm:block',
|
||||||
|
active ? 'text-[#1A1A1A] dark:text-white' : 'text-[#AAAAAA]'
|
||||||
|
)}>{labels[i]}</span>
|
||||||
|
</div>
|
||||||
|
{i < 2 && <div className="flex-1 h-px bg-[#E5E3DF] dark:bg-white/10" />}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* ── Sous-étape 1 : Format ─── */}
|
||||||
|
{fileWizardStep === 1 && (
|
||||||
|
<div className="space-y-5">
|
||||||
|
<div className="bg-white dark:bg-[#141414] border border-[#E5E3DF] dark:border-white/5 rounded-2xl p-6 space-y-5">
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<p className="text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-3">Format attendu</p>
|
||||||
|
<p className="text-[12px] text-[#555555] dark:text-white/50 font-light mb-4 leading-relaxed">
|
||||||
|
Votre fichier doit avoir <strong className="text-[#1A1A1A] dark:text-white">2 colonnes</strong> : la première pour les termes source, la seconde pour les termes cibles.
|
||||||
|
La première ligne peut être un en-tête (elle sera ignorée automatiquement).
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{/* Tableau exemple */}
|
||||||
|
<div className="rounded-xl overflow-hidden border border-[#E5E3DF] dark:border-white/10 text-[11px] font-mono">
|
||||||
|
<div className="grid grid-cols-2 bg-[#F5F3EF] dark:bg-white/5 border-b border-[#E5E3DF] dark:border-white/10">
|
||||||
|
<div className="px-4 py-2 font-bold text-[#8B6F47] uppercase tracking-wider">source</div>
|
||||||
|
<div className="px-4 py-2 font-bold text-[#8B6F47] uppercase tracking-wider border-l border-[#E5E3DF] dark:border-white/10">target</div>
|
||||||
|
</div>
|
||||||
|
{[['bénéfice', 'profit'], ['flux de trésorerie', 'cash flow'], ['bilan', 'balance sheet'], ['résiliation', 'termination']].map(([s, t]) => (
|
||||||
|
<div key={s} className="grid grid-cols-2 border-b border-[#E5E3DF] dark:border-white/5 last:border-0 hover:bg-[#FAFAF8] dark:hover:bg-white/[0.02]">
|
||||||
|
<div className="px-4 py-2 text-[#1A1A1A] dark:text-white">{s}</div>
|
||||||
|
<div className="px-4 py-2 text-[#555555] dark:text-white/60 border-l border-[#E5E3DF] dark:border-white/10">{t}</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Formats acceptés */}
|
||||||
|
<div className="flex flex-wrap gap-2">
|
||||||
|
{['CSV (.csv)', 'Excel (.xlsx)', 'Excel (.xls)', 'ODS (.ods)', 'TSV (.tsv)', 'Texte (.txt)'].map(f => (
|
||||||
|
<span key={f} className="px-2.5 py-1 rounded-lg bg-[#F0EDE8] dark:bg-white/5 text-[10px] font-bold text-[#8B6F47] dark:text-brand-accent uppercase tracking-wider">{f}</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Règles */}
|
||||||
|
<div className="space-y-2">
|
||||||
|
{[
|
||||||
|
'Séparateur : virgule (CSV) ou tabulation (TSV)',
|
||||||
|
'Encodage recommandé : UTF-8',
|
||||||
|
'Taille maximale : 5 MB',
|
||||||
|
'Limite : 500 termes par glossaire',
|
||||||
|
'Les champs contenant des virgules doivent être entre guillemets',
|
||||||
|
].map(rule => (
|
||||||
|
<div key={rule} className="flex items-start gap-2 text-[11px] text-[#555555] dark:text-white/50">
|
||||||
|
<CheckCircle2 size={12} className="text-emerald-500 mt-0.5 shrink-0" />
|
||||||
|
{rule}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Télécharger exemple */}
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<button
|
||||||
|
onClick={handleDownloadSample}
|
||||||
|
className="flex items-center gap-2 text-[11px] font-bold text-[#8B6F47] dark:text-brand-accent hover:underline cursor-pointer"
|
||||||
|
>
|
||||||
|
<Upload size={12} /> Télécharger un exemple CSV
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={() => setFileWizardStep(2)}
|
||||||
|
className="flex items-center gap-2 px-6 py-2.5 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all cursor-pointer"
|
||||||
|
>
|
||||||
|
Suivant <ArrowRight size={13} />
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* ── Sous-étape 2 : Fichier ─── */}
|
||||||
|
{fileWizardStep === 2 && (
|
||||||
|
<div className="space-y-5">
|
||||||
|
{/* Zone de dépôt */}
|
||||||
|
<div
|
||||||
|
onDragOver={e => { e.preventDefault(); setIsDragging(true); }}
|
||||||
|
onDragLeave={() => setIsDragging(false)}
|
||||||
|
onDrop={e => {
|
||||||
|
e.preventDefault(); setIsDragging(false);
|
||||||
|
const f = e.dataTransfer.files[0];
|
||||||
|
if (f) processFile(f);
|
||||||
|
}}
|
||||||
|
onClick={() => fileInputRef.current?.click()}
|
||||||
|
className={cn(
|
||||||
|
'flex flex-col items-center justify-center gap-4 p-12 rounded-2xl border-2 border-dashed cursor-pointer transition-all min-h-[220px]',
|
||||||
|
isDragging ? 'border-[#C5A17A] bg-[#F5F0EA]' : 'border-[#D9D6D0] dark:border-white/10 hover:border-[#C5A17A] hover:bg-[#FAFAF8] dark:hover:bg-white/[0.02]',
|
||||||
|
fileStatus === 'error' && 'border-red-400 bg-red-50 dark:bg-red-500/5'
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
<input
|
||||||
|
ref={fileInputRef}
|
||||||
|
type="file"
|
||||||
|
accept=".csv,.xlsx,.xls,.ods,.txt,.tsv"
|
||||||
|
className="hidden"
|
||||||
|
onChange={e => { const f = e.target.files?.[0]; if (f) processFile(f); e.target.value = ''; }}
|
||||||
|
/>
|
||||||
|
{fileStatus === 'parsing' && (
|
||||||
|
<><Loader2 size={32} className="animate-spin text-[#C5A17A]" /><p className="text-sm font-medium text-[#555555] dark:text-white/50">Lecture du fichier…</p></>
|
||||||
|
)}
|
||||||
|
{fileStatus === 'success' && (
|
||||||
|
<>
|
||||||
|
<CheckCircle2 size={36} className="text-emerald-600" />
|
||||||
|
<div className="text-center">
|
||||||
|
<p className="text-sm font-bold text-emerald-700 dark:text-emerald-400">{parsedTerms.length} termes détectés</p>
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/40 mt-1">Cliquez pour changer de fichier</p>
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{fileStatus === 'error' && (
|
||||||
|
<>
|
||||||
|
<AlertCircle size={36} className="text-red-500" />
|
||||||
|
<div className="text-center">
|
||||||
|
<p className="text-sm font-medium text-red-600">{fileError}</p>
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/40 mt-1">Cliquez pour réessayer</p>
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{fileStatus === 'idle' && (
|
||||||
|
<>
|
||||||
|
<div className="w-16 h-16 rounded-2xl bg-[#F0EDE8] dark:bg-white/5 flex items-center justify-center text-[#8B6F47]">
|
||||||
|
<Upload size={32} />
|
||||||
|
</div>
|
||||||
|
<div className="text-center">
|
||||||
|
<p className="text-sm font-serif font-bold text-[#1A1A1A] dark:text-white">Glissez votre fichier ici</p>
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/40 mt-1">CSV, Excel, ODS, TSV — max 5 MB</p>
|
||||||
|
</div>
|
||||||
|
<span className="text-[10px] text-[#AAAAAA] font-light">ou cliquez pour parcourir</span>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<button onClick={() => setFileWizardStep(1)} className="flex items-center gap-2 text-[11px] font-bold uppercase tracking-wider text-[#555555] hover:text-[#1A1A1A] dark:hover:text-white transition-colors cursor-pointer">
|
||||||
|
<ArrowLeft size={13} /> Retour
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={() => setFileWizardStep(3)}
|
||||||
|
disabled={fileStatus !== 'success'}
|
||||||
|
className="flex items-center gap-2 px-6 py-2.5 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all disabled:opacity-40 disabled:cursor-not-allowed cursor-pointer"
|
||||||
|
>
|
||||||
|
Suivant <ArrowRight size={13} />
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* ── Sous-étape 3 : Configurer ─── */}
|
||||||
|
{fileWizardStep === 3 && (
|
||||||
|
<div className="space-y-5">
|
||||||
|
<div className="bg-white dark:bg-[#141414] border border-[#E5E3DF] dark:border-white/5 rounded-2xl p-6 space-y-6">
|
||||||
|
|
||||||
|
{/* Résumé fichier */}
|
||||||
|
<div className="flex items-center gap-3 p-3 rounded-xl bg-emerald-50 dark:bg-emerald-500/5 border border-emerald-200 dark:border-emerald-500/20">
|
||||||
|
<CheckCircle2 size={16} className="text-emerald-600 shrink-0" />
|
||||||
|
<div>
|
||||||
|
<p className="text-[11px] font-bold text-emerald-800 dark:text-emerald-400">{parsedTerms.length} termes prêts à importer</p>
|
||||||
|
<button onClick={() => { setFileWizardStep(2); setFileStatus('idle'); setParsedTerms([]); }} className="text-[10px] text-emerald-700 dark:text-emerald-500 hover:underline cursor-pointer">
|
||||||
|
Changer de fichier
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Nom */}
|
||||||
|
<div>
|
||||||
|
<label className="block text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-2">
|
||||||
|
Nom du glossaire <span className="text-red-500">*</span>
|
||||||
|
</label>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
value={parsedFileName}
|
||||||
|
onChange={e => setParsedFileName(e.target.value)}
|
||||||
|
placeholder="Ex : Termes RH internes, Glossaire juridique…"
|
||||||
|
className="w-full px-4 py-3 rounded-xl border border-[#D9D6D0] dark:border-white/10 bg-[#FAFAF8] dark:bg-[#1A1A1A] text-[#1A1A1A] dark:text-white text-sm placeholder:text-[#AAAAAA] dark:placeholder:text-white/25 focus:outline-none focus:ring-2 focus:ring-brand-accent/30 focus:border-brand-accent/50"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Langues */}
|
||||||
|
<div className="grid grid-cols-2 gap-5">
|
||||||
|
<div>
|
||||||
|
<label className="block text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-2">
|
||||||
|
Langue source
|
||||||
|
</label>
|
||||||
|
<select
|
||||||
|
value={fileSrcLang}
|
||||||
|
onChange={e => setFileSrcLang(e.target.value)}
|
||||||
|
className="w-full px-4 py-3 rounded-xl border border-[#D9D6D0] dark:border-white/10 bg-[#FAFAF8] dark:bg-[#1A1A1A] text-[#1A1A1A] dark:text-white text-sm focus:outline-none focus:ring-2 focus:ring-brand-accent/30"
|
||||||
|
>
|
||||||
|
{SUPPORTED_LANGUAGES.filter(l => l.code !== 'multi').map(l => (
|
||||||
|
<option key={l.code} value={l.code}>{l.flag} {l.label}</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
<p className="text-[10px] text-[#AAAAAA] dark:text-white/30 mt-1.5 font-light">Langue de la 1ère colonne de votre fichier</p>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<label className="block text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-2">
|
||||||
|
Langue cible
|
||||||
|
</label>
|
||||||
|
<select
|
||||||
|
value={fileTgtLang}
|
||||||
|
onChange={e => setFileTgtLang(e.target.value)}
|
||||||
|
className="w-full px-4 py-3 rounded-xl border border-[#D9D6D0] dark:border-white/10 bg-[#FAFAF8] dark:bg-[#1A1A1A] text-[#1A1A1A] dark:text-white text-sm focus:outline-none focus:ring-2 focus:ring-brand-accent/30"
|
||||||
|
>
|
||||||
|
{SUPPORTED_LANGUAGES.map(l => (
|
||||||
|
<option key={l.code} value={l.code}>{l.flag} {l.label}</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
<p className="text-[10px] text-[#AAAAAA] dark:text-white/30 mt-1.5 font-light">Langue de la 2ème colonne de votre fichier</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Info */}
|
||||||
|
<div className="p-3.5 rounded-xl bg-[#F5F3EF] dark:bg-white/[0.02] border border-[#D9D6D0] dark:border-white/5">
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/50 font-light leading-relaxed">
|
||||||
|
<strong className="font-bold text-[#1A1A1A] dark:text-white/80">Conseil :</strong>{' '}
|
||||||
|
Choisissez <strong>Multilingue</strong> comme langue cible si vous prévoyez d'utiliser ce glossaire pour traduire vers plusieurs langues. Vous pourrez ajouter des traductions dans l'éditeur.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<button onClick={() => setFileWizardStep(2)} className="flex items-center gap-2 text-[11px] font-bold uppercase tracking-wider text-[#555555] hover:text-[#1A1A1A] dark:hover:text-white transition-colors cursor-pointer">
|
||||||
|
<ArrowLeft size={13} /> Retour
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={handleCreateFromFile}
|
||||||
|
disabled={!parsedFileName.trim() || isProcessing}
|
||||||
|
className="flex items-center gap-2 px-7 py-3 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all disabled:opacity-40 disabled:cursor-not-allowed cursor-pointer"
|
||||||
|
>
|
||||||
|
{isCreating ? <><Loader2 size={14} className="animate-spin" /> Création…</> : <><CheckCircle2 size={14} /> Créer le glossaire</>}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* ── CAS C : Manuel ─── */}
|
||||||
|
{method === 'manual' && (
|
||||||
|
<div>
|
||||||
|
<h1 className="text-2xl font-serif font-medium text-[#1A1A1A] dark:text-white mb-2">
|
||||||
|
Créez votre glossaire <span className="italic">manuellement</span>
|
||||||
|
</h1>
|
||||||
|
<p className="text-[#555555] dark:text-white/50 text-sm font-light mb-8">
|
||||||
|
Donnez un nom à votre glossaire et choisissez les langues. Vous ajouterez les termes dans l'éditeur.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div className="bg-white dark:bg-[#141414] border border-[#E5E3DF] dark:border-white/5 rounded-2xl p-8 space-y-6">
|
||||||
|
|
||||||
|
{/* Nom */}
|
||||||
|
<div>
|
||||||
|
<label className="block text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-2">
|
||||||
|
Nom du glossaire <span className="text-red-500">*</span>
|
||||||
|
</label>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
value={manualName}
|
||||||
|
onChange={e => setManualName(e.target.value)}
|
||||||
|
placeholder="Ex : Termes juridiques internes, Glossaire RH…"
|
||||||
|
className="w-full px-4 py-3 rounded-xl border border-[#D9D6D0] dark:border-white/10 bg-[#FAFAF8] dark:bg-[#1A1A1A] text-[#1A1A1A] dark:text-white text-sm placeholder:text-[#AAAAAA] dark:placeholder:text-white/25 focus:outline-none focus:ring-2 focus:ring-brand-accent/30 focus:border-brand-accent/50"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Langues */}
|
||||||
|
<div className="grid grid-cols-2 gap-5">
|
||||||
|
<div>
|
||||||
|
<label className="block text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-2">
|
||||||
|
Langue source
|
||||||
|
</label>
|
||||||
|
<select
|
||||||
|
value={manualSrc}
|
||||||
|
onChange={e => setManualSrc(e.target.value)}
|
||||||
|
className="w-full px-4 py-3 rounded-xl border border-[#D9D6D0] dark:border-white/10 bg-[#FAFAF8] dark:bg-[#1A1A1A] text-[#1A1A1A] dark:text-white text-sm focus:outline-none focus:ring-2 focus:ring-brand-accent/30"
|
||||||
|
>
|
||||||
|
{SUPPORTED_LANGUAGES.filter(l => l.code !== 'multi').map(l => (
|
||||||
|
<option key={l.code} value={l.code}>{l.flag} {l.label}</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<label className="block text-xs font-bold text-[#333333] dark:text-white/70 uppercase tracking-wider mb-2">
|
||||||
|
Langue cible
|
||||||
|
</label>
|
||||||
|
<select
|
||||||
|
value={manualTgt}
|
||||||
|
onChange={e => setManualTgt(e.target.value)}
|
||||||
|
className="w-full px-4 py-3 rounded-xl border border-[#D9D6D0] dark:border-white/10 bg-[#FAFAF8] dark:bg-[#1A1A1A] text-[#1A1A1A] dark:text-white text-sm focus:outline-none focus:ring-2 focus:ring-brand-accent/30"
|
||||||
|
>
|
||||||
|
{SUPPORTED_LANGUAGES.map(l => (
|
||||||
|
<option key={l.code} value={l.code}>{l.flag} {l.label}</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Info */}
|
||||||
|
<div className="p-3.5 rounded-xl bg-[#F5F3EF] dark:bg-white/[0.02] border border-[#D9D6D0] dark:border-white/5">
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/50 font-light leading-relaxed">
|
||||||
|
<strong className="font-bold text-[#1A1A1A] dark:text-white/80">Que se passe-t-il ensuite ?</strong>{' '}
|
||||||
|
Un glossaire vide sera créé. Vous serez redirigé vers l'éditeur pour ajouter vos termes.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex justify-end mt-6">
|
||||||
|
<button
|
||||||
|
onClick={handleCreateManual}
|
||||||
|
disabled={!manualName.trim() || isProcessing}
|
||||||
|
className="flex items-center gap-2 px-7 py-3 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all disabled:opacity-40 disabled:cursor-not-allowed cursor-pointer"
|
||||||
|
>
|
||||||
|
{isCreating ? <><Loader2 size={14} className="animate-spin" /> Création…</> : <><Plus size={14} /> Créer le glossaire</>}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -1,131 +1,41 @@
|
|||||||
'use client';
|
'use client';
|
||||||
|
|
||||||
import { useState, useEffect, useMemo, useRef } from 'react';
|
import { useState, useEffect, useMemo } from 'react';
|
||||||
import {
|
import {
|
||||||
BookText, Plus, Library, Calendar, Hash,
|
Library, Calendar, Hash, MessageSquare, Save, Trash2, Loader2,
|
||||||
MessageSquare, Save, Trash2, Loader2,
|
CheckCircle2, AlertCircle, ArrowRight, Info, Search, Plus,
|
||||||
CheckCircle2, AlertCircle, ArrowRight, Info, ExternalLink, Search,
|
Sparkles, Zap
|
||||||
Upload, Scale, Cpu, TrendingUp, HeartPulse, Megaphone, Users, FlaskConical, ShoppingCart, Zap, PenLine
|
|
||||||
} from 'lucide-react';
|
} from 'lucide-react';
|
||||||
import Link from 'next/link';
|
import Link from 'next/link';
|
||||||
import { useRouter } from 'next/navigation';
|
import { useRouter } from 'next/navigation';
|
||||||
import { cn } from '@/lib/utils';
|
import { cn } from '@/lib/utils';
|
||||||
import { useUser } from '@/app/dashboard/useUser';
|
import { useUser } from '@/app/dashboard/useUser';
|
||||||
import { useI18n } from '@/lib/i18n';
|
import { useI18n } from '@/lib/i18n';
|
||||||
import { useGlossaries, useGlossaryTemplates } from './useGlossaries';
|
import { useGlossaries } from './useGlossaries';
|
||||||
import type { GlossaryListItem, GlossaryTermInput } from './types';
|
import type { GlossaryListItem } from './types';
|
||||||
import { ProUpgradePrompt } from './ProUpgradePrompt';
|
import { ProUpgradePrompt } from './ProUpgradePrompt';
|
||||||
import { CreateGlossaryDialog } from './CreateGlossaryDialog';
|
|
||||||
import { useToast } from '@/components/ui/toast';
|
import { useToast } from '@/components/ui/toast';
|
||||||
import { SUPPORTED_LANGUAGES } from './types';
|
import { SUPPORTED_LANGUAGES } from './types';
|
||||||
import { useTranslationStore } from '@/lib/store';
|
import { useTranslationStore } from '@/lib/store';
|
||||||
import { parseFileToTerms } from './csvUtils';
|
|
||||||
|
|
||||||
const TEMPLATE_ICONS: Record<string, React.ComponentType<{ className?: string }>> = {
|
// ── Chips de suggestions pour les consignes de contexte ─────────────────────
|
||||||
legal: Scale,
|
const CONTEXT_SUGGESTIONS = [
|
||||||
technology: Cpu,
|
{ label: 'Ton formel', value: 'Utilise toujours un ton formel et professionnel dans tes traductions.' },
|
||||||
finance: TrendingUp,
|
{ label: 'Noms propres', value: 'Ne traduis pas les noms propres, marques et noms de personnes.' },
|
||||||
medical: HeartPulse,
|
{ label: 'Chiffres', value: 'Garde tous les chiffres, pourcentages et montants tels quels sans les modifier.' },
|
||||||
marketing: Megaphone,
|
{ label: 'Placeholders', value: 'Ne traduis pas les variables entre accolades comme {nom}, {date}, {montant}.' },
|
||||||
hr: Users,
|
{ label: 'Termes techniques', value: 'Conserve les termes techniques en langue originale et ne les traduis pas.' },
|
||||||
scientific: FlaskConical,
|
{ label: 'Style concis', value: 'Préfère des formulations courtes et directes. Évite les périphrases.' },
|
||||||
ecommerce: ShoppingCart,
|
];
|
||||||
};
|
|
||||||
|
|
||||||
function FileUploadZone({
|
|
||||||
onTermsParsed,
|
|
||||||
disabled,
|
|
||||||
t,
|
|
||||||
}: {
|
|
||||||
onTermsParsed: (terms: GlossaryTermInput[], filename: string) => void;
|
|
||||||
disabled: boolean;
|
|
||||||
t: (key: string, params?: any) => string;
|
|
||||||
}) {
|
|
||||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
|
||||||
const [isDragging, setIsDragging] = useState(false);
|
|
||||||
const [status, setStatus] = useState<'idle' | 'parsing' | 'success' | 'error'>('idle');
|
|
||||||
const [errorMsg, setErrorMsg] = useState('');
|
|
||||||
|
|
||||||
const processFile = async (file: File) => {
|
|
||||||
const ext = file.name.split('.').pop()?.toLowerCase();
|
|
||||||
const allowed = ['csv', 'xlsx', 'xls', 'ods', 'txt', 'tsv'];
|
|
||||||
if (!ext || !allowed.includes(ext)) {
|
|
||||||
setStatus('error');
|
|
||||||
setErrorMsg(t('glossaries.dialog.errorFormat') || 'Format non supporté');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (file.size > 5 * 1024 * 1024) {
|
|
||||||
setStatus('error');
|
|
||||||
setErrorMsg(t('glossaries.dialog.errorSize', { max: '5' }) || 'Fichier trop volumineux (max 5MB)');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
setStatus('parsing');
|
|
||||||
setErrorMsg('');
|
|
||||||
try {
|
|
||||||
const terms = await parseFileToTerms(file);
|
|
||||||
if (terms.length === 0) {
|
|
||||||
setStatus('error');
|
|
||||||
setErrorMsg(t('glossaries.dialog.errorEmpty') || 'Fichier vide');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
setStatus('success');
|
|
||||||
onTermsParsed(terms, file.name);
|
|
||||||
setTimeout(() => setStatus('idle'), 2000);
|
|
||||||
} catch {
|
|
||||||
setStatus('error');
|
|
||||||
setErrorMsg(t('glossaries.dialog.errorRead') || 'Erreur de lecture');
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
|
function renderTitle(title: string) {
|
||||||
|
const lastSpaceIndex = title.lastIndexOf(' ');
|
||||||
|
if (lastSpaceIndex === -1) return title;
|
||||||
return (
|
return (
|
||||||
<div
|
|
||||||
onDragOver={(e) => { e.preventDefault(); setIsDragging(true); }}
|
|
||||||
onDragLeave={() => setIsDragging(false)}
|
|
||||||
onDrop={(e) => { e.preventDefault(); setIsDragging(false); const file = e.dataTransfer.files[0]; if (file) processFile(file); }}
|
|
||||||
onClick={() => !disabled && fileInputRef.current?.click()}
|
|
||||||
className={cn(
|
|
||||||
'editorial-card flex flex-col items-center justify-center gap-3 p-6 text-center transition-all cursor-pointer min-h-[140px]',
|
|
||||||
isDragging ? 'border-brand-accent bg-brand-accent/5' : 'border-black/5 dark:border-white/5 hover:border-brand-accent/30 hover:bg-brand-muted/30 dark:hover:bg-white/[0.02]',
|
|
||||||
disabled && 'opacity-50 cursor-not-allowed',
|
|
||||||
status === 'error' && 'border-destructive/30 bg-destructive/5'
|
|
||||||
)}
|
|
||||||
>
|
|
||||||
<input
|
|
||||||
ref={fileInputRef}
|
|
||||||
type="file"
|
|
||||||
accept=".csv,.xlsx,.xls,.ods,.txt,.tsv"
|
|
||||||
className="hidden"
|
|
||||||
onChange={(e) => { const file = e.target.files?.[0]; if (file) processFile(file); e.target.value = ''; }}
|
|
||||||
disabled={disabled}
|
|
||||||
/>
|
|
||||||
{status === 'parsing' ? (
|
|
||||||
<>
|
<>
|
||||||
<Loader2 className="size-6 animate-spin text-brand-accent" />
|
{title.substring(0, lastSpaceIndex)}{' '}
|
||||||
<p className="text-[11px] text-brand-dark/50 dark:text-white/40 font-light">{t('glossaries.dialog.parsing') || 'Analyse…'}</p>
|
<span className="italic">{title.substring(lastSpaceIndex + 1)}</span>
|
||||||
</>
|
</>
|
||||||
) : status === 'error' ? (
|
|
||||||
<>
|
|
||||||
<AlertCircle className="size-6 text-destructive" />
|
|
||||||
<p className="text-[11px] font-medium text-destructive leading-tight">{errorMsg}</p>
|
|
||||||
<span className="text-[9px] font-bold uppercase tracking-wider text-brand-dark/40 dark:text-white/35">{t('glossaries.dialog.retry') || 'Réessayer'}</span>
|
|
||||||
</>
|
|
||||||
) : status === 'success' ? (
|
|
||||||
<>
|
|
||||||
<CheckCircle2 className="size-6 text-emerald-600 animate-bounce" />
|
|
||||||
<p className="text-[11px] font-bold text-emerald-700 dark:text-emerald-400">{t('glossaries.toast.imported') || 'Importé avec succès'}</p>
|
|
||||||
</>
|
|
||||||
) : (
|
|
||||||
<>
|
|
||||||
<div className="w-10 h-10 rounded-xl bg-brand-accent/10 dark:bg-brand-accent/15 flex items-center justify-center text-brand-accent shrink-0 animate-pulse">
|
|
||||||
<Upload className="size-5" />
|
|
||||||
</div>
|
|
||||||
<div>
|
|
||||||
<p className="text-xs font-serif font-semibold text-brand-dark dark:text-white leading-tight tracking-tight">{t('glossaries.dialog.tabFile') || 'Glissez un fichier CSV'}</p>
|
|
||||||
<p className="text-[10px] text-brand-dark/40 dark:text-white/30 mt-1 font-light leading-snug">{t('glossaries.dialog.dropFormats') || 'Format supporté: CSV, Excel'}</p>
|
|
||||||
</div>
|
|
||||||
</>
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -133,25 +43,14 @@ export default function GlossariesPage() {
|
|||||||
const { t } = useI18n();
|
const { t } = useI18n();
|
||||||
const router = useRouter();
|
const router = useRouter();
|
||||||
const { data: user, isLoading: isLoadingUser } = useUser();
|
const { data: user, isLoading: isLoadingUser } = useUser();
|
||||||
const {
|
const { glossaries, isLoading: isLoadingGlossaries } = useGlossaries();
|
||||||
glossaries,
|
|
||||||
total,
|
|
||||||
isLoading: isLoadingGlossaries,
|
|
||||||
isCreating,
|
|
||||||
isImportingTemplate,
|
|
||||||
createGlossary,
|
|
||||||
importTemplate,
|
|
||||||
} = useGlossaries();
|
|
||||||
const { templates, isLoading: isLoadingTemplates } = useGlossaryTemplates();
|
|
||||||
const { toast } = useToast();
|
const { toast } = useToast();
|
||||||
const { settings, updateSettings } = useTranslationStore();
|
const { settings, updateSettings } = useTranslationStore();
|
||||||
|
|
||||||
const [createDialogOpen, setCreateDialogOpen] = useState(false);
|
|
||||||
const [systemPrompt, setSystemPrompt] = useState(settings.systemPrompt);
|
const [systemPrompt, setSystemPrompt] = useState(settings.systemPrompt);
|
||||||
const [isSavingPrompt, setIsSavingPrompt] = useState(false);
|
const [isSavingPrompt, setIsSavingPrompt] = useState(false);
|
||||||
const [promptSaved, setPromptSaved] = useState(false);
|
const [promptSaved, setPromptSaved] = useState(false);
|
||||||
const [searchQuery, setSearchQuery] = useState('');
|
const [searchQuery, setSearchQuery] = useState('');
|
||||||
const [importingPresetId, setImportingPresetId] = useState<string | null>(null);
|
|
||||||
const [activeTab, setActiveTab] = useState<'glossaries' | 'context'>('glossaries');
|
const [activeTab, setActiveTab] = useState<'glossaries' | 'context'>('glossaries');
|
||||||
|
|
||||||
const isPro = user?.tier === 'pro';
|
const isPro = user?.tier === 'pro';
|
||||||
@@ -186,79 +85,12 @@ export default function GlossariesPage() {
|
|||||||
setSystemPrompt('');
|
setSystemPrompt('');
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleCreateGlossary = async (data: { name: string; source_language: string; target_language: string; terms: GlossaryTermInput[] }) => {
|
const handleAddSuggestion = (value: string) => {
|
||||||
try {
|
const current = systemPrompt.trim();
|
||||||
await createGlossary(data);
|
const newPrompt = current ? `${current}\n${value}` : value;
|
||||||
setCreateDialogOpen(false);
|
setSystemPrompt(newPrompt);
|
||||||
toast({
|
|
||||||
title: t('glossaries.toast.created'),
|
|
||||||
description: t('glossaries.toast.createdDesc', { name: data.name }),
|
|
||||||
});
|
|
||||||
} catch (error) {
|
|
||||||
toast({
|
|
||||||
variant: 'destructive',
|
|
||||||
title: t('glossaries.toast.error'),
|
|
||||||
description: t('glossaries.toast.errorCreate'),
|
|
||||||
});
|
|
||||||
throw error;
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleImportPreset = async (templateId: string, name?: string) => {
|
|
||||||
setImportingPresetId(templateId);
|
|
||||||
try {
|
|
||||||
await importTemplate(templateId, name);
|
|
||||||
toast({
|
|
||||||
title: t('glossaries.toast.imported'),
|
|
||||||
description: name
|
|
||||||
? t('glossaries.toast.importedDesc', { name })
|
|
||||||
: t('glossaries.toast.importedDesc', { name: templateId }),
|
|
||||||
});
|
|
||||||
} catch (error) {
|
|
||||||
toast({
|
|
||||||
variant: 'destructive',
|
|
||||||
title: t('glossaries.toast.error'),
|
|
||||||
description: t('glossaries.toast.errorImport'),
|
|
||||||
});
|
|
||||||
} finally {
|
|
||||||
setImportingPresetId(null);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
const handleFileTermsImport = async (parsedTerms: GlossaryTermInput[], filename: string) => {
|
|
||||||
try {
|
|
||||||
const baseName = filename.replace(/\.[^.]+$/, '').replace(/[_-]/g, ' ');
|
|
||||||
await createGlossary({
|
|
||||||
name: baseName,
|
|
||||||
source_language: 'fr',
|
|
||||||
target_language: 'multi',
|
|
||||||
terms: parsedTerms,
|
|
||||||
});
|
|
||||||
toast({
|
|
||||||
title: t('glossaries.toast.created'),
|
|
||||||
description: t('glossaries.toast.createdDesc', { name: baseName }),
|
|
||||||
});
|
|
||||||
} catch (error) {
|
|
||||||
toast({
|
|
||||||
variant: 'destructive',
|
|
||||||
title: t('glossaries.toast.error'),
|
|
||||||
description: t('glossaries.toast.errorCreate'),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
const importedTemplateIds = useMemo(() => {
|
|
||||||
return new Set(
|
|
||||||
glossaries
|
|
||||||
.map((g: GlossaryListItem) => g.template_id)
|
|
||||||
.filter(Boolean) as string[]
|
|
||||||
);
|
|
||||||
}, [glossaries]);
|
|
||||||
|
|
||||||
const availableTemplates = useMemo(() => {
|
|
||||||
return templates.filter(t => !importedTemplateIds.has(t.id));
|
|
||||||
}, [templates, importedTemplateIds]);
|
|
||||||
|
|
||||||
const filteredGlossaries = useMemo(() => {
|
const filteredGlossaries = useMemo(() => {
|
||||||
if (!searchQuery.trim()) return glossaries;
|
if (!searchQuery.trim()) return glossaries;
|
||||||
const q = searchQuery.toLowerCase();
|
const q = searchQuery.toLowerCase();
|
||||||
@@ -270,7 +102,7 @@ export default function GlossariesPage() {
|
|||||||
<div className="flex items-center justify-center min-h-[60vh]">
|
<div className="flex items-center justify-center min-h-[60vh]">
|
||||||
<div className="text-center space-y-4">
|
<div className="text-center space-y-4">
|
||||||
<div className="animate-spin rounded-full h-8 w-8 border-4 border-brand-muted border-t-brand-accent mx-auto"></div>
|
<div className="animate-spin rounded-full h-8 w-8 border-4 border-brand-muted border-t-brand-accent mx-auto"></div>
|
||||||
<p className="text-xs text-brand-dark/40 dark:text-white/40 font-light">{t('glossaries.loading')}</p>
|
<p className="text-xs text-[#555555] dark:text-white/40 font-light">{t('glossaries.loading')}</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
@@ -280,149 +112,159 @@ export default function GlossariesPage() {
|
|||||||
return <ProUpgradePrompt />;
|
return <ProUpgradePrompt />;
|
||||||
}
|
}
|
||||||
|
|
||||||
const isProcessing = isCreating || isImportingTemplate || !!importingPresetId;
|
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="max-w-6xl mx-auto w-full p-6 lg:p-8">
|
<div className="max-w-6xl mx-auto w-full p-6 lg:p-8">
|
||||||
|
|
||||||
{/* ── Editorial Header ───────────────────────────────────── */}
|
{/* ── Header ────────────────────────────────────────────────── */}
|
||||||
<div className="flex flex-col sm:flex-row justify-between items-start sm:items-end mb-10 gap-6">
|
<div className="flex flex-col sm:flex-row justify-between items-start sm:items-end mb-10 gap-6">
|
||||||
<div>
|
<div>
|
||||||
<span className="accent-pill mb-3 block w-fit font-medium text-[10px] uppercase tracking-widest">
|
<span className="accent-pill mb-3 block w-fit font-medium text-[10px] uppercase tracking-widest">
|
||||||
{t('glossaries.yourGlossaries') || "Vos Glossaires"}
|
{t('glossaries.yourGlossaries') || 'Vos Glossaires'}
|
||||||
</span>
|
</span>
|
||||||
<h1 className="text-4xl md:text-5xl mb-3 leading-tight text-brand-dark dark:text-white font-serif font-medium tracking-tight">
|
<h1 className="text-4xl md:text-5xl mb-3 leading-tight text-brand-dark dark:text-white font-serif font-medium tracking-tight">
|
||||||
{renderTitle(t('glossaries.title') || "Glossaires & Contexte")}
|
{renderTitle(t('glossaries.title') || 'Glossaires & Contexte')}
|
||||||
</h1>
|
</h1>
|
||||||
<p className="text-brand-dark/50 dark:text-white/50 text-sm font-light leading-relaxed">
|
<p className="text-[#555555] dark:text-white/60 text-sm font-light leading-relaxed">
|
||||||
{t('glossaries.description') || "Gérez vos glossaires et instructions de contexte pour des traductions plus précises."}
|
{t('glossaries.description') || 'Gérez vos glossaires et instructions de contexte pour des traductions plus précises.'}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
{/* Bouton principal — unique point d'entrée pour créer */}
|
||||||
|
<Link
|
||||||
|
href="/dashboard/glossaries/new"
|
||||||
|
className="shrink-0 flex items-center gap-2 px-5 py-2.5 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all"
|
||||||
|
>
|
||||||
|
<Plus size={14} />
|
||||||
|
Nouveau glossaire
|
||||||
|
</Link>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* ── How it works ───────────────────────────────────────── */}
|
{/* ── Bandeau informatif ────────────────────────────────────── */}
|
||||||
<div className="mb-10 p-5 rounded-2xl bg-brand-accent/5 dark:bg-brand-accent/10 border border-brand-accent/20 dark:border-brand-accent/15">
|
<div className="mb-8 flex items-center gap-3 p-4 rounded-xl bg-[#F5F0EA] dark:bg-brand-accent/10 border border-[#D4B896] dark:border-brand-accent/20">
|
||||||
<div className="flex items-center gap-2 mb-4">
|
<div className="w-7 h-7 rounded-full bg-[#8B6F47] text-white flex items-center justify-center shrink-0">
|
||||||
<Info size={15} className="text-brand-accent shrink-0" />
|
<Info size={13} />
|
||||||
<span className="text-xs font-bold uppercase tracking-widest text-brand-accent">{t('glossaries.howItWorks.title')}</span>
|
|
||||||
</div>
|
</div>
|
||||||
<div className="grid grid-cols-1 sm:grid-cols-3 gap-4">
|
<p className="text-[11px] text-[#2D2D2D] dark:text-white/70 font-medium leading-relaxed flex-1">
|
||||||
{/* Step 1 */}
|
<strong className="font-bold text-[#1A1A1A] dark:text-white">Pour utiliser un glossaire dans une traduction :</strong>{' '}
|
||||||
<div className="flex items-start gap-3">
|
cliquez sur <span className="font-bold text-[#7A5C35] dark:text-brand-accent">« Utiliser »</span> sur la carte souhaitée.
|
||||||
<div className="w-7 h-7 rounded-full bg-brand-accent text-white flex items-center justify-center text-[11px] font-black shrink-0 mt-0.5">1</div>
|
Vous serez redirigé sur la page Traduire avec ce glossaire déjà sélectionné.
|
||||||
<div>
|
|
||||||
<p className="text-xs font-bold text-brand-dark dark:text-white mb-1">{t('glossaries.howItWorks.step1Title')}</p>
|
|
||||||
<p className="text-[11px] text-brand-dark/55 dark:text-white/50 font-light leading-relaxed">
|
|
||||||
{t('glossaries.howItWorks.step1Desc')}
|
|
||||||
</p>
|
</p>
|
||||||
</div>
|
<Link
|
||||||
</div>
|
href="/dashboard/translate"
|
||||||
{/* Arrow */}
|
className="shrink-0 flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-[#7A5C35] dark:text-brand-accent hover:underline whitespace-nowrap"
|
||||||
<div className="hidden sm:flex items-center justify-center">
|
>
|
||||||
<ArrowRight size={18} className="text-brand-accent/40" />
|
Aller traduire <ArrowRight size={11} />
|
||||||
</div>
|
</Link>
|
||||||
{/* Step 2 */}
|
|
||||||
<div className="flex items-start gap-3">
|
|
||||||
<div className="w-7 h-7 rounded-full bg-brand-accent text-white flex items-center justify-center text-[11px] font-black shrink-0 mt-0.5">2</div>
|
|
||||||
<div>
|
|
||||||
<p className="text-xs font-bold text-brand-dark dark:text-white mb-1">{t('glossaries.howItWorks.step2Title')}</p>
|
|
||||||
<p className="text-[11px] text-brand-dark/55 dark:text-white/50 font-light leading-relaxed">
|
|
||||||
{t('glossaries.howItWorks.step2Desc')}
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div className="mt-4 pt-4 border-t border-brand-accent/10">
|
|
||||||
<p className="text-[11px] text-brand-dark/45 dark:text-white/40 font-light">
|
|
||||||
{t('glossaries.howItWorks.warning')}
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* ── Tab Switcher ───────────────────────────────────────── */}
|
{/* ── Onglets ───────────────────────────────────────────────── */}
|
||||||
<div className="flex border-b border-black/5 dark:border-white/5 mb-8">
|
<div className="flex border-b border-black/10 dark:border-white/10 mb-8">
|
||||||
<button
|
<button
|
||||||
onClick={() => setActiveTab('glossaries')}
|
onClick={() => setActiveTab('glossaries')}
|
||||||
className={cn(
|
className={cn(
|
||||||
"pb-4 px-6 text-xs uppercase tracking-widest font-bold border-b-2 transition-all cursor-pointer",
|
'pb-4 px-6 text-xs uppercase tracking-widest font-bold border-b-2 transition-all cursor-pointer',
|
||||||
activeTab === 'glossaries'
|
activeTab === 'glossaries'
|
||||||
? "border-brand-accent text-brand-dark dark:text-white"
|
? 'border-brand-accent text-[#1A1A1A] dark:text-white'
|
||||||
: "border-transparent text-brand-dark/40 dark:text-white/40 hover:text-brand-dark/70 dark:hover:text-white/70"
|
: 'border-transparent text-[#555555] dark:text-white/40 hover:text-[#1A1A1A] dark:hover:text-white/80'
|
||||||
)}
|
)}
|
||||||
>
|
>
|
||||||
{t('glossaries.tabs.glossaries') || "Glossaires terminologiques"}
|
{t('glossaries.tabs.glossaries') || 'Glossaires terminologiques'}
|
||||||
</button>
|
</button>
|
||||||
<button
|
<button
|
||||||
onClick={() => setActiveTab('context')}
|
onClick={() => setActiveTab('context')}
|
||||||
className={cn(
|
className={cn(
|
||||||
"pb-4 px-6 text-xs uppercase tracking-widest font-bold border-b-2 transition-all cursor-pointer",
|
'pb-4 px-6 text-xs uppercase tracking-widest font-bold border-b-2 transition-all cursor-pointer',
|
||||||
activeTab === 'context'
|
activeTab === 'context'
|
||||||
? "border-brand-accent text-brand-dark dark:text-white"
|
? 'border-brand-accent text-[#1A1A1A] dark:text-white'
|
||||||
: "border-transparent text-brand-dark/40 dark:text-white/40 hover:text-brand-dark/70 dark:hover:text-white/70"
|
: 'border-transparent text-[#555555] dark:text-white/40 hover:text-[#1A1A1A] dark:hover:text-white/80'
|
||||||
)}
|
)}
|
||||||
>
|
>
|
||||||
{t('glossaries.tabs.context') || "Consignes de contexte (IA)"}
|
{t('glossaries.tabs.context') || 'Consignes de contexte (IA)'}
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="space-y-10">
|
<div className="space-y-10">
|
||||||
{activeTab === 'context' ? (
|
{activeTab === 'context' ? (
|
||||||
/* ── System Prompt (Context) ─────────────────────────────── */
|
|
||||||
|
/* ── Onglet Consignes de Contexte ─────────────────────── */
|
||||||
<section className="editorial-card p-8 lg:p-10 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl shadow-sm animate-fade-in">
|
<section className="editorial-card p-8 lg:p-10 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl shadow-sm animate-fade-in">
|
||||||
<div className="flex items-center justify-between mb-3">
|
|
||||||
<div className="flex items-center gap-3 text-brand-accent">
|
{/* En-tête section */}
|
||||||
|
<div className="flex items-start justify-between mb-6 gap-4">
|
||||||
|
<div className="flex items-start gap-3">
|
||||||
|
<div className="w-9 h-9 rounded-xl bg-brand-accent/10 flex items-center justify-center text-[#8B6F47] shrink-0">
|
||||||
<MessageSquare size={18} />
|
<MessageSquare size={18} />
|
||||||
<h3 className="text-sm font-serif font-medium text-brand-dark dark:text-white tracking-tight">
|
|
||||||
{t('context.instructions.title')}
|
|
||||||
</h3>
|
|
||||||
</div>
|
</div>
|
||||||
|
<div>
|
||||||
|
<h2 className="text-base font-serif font-semibold text-[#1A1A1A] dark:text-white tracking-tight">
|
||||||
|
Consignes de contexte
|
||||||
|
</h2>
|
||||||
|
<p className="text-[11px] text-[#555555] dark:text-white/50 font-light mt-0.5">
|
||||||
|
Ces instructions sont envoyées automatiquement à l'IA avant chaque traduction en mode Pro LLM.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{/* Badge d'état */}
|
||||||
{promptHasUnsavedChanges ? (
|
{promptHasUnsavedChanges ? (
|
||||||
<span className="flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-amber-600 dark:text-amber-400 bg-amber-500/10 px-3 py-1 rounded-full border border-amber-500/20">
|
<span className="shrink-0 flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-amber-700 bg-amber-100 dark:text-amber-400 dark:bg-amber-500/10 px-3 py-1.5 rounded-full border border-amber-300 dark:border-amber-500/20">
|
||||||
<AlertCircle size={11} />
|
<AlertCircle size={11} />
|
||||||
{t('glossaries.status.unsaved')}
|
Non sauvegardé
|
||||||
</span>
|
</span>
|
||||||
) : promptIsActive ? (
|
) : promptIsActive ? (
|
||||||
<span className="flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-emerald-600 dark:text-emerald-400 bg-emerald-500/10 px-3 py-1 rounded-full border border-emerald-500/20">
|
<span className="shrink-0 flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-emerald-700 bg-emerald-100 dark:text-emerald-400 dark:bg-emerald-500/10 px-3 py-1.5 rounded-full border border-emerald-300 dark:border-emerald-500/20">
|
||||||
<CheckCircle2 size={11} />
|
<CheckCircle2 size={11} />
|
||||||
{t('glossaries.status.active')}
|
Actif
|
||||||
</span>
|
</span>
|
||||||
) : (
|
) : (
|
||||||
<span className="flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-brand-dark/30 dark:text-white/30 bg-brand-muted dark:bg-white/5 px-3 py-1 rounded-full border border-black/5 dark:border-white/5">
|
<span className="shrink-0 flex items-center gap-1.5 text-[10px] font-bold uppercase tracking-wider text-[#555555] bg-[#EBEBEB] dark:text-white/30 dark:bg-white/5 px-3 py-1.5 rounded-full border border-[#CCCCCC] dark:border-white/10">
|
||||||
{t('glossaries.status.inactive')}
|
Inactif
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Explanation box */}
|
{/* Chips de suggestions rapides */}
|
||||||
<div className="mb-5 p-3.5 rounded-xl bg-brand-muted/40 dark:bg-white/[0.03] border border-black/5 dark:border-white/5">
|
<div className="mb-5">
|
||||||
<p className="text-[11px] text-brand-dark/60 dark:text-white/50 font-light leading-relaxed">
|
<div className="flex items-center gap-2 mb-3">
|
||||||
<strong className="font-bold text-brand-dark/80 dark:text-white/80">{t('glossaries.instructions.whatForBold')}</strong> {t('glossaries.instructions.whatForDesc')}
|
<Sparkles size={13} className="text-[#8B6F47]" />
|
||||||
</p>
|
<span className="text-[10px] font-bold uppercase tracking-widest text-[#555555] dark:text-white/40">
|
||||||
<p className="text-[11px] text-brand-accent/80 dark:text-brand-accent/70 font-medium mt-2 italic">
|
Suggestions rapides — cliquez pour ajouter
|
||||||
{t('glossaries.instructions.example')}
|
</span>
|
||||||
</p>
|
</div>
|
||||||
<p className="text-[10px] text-brand-accent font-bold mt-2.5 flex items-center gap-1.5 border-t border-black/5 dark:border-white/5 pt-2">
|
<div className="flex flex-wrap gap-2">
|
||||||
<Info size={11} />
|
{CONTEXT_SUGGESTIONS.map((s) => (
|
||||||
Remarque : Ces consignes s'appliquent automatiquement à toutes vos traductions réalisées en mode Pro LLM.
|
<button
|
||||||
</p>
|
key={s.label}
|
||||||
|
onClick={() => handleAddSuggestion(s.value)}
|
||||||
|
className="px-3 py-1.5 rounded-lg text-[11px] font-semibold text-[#3D3D3D] dark:text-white/70 bg-[#F0EDE8] dark:bg-white/5 border border-[#D9D5CE] dark:border-white/10 hover:bg-[#E8E2DA] hover:border-[#C5A17A] dark:hover:bg-brand-accent/10 dark:hover:border-brand-accent/30 transition-all cursor-pointer"
|
||||||
|
>
|
||||||
|
+ {s.label}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Zone de texte */}
|
||||||
<textarea
|
<textarea
|
||||||
value={systemPrompt}
|
value={systemPrompt}
|
||||||
onChange={e => { setSystemPrompt(e.target.value); setPromptSaved(false); }}
|
onChange={e => { setSystemPrompt(e.target.value); setPromptSaved(false); }}
|
||||||
placeholder={t('context.instructions.placeholder')}
|
placeholder="Ex : Vous traduisez des documents techniques HVAC. Utilisez une terminologie d'ingénierie précise et un ton professionnel…"
|
||||||
className="w-full h-40 p-4 bg-brand-muted/30 dark:bg-white/[0.02] rounded-xl border border-black/5 dark:border-white/10 text-xs focus:ring-2 focus:ring-brand-accent/20 focus:border-brand-accent/30 transition-all outline-none resize-y"
|
className="w-full h-44 p-4 bg-[#FAFAF8] dark:bg-white/[0.02] rounded-xl border border-[#D9D6D0] dark:border-white/10 text-[#1A1A1A] dark:text-white text-xs leading-relaxed focus:ring-2 focus:ring-brand-accent/30 focus:border-brand-accent/50 transition-all outline-none resize-y placeholder:text-[#AAAAAA] dark:placeholder:text-white/25"
|
||||||
/>
|
/>
|
||||||
|
|
||||||
|
{/* Pied de section */}
|
||||||
<div className="flex justify-between items-center mt-4">
|
<div className="flex justify-between items-center mt-4">
|
||||||
<p className="text-[10px] text-brand-dark/30 dark:text-white/25 font-light">
|
<p className="text-[10px] text-[#777777] dark:text-white/25 font-light">
|
||||||
{systemPrompt.length > 0 ? t('glossaries.instructions.charCount', { count: systemPrompt.length }) : t('glossaries.instructions.emptyHint')}
|
{systemPrompt.length > 0
|
||||||
|
? `${systemPrompt.length} caractères`
|
||||||
|
: 'Aucune consigne enregistrée'}
|
||||||
</p>
|
</p>
|
||||||
<div className="flex gap-3">
|
<div className="flex gap-3">
|
||||||
<button
|
<button
|
||||||
onClick={handleClearPrompt}
|
onClick={handleClearPrompt}
|
||||||
className="px-5 py-2.5 bg-brand-muted dark:bg-white/5 text-brand-dark/50 dark:text-white/40 rounded-lg text-xs font-bold uppercase tracking-wider hover:text-brand-dark dark:hover:text-white transition-all cursor-pointer"
|
disabled={!systemPrompt}
|
||||||
|
className="px-5 py-2.5 bg-[#EBEBEB] dark:bg-white/5 text-[#444444] dark:text-white/40 rounded-lg text-xs font-bold uppercase tracking-wider hover:text-[#1A1A1A] hover:bg-[#E0E0E0] dark:hover:text-white transition-all cursor-pointer disabled:opacity-40 disabled:cursor-not-allowed"
|
||||||
>
|
>
|
||||||
<Trash2 size={12} className="inline mr-1.5" />{t('glossaries.instructions.clearAll')}
|
<Trash2 size={12} className="inline mr-1.5" />
|
||||||
|
Tout effacer
|
||||||
</button>
|
</button>
|
||||||
<button
|
<button
|
||||||
onClick={handleSavePrompt}
|
onClick={handleSavePrompt}
|
||||||
@@ -430,77 +272,84 @@ export default function GlossariesPage() {
|
|||||||
className="premium-button px-8 py-2.5 text-xs uppercase tracking-widest !rounded-lg flex items-center gap-2 disabled:opacity-50 cursor-pointer font-bold"
|
className="premium-button px-8 py-2.5 text-xs uppercase tracking-widest !rounded-lg flex items-center gap-2 disabled:opacity-50 cursor-pointer font-bold"
|
||||||
>
|
>
|
||||||
{isSavingPrompt
|
{isSavingPrompt
|
||||||
? <><Loader2 size={14} className="animate-spin" /> {t('glossaries.instructions.saving')}</>
|
? <><Loader2 size={14} className="animate-spin" /> Enregistrement…</>
|
||||||
: promptSaved
|
: promptSaved
|
||||||
? <><CheckCircle2 size={14} /> {t('glossaries.instructions.saved')}</>
|
? <><CheckCircle2 size={14} /> Enregistré</>
|
||||||
: <><Save size={14} /> {t('context.save')}</>
|
: <><Save size={14} /> Enregistrer</>
|
||||||
}
|
}
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
) : (
|
) : (
|
||||||
<div className="space-y-12 animate-fade-in">
|
|
||||||
{/* ── Section 1 : Vos Glossaires (En premier !) ───────────────── */}
|
/* ── Onglet Glossaires ──────────────────────────────────── */
|
||||||
<section>
|
<div className="space-y-6 animate-fade-in">
|
||||||
<div className="flex items-center justify-between mb-5 gap-4">
|
|
||||||
|
{/* En-tête de section + compteur */}
|
||||||
|
<div className="flex items-center justify-between gap-4">
|
||||||
<div>
|
<div>
|
||||||
<h2 className="text-lg font-serif font-medium text-brand-dark dark:text-white tracking-tight">
|
<h2 className="text-lg font-serif font-medium text-[#1A1A1A] dark:text-white tracking-tight">
|
||||||
{t('glossaries.grid.title')} <span className="italic">{t('glossaries.grid.titleHighlight')}</span>
|
{t('glossaries.grid.title') || 'Vos'}{' '}
|
||||||
|
<span className="italic">{t('glossaries.grid.titleHighlight') || 'glossaires'}</span>
|
||||||
</h2>
|
</h2>
|
||||||
<p className="text-[11px] text-brand-dark/45 dark:text-white/40 font-light mt-1">
|
<p className="text-[11px] text-[#555555] dark:text-white/40 font-light mt-1">
|
||||||
{glossaries.length > 0
|
{glossaries.length > 0
|
||||||
? t('glossaries.grid.countWithAction', { count: glossaries.length, plural: glossaries.length > 1 ? 's' : '' })
|
? `${glossaries.length} glossaire${glossaries.length > 1 ? 's' : ''} — cliquez sur une carte pour la modifier`
|
||||||
: t('glossaries.grid.emptyAction')}
|
: 'Aucun glossaire — créez-en un pour commencer'}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<div className="flex items-center gap-3 shrink-0">
|
|
||||||
{currentTargetInfo && (
|
{currentTargetInfo && (
|
||||||
<span className="flex items-center gap-1.5 text-[10px] font-bold text-brand-dark/50 dark:text-white/40 bg-brand-muted dark:bg-white/5 border border-black/5 dark:border-white/5 px-3 py-1.5 rounded-full">
|
<span className="flex items-center gap-1.5 text-[10px] font-bold text-[#444444] dark:text-white/40 bg-[#EBEBEB] dark:bg-white/5 border border-[#CCCCCC] dark:border-white/5 px-3 py-1.5 rounded-full shrink-0">
|
||||||
<span>{t('glossaries.grid.activeTranslation')}</span>
|
<span>Traduction active :</span>
|
||||||
<span>{currentTargetInfo.flag} {currentTargetInfo.label}</span>
|
<span>{currentTargetInfo.flag} {currentTargetInfo.label}</span>
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
|
||||||
|
|
||||||
{/* Search bar (only if more than 3 glossaries) */}
|
{/* Barre de recherche (si > 3 glossaires) */}
|
||||||
{glossaries.length > 3 && (
|
{glossaries.length > 3 && (
|
||||||
<div className="relative mb-5">
|
<div className="relative">
|
||||||
<Search size={14} className="absolute left-3 top-1/2 -translate-y-1/2 text-brand-dark/30 dark:text-white/30" />
|
<Search size={14} className="absolute left-3 top-1/2 -translate-y-1/2 text-[#888888] dark:text-white/30" />
|
||||||
<input
|
<input
|
||||||
type="text"
|
type="text"
|
||||||
value={searchQuery}
|
value={searchQuery}
|
||||||
onChange={e => setSearchQuery(e.target.value)}
|
onChange={e => setSearchQuery(e.target.value)}
|
||||||
placeholder={t('glossaries.grid.searchPlaceholder') || "Rechercher un glossaire…"}
|
placeholder="Rechercher un glossaire…"
|
||||||
className="w-full pl-9 pr-3 py-2.5 text-xs rounded-lg border border-black/5 dark:border-white/10 bg-white dark:bg-[#141414] focus:outline-none focus:ring-2 focus:ring-brand-accent/20 focus:border-brand-accent/30"
|
className="w-full pl-9 pr-3 py-2.5 text-xs rounded-lg border border-[#D9D6D0] dark:border-white/10 bg-white dark:bg-[#141414] text-[#1A1A1A] dark:text-white placeholder:text-[#AAAAAA] dark:placeholder:text-white/25 focus:outline-none focus:ring-2 focus:ring-brand-accent/20 focus:border-brand-accent/30"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{/* État vide */}
|
||||||
{glossaries.length === 0 ? (
|
{glossaries.length === 0 ? (
|
||||||
<div className="editorial-card p-12 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl shadow-sm text-center">
|
<div className="editorial-card p-16 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl shadow-sm text-center">
|
||||||
<div className="w-12 h-12 bg-brand-muted dark:bg-white/10 rounded-xl flex items-center justify-center text-brand-accent mx-auto mb-4">
|
<div className="w-14 h-14 bg-[#F0EDE8] dark:bg-white/10 rounded-2xl flex items-center justify-center text-[#8B6F47] mx-auto mb-5">
|
||||||
<Library size={24} />
|
<Library size={28} />
|
||||||
</div>
|
</div>
|
||||||
<p className="text-base font-serif font-medium text-brand-dark dark:text-white mb-1">{t('glossaries.empty')}</p>
|
<p className="text-base font-serif font-semibold text-[#1A1A1A] dark:text-white mb-2">
|
||||||
<p className="text-xs text-brand-dark/40 dark:text-white/40 font-light mb-5">{t('glossaries.emptyDesc')}</p>
|
{t('glossaries.empty') || 'Aucun glossaire'}
|
||||||
<button
|
</p>
|
||||||
onClick={() => setCreateDialogOpen(true)}
|
<p className="text-sm text-[#555555] dark:text-white/40 font-light mb-6 max-w-xs mx-auto">
|
||||||
className="premium-button px-6 py-2.5 text-[11px] uppercase tracking-widest !rounded-lg inline-flex items-center gap-2 cursor-pointer font-bold"
|
{t('glossaries.emptyDesc') || 'Créez votre premier glossaire pour garantir une terminologie cohérente dans vos traductions.'}
|
||||||
|
</p>
|
||||||
|
<Link
|
||||||
|
href="/dashboard/glossaries/new"
|
||||||
|
className="inline-flex items-center gap-2 px-6 py-2.5 rounded-xl bg-[#1A1A1A] hover:bg-[#333333] text-white text-xs font-bold uppercase tracking-widest transition-all"
|
||||||
>
|
>
|
||||||
<Plus size={12} />
|
<Plus size={12} />
|
||||||
{t('glossaries.createNew') || "Créer un glossaire"}
|
Créer mon premier glossaire
|
||||||
</button>
|
</Link>
|
||||||
</div>
|
</div>
|
||||||
) : filteredGlossaries.length === 0 ? (
|
) : filteredGlossaries.length === 0 ? (
|
||||||
<div className="editorial-card p-8 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl shadow-sm text-center">
|
<div className="p-8 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl text-center">
|
||||||
<p className="text-xs text-brand-dark/50 dark:text-white/50 font-light">
|
<p className="text-sm text-[#555555] dark:text-white/40 font-light">
|
||||||
{t('glossaries.grid.noResults') || "Aucun résultat pour cette recherche."}
|
Aucun résultat pour « {searchQuery} »
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
) : (
|
) : (
|
||||||
<div className="grid md:grid-cols-2 lg:grid-cols-3 gap-6">
|
<div className="grid md:grid-cols-2 lg:grid-cols-3 gap-5">
|
||||||
{filteredGlossaries.map((glossary: GlossaryListItem) => {
|
{filteredGlossaries.map((glossary: GlossaryListItem) => {
|
||||||
const termCount = glossary.terms_count ?? 0;
|
const termCount = glossary.terms_count ?? 0;
|
||||||
const srcInfo = SUPPORTED_LANGUAGES.find(l => l.code === glossary.source_language);
|
const srcInfo = SUPPORTED_LANGUAGES.find(l => l.code === glossary.source_language);
|
||||||
@@ -508,72 +357,76 @@ export default function GlossariesPage() {
|
|||||||
const isMultilingual = glossary.target_language === 'multi';
|
const isMultilingual = glossary.target_language === 'multi';
|
||||||
const matchesTarget = currentTargetLang && (isMultilingual || glossary.target_language === currentTargetLang);
|
const matchesTarget = currentTargetLang && (isMultilingual || glossary.target_language === currentTargetLang);
|
||||||
const mismatch = currentTargetLang && !isMultilingual && glossary.target_language && glossary.target_language !== currentTargetLang;
|
const mismatch = currentTargetLang && !isMultilingual && glossary.target_language && glossary.target_language !== currentTargetLang;
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div
|
<div
|
||||||
key={glossary.id}
|
key={glossary.id}
|
||||||
className={cn(
|
className={cn(
|
||||||
'editorial-card p-6 bg-white dark:bg-[#141414] border rounded-2xl shadow-sm group transition-all relative text-left w-full flex flex-col justify-between min-h-[200px]',
|
'bg-white dark:bg-[#141414] border rounded-2xl p-6 shadow-sm group transition-all relative flex flex-col justify-between min-h-[200px]',
|
||||||
matchesTarget
|
matchesTarget
|
||||||
? 'border-brand-accent/40 ring-1 ring-brand-accent/20 hover:border-brand-accent/60'
|
? 'border-brand-accent/50 ring-1 ring-brand-accent/20 hover:border-brand-accent/70'
|
||||||
: mismatch
|
: mismatch
|
||||||
? 'border-amber-300/40 dark:border-amber-500/20 hover:border-amber-400/60 opacity-75 hover:opacity-100'
|
? 'border-amber-300/50 dark:border-amber-500/20 hover:border-amber-400/70 opacity-80 hover:opacity-100'
|
||||||
: 'border-black/5 dark:border-white/5 hover:border-brand-accent/30'
|
: 'border-[#E5E3DF] dark:border-white/5 hover:border-[#C5A17A]/40'
|
||||||
)}
|
)}
|
||||||
>
|
>
|
||||||
|
{/* Badge compatible / autre langue */}
|
||||||
{matchesTarget && (
|
{matchesTarget && (
|
||||||
<div className="absolute top-3 right-3 flex items-center gap-1 bg-brand-accent/10 text-brand-accent px-2 py-0.5 rounded-full text-[9px] font-black uppercase tracking-wider">
|
<div className="absolute top-3 right-3 flex items-center gap-1 bg-emerald-100 dark:bg-emerald-500/10 text-emerald-700 dark:text-emerald-400 px-2 py-0.5 rounded-full text-[9px] font-black uppercase tracking-wider border border-emerald-300 dark:border-emerald-500/20">
|
||||||
<CheckCircle2 size={9} /> {t('glossaries.badge.compatible')}
|
<CheckCircle2 size={9} /> Compatible
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
{mismatch && (
|
{mismatch && (
|
||||||
<div className="absolute top-3 right-3 flex items-center gap-1 bg-amber-500/10 text-amber-600 dark:text-amber-400 px-2 py-0.5 rounded-full text-[9px] font-black uppercase tracking-wider">
|
<div className="absolute top-3 right-3 flex items-center gap-1 bg-amber-100 dark:bg-amber-500/10 text-amber-700 dark:text-amber-400 px-2 py-0.5 rounded-full text-[9px] font-black uppercase tracking-wider border border-amber-300 dark:border-amber-500/20">
|
||||||
<AlertCircle size={9} /> {t('glossaries.badge.otherTarget')}
|
<AlertCircle size={9} /> Autre langue
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{/* Corps cliquable → page de détail */}
|
||||||
<div className="cursor-pointer flex-1" onClick={() => router.push(`/dashboard/glossaries/${glossary.id}`)}>
|
<div className="cursor-pointer flex-1" onClick={() => router.push(`/dashboard/glossaries/${glossary.id}`)}>
|
||||||
<div className="flex items-start gap-3 mb-4">
|
<div className="flex items-start gap-3 mb-4">
|
||||||
<div className="w-10 h-10 bg-brand-muted dark:bg-white/10 rounded-xl flex items-center justify-center text-brand-accent shrink-0">
|
<div className="w-10 h-10 bg-[#F0EDE8] dark:bg-white/10 rounded-xl flex items-center justify-center text-[#8B6F47] shrink-0">
|
||||||
<Library size={18} />
|
<Library size={18} />
|
||||||
</div>
|
</div>
|
||||||
<div className="flex-1 min-w-0 pt-0.5">
|
<div className="flex-1 min-w-0 pt-0.5">
|
||||||
<h3 className="text-sm font-serif font-semibold text-brand-dark dark:text-white tracking-tight leading-snug line-clamp-2 group-hover:text-brand-accent transition-colors">
|
<h3 className="text-sm font-serif font-semibold text-[#1A1A1A] dark:text-white tracking-tight leading-snug line-clamp-2 group-hover:text-[#8B6F47] transition-colors">
|
||||||
{glossary.name}
|
{glossary.name}
|
||||||
</h3>
|
</h3>
|
||||||
<p className="text-[11px] text-brand-dark/40 dark:text-white/40 font-medium flex items-center gap-1 mt-0.5">
|
<p className="text-[11px] text-[#555555] dark:text-white/50 font-medium flex items-center gap-1 mt-0.5">
|
||||||
<span>{srcInfo?.flag ?? '🌐'}</span>
|
<span>{srcInfo?.flag ?? '🌐'}</span>
|
||||||
<span>{srcInfo?.label ?? glossary.source_language}</span>
|
<span>{srcInfo?.label ?? glossary.source_language}</span>
|
||||||
<span className="text-brand-accent font-bold">→</span>
|
<span className="text-[#7A5C35] dark:text-brand-accent font-bold">→</span>
|
||||||
<span>{tgtInfo?.flag ?? '🌐'}</span>
|
<span>{tgtInfo?.flag ?? '🌐'}</span>
|
||||||
<span>{tgtInfo?.label ?? glossary.target_language}</span>
|
<span>{tgtInfo?.label ?? glossary.target_language}</span>
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="flex justify-between items-center pb-4 text-xs text-brand-dark/40 dark:text-white/40">
|
<div className="flex justify-between items-center pb-4">
|
||||||
<span className="flex items-center gap-1 font-semibold text-brand-dark/65 dark:text-white/65">
|
<span className="flex items-center gap-1 text-xs font-semibold text-[#333333] dark:text-white/65">
|
||||||
<Hash size={12} className="text-brand-accent" />
|
<Hash size={12} className="text-[#8B6F47] dark:text-brand-accent" />
|
||||||
{termCount} {t('glossaries.defineTerms') || "termes"}
|
{termCount} terme{termCount > 1 ? 's' : ''}
|
||||||
</span>
|
</span>
|
||||||
<span className="flex items-center gap-1 font-mono text-[9px] opacity-75">
|
<span className="flex items-center gap-1 font-mono text-[9px] text-[#666666] dark:text-white/40">
|
||||||
<Calendar size={11} />
|
<Calendar size={11} />
|
||||||
{new Date(glossary.created_at).toLocaleDateString()}
|
{new Date(glossary.created_at).toLocaleDateString('fr-FR')}
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="flex gap-2 pt-4 border-t border-black/5 dark:border-white/10 mt-auto shrink-0 w-full">
|
{/* Actions */}
|
||||||
|
<div className="flex gap-2 pt-4 border-t border-[#F0EDE8] dark:border-white/10 mt-auto shrink-0">
|
||||||
<Link
|
<Link
|
||||||
href={`/dashboard/glossaries/${glossary.id}`}
|
href={`/dashboard/glossaries/${glossary.id}`}
|
||||||
className="flex-1 text-center py-2 rounded-lg bg-brand-muted/65 dark:bg-white/5 hover:bg-brand-accent/10 text-brand-dark/70 dark:text-white/60 hover:text-brand-accent text-[10px] font-bold uppercase tracking-wider transition-all"
|
className="flex-none px-3 py-2 rounded-lg bg-[#EBEBEB] dark:bg-white/5 hover:bg-[#E0D9D1] text-[#333333] dark:text-white/50 hover:text-[#1A1A1A] text-[10px] font-bold uppercase tracking-wider transition-all border border-[#CCCCCC] dark:border-white/10"
|
||||||
>
|
>
|
||||||
Détails / Éditer
|
Éditer
|
||||||
</Link>
|
</Link>
|
||||||
<Link
|
<Link
|
||||||
href={`/dashboard/translate?glossaryId=${glossary.id}`}
|
href={`/dashboard/translate?glossaryId=${glossary.id}`}
|
||||||
className="flex-1 text-center py-2 rounded-lg bg-brand-accent hover:bg-brand-accent/90 text-white text-[10px] font-bold uppercase tracking-wider transition-all flex items-center justify-center gap-1.5"
|
className="flex-1 text-center py-2 rounded-lg bg-[#1A1A1A] hover:bg-[#333333] text-white text-[10px] font-bold uppercase tracking-wider transition-all flex items-center justify-center gap-1.5"
|
||||||
>
|
>
|
||||||
Traduire
|
Utiliser <ArrowRight size={11} />
|
||||||
</Link>
|
</Link>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -581,172 +434,9 @@ export default function GlossariesPage() {
|
|||||||
})}
|
})}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</section>
|
|
||||||
|
|
||||||
{/* ── Section 2 : Créer ou Importer ─────────────────────────── */}
|
|
||||||
<section className="editorial-card p-8 lg:p-10 bg-white dark:bg-[#141414] border border-black/5 dark:border-white/5 rounded-2xl shadow-sm">
|
|
||||||
<div className="mb-6">
|
|
||||||
<div className="flex items-center gap-2 text-brand-accent mb-2">
|
|
||||||
<Zap size={18} />
|
|
||||||
<h2 className="text-sm font-serif font-medium text-brand-dark dark:text-white tracking-tight">
|
|
||||||
{t('glossaries.presets.whatForBold') || 'Créer un nouveau glossaire'}
|
|
||||||
</h2>
|
|
||||||
</div>
|
|
||||||
<p className="text-xs text-brand-dark/50 dark:text-white/40 font-light">
|
|
||||||
{t('glossaries.presets.whatForDesc') || 'Choisissez un modèle professionnel pré-rempli ou importez vos propres termes.'}
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div className="grid grid-cols-1 lg:grid-cols-3 gap-8">
|
|
||||||
{/* Grille des Modèles (2/3 de largeur) */}
|
|
||||||
<div className="lg:col-span-2 space-y-3">
|
|
||||||
<div className="flex items-center gap-1.5 text-brand-accent">
|
|
||||||
<BookText size={14} />
|
|
||||||
<span className="text-[10px] font-bold uppercase tracking-wider">{t('context.presets.title') || 'Modèles professionnels'}</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{isLoadingTemplates ? (
|
|
||||||
<div className="flex items-center justify-center py-12">
|
|
||||||
<Loader2 className="size-6 animate-spin text-brand-muted" />
|
|
||||||
</div>
|
|
||||||
) : availableTemplates.length === 0 ? (
|
|
||||||
<div className="p-8 bg-brand-muted/20 dark:bg-white/[0.01] border border-black/5 dark:border-white/5 rounded-xl text-center w-full">
|
|
||||||
<p className="text-xs text-brand-dark/40 dark:text-white/30 font-light">
|
|
||||||
{t('glossaries.presets.allImported') || 'Tous les modèles professionnels ont été importés.'}
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
) : (
|
|
||||||
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
|
|
||||||
{availableTemplates.map((template) => {
|
|
||||||
const Icon = TEMPLATE_ICONS[template.id] || BookText;
|
|
||||||
const isImported = importedTemplateIds.has(template.id);
|
|
||||||
const isProcessingThis = importingPresetId === template.id;
|
|
||||||
|
|
||||||
return (
|
|
||||||
<div
|
|
||||||
key={template.id}
|
|
||||||
className={cn(
|
|
||||||
'relative p-4 rounded-xl text-left border transition-all min-h-[110px] flex flex-col justify-between group',
|
|
||||||
isImported
|
|
||||||
? 'bg-emerald-500/5 dark:bg-emerald-500/5 border-emerald-500/20 dark:border-emerald-500/10 opacity-90'
|
|
||||||
: 'bg-brand-muted/40 dark:bg-white/5 border-black/5 dark:border-white/5 hover:border-brand-accent/20 hover:bg-brand-accent/[0.02]'
|
|
||||||
)}
|
|
||||||
>
|
|
||||||
<div className="flex items-center justify-between gap-2 w-full">
|
|
||||||
<div className="p-1.5 bg-brand-accent/10 rounded-lg text-brand-accent group-hover:scale-115 transition-transform">
|
|
||||||
{isProcessingThis ? <Loader2 size={16} className="animate-spin" /> : <Icon className="size-4" />}
|
|
||||||
</div>
|
|
||||||
{isImported ? (
|
|
||||||
<div className="flex items-center gap-2">
|
|
||||||
<span className="flex items-center gap-1 text-[9px] font-bold uppercase tracking-wider text-emerald-600 dark:text-emerald-400 bg-emerald-500/10 px-2 py-0.5 rounded-full border border-emerald-500/20">
|
|
||||||
<CheckCircle2 size={9} /> {t('glossaries.presets.alreadyImported') || 'Importé'}
|
|
||||||
</span>
|
|
||||||
{(() => {
|
|
||||||
const matchingGlossary = glossaries.find((g) => g.template_id === template.id);
|
|
||||||
return matchingGlossary ? (
|
|
||||||
<Link
|
|
||||||
href={`/dashboard/glossaries/${matchingGlossary.id}`}
|
|
||||||
className="text-[9px] font-bold uppercase tracking-wider text-brand-accent hover:underline flex items-center gap-0.5"
|
|
||||||
>
|
|
||||||
Modifier
|
|
||||||
</Link>
|
|
||||||
) : null;
|
|
||||||
})()}
|
|
||||||
</div>
|
|
||||||
) : (
|
|
||||||
<button
|
|
||||||
disabled={isProcessing}
|
|
||||||
onClick={() => handleImportPreset(template.id, template.name)}
|
|
||||||
className="accent-pill !px-3 !py-1 !text-[9px] font-bold uppercase tracking-wider bg-brand-accent text-white hover:bg-brand-accent/90 rounded-md transition-colors cursor-pointer"
|
|
||||||
>
|
|
||||||
{t('glossaries.presets.importBtn') || 'Importer'}
|
|
||||||
</button>
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
<div className="mt-3">
|
|
||||||
<div className="flex justify-between items-baseline gap-2">
|
|
||||||
<p className="text-xs font-serif font-bold text-brand-dark dark:text-white leading-tight">
|
|
||||||
{template.name.split(' - ')[0]}
|
|
||||||
</p>
|
|
||||||
<span className="text-[9px] text-brand-dark/40 dark:text-white/30 font-semibold font-mono shrink-0">
|
|
||||||
{template.terms_count} {t('glossaries.defineTerms') || 'termes'}
|
|
||||||
</span>
|
|
||||||
</div>
|
|
||||||
<p className="text-[10px] text-brand-dark/45 dark:text-white/35 font-light leading-normal mt-1 line-clamp-2">
|
|
||||||
{template.description}
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
);
|
|
||||||
})}
|
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Fichier & Manuel (1/3 de largeur) */}
|
|
||||||
<div className="space-y-5">
|
|
||||||
{/* Import Fichier */}
|
|
||||||
<div className="space-y-2.5">
|
|
||||||
<div className="flex items-center gap-1.5 text-brand-accent">
|
|
||||||
<Upload size={14} />
|
|
||||||
<span className="text-[10px] font-bold uppercase tracking-wider">{t('glossaries.dialog.tabFile') || 'Importer un fichier'}</span>
|
|
||||||
</div>
|
|
||||||
<FileUploadZone
|
|
||||||
onTermsParsed={handleFileTermsImport}
|
|
||||||
disabled={isProcessing}
|
|
||||||
t={t}
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{/* Création Manuelle */}
|
|
||||||
<div className="space-y-2.5">
|
|
||||||
<div className="flex items-center gap-1.5 text-brand-accent">
|
|
||||||
<PenLine size={14} />
|
|
||||||
<span className="text-[10px] font-bold uppercase tracking-wider">{t('glossaries.dialog.tabManual') || 'Création manuelle'}</span>
|
|
||||||
</div>
|
|
||||||
<button
|
|
||||||
onClick={() => setCreateDialogOpen(true)}
|
|
||||||
disabled={isProcessing}
|
|
||||||
className="editorial-card w-full flex items-center justify-between p-5 text-left transition-all hover:border-brand-accent/30 hover:bg-brand-muted/30 dark:hover:bg-white/[0.02] cursor-pointer group border-black/5 dark:border-white/5"
|
|
||||||
>
|
|
||||||
<div className="flex items-center gap-3">
|
|
||||||
<div className="w-9 h-9 rounded-lg bg-brand-accent/10 dark:bg-brand-accent/15 flex items-center justify-center text-brand-accent shrink-0 group-hover:scale-110 transition-transform">
|
|
||||||
<Plus size={16} />
|
|
||||||
</div>
|
|
||||||
<div>
|
|
||||||
<p className="text-xs font-serif font-bold text-brand-dark dark:text-white leading-tight">{t('glossaries.createNew') || 'Créer manuellement'}</p>
|
|
||||||
<p className="text-[10px] text-brand-dark/40 dark:text-white/35 mt-1 font-light leading-tight">{t('glossaries.dialog.createEmpty') || 'À partir de zéro'}</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<ArrowRight className="size-4 text-brand-dark/30 dark:text-white/20 group-hover:text-brand-accent group-hover:translate-x-1 transition-all" />
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
</div>
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{/* Dialogs */}
|
|
||||||
<CreateGlossaryDialog
|
|
||||||
open={createDialogOpen}
|
|
||||||
onOpenChange={setCreateDialogOpen}
|
|
||||||
onCreate={handleCreateGlossary}
|
|
||||||
isCreating={isCreating}
|
|
||||||
/>
|
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
function renderTitle(title: string) {
|
|
||||||
const lastSpaceIndex = title.lastIndexOf(' ');
|
|
||||||
if (lastSpaceIndex === -1) return title;
|
|
||||||
const firstPart = title.substring(0, lastSpaceIndex);
|
|
||||||
const lastWord = title.substring(lastSpaceIndex + 1);
|
|
||||||
return (
|
|
||||||
<>
|
|
||||||
{firstPart} <span className="italic">{lastWord}</span>
|
|
||||||
</>
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -742,9 +742,14 @@ export default function TranslatePage() {
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* ── MOMENTO PROMO BANNER ──────────────────────────────── */}
|
{/* ── MEMENTO PROMO BANNER ──────────────────────────────── */}
|
||||||
{(showUpload || showConfiguring || showFailed) && (
|
{(showUpload || showConfiguring || showFailed) && (
|
||||||
<div className="mt-12 editorial-card p-10 bg-white dark:bg-[#141414] border-none shadow-editorial flex flex-col md:flex-row items-center gap-8 group overflow-hidden relative">
|
<a
|
||||||
|
href={t('memento.url', { defaultValue: 'https://memento-note.com/' })}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="block mt-12 editorial-card p-10 bg-white dark:bg-[#141414] border-none shadow-editorial flex flex-col md:flex-row items-center gap-8 group overflow-hidden relative cursor-pointer"
|
||||||
|
>
|
||||||
<div className="absolute -right-20 -top-20 w-64 h-64 bg-brand-accent/5 rounded-full blur-3xl group-hover:bg-brand-accent/10 transition-colors pointer-events-none" />
|
<div className="absolute -right-20 -top-20 w-64 h-64 bg-brand-accent/5 rounded-full blur-3xl group-hover:bg-brand-accent/10 transition-colors pointer-events-none" />
|
||||||
|
|
||||||
<div className="w-16 h-16 bg-brand-dark dark:bg-brand-accent rounded-[24px] flex items-center justify-center text-white dark:text-brand-dark text-3xl font-black shadow-2xl shrink-0 group-hover:rotate-6 transition-transform duration-500">
|
<div className="w-16 h-16 bg-brand-dark dark:bg-brand-accent rounded-[24px] flex items-center justify-center text-white dark:text-brand-dark text-3xl font-black shadow-2xl shrink-0 group-hover:rotate-6 transition-transform duration-500">
|
||||||
@@ -762,14 +767,14 @@ export default function TranslatePage() {
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="flex flex-col sm:flex-row gap-3 shrink-0 w-full md:w-auto">
|
<div className="flex flex-col sm:flex-row gap-3 shrink-0 w-full md:w-auto">
|
||||||
<button className="premium-button px-8 py-3.5 text-[9px] uppercase tracking-widest !rounded-xl text-center">
|
<span className="premium-button px-8 py-3.5 text-[9px] uppercase tracking-widest !rounded-xl text-center">
|
||||||
{t('memento.ctaFree')}
|
{t('memento.ctaFree')}
|
||||||
</button>
|
</span>
|
||||||
<button className="px-8 py-3.5 border border-black/5 bg-brand-muted text-brand-dark/40 rounded-xl text-[9px] font-bold uppercase tracking-widest hover:text-brand-dark dark:border-white/5 dark:bg-white/5 dark:text-white/40 dark:hover:text-white hover:bg-brand-muted/70 transition-all text-center">
|
<span className="px-8 py-3.5 border border-black/5 bg-brand-muted text-brand-dark/40 rounded-xl text-[9px] font-bold uppercase tracking-widest hover:text-brand-dark dark:border-white/5 dark:bg-white/5 dark:text-white/40 dark:hover:text-white hover:bg-brand-muted/70 transition-all text-center">
|
||||||
{t('memento.ctaMore')}
|
{t('memento.ctaMore')}
|
||||||
</button>
|
</span>
|
||||||
</div>
|
|
||||||
</div>
|
</div>
|
||||||
|
</a>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Mobile Sticky Action Bar (visible on mobile, hidden on lg) */}
|
{/* Mobile Sticky Action Bar (visible on mobile, hidden on lg) */}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "اكتشف Momento",
|
"memento.title": "اكتشف Memento",
|
||||||
"memento.slogan": "Momento ليس مجرد تطبيق ملاحظات. إنه نظام بيئي ذكي يربط ويحلل ويطور أفكارك في الوقت الفعلي باستخدام 6 وكلاء ذكاء اصطناعي وبحث دلالي متقدم.",
|
"memento.slogan": "Memento ليس مجرد تطبيق ملاحظات. إنه نظام بيئي ذكي يربط ويحلل ويطور أفكارك في الوقت الفعلي باستخدام 6 وكلاء ذكاء اصطناعي وبحث دلالي متقدم.",
|
||||||
"memento.ctaFree": "ابدأ مجاناً",
|
"memento.ctaFree": "ابدأ مجاناً",
|
||||||
"memento.ctaMore": "اعرف المزيد"
|
"memento.ctaMore": "اعرف المزيد",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Entdecken Sie Momento",
|
"memento.title": "Entdecken Sie Memento",
|
||||||
"memento.slogan": "Momento ist mehr als nur eine Notizen-App. Es ist ein intelligentes Ökosystem, das Ihre Ideen in Echtzeit verbindet, analysiert und weiterentwickelt – mit 6 KI-Agenten und semantischer Suche.",
|
"memento.slogan": "Memento ist mehr als nur eine Notizen-App. Es ist ein intelligentes Ökosystem, das Ihre Ideen in Echtzeit verbindet, analysiert und weiterentwickelt – mit 6 KI-Agenten und semantischer Suche.",
|
||||||
"memento.ctaFree": "Kostenlos starten",
|
"memento.ctaFree": "Kostenlos starten",
|
||||||
"memento.ctaMore": "Mehr erfahren"
|
"memento.ctaMore": "Mehr erfahren",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -91,6 +91,8 @@
|
|||||||
"glossaries.upgrade.proLabel": "Pro",
|
"glossaries.upgrade.proLabel": "Pro",
|
||||||
"glossaries.upgrade.upgradeBtn": "Upgrade to Pro",
|
"glossaries.upgrade.upgradeBtn": "Upgrade to Pro",
|
||||||
"glossaries.loading": "Loading...",
|
"glossaries.loading": "Loading...",
|
||||||
|
"glossaries.tabs.glossaries": "Terminology Glossaries",
|
||||||
|
"glossaries.tabs.context": "Context Instructions (AI)",
|
||||||
"glossaries.howItWorks.title": "How these settings are used",
|
"glossaries.howItWorks.title": "How these settings are used",
|
||||||
"glossaries.howItWorks.step1Title": "Configure here",
|
"glossaries.howItWorks.step1Title": "Configure here",
|
||||||
"glossaries.howItWorks.step1Desc": "Write your context instructions or create/import a glossary of terms.",
|
"glossaries.howItWorks.step1Desc": "Write your context instructions or create/import a glossary of terms.",
|
||||||
@@ -114,6 +116,8 @@
|
|||||||
"glossaries.presets.clickHint": "Click a card → glossary created → select it in Translate",
|
"glossaries.presets.clickHint": "Click a card → glossary created → select it in Translate",
|
||||||
"glossaries.presets.creating": "Creating...",
|
"glossaries.presets.creating": "Creating...",
|
||||||
"glossaries.presets.alreadyImported": "Imported",
|
"glossaries.presets.alreadyImported": "Imported",
|
||||||
|
"glossaries.presets.importBtn": "Import",
|
||||||
|
"glossaries.presets.allImported": "All professional templates have been imported.",
|
||||||
"glossaries.presets.it.title": "IT / Software",
|
"glossaries.presets.it.title": "IT / Software",
|
||||||
"glossaries.presets.it.desc": "Development, infrastructure, DevOps",
|
"glossaries.presets.it.desc": "Development, infrastructure, DevOps",
|
||||||
"glossaries.presets.legal.title": "Legal / Contracts",
|
"glossaries.presets.legal.title": "Legal / Contracts",
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Discover Momento",
|
"memento.title": "Discover Memento",
|
||||||
"memento.slogan": "Momento is more than just a notes application. It's an intelligent ecosystem that connects, analyzes, and develops your ideas in real-time using 6 AI agents and cutting-edge semantic search.",
|
"memento.slogan": "Memento is more than just a notes application. It's an intelligent ecosystem that connects, analyzes, and develops your ideas in real-time using 6 AI agents and cutting-edge semantic search.",
|
||||||
"memento.ctaFree": "Start for free",
|
"memento.ctaFree": "Start for free",
|
||||||
"memento.ctaMore": "Learn more"
|
"memento.ctaMore": "Learn more",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Descubre Momento",
|
"memento.title": "Descubre Memento",
|
||||||
"memento.slogan": "Momento no es solo una aplicación de notas. Es un ecosistema inteligente que conecta, analiza y desarrolla tus ideas en tiempo real usando 6 agentes de IA y búsqueda semántica avanzada.",
|
"memento.slogan": "Memento no es solo una aplicación de notas. Es un ecosistema inteligente que conecta, analiza y desarrolla tus ideas en tiempo real usando 6 agentes de IA y búsqueda semántica avanzada.",
|
||||||
"memento.ctaFree": "Empezar gratis",
|
"memento.ctaFree": "Empezar gratis",
|
||||||
"memento.ctaMore": "Saber más"
|
"memento.ctaMore": "Saber más",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Momento را کشف کنید",
|
"memento.title": "Memento را کشف کنید",
|
||||||
"memento.slogan": "Momento فقط یک برنامه یادداشت نیست. یک اکوسیستم هوشمند است که با استفاده از ۶ عامل هوش مصنوعی و جستجوی معنایی پیشرفته، ایدههای شما را در زمان واقعی متصل، تحلیل و توسعه میدهد.",
|
"memento.slogan": "Memento فقط یک برنامه یادداشت نیست. یک اکوسیستم هوشمند است که با استفاده از ۶ عامل هوش مصنوعی و جستجوی معنایی پیشرفته، ایدههای شما را در زمان واقعی متصل، تحلیل و توسعه میدهد.",
|
||||||
"memento.ctaFree": "رایگان شروع کنید",
|
"memento.ctaFree": "رایگان شروع کنید",
|
||||||
"memento.ctaMore": "بیشتر بدانید"
|
"memento.ctaMore": "بیشتر بدانید",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -91,6 +91,8 @@
|
|||||||
"glossaries.upgrade.proLabel": "Pro",
|
"glossaries.upgrade.proLabel": "Pro",
|
||||||
"glossaries.upgrade.upgradeBtn": "Passer à Pro",
|
"glossaries.upgrade.upgradeBtn": "Passer à Pro",
|
||||||
"glossaries.loading": "Chargement...",
|
"glossaries.loading": "Chargement...",
|
||||||
|
"glossaries.tabs.glossaries": "Glossaires terminologiques",
|
||||||
|
"glossaries.tabs.context": "Consignes de contexte (IA)",
|
||||||
"glossaries.howItWorks.title": "Comment ces paramètres sont utilisés",
|
"glossaries.howItWorks.title": "Comment ces paramètres sont utilisés",
|
||||||
"glossaries.howItWorks.step1Title": "Configurez ici",
|
"glossaries.howItWorks.step1Title": "Configurez ici",
|
||||||
"glossaries.howItWorks.step1Desc": "Rédigez vos instructions de contexte ou créez/importez un glossaire de termes.",
|
"glossaries.howItWorks.step1Desc": "Rédigez vos instructions de contexte ou créez/importez un glossaire de termes.",
|
||||||
@@ -114,6 +116,8 @@
|
|||||||
"glossaries.presets.clickHint": "Cliquez sur une carte → glossaire créé → sélectionnez-le dans Traduire",
|
"glossaries.presets.clickHint": "Cliquez sur une carte → glossaire créé → sélectionnez-le dans Traduire",
|
||||||
"glossaries.presets.creating": "Création…",
|
"glossaries.presets.creating": "Création…",
|
||||||
"glossaries.presets.alreadyImported": "Déjà importé",
|
"glossaries.presets.alreadyImported": "Déjà importé",
|
||||||
|
"glossaries.presets.importBtn": "Importer",
|
||||||
|
"glossaries.presets.allImported": "Tous les modèles professionnels ont été importés.",
|
||||||
"glossaries.presets.it.title": "IT / Logiciel",
|
"glossaries.presets.it.title": "IT / Logiciel",
|
||||||
"glossaries.presets.it.desc": "Développement, infrastructure, DevOps",
|
"glossaries.presets.it.desc": "Développement, infrastructure, DevOps",
|
||||||
"glossaries.presets.legal.title": "Juridique / Contrats",
|
"glossaries.presets.legal.title": "Juridique / Contrats",
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Découvrez Momento",
|
"memento.title": "Découvrez Memento",
|
||||||
"memento.slogan": "Momento n'est pas qu'une simple application de notes. C'est un écosystème intelligent qui connecte, analyse et développe vos idées en temps réel grâce à 6 types d'agents IA et une recherche sémantique de pointe.",
|
"memento.slogan": "Memento n'est pas qu'une simple application de notes. C'est un écosystème intelligent qui connecte, analyse et développe vos idées en temps réel grâce à 6 types d'agents IA et une recherche sémantique de pointe.",
|
||||||
"memento.ctaFree": "Commencer gratuitement",
|
"memento.ctaFree": "Commencer gratuitement",
|
||||||
"memento.ctaMore": "Voir plus"
|
"memento.ctaMore": "Voir plus",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Scopri Momento",
|
"memento.title": "Scopri Memento",
|
||||||
"memento.slogan": "Momento non è solo un'app di note. È un ecosistema intelligente che connette, analizza e sviluppa le tue idee in tempo reale usando 6 agenti IA e ricerca semantica avanzata.",
|
"memento.slogan": "Memento non è solo un'app di note. È un ecosistema intelligente che connette, analizza e sviluppa le tue idee in tempo reale usando 6 agenti IA e ricerca semantica avanzata.",
|
||||||
"memento.ctaFree": "Inizia gratis",
|
"memento.ctaFree": "Inizia gratis",
|
||||||
"memento.ctaMore": "Scopri di più"
|
"memento.ctaMore": "Scopri di più",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Momentoを発見",
|
"memento.title": "Mementoを発見",
|
||||||
"memento.slogan": "Momentoは単なるメモアプリではありません。6つのAIエージェントと高度なセマンティック検索を使用して、アイデアをリアルタイムで接続、分析、発展させるインテリジェントなエコシステムです。",
|
"memento.slogan": "Mementoは単なるメモアプリではありません。6つのAIエージェントと高度なセマンティック検索を使用して、アイデアをリアルタイムで接続、分析、発展させるインテリジェントなエコシステムです。",
|
||||||
"memento.ctaFree": "無料で始める",
|
"memento.ctaFree": "無料で始める",
|
||||||
"memento.ctaMore": "詳しく見る"
|
"memento.ctaMore": "詳しく見る",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Momento 발견하기",
|
"memento.title": "Memento 발견하기",
|
||||||
"memento.slogan": "Momento는 단순한 메모 앱이 아닙니다. 6개의 AI 에이전트와 최첨단 시맨틱 검색을 사용하여 아이디어를 실시간으로 연결, 분석, 발전시키는 지능형 생태계입니다.",
|
"memento.slogan": "Memento는 단순한 메모 앱이 아닙니다. 6개의 AI 에이전트와 최첨단 시맨틱 검색을 사용하여 아이디어를 실시간으로 연결, 분석, 발전시키는 지능형 생태계입니다.",
|
||||||
"memento.ctaFree": "무료로 시작",
|
"memento.ctaFree": "무료로 시작",
|
||||||
"memento.ctaMore": "자세히 보기"
|
"memento.ctaMore": "자세히 보기",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Ontdek Momento",
|
"memento.title": "Ontdek Memento",
|
||||||
"memento.slogan": "Momento is meer dan alleen een notitie-app. Het is een intelligent ecosysteem dat uw ideeën in realtime verbindt, analyseert en verder ontwikkelt met 6 AI-agents en geavanceerde semantische zoekfuncties.",
|
"memento.slogan": "Memento is meer dan alleen een notitie-app. Het is een intelligent ecosysteem dat uw ideeën in realtime verbindt, analyseert en verder ontwikkelt met 6 AI-agents en geavanceerde semantische zoekfuncties.",
|
||||||
"memento.ctaFree": "Gratis beginnen",
|
"memento.ctaFree": "Gratis beginnen",
|
||||||
"memento.ctaMore": "Meer informatie"
|
"memento.ctaMore": "Meer informatie",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Descubra o Momento",
|
"memento.title": "Descubra o Memento",
|
||||||
"memento.slogan": "O Momento é mais do que uma aplicação de notas. É um ecossistema inteligente que conecta, analisa e desenvolve suas ideias em tempo real usando 6 agentes de IA e busca semântica avançada.",
|
"memento.slogan": "O Memento é mais do que uma aplicação de notas. É um ecossistema inteligente que conecta, analisa e desenvolve suas ideias em tempo real usando 6 agentes de IA e busca semântica avançada.",
|
||||||
"memento.ctaFree": "Começar grátis",
|
"memento.ctaFree": "Começar grátis",
|
||||||
"memento.ctaMore": "Saiba mais"
|
"memento.ctaMore": "Saiba mais",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "Откройте Momento",
|
"memento.title": "Откройте Memento",
|
||||||
"memento.slogan": "Momento — это больше, чем приложение для заметок. Это интеллектуальная экосистема, которая связывает, анализирует и развивает ваши идеи в реальном времени с помощью 6 ИИ-агентов и продвинутого семантического поиска.",
|
"memento.slogan": "Memento — это больше, чем приложение для заметок. Это интеллектуальная экосистема, которая связывает, анализирует и развивает ваши идеи в реальном времени с помощью 6 ИИ-агентов и продвинутого семантического поиска.",
|
||||||
"memento.ctaFree": "Начать бесплатно",
|
"memento.ctaFree": "Начать бесплатно",
|
||||||
"memento.ctaMore": "Узнать больше"
|
"memento.ctaMore": "Узнать больше",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
{
|
{
|
||||||
"memento.title": "探索 Momento",
|
"memento.title": "探索 Memento",
|
||||||
"memento.slogan": "Momento 不仅仅是一个笔记应用。它是一个智能生态系统,利用 6 个 AI 代理和先进的语义搜索,实时连接、分析和发展您的想法。",
|
"memento.slogan": "Memento 不仅仅是一个笔记应用。它是一个智能生态系统,利用 6 个 AI 代理和先进的语义搜索,实时连接、分析和发展您的想法。",
|
||||||
"memento.ctaFree": "免费开始",
|
"memento.ctaFree": "免费开始",
|
||||||
"memento.ctaMore": "了解更多"
|
"memento.ctaMore": "了解更多",
|
||||||
|
"memento.url": "https://memento-note.com/"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -42,6 +42,72 @@ file_size_bytes = Histogram(
|
|||||||
buckets=(100_000, 500_000, 1_000_000, 5_000_000, 10_000_000, 25_000_000, 50_000_000),
|
buckets=(100_000, 500_000, 1_000_000, 5_000_000, 10_000_000, 25_000_000, 50_000_000),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# ---- Quality pipeline metrics (L0 + L1) ----
|
||||||
|
|
||||||
|
quality_l0_checks_total = Counter(
|
||||||
|
"quality_l0_checks_total",
|
||||||
|
"Total L0 (script+length+pattern) quality checks",
|
||||||
|
["result", "file_type"], # result: pass | fail | error
|
||||||
|
)
|
||||||
|
|
||||||
|
quality_l1_judge_total = Counter(
|
||||||
|
"quality_l1_judge_total",
|
||||||
|
"Total L1 (LLM judge) verdicts",
|
||||||
|
["verdict", "model"], # verdict: pass | fail | skip | error
|
||||||
|
)
|
||||||
|
|
||||||
|
quality_l1_judge_duration_seconds = Histogram(
|
||||||
|
"quality_l1_judge_duration_seconds",
|
||||||
|
"L1 LLM judge call duration in seconds",
|
||||||
|
["model"],
|
||||||
|
buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30),
|
||||||
|
)
|
||||||
|
|
||||||
|
quality_l1_judge_cost_usd = Histogram(
|
||||||
|
"quality_l1_judge_cost_usd",
|
||||||
|
"L1 LLM judge estimated cost in USD",
|
||||||
|
["model"],
|
||||||
|
buckets=(0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1),
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- L2 Pro premium judge (Track A4) ----
|
||||||
|
|
||||||
|
quality_l2_judge_total = Counter(
|
||||||
|
"quality_l2_judge_total",
|
||||||
|
"Total L2 (Pro premium judge, 8-dim) verdicts",
|
||||||
|
["verdict", "model", "tier"], # verdict: pass | fail | skip | error
|
||||||
|
)
|
||||||
|
|
||||||
|
quality_l2_judge_duration_seconds = Histogram(
|
||||||
|
"quality_l2_judge_duration_seconds",
|
||||||
|
"L2 Pro judge call duration in seconds",
|
||||||
|
["model"],
|
||||||
|
buckets=(0.5, 1, 2, 5, 10, 20, 30, 60),
|
||||||
|
)
|
||||||
|
|
||||||
|
quality_l2_judge_cost_usd = Histogram(
|
||||||
|
"quality_l2_judge_cost_usd",
|
||||||
|
"L2 Pro judge estimated cost in USD",
|
||||||
|
["model"],
|
||||||
|
buckets=(0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0),
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- Retry metrics ----
|
||||||
|
|
||||||
|
translation_retry_total = Counter(
|
||||||
|
"translation_retry_total",
|
||||||
|
"Total translation retries triggered by quality issues",
|
||||||
|
["reason", "tier"], # reason: l0_fail | l1_fail | format_loss
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- Format preservation metrics ----
|
||||||
|
|
||||||
|
format_elements_lost_total = Counter(
|
||||||
|
"format_elements_lost_total",
|
||||||
|
"Format elements (hyperlinks, footnotes, charts, ...) lost during translation",
|
||||||
|
["format", "element_type"], # format: docx | xlsx | pptx | pdf
|
||||||
|
)
|
||||||
|
|
||||||
# Paths to skip from metrics (noisy health checks)
|
# Paths to skip from metrics (noisy health checks)
|
||||||
_SKIP_PATHS = {"/health", "/ready", "/metrics", "/favicon.ico"}
|
_SKIP_PATHS = {"/health", "/ready", "/metrics", "/favicon.ico"}
|
||||||
|
|
||||||
@@ -55,6 +121,88 @@ def record_file_size(file_type: str, size_bytes: int):
|
|||||||
file_size_bytes.labels(file_type=file_type).observe(size_bytes)
|
file_size_bytes.labels(file_type=file_type).observe(size_bytes)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- Quality helpers ----
|
||||||
|
|
||||||
|
def record_l0_result(passed: bool, file_type: str = "unknown"):
|
||||||
|
"""Record an L0 quality check outcome.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
passed: True if the L0 check passed.
|
||||||
|
file_type: docx | xlsx | pptx | pdf | unknown
|
||||||
|
"""
|
||||||
|
result = "pass" if passed else "fail"
|
||||||
|
quality_l0_checks_total.labels(result=result, file_type=file_type).inc()
|
||||||
|
|
||||||
|
|
||||||
|
def record_l0_error(file_type: str = "unknown"):
|
||||||
|
"""Record an L0 internal error (so we can spot broken checks)."""
|
||||||
|
quality_l0_checks_total.labels(result="error", file_type=file_type).inc()
|
||||||
|
|
||||||
|
|
||||||
|
def record_l1_verdict(
|
||||||
|
verdict: str,
|
||||||
|
model: str = "unknown",
|
||||||
|
duration_seconds: float = None,
|
||||||
|
cost_usd: float = None,
|
||||||
|
):
|
||||||
|
"""Record an L1 judge verdict.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
verdict: pass | fail | skip | error
|
||||||
|
model: model name (e.g. deepseek-chat)
|
||||||
|
duration_seconds: optional, observed in histogram
|
||||||
|
cost_usd: optional, observed in histogram
|
||||||
|
"""
|
||||||
|
quality_l1_judge_total.labels(verdict=verdict, model=model).inc()
|
||||||
|
if duration_seconds is not None:
|
||||||
|
quality_l1_judge_duration_seconds.labels(model=model).observe(duration_seconds)
|
||||||
|
if cost_usd is not None:
|
||||||
|
quality_l1_judge_cost_usd.labels(model=model).observe(cost_usd)
|
||||||
|
|
||||||
|
|
||||||
|
def record_l2_verdict(
|
||||||
|
verdict: str,
|
||||||
|
model: str = "unknown",
|
||||||
|
tier: str = "pro",
|
||||||
|
duration_seconds: float = None,
|
||||||
|
cost_usd: float = None,
|
||||||
|
):
|
||||||
|
"""Record an L2 Pro judge verdict.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
verdict: pass | fail | skip | error
|
||||||
|
model: model name (e.g. gpt-4o)
|
||||||
|
tier: pro | business | enterprise
|
||||||
|
duration_seconds: optional, observed in histogram
|
||||||
|
cost_usd: optional, observed in histogram
|
||||||
|
"""
|
||||||
|
quality_l2_judge_total.labels(verdict=verdict, model=model, tier=tier).inc()
|
||||||
|
if duration_seconds is not None:
|
||||||
|
quality_l2_judge_duration_seconds.labels(model=model).observe(duration_seconds)
|
||||||
|
if cost_usd is not None:
|
||||||
|
quality_l2_judge_cost_usd.labels(model=model).observe(cost_usd)
|
||||||
|
|
||||||
|
|
||||||
|
def record_translation_retry(reason: str, tier: str = "free"):
|
||||||
|
"""Record a translation retry triggered by a quality issue.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
reason: l0_fail | l1_fail | format_loss | user_request
|
||||||
|
tier: free | pro | enterprise
|
||||||
|
"""
|
||||||
|
translation_retry_total.labels(reason=reason, tier=tier).inc()
|
||||||
|
|
||||||
|
|
||||||
|
def record_format_loss(format: str, element_type: str):
|
||||||
|
"""Record a format element that was lost during translation.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
format: docx | xlsx | pptx | pdf
|
||||||
|
element_type: hyperlink | footnote | chart | diagram | comment | image
|
||||||
|
"""
|
||||||
|
format_elements_lost_total.labels(format=format, element_type=element_type).inc()
|
||||||
|
|
||||||
|
|
||||||
class PrometheusMiddleware(BaseHTTPMiddleware):
|
class PrometheusMiddleware(BaseHTTPMiddleware):
|
||||||
async def dispatch(self, request: Request, call_next):
|
async def dispatch(self, request: Request, call_next):
|
||||||
if request.url.path in _SKIP_PATHS:
|
if request.url.path in _SKIP_PATHS:
|
||||||
|
|||||||
@@ -1477,10 +1477,14 @@ async def get_pricing_config(admin_id: str = Depends(require_admin)):
|
|||||||
}
|
}
|
||||||
|
|
||||||
stripe_configured = bool(os.getenv("STRIPE_SECRET_KEY", "").strip().startswith("sk_"))
|
stripe_configured = bool(os.getenv("STRIPE_SECRET_KEY", "").strip().startswith("sk_"))
|
||||||
|
mode = pricing_cfg.stripe_mode()
|
||||||
|
is_test_in_prod = pricing_cfg.is_test_mode_in_production()
|
||||||
return JSONResponse(status_code=200, content={
|
return JSONResponse(status_code=200, content={
|
||||||
"data": result,
|
"data": result,
|
||||||
"stripe": {
|
"stripe": {
|
||||||
"configured": stripe_configured,
|
"configured": stripe_configured,
|
||||||
|
"mode": mode, # "live" | "test" | "unknown"
|
||||||
|
"is_test_mode_in_production": is_test_in_prod,
|
||||||
"has_secret_key": bool(os.getenv("STRIPE_SECRET_KEY", "").strip()),
|
"has_secret_key": bool(os.getenv("STRIPE_SECRET_KEY", "").strip()),
|
||||||
"has_publishable_key": bool(os.getenv("STRIPE_PUBLISHABLE_KEY", "").strip()),
|
"has_publishable_key": bool(os.getenv("STRIPE_PUBLISHABLE_KEY", "").strip()),
|
||||||
"has_webhook_secret": bool(os.getenv("STRIPE_WEBHOOK_SECRET", "").strip()),
|
"has_webhook_secret": bool(os.getenv("STRIPE_WEBHOOK_SECRET", "").strip()),
|
||||||
@@ -1489,6 +1493,7 @@ async def get_pricing_config(admin_id: str = Depends(require_admin)):
|
|||||||
"meta": {
|
"meta": {
|
||||||
"annual_discount_percent": pricing_cfg.ANNUAL_DISCOUNT_PERCENT,
|
"annual_discount_percent": pricing_cfg.ANNUAL_DISCOUNT_PERCENT,
|
||||||
"yearly_discount_factor": pricing_cfg.YEARLY_DISCOUNT_FACTOR,
|
"yearly_discount_factor": pricing_cfg.YEARLY_DISCOUNT_FACTOR,
|
||||||
|
"env": (os.getenv("ENV", os.getenv("ENVIRONMENT", "development")) or "development").lower(),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -1591,7 +1596,7 @@ async def update_pricing_config(
|
|||||||
|
|
||||||
|
|
||||||
@router.post("/pricing/setup-stripe")
|
@router.post("/pricing/setup-stripe")
|
||||||
async def setup_stripe_products(admin_id: str = Depends(require_admin)):
|
async def setup_stripe_products(request: Request, admin_id: str = Depends(require_admin)):
|
||||||
"""Auto-create Stripe products & prices from backend, then save IDs to .env."""
|
"""Auto-create Stripe products & prices from backend, then save IDs to .env."""
|
||||||
secret_key = os.getenv("STRIPE_SECRET_KEY", "").strip()
|
secret_key = os.getenv("STRIPE_SECRET_KEY", "").strip()
|
||||||
if not secret_key or not secret_key.startswith("sk_"):
|
if not secret_key or not secret_key.startswith("sk_"):
|
||||||
@@ -1600,6 +1605,27 @@ async def setup_stripe_products(admin_id: str = Depends(require_admin)):
|
|||||||
"message": "STRIPE_SECRET_KEY manquante ou invalide. Configurez-la d'abord.",
|
"message": "STRIPE_SECRET_KEY manquante ou invalide. Configurez-la d'abord.",
|
||||||
})
|
})
|
||||||
|
|
||||||
|
# Block test-mode auto-setup in production unless the admin forces it.
|
||||||
|
# Otherwise we'd create TEST products in the LIVE account context,
|
||||||
|
# which silently fails for real users and corrupts the prod pricing.
|
||||||
|
body: dict = {}
|
||||||
|
try:
|
||||||
|
body = await request.json()
|
||||||
|
except Exception:
|
||||||
|
body = {}
|
||||||
|
force = bool(body.get("force", False))
|
||||||
|
if pricing_cfg.is_test_mode_in_production() and not force:
|
||||||
|
return JSONResponse(status_code=409, content={
|
||||||
|
"error": "TEST_MODE_IN_PRODUCTION",
|
||||||
|
"message": (
|
||||||
|
"Vous êtes en production (ENV=production) mais STRIPE_SECRET_KEY "
|
||||||
|
"est une clé de test (sk_test_…). Créez des produits TEST dans un "
|
||||||
|
"compte TEST ou basculez sur une clé sk_live_… . Pour forcer, "
|
||||||
|
"relancez avec {\"force\": true} (à vos risques)."
|
||||||
|
),
|
||||||
|
"mode": pricing_cfg.stripe_mode(),
|
||||||
|
})
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import stripe as _stripe
|
import stripe as _stripe
|
||||||
_stripe.api_key = secret_key
|
_stripe.api_key = secret_key
|
||||||
@@ -1689,3 +1715,160 @@ async def setup_stripe_products(admin_id: str = Depends(require_admin)):
|
|||||||
"env_updated": list(env_updates.keys()),
|
"env_updated": list(env_updates.keys()),
|
||||||
"meta": {},
|
"meta": {},
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/pricing/setup-webhook")
|
||||||
|
async def setup_stripe_webhook(request: Request, admin_id: str = Depends(require_admin)):
|
||||||
|
"""Auto-create the Stripe webhook endpoint and save its signing secret.
|
||||||
|
|
||||||
|
The webhook URL is derived from the request's host headers
|
||||||
|
(X-Forwarded-Proto + Host). In production, the URL must be HTTPS
|
||||||
|
and publicly reachable from Stripe.
|
||||||
|
|
||||||
|
Body (optional):
|
||||||
|
- ``url_override`` (str): explicit webhook URL if auto-detection
|
||||||
|
doesn't match your deployment (e.g. behind a custom reverse proxy)
|
||||||
|
- ``force`` (bool): if true, allow auto-setup in test mode even
|
||||||
|
when running in production (useful for staging)
|
||||||
|
"""
|
||||||
|
secret_key = os.getenv("STRIPE_SECRET_KEY", "").strip()
|
||||||
|
if not secret_key or not secret_key.startswith("sk_"):
|
||||||
|
return JSONResponse(status_code=400, content={
|
||||||
|
"error": "STRIPE_NOT_CONFIGURED",
|
||||||
|
"message": "STRIPE_SECRET_KEY manquante ou invalide. Configurez-la d'abord.",
|
||||||
|
})
|
||||||
|
|
||||||
|
mode = pricing_cfg.stripe_mode()
|
||||||
|
is_test_in_prod = pricing_cfg.is_test_mode_in_production()
|
||||||
|
|
||||||
|
# Read optional body
|
||||||
|
body: dict = {}
|
||||||
|
try:
|
||||||
|
body = await request.json()
|
||||||
|
except Exception:
|
||||||
|
body = {}
|
||||||
|
|
||||||
|
force = bool(body.get("force", False))
|
||||||
|
url_override = (body.get("url_override") or "").strip()
|
||||||
|
|
||||||
|
if is_test_in_prod and not force:
|
||||||
|
return JSONResponse(status_code=409, content={
|
||||||
|
"error": "TEST_MODE_IN_PRODUCTION",
|
||||||
|
"message": (
|
||||||
|
"Vous êtes en production (ENV=production) mais STRIPE_SECRET_KEY "
|
||||||
|
"est une clé de test (sk_test_…). Basculez sur sk_live_… ou "
|
||||||
|
"relancez avec {\"force\": true} pour créer un webhook test "
|
||||||
|
"explicitement (à des fins de staging)."
|
||||||
|
),
|
||||||
|
"mode": mode,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Build the webhook URL from the request (host + forwarded proto)
|
||||||
|
if url_override:
|
||||||
|
webhook_url = url_override
|
||||||
|
else:
|
||||||
|
xfp = request.headers.get("x-forwarded-proto", "")
|
||||||
|
scheme = xfp.split(",")[0].strip() if xfp else (request.url.scheme or "https")
|
||||||
|
host = (
|
||||||
|
request.headers.get("x-forwarded-host")
|
||||||
|
or request.headers.get("host")
|
||||||
|
or request.url.netloc
|
||||||
|
)
|
||||||
|
webhook_url = f"{scheme}://{host}/api/v1/webhook/stripe"
|
||||||
|
|
||||||
|
# Basic sanity check
|
||||||
|
if not webhook_url.startswith(("http://", "https://")):
|
||||||
|
return JSONResponse(status_code=400, content={
|
||||||
|
"error": "INVALID_URL",
|
||||||
|
"message": f"URL webhook invalide : {webhook_url!r}",
|
||||||
|
})
|
||||||
|
if mode == "live" and webhook_url.startswith("http://"):
|
||||||
|
return JSONResponse(status_code=400, content={
|
||||||
|
"error": "INSECURE_URL_FOR_LIVE_MODE",
|
||||||
|
"message": (
|
||||||
|
"En mode Live, l'URL du webhook doit être HTTPS. "
|
||||||
|
f"URL actuelle : {webhook_url}. Utilisez url_override=https://…"
|
||||||
|
),
|
||||||
|
})
|
||||||
|
|
||||||
|
try:
|
||||||
|
import stripe as _stripe
|
||||||
|
_stripe.api_key = secret_key
|
||||||
|
|
||||||
|
# The events the backend actually handles (see payment_service.handle_webhook).
|
||||||
|
# If we send more events than the backend expects, Stripe will retry on
|
||||||
|
# signature mismatches and waste API quota. Keep this list aligned with
|
||||||
|
# the handler.
|
||||||
|
events = [
|
||||||
|
"checkout.session.completed",
|
||||||
|
"customer.subscription.created",
|
||||||
|
"customer.subscription.updated",
|
||||||
|
"customer.subscription.deleted",
|
||||||
|
"invoice.paid",
|
||||||
|
"invoice.payment_failed",
|
||||||
|
]
|
||||||
|
|
||||||
|
# Avoid duplicates: list existing endpoints and reuse one that already
|
||||||
|
# points to the same URL.
|
||||||
|
existing_list = _stripe.WebhookEndpoint.list(limit=20)
|
||||||
|
existing = next(
|
||||||
|
(e for e in existing_list.data if e.url == webhook_url),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if existing:
|
||||||
|
# Update the existing endpoint to make sure its events are in sync
|
||||||
|
webhook = _stripe.WebhookEndpoint.update(
|
||||||
|
existing.id,
|
||||||
|
enabled_events=events,
|
||||||
|
)
|
||||||
|
created = False
|
||||||
|
else:
|
||||||
|
webhook = _stripe.WebhookEndpoint.create(
|
||||||
|
url=webhook_url,
|
||||||
|
enabled_events=events,
|
||||||
|
description="Office Translator — auto-created by admin panel",
|
||||||
|
)
|
||||||
|
created = True
|
||||||
|
|
||||||
|
# ``webhook.secret`` is only returned at creation time by Stripe. On
|
||||||
|
# update it is None, so we re-use the existing STRIPE_WEBHOOK_SECRET
|
||||||
|
# if the endpoint already existed.
|
||||||
|
new_secret = webhook.get("secret") or os.getenv("STRIPE_WEBHOOK_SECRET", "").strip()
|
||||||
|
|
||||||
|
env_updates = {}
|
||||||
|
if new_secret:
|
||||||
|
pricing_cfg.validate_stripe_webhook_secret(new_secret)
|
||||||
|
env_updates["STRIPE_WEBHOOK_SECRET"] = new_secret
|
||||||
|
if env_updates:
|
||||||
|
_update_env_file(env_updates)
|
||||||
|
pricing_cfg.apply_runtime_config_after_admin_write()
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"admin_stripe_webhook_setup by {admin_id}: url={webhook_url} "
|
||||||
|
f"created={created} mode={mode}"
|
||||||
|
)
|
||||||
|
return JSONResponse(status_code=200, content={
|
||||||
|
"data": {
|
||||||
|
"id": webhook.id,
|
||||||
|
"url": webhook.url,
|
||||||
|
"enabled_events": webhook.enabled_events,
|
||||||
|
"status": webhook.status,
|
||||||
|
"created": created,
|
||||||
|
"secret_saved": bool(new_secret),
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"mode": mode,
|
||||||
|
"env": (os.getenv("ENV", os.getenv("ENVIRONMENT", "development")) or "development").lower(),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
except ImportError:
|
||||||
|
return JSONResponse(status_code=500, content={
|
||||||
|
"error": "STRIPE_NOT_INSTALLED",
|
||||||
|
"message": "Le module stripe n'est pas installé sur le serveur.",
|
||||||
|
})
|
||||||
|
except Exception as e:
|
||||||
|
logger.exception("admin_stripe_webhook_setup_failed")
|
||||||
|
return JSONResponse(status_code=500, content={
|
||||||
|
"error": "STRIPE_API_ERROR",
|
||||||
|
"message": str(e),
|
||||||
|
})
|
||||||
|
|||||||
@@ -1354,6 +1354,150 @@ async def _run_translation_job(
|
|||||||
f"{changed}/{attempted} ({ratio:.1%})"
|
f"{changed}/{attempted} ({ratio:.1%})"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Quality L0 layer (Track A1 — observability only)
|
||||||
|
# Extract a small sample of text from the output file and run the
|
||||||
|
# L0 quality checks. NEVER blocks the job, NEVER modifies the file.
|
||||||
|
# Enabled by feature flag QUALITY_L0_ENABLED (default: false).
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
quality_samples: list = [] # captured here for L1 to reuse
|
||||||
|
l0_failed_indices: set = set() # captured for L1 sampling
|
||||||
|
if getattr(config, "QUALITY_L0_ENABLED", False):
|
||||||
|
try:
|
||||||
|
from services.quality import run_l0_check, extract_sample
|
||||||
|
quality_samples = extract_sample(
|
||||||
|
output_path,
|
||||||
|
file_extension,
|
||||||
|
max_samples=getattr(config, "QUALITY_L0_SAMPLE_SIZE", 20),
|
||||||
|
)
|
||||||
|
translated_chunks = [s["translated"] for s in quality_samples]
|
||||||
|
l0_result = run_l0_check(
|
||||||
|
source_chunks=[""] * len(translated_chunks), # L0 doesn't need source
|
||||||
|
translated_chunks=translated_chunks,
|
||||||
|
target_lang=target_lang,
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
)
|
||||||
|
# Build the set of indices that L0 flagged as bad
|
||||||
|
if not l0_result.passed and l0_result.samples:
|
||||||
|
l0_failed_indices = {s["index"] for s in l0_result.samples}
|
||||||
|
except Exception as l0_err:
|
||||||
|
# Quality L0 must NEVER break a job. Log and continue.
|
||||||
|
logger.warning(
|
||||||
|
f"Job {job_id}: quality L0 layer failed: {l0_err}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Quality L1 layer (Track A3 — API-based LLM judge, observability first)
|
||||||
|
# Sends a small sample of chunks to a cheap LLM (deepseek-chat by
|
||||||
|
# default) and logs a binary verdict (pass/fail). Log-only by
|
||||||
|
# default — the verdict NEVER blocks a job, NEVER triggers a retry.
|
||||||
|
# After 2 weeks of observation, set QUALITY_L1_LOG_ONLY=false to
|
||||||
|
# enable auto-retry.
|
||||||
|
# Cost: ~$0.0003/job (deepseek) or ~$0.001/job (gpt-4o-mini).
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
if getattr(config, "QUALITY_L1_ENABLED", False) and quality_samples:
|
||||||
|
try:
|
||||||
|
from services.quality import run_l1_check
|
||||||
|
translated_chunks_for_l1 = [s["translated"] for s in quality_samples]
|
||||||
|
# The samples are extracted from the OUTPUT, not the source —
|
||||||
|
# we don't have the source here. L1 still works because the
|
||||||
|
# judge can spot language confusion, gibberish, repetition,
|
||||||
|
# and prompt leaks without the source. (Source IS used for
|
||||||
|
# the "accurate" check, so the verdict on that dimension is
|
||||||
|
# conservative when no source is available — we expect
|
||||||
|
# the judge to be honest about what it can verify.)
|
||||||
|
l1_result = await run_l1_check(
|
||||||
|
source_chunks=[""] * len(translated_chunks_for_l1),
|
||||||
|
translated_chunks=translated_chunks_for_l1,
|
||||||
|
target_lang=target_lang,
|
||||||
|
l0_failed_indices=l0_failed_indices,
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
max_samples=getattr(config, "QUALITY_L1_SAMPLE_SIZE", 5),
|
||||||
|
min_chunks=getattr(config, "QUALITY_L1_MIN_CHUNKS", 10),
|
||||||
|
log_only=getattr(config, "QUALITY_L1_LOG_ONLY", True),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When not in log-only mode, an L1 fail could trigger a
|
||||||
|
# retry. Record the intent (best-effort, never breaks job).
|
||||||
|
if (
|
||||||
|
l1_result is not None
|
||||||
|
and l1_result.verdict == "fail"
|
||||||
|
and not getattr(config, "QUALITY_L1_LOG_ONLY", True)
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
from middleware.metrics import record_translation_retry
|
||||||
|
record_translation_retry(
|
||||||
|
reason="l1_fail",
|
||||||
|
tier=_tier_for_quota(
|
||||||
|
current_user.plan if current_user else None
|
||||||
|
) if current_user else "free",
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
except Exception as l1_err:
|
||||||
|
# L1 must NEVER break a job. Log and continue.
|
||||||
|
logger.warning(
|
||||||
|
f"Job {job_id}: quality L1 layer failed: {l1_err}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Record L0 fail as a "potential retry" (also best-effort).
|
||||||
|
if l0_failed_indices and not getattr(config, "QUALITY_L1_LOG_ONLY", True):
|
||||||
|
try:
|
||||||
|
from middleware.metrics import record_translation_retry
|
||||||
|
record_translation_retry(
|
||||||
|
reason="l0_fail",
|
||||||
|
tier=_tier_for_quota(
|
||||||
|
current_user.plan if current_user else None
|
||||||
|
) if current_user else "free",
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Quality L2 layer (Track A4 — Pro premium judge)
|
||||||
|
# Stronger LLM (gpt-4o default), 8 dimensions, 15 samples.
|
||||||
|
# Gated to Pro+ plans (configurable via QUALITY_L2_TIER_GATE).
|
||||||
|
# Default OFF everywhere — observation first.
|
||||||
|
# Cost: ~$0.005–$0.02/job (gpt-4o), ~$0.001/job (gpt-4o-mini).
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
if getattr(config, "QUALITY_L2_ENABLED", False) and quality_samples:
|
||||||
|
try:
|
||||||
|
from services.quality import run_l2_check
|
||||||
|
|
||||||
|
# Tier gate: Pro+ plans only (unless gate is disabled)
|
||||||
|
user_tier = (
|
||||||
|
_tier_for_quota(current_user.plan) if current_user else "free"
|
||||||
|
)
|
||||||
|
tier_gate_on = getattr(config, "QUALITY_L2_TIER_GATE", True)
|
||||||
|
if not tier_gate_on or user_tier in ("pro", "business", "enterprise"):
|
||||||
|
translated_chunks_for_l2 = [s["translated"] for s in quality_samples]
|
||||||
|
l2_result = await run_l2_check(
|
||||||
|
source_chunks=[""] * len(translated_chunks_for_l2),
|
||||||
|
translated_chunks=translated_chunks_for_l2,
|
||||||
|
target_lang=target_lang,
|
||||||
|
l0_failed_indices=l0_failed_indices,
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
max_samples=getattr(config, "QUALITY_L2_SAMPLE_SIZE", 15),
|
||||||
|
min_chunks=getattr(config, "QUALITY_L2_MIN_CHUNKS", 20),
|
||||||
|
log_only=getattr(config, "QUALITY_L2_LOG_ONLY", True),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Free/Starter user — skip L2 silently (gated)
|
||||||
|
logger.info(
|
||||||
|
"quality_l2_check_skipped",
|
||||||
|
job_id=job_id,
|
||||||
|
reason="tier_gated",
|
||||||
|
tier=user_tier,
|
||||||
|
)
|
||||||
|
except Exception as l2_err:
|
||||||
|
# L2 must NEVER break a job. Log and continue.
|
||||||
|
logger.warning(
|
||||||
|
f"Job {job_id}: quality L2 layer failed: {l2_err}"
|
||||||
|
)
|
||||||
|
|
||||||
if user_id:
|
if user_id:
|
||||||
# Determine cost factor based on selected provider and model
|
# Determine cost factor based on selected provider and model
|
||||||
cost_factor = 1
|
cost_factor = 1
|
||||||
|
|||||||
BIN
sample_files/test_corpus/test_excel.xlsx
Normal file
BIN
sample_files/test_corpus/test_excel.xlsx
Normal file
Binary file not shown.
BIN
sample_files/test_corpus/test_pdf.pdf
Normal file
BIN
sample_files/test_corpus/test_pdf.pdf
Normal file
Binary file not shown.
BIN
sample_files/test_corpus/test_pdf_translated.pdf
Normal file
BIN
sample_files/test_corpus/test_pdf_translated.pdf
Normal file
Binary file not shown.
BIN
sample_files/test_corpus/test_pptx.pptx
Normal file
BIN
sample_files/test_corpus/test_pptx.pptx
Normal file
Binary file not shown.
BIN
sample_files/test_corpus/test_word.docx
Normal file
BIN
sample_files/test_corpus/test_word.docx
Normal file
Binary file not shown.
107
scripts/compare_pdfs.py
Normal file
107
scripts/compare_pdfs.py
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
"""
|
||||||
|
Deep comparison of original vs translated PDF.
|
||||||
|
|
||||||
|
Outputs a detailed diff to identify exactly what format elements were lost.
|
||||||
|
"""
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
|
||||||
|
original = Path("sample_files/test_corpus/test_pdf.pdf")
|
||||||
|
translated = Path("sample_files/test_corpus/test_pdf_translated.pdf")
|
||||||
|
|
||||||
|
print(f"Original: {original.stat().st_size:>10} bytes")
|
||||||
|
print(f"Translated: {translated.stat().st_size:>10} bytes")
|
||||||
|
print(f"Size diff: {translated.stat().st_size - original.stat().st_size:>10} bytes ({100 * translated.stat().st_size / original.stat().st_size:.1f}%)")
|
||||||
|
print()
|
||||||
|
|
||||||
|
doc_orig = fitz.open(str(original))
|
||||||
|
doc_trans = fitz.open(str(translated))
|
||||||
|
|
||||||
|
print("=" * 70)
|
||||||
|
print("PAGE COUNT")
|
||||||
|
print("=" * 70)
|
||||||
|
print(f" Original: {len(doc_orig)} pages")
|
||||||
|
print(f" Translated: {len(doc_trans)} pages")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print("=" * 70)
|
||||||
|
print("PER-PAGE COMPARISON")
|
||||||
|
print("=" * 70)
|
||||||
|
for i in range(max(len(doc_orig), len(doc_trans))):
|
||||||
|
o_page = doc_orig[i] if i < len(doc_orig) else None
|
||||||
|
t_page = doc_trans[i] if i < len(doc_trans) else None
|
||||||
|
print(f"\n--- Page {i+1} ---")
|
||||||
|
if o_page:
|
||||||
|
o_text = len(o_page.get_text())
|
||||||
|
o_links = len(o_page.get_links())
|
||||||
|
o_images = len(o_page.get_images())
|
||||||
|
o_drawings = len(o_page.get_drawings())
|
||||||
|
print(f" Original: {o_text:>4} chars, {o_links:>2} links, {o_images:>2} images, {o_drawings:>3} drawings")
|
||||||
|
if t_page:
|
||||||
|
t_text = len(t_page.get_text())
|
||||||
|
t_links = len(t_page.get_links())
|
||||||
|
t_images = len(t_page.get_images())
|
||||||
|
t_drawings = len(t_page.get_drawings())
|
||||||
|
print(f" Translated: {t_text:>4} chars, {t_links:>2} links, {t_images:>2} images, {t_drawings:>3} drawings")
|
||||||
|
if o_page and t_page:
|
||||||
|
print(f" DIFF: text {t_text - o_text:+d}, links {t_links - o_links:+d}, images {t_images - o_images:+d}, drawings {t_drawings - o_drawings:+d}")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 70)
|
||||||
|
print("HYPERLINK COMPARISON")
|
||||||
|
print("=" * 70)
|
||||||
|
for i in range(max(len(doc_orig), len(doc_trans))):
|
||||||
|
o_page = doc_orig[i] if i < len(doc_orig) else None
|
||||||
|
t_page = doc_trans[i] if i < len(doc_trans) else None
|
||||||
|
o_links = o_page.get_links() if o_page else []
|
||||||
|
t_links = t_page.get_links() if t_page else []
|
||||||
|
if o_links or t_links:
|
||||||
|
print(f"\n Page {i+1}:")
|
||||||
|
print(f" Original has {len(o_links)} links:")
|
||||||
|
for link in o_links:
|
||||||
|
kind = "URI" if link.get("uri") else "GOTO" if link.get("page") is not None else "?"
|
||||||
|
uri_or_page = link.get("uri") or f"page {link.get('page')}"
|
||||||
|
print(f" - {kind}: {uri_or_page[:80]}")
|
||||||
|
print(f" Translated has {len(t_links)} links:")
|
||||||
|
for link in t_links:
|
||||||
|
kind = "URI" if link.get("uri") else "GOTO" if link.get("page") is not None else "?"
|
||||||
|
uri_or_page = link.get("uri") or f"page {link.get('page')}"
|
||||||
|
print(f" - {kind}: {uri_or_page[:80]}")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 70)
|
||||||
|
print("IMAGE COMPARISON")
|
||||||
|
print("=" * 70)
|
||||||
|
total_o_imgs = sum(len(p.get_images()) for p in doc_orig)
|
||||||
|
total_t_imgs = sum(len(p.get_images()) for p in doc_trans)
|
||||||
|
print(f" Original: {total_o_imgs} images")
|
||||||
|
print(f" Translated: {total_t_imgs} images")
|
||||||
|
if total_o_imgs > 0 and total_t_imgs == 0:
|
||||||
|
print(" ❌ ALL IMAGES LOST in translation!")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 70)
|
||||||
|
print("DRAWING COMPARISON (rects, lines, etc.)")
|
||||||
|
print("=" * 70)
|
||||||
|
total_o_dr = sum(len(p.get_drawings()) for p in doc_orig)
|
||||||
|
total_t_dr = sum(len(p.get_drawings()) for p in doc_trans)
|
||||||
|
print(f" Original: {total_o_dr} drawings")
|
||||||
|
print(f" Translated: {total_t_dr} drawings")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 70)
|
||||||
|
print("SAMPLE TEXT — Page 1 of translated")
|
||||||
|
print("=" * 70)
|
||||||
|
if len(doc_trans) > 0:
|
||||||
|
print(doc_trans[0].get_text()[:500])
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 70)
|
||||||
|
print("SAMPLE TEXT — Page 1 of original")
|
||||||
|
print("=" * 70)
|
||||||
|
if len(doc_orig) > 0:
|
||||||
|
print(doc_orig[0].get_text()[:500])
|
||||||
|
|
||||||
|
doc_orig.close()
|
||||||
|
doc_trans.close()
|
||||||
40
scripts/debug_rc_real.py
Normal file
40
scripts/debug_rc_real.py
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
"""Debug: show actual rc values for the title block in real translation."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
# Patch _try_insert to log rc values
|
||||||
|
original_try_insert = pdf_mod.PDFTranslator._try_insert
|
||||||
|
TITLES_TO_TRACE = {"Spécification technique : Office Translator v3.0", "Technical Specification: Office Translator v3.0"}
|
||||||
|
|
||||||
|
def debug_try_insert(self, page, rect, text, fontsize, fontname, fontfile, color, align):
|
||||||
|
rc = page.insert_textbox(
|
||||||
|
rect, text,
|
||||||
|
fontsize=fontsize,
|
||||||
|
fontname=fontname,
|
||||||
|
fontfile=fontfile,
|
||||||
|
color=color,
|
||||||
|
align=align,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
# Only trace the title
|
||||||
|
is_title = any(t in text for t in ["Spécification technique", "Technical Specification"])
|
||||||
|
if is_title:
|
||||||
|
print(f" [TITLE] size={fontsize:.1f} rect={rect.width:.0f}x{rect.height:.0f} -> rc={rc}", flush=True)
|
||||||
|
return rc
|
||||||
|
|
||||||
|
pdf_mod.PDFTranslator._try_insert = debug_try_insert
|
||||||
|
|
||||||
|
PDFTranslator = pdf_mod.PDFTranslator
|
||||||
|
PDFTranslator().translate_file(
|
||||||
|
Path('sample_files/test_corpus/test_pdf.pdf'),
|
||||||
|
Path('sample_files/test_corpus/test_pdf_translated.pdf'),
|
||||||
|
target_language='fr',
|
||||||
|
source_language='en',
|
||||||
|
)
|
||||||
37
scripts/debug_rc_values.py
Normal file
37
scripts/debug_rc_values.py
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
"""Debug the actual rc values returned by insert_textbox during real translation."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
# Monkey-patch _try_insert to print rc values
|
||||||
|
original_try_insert = pdf_mod.PDFTranslator._try_insert
|
||||||
|
|
||||||
|
def debug_try_insert(self, page, rect, text, fontsize, fontname, fontfile, color, align):
|
||||||
|
rc = page.insert_textbox(
|
||||||
|
rect, text,
|
||||||
|
fontsize=fontsize,
|
||||||
|
fontname=fontname,
|
||||||
|
fontfile=fontfile,
|
||||||
|
color=color,
|
||||||
|
align=align,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
text_preview = text[:30].replace('\n', '\\n')
|
||||||
|
print(f" TRY: size={fontsize:.1f} rect={rect.width:.0f}x{rect.height:.0f} text={text_preview!r} -> rc={rc}", flush=True)
|
||||||
|
return rc
|
||||||
|
|
||||||
|
pdf_mod.PDFTranslator._try_insert = debug_try_insert
|
||||||
|
|
||||||
|
PDFTranslator = pdf_mod.PDFTranslator
|
||||||
|
PDFTranslator().translate_file(
|
||||||
|
Path('sample_files/test_corpus/test_pdf.pdf'),
|
||||||
|
Path('sample_files/test_corpus/test_pdf_translated.pdf'),
|
||||||
|
target_language='fr',
|
||||||
|
source_language='en',
|
||||||
|
)
|
||||||
81
scripts/debug_smart_fit.py
Normal file
81
scripts/debug_smart_fit.py
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
"""Debug the smart-fit behavior on a real PDF."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location("pdf_mod", "translators/pdf_translator.py")
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
# Open the test PDF
|
||||||
|
doc = fitz.open("sample_files/test_corpus/test_pdf.pdf")
|
||||||
|
page = doc[0]
|
||||||
|
|
||||||
|
# Extract blocks
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
raw_blocks = translator._extract_text_blocks(page)
|
||||||
|
merged = translator._merge_adjacent_blocks(raw_blocks, page.rect)
|
||||||
|
|
||||||
|
# Show block info
|
||||||
|
for i, block in enumerate(merged[:5]):
|
||||||
|
print(f"Block {i}:")
|
||||||
|
print(f" bbox: {block['bbox']}")
|
||||||
|
print(f" font_size: {block['font_size']}")
|
||||||
|
print(f" text: {block['text'][:50]!r}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Manually try the smart-fit on the title block
|
||||||
|
title = merged[0]
|
||||||
|
print("=" * 60)
|
||||||
|
print("Smart-fit trace on title block:")
|
||||||
|
print("=" * 60)
|
||||||
|
import fitz
|
||||||
|
original_rect = fitz.Rect(title["bbox"])
|
||||||
|
page_rect = page.rect
|
||||||
|
print(f" original_rect: {original_rect}")
|
||||||
|
print(f" page_rect: {page_rect}")
|
||||||
|
print(f" page width: {page_rect.x1 - page_rect.x0}")
|
||||||
|
print(f" original width: {original_rect.width}, height: {original_rect.height}")
|
||||||
|
|
||||||
|
# Simulate the expansion
|
||||||
|
margin = 18
|
||||||
|
max_x1 = page_rect.x1 - margin
|
||||||
|
expanded_h = fitz.Rect(
|
||||||
|
max(original_rect.x0, page_rect.x0 + margin),
|
||||||
|
original_rect.y0,
|
||||||
|
max_x1,
|
||||||
|
original_rect.y1,
|
||||||
|
)
|
||||||
|
print(f" expanded_h: {expanded_h}, width: {expanded_h.width}")
|
||||||
|
|
||||||
|
next_block_y = page_rect.y1 - margin
|
||||||
|
max_expand_y = min(
|
||||||
|
next_block_y - original_rect.y1,
|
||||||
|
original_rect.height * pdf_mod.MAX_VERTICAL_EXPANSION,
|
||||||
|
)
|
||||||
|
expanded_v = fitz.Rect(
|
||||||
|
expanded_h.x0,
|
||||||
|
expanded_h.y0,
|
||||||
|
expanded_h.x1,
|
||||||
|
expanded_h.y1 + max_expand_y,
|
||||||
|
)
|
||||||
|
print(f" expanded_v: {expanded_v}, width: {expanded_v.width}, height: {expanded_v.height}")
|
||||||
|
|
||||||
|
# Try the actual insert
|
||||||
|
from fitz import TEXT_ALIGN_LEFT
|
||||||
|
french_text = "Spécification technique : Office Translator v3.0"
|
||||||
|
rc = page.insert_textbox(
|
||||||
|
expanded_v,
|
||||||
|
french_text,
|
||||||
|
fontsize=title["font_size"],
|
||||||
|
fontname="helv",
|
||||||
|
color=(0.1, 0.2, 0.5),
|
||||||
|
align=TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f" insert_textbox rc (with expanded_v at original size): {rc}")
|
||||||
|
print(f" >= 0 = fit, < 0 = overflow of {rc} points")
|
||||||
|
|
||||||
|
doc.close()
|
||||||
55
scripts/debug_smart_fit_trace.py
Normal file
55
scripts/debug_smart_fit_trace.py
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
"""Debug trace the smart-fit behavior on a real PDF."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
original_write = pdf_mod.PDFTranslator._write_translated_block
|
||||||
|
|
||||||
|
def debug_write(self, page, block, font_path, is_rtl):
|
||||||
|
target_size = block['font_size']
|
||||||
|
original_rect = fitz.Rect(block['bbox'])
|
||||||
|
page_rect = page.rect
|
||||||
|
margin = 18
|
||||||
|
is_heading = target_size >= 14
|
||||||
|
if is_heading:
|
||||||
|
expanded_h = original_rect
|
||||||
|
else:
|
||||||
|
expanded_h = fitz.Rect(
|
||||||
|
max(original_rect.x0, page_rect.x0 + margin),
|
||||||
|
original_rect.y0,
|
||||||
|
page_rect.x1 - margin,
|
||||||
|
original_rect.y1,
|
||||||
|
)
|
||||||
|
next_block_y = page_rect.y1 - margin
|
||||||
|
max_expand_y = min(
|
||||||
|
next_block_y - original_rect.y1,
|
||||||
|
original_rect.height * 3.0,
|
||||||
|
)
|
||||||
|
expanded_v = fitz.Rect(
|
||||||
|
expanded_h.x0,
|
||||||
|
expanded_h.y0,
|
||||||
|
expanded_h.x1,
|
||||||
|
expanded_h.y1 + max_expand_y,
|
||||||
|
)
|
||||||
|
text = block['translated'][:40]
|
||||||
|
is_heading_str = "HEAD" if is_heading else "BODY"
|
||||||
|
print(f" [{is_heading_str}] text={text!r} | orig={original_rect.width:.0f}x{original_rect.height:.0f} | exp_v={expanded_v.width:.0f}x{expanded_v.height:.0f} | size={target_size}", flush=True)
|
||||||
|
result = original_write(self, page, block, font_path, is_rtl)
|
||||||
|
print(f" result: {result}", flush=True)
|
||||||
|
return result
|
||||||
|
|
||||||
|
pdf_mod.PDFTranslator._write_translated_block = debug_write
|
||||||
|
|
||||||
|
PDFTranslator = pdf_mod.PDFTranslator
|
||||||
|
PDFTranslator().translate_file(
|
||||||
|
Path('sample_files/test_corpus/test_pdf.pdf'),
|
||||||
|
Path('sample_files/test_corpus/test_pdf_translated.pdf'),
|
||||||
|
target_language='fr',
|
||||||
|
source_language='en',
|
||||||
|
)
|
||||||
71
scripts/diagnose_pdf_layout.py
Normal file
71
scripts/diagnose_pdf_layout.py
Normal file
@@ -0,0 +1,71 @@
|
|||||||
|
"""
|
||||||
|
Deeper investigation: where exactly is the text on each page?
|
||||||
|
|
||||||
|
Compares bounding boxes of the original text vs translated text.
|
||||||
|
If they're in totally different positions, the layout is broken.
|
||||||
|
"""
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
orig = fitz.open("sample_files/test_corpus/test_pdf.pdf")
|
||||||
|
trans = fitz.open("sample_files/test_corpus/test_pdf_translated.pdf")
|
||||||
|
|
||||||
|
for i in range(min(3, len(orig), len(trans))):
|
||||||
|
o = orig[i]
|
||||||
|
t = trans[i]
|
||||||
|
print(f"\n{'=' * 70}")
|
||||||
|
print(f"PAGE {i+1}")
|
||||||
|
print('=' * 70)
|
||||||
|
|
||||||
|
# Get text blocks with positions for original
|
||||||
|
print(f"\n--- Original (page {i+1}) text blocks ---")
|
||||||
|
o_blocks = o.get_text("blocks")
|
||||||
|
for j, b in enumerate(o_blocks[:10]):
|
||||||
|
x0, y0, x1, y1, text, *_ = b
|
||||||
|
text_preview = text[:50].replace("\n", " ")
|
||||||
|
print(f" [{j}] ({x0:.0f},{y0:.0f})-({x1:.0f},{y1:.0f}) '{text_preview}'")
|
||||||
|
|
||||||
|
print(f"\n--- Translated (page {i+1}) text blocks ---")
|
||||||
|
t_blocks = t.get_text("blocks")
|
||||||
|
for j, b in enumerate(t_blocks[:10]):
|
||||||
|
x0, y0, x1, y1, text, *_ = b
|
||||||
|
text_preview = text[:50].replace("\n", " ")
|
||||||
|
print(f" [{j}] ({x0:.0f},{y0:.0f})-({x1:.0f},{y1:.0f}) '{text_preview}'")
|
||||||
|
|
||||||
|
# Check fonts
|
||||||
|
print(f"\n--- Fonts (page {i+1}) ---")
|
||||||
|
o_fonts = set()
|
||||||
|
t_fonts = set()
|
||||||
|
for blk in o_blocks:
|
||||||
|
for line in blk[4].split("\n") if blk[4] else []:
|
||||||
|
pass
|
||||||
|
# Get fonts from the dict format
|
||||||
|
o_dict = o.get_text("dict")
|
||||||
|
t_dict = t.get_text("dict")
|
||||||
|
for block in o_dict.get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
o_fonts.add(f"{span.get('font', '?')}@{span.get('size', 0):.1f}")
|
||||||
|
for block in t_dict.get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
t_fonts.add(f"{span.get('font', '?')}@{span.get('size', 0):.1f}")
|
||||||
|
print(f" Original fonts: {sorted(o_fonts)[:5]}")
|
||||||
|
print(f" Translated fonts: {sorted(t_fonts)[:5]}")
|
||||||
|
|
||||||
|
# Check colors used
|
||||||
|
print(f"\n--- Colors (page {i+1}) ---")
|
||||||
|
o_colors = set()
|
||||||
|
t_colors = set()
|
||||||
|
for block in o_dict.get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
o_colors.add(span.get("color", 0))
|
||||||
|
for block in t_dict.get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
t_colors.add(span.get("color", 0))
|
||||||
|
print(f" Original colors: {sorted(o_colors)[:5]}")
|
||||||
|
print(f" Translated colors: {sorted(t_colors)[:5]}")
|
||||||
|
|
||||||
|
orig.close()
|
||||||
|
trans.close()
|
||||||
1304
scripts/generate_test_files.py
Normal file
1304
scripts/generate_test_files.py
Normal file
File diff suppressed because it is too large
Load Diff
103
scripts/migrate_glossaries_to_multilingual_by_translation.py
Normal file
103
scripts/migrate_glossaries_to_multilingual_by_translation.py
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Migrate and translate all non-multilingual glossaries in the database to multilingual glossaries.
|
||||||
|
It uses the free Google Translate provider to fill in the translations for target languages
|
||||||
|
(de, es, it, pt, nl, ru, ja, ko, zh, ar, fa, en) in the glossary_terms table.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
|
||||||
|
from database.connection import get_sync_session
|
||||||
|
from database.models import Glossary, GlossaryTerm
|
||||||
|
from services.providers.google_provider import get_legacy_google_adapter
|
||||||
|
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
)
|
||||||
|
logger = logging.getLogger("translate_non_multilingual_glossaries")
|
||||||
|
|
||||||
|
TARGET_LANGUAGES = ["de", "es", "it", "pt", "nl", "ru", "ja", "ko", "zh", "ar", "fa"]
|
||||||
|
|
||||||
|
def translate_term(adapter, text_to_translate: str, src_lang: str, target_langs: list[str]) -> dict:
|
||||||
|
translations = {}
|
||||||
|
for lang in target_langs:
|
||||||
|
try:
|
||||||
|
translated = adapter.translate(text_to_translate, target_language=lang, source_language=src_lang)
|
||||||
|
translations[lang] = translated.strip()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to translate '{text_to_translate}' to {lang}: {e}")
|
||||||
|
translations[lang] = ""
|
||||||
|
return translations
|
||||||
|
|
||||||
|
def main():
|
||||||
|
logger.info("Connecting to database and fetching all glossaries...")
|
||||||
|
|
||||||
|
adapter = get_legacy_google_adapter()
|
||||||
|
|
||||||
|
with get_sync_session() as session:
|
||||||
|
# Find all glossaries
|
||||||
|
all_glossaries = session.query(Glossary).all()
|
||||||
|
|
||||||
|
if not all_glossaries:
|
||||||
|
logger.info("No glossaries found in database.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
logger.info(f"Found {len(all_glossaries)} glossaries to process.")
|
||||||
|
|
||||||
|
for glossary in all_glossaries:
|
||||||
|
logger.info(f"Processing glossary '{glossary.name}' (ID: {glossary.id})")
|
||||||
|
|
||||||
|
# Change target_language to 'multi' if not already
|
||||||
|
if glossary.target_language != 'multi':
|
||||||
|
glossary.target_language = 'multi'
|
||||||
|
|
||||||
|
# Rename the glossary to indicate it's now multilingual (e.g. replace "Anglais" with "Multilingue")
|
||||||
|
if "Anglais" in glossary.name:
|
||||||
|
glossary.name = glossary.name.replace("Anglais", "Multilingue")
|
||||||
|
elif "English" in glossary.name:
|
||||||
|
glossary.name = glossary.name.replace("English", "Multilingue")
|
||||||
|
else:
|
||||||
|
glossary.name = f"{glossary.name} → Multilingue"
|
||||||
|
|
||||||
|
src_lang = glossary.source_language or "fr"
|
||||||
|
|
||||||
|
terms = session.query(GlossaryTerm).filter(GlossaryTerm.glossary_id == glossary.id).all()
|
||||||
|
logger.info(f"Checking translations for {len(terms)} terms in '{glossary.name}'...")
|
||||||
|
|
||||||
|
updated_terms_count = 0
|
||||||
|
for idx, term in enumerate(terms):
|
||||||
|
translations = term.translations or {}
|
||||||
|
|
||||||
|
# Check if we already have translations for the target languages
|
||||||
|
missing_langs = [lang for lang in TARGET_LANGUAGES if lang not in translations or not translations[lang]]
|
||||||
|
|
||||||
|
if missing_langs:
|
||||||
|
# Translate
|
||||||
|
logger.info(f" [{idx+1}/{len(terms)}] Translating '{term.source}' to {missing_langs}...")
|
||||||
|
new_translations = translate_term(adapter, term.source, src_lang, missing_langs)
|
||||||
|
translations.update(new_translations)
|
||||||
|
updated_terms_count += 1
|
||||||
|
|
||||||
|
# Ensure the original default target (e.g. English translation) is in the translations dict under 'en'
|
||||||
|
if 'en' not in translations or not translations['en']:
|
||||||
|
translations['en'] = term.target
|
||||||
|
|
||||||
|
term.translations = translations
|
||||||
|
|
||||||
|
if updated_terms_count > 0:
|
||||||
|
session.commit()
|
||||||
|
logger.info(f"Glossary '{glossary.name}' updated with {updated_terms_count} translated terms.")
|
||||||
|
else:
|
||||||
|
logger.info(f"No translations were missing for glossary '{glossary.name}'.")
|
||||||
|
|
||||||
|
logger.info("Migration complete!")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
sys.exit(main())
|
||||||
45
scripts/reproduce_image_issues.py
Normal file
45
scripts/reproduce_image_issues.py
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
"""Reproduce the image issues: title overflow + white box on colored block."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Open the translated PDF from the user
|
||||||
|
src = Path('sample_files/test_corpus/test_pdf_translated (1).pdf')
|
||||||
|
doc = fitz.open(str(src))
|
||||||
|
|
||||||
|
# Page 1
|
||||||
|
p = doc[0]
|
||||||
|
print("=== Page 1 text blocks ===")
|
||||||
|
for i, b in enumerate(p.get_text("dict").get("blocks", [])):
|
||||||
|
bbox = b.get("bbox", (0, 0, 0, 0))
|
||||||
|
if b.get("type") != 0: # not text
|
||||||
|
continue
|
||||||
|
text_lines = []
|
||||||
|
for line in b.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
text_lines.append(span.get("text", ""))
|
||||||
|
text = " ".join(text_lines)[:60]
|
||||||
|
print(f" [{i}] bbox=({bbox[0]:.0f},{bbox[1]:.0f},{bbox[2]:.0f},{bbox[3]:.0f}) text={text!r}")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=== Page 1 drawings ===")
|
||||||
|
for i, d in enumerate(p.get_drawings()):
|
||||||
|
rect = d.get("rect")
|
||||||
|
fill = d.get("fill")
|
||||||
|
color = d.get("color")
|
||||||
|
print(f" [{i}] rect=({rect.x0:.0f},{rect.y0:.0f},{rect.x1:.0f},{rect.y1:.0f}) fill={fill} color={color}")
|
||||||
|
|
||||||
|
# Open the original to compare
|
||||||
|
orig = fitz.open('sample_files/test_corpus/test_pdf.pdf')
|
||||||
|
p_orig = orig[0]
|
||||||
|
print()
|
||||||
|
print("=== Original Page 1 drawings (for comparison) ===")
|
||||||
|
for i, d in enumerate(p_orig.get_drawings()):
|
||||||
|
rect = d.get("rect")
|
||||||
|
fill = d.get("fill")
|
||||||
|
color = d.get("color")
|
||||||
|
print(f" [{i}] rect=({rect.x0:.0f},{rect.y0:.0f},{rect.x1:.0f},{rect.y1:.0f}) fill={fill} color={color}")
|
||||||
|
|
||||||
|
doc.close()
|
||||||
|
orig.close()
|
||||||
33
scripts/rerun_translation.py
Normal file
33
scripts/rerun_translation.py
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
"""Re-run translation with the new B3.5 code and save to disk."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
src = Path('sample_files/test_corpus/test_pdf.pdf')
|
||||||
|
dst = Path('sample_files/test_corpus/test_pdf_translated.pdf')
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
PDFTranslator().translate_file(src, dst, target_language='fr', source_language='en')
|
||||||
|
|
||||||
|
o = fitz.open(src)
|
||||||
|
n = fitz.open(dst)
|
||||||
|
print(f'Original: {len(o)} pages, {src.stat().st_size} bytes')
|
||||||
|
print(f'Translated: {len(n)} pages, {dst.stat().st_size} bytes')
|
||||||
|
print()
|
||||||
|
print('Page 1 font sizes:')
|
||||||
|
o_sizes = set()
|
||||||
|
n_sizes = set()
|
||||||
|
for b in o[0].get_text("dict").get("blocks", []):
|
||||||
|
for l in b.get("lines", []):
|
||||||
|
for s in l.get("spans", []):
|
||||||
|
o_sizes.add(round(s.get("size", 0), 1))
|
||||||
|
for b in n[0].get_text("dict").get("blocks", []):
|
||||||
|
for l in b.get("lines", []):
|
||||||
|
for s in l.get("spans", []):
|
||||||
|
n_sizes.add(round(s.get("size", 0), 1))
|
||||||
|
print(' Original: ', sorted(o_sizes))
|
||||||
|
print(' Translated:', sorted(n_sizes))
|
||||||
|
o.close()
|
||||||
|
n.close()
|
||||||
@@ -1,77 +0,0 @@
|
|||||||
"""
|
|
||||||
Script de setup Stripe : crée les produits et prix dans Stripe Dashboard.
|
|
||||||
Lance avec : python scripts/stripe_setup.py
|
|
||||||
"""
|
|
||||||
import stripe
|
|
||||||
import sys
|
|
||||||
|
|
||||||
SK = "sk_test_51SkSHkCKXUJE51jnbEtXZ0nKiTHTa8ohDwLH8fZiDVEx6Ze0g5dg4fGJJgX1VgNHvF93GE3HTramT3oQrCaqOxid00OXTcZlsW"
|
|
||||||
stripe.api_key = SK
|
|
||||||
|
|
||||||
PLANS = [
|
|
||||||
{"id": "starter", "name": "Wordly Starter", "monthly_eur": 900, "yearly_eur": 8640},
|
|
||||||
{"id": "pro", "name": "Wordly Pro", "monthly_eur": 1900, "yearly_eur": 18240},
|
|
||||||
{"id": "business","name": "Wordly Business","monthly_eur": 4900, "yearly_eur": 47040},
|
|
||||||
]
|
|
||||||
|
|
||||||
results = {}
|
|
||||||
for plan in PLANS:
|
|
||||||
print(f"\n>> Creation produit {plan['name']}...")
|
|
||||||
# Search existing product first
|
|
||||||
existing = stripe.Product.search(query=f"name:'{plan['name']}'", limit=1)
|
|
||||||
if existing.data:
|
|
||||||
product = existing.data[0]
|
|
||||||
print(f" Produit existant: {product.id}")
|
|
||||||
else:
|
|
||||||
product = stripe.Product.create(
|
|
||||||
name=plan["name"],
|
|
||||||
metadata={"plan_id": plan["id"]}
|
|
||||||
)
|
|
||||||
print(f" Produit créé: {product.id}")
|
|
||||||
|
|
||||||
# Monthly price
|
|
||||||
monthly_prices = stripe.Price.list(product=product.id, active=True, limit=10)
|
|
||||||
monthly_price = None
|
|
||||||
yearly_price = None
|
|
||||||
for p in monthly_prices.data:
|
|
||||||
if p.recurring and p.recurring.interval == "month" and p.unit_amount == plan["monthly_eur"]:
|
|
||||||
monthly_price = p
|
|
||||||
if p.recurring and p.recurring.interval == "year" and p.unit_amount == plan["yearly_eur"]:
|
|
||||||
yearly_price = p
|
|
||||||
|
|
||||||
if not monthly_price:
|
|
||||||
monthly_price = stripe.Price.create(
|
|
||||||
product=product.id,
|
|
||||||
unit_amount=plan["monthly_eur"],
|
|
||||||
currency="eur",
|
|
||||||
recurring={"interval": "month"},
|
|
||||||
metadata={"plan_id": plan["id"], "billing": "monthly"}
|
|
||||||
)
|
|
||||||
print(f" Prix mensuel créé: {monthly_price.id}")
|
|
||||||
else:
|
|
||||||
print(f" Prix mensuel existant: {monthly_price.id}")
|
|
||||||
|
|
||||||
if not yearly_price:
|
|
||||||
yearly_price = stripe.Price.create(
|
|
||||||
product=product.id,
|
|
||||||
unit_amount=plan["yearly_eur"],
|
|
||||||
currency="eur",
|
|
||||||
recurring={"interval": "year"},
|
|
||||||
metadata={"plan_id": plan["id"], "billing": "yearly"}
|
|
||||||
)
|
|
||||||
print(f" Prix annuel créé: {yearly_price.id}")
|
|
||||||
else:
|
|
||||||
print(f" Prix annuel existant: {yearly_price.id}")
|
|
||||||
|
|
||||||
results[plan["id"]] = {
|
|
||||||
"monthly": monthly_price.id,
|
|
||||||
"yearly": yearly_price.id,
|
|
||||||
}
|
|
||||||
|
|
||||||
print("\n" + "="*60)
|
|
||||||
print("Variables à ajouter dans .env.production et .env :")
|
|
||||||
print("="*60)
|
|
||||||
for plan_id, ids in results.items():
|
|
||||||
print(f"STRIPE_PRICE_{plan_id.upper()}_MONTHLY={ids['monthly']}")
|
|
||||||
print(f"STRIPE_PRICE_{plan_id.upper()}_YEARLY={ids['yearly']}")
|
|
||||||
print("="*60)
|
|
||||||
72
scripts/test_full_pipeline.py
Normal file
72
scripts/test_full_pipeline.py
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
"""Test if the full pipeline (extract, redact, re-insert links) affects insert_textbox."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
doc = fitz.open('sample_files/test_corpus/test_pdf.pdf')
|
||||||
|
page = doc[0]
|
||||||
|
|
||||||
|
# Step 1: Extract blocks
|
||||||
|
raw_blocks = pdf_mod.PDFTranslator()._extract_text_blocks(page)
|
||||||
|
title = raw_blocks[0]
|
||||||
|
print(f"Title bbox: {title['bbox']}")
|
||||||
|
|
||||||
|
# Step 2: Simulate redaction
|
||||||
|
title_rect = fitz.Rect(title['bbox'])
|
||||||
|
page.add_redact_annot(title_rect, fill=(1, 1, 1))
|
||||||
|
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
||||||
|
print("After redaction")
|
||||||
|
|
||||||
|
# Step 3: Simulate link reinsertion (we have TOC links)
|
||||||
|
links = list(page.get_links())
|
||||||
|
print(f"Got {len(links)} links")
|
||||||
|
for link in links:
|
||||||
|
from_rect = link.get("from")
|
||||||
|
if link.get("page") is not None:
|
||||||
|
page.insert_link({
|
||||||
|
"kind": fitz.LINK_GOTO,
|
||||||
|
"from": from_rect,
|
||||||
|
"page": link["page"],
|
||||||
|
"to": fitz.Point(72, 72),
|
||||||
|
})
|
||||||
|
elif link.get("uri"):
|
||||||
|
page.insert_link({
|
||||||
|
"kind": fitz.LINK_URI,
|
||||||
|
"from": from_rect,
|
||||||
|
"uri": link["uri"],
|
||||||
|
})
|
||||||
|
print("After link reinsertion")
|
||||||
|
|
||||||
|
# Step 4: NOW try insert_textbox
|
||||||
|
french = "Spécification technique : Office Translator v3.0"
|
||||||
|
rect = fitz.Rect(title['bbox'])
|
||||||
|
expanded_v = fitz.Rect(rect.x0, rect.y0, rect.x1, rect.y1 + rect.height * 3)
|
||||||
|
|
||||||
|
# Test on original bbox
|
||||||
|
rc1 = page.insert_textbox(
|
||||||
|
rect, french,
|
||||||
|
fontsize=22,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
color=(0.1, 0.2, 0.5),
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"After full pipeline, original rect: rc={rc1}")
|
||||||
|
|
||||||
|
# Test on expanded_v
|
||||||
|
rc2 = page.insert_textbox(
|
||||||
|
expanded_v, french,
|
||||||
|
fontsize=22,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
color=(0.1, 0.2, 0.5),
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"After full pipeline, expanded_v: rc={rc2}")
|
||||||
|
|
||||||
|
doc.close()
|
||||||
87
scripts/test_merged_block.py
Normal file
87
scripts/test_merged_block.py
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
"""Test if merged blocks (TOC) fit in expanded_v."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
doc = fitz.open('sample_files/test_corpus/test_pdf.pdf')
|
||||||
|
page = doc[0]
|
||||||
|
|
||||||
|
# Extract and merge
|
||||||
|
raw_blocks = pdf_mod.PDFTranslator()._extract_text_blocks(page)
|
||||||
|
merged = pdf_mod.PDFTranslator()._merge_adjacent_blocks(raw_blocks, page.rect)
|
||||||
|
|
||||||
|
# Find the TOC block (block 3 in our trace)
|
||||||
|
toc = merged[3]
|
||||||
|
print(f"TOC bbox: {toc['bbox']}")
|
||||||
|
print(f"TOC font_size: {toc['font_size']}")
|
||||||
|
print(f"TOC text length: {len(toc['text'])}")
|
||||||
|
print(f"TOC text preview: {toc['text'][:80]!r}")
|
||||||
|
|
||||||
|
# Redact
|
||||||
|
toc_rect = fitz.Rect(toc['bbox'])
|
||||||
|
page.add_redact_annot(toc_rect, fill=(1, 1, 1))
|
||||||
|
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
||||||
|
|
||||||
|
# Test insert with expanded_v
|
||||||
|
margin = 18
|
||||||
|
page_rect = page.rect
|
||||||
|
expanded_h = fitz.Rect(
|
||||||
|
max(toc_rect.x0, page_rect.x0 + margin),
|
||||||
|
toc_rect.y0,
|
||||||
|
page_rect.x1 - margin,
|
||||||
|
toc_rect.y1,
|
||||||
|
)
|
||||||
|
max_expand_y = min(
|
||||||
|
page_rect.y1 - margin - toc_rect.y1,
|
||||||
|
toc_rect.height * 3.0,
|
||||||
|
)
|
||||||
|
expanded_v = fitz.Rect(
|
||||||
|
expanded_h.x0, expanded_h.y0, expanded_h.x1,
|
||||||
|
expanded_h.y1 + max_expand_y,
|
||||||
|
)
|
||||||
|
print(f"expanded_v: {expanded_v.width}x{expanded_v.height}")
|
||||||
|
|
||||||
|
# Translate the TOC text
|
||||||
|
french_toc = """1. Introduction
|
||||||
|
.......... 1
|
||||||
|
2. Installation et configuration
|
||||||
|
.......... 2
|
||||||
|
3. Moteur de traduction
|
||||||
|
.......... 3
|
||||||
|
4. Préservation des formats
|
||||||
|
.......... 4
|
||||||
|
5. Assurance qualité
|
||||||
|
.......... 5
|
||||||
|
6. Performances et mise à l'échelle
|
||||||
|
.......... 6
|
||||||
|
7. Référence API
|
||||||
|
.......... 7
|
||||||
|
8. Dépannage
|
||||||
|
.......... 8"""
|
||||||
|
|
||||||
|
# Test on original
|
||||||
|
rc1 = page.insert_textbox(
|
||||||
|
toc_rect, french_toc,
|
||||||
|
fontsize=12,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"Original rect: rc={rc1}")
|
||||||
|
|
||||||
|
# Test on expanded_v
|
||||||
|
rc2 = page.insert_textbox(
|
||||||
|
expanded_v, french_toc,
|
||||||
|
fontsize=12,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"expanded_v: rc={rc2}")
|
||||||
|
|
||||||
|
doc.close()
|
||||||
46
scripts/test_smart_fit_no_translation.py
Normal file
46
scripts/test_smart_fit_no_translation.py
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
"""Test smart-fit with mock translation (same length) to isolate the issue."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
# Mock the translation to return the original text (no length change)
|
||||||
|
def mock_translate(self, text, target_lang, source_lang):
|
||||||
|
return text # No change
|
||||||
|
|
||||||
|
PDFTranslator = pdf_mod.PDFTranslator
|
||||||
|
with patch.object(PDFTranslator, '_translate_single', mock_translate):
|
||||||
|
PDFTranslator().translate_file(
|
||||||
|
Path('sample_files/test_corpus/test_pdf.pdf'),
|
||||||
|
Path('sample_files/test_corpus/test_pdf_no_translation.pdf'),
|
||||||
|
target_language='fr',
|
||||||
|
source_language='en',
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check fonts
|
||||||
|
o = fitz.open('sample_files/test_corpus/test_pdf.pdf')
|
||||||
|
n = fitz.open('sample_files/test_corpus/test_pdf_no_translation.pdf')
|
||||||
|
print(f"=== Mock translation (no length change) ===")
|
||||||
|
print(f"Original: {len(o)} pages, {o.page_count} blocks")
|
||||||
|
print(f"Translated: {len(n)} pages")
|
||||||
|
for i in [0]:
|
||||||
|
o_sizes = set()
|
||||||
|
n_sizes = set()
|
||||||
|
for b in o[i].get_text("dict").get("blocks", []):
|
||||||
|
for l in b.get("lines", []):
|
||||||
|
for s in l.get("spans", []):
|
||||||
|
o_sizes.add(round(s.get("size", 0), 1))
|
||||||
|
for b in n[i].get_text("dict").get("blocks", []):
|
||||||
|
for l in b.get("lines", []):
|
||||||
|
for s in l.get("spans", []):
|
||||||
|
n_sizes.add(round(s.get("size", 0), 1))
|
||||||
|
print(f" Page 1 Original: {sorted(o_sizes)}")
|
||||||
|
print(f" Page 1 Translated: {sorted(n_sizes)}")
|
||||||
|
o.close()
|
||||||
|
n.close()
|
||||||
59
scripts/test_with_redaction.py
Normal file
59
scripts/test_with_redaction.py
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
"""Test if redaction affects insert_textbox behavior."""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import importlib.util
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location('pdf_mod', 'translators/pdf_translator.py')
|
||||||
|
pdf_mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(pdf_mod)
|
||||||
|
|
||||||
|
doc = fitz.open('sample_files/test_corpus/test_pdf.pdf')
|
||||||
|
page = doc[0]
|
||||||
|
|
||||||
|
# Get the title block info
|
||||||
|
raw_blocks = pdf_mod.PDFTranslator()._extract_text_blocks(page)
|
||||||
|
title = raw_blocks[0]
|
||||||
|
print(f"Title bbox: {title['bbox']}")
|
||||||
|
print(f"Title text: {title['text']!r}")
|
||||||
|
|
||||||
|
# Test 1: insert_textbox on fresh page
|
||||||
|
french = "Spécification technique : Office Translator v3.0"
|
||||||
|
rect = fitz.Rect(title['bbox'])
|
||||||
|
rc1 = page.insert_textbox(
|
||||||
|
rect, french,
|
||||||
|
fontsize=22,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
color=(0.1, 0.2, 0.5),
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"Fresh page: rc={rc1} (negative = overflow, positive = fit)")
|
||||||
|
|
||||||
|
# Test 2: Apply redaction, then insert_textbox
|
||||||
|
page.add_redact_annot(rect, fill=(1, 1, 1))
|
||||||
|
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
||||||
|
|
||||||
|
rc2 = page.insert_textbox(
|
||||||
|
rect, french,
|
||||||
|
fontsize=22,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
color=(0.1, 0.2, 0.5),
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"After redaction: rc={rc2}")
|
||||||
|
|
||||||
|
# Test 3: With expanded_v
|
||||||
|
expanded_v = fitz.Rect(rect.x0, rect.y0, rect.x1, rect.y1 + rect.height * 3)
|
||||||
|
rc3 = page.insert_textbox(
|
||||||
|
expanded_v, french,
|
||||||
|
fontsize=22,
|
||||||
|
fontfile='C:/Windows/Fonts/arial.ttf',
|
||||||
|
color=(0.1, 0.2, 0.5),
|
||||||
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
|
overlay=True,
|
||||||
|
)
|
||||||
|
print(f"After redaction, expanded_v ({expanded_v.width}x{expanded_v.height}): rc={rc3}")
|
||||||
|
|
||||||
|
doc.close()
|
||||||
62
scripts/verify_b3_5_fix.py
Normal file
62
scripts/verify_b3_5_fix.py
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
"""
|
||||||
|
End-to-end verification: run the smart-fit PDF translator on the
|
||||||
|
real test PDF and verify the output preserves the font hierarchy.
|
||||||
|
"""
|
||||||
|
import fitz
|
||||||
|
import shutil
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Create a temp copy of the test PDF
|
||||||
|
src = Path("sample_files/test_corpus/test_pdf.pdf")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
work = Path(tmp) / "test.pdf"
|
||||||
|
shutil.copy(src, work)
|
||||||
|
out = Path(tmp) / "out.pdf"
|
||||||
|
|
||||||
|
# Run the translator
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
translator.translate_file(
|
||||||
|
work, out,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Compare fonts
|
||||||
|
orig = fitz.open(src)
|
||||||
|
new = fitz.open(out)
|
||||||
|
|
||||||
|
print(f"Original: {len(orig)} pages, {src.stat().st_size} bytes")
|
||||||
|
print(f"Translated: {len(new)} pages, {out.stat().st_size} bytes")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print(f"{'Page':<6}{'Original fonts':<40}{'Translated fonts':<40}")
|
||||||
|
print("=" * 90)
|
||||||
|
for i in range(min(len(orig), len(new))):
|
||||||
|
o = orig[i]
|
||||||
|
t = new[i]
|
||||||
|
|
||||||
|
o_fonts = set()
|
||||||
|
t_fonts = set()
|
||||||
|
for block in o.get_text("dict").get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
o_fonts.add(round(span.get("size", 0), 1))
|
||||||
|
for block in t.get_text("dict").get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
t_fonts.add(round(span.get("size", 0), 1))
|
||||||
|
|
||||||
|
# Filter out the 6.0pt placeholder
|
||||||
|
t_fonts_real = {s for s in t_fonts if s > 6.5}
|
||||||
|
|
||||||
|
print(f"{i+1:<6}{sorted(o_fonts)!s:<40}{sorted(t_fonts_real)!s:<40}")
|
||||||
|
|
||||||
|
orig.close()
|
||||||
|
new.close()
|
||||||
|
|
||||||
|
# Save the translated output for the user to inspect
|
||||||
|
shutil.copy(out, "sample_files/test_corpus/test_pdf_translated_v2.pdf")
|
||||||
|
print()
|
||||||
|
print("Translated output saved to: sample_files/test_corpus/test_pdf_translated_v2.pdf")
|
||||||
228
scripts/verify_b3_6_fix.py
Normal file
228
scripts/verify_b3_6_fix.py
Normal file
@@ -0,0 +1,228 @@
|
|||||||
|
"""
|
||||||
|
B3.6 visual verification script.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Re-translate the test PDF using a mock provider (cheap, no API cost).
|
||||||
|
2. Render both the OLD broken PDF and the NEW (B3.6) PDF as PNGs.
|
||||||
|
3. Inspect drawings: the blue "Avis important" box must still be present
|
||||||
|
in the new PDF but was lost in the old PDF.
|
||||||
|
4. Inspect title block: it must not overlap the next block ("Version du document...").
|
||||||
|
5. Open both PNGs side-by-side so we can eyeball the difference.
|
||||||
|
|
||||||
|
Usage: python scripts/verify_b3_6_fix.py
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
# Make repo root importable
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||||
|
|
||||||
|
import fitz # PyMuPDF
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
OLD_PDF = REPO_ROOT / "sample_files" / "test_corpus" / "test_pdf_translated (1).pdf"
|
||||||
|
ORIG_PDF = REPO_ROOT / "sample_files" / "test_corpus" / "test_pdf.pdf"
|
||||||
|
NEW_PDF = REPO_ROOT / "sample_files" / "test_corpus" / "test_pdf_translated_fresh.pdf"
|
||||||
|
RENDER_DIR = REPO_ROOT / "scripts" / "_renders"
|
||||||
|
RENDER_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_pdf_page(pdf_path: Path, page_num: int = 0, out_png: Path = None,
|
||||||
|
zoom: float = 1.5) -> Path:
|
||||||
|
"""Render a single PDF page as PNG."""
|
||||||
|
doc = fitz.open(str(pdf_path))
|
||||||
|
page = doc[page_num]
|
||||||
|
mat = fitz.Matrix(zoom, zoom)
|
||||||
|
pix = page.get_pixmap(matrix=mat)
|
||||||
|
if out_png is None:
|
||||||
|
out_png = RENDER_DIR / f"{pdf_path.stem}_p{page_num + 1}.png"
|
||||||
|
pix.save(str(out_png))
|
||||||
|
doc.close()
|
||||||
|
return out_png
|
||||||
|
|
||||||
|
|
||||||
|
def _translate_with_mock(input_path: Path, output_path: Path) -> Path:
|
||||||
|
"""Translate using a mock provider that returns the French text we want."""
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
|
||||||
|
translator = PDFTranslator()
|
||||||
|
|
||||||
|
# Mock provider: a fake French translation that's longer than the English original
|
||||||
|
# (this is the realistic case that triggered the overflow bug)
|
||||||
|
def mock_translate(text, target_language, source_language):
|
||||||
|
# Return a slightly longer French version to stress the layout
|
||||||
|
if not text:
|
||||||
|
return text
|
||||||
|
# Replace common patterns with longer French to trigger overflow
|
||||||
|
replacements = {
|
||||||
|
"Technical Specification": "Spécification technique : Office Translator",
|
||||||
|
"Document Version": "Version du document",
|
||||||
|
"Table of Contents": "Table des matières",
|
||||||
|
"1. Introduction": "1. Introduction",
|
||||||
|
"2. Installation and Configuration": "2. Installation et configuration",
|
||||||
|
"3. Translation Engine": "3. Moteur de traduction",
|
||||||
|
"4. Format Preservation": "4. Préservation des formats",
|
||||||
|
"5. Quality Assurance": "5. Assurance qualité",
|
||||||
|
"6. Performance and Scaling": "6. Performances et mise à l'échelle",
|
||||||
|
"7. API Reference": "7. Référence API",
|
||||||
|
"8. Troubleshooting": "8. Résolution des problèmes",
|
||||||
|
"Important Notice": "Avis important",
|
||||||
|
"This is a preliminary document.": "Ceci est un document préliminaire.",
|
||||||
|
"Subject to change without notice.": "Sujet à changement sans préavis.",
|
||||||
|
"For the latest version of this document, visit:": "Pour la dernière version de ce document, visitez :",
|
||||||
|
"http://docs.translator.example.com/v3": "http://docs.translator.example.com/v3",
|
||||||
|
}
|
||||||
|
result = text
|
||||||
|
for eng, fr in replacements.items():
|
||||||
|
result = result.replace(eng, fr)
|
||||||
|
return result
|
||||||
|
|
||||||
|
mock = MagicMock()
|
||||||
|
mock.translate_batch = MagicMock(side_effect=lambda texts, tgt, src: [mock_translate(t, tgt, src) for t in texts])
|
||||||
|
|
||||||
|
translator.set_provider(mock)
|
||||||
|
translator.translate_file(
|
||||||
|
input_path, output_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
return output_path
|
||||||
|
|
||||||
|
|
||||||
|
def _inspect_drawings(pdf_path: Path) -> dict:
|
||||||
|
"""Inspect page 1: count blue background drawings (the "Avis important" box)."""
|
||||||
|
doc = fitz.open(str(pdf_path))
|
||||||
|
page = doc[0]
|
||||||
|
drawings = page.get_drawings()
|
||||||
|
# The "Avis important" box has fill (0.85, 0.92, 1.0) — light blue
|
||||||
|
blue_boxes = []
|
||||||
|
for d in drawings:
|
||||||
|
fill = d.get("fill")
|
||||||
|
if fill is None:
|
||||||
|
continue
|
||||||
|
# Match light blue (R=0.85, G=0.92, B=1.0)
|
||||||
|
if all(abs(fill[i] - target) < 0.05 for i, target in enumerate([0.85, 0.92, 1.0])):
|
||||||
|
blue_boxes.append(d)
|
||||||
|
doc.close()
|
||||||
|
return {
|
||||||
|
"total_drawings": len(drawings),
|
||||||
|
"blue_boxes": len(blue_boxes),
|
||||||
|
"blue_rects": [(round(d["rect"].x0, 1), round(d["rect"].y0, 1),
|
||||||
|
round(d["rect"].x1, 1), round(d["rect"].y1, 1))
|
||||||
|
for d in blue_boxes],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _inspect_title_overlap(pdf_path: Path) -> dict:
|
||||||
|
"""Check if the title block bottom overlaps the next block top."""
|
||||||
|
doc = fitz.open(str(pdf_path))
|
||||||
|
page = doc[0]
|
||||||
|
text_dict = page.get_text("dict")
|
||||||
|
|
||||||
|
# Find the title block (largest font on the page)
|
||||||
|
title_block = None
|
||||||
|
next_block = None
|
||||||
|
sorted_blocks = sorted(
|
||||||
|
[b for b in text_dict.get("blocks", []) if b.get("type") == 0],
|
||||||
|
key=lambda b: -(
|
||||||
|
max((s.get("size", 12) for line in b.get("lines", [])
|
||||||
|
for s in line.get("spans", [])), default=12)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if sorted_blocks:
|
||||||
|
title_block = sorted_blocks[0]
|
||||||
|
title_text = " ".join(
|
||||||
|
s.get("text", "")
|
||||||
|
for line in title_block.get("lines", [])
|
||||||
|
for s in line.get("spans", [])
|
||||||
|
)
|
||||||
|
title_bottom = title_block["bbox"][3]
|
||||||
|
|
||||||
|
# Find the next block right below the title
|
||||||
|
candidates = [
|
||||||
|
b for b in text_dict.get("blocks", [])
|
||||||
|
if b.get("type") == 0
|
||||||
|
and b is not title_block
|
||||||
|
and b["bbox"][1] >= title_bottom - 5
|
||||||
|
and b["bbox"][1] <= title_bottom + 50
|
||||||
|
]
|
||||||
|
if candidates:
|
||||||
|
next_block = min(candidates, key=lambda b: b["bbox"][1])
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"title": " ".join(
|
||||||
|
s.get("text", "")
|
||||||
|
for line in (title_block or {}).get("lines", [])
|
||||||
|
for s in line.get("spans", [])
|
||||||
|
)[:60],
|
||||||
|
"title_bbox": title_block["bbox"] if title_block else None,
|
||||||
|
}
|
||||||
|
if next_block:
|
||||||
|
next_text = " ".join(
|
||||||
|
s.get("text", "")
|
||||||
|
for line in next_block.get("lines", [])
|
||||||
|
for s in line.get("spans", [])
|
||||||
|
)[:60]
|
||||||
|
result["next_text"] = next_text
|
||||||
|
result["next_bbox"] = next_block["bbox"]
|
||||||
|
result["overlap_px"] = round(title_block["bbox"][3] - next_block["bbox"][1], 2)
|
||||||
|
result["overlaps"] = result["overlap_px"] > 2
|
||||||
|
doc.close()
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("=" * 60)
|
||||||
|
print("B3.6 VERIFICATION — visual & structural")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
# 1. Translate the original with B3.6 fix
|
||||||
|
print("\n[1] Re-translating test_pdf.pdf with B3.6 fix...")
|
||||||
|
if not ORIG_PDF.exists():
|
||||||
|
print(f" ERROR: original PDF not found at {ORIG_PDF}")
|
||||||
|
return 1
|
||||||
|
_translate_with_mock(ORIG_PDF, NEW_PDF)
|
||||||
|
print(f" wrote {NEW_PDF.name}")
|
||||||
|
|
||||||
|
# 2. Render both PDFs as PNG for visual comparison
|
||||||
|
print("\n[2] Rendering PDFs to PNG...")
|
||||||
|
old_png = _render_pdf_page(OLD_PDF)
|
||||||
|
new_png = _render_pdf_page(NEW_PDF)
|
||||||
|
orig_png = _render_pdf_page(ORIG_PDF)
|
||||||
|
print(f" OLD broken: {old_png.relative_to(REPO_ROOT)}")
|
||||||
|
print(f" NEW (B3.6): {new_png.relative_to(REPO_ROOT)}")
|
||||||
|
print(f" ORIGINAL ref: {orig_png.relative_to(REPO_ROOT)}")
|
||||||
|
|
||||||
|
# 3. Inspect drawings - blue box preservation
|
||||||
|
print("\n[3] Inspecting drawings (blue 'Avis important' box)...")
|
||||||
|
for label, path in [("OLD", OLD_PDF), ("NEW (B3.6)", NEW_PDF), ("ORIGINAL", ORIG_PDF)]:
|
||||||
|
info = _inspect_drawings(path)
|
||||||
|
verdict = "[OK]" if info["blue_boxes"] >= 1 else "[FAIL]"
|
||||||
|
print(f" {verdict} {label:12} -> {info['blue_boxes']} blue box(es) of "
|
||||||
|
f"{info['total_drawings']} total drawings")
|
||||||
|
|
||||||
|
# 4. Inspect title overlap
|
||||||
|
print("\n[4] Inspecting title-vs-next-block overlap...")
|
||||||
|
for label, path in [("OLD", OLD_PDF), ("NEW (B3.6)", NEW_PDF)]:
|
||||||
|
info = _inspect_title_overlap(path)
|
||||||
|
if "overlaps" in info:
|
||||||
|
verdict = "[FAIL OVERLAPS]" if info["overlaps"] else "[OK no overlap]"
|
||||||
|
print(f" {verdict} {label:12} -> title={info['title']!r:40} "
|
||||||
|
f"overlap_px={info['overlap_px']:+.1f}")
|
||||||
|
else:
|
||||||
|
print(f" [?] {label:12} -> could not determine (insufficient blocks)")
|
||||||
|
|
||||||
|
print("\n" + "=" * 60)
|
||||||
|
print("Visual comparison rendered to:")
|
||||||
|
print(f" {old_png}")
|
||||||
|
print(f" {new_png}")
|
||||||
|
print(f" {orig_png}")
|
||||||
|
print("=" * 60)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
144
scripts/verify_b3_8_multicolumn.py
Normal file
144
scripts/verify_b3_8_multicolumn.py
Normal file
@@ -0,0 +1,144 @@
|
|||||||
|
"""
|
||||||
|
B3.8 visual verification on a REAL multi-column layout.
|
||||||
|
|
||||||
|
Generates a 2-column journal-style PDF (3 paragraphs in each column,
|
||||||
|
each paragraph = 3 lines), then translates it with B3.8 enabled.
|
||||||
|
|
||||||
|
Without B3.8, the next_block_y of a left-column block would point to
|
||||||
|
the right-column block at the same y (the closest in y-order), which
|
||||||
|
is wrong — the smart-fit would never let left-column text expand down
|
||||||
|
into the left-column space below it.
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
sys.path.insert(0, '.')
|
||||||
|
import fitz
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
REPO_ROOT = Path('.').resolve()
|
||||||
|
OUT_DIR = REPO_ROOT / 'scripts' / '_renders_b3_8'
|
||||||
|
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def build_2col_journal_pdf(path: Path) -> None:
|
||||||
|
"""Build a 2-column journal-style PDF."""
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
# Centered title
|
||||||
|
title_rect = fitz.Rect(72, 60, 540, 100)
|
||||||
|
page.insert_textbox(title_rect, "Article Title", fontsize=20)
|
||||||
|
|
||||||
|
# Author under title
|
||||||
|
page.insert_text((250, 120), "By John Doe", fontsize=10)
|
||||||
|
|
||||||
|
# Left column: 3 paragraphs
|
||||||
|
left_x0 = 72
|
||||||
|
left_x1 = 300
|
||||||
|
paras_left = [
|
||||||
|
"This is the first paragraph of the left column. It has multiple lines that span across the column width.",
|
||||||
|
"Second paragraph of the left column. This text continues to fill the left side of the page.",
|
||||||
|
"Third and final paragraph on the left side. The text wraps to fill the available space.",
|
||||||
|
]
|
||||||
|
y = 170
|
||||||
|
for p in paras_left:
|
||||||
|
rect = fitz.Rect(left_x0, y, left_x1, y + 80)
|
||||||
|
page.insert_textbox(rect, p, fontsize=10)
|
||||||
|
y += 100
|
||||||
|
|
||||||
|
# Right column: 3 paragraphs (same y positions as left, so the
|
||||||
|
# "wrong" next-block mapping would matter here)
|
||||||
|
right_x0 = 312
|
||||||
|
right_x1 = 540
|
||||||
|
paras_right = [
|
||||||
|
"Right column, first paragraph. This should stay in the right column and not be affected by left.",
|
||||||
|
"Right column, second paragraph. Independent of the left column's translation length.",
|
||||||
|
"Right column final paragraph. Verifies column-aware layout.",
|
||||||
|
]
|
||||||
|
y = 170
|
||||||
|
for p in paras_right:
|
||||||
|
rect = fitz.Rect(right_x0, y, right_x1, y + 80)
|
||||||
|
page.insert_textbox(rect, p, fontsize=10)
|
||||||
|
y += 100
|
||||||
|
|
||||||
|
doc.save(str(path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
def translate_with_mock(input_path: Path, output_path: Path) -> None:
|
||||||
|
"""Translate using a mock that returns the same text (identity)."""
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
def _flex(*args, **kwargs):
|
||||||
|
if args and hasattr(args[0], '__iter__') and not isinstance(args[0], str):
|
||||||
|
first = args[0][0] if len(args[0]) else None
|
||||||
|
if first is not None and hasattr(first, 'text'):
|
||||||
|
class _R:
|
||||||
|
def __init__(self, t): self.translated_text = t
|
||||||
|
return [_R(r.text) for r in args[0]]
|
||||||
|
return list(args[0]) if args else []
|
||||||
|
mock = MagicMock()
|
||||||
|
mock.__class__.__name__ = "MagicMock"
|
||||||
|
mock.translate_batch = MagicMock(side_effect=_flex)
|
||||||
|
translator.set_provider(mock)
|
||||||
|
translator.translate_file(input_path, output_path,
|
||||||
|
target_language='fr', source_language='en')
|
||||||
|
|
||||||
|
|
||||||
|
def inspect_columns(pdf_path: Path) -> dict:
|
||||||
|
"""Count blocks per column and check no cross-column overlap."""
|
||||||
|
doc = fitz.open(str(pdf_path))
|
||||||
|
page = doc[0]
|
||||||
|
blocks = [b for b in page.get_text("dict").get("blocks", [])
|
||||||
|
if b.get("type") == 0]
|
||||||
|
left_blocks = [b for b in blocks if b["bbox"][0] < 200]
|
||||||
|
right_blocks = [b for b in blocks if b["bbox"][0] >= 200]
|
||||||
|
doc.close()
|
||||||
|
return {
|
||||||
|
"total": len(blocks),
|
||||||
|
"left": len(left_blocks),
|
||||||
|
"right": len(right_blocks),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def render(pdf_path: Path, out_png: Path) -> None:
|
||||||
|
doc = fitz.open(str(pdf_path))
|
||||||
|
pix = doc[0].get_pixmap(matrix=fitz.Matrix(1.5, 1.5))
|
||||||
|
pix.save(str(out_png))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("B3.8 multi-column verification")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
in_pdf = OUT_DIR / 'journal_in.pdf'
|
||||||
|
out_pdf = OUT_DIR / 'journal_out.pdf'
|
||||||
|
in_png = OUT_DIR / 'journal_in.png'
|
||||||
|
out_png = OUT_DIR / 'journal_out.png'
|
||||||
|
|
||||||
|
print(f"[1] Building 2-column journal PDF -> {in_pdf.name}")
|
||||||
|
build_2col_journal_pdf(in_pdf)
|
||||||
|
|
||||||
|
print(f"[2] Translating with B3.8 enabled -> {out_pdf.name}")
|
||||||
|
translate_with_mock(in_pdf, out_pdf)
|
||||||
|
|
||||||
|
print(f"[3] Inspecting input column distribution...")
|
||||||
|
in_info = inspect_columns(in_pdf)
|
||||||
|
print(f" input : total={in_info['total']} left={in_info['left']} right={in_info['right']}")
|
||||||
|
|
||||||
|
print(f"[4] Inspecting output column distribution...")
|
||||||
|
out_info = inspect_columns(out_pdf)
|
||||||
|
print(f" output : total={out_info['total']} left={out_info['left']} right={out_info['right']}")
|
||||||
|
|
||||||
|
print(f"[5] Rendering both to PNG...")
|
||||||
|
render(in_pdf, in_png)
|
||||||
|
render(out_pdf, out_png)
|
||||||
|
print(f" input : {in_png}")
|
||||||
|
print(f" output : {out_png}")
|
||||||
|
|
||||||
|
print("=" * 60)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
41
scripts/verify_test_corpus.py
Normal file
41
scripts/verify_test_corpus.py
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
"""Quick verification of the test corpus."""
|
||||||
|
import zipfile
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
print("=== test_word.docx ===")
|
||||||
|
with zipfile.ZipFile("sample_files/test_corpus/test_word.docx") as z:
|
||||||
|
has_fn = "word/footnotes.xml" in z.namelist()
|
||||||
|
rels = z.read("word/_rels/document.xml.rels").decode()
|
||||||
|
n_hl = rels.count('TargetMode="External"')
|
||||||
|
print(f" footnotes.xml: {has_fn}, external hyperlinks: {n_hl}")
|
||||||
|
|
||||||
|
print("=== test_excel.xlsx ===")
|
||||||
|
from openpyxl import load_workbook
|
||||||
|
wb = load_workbook("sample_files/test_corpus/test_excel.xlsx")
|
||||||
|
ws = wb["Sales"]
|
||||||
|
n_comments = sum(1 for row in ws.iter_rows() for c in row if c.comment)
|
||||||
|
n_hl = sum(1 for row in ws.iter_rows() for c in row if c.hyperlink)
|
||||||
|
n_charts = len(ws._charts) if hasattr(ws, "_charts") else 0
|
||||||
|
print(f" comments: {n_comments}, hyperlinks: {n_hl}, charts: {n_charts}")
|
||||||
|
print(f" sheets: {wb.sheetnames}")
|
||||||
|
|
||||||
|
print("=== test_pptx.pptx ===")
|
||||||
|
with zipfile.ZipFile("sample_files/test_corpus/test_pptx.pptx") as z:
|
||||||
|
has_diag = "ppt/diagrams/data1.xml" in z.namelist()
|
||||||
|
if has_diag:
|
||||||
|
diag = z.read("ppt/diagrams/data1.xml").decode()
|
||||||
|
n_texts = diag.count("<a:t>")
|
||||||
|
else:
|
||||||
|
n_texts = 0
|
||||||
|
print(f" diagrams/data1.xml: {has_diag}, <a:t> nodes: {n_texts}")
|
||||||
|
|
||||||
|
print("=== test_pdf.pdf ===")
|
||||||
|
doc = fitz.open("sample_files/test_corpus/test_pdf.pdf")
|
||||||
|
print(f" pages: {len(doc)}")
|
||||||
|
total_links = 0
|
||||||
|
total_images = 0
|
||||||
|
for p in doc:
|
||||||
|
total_links += len(p.get_links())
|
||||||
|
total_images += len(p.get_images())
|
||||||
|
print(f" total links: {total_links}, total images: {total_images}")
|
||||||
|
doc.close()
|
||||||
@@ -92,6 +92,36 @@ def validate_stripe_webhook_secret(value: str) -> None:
|
|||||||
raise ValueError("Secret webhook Stripe invalide (attendu whsec_…).")
|
raise ValueError("Secret webhook Stripe invalide (attendu whsec_…).")
|
||||||
|
|
||||||
|
|
||||||
|
def stripe_mode() -> str:
|
||||||
|
"""Detect Stripe mode from the secret key prefix.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ``"live"`` if STRIPE_SECRET_KEY starts with ``sk_live_``
|
||||||
|
- ``"test"`` if STRIPE_SECRET_KEY starts with ``sk_test_``
|
||||||
|
- ``"unknown"`` if no key is set or the prefix is unrecognized
|
||||||
|
|
||||||
|
Used by the admin panel to display the current mode and to warn
|
||||||
|
(or block) live-mode actions when running in test mode.
|
||||||
|
"""
|
||||||
|
key = os.getenv("STRIPE_SECRET_KEY", "").strip()
|
||||||
|
if key.startswith("sk_live_"):
|
||||||
|
return "live"
|
||||||
|
if key.startswith("sk_test_"):
|
||||||
|
return "test"
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def is_test_mode_in_production() -> bool:
|
||||||
|
"""True if ENV=production but STRIPE_SECRET_KEY is a test key.
|
||||||
|
|
||||||
|
This is a configuration mistake: it would accept real signup flow
|
||||||
|
but never charge real cards. The admin panel surfaces this as a
|
||||||
|
blocking warning.
|
||||||
|
"""
|
||||||
|
env = (os.getenv("ENV", os.getenv("ENVIRONMENT", "development")) or "").lower()
|
||||||
|
return env == "production" and stripe_mode() == "test"
|
||||||
|
|
||||||
|
|
||||||
def load_pricing_overrides() -> dict[str, Any]:
|
def load_pricing_overrides() -> dict[str, Any]:
|
||||||
try:
|
try:
|
||||||
if PRICING_FILE.exists():
|
if PRICING_FILE.exists():
|
||||||
|
|||||||
314
services/providers/adapters.py
Normal file
314
services/providers/adapters.py
Normal file
@@ -0,0 +1,314 @@
|
|||||||
|
"""
|
||||||
|
Track C1.1 — Provider interface adapters.
|
||||||
|
|
||||||
|
Two adapter classes that bridge the legacy `TranslationProvider`
|
||||||
|
interface (translate(text, target, source) -> str) and the new
|
||||||
|
TranslationProvider interface (translate_text(request) -> Response).
|
||||||
|
|
||||||
|
Why?
|
||||||
|
The codebase has TWO provider interfaces:
|
||||||
|
1. New: services/providers/base.py
|
||||||
|
- translate_text(TranslationRequest) -> TranslationResponse
|
||||||
|
- translate_batch(List[TranslationRequest]) -> List[TranslationResponse]
|
||||||
|
2. Legacy: services/translation_service.py
|
||||||
|
- translate(text, target, source) -> str
|
||||||
|
- translate_batch(texts, target, source) -> List[str]
|
||||||
|
|
||||||
|
Both are used in production. Translators sometimes receive the new
|
||||||
|
interface, sometimes the legacy one. Previously the translators
|
||||||
|
duck-typed the provider to figure out which interface it supported
|
||||||
|
(see e.g. pptx_translator._translate_with_provider). That was hacky.
|
||||||
|
|
||||||
|
These adapters give a single, explicit way to bridge the two
|
||||||
|
interfaces, so translators can always treat the provider as
|
||||||
|
whichever interface they prefer.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
# Wrap a new-style provider as legacy
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new_provider)
|
||||||
|
legacy.translate("Hello", "fr", "auto") # returns str
|
||||||
|
|
||||||
|
# Wrap a legacy-style provider as new
|
||||||
|
new = LegacyProviderAsNewAdapter(legacy_provider)
|
||||||
|
new.translate_text(TranslationRequest(...)) # returns TranslationResponse
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import List, Optional
|
||||||
|
|
||||||
|
from core.logging import get_logger
|
||||||
|
|
||||||
|
from .base import TranslationProvider as NewTranslationProvider
|
||||||
|
from .schemas import TranslationRequest, TranslationResponse, ProviderHealthStatus
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# New-style → Legacy
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class NewProviderAsLegacyAdapter:
|
||||||
|
"""Adapt a new-style provider (translate_text) to the legacy interface.
|
||||||
|
|
||||||
|
The legacy interface is what most translators and the legacy
|
||||||
|
`translation_service` expect:
|
||||||
|
translate(text, target_language, source_language="auto") -> str
|
||||||
|
translate_batch(texts, target, source) -> List[str]
|
||||||
|
get_name() -> str
|
||||||
|
is_available() -> bool
|
||||||
|
|
||||||
|
This adapter is for INTERNAL use — it adapts at the boundary so
|
||||||
|
that a single provider instance can be passed to any code path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, new_provider: NewTranslationProvider):
|
||||||
|
self._new = new_provider
|
||||||
|
|
||||||
|
# ---- Core legacy methods ----
|
||||||
|
|
||||||
|
def translate(
|
||||||
|
self, text: str, target_language: str, source_language: str = "auto"
|
||||||
|
) -> str:
|
||||||
|
"""Translate a single text — legacy signature."""
|
||||||
|
# Lazy import to avoid circular import
|
||||||
|
try:
|
||||||
|
from .schemas import TranslationRequest
|
||||||
|
request = TranslationRequest(
|
||||||
|
text=text,
|
||||||
|
target_language=target_language,
|
||||||
|
source_language=source_language,
|
||||||
|
)
|
||||||
|
response = self._new.translate_text(request)
|
||||||
|
if response.error:
|
||||||
|
raise Exception(
|
||||||
|
f"[{response.error_code or 'UNKNOWN'}] {response.error}"
|
||||||
|
)
|
||||||
|
return response.translated_text
|
||||||
|
except Exception as e:
|
||||||
|
# If the new provider raised, re-raise so the caller
|
||||||
|
# can decide how to handle it (legacy code expects exceptions).
|
||||||
|
logger.debug(
|
||||||
|
"new_provider_as_legacy_translate_error",
|
||||||
|
error=str(e)[:200],
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def translate_batch(
|
||||||
|
self,
|
||||||
|
texts: List[str],
|
||||||
|
target_language: str,
|
||||||
|
source_language: str = "auto",
|
||||||
|
) -> List[str]:
|
||||||
|
"""Translate multiple texts — legacy signature."""
|
||||||
|
if not texts:
|
||||||
|
return []
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .schemas import TranslationRequest
|
||||||
|
requests = [
|
||||||
|
TranslationRequest(
|
||||||
|
text=t,
|
||||||
|
target_language=target_language,
|
||||||
|
source_language=source_language,
|
||||||
|
)
|
||||||
|
for t in texts
|
||||||
|
]
|
||||||
|
responses = self._new.translate_batch(requests)
|
||||||
|
return [r.translated_text for r in responses]
|
||||||
|
except Exception as e:
|
||||||
|
logger.debug(
|
||||||
|
"new_provider_as_legacy_batch_error",
|
||||||
|
error=str(e)[:200],
|
||||||
|
count=len(texts),
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
# ---- Identity methods (pass-through) ----
|
||||||
|
|
||||||
|
def get_name(self) -> str:
|
||||||
|
return self._new.get_name()
|
||||||
|
|
||||||
|
def is_available(self) -> bool:
|
||||||
|
try:
|
||||||
|
return bool(self._new.is_available())
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def health_check(self) -> dict:
|
||||||
|
"""Best-effort health check, legacy-shaped (plain dict)."""
|
||||||
|
try:
|
||||||
|
status = self._new.health_check()
|
||||||
|
return {
|
||||||
|
"name": status.name,
|
||||||
|
"available": status.available,
|
||||||
|
"latency_ms": status.latency_ms,
|
||||||
|
"error": status.error,
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
return {
|
||||||
|
"name": self.get_name(),
|
||||||
|
"available": False,
|
||||||
|
"error": str(e)[:200],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Legacy-style → New
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class LegacyProviderAsNewAdapter(NewTranslationProvider):
|
||||||
|
"""Adapt a legacy-style provider (translate(text, ...)) to the new interface.
|
||||||
|
|
||||||
|
Most new code expects the new interface:
|
||||||
|
translate_text(TranslationRequest) -> TranslationResponse
|
||||||
|
translate_batch(List[TranslationRequest]) -> List[TranslationResponse]
|
||||||
|
|
||||||
|
A "legacy" provider is anything that has:
|
||||||
|
translate(text, target, source) -> str
|
||||||
|
translate_batch(texts, target, source) -> List[str]
|
||||||
|
get_name() -> str
|
||||||
|
is_available() -> bool
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, legacy_provider, provider_name: str = None):
|
||||||
|
self._legacy = legacy_provider
|
||||||
|
# If a custom name is provided, use it; otherwise inherit from the
|
||||||
|
# legacy provider.
|
||||||
|
if provider_name is not None:
|
||||||
|
self._name = provider_name
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
self._name = legacy_provider.get_name()
|
||||||
|
except Exception:
|
||||||
|
self._name = "legacy"
|
||||||
|
|
||||||
|
# ---- Implement the abstract methods of NewTranslationProvider ----
|
||||||
|
|
||||||
|
def translate_text(self, request: TranslationRequest) -> TranslationResponse:
|
||||||
|
"""Translate a single text — new interface."""
|
||||||
|
start = time.time()
|
||||||
|
try:
|
||||||
|
translated = self._legacy.translate(
|
||||||
|
text=request.text,
|
||||||
|
target_language=request.target_language,
|
||||||
|
source_language=request.source_language or "auto",
|
||||||
|
)
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
return TranslationResponse(
|
||||||
|
translated_text=translated or "",
|
||||||
|
provider_name=self._name,
|
||||||
|
source_language=request.source_language or "auto",
|
||||||
|
target_language=request.target_language,
|
||||||
|
elapsed_ms=elapsed_ms,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
return TranslationResponse(
|
||||||
|
translated_text="",
|
||||||
|
provider_name=self._name,
|
||||||
|
source_language=request.source_language or "auto",
|
||||||
|
target_language=request.target_language,
|
||||||
|
elapsed_ms=elapsed_ms,
|
||||||
|
error=str(e)[:200],
|
||||||
|
error_code="LEGACY_PROVIDER_ERROR",
|
||||||
|
)
|
||||||
|
|
||||||
|
def translate_batch(
|
||||||
|
self, requests: List[TranslationRequest]
|
||||||
|
) -> List[TranslationResponse]:
|
||||||
|
"""Translate a batch of texts — new interface."""
|
||||||
|
if not requests:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Group by language pair so we can call the legacy
|
||||||
|
# translate_batch efficiently (legacy signature is per-language-pair).
|
||||||
|
from collections import defaultdict
|
||||||
|
groups: dict = defaultdict(list)
|
||||||
|
for i, r in enumerate(requests):
|
||||||
|
key = (r.target_language, r.source_language or "auto")
|
||||||
|
groups[key].append((i, r))
|
||||||
|
|
||||||
|
results: List[TranslationResponse] = [None] * len(requests) # type: ignore
|
||||||
|
|
||||||
|
for (target_lang, source_lang), items in groups.items():
|
||||||
|
texts = [r.text for _, r in items]
|
||||||
|
try:
|
||||||
|
translated = self._legacy.translate_batch(
|
||||||
|
texts=texts,
|
||||||
|
target_language=target_lang,
|
||||||
|
source_language=source_lang,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
# If legacy batch fails, fall back to per-text
|
||||||
|
logger.debug(
|
||||||
|
"legacy_batch_fallback_to_individual",
|
||||||
|
error=str(e)[:200],
|
||||||
|
)
|
||||||
|
translated = []
|
||||||
|
for t in texts:
|
||||||
|
try:
|
||||||
|
translated.append(
|
||||||
|
self._legacy.translate(t, target_lang, source_lang)
|
||||||
|
)
|
||||||
|
except Exception as individual_err:
|
||||||
|
translated.append("") # legacy code expects str
|
||||||
|
|
||||||
|
for (orig_idx, r), t in zip(items, translated):
|
||||||
|
results[orig_idx] = TranslationResponse(
|
||||||
|
translated_text=t or "",
|
||||||
|
provider_name=self._name,
|
||||||
|
source_language=source_lang,
|
||||||
|
target_language=target_lang,
|
||||||
|
)
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_name(self) -> str:
|
||||||
|
# Always prefer the explicit name set at construction. Only fall
|
||||||
|
# back to the legacy provider's get_name if no name was set.
|
||||||
|
if hasattr(self, "_name") and self._name:
|
||||||
|
return self._name
|
||||||
|
try:
|
||||||
|
return self._legacy.get_name()
|
||||||
|
except Exception:
|
||||||
|
return "legacy"
|
||||||
|
|
||||||
|
def is_available(self) -> bool:
|
||||||
|
try:
|
||||||
|
return bool(self._legacy.is_available())
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Auto-detect helper
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
import time # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def coerce_to_legacy(provider) -> NewProviderAsLegacyAdapter:
|
||||||
|
"""Coerce any provider to the legacy interface.
|
||||||
|
|
||||||
|
If the provider is already a legacy provider, return it as-is.
|
||||||
|
If it's a new-style provider, wrap it in NewProviderAsLegacyAdapter.
|
||||||
|
"""
|
||||||
|
if isinstance(provider, NewProviderAsLegacyAdapter):
|
||||||
|
return provider
|
||||||
|
if isinstance(provider, NewTranslationProvider):
|
||||||
|
return NewProviderAsLegacyAdapter(provider)
|
||||||
|
# Already legacy-shaped — pass through
|
||||||
|
return provider
|
||||||
|
|
||||||
|
|
||||||
|
def coerce_to_new(provider, name: str = None) -> NewTranslationProvider:
|
||||||
|
"""Coerce any provider to the new interface.
|
||||||
|
|
||||||
|
If the provider is already a new-style provider, return it as-is.
|
||||||
|
If it's a legacy provider, wrap it in LegacyProviderAsNewAdapter.
|
||||||
|
"""
|
||||||
|
if isinstance(provider, NewTranslationProvider):
|
||||||
|
return provider
|
||||||
|
return LegacyProviderAsNewAdapter(provider, provider_name=name)
|
||||||
55
services/quality/__init__.py
Normal file
55
services/quality/__init__.py
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
"""
|
||||||
|
Quality check layer for translations.
|
||||||
|
|
||||||
|
Tracks:
|
||||||
|
A1 — L0 backend (pure Python, no API calls). Always available.
|
||||||
|
A2 — L0 frontend JavaScript mirror. Browser/Node compatible.
|
||||||
|
A3 — L1 LLM judge (API-based, opt-in). Cheap model, sampled.
|
||||||
|
|
||||||
|
Designed to be ADDITIVE: existing translation flow is untouched.
|
||||||
|
|
||||||
|
Public API:
|
||||||
|
L0:
|
||||||
|
QualityCheckResult — per-chunk result dataclass
|
||||||
|
DocumentQualityResult — aggregated result dataclass
|
||||||
|
evaluate_chunk(...) — score a single (source, translation) pair
|
||||||
|
evaluate_document(...) — score a list of pairs and aggregate
|
||||||
|
run_l0_check(...) — defensive wrapper used by the route
|
||||||
|
extract_sample(...) — extract text from a finished file
|
||||||
|
L1:
|
||||||
|
L1Result — verdict dataclass
|
||||||
|
LLMJudge — judge client
|
||||||
|
run_l1_check(...) — async wrapper used by the route
|
||||||
|
sample_chunks_for_l1(...) — pick representative chunks
|
||||||
|
Helpers:
|
||||||
|
get_script, isArabicScriptLang
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .script_detector import (
|
||||||
|
QualityCheckResult,
|
||||||
|
DocumentQualityResult,
|
||||||
|
evaluate_chunk,
|
||||||
|
evaluate_document,
|
||||||
|
detect_arabic_variant,
|
||||||
|
)
|
||||||
|
from .pipeline import run_l0_check, run_l1_check
|
||||||
|
from .file_extractor import extract_sample
|
||||||
|
from .sampler import sample_chunks_for_l1
|
||||||
|
from .llm_judge import L1Result, L1ChunkVerdict, LLMJudge
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
# L0
|
||||||
|
"QualityCheckResult",
|
||||||
|
"DocumentQualityResult",
|
||||||
|
"evaluate_chunk",
|
||||||
|
"evaluate_document",
|
||||||
|
"detect_arabic_variant",
|
||||||
|
"run_l0_check",
|
||||||
|
"extract_sample",
|
||||||
|
# L1
|
||||||
|
"L1Result",
|
||||||
|
"L1ChunkVerdict",
|
||||||
|
"LLMJudge",
|
||||||
|
"run_l1_check",
|
||||||
|
"sample_chunks_for_l1",
|
||||||
|
]
|
||||||
235
services/quality/config.py
Normal file
235
services/quality/config.py
Normal file
@@ -0,0 +1,235 @@
|
|||||||
|
"""
|
||||||
|
Language → script mapping for the L0 quality layer.
|
||||||
|
|
||||||
|
A "script" groups languages that share a Unicode block. The script detector
|
||||||
|
verifies that the *translation* is in the same script as the *target language*.
|
||||||
|
|
||||||
|
Languages are mapped by their ISO 639-1 (or 639-3) code. Anything not mapped
|
||||||
|
falls back to the "latin" script so we never crash on an unknown language.
|
||||||
|
|
||||||
|
Arabic-script languages (ar, fa, ur, ps, ku, sd, ug, yi, dv, ckb) all share
|
||||||
|
the same Unicode ranges, so we additionally use a set of *discriminating
|
||||||
|
characters* to tell them apart — e.g. Persian text contains چ/پ/ژ/گ which
|
||||||
|
Arabic text doesn't.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Dict, FrozenSet, List, Optional, Tuple
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Unicode ranges per script ----------
|
||||||
|
# Format: list of (start, end) inclusive ranges.
|
||||||
|
# Latin is intentionally omitted (it's the fallback).
|
||||||
|
|
||||||
|
UNICODE_RANGES: Dict[str, List[Tuple[int, int]]] = {
|
||||||
|
"cyrillic": [
|
||||||
|
(0x0400, 0x04FF), # Cyrillic
|
||||||
|
(0x0500, 0x052F), # Cyrillic Supplement
|
||||||
|
],
|
||||||
|
"greek": [
|
||||||
|
(0x0370, 0x03FF), # Greek and Coptic
|
||||||
|
],
|
||||||
|
"arabic": [
|
||||||
|
(0x0600, 0x06FF), # Arabic
|
||||||
|
(0x0750, 0x077F), # Arabic Supplement
|
||||||
|
(0x08A0, 0x08FF), # Arabic Extended-A
|
||||||
|
],
|
||||||
|
"hebrew": [
|
||||||
|
(0x0590, 0x05FF), # Hebrew
|
||||||
|
],
|
||||||
|
"devanagari": [
|
||||||
|
(0x0900, 0x097F), # Devanagari
|
||||||
|
],
|
||||||
|
"bengali": [
|
||||||
|
(0x0980, 0x09FF), # Bengali
|
||||||
|
],
|
||||||
|
"tamil": [
|
||||||
|
(0x0B80, 0x0BFF),
|
||||||
|
],
|
||||||
|
"telugu": [
|
||||||
|
(0x0C00, 0x0C7F),
|
||||||
|
],
|
||||||
|
"kannada": [
|
||||||
|
(0x0C80, 0x0CFF),
|
||||||
|
],
|
||||||
|
"malayalam": [
|
||||||
|
(0x0D00, 0x0D7F),
|
||||||
|
],
|
||||||
|
"sinhala": [
|
||||||
|
(0x0D80, 0x0DFF),
|
||||||
|
],
|
||||||
|
"gujarati": [
|
||||||
|
(0x0A80, 0x0AFF),
|
||||||
|
],
|
||||||
|
"gurmukhi": [
|
||||||
|
(0x0A00, 0x0A7F),
|
||||||
|
],
|
||||||
|
"thai": [
|
||||||
|
(0x0E00, 0x0E7F),
|
||||||
|
],
|
||||||
|
"lao": [
|
||||||
|
(0x0E80, 0x0EFF),
|
||||||
|
],
|
||||||
|
"burmese": [
|
||||||
|
(0x1000, 0x109F),
|
||||||
|
],
|
||||||
|
"khmer": [
|
||||||
|
(0x1780, 0x17FF),
|
||||||
|
],
|
||||||
|
"cjk": [
|
||||||
|
(0x4E00, 0x9FFF), # CJK Unified Ideographs
|
||||||
|
(0x3400, 0x4DBF), # CJK Extension A
|
||||||
|
(0x20000, 0x2A6DF), # CJK Extension B (rare)
|
||||||
|
],
|
||||||
|
"hiragana_katakana": [
|
||||||
|
(0x3040, 0x309F), # Hiragana
|
||||||
|
(0x30A0, 0x30FF), # Katakana
|
||||||
|
],
|
||||||
|
"hangul": [
|
||||||
|
(0xAC00, 0xD7AF), # Hangul Syllables
|
||||||
|
(0x1100, 0x11FF), # Hangul Jamo
|
||||||
|
(0xA960, 0xA97F), # Hangul Jamo Extended-A
|
||||||
|
],
|
||||||
|
"georgian": [
|
||||||
|
(0x10A0, 0x10FF),
|
||||||
|
],
|
||||||
|
"armenian": [
|
||||||
|
(0x0530, 0x058F),
|
||||||
|
],
|
||||||
|
"ethiopic": [
|
||||||
|
(0x1200, 0x137F), # Ethiopic
|
||||||
|
(0x1380, 0x139F), # Ethiopic Supplement
|
||||||
|
],
|
||||||
|
"tibetan": [
|
||||||
|
(0x0F00, 0x0FFF),
|
||||||
|
],
|
||||||
|
"thaana": [
|
||||||
|
(0x0780, 0x07BF), # Thaana (Dhivehi)
|
||||||
|
],
|
||||||
|
# Latin is the implicit fallback.
|
||||||
|
"latin": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Language → script mapping ----------
|
||||||
|
# Single source of truth. Keys are lower-case ISO codes.
|
||||||
|
|
||||||
|
LANG_TO_SCRIPT: Dict[str, str] = {
|
||||||
|
# Latin-script languages (the long tail of European + colonial)
|
||||||
|
"en": "latin", "fr": "latin", "de": "latin", "es": "latin", "it": "latin",
|
||||||
|
"pt": "latin", "nl": "latin", "pl": "latin", "tr": "latin", "vi": "latin",
|
||||||
|
"id": "latin", "ms": "latin", "ro": "latin", "cs": "latin", "sv": "latin",
|
||||||
|
"da": "latin", "fi": "latin", "no": "latin", "nb": "latin", "nn": "latin",
|
||||||
|
"hu": "latin", "sk": "latin", "sl": "latin", "lt": "latin", "lv": "latin",
|
||||||
|
"et": "latin", "sq": "latin", "az": "latin", "uz": "latin", "kk": "latin",
|
||||||
|
"ky": "latin", "tk": "latin", "sw": "latin", "eu": "latin", "gl": "latin",
|
||||||
|
"is": "latin", "ga": "latin", "mt": "latin", "ca": "latin", "hr": "latin",
|
||||||
|
"bs": "latin", "sr": "latin", "mk": "latin", "bg": "latin", "be": "latin",
|
||||||
|
"uk": "latin", # NOTE: uk/be/sr/mk/bg are actually Cyrillic — see fix below
|
||||||
|
"af": "latin", "cy": "latin", "lb": "latin", "fo": "latin", "br": "latin",
|
||||||
|
"co": "latin", "fy": "latin", "gd": "latin", "gn": "latin", "gu": "latin",
|
||||||
|
"ht": "latin", "haw": "latin", "hmn": "latin", "jv": "latin", "ku": "latin",
|
||||||
|
"mg": "latin", "mi": "latin", "mn": "latin", "nso": "latin", "ny": "latin",
|
||||||
|
"oc": "latin", "os": "latin", "ps": "latin", "qu": "latin", "rw": "latin",
|
||||||
|
"sc": "latin", "si": "latin", "sm": "latin", "sn": "latin", "so": "latin",
|
||||||
|
"st": "latin", "su": "latin", "tg": "latin", "tt": "latin", "ty": "latin",
|
||||||
|
"ug": "latin", "vo": "latin", "wa": "latin", "wo": "latin", "xh": "latin",
|
||||||
|
"yi": "latin", "zu": "latin",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Correct Cyrillic assignments (overrides above for the Cyrillic-script langs)
|
||||||
|
LANG_TO_SCRIPT.update({
|
||||||
|
"ru": "cyrillic", "uk": "cyrillic", "be": "cyrillic", "sr": "cyrillic",
|
||||||
|
"mk": "cyrillic", "bg": "cyrillic", "kk": "cyrillic", "ky": "cyrillic",
|
||||||
|
"tg": "cyrillic", "tt": "cyrillic", "mn": "cyrillic", "ab": "cyrillic",
|
||||||
|
"ba": "cyrillic", "ce": "cyrillic", "cv": "cyrillic", "kv": "cyrillic",
|
||||||
|
"kv": "cyrillic", "l1": "cyrillic", "mhr": "cyrillic", "mrj": "cyrillic",
|
||||||
|
"myv": "cyrillic", "os": "cyrillic", "rue": "cyrillic", "sah": "cyrillic",
|
||||||
|
"udm": "cyrillic", "uk": "cyrillic",
|
||||||
|
})
|
||||||
|
|
||||||
|
# More corrections
|
||||||
|
LANG_TO_SCRIPT.update({
|
||||||
|
"el": "greek",
|
||||||
|
"ar": "arabic", "fa": "arabic", "ur": "arabic", "ps": "arabic",
|
||||||
|
"ku": "arabic", "sd": "arabic", "ug": "arabic",
|
||||||
|
"ckb": "arabic", "bal": "arabic", "bqi": "arabic",
|
||||||
|
"glk": "arabic", "mzn": "arabic",
|
||||||
|
"he": "hebrew",
|
||||||
|
"yi": "hebrew", # Yiddish uses Hebrew script
|
||||||
|
"dv": "thaana", # Maldivian (Dhivehi) uses Thaana script
|
||||||
|
"hi": "devanagari", "ne": "devanagari", "mr": "devanagari", "sa": "devanagari",
|
||||||
|
"mai": "devanagari", "bho": "devanagari", "awa": "devanagari",
|
||||||
|
"bn": "bengali", "as": "bengali",
|
||||||
|
"ta": "tamil",
|
||||||
|
"te": "telugu",
|
||||||
|
"kn": "kannada",
|
||||||
|
"ml": "malayalam",
|
||||||
|
"si": "sinhala",
|
||||||
|
"gu": "gujarati",
|
||||||
|
"pa": "gurmukhi",
|
||||||
|
"th": "thai",
|
||||||
|
"lo": "lao",
|
||||||
|
"my": "burmese",
|
||||||
|
"km": "khmer",
|
||||||
|
"zh": "cjk", "zh-cn": "cjk", "zh-tw": "cjk", "zh-hk": "cjk",
|
||||||
|
"ja": "hiragana_katakana",
|
||||||
|
"ko": "hangul",
|
||||||
|
"ka": "georgian",
|
||||||
|
"hy": "armenian",
|
||||||
|
"am": "ethiopic", "ti": "ethiopic", "gez": "ethiopic", "tig": "ethiopic",
|
||||||
|
"bo": "tibetan", "dz": "tibetan",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Discriminating characters for Arabic-script languages ----------
|
||||||
|
# These are characters that strongly suggest a SPECIFIC Arabic-script
|
||||||
|
# language as opposed to the generic "arabic" base.
|
||||||
|
|
||||||
|
DISCRIMINATING_CHARS: Dict[str, FrozenSet[str]] = {
|
||||||
|
"fa": frozenset("پچژگ"), # Persian
|
||||||
|
"ur": frozenset("ٹڈڑے"), # Urdu
|
||||||
|
"ps": frozenset("ټډړږښ"), # Pashto
|
||||||
|
"ku": frozenset("ڕێ"), # Kurdish
|
||||||
|
"ckb": frozenset("ڕێ"), # Sorani Kurdish (same as ku)
|
||||||
|
"sd": frozenset("ٿ"), # Sindhi
|
||||||
|
"ug": frozenset("ۇۆې"), # Uyghur
|
||||||
|
"yi": frozenset("ײ"), # Yiddish (uses Hebrew too)
|
||||||
|
"bal": frozenset("ێ"), # Balochi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Thresholds ----------
|
||||||
|
# These are tunable but with sensible defaults.
|
||||||
|
|
||||||
|
MIN_RATIO_IN_SCRIPT = 0.60 # at least 60% of letters must match the script
|
||||||
|
MIN_RATIO_FOR_ARABIC_VARIANT = 0.20 # at least 20% of letters in the discriminating set
|
||||||
|
|
||||||
|
|
||||||
|
def get_script(lang_code: Optional[str]) -> str:
|
||||||
|
"""
|
||||||
|
Return the script id (e.g. 'cyrillic', 'cjk') for a given language code.
|
||||||
|
Returns 'latin' for unknown codes (which is the safe default — Latin
|
||||||
|
covers the largest number of languages).
|
||||||
|
"""
|
||||||
|
if not lang_code:
|
||||||
|
return "latin"
|
||||||
|
return LANG_TO_SCRIPT.get(lang_code.lower(), "latin")
|
||||||
|
|
||||||
|
|
||||||
|
def is_arabic_script_lang(lang_code: Optional[str]) -> bool:
|
||||||
|
"""True if lang_code is one of the Arabic-script languages we discriminate."""
|
||||||
|
if not lang_code:
|
||||||
|
return False
|
||||||
|
return get_script(lang_code) == "arabic"
|
||||||
|
|
||||||
|
|
||||||
|
def get_ranges(script_id: str) -> List[Tuple[int, int]]:
|
||||||
|
"""Return the Unicode ranges for a script id. Empty list for 'latin'."""
|
||||||
|
return UNICODE_RANGES.get(script_id, [])
|
||||||
|
|
||||||
|
|
||||||
|
def get_discriminating_chars(lang_code: Optional[str]) -> FrozenSet[str]:
|
||||||
|
"""Return the discriminating characters for a specific Arabic-script language."""
|
||||||
|
if not lang_code:
|
||||||
|
return frozenset()
|
||||||
|
return DISCRIMINATING_CHARS.get(lang_code.lower(), frozenset())
|
||||||
146
services/quality/file_extractor.py
Normal file
146
services/quality/file_extractor.py
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
"""
|
||||||
|
File text extractor for the L0 quality layer.
|
||||||
|
|
||||||
|
Extracts a small sample of text from a translated file so the L0 checks
|
||||||
|
can run on a real output without requiring the translators to expose
|
||||||
|
their internal chunk data.
|
||||||
|
|
||||||
|
This module depends on the same libraries the translators use
|
||||||
|
(python-docx, openpyxl, python-pptx, PyMuPDF). All imports are lazy
|
||||||
|
and guarded, so a missing library only blocks the matching format —
|
||||||
|
other formats keep working.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Optional, TypedDict
|
||||||
|
|
||||||
|
|
||||||
|
class TextSample(TypedDict):
|
||||||
|
"""A (source_placeholder, translated) pair from an output file."""
|
||||||
|
source: str
|
||||||
|
translated: str
|
||||||
|
|
||||||
|
|
||||||
|
# Maximum samples per format — keeps the L0 check fast.
|
||||||
|
DEFAULT_MAX_SAMPLES = 20
|
||||||
|
|
||||||
|
|
||||||
|
def extract_sample(
|
||||||
|
file_path: Path,
|
||||||
|
file_extension: str,
|
||||||
|
max_samples: int = DEFAULT_MAX_SAMPLES,
|
||||||
|
) -> List[TextSample]:
|
||||||
|
"""
|
||||||
|
Extract a sample of translated text strings from a finished file.
|
||||||
|
|
||||||
|
The "source" field is always empty — we don't have the original
|
||||||
|
document at this point. The L0 checks that care about source/target
|
||||||
|
ratio (length_checker) handle empty source gracefully.
|
||||||
|
|
||||||
|
Returns an empty list if the file cannot be read.
|
||||||
|
"""
|
||||||
|
if not file_path or not Path(file_path).exists():
|
||||||
|
return []
|
||||||
|
|
||||||
|
ext = (file_extension or Path(file_path).suffix).lower()
|
||||||
|
try:
|
||||||
|
if ext == ".docx":
|
||||||
|
return _extract_docx(file_path, max_samples)
|
||||||
|
if ext == ".xlsx":
|
||||||
|
return _extract_xlsx(file_path, max_samples)
|
||||||
|
if ext == ".pptx":
|
||||||
|
return _extract_pptx(file_path, max_samples)
|
||||||
|
if ext == ".pdf":
|
||||||
|
return _extract_pdf(file_path, max_samples)
|
||||||
|
except Exception:
|
||||||
|
# Any failure: return an empty list. The route will log it.
|
||||||
|
return []
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_docx(path: Path, max_samples: int) -> List[TextSample]:
|
||||||
|
"""Extract text from a Word document."""
|
||||||
|
from docx import Document
|
||||||
|
doc = Document(str(path))
|
||||||
|
samples: List[TextSample] = []
|
||||||
|
for para in doc.paragraphs:
|
||||||
|
text = (para.text or "").strip()
|
||||||
|
if text and len(text) > 5:
|
||||||
|
samples.append({"source": "", "translated": text})
|
||||||
|
if len(samples) >= max_samples:
|
||||||
|
break
|
||||||
|
# If body had nothing, try tables.
|
||||||
|
if not samples:
|
||||||
|
for table in doc.tables:
|
||||||
|
for row in table.rows:
|
||||||
|
for cell in row.cells:
|
||||||
|
text = (cell.text or "").strip()
|
||||||
|
if text and len(text) > 5:
|
||||||
|
samples.append({"source": "", "translated": text})
|
||||||
|
if len(samples) >= max_samples:
|
||||||
|
return samples
|
||||||
|
return samples
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_xlsx(path: Path, max_samples: int) -> List[TextSample]:
|
||||||
|
"""Extract text from an Excel file."""
|
||||||
|
from openpyxl import load_workbook
|
||||||
|
wb = load_workbook(str(path), data_only=True, read_only=True)
|
||||||
|
samples: List[TextSample] = []
|
||||||
|
try:
|
||||||
|
for ws in wb.worksheets:
|
||||||
|
for row in ws.iter_rows():
|
||||||
|
for cell in row:
|
||||||
|
val = cell.value
|
||||||
|
if isinstance(val, str):
|
||||||
|
text = val.strip()
|
||||||
|
if text and len(text) > 3:
|
||||||
|
samples.append({"source": "", "translated": text})
|
||||||
|
if len(samples) >= max_samples:
|
||||||
|
return samples
|
||||||
|
finally:
|
||||||
|
wb.close()
|
||||||
|
return samples
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_pptx(path: Path, max_samples: int) -> List[TextSample]:
|
||||||
|
"""Extract text from a PowerPoint file."""
|
||||||
|
from pptx import Presentation
|
||||||
|
pres = Presentation(str(path))
|
||||||
|
samples: List[TextSample] = []
|
||||||
|
for slide in pres.slides:
|
||||||
|
for shape in slide.shapes:
|
||||||
|
if not shape.has_text_frame:
|
||||||
|
continue
|
||||||
|
for para in shape.text_frame.paragraphs:
|
||||||
|
text = (para.text or "").strip()
|
||||||
|
if text and len(text) > 3:
|
||||||
|
samples.append({"source": "", "translated": text})
|
||||||
|
if len(samples) >= max_samples:
|
||||||
|
return samples
|
||||||
|
return samples
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_pdf(path: Path, max_samples: int) -> List[TextSample]:
|
||||||
|
"""Extract text from a PDF file (best-effort, layout-preserving format)."""
|
||||||
|
try:
|
||||||
|
import fitz # PyMuPDF
|
||||||
|
except ImportError:
|
||||||
|
return []
|
||||||
|
samples: List[TextSample] = []
|
||||||
|
doc = fitz.open(str(path))
|
||||||
|
try:
|
||||||
|
for page in doc:
|
||||||
|
text = page.get_text("text") or ""
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if line and len(line) > 5:
|
||||||
|
samples.append({"source": "", "translated": line})
|
||||||
|
if len(samples) >= max_samples:
|
||||||
|
return samples
|
||||||
|
finally:
|
||||||
|
doc.close()
|
||||||
|
return samples
|
||||||
395
services/quality/l2_judge.py
Normal file
395
services/quality/l2_judge.py
Normal file
@@ -0,0 +1,395 @@
|
|||||||
|
"""
|
||||||
|
L2 Pro Premium Judge — stronger model, more dimensions, Pro-tier only.
|
||||||
|
|
||||||
|
Why a SEPARATE module from L1?
|
||||||
|
- L1 is fast + cheap (deepseek-chat, 4 dimensions, 5 samples)
|
||||||
|
- L2 is slow + expensive (gpt-4o, 8 dimensions, 15 samples)
|
||||||
|
- Different defaults, different config, different metrics
|
||||||
|
- L2 is gated to the Pro plan; L1 is universal
|
||||||
|
|
||||||
|
L2 dimensions (8):
|
||||||
|
1. accurate — meaning preserved
|
||||||
|
2. fluent — natural in target language
|
||||||
|
3. correct_lang — in the target language (not source leakage)
|
||||||
|
4. no_leaks — no prompt artifacts
|
||||||
|
5. terminology — domain terms correctly handled
|
||||||
|
6. style — appropriate register (formal/informal/technical)
|
||||||
|
7. completeness — no content dropped or added
|
||||||
|
8. formatting — codes, numbers, units preserved
|
||||||
|
|
||||||
|
L1 was a binary pass/fail. L2 returns per-dimension scores (0/1) so the
|
||||||
|
caller can decide which dimensions matter for a given job.
|
||||||
|
|
||||||
|
DESIGN CONSTRAINTS (same as L1):
|
||||||
|
- 100% API-based. No local models, no GPU.
|
||||||
|
- Async, with a hard timeout.
|
||||||
|
- Defensive: never raises.
|
||||||
|
- Output is structured JSON.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from typing import List, Optional, Tuple, Dict, Any
|
||||||
|
|
||||||
|
from core.logging import get_logger
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Result dataclasses ----------
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class L2DimensionVerdict:
|
||||||
|
"""8-dimension verdict for a single chunk."""
|
||||||
|
accurate: bool = False
|
||||||
|
fluent: bool = False
|
||||||
|
correct_lang: bool = False
|
||||||
|
no_leaks: bool = False
|
||||||
|
terminology: bool = False
|
||||||
|
style: bool = False
|
||||||
|
completeness: bool = False
|
||||||
|
formatting: bool = False
|
||||||
|
reason: str = ""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def passed_count(self) -> int:
|
||||||
|
return sum([
|
||||||
|
self.accurate, self.fluent, self.correct_lang, self.no_leaks,
|
||||||
|
self.terminology, self.style, self.completeness, self.formatting,
|
||||||
|
])
|
||||||
|
|
||||||
|
@property
|
||||||
|
def total(self) -> int:
|
||||||
|
return 8
|
||||||
|
|
||||||
|
@property
|
||||||
|
def score(self) -> float:
|
||||||
|
return self.passed_count / self.total
|
||||||
|
|
||||||
|
@property
|
||||||
|
def passed(self) -> bool:
|
||||||
|
# L2 is conservative: any single fail = chunk fails
|
||||||
|
return self.passed_count == self.total
|
||||||
|
|
||||||
|
def to_log_dict(self) -> dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class L2Result:
|
||||||
|
"""Aggregate result of an L2 check on a sample of chunks."""
|
||||||
|
verdict: str # "pass", "fail", "skip"
|
||||||
|
chunks_evaluated: int = 0
|
||||||
|
chunks_passed: int = 0
|
||||||
|
chunks_failed: int = 0
|
||||||
|
failure_rate: float = 0.0
|
||||||
|
average_score: float = 0.0 # mean of per-chunk scores (0.0 to 1.0)
|
||||||
|
dimension_pass_rates: Dict[str, float] = field(default_factory=dict)
|
||||||
|
samples: List[dict] = field(default_factory=list)
|
||||||
|
model_used: str = ""
|
||||||
|
elapsed_ms: float = 0.0
|
||||||
|
cost_estimate_usd: float = 0.0
|
||||||
|
error: str = ""
|
||||||
|
|
||||||
|
def to_log_dict(self) -> dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Prompt template (8 dimensions) ----------
|
||||||
|
|
||||||
|
L2_JUDGE_SYSTEM_PROMPT = """You are an expert translation quality evaluator using MQM-inspired criteria.
|
||||||
|
|
||||||
|
For each (SOURCE, TRANSLATION) pair, check these 8 criteria (yes/no for each):
|
||||||
|
|
||||||
|
1. ACCURATE — Does the translation preserve the meaning of the source?
|
||||||
|
2. FLUENT — Is the translation natural and grammatical in {target_lang_name}?
|
||||||
|
3. CORRECT_LANG — Is the translation actually in {target_lang_name} (ISO: {target_lang})?
|
||||||
|
4. NO_LEAKS — Is the translation free of prompt artifacts, source-language text, or meta-commentary?
|
||||||
|
5. TERMINOLOGY — Are domain-specific terms (technical, legal, medical, etc.) correctly translated?
|
||||||
|
6. STYLE — Is the register/tone appropriate (formal/informal/technical matching the source)?
|
||||||
|
7. COMPLETENESS — Is all content present, with nothing added or dropped?
|
||||||
|
8. FORMATTING — Are codes, numbers, dates, and units preserved exactly?
|
||||||
|
|
||||||
|
A translation FAILS if ANY criterion is "no". The "reason" must be in English and ≤ 20 words.
|
||||||
|
|
||||||
|
Respond with a JSON array, one object per pair, in the same order. NO other text, NO markdown fences:
|
||||||
|
[
|
||||||
|
{{"accurate": "yes"|"no", "fluent": "yes"|"no", "correct_lang": "yes"|"no", "no_leaks": "yes"|"no", "terminology": "yes"|"no", "style": "yes"|"no", "completeness": "yes"|"no", "formatting": "yes"|"no", "reason": "short justification"}}
|
||||||
|
]
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- LLM client ----------
|
||||||
|
|
||||||
|
class L2ProJudge:
|
||||||
|
"""
|
||||||
|
Calls a STRONG LLM via the OpenAI-compatible API to judge translation
|
||||||
|
quality across 8 dimensions. Pro-tier only.
|
||||||
|
|
||||||
|
Default model: gpt-4o (strongest general judge we can afford at scale).
|
||||||
|
For sub-$0.01/job cost, we limit to 15 samples per job and 8 dimensions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
api_key: str,
|
||||||
|
base_url: str = "https://api.openai.com/v1",
|
||||||
|
model: str = "gpt-4o",
|
||||||
|
timeout_seconds: float = 20.0,
|
||||||
|
max_retries: int = 1,
|
||||||
|
):
|
||||||
|
if not api_key:
|
||||||
|
raise ValueError("api_key is required for L2ProJudge")
|
||||||
|
self._api_key = api_key
|
||||||
|
self._base_url = base_url.rstrip("/")
|
||||||
|
self._model = model
|
||||||
|
self._timeout = timeout_seconds
|
||||||
|
self._max_retries = max_retries
|
||||||
|
self._client = None
|
||||||
|
|
||||||
|
def _get_client(self):
|
||||||
|
if self._client is None:
|
||||||
|
try:
|
||||||
|
import openai
|
||||||
|
self._client = openai.AsyncOpenAI(
|
||||||
|
api_key=self._api_key,
|
||||||
|
base_url=self._base_url,
|
||||||
|
timeout=self._timeout,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("l2_judge_client_init_failed", error=str(e))
|
||||||
|
return None
|
||||||
|
return self._client
|
||||||
|
|
||||||
|
async def judge_batch(
|
||||||
|
self,
|
||||||
|
pairs: List[Tuple[str, str]],
|
||||||
|
target_lang: str,
|
||||||
|
target_lang_name: str = "",
|
||||||
|
) -> L2Result:
|
||||||
|
"""
|
||||||
|
Judge a batch of (source, translation) pairs across 8 dimensions.
|
||||||
|
|
||||||
|
Returns an L2Result with verdict="skip" on any internal error.
|
||||||
|
Never raises.
|
||||||
|
"""
|
||||||
|
start = time.time()
|
||||||
|
empty = L2Result(verdict="skip", error="not_run")
|
||||||
|
|
||||||
|
if not pairs:
|
||||||
|
return L2Result(verdict="skip", error="empty pairs",
|
||||||
|
elapsed_ms=round((time.time() - start) * 1000, 2))
|
||||||
|
|
||||||
|
client = self._get_client()
|
||||||
|
if client is None:
|
||||||
|
return L2Result(verdict="skip", error="client unavailable",
|
||||||
|
elapsed_ms=round((time.time() - start) * 1000, 2))
|
||||||
|
|
||||||
|
user_lines = [f"Target language: {target_lang} ({target_lang_name})\n"]
|
||||||
|
for i, (src, trans) in enumerate(pairs, 1):
|
||||||
|
user_lines.append(f"\n--- Pair {i} ---")
|
||||||
|
user_lines.append(f"SOURCE: {src}")
|
||||||
|
user_lines.append(f"TRANSLATION: {trans}")
|
||||||
|
user_msg = "\n".join(user_lines)
|
||||||
|
|
||||||
|
system_prompt = L2_JUDGE_SYSTEM_PROMPT.format(
|
||||||
|
target_lang=target_lang,
|
||||||
|
target_lang_name=target_lang_name or target_lang,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = await asyncio.wait_for(
|
||||||
|
self._call_with_retries(client, system_prompt, user_msg),
|
||||||
|
timeout=self._timeout + 5.0,
|
||||||
|
)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
logger.warning("l2_judge_timeout",
|
||||||
|
timeout_s=self._timeout, elapsed_ms=elapsed_ms)
|
||||||
|
return L2Result(verdict="skip", error="timeout", elapsed_ms=elapsed_ms)
|
||||||
|
except Exception as e:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
logger.warning("l2_judge_error",
|
||||||
|
error=str(e)[:200], elapsed_ms=elapsed_ms)
|
||||||
|
return L2Result(verdict="skip", error=str(e)[:200],
|
||||||
|
elapsed_ms=elapsed_ms)
|
||||||
|
|
||||||
|
verdicts = self._parse_response(response, len(pairs))
|
||||||
|
|
||||||
|
if not verdicts:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
return L2Result(verdict="skip", error="parse_failed",
|
||||||
|
elapsed_ms=elapsed_ms)
|
||||||
|
|
||||||
|
# Aggregate
|
||||||
|
passed = sum(1 for v in verdicts if v.passed)
|
||||||
|
failed = len(verdicts) - passed
|
||||||
|
failure_rate = failed / len(verdicts) if verdicts else 0.0
|
||||||
|
average_score = sum(v.score for v in verdicts) / len(verdicts)
|
||||||
|
|
||||||
|
# Per-dimension pass rates
|
||||||
|
dimensions = [
|
||||||
|
"accurate", "fluent", "correct_lang", "no_leaks",
|
||||||
|
"terminology", "style", "completeness", "formatting",
|
||||||
|
]
|
||||||
|
dim_pass_rates = {}
|
||||||
|
for dim in dimensions:
|
||||||
|
count = sum(1 for v in verdicts if getattr(v, dim))
|
||||||
|
dim_pass_rates[dim] = round(count / len(verdicts), 3) if verdicts else 0.0
|
||||||
|
|
||||||
|
# L2 verdict: strict — any chunk fail = overall fail
|
||||||
|
verdict = "pass" if failed == 0 else "fail"
|
||||||
|
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
cost_estimate = self._estimate_cost(len(pairs))
|
||||||
|
|
||||||
|
return L2Result(
|
||||||
|
verdict=verdict,
|
||||||
|
chunks_evaluated=len(verdicts),
|
||||||
|
chunks_passed=passed,
|
||||||
|
chunks_failed=failed,
|
||||||
|
failure_rate=round(failure_rate, 3),
|
||||||
|
average_score=round(average_score, 3),
|
||||||
|
dimension_pass_rates=dim_pass_rates,
|
||||||
|
samples=[v.to_log_dict() for v in verdicts],
|
||||||
|
model_used=self._model,
|
||||||
|
elapsed_ms=elapsed_ms,
|
||||||
|
cost_estimate_usd=cost_estimate,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _call_with_retries(self, client, system_prompt: str, user_msg: str):
|
||||||
|
"""Call the LLM with retry on transient errors."""
|
||||||
|
last_exc = None
|
||||||
|
for attempt in range(self._max_retries + 1):
|
||||||
|
try:
|
||||||
|
response = await client.chat.completions.create(
|
||||||
|
model=self._model,
|
||||||
|
messages=[
|
||||||
|
{"role": "system", "content": system_prompt},
|
||||||
|
{"role": "user", "content": user_msg},
|
||||||
|
],
|
||||||
|
temperature=0.0,
|
||||||
|
max_tokens=1200, # larger than L1 (more dimensions)
|
||||||
|
response_format={"type": "json_object"},
|
||||||
|
)
|
||||||
|
return response
|
||||||
|
except Exception as e:
|
||||||
|
last_exc = e
|
||||||
|
if attempt < self._max_retries:
|
||||||
|
await asyncio.sleep(0.8)
|
||||||
|
raise last_exc
|
||||||
|
|
||||||
|
def _parse_response(self, response, expected_count: int) -> List[L2DimensionVerdict]:
|
||||||
|
"""Parse the LLM response into a list of 8-dimension verdicts."""
|
||||||
|
try:
|
||||||
|
content = response.choices[0].message.content or ""
|
||||||
|
except (AttributeError, IndexError) as e:
|
||||||
|
logger.warning("l2_judge_bad_response", error=str(e))
|
||||||
|
return []
|
||||||
|
|
||||||
|
content = content.strip()
|
||||||
|
if content.startswith("```"):
|
||||||
|
content = re.sub(r"^```(?:json)?\s*\n?", "", content)
|
||||||
|
content = re.sub(r"\n?```\s*$", "", content)
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = json.loads(content)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
logger.warning("l2_judge_json_parse_error",
|
||||||
|
error=str(e), content_preview=content[:200])
|
||||||
|
return []
|
||||||
|
|
||||||
|
if isinstance(data, dict):
|
||||||
|
items = None
|
||||||
|
for key in ("verdicts", "results", "translations", "data"):
|
||||||
|
if key in data and isinstance(data[key], list):
|
||||||
|
items = data[key]
|
||||||
|
break
|
||||||
|
if items is None:
|
||||||
|
for v in data.values():
|
||||||
|
if isinstance(v, list):
|
||||||
|
items = v
|
||||||
|
break
|
||||||
|
if items is None:
|
||||||
|
logger.warning("l2_judge_no_list_in_response")
|
||||||
|
return []
|
||||||
|
elif isinstance(data, list):
|
||||||
|
items = data
|
||||||
|
else:
|
||||||
|
logger.warning("l2_judge_unexpected_response_type",
|
||||||
|
type_=type(data).__name__)
|
||||||
|
return []
|
||||||
|
|
||||||
|
verdicts: List[L2DimensionVerdict] = []
|
||||||
|
for item in items:
|
||||||
|
try:
|
||||||
|
v = L2DimensionVerdict(
|
||||||
|
accurate=str(item.get("accurate", "")).lower() == "yes",
|
||||||
|
fluent=str(item.get("fluent", "")).lower() == "yes",
|
||||||
|
correct_lang=str(item.get("correct_lang", "")).lower() == "yes",
|
||||||
|
no_leaks=str(item.get("no_leaks", "")).lower() == "yes",
|
||||||
|
terminology=str(item.get("terminology", "")).lower() == "yes",
|
||||||
|
style=str(item.get("style", "")).lower() == "yes",
|
||||||
|
completeness=str(item.get("completeness", "")).lower() == "yes",
|
||||||
|
formatting=str(item.get("formatting", "")).lower() == "yes",
|
||||||
|
reason=str(item.get("reason", ""))[:300],
|
||||||
|
)
|
||||||
|
verdicts.append(v)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("l2_judge_item_parse_error",
|
||||||
|
error=str(e), item=str(item)[:200])
|
||||||
|
|
||||||
|
return verdicts
|
||||||
|
|
||||||
|
def _estimate_cost(self, num_pairs: int) -> float:
|
||||||
|
"""Rough USD cost estimate for the call."""
|
||||||
|
# L2 has more dimensions = longer output
|
||||||
|
input_tokens = 250 + (num_pairs * 280)
|
||||||
|
output_tokens = num_pairs * 110
|
||||||
|
model_lower = self._model.lower()
|
||||||
|
# IMPORTANT: check 'mini' BEFORE full 'gpt-4o' because
|
||||||
|
# 'gpt-4o-mini' contains 'gpt-4o'.
|
||||||
|
if "gpt-4o-mini" in model_lower:
|
||||||
|
input_cost = input_tokens / 1_000_000 * 0.15
|
||||||
|
output_cost = output_tokens / 1_000_000 * 0.60
|
||||||
|
elif "gpt-4o" in model_lower:
|
||||||
|
input_cost = input_tokens / 1_000_000 * 2.50
|
||||||
|
output_cost = output_tokens / 1_000_000 * 10.00
|
||||||
|
elif "claude" in model_lower:
|
||||||
|
input_cost = input_tokens / 1_000_000 * 3.00
|
||||||
|
output_cost = output_tokens / 1_000_000 * 15.00
|
||||||
|
else:
|
||||||
|
# Generic conservative
|
||||||
|
input_cost = input_tokens / 1_000_000 * 1.00
|
||||||
|
output_cost = output_tokens / 1_000_000 * 3.00
|
||||||
|
return round(input_cost + output_cost, 6)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Convenience factory ----------
|
||||||
|
|
||||||
|
def make_l2_judge_from_env() -> Optional[L2ProJudge]:
|
||||||
|
"""
|
||||||
|
Build an L2ProJudge from environment variables. Returns None if
|
||||||
|
no API key is configured.
|
||||||
|
|
||||||
|
Reads:
|
||||||
|
- L2_JUDGE_API_KEY (required)
|
||||||
|
- L2_JUDGE_BASE_URL (default: OpenAI)
|
||||||
|
- L2_JUDGE_MODEL (default: gpt-4o)
|
||||||
|
- L2_JUDGE_TIMEOUT (default: 20.0)
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
api_key = os.getenv("L2_JUDGE_API_KEY", "").strip()
|
||||||
|
if not api_key:
|
||||||
|
return None
|
||||||
|
return L2ProJudge(
|
||||||
|
api_key=api_key,
|
||||||
|
base_url=os.getenv("L2_JUDGE_BASE_URL", "https://api.openai.com/v1"),
|
||||||
|
model=os.getenv("L2_JUDGE_MODEL", "gpt-4o"),
|
||||||
|
timeout_seconds=float(os.getenv("L2_JUDGE_TIMEOUT", "20.0")),
|
||||||
|
)
|
||||||
125
services/quality/length_checker.py
Normal file
125
services/quality/length_checker.py
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
"""
|
||||||
|
Length sanity check for the L0 quality layer.
|
||||||
|
|
||||||
|
A translation that's 10× longer or 10× shorter than the source is almost
|
||||||
|
certainly a hallucination or a truncation. We flag these as warnings
|
||||||
|
(not failures) so the caller can decide what to do.
|
||||||
|
|
||||||
|
Thresholds are tunable via env vars if needed, but the defaults work
|
||||||
|
well for prose documents. Tables and bullet lists will naturally have
|
||||||
|
shorter translations, so we keep the lower bound loose.
|
||||||
|
|
||||||
|
Special cases:
|
||||||
|
* If the source is mostly digits/punctuation, the translation can also
|
||||||
|
be short (e.g. "Price: 100$" → "100€") — skip the check.
|
||||||
|
* If the source is empty/very short, skip the check entirely.
|
||||||
|
* If the source contains an embedded URL or email, the translation may
|
||||||
|
legitimately shrink — skip the check.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict
|
||||||
|
|
||||||
|
# Default thresholds — generous enough to handle tables / short strings.
|
||||||
|
RATIO_MAX = 3.5
|
||||||
|
RATIO_MIN = 0.15
|
||||||
|
# Hard lower bound: a translation shorter than this is very suspect,
|
||||||
|
# UNLESS the source is also very short.
|
||||||
|
ABSOLUTE_MIN_LENGTH = 2
|
||||||
|
# If source is short (under this many chars), skip the ratio check entirely
|
||||||
|
# — short strings are too noisy to be useful for length analysis.
|
||||||
|
MIN_SOURCE_LENGTH_FOR_RATIO = 20
|
||||||
|
|
||||||
|
# Pattern to detect text that is mostly digits / numbers / simple symbols.
|
||||||
|
# E.g. "Price: 100€", "+33 6 12 34 56 78", "192.168.1.1".
|
||||||
|
_MOSTLY_NUMERIC_RE = re.compile(r"^[\d\s\W]*$", re.UNICODE)
|
||||||
|
_NUMERIC_RATIO_THRESHOLD = 0.5 # 50% of letters are digits
|
||||||
|
|
||||||
|
|
||||||
|
def check(source_text: str, translated_text: str) -> Dict:
|
||||||
|
"""
|
||||||
|
Returns a dict like:
|
||||||
|
{
|
||||||
|
"issue": None | "length_outlier" | "truncation_suspect",
|
||||||
|
"ratio": float,
|
||||||
|
"source_length": int,
|
||||||
|
"translated_length": int,
|
||||||
|
}
|
||||||
|
Never raises.
|
||||||
|
"""
|
||||||
|
if not source_text:
|
||||||
|
return {
|
||||||
|
"issue": None,
|
||||||
|
"ratio": None,
|
||||||
|
"source_length": 0,
|
||||||
|
"translated_length": len(translated_text or ""),
|
||||||
|
}
|
||||||
|
|
||||||
|
src_len = len(source_text.strip())
|
||||||
|
trans_len = len(translated_text.strip())
|
||||||
|
|
||||||
|
# Empty translation — always suspect.
|
||||||
|
if trans_len == 0:
|
||||||
|
return {
|
||||||
|
"issue": "truncation_suspect",
|
||||||
|
"ratio": 0.0,
|
||||||
|
"source_length": src_len,
|
||||||
|
"translated_length": trans_len,
|
||||||
|
}
|
||||||
|
|
||||||
|
# If source is mostly digits/numbers, the translation can also be short
|
||||||
|
# (e.g. "Price: 100" → "100"). Don't flag length in this case.
|
||||||
|
if _is_mostly_numeric(source_text):
|
||||||
|
return {
|
||||||
|
"issue": None,
|
||||||
|
"ratio": None,
|
||||||
|
"source_length": src_len,
|
||||||
|
"translated_length": trans_len,
|
||||||
|
"note": "skipped: numeric source",
|
||||||
|
}
|
||||||
|
|
||||||
|
# If source is very short, skip the ratio check.
|
||||||
|
if src_len < MIN_SOURCE_LENGTH_FOR_RATIO:
|
||||||
|
return {
|
||||||
|
"issue": None,
|
||||||
|
"ratio": None,
|
||||||
|
"source_length": src_len,
|
||||||
|
"translated_length": trans_len,
|
||||||
|
}
|
||||||
|
|
||||||
|
ratio = trans_len / src_len
|
||||||
|
|
||||||
|
if ratio > RATIO_MAX:
|
||||||
|
return {
|
||||||
|
"issue": "length_outlier",
|
||||||
|
"ratio": round(ratio, 2),
|
||||||
|
"source_length": src_len,
|
||||||
|
"translated_length": trans_len,
|
||||||
|
}
|
||||||
|
if ratio < RATIO_MIN:
|
||||||
|
return {
|
||||||
|
"issue": "truncation_suspect",
|
||||||
|
"ratio": round(ratio, 2),
|
||||||
|
"source_length": src_len,
|
||||||
|
"translated_length": trans_len,
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"issue": None,
|
||||||
|
"ratio": round(ratio, 2),
|
||||||
|
"source_length": src_len,
|
||||||
|
"translated_length": trans_len,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _is_mostly_numeric(text: str) -> bool:
|
||||||
|
"""True if at least 50% of non-whitespace characters are digits."""
|
||||||
|
if not text:
|
||||||
|
return False
|
||||||
|
chars = [c for c in text if not c.isspace()]
|
||||||
|
if not chars:
|
||||||
|
return False
|
||||||
|
digit_count = sum(1 for c in chars if c.isdigit())
|
||||||
|
return (digit_count / len(chars)) >= _NUMERIC_RATIO_THRESHOLD
|
||||||
365
services/quality/llm_judge.py
Normal file
365
services/quality/llm_judge.py
Normal file
@@ -0,0 +1,365 @@
|
|||||||
|
"""
|
||||||
|
L1 LLM Judge — uses a cheap LLM via the OpenAI-compatible API to validate
|
||||||
|
the quality of a translation.
|
||||||
|
|
||||||
|
Why a SEPARATE LLM (not the one that did the translation)?
|
||||||
|
- Self-evaluation bias: a model tends to rate its own output as good
|
||||||
|
(Meta/Stanford papers on LLM-as-judge bias, 2024-2025).
|
||||||
|
- Independence gives a more reliable signal.
|
||||||
|
|
||||||
|
Design constraints:
|
||||||
|
- 100% API-based. No local models, no GPU.
|
||||||
|
- Async, with a hard timeout.
|
||||||
|
- Defensive: never raises. Returns a "skip" verdict on any internal error.
|
||||||
|
- Output is structured JSON for reliable parsing.
|
||||||
|
- Sampled (we never judge all chunks, just a small subset).
|
||||||
|
|
||||||
|
The LLM is asked a binary question per chunk: "Is this translation
|
||||||
|
accurate and fluent, in the correct language?" with a short reason.
|
||||||
|
We do NOT ask for nuanced MQM scores — binary is more reliable and
|
||||||
|
cheaper.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from typing import List, Optional, Tuple
|
||||||
|
|
||||||
|
from core.logging import get_logger
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Result dataclasses ----------
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class L1ChunkVerdict:
|
||||||
|
"""Verdict for a single (source, translation) pair."""
|
||||||
|
accurate: bool
|
||||||
|
fluent: bool
|
||||||
|
correct_language: bool
|
||||||
|
no_leaks: bool
|
||||||
|
reason: str = ""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def passed(self) -> bool:
|
||||||
|
return self.accurate and self.fluent and self.correct_language and self.no_leaks
|
||||||
|
|
||||||
|
def to_log_dict(self) -> dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class L1Result:
|
||||||
|
"""Aggregate result of an L1 check on a sample of chunks."""
|
||||||
|
verdict: str # "pass", "fail", "skip"
|
||||||
|
chunks_evaluated: int
|
||||||
|
chunks_passed: int
|
||||||
|
chunks_failed: int
|
||||||
|
failure_rate: float # 0.0 to 1.0
|
||||||
|
samples: List[dict] = field(default_factory=list)
|
||||||
|
model_used: str = ""
|
||||||
|
elapsed_ms: float = 0.0
|
||||||
|
cost_estimate_usd: float = 0.0
|
||||||
|
error: str = ""
|
||||||
|
|
||||||
|
def to_log_dict(self) -> dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Prompt template ----------
|
||||||
|
|
||||||
|
JUDGE_SYSTEM_PROMPT = """You are a strict translation quality evaluator. Your job is to detect translation failures.
|
||||||
|
|
||||||
|
For each (SOURCE, TRANSLATION) pair, check these 4 criteria:
|
||||||
|
|
||||||
|
1. ACCURATE — Does the translation preserve the meaning of the source? (yes/no)
|
||||||
|
2. FLUENT — Is the translation grammatically correct in the target language? (yes/no)
|
||||||
|
3. CORRECT_LANGUAGE — Is the translation actually in {target_lang_name} (ISO code: {target_lang})? (yes/no)
|
||||||
|
4. NO_LEAKS — Is the translation free of prompt artifacts, source-language text, or meta-commentary? (yes/no)
|
||||||
|
|
||||||
|
A translation FAILS if ANY of the 4 checks is "no".
|
||||||
|
|
||||||
|
Respond with a JSON array, one object per pair, in the same order. NO other text, NO markdown fences:
|
||||||
|
[
|
||||||
|
{{"accurate": "yes"|"no", "fluent": "yes"|"no", "correct_language": "yes"|"no", "no_leaks": "yes"|"no", "reason": "one short sentence"}}
|
||||||
|
]
|
||||||
|
|
||||||
|
Be honest: if the translation looks suspicious, say "no". The "reason" must be in English and ≤ 12 words.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- LLM client ----------
|
||||||
|
|
||||||
|
class LLMJudge:
|
||||||
|
"""
|
||||||
|
Calls a cheap LLM via the OpenAI-compatible API to judge translation quality.
|
||||||
|
|
||||||
|
Default configuration targets deepseek-chat via the OpenAI-compatible
|
||||||
|
DeepSeek API. The judge can be reconfigured to use any OpenAI-compatible
|
||||||
|
endpoint (OpenAI, OpenRouter, etc.) by passing different params.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
api_key: str,
|
||||||
|
base_url: str = "https://api.deepseek.com/v1",
|
||||||
|
model: str = "deepseek-chat",
|
||||||
|
timeout_seconds: float = 8.0,
|
||||||
|
max_retries: int = 1,
|
||||||
|
):
|
||||||
|
if not api_key:
|
||||||
|
raise ValueError("api_key is required for LLMJudge")
|
||||||
|
self._api_key = api_key
|
||||||
|
self._base_url = base_url.rstrip("/")
|
||||||
|
self._model = model
|
||||||
|
self._timeout = timeout_seconds
|
||||||
|
self._max_retries = max_retries
|
||||||
|
# Lazy import — keep services/quality free of the openai dep at import time
|
||||||
|
self._client = None
|
||||||
|
|
||||||
|
def _get_client(self):
|
||||||
|
if self._client is None:
|
||||||
|
try:
|
||||||
|
import openai
|
||||||
|
self._client = openai.AsyncOpenAI(
|
||||||
|
api_key=self._api_key,
|
||||||
|
base_url=self._base_url,
|
||||||
|
timeout=self._timeout,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("llm_judge_client_init_failed", error=str(e))
|
||||||
|
return None
|
||||||
|
return self._client
|
||||||
|
|
||||||
|
async def judge_batch(
|
||||||
|
self,
|
||||||
|
pairs: List[Tuple[str, str]],
|
||||||
|
target_lang: str,
|
||||||
|
target_lang_name: str = "",
|
||||||
|
) -> L1Result:
|
||||||
|
"""
|
||||||
|
Judge a batch of (source, translation) pairs.
|
||||||
|
|
||||||
|
Returns an L1Result with verdict="skip" on any internal error.
|
||||||
|
Never raises.
|
||||||
|
"""
|
||||||
|
start = time.time()
|
||||||
|
if not pairs:
|
||||||
|
return L1Result(verdict="skip", chunks_evaluated=0, chunks_passed=0,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
error="empty pairs")
|
||||||
|
|
||||||
|
client = self._get_client()
|
||||||
|
if client is None:
|
||||||
|
return L1Result(verdict="skip", chunks_evaluated=0, chunks_passed=0,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
error="client unavailable")
|
||||||
|
|
||||||
|
# Build the user message
|
||||||
|
user_lines = [f"Target language: {target_lang} ({target_lang_name})\n"]
|
||||||
|
for i, (src, trans) in enumerate(pairs, 1):
|
||||||
|
user_lines.append(f"\n--- Pair {i} ---")
|
||||||
|
user_lines.append(f"SOURCE: {src}")
|
||||||
|
user_lines.append(f"TRANSLATION: {trans}")
|
||||||
|
user_msg = "\n".join(user_lines)
|
||||||
|
|
||||||
|
system_prompt = JUDGE_SYSTEM_PROMPT.format(
|
||||||
|
target_lang=target_lang,
|
||||||
|
target_lang_name=target_lang_name or target_lang,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = await asyncio.wait_for(
|
||||||
|
self._call_with_retries(client, system_prompt, user_msg),
|
||||||
|
timeout=self._timeout + 2.0, # hard ceiling
|
||||||
|
)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
logger.warning("llm_judge_timeout", timeout_s=self._timeout, elapsed_ms=elapsed_ms)
|
||||||
|
return L1Result(verdict="skip", chunks_evaluated=0, chunks_passed=0,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
error="timeout", elapsed_ms=elapsed_ms)
|
||||||
|
except Exception as e:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
logger.warning("llm_judge_error", error=str(e)[:200], elapsed_ms=elapsed_ms)
|
||||||
|
return L1Result(verdict="skip", chunks_evaluated=0, chunks_passed=0,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
error=str(e)[:200], elapsed_ms=elapsed_ms)
|
||||||
|
|
||||||
|
# Parse the response
|
||||||
|
verdicts = self._parse_response(response, len(pairs))
|
||||||
|
|
||||||
|
passed = sum(1 for v in verdicts if v.passed)
|
||||||
|
failed = len(verdicts) - passed
|
||||||
|
failure_rate = (failed / len(verdicts)) if verdicts else 0.0
|
||||||
|
|
||||||
|
# Aggregate verdict
|
||||||
|
if failed == 0:
|
||||||
|
verdict = "pass"
|
||||||
|
elif failure_rate >= 0.5:
|
||||||
|
verdict = "fail"
|
||||||
|
else:
|
||||||
|
# Some pass, some fail — degraded but not catastrophic.
|
||||||
|
# Caller can decide what to do.
|
||||||
|
verdict = "fail" # conservative: any failure = fail
|
||||||
|
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
|
||||||
|
# Cost estimate: deepseek-chat at $0.14/M in, $0.28/M out
|
||||||
|
# Rough estimate: 500 input tokens + 100 output tokens per call
|
||||||
|
cost_estimate = self._estimate_cost(len(pairs))
|
||||||
|
|
||||||
|
return L1Result(
|
||||||
|
verdict=verdict,
|
||||||
|
chunks_evaluated=len(verdicts),
|
||||||
|
chunks_passed=passed,
|
||||||
|
chunks_failed=failed,
|
||||||
|
failure_rate=round(failure_rate, 3),
|
||||||
|
samples=[v.to_log_dict() for v in verdicts],
|
||||||
|
model_used=self._model,
|
||||||
|
elapsed_ms=elapsed_ms,
|
||||||
|
cost_estimate_usd=cost_estimate,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _call_with_retries(self, client, system_prompt: str, user_msg: str):
|
||||||
|
"""Call the LLM with retry on transient errors."""
|
||||||
|
last_exc = None
|
||||||
|
for attempt in range(self._max_retries + 1):
|
||||||
|
try:
|
||||||
|
response = await client.chat.completions.create(
|
||||||
|
model=self._model,
|
||||||
|
messages=[
|
||||||
|
{"role": "system", "content": system_prompt},
|
||||||
|
{"role": "user", "content": user_msg},
|
||||||
|
],
|
||||||
|
temperature=0.0, # deterministic
|
||||||
|
max_tokens=500, # enough for ~5 verdicts
|
||||||
|
response_format={"type": "json_object"}, # if supported
|
||||||
|
)
|
||||||
|
return response
|
||||||
|
except Exception as e:
|
||||||
|
last_exc = e
|
||||||
|
if attempt < self._max_retries:
|
||||||
|
await asyncio.sleep(0.5)
|
||||||
|
raise last_exc
|
||||||
|
|
||||||
|
def _parse_response(self, response, expected_count: int) -> List[L1ChunkVerdict]:
|
||||||
|
"""
|
||||||
|
Parse the LLM response into a list of verdicts.
|
||||||
|
Robust to JSON wrapped in code fences or with extra commentary.
|
||||||
|
"""
|
||||||
|
# Extract the message content
|
||||||
|
try:
|
||||||
|
content = response.choices[0].message.content or ""
|
||||||
|
except (AttributeError, IndexError) as e:
|
||||||
|
logger.warning("llm_judge_bad_response", error=str(e))
|
||||||
|
return []
|
||||||
|
|
||||||
|
content = content.strip()
|
||||||
|
|
||||||
|
# Strip markdown code fences if present
|
||||||
|
if content.startswith("```"):
|
||||||
|
content = re.sub(r"^```(?:json)?\s*\n?", "", content)
|
||||||
|
content = re.sub(r"\n?```\s*$", "", content)
|
||||||
|
|
||||||
|
# Try to parse as JSON object (deepseek with json_object format) or array
|
||||||
|
try:
|
||||||
|
data = json.loads(content)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
logger.warning("llm_judge_json_parse_error", error=str(e), content_preview=content[:200])
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Handle both {"verdicts": [...]} and direct [...]
|
||||||
|
if isinstance(data, dict):
|
||||||
|
# Look for a list-typed value
|
||||||
|
items = None
|
||||||
|
for key in ("verdicts", "results", "translations", "data"):
|
||||||
|
if key in data and isinstance(data[key], list):
|
||||||
|
items = data[key]
|
||||||
|
break
|
||||||
|
if items is None:
|
||||||
|
# Take the first list-typed value
|
||||||
|
for v in data.values():
|
||||||
|
if isinstance(v, list):
|
||||||
|
items = v
|
||||||
|
break
|
||||||
|
if items is None:
|
||||||
|
logger.warning("llm_judge_no_list_in_response")
|
||||||
|
return []
|
||||||
|
elif isinstance(data, list):
|
||||||
|
items = data
|
||||||
|
else:
|
||||||
|
logger.warning("llm_judge_unexpected_response_type", type_=type(data).__name__)
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Parse each item
|
||||||
|
verdicts: List[L1ChunkVerdict] = []
|
||||||
|
for item in items:
|
||||||
|
try:
|
||||||
|
v = L1ChunkVerdict(
|
||||||
|
accurate=str(item.get("accurate", "")).lower() == "yes",
|
||||||
|
fluent=str(item.get("fluent", "")).lower() == "yes",
|
||||||
|
correct_language=str(item.get("correct_language", "")).lower() == "yes",
|
||||||
|
no_leaks=str(item.get("no_leaks", "")).lower() == "yes",
|
||||||
|
reason=str(item.get("reason", ""))[:200],
|
||||||
|
)
|
||||||
|
verdicts.append(v)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("llm_judge_item_parse_error", error=str(e), item=str(item)[:200])
|
||||||
|
# Continue with what we have
|
||||||
|
|
||||||
|
return verdicts
|
||||||
|
|
||||||
|
def _estimate_cost(self, num_pairs: int) -> float:
|
||||||
|
"""Rough USD cost estimate for the call. Conservative (rounded up)."""
|
||||||
|
# Approximation: 250 input tokens per pair + 50 output tokens per pair
|
||||||
|
# (rough based on the prompt template + JSON output)
|
||||||
|
input_tokens = 200 + (num_pairs * 250)
|
||||||
|
output_tokens = num_pairs * 50
|
||||||
|
# deepseek-chat pricing
|
||||||
|
if "deepseek" in self._model.lower():
|
||||||
|
input_cost = input_tokens / 1_000_000 * 0.14
|
||||||
|
output_cost = output_tokens / 1_000_000 * 0.28
|
||||||
|
elif "gpt-4o-mini" in self._model.lower():
|
||||||
|
input_cost = input_tokens / 1_000_000 * 0.15
|
||||||
|
output_cost = output_tokens / 1_000_000 * 0.60
|
||||||
|
elif "gemini" in self._model.lower() and "flash" in self._model.lower():
|
||||||
|
# gemini-2.5-flash-lite ~ $0.10/M in, $0.40/M out
|
||||||
|
input_cost = input_tokens / 1_000_000 * 0.10
|
||||||
|
output_cost = output_tokens / 1_000_000 * 0.40
|
||||||
|
else:
|
||||||
|
# Generic conservative estimate
|
||||||
|
input_cost = input_tokens / 1_000_000 * 0.50
|
||||||
|
output_cost = output_tokens / 1_000_000 * 1.00
|
||||||
|
return round(input_cost + output_cost, 6)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Convenience factory ----------
|
||||||
|
|
||||||
|
def make_judge_from_env() -> Optional[LLMJudge]:
|
||||||
|
"""
|
||||||
|
Build an LLMJudge from environment variables. Returns None if no
|
||||||
|
API key is configured.
|
||||||
|
|
||||||
|
Reads:
|
||||||
|
- L1_JUDGE_API_KEY (required)
|
||||||
|
- L1_JUDGE_BASE_URL (default: DeepSeek)
|
||||||
|
- L1_JUDGE_MODEL (default: deepseek-chat)
|
||||||
|
- L1_JUDGE_TIMEOUT (default: 8.0)
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
api_key = os.getenv("L1_JUDGE_API_KEY", "").strip()
|
||||||
|
if not api_key:
|
||||||
|
return None
|
||||||
|
return LLMJudge(
|
||||||
|
api_key=api_key,
|
||||||
|
base_url=os.getenv("L1_JUDGE_BASE_URL", "https://api.deepseek.com/v1"),
|
||||||
|
model=os.getenv("L1_JUDGE_MODEL", "deepseek-chat"),
|
||||||
|
timeout_seconds=float(os.getenv("L1_JUDGE_TIMEOUT", "8.0")),
|
||||||
|
)
|
||||||
126
services/quality/pattern_leak.py
Normal file
126
services/quality/pattern_leak.py
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
"""
|
||||||
|
Pattern leak detection for the L0 quality layer.
|
||||||
|
|
||||||
|
Detects common failure modes where the LLM translator:
|
||||||
|
1. Leaks parts of the system prompt (e.g. starts with "Translation:" or "Voici la traduction :")
|
||||||
|
2. Hallucinates by repeating the same word/phrase many times in a row
|
||||||
|
3. Returns a chain-of-thought / explanation instead of a translation
|
||||||
|
|
||||||
|
These checks are pure regex / counting — no model, no network.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Prompt leak patterns ----------
|
||||||
|
|
||||||
|
# Phrases that strongly suggest the LLM leaked its prompt or a thought process
|
||||||
|
# into the output. We check only at the START of the translation (after
|
||||||
|
# stripping whitespace) to avoid false positives on legitimate text.
|
||||||
|
LEAK_PREFIX_PATTERNS: List[re.Pattern] = [
|
||||||
|
re.compile(r"^(translation|translated text|here is the translation|here'?s the translation)\s*[::-]", re.IGNORECASE),
|
||||||
|
re.compile(r"^(voici (la |ma )?traduction|traduction\s*[::-])\b", re.IGNORECASE),
|
||||||
|
re.compile(r"^(原文|译|翻译|译为|以下是)\s*[::]?", re.UNICODE),
|
||||||
|
re.compile(r"^(sure,?\s+here'?s?\s+(the\s+)?translation|of course,?\s+here)", re.IGNORECASE),
|
||||||
|
re.compile(r"^(\*\*|__|\#)\s*translation", re.IGNORECASE),
|
||||||
|
re.compile(r"^translated from\s+\w+\s+to\s+\w+\s*[::-]", re.IGNORECASE),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Repetition detection ----------
|
||||||
|
|
||||||
|
# A "word" for the repetition check — uses Unicode word boundaries so
|
||||||
|
# it works on Chinese, Japanese, Korean, etc.
|
||||||
|
_WORD_RE = re.compile(r"\S+", re.UNICODE)
|
||||||
|
|
||||||
|
# Threshold: a word (or token) repeated 5+ times consecutively is almost
|
||||||
|
# always a hallucination. We allow up to 4 to handle legitimate text
|
||||||
|
# like "the the" (typo) without false positives.
|
||||||
|
REPETITION_THRESHOLD = 5
|
||||||
|
# Same character repeated 20+ times in a row is also a hallucination
|
||||||
|
# (catches cases like "xxxxxxxxxxx" or "==========").
|
||||||
|
CHAR_REPETITION_THRESHOLD = 20
|
||||||
|
|
||||||
|
|
||||||
|
def check(text: str) -> Dict:
|
||||||
|
"""
|
||||||
|
Returns a dict like:
|
||||||
|
{
|
||||||
|
"issue": None | "prompt_leak" | "repetition_hallucination",
|
||||||
|
"matched_pattern": "..." | None,
|
||||||
|
"repetition_count": int | None,
|
||||||
|
}
|
||||||
|
Never raises.
|
||||||
|
"""
|
||||||
|
if not text or not text.strip():
|
||||||
|
return {"issue": None, "matched_pattern": None, "repetition_count": None}
|
||||||
|
|
||||||
|
stripped = text.lstrip()
|
||||||
|
|
||||||
|
# 1. Prompt leak
|
||||||
|
for pat in LEAK_PREFIX_PATTERNS:
|
||||||
|
m = pat.match(stripped)
|
||||||
|
if m:
|
||||||
|
return {
|
||||||
|
"issue": "prompt_leak",
|
||||||
|
"matched_pattern": pat.pattern,
|
||||||
|
"repetition_count": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
# 2. Token-level repetition
|
||||||
|
tokens = _WORD_RE.findall(stripped)
|
||||||
|
rep_count = _max_consecutive_repetition(tokens)
|
||||||
|
if rep_count >= REPETITION_THRESHOLD:
|
||||||
|
return {
|
||||||
|
"issue": "repetition_hallucination",
|
||||||
|
"matched_pattern": None,
|
||||||
|
"repetition_count": rep_count,
|
||||||
|
}
|
||||||
|
|
||||||
|
# 3. Character-level repetition (catches "xxxxxx" without spaces)
|
||||||
|
char_rep = _max_consecutive_char_repetition(stripped)
|
||||||
|
if char_rep >= CHAR_REPETITION_THRESHOLD:
|
||||||
|
return {
|
||||||
|
"issue": "repetition_hallucination",
|
||||||
|
"matched_pattern": None,
|
||||||
|
"repetition_count": char_rep,
|
||||||
|
}
|
||||||
|
|
||||||
|
return {"issue": None, "matched_pattern": None, "repetition_count": max(rep_count, char_rep) or None}
|
||||||
|
|
||||||
|
|
||||||
|
def _max_consecutive_repetition(tokens: List[str]) -> int:
|
||||||
|
"""Return the maximum number of times the same token appears consecutively."""
|
||||||
|
if not tokens:
|
||||||
|
return 0
|
||||||
|
# Normalize: lower-case + strip basic punctuation for comparison
|
||||||
|
norm = [t.lower().strip(".,!?;:\"'`()[]{}") for t in tokens]
|
||||||
|
max_run = 1
|
||||||
|
current_run = 1
|
||||||
|
for i in range(1, len(norm)):
|
||||||
|
if norm[i] and norm[i] == norm[i - 1]:
|
||||||
|
current_run += 1
|
||||||
|
if current_run > max_run:
|
||||||
|
max_run = current_run
|
||||||
|
else:
|
||||||
|
current_run = 1
|
||||||
|
return max_run
|
||||||
|
|
||||||
|
|
||||||
|
def _max_consecutive_char_repetition(text: str) -> int:
|
||||||
|
"""Return the maximum number of times the same character appears consecutively."""
|
||||||
|
if not text:
|
||||||
|
return 0
|
||||||
|
max_run = 1
|
||||||
|
current_run = 1
|
||||||
|
for i in range(1, len(text)):
|
||||||
|
if text[i] == text[i - 1] and not text[i].isspace():
|
||||||
|
current_run += 1
|
||||||
|
if current_run > max_run:
|
||||||
|
max_run = current_run
|
||||||
|
else:
|
||||||
|
current_run = 1
|
||||||
|
return max_run
|
||||||
426
services/quality/pipeline.py
Normal file
426
services/quality/pipeline.py
Normal file
@@ -0,0 +1,426 @@
|
|||||||
|
"""
|
||||||
|
Quality pipeline — defensive wrapper around the L0 and L1 checks.
|
||||||
|
|
||||||
|
The pipeline is the integration point for the route. It:
|
||||||
|
1. Catches all exceptions (quality checks must NEVER break a translation job)
|
||||||
|
2. Adds timing
|
||||||
|
3. Emits a single structured log line per job
|
||||||
|
4. For L1, reads configuration from environment and is opt-in via
|
||||||
|
a flag passed in by the route (the route reads config.py).
|
||||||
|
|
||||||
|
L0 = pure-Python script + length + pattern checks. No API calls.
|
||||||
|
L1 = cheap LLM judge via API. Sampled (5 chunks per job by default).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from typing import List, Optional, Set
|
||||||
|
|
||||||
|
from core.logging import get_logger
|
||||||
|
|
||||||
|
from .script_detector import evaluate_document, DocumentQualityResult
|
||||||
|
from .sampler import sample_chunks_for_l1
|
||||||
|
from .llm_judge import L1Result, LLMJudge
|
||||||
|
from .l2_judge import L2Result, L2ProJudge
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- L0 ----------
|
||||||
|
|
||||||
|
def run_l0_check(
|
||||||
|
source_chunks: List[str],
|
||||||
|
translated_chunks: List[str],
|
||||||
|
target_lang: Optional[str],
|
||||||
|
job_id: Optional[str] = None,
|
||||||
|
file_extension: Optional[str] = None,
|
||||||
|
) -> DocumentQualityResult:
|
||||||
|
"""
|
||||||
|
Run the L0 quality checks defensively. Never raises.
|
||||||
|
"""
|
||||||
|
start = time.time()
|
||||||
|
empty = DocumentQualityResult(
|
||||||
|
passed=True,
|
||||||
|
score=0.0,
|
||||||
|
chunk_count=0,
|
||||||
|
failed_chunk_count=0,
|
||||||
|
issues={"internal_error": 1},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Best-effort import of metrics; never let it break the check.
|
||||||
|
try:
|
||||||
|
from middleware.metrics import record_l0_result, record_l0_error
|
||||||
|
except Exception:
|
||||||
|
record_l0_result = None
|
||||||
|
record_l0_error = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = evaluate_document(source_chunks, translated_chunks, target_lang)
|
||||||
|
except Exception as e:
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
logger.warning(
|
||||||
|
"quality_l0_check_failed",
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
error=str(e)[:200],
|
||||||
|
error_type=type(e).__name__,
|
||||||
|
elapsed_ms=elapsed_ms,
|
||||||
|
)
|
||||||
|
if record_l0_error:
|
||||||
|
try:
|
||||||
|
record_l0_error(file_type=file_extension or "unknown")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return empty
|
||||||
|
|
||||||
|
elapsed_ms = round((time.time() - start) * 1000, 2)
|
||||||
|
logger.info(
|
||||||
|
"quality_l0_check",
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
target_lang=target_lang,
|
||||||
|
chunk_count=result.chunk_count,
|
||||||
|
failed_chunk_count=result.failed_chunk_count,
|
||||||
|
score=result.score,
|
||||||
|
passed=result.passed,
|
||||||
|
issues=result.issues,
|
||||||
|
elapsed_ms=elapsed_ms,
|
||||||
|
)
|
||||||
|
if record_l0_result:
|
||||||
|
try:
|
||||||
|
record_l0_result(
|
||||||
|
passed=bool(result.passed),
|
||||||
|
file_type=file_extension or "unknown",
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- L1 ----------
|
||||||
|
|
||||||
|
# Common ISO codes → language name for the prompt
|
||||||
|
_LANG_NAMES = {
|
||||||
|
"en": "English", "fr": "French", "es": "Spanish", "de": "German",
|
||||||
|
"it": "Italian", "pt": "Portuguese", "nl": "Dutch", "ru": "Russian",
|
||||||
|
"ja": "Japanese", "ko": "Korean", "zh": "Chinese", "ar": "Arabic",
|
||||||
|
"fa": "Persian", "hi": "Hindi", "tr": "Turkish", "pl": "Polish",
|
||||||
|
"vi": "Vietnamese", "th": "Thai", "id": "Indonesian", "ms": "Malay",
|
||||||
|
"uk": "Ukrainian", "cs": "Czech", "sv": "Swedish", "ro": "Romanian",
|
||||||
|
"hu": "Hungarian", "el": "Greek", "he": "Hebrew", "da": "Danish",
|
||||||
|
"fi": "Finnish", "no": "Norwegian", "bg": "Bulgarian", "hr": "Croatian",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def run_l1_check(
|
||||||
|
source_chunks: List[str],
|
||||||
|
translated_chunks: List[str],
|
||||||
|
target_lang: Optional[str],
|
||||||
|
l0_failed_indices: Optional[Set[int]] = None,
|
||||||
|
job_id: Optional[str] = None,
|
||||||
|
file_extension: Optional[str] = None,
|
||||||
|
max_samples: int = 5,
|
||||||
|
min_chunks: int = 10,
|
||||||
|
judge: Optional[LLMJudge] = None,
|
||||||
|
log_only: bool = True,
|
||||||
|
) -> L1Result:
|
||||||
|
"""
|
||||||
|
Run the L1 LLM judge check on a sample of chunks.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
source_chunks: Original texts.
|
||||||
|
translated_chunks: Translated texts.
|
||||||
|
target_lang: Target language code (e.g. "fr", "en").
|
||||||
|
l0_failed_indices: Indices that L0 flagged as bad — skipped.
|
||||||
|
job_id: For logging.
|
||||||
|
file_extension: For logging.
|
||||||
|
max_samples: How many chunks to send to the LLM.
|
||||||
|
min_chunks: Skip the check if document has fewer chunks.
|
||||||
|
judge: An LLMJudge instance. If None, created from env vars.
|
||||||
|
log_only: If True, never propagate the verdict (observation mode).
|
||||||
|
If False, the caller can decide what to do with the verdict.
|
||||||
|
|
||||||
|
Returns an L1Result. verdict="skip" on any internal error.
|
||||||
|
Never raises — defensive wrapper.
|
||||||
|
"""
|
||||||
|
skip = L1Result(
|
||||||
|
verdict="skip",
|
||||||
|
chunks_evaluated=0, chunks_passed=0, chunks_failed=0,
|
||||||
|
failure_rate=0.0, error="not_run",
|
||||||
|
)
|
||||||
|
|
||||||
|
if l0_failed_indices is None:
|
||||||
|
l0_failed_indices = set()
|
||||||
|
|
||||||
|
# Sample
|
||||||
|
sample = sample_chunks_for_l1(
|
||||||
|
source_chunks, translated_chunks, l0_failed_indices,
|
||||||
|
max_samples=max_samples, min_chunks=min_chunks,
|
||||||
|
)
|
||||||
|
if not sample:
|
||||||
|
logger.info(
|
||||||
|
"quality_l1_check_skipped",
|
||||||
|
job_id=job_id,
|
||||||
|
reason="insufficient_chunks_or_all_flagged",
|
||||||
|
chunk_count=len(source_chunks),
|
||||||
|
)
|
||||||
|
_record_l1_metric(verdict="skip", model="none")
|
||||||
|
return skip
|
||||||
|
|
||||||
|
# Get the judge
|
||||||
|
if judge is None:
|
||||||
|
judge = make_judge_from_env_safe()
|
||||||
|
|
||||||
|
if judge is None:
|
||||||
|
logger.info(
|
||||||
|
"quality_l1_check_skipped",
|
||||||
|
job_id=job_id,
|
||||||
|
reason="no_judge_configured",
|
||||||
|
)
|
||||||
|
_record_l1_metric(verdict="skip", model="none")
|
||||||
|
return skip
|
||||||
|
|
||||||
|
# Get the language name for the prompt
|
||||||
|
target_lang_name = _LANG_NAMES.get((target_lang or "").lower(), target_lang or "auto")
|
||||||
|
|
||||||
|
# Call the LLM
|
||||||
|
try:
|
||||||
|
result = await judge.judge_batch(sample, target_lang or "auto", target_lang_name)
|
||||||
|
except Exception as e:
|
||||||
|
elapsed_ms = 0.0
|
||||||
|
logger.warning(
|
||||||
|
"quality_l1_check_failed",
|
||||||
|
job_id=job_id,
|
||||||
|
error=str(e)[:200],
|
||||||
|
error_type=type(e).__name__,
|
||||||
|
)
|
||||||
|
_record_l1_metric(verdict="error", model="unknown")
|
||||||
|
return L1Result(
|
||||||
|
verdict="skip",
|
||||||
|
chunks_evaluated=0, chunks_passed=0, chunks_failed=0,
|
||||||
|
failure_rate=0.0, error=str(e)[:200], elapsed_ms=elapsed_ms,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Log (always) — caller decides what to do
|
||||||
|
logger.info(
|
||||||
|
"quality_l1_check",
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
target_lang=target_lang,
|
||||||
|
verdict=result.verdict,
|
||||||
|
chunks_evaluated=result.chunks_evaluated,
|
||||||
|
chunks_passed=result.chunks_passed,
|
||||||
|
chunks_failed=result.chunks_failed,
|
||||||
|
failure_rate=result.failure_rate,
|
||||||
|
model=result.model_used,
|
||||||
|
elapsed_ms=result.elapsed_ms,
|
||||||
|
cost_estimate_usd=result.cost_estimate_usd,
|
||||||
|
log_only=log_only,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Record Prometheus metric (best-effort, never breaks the job)
|
||||||
|
duration_s = None
|
||||||
|
if result.elapsed_ms is not None:
|
||||||
|
try:
|
||||||
|
duration_s = float(result.elapsed_ms) / 1000.0
|
||||||
|
except Exception:
|
||||||
|
duration_s = None
|
||||||
|
_record_l1_metric(
|
||||||
|
verdict=result.verdict or "skip",
|
||||||
|
model=result.model_used or "unknown",
|
||||||
|
duration_seconds=duration_s,
|
||||||
|
cost_usd=result.cost_estimate_usd,
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _record_l1_metric(
|
||||||
|
verdict: str,
|
||||||
|
model: str = "unknown",
|
||||||
|
duration_seconds: float = None,
|
||||||
|
cost_usd: float = None,
|
||||||
|
) -> None:
|
||||||
|
"""Best-effort Prometheus metric emission for L1.
|
||||||
|
|
||||||
|
Never raises — if the metrics module fails to import or the
|
||||||
|
counter is unavailable, we silently skip.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from middleware.metrics import record_l1_verdict
|
||||||
|
record_l1_verdict(
|
||||||
|
verdict=verdict,
|
||||||
|
model=model,
|
||||||
|
duration_seconds=duration_seconds,
|
||||||
|
cost_usd=cost_usd,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def make_judge_from_env_safe() -> Optional[LLMJudge]:
|
||||||
|
"""Read env vars and build a judge, or return None if not configured.
|
||||||
|
|
||||||
|
This is a thin wrapper around `llm_judge.make_judge_from_env` that
|
||||||
|
catches any import-time or init-time error and returns None — so
|
||||||
|
a misconfigured L1 environment NEVER breaks a translation job.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from .llm_judge import make_judge_from_env
|
||||||
|
return make_judge_from_env()
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("l1_judge_init_failed", error=str(e)[:200])
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- L2 (Pro tier) ----------
|
||||||
|
|
||||||
|
async def run_l2_check(
|
||||||
|
source_chunks: List[str],
|
||||||
|
translated_chunks: List[str],
|
||||||
|
target_lang: Optional[str],
|
||||||
|
l0_failed_indices: Optional[Set[int]] = None,
|
||||||
|
job_id: Optional[str] = None,
|
||||||
|
file_extension: Optional[str] = None,
|
||||||
|
max_samples: int = 15,
|
||||||
|
min_chunks: int = 20,
|
||||||
|
judge: Optional[L2ProJudge] = None,
|
||||||
|
log_only: bool = True,
|
||||||
|
) -> L2Result:
|
||||||
|
"""
|
||||||
|
Run the L2 Pro premium judge (8 dimensions, gpt-4o default).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
source_chunks: Original texts.
|
||||||
|
translated_chunks: Translated texts.
|
||||||
|
target_lang: Target language code (e.g. "fr", "en").
|
||||||
|
l0_failed_indices: Indices that L0 flagged as bad — skipped.
|
||||||
|
job_id: For logging.
|
||||||
|
file_extension: For logging.
|
||||||
|
max_samples: How many chunks to send to the LLM.
|
||||||
|
min_chunks: Skip the check if document has fewer chunks.
|
||||||
|
judge: An L2ProJudge instance. If None, created from env vars.
|
||||||
|
log_only: If True, never propagate the verdict (observation mode).
|
||||||
|
If False, the caller can decide what to do with the verdict.
|
||||||
|
|
||||||
|
Returns an L2Result. verdict="skip" on any internal error.
|
||||||
|
Never raises — defensive wrapper.
|
||||||
|
"""
|
||||||
|
skip = L2Result(verdict="skip", error="not_run")
|
||||||
|
|
||||||
|
if l0_failed_indices is None:
|
||||||
|
l0_failed_indices = set()
|
||||||
|
|
||||||
|
# Sample (reuse the L1 sampler — it's just chunk selection, model-agnostic)
|
||||||
|
sample = sample_chunks_for_l1(
|
||||||
|
source_chunks, translated_chunks, l0_failed_indices,
|
||||||
|
max_samples=max_samples, min_chunks=min_chunks,
|
||||||
|
)
|
||||||
|
if not sample:
|
||||||
|
logger.info(
|
||||||
|
"quality_l2_check_skipped",
|
||||||
|
job_id=job_id,
|
||||||
|
reason="insufficient_chunks_or_all_flagged",
|
||||||
|
chunk_count=len(source_chunks),
|
||||||
|
)
|
||||||
|
_record_l2_metric(verdict="skip", model="none")
|
||||||
|
return skip
|
||||||
|
|
||||||
|
# Get the judge
|
||||||
|
if judge is None:
|
||||||
|
judge = make_l2_judge_from_env_safe()
|
||||||
|
|
||||||
|
if judge is None:
|
||||||
|
logger.info(
|
||||||
|
"quality_l2_check_skipped",
|
||||||
|
job_id=job_id,
|
||||||
|
reason="no_l2_judge_configured",
|
||||||
|
)
|
||||||
|
_record_l2_metric(verdict="skip", model="none")
|
||||||
|
return skip
|
||||||
|
|
||||||
|
# Get the language name for the prompt
|
||||||
|
target_lang_name = _LANG_NAMES.get((target_lang or "").lower(), target_lang or "auto")
|
||||||
|
|
||||||
|
# Call the LLM
|
||||||
|
try:
|
||||||
|
result = await judge.judge_batch(sample, target_lang or "auto", target_lang_name)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
"quality_l2_check_failed",
|
||||||
|
job_id=job_id,
|
||||||
|
error=str(e)[:200],
|
||||||
|
error_type=type(e).__name__,
|
||||||
|
)
|
||||||
|
_record_l2_metric(verdict="error", model="unknown")
|
||||||
|
return L2Result(verdict="skip", error=str(e)[:200])
|
||||||
|
|
||||||
|
# Log (always) — caller decides what to do
|
||||||
|
logger.info(
|
||||||
|
"quality_l2_check",
|
||||||
|
job_id=job_id,
|
||||||
|
file_extension=file_extension,
|
||||||
|
target_lang=target_lang,
|
||||||
|
verdict=result.verdict,
|
||||||
|
chunks_evaluated=result.chunks_evaluated,
|
||||||
|
chunks_passed=result.chunks_passed,
|
||||||
|
chunks_failed=result.chunks_failed,
|
||||||
|
failure_rate=result.failure_rate,
|
||||||
|
average_score=result.average_score,
|
||||||
|
dimension_pass_rates=result.dimension_pass_rates,
|
||||||
|
model=result.model_used,
|
||||||
|
elapsed_ms=result.elapsed_ms,
|
||||||
|
cost_estimate_usd=result.cost_estimate_usd,
|
||||||
|
log_only=log_only,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Record Prometheus metric
|
||||||
|
duration_s = None
|
||||||
|
if result.elapsed_ms is not None:
|
||||||
|
try:
|
||||||
|
duration_s = float(result.elapsed_ms) / 1000.0
|
||||||
|
except Exception:
|
||||||
|
duration_s = None
|
||||||
|
_record_l2_metric(
|
||||||
|
verdict=result.verdict or "skip",
|
||||||
|
model=result.model_used or "unknown",
|
||||||
|
duration_seconds=duration_s,
|
||||||
|
cost_usd=result.cost_estimate_usd,
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _record_l2_metric(
|
||||||
|
verdict: str,
|
||||||
|
model: str = "unknown",
|
||||||
|
duration_seconds: float = None,
|
||||||
|
cost_usd: float = None,
|
||||||
|
) -> None:
|
||||||
|
"""Best-effort Prometheus metric emission for L2.
|
||||||
|
|
||||||
|
Never raises.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from middleware.metrics import record_l2_verdict
|
||||||
|
record_l2_verdict(
|
||||||
|
verdict=verdict,
|
||||||
|
model=model,
|
||||||
|
duration_seconds=duration_seconds,
|
||||||
|
cost_usd=cost_usd,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def make_l2_judge_from_env_safe() -> Optional[L2ProJudge]:
|
||||||
|
"""Read env vars and build an L2 judge, or return None if not configured.
|
||||||
|
|
||||||
|
Defensive wrapper — a misconfigured L2 environment NEVER breaks a job.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from .l2_judge import make_l2_judge_from_env
|
||||||
|
return make_l2_judge_from_env()
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("l2_judge_init_failed", error=str(e)[:200])
|
||||||
|
return None
|
||||||
79
services/quality/sampler.py
Normal file
79
services/quality/sampler.py
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
"""
|
||||||
|
Stratified sampler for the L1 quality layer.
|
||||||
|
|
||||||
|
The L1 layer sends a small number of (source, translation) pairs to a
|
||||||
|
cheap LLM for a quality verdict. We don't want to send ALL chunks
|
||||||
|
(cost, latency) and we don't want to send random chunks (might miss
|
||||||
|
the problematic ones). We use a simple but effective strategy:
|
||||||
|
|
||||||
|
- Prefer the LONGEST chunks (they contain the most diagnostic
|
||||||
|
information per call).
|
||||||
|
- Skip chunks that the L0 already flagged (we already know they're
|
||||||
|
bad; we don't need the LLM to confirm).
|
||||||
|
- Never sample more than `max_samples` chunks.
|
||||||
|
- If `min_chunks` chunks aren't available, skip the L1 entirely
|
||||||
|
(small documents don't need it).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import List, Tuple
|
||||||
|
|
||||||
|
|
||||||
|
def sample_chunks_for_l1(
|
||||||
|
source_chunks: List[str],
|
||||||
|
translated_chunks: List[str],
|
||||||
|
failed_indices: set,
|
||||||
|
max_samples: int = 5,
|
||||||
|
min_chunks: int = 10,
|
||||||
|
) -> List[Tuple[str, str]]:
|
||||||
|
"""
|
||||||
|
Select a sample of (source, translation) pairs to send to the L1 judge.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
source_chunks: Original texts.
|
||||||
|
translated_chunks: Translated texts (same length as source_chunks).
|
||||||
|
failed_indices: Set of indices that L0 already flagged as bad.
|
||||||
|
These are SKIPPED — we want fresh signal, not
|
||||||
|
confirmation of an already-detected failure.
|
||||||
|
max_samples: Maximum number of pairs to return.
|
||||||
|
min_chunks: If the document has fewer than this many chunks,
|
||||||
|
return an empty list (not enough signal to bother
|
||||||
|
calling the LLM).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A list of (source, translated) tuples, ready to send to the LLM
|
||||||
|
judge. Order: longest chunks first.
|
||||||
|
"""
|
||||||
|
n = min(len(source_chunks), len(translated_chunks))
|
||||||
|
|
||||||
|
if n < min_chunks:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Build candidate list, excluding L0 failures
|
||||||
|
candidates: List[Tuple[int, int, str, str]] = []
|
||||||
|
for i in range(n):
|
||||||
|
if i in failed_indices:
|
||||||
|
continue
|
||||||
|
src = (source_chunks[i] or "").strip()
|
||||||
|
trans = (translated_chunks[i] or "").strip()
|
||||||
|
# Skip very short pairs (not enough context for the LLM to judge)
|
||||||
|
if len(src) < 5 and len(trans) < 5:
|
||||||
|
continue
|
||||||
|
# Skip pairs where source and translation are identical
|
||||||
|
# (probably a non-translated cell like a number, code, or brand)
|
||||||
|
if src == trans:
|
||||||
|
continue
|
||||||
|
# Rank by length of the translation (longer = more diagnostic)
|
||||||
|
rank = len(trans) + len(src) // 2
|
||||||
|
candidates.append((rank, i, src, trans))
|
||||||
|
|
||||||
|
# Sort by length descending and take top N
|
||||||
|
candidates.sort(key=lambda c: c[0], reverse=True)
|
||||||
|
selected = candidates[:max_samples]
|
||||||
|
|
||||||
|
# Return in original document order (helps the LLM judge maintain
|
||||||
|
# context, and makes the verdict easier to interpret)
|
||||||
|
selected.sort(key=lambda c: c[1])
|
||||||
|
|
||||||
|
return [(src, trans) for _rank, _i, src, trans in selected]
|
||||||
374
services/quality/script_detector.py
Normal file
374
services/quality/script_detector.py
Normal file
@@ -0,0 +1,374 @@
|
|||||||
|
"""
|
||||||
|
L0 script detector.
|
||||||
|
|
||||||
|
Verifies that a translated string is actually written in the script expected
|
||||||
|
for the target language.
|
||||||
|
|
||||||
|
This is the first line of defense against the most common translation
|
||||||
|
failure mode: the LLM hallucinates text in the wrong language or wrong
|
||||||
|
script (e.g. user asks for Persian, model returns Arabic, or user asks
|
||||||
|
for Hindi, model returns Arabic). The check is purely heuristic — it
|
||||||
|
counts code points in the relevant Unicode ranges and compares to a
|
||||||
|
threshold.
|
||||||
|
|
||||||
|
Pure Python. No network calls. No new dependencies.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
|
||||||
|
from . import config as _config
|
||||||
|
from . import length_checker
|
||||||
|
from . import pattern_leak
|
||||||
|
|
||||||
|
from core.logging import get_logger
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Result dataclasses ----------
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class QualityCheckResult:
|
||||||
|
"""Result of evaluating a single (source, translation) pair."""
|
||||||
|
passed: bool
|
||||||
|
score: float # 0.0 to 1.0
|
||||||
|
issues: List[str] = field(default_factory=list)
|
||||||
|
detected_script: Optional[str] = None
|
||||||
|
expected_script: Optional[str] = None
|
||||||
|
details: Dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
def to_log_dict(self) -> Dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class DocumentQualityResult:
|
||||||
|
"""Aggregated result for a list of (source, translation) pairs."""
|
||||||
|
passed: bool
|
||||||
|
score: float # mean score across chunks
|
||||||
|
chunk_count: int
|
||||||
|
failed_chunk_count: int
|
||||||
|
issues: Dict[str, int] = field(default_factory=dict) # issue -> count
|
||||||
|
samples: List[Dict] = field(default_factory=list) # a few example failures
|
||||||
|
|
||||||
|
def to_log_dict(self) -> Dict:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Core helpers ----------
|
||||||
|
|
||||||
|
def _char_in_ranges(code_point: int, ranges: list) -> bool:
|
||||||
|
"""True if a code point falls in any of the (start, end) ranges."""
|
||||||
|
for start, end in ranges:
|
||||||
|
if start <= code_point <= end:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _count_letters(text: str) -> int:
|
||||||
|
"""Count alphabetic characters (using Python's built-in isalpha)."""
|
||||||
|
return sum(1 for c in text if c.isalpha())
|
||||||
|
|
||||||
|
|
||||||
|
def _count_in_script(text: str, ranges: list) -> int:
|
||||||
|
"""Count how many alphabetic characters fall within the given Unicode ranges."""
|
||||||
|
if not ranges:
|
||||||
|
# 'latin' or unknown — treat all letters as matching.
|
||||||
|
return _count_letters(text)
|
||||||
|
return sum(
|
||||||
|
1 for c in text
|
||||||
|
if c.isalpha() and _char_in_ranges(ord(c), ranges)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Arabic-script variant detection ----------
|
||||||
|
|
||||||
|
def detect_arabic_variant(
|
||||||
|
text: str,
|
||||||
|
claimed_lang: Optional[str],
|
||||||
|
) -> Dict:
|
||||||
|
"""
|
||||||
|
For text that is in the Arabic script block, check whether it matches
|
||||||
|
the specific variant the user asked for (Persian, Urdu, Pashto, etc.).
|
||||||
|
|
||||||
|
Returns a dict like:
|
||||||
|
{
|
||||||
|
"verdict": "pass" | "fail" | "skip",
|
||||||
|
"claimed_lang": "fa",
|
||||||
|
"detected_variants": ["fa"],
|
||||||
|
"reason": "...",
|
||||||
|
}
|
||||||
|
|
||||||
|
Detection logic:
|
||||||
|
1. If the text has < 60% Arabic-script letters overall, verdict = "skip"
|
||||||
|
(the script-detector will catch the mismatch).
|
||||||
|
2. If claimed_lang is NOT an Arabic-script language, verdict = "fail"
|
||||||
|
(this case should have been caught upstream — defensive double-check).
|
||||||
|
3. Scan the text for any discriminating character from any
|
||||||
|
Arabic-script language. If a discriminating character of a
|
||||||
|
DIFFERENT language is found, verdict = "fail".
|
||||||
|
4. Otherwise verdict = "pass".
|
||||||
|
"""
|
||||||
|
if not text or not text.strip():
|
||||||
|
return {"verdict": "skip", "claimed_lang": claimed_lang, "reason": "empty text"}
|
||||||
|
|
||||||
|
arabic_ranges = _config.get_ranges("arabic")
|
||||||
|
letters = _count_letters(text)
|
||||||
|
if letters == 0:
|
||||||
|
return {"verdict": "skip", "claimed_lang": claimed_lang, "reason": "no letters"}
|
||||||
|
|
||||||
|
in_arabic = _count_in_script(text, arabic_ranges)
|
||||||
|
arabic_ratio = in_arabic / letters
|
||||||
|
|
||||||
|
if arabic_ratio < _config.MIN_RATIO_IN_SCRIPT:
|
||||||
|
# Not really Arabic-script — let the main script_detector handle it.
|
||||||
|
return {
|
||||||
|
"verdict": "skip",
|
||||||
|
"claimed_lang": claimed_lang,
|
||||||
|
"arabic_ratio": round(arabic_ratio, 3),
|
||||||
|
"reason": "not in Arabic script",
|
||||||
|
}
|
||||||
|
|
||||||
|
if not _config.is_arabic_script_lang(claimed_lang):
|
||||||
|
# The translation IS in Arabic but the target wasn't Arabic.
|
||||||
|
# The main script_detector will fail on this; we just return skip.
|
||||||
|
return {
|
||||||
|
"verdict": "skip",
|
||||||
|
"claimed_lang": claimed_lang,
|
||||||
|
"arabic_ratio": round(arabic_ratio, 3),
|
||||||
|
"reason": "target is not an Arabic-script language",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Now: text is Arabic-script AND target is Arabic-script. Check the variant.
|
||||||
|
detected = set()
|
||||||
|
for lang_code, chars in _config.DISCRIMINATING_CHARS.items():
|
||||||
|
if not chars:
|
||||||
|
continue
|
||||||
|
if any(c in chars for c in text):
|
||||||
|
detected.add(lang_code)
|
||||||
|
|
||||||
|
if detected and claimed_lang and claimed_lang.lower() not in detected:
|
||||||
|
return {
|
||||||
|
"verdict": "fail",
|
||||||
|
"claimed_lang": claimed_lang,
|
||||||
|
"detected_variants": sorted(detected),
|
||||||
|
"arabic_ratio": round(arabic_ratio, 3),
|
||||||
|
"reason": (
|
||||||
|
f"target={claimed_lang} but text contains characters typical of "
|
||||||
|
f"{', '.join(sorted(detected))}"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"verdict": "pass",
|
||||||
|
"claimed_lang": claimed_lang,
|
||||||
|
"detected_variants": sorted(detected) if detected else [claimed_lang],
|
||||||
|
"arabic_ratio": round(arabic_ratio, 3),
|
||||||
|
"reason": "ok",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Per-chunk evaluation ----------
|
||||||
|
|
||||||
|
def evaluate_chunk(
|
||||||
|
source_text: str,
|
||||||
|
translated_text: str,
|
||||||
|
target_lang: Optional[str],
|
||||||
|
) -> QualityCheckResult:
|
||||||
|
"""
|
||||||
|
Run the L0 checks on a single (source, translation) pair.
|
||||||
|
|
||||||
|
Returns a QualityCheckResult. The function is purely defensive — it
|
||||||
|
never raises; any internal error results in a "skip" result.
|
||||||
|
"""
|
||||||
|
if translated_text is None:
|
||||||
|
return QualityCheckResult(
|
||||||
|
passed=True, score=0.0, issues=["empty_translation"],
|
||||||
|
details={"reason": "translation is None"},
|
||||||
|
)
|
||||||
|
|
||||||
|
text = translated_text.strip()
|
||||||
|
if not text:
|
||||||
|
return QualityCheckResult(
|
||||||
|
passed=True, score=0.0, issues=["empty_translation"],
|
||||||
|
details={"reason": "translation is empty or whitespace-only"},
|
||||||
|
)
|
||||||
|
|
||||||
|
target_lang = (target_lang or "").lower() or None
|
||||||
|
issues: List[str] = []
|
||||||
|
details: Dict = {}
|
||||||
|
|
||||||
|
# --- Script detection ---
|
||||||
|
expected_script = _config.get_script(target_lang)
|
||||||
|
expected_ranges = _config.get_ranges(expected_script)
|
||||||
|
letters = _count_letters(text)
|
||||||
|
|
||||||
|
if letters == 0:
|
||||||
|
# No alphabetic characters — could be numbers, punctuation, or
|
||||||
|
# a single non-Latin symbol. Skip script check.
|
||||||
|
script_score = 1.0
|
||||||
|
detected_script = expected_script
|
||||||
|
details["script_check"] = "skipped: no alphabetic characters"
|
||||||
|
else:
|
||||||
|
# Always try to determine the ACTUAL script of the text — used for
|
||||||
|
# diagnostics and for catching language confusion when the target
|
||||||
|
# is Latin (e.g. user asks fr, we get Arabic text).
|
||||||
|
detected_script = _detect_actual_script(text)
|
||||||
|
in_expected = _count_in_script(text, expected_ranges)
|
||||||
|
script_score = in_expected / letters
|
||||||
|
|
||||||
|
details["script_score"] = round(script_score, 3)
|
||||||
|
details["letters_in_text"] = letters
|
||||||
|
details["letters_in_script"] = in_expected
|
||||||
|
details["detected_script"] = detected_script
|
||||||
|
details["expected_script"] = expected_script
|
||||||
|
details["min_ratio"] = _config.MIN_RATIO_IN_SCRIPT
|
||||||
|
|
||||||
|
# Two failure modes:
|
||||||
|
# 1. Target is a SPECIFIC non-Latin script (cyrillic, arabic, cjk...)
|
||||||
|
# and the text doesn't match it enough.
|
||||||
|
# 2. Target is Latin but the text is clearly in a SPECIFIC other
|
||||||
|
# script (cyrillic, arabic, devanagari, cjk...) — language
|
||||||
|
# confusion.
|
||||||
|
if expected_script != "latin" and expected_ranges:
|
||||||
|
# Specific non-Latin target.
|
||||||
|
if script_score < _config.MIN_RATIO_IN_SCRIPT:
|
||||||
|
issues.append("wrong_script")
|
||||||
|
details["reason"] = (
|
||||||
|
f"only {script_score:.0%} of letters match {expected_script} script; "
|
||||||
|
f"text appears to be in {detected_script}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Latin target. If detected script is clearly non-Latin, fail.
|
||||||
|
if detected_script and detected_script != "latin" and detected_script != "unknown":
|
||||||
|
# Measure how confident we are that the text is non-Latin.
|
||||||
|
non_latin_ranges = _config.get_ranges(detected_script)
|
||||||
|
in_detected = _count_in_script(text, non_latin_ranges)
|
||||||
|
non_latin_confidence = in_detected / letters
|
||||||
|
if non_latin_confidence >= 0.7:
|
||||||
|
issues.append("wrong_script")
|
||||||
|
details["reason"] = (
|
||||||
|
f"target is Latin but {non_latin_confidence:.0%} of letters "
|
||||||
|
f"are in {detected_script} script — language confusion"
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- Arabic-script variant detection ---
|
||||||
|
if _config.is_arabic_script_lang(target_lang):
|
||||||
|
variant_result = detect_arabic_variant(text, target_lang)
|
||||||
|
details["arabic_variant"] = variant_result
|
||||||
|
if variant_result["verdict"] == "fail":
|
||||||
|
issues.append("wrong_arabic_variant")
|
||||||
|
|
||||||
|
# --- Length sanity ---
|
||||||
|
length_result = length_checker.check(source_text, text)
|
||||||
|
details["length"] = length_result
|
||||||
|
if length_result.get("issue"):
|
||||||
|
issues.append(length_result["issue"])
|
||||||
|
|
||||||
|
# --- Pattern leak / repetition ---
|
||||||
|
leak_result = pattern_leak.check(text)
|
||||||
|
details["pattern_check"] = leak_result
|
||||||
|
if leak_result.get("issue"):
|
||||||
|
issues.append(leak_result["issue"])
|
||||||
|
|
||||||
|
# --- Aggregate ---
|
||||||
|
passed = len(issues) == 0
|
||||||
|
# Simple score: how many of the 3 main checks passed.
|
||||||
|
n_checks = 3
|
||||||
|
n_failed = sum(
|
||||||
|
1 for issue in issues if issue in (
|
||||||
|
"wrong_script", "wrong_arabic_variant",
|
||||||
|
"length_outlier", "truncation_suspect",
|
||||||
|
"prompt_leak", "repetition_hallucination",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
score = max(0.0, 1.0 - (n_failed / n_checks))
|
||||||
|
|
||||||
|
return QualityCheckResult(
|
||||||
|
passed=passed,
|
||||||
|
score=round(score, 3),
|
||||||
|
issues=issues,
|
||||||
|
detected_script=detected_script,
|
||||||
|
expected_script=expected_script,
|
||||||
|
details=details,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _detect_actual_script(text: str) -> str:
|
||||||
|
"""
|
||||||
|
Heuristically determine which script a string is in. Used only for
|
||||||
|
diagnostics — never for the verdict. Returns the first script (in
|
||||||
|
priority order) whose ratio exceeds the threshold.
|
||||||
|
"""
|
||||||
|
letters = _count_letters(text)
|
||||||
|
if letters == 0:
|
||||||
|
return "unknown"
|
||||||
|
# Priority order: more specific scripts first.
|
||||||
|
order = [
|
||||||
|
"hiragana_katakana", "hangul", "cjk", "thai", "lao", "burmese",
|
||||||
|
"khmer", "devanagari", "bengali", "tamil", "telugu", "kannada",
|
||||||
|
"malayalam", "sinhala", "gujarati", "gurmukhi",
|
||||||
|
"arabic", "hebrew", "cyrillic", "greek", "armenian", "georgian",
|
||||||
|
"ethiopic", "tibetan", "thaana",
|
||||||
|
]
|
||||||
|
for script_id in order:
|
||||||
|
ranges = _config.get_ranges(script_id)
|
||||||
|
in_script = _count_in_script(text, ranges)
|
||||||
|
if in_script / letters > 0.4:
|
||||||
|
return script_id
|
||||||
|
return "latin"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Document-level aggregation ----------
|
||||||
|
|
||||||
|
def evaluate_document(
|
||||||
|
source_chunks: List[str],
|
||||||
|
translated_chunks: List[str],
|
||||||
|
target_lang: Optional[str],
|
||||||
|
sample_size: int = 50,
|
||||||
|
) -> DocumentQualityResult:
|
||||||
|
"""
|
||||||
|
Evaluate all (source, translation) pairs and return a document-level
|
||||||
|
summary. The full list is processed but only the first `sample_size`
|
||||||
|
failing chunks are kept in `samples` to keep logs compact.
|
||||||
|
"""
|
||||||
|
n = min(len(source_chunks), len(translated_chunks))
|
||||||
|
chunk_results: List[QualityCheckResult] = []
|
||||||
|
issues_count: Dict[str, int] = {}
|
||||||
|
samples: List[Dict] = []
|
||||||
|
score_sum = 0.0
|
||||||
|
failed_count = 0
|
||||||
|
|
||||||
|
for i in range(n):
|
||||||
|
r = evaluate_chunk(source_chunks[i], translated_chunks[i], target_lang)
|
||||||
|
chunk_results.append(r)
|
||||||
|
score_sum += r.score
|
||||||
|
for issue in r.issues:
|
||||||
|
issues_count[issue] = issues_count.get(issue, 0) + 1
|
||||||
|
if not r.passed:
|
||||||
|
failed_count += 1
|
||||||
|
if len(samples) < sample_size:
|
||||||
|
src_preview = (source_chunks[i] or "")[:80]
|
||||||
|
trans_preview = (translated_chunks[i] or "")[:80]
|
||||||
|
samples.append({
|
||||||
|
"index": i,
|
||||||
|
"issues": r.issues,
|
||||||
|
"source_preview": src_preview,
|
||||||
|
"translated_preview": trans_preview,
|
||||||
|
"details": r.details,
|
||||||
|
})
|
||||||
|
|
||||||
|
mean_score = (score_sum / n) if n > 0 else 0.0
|
||||||
|
passed = failed_count == 0
|
||||||
|
|
||||||
|
return DocumentQualityResult(
|
||||||
|
passed=passed,
|
||||||
|
score=round(mean_score, 3),
|
||||||
|
chunk_count=n,
|
||||||
|
failed_chunk_count=failed_count,
|
||||||
|
issues=issues_count,
|
||||||
|
samples=samples,
|
||||||
|
)
|
||||||
427
services/translation_cache.py
Normal file
427
services/translation_cache.py
Normal file
@@ -0,0 +1,427 @@
|
|||||||
|
"""
|
||||||
|
Track C2 — Translation cache with Redis backend.
|
||||||
|
|
||||||
|
This module provides a translation cache that:
|
||||||
|
- Defaults to in-process LRU (same as the previous behavior)
|
||||||
|
- Optionally uses Redis when REDIS_CACHE_ENABLED=true
|
||||||
|
- Namespaces keys by user_id (so free and pro users don't share)
|
||||||
|
- Includes custom_prompt_hash + glossary_hash in the key (so two
|
||||||
|
users with different prompts/glossaries don't get cross-contamination)
|
||||||
|
- Falls back to LRU if Redis is unavailable (transient connection
|
||||||
|
error, etc.) — a Redis outage MUST NOT break translation jobs
|
||||||
|
|
||||||
|
Public API:
|
||||||
|
TranslationCache — single instance per process, async-friendly.
|
||||||
|
get_cache() — singleton accessor.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
cache = get_cache()
|
||||||
|
cached = await cache.get(text, target, source, provider, user_id, custom_prompt_hash, glossary_hash)
|
||||||
|
if cached is None:
|
||||||
|
translated = await provider.translate(text, target, source)
|
||||||
|
await cache.set(text, target, source, provider, user_id, custom_prompt_hash, glossary_hash, translated)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import asyncio
|
||||||
|
import threading
|
||||||
|
from collections import OrderedDict
|
||||||
|
from typing import Optional, Dict, List
|
||||||
|
|
||||||
|
from core.logging import get_logger
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Key construction
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
def make_cache_key(
|
||||||
|
text: str,
|
||||||
|
target_language: str,
|
||||||
|
source_language: str,
|
||||||
|
provider: str,
|
||||||
|
user_id: Optional[str] = None,
|
||||||
|
custom_prompt_hash: Optional[str] = None,
|
||||||
|
glossary_hash: Optional[str] = None,
|
||||||
|
) -> str:
|
||||||
|
"""Build a stable cache key.
|
||||||
|
|
||||||
|
The key includes:
|
||||||
|
- The text being translated
|
||||||
|
- The language pair
|
||||||
|
- The provider
|
||||||
|
- The user (so free/pro users don't collide)
|
||||||
|
- The custom_prompt hash (so different prompts are isolated)
|
||||||
|
- The glossary hash (so different glossaries are isolated)
|
||||||
|
"""
|
||||||
|
parts = [
|
||||||
|
f"provider={provider}",
|
||||||
|
f"src={source_language or 'auto'}",
|
||||||
|
f"tgt={target_language}",
|
||||||
|
f"user={user_id or 'anon'}",
|
||||||
|
f"prompt={custom_prompt_hash or 'none'}",
|
||||||
|
f"glossary={glossary_hash or 'none'}",
|
||||||
|
f"text={text}",
|
||||||
|
]
|
||||||
|
content = "|".join(parts)
|
||||||
|
return hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def hash_prompt(prompt: Optional[str]) -> Optional[str]:
|
||||||
|
"""Hash a custom prompt for use in a cache key. Returns None for None/empty."""
|
||||||
|
if not prompt:
|
||||||
|
return None
|
||||||
|
return hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
def hash_glossary_terms(terms: Optional[List[dict]]) -> Optional[str]:
|
||||||
|
"""Hash a glossary's term list for use in a cache key.
|
||||||
|
|
||||||
|
We only hash the source/target pairs (not metadata) so that two
|
||||||
|
glossaries with the same content but different names still share
|
||||||
|
cache entries.
|
||||||
|
"""
|
||||||
|
if not terms:
|
||||||
|
return None
|
||||||
|
# Stable representation: list of (source, target) tuples
|
||||||
|
normalized = sorted(
|
||||||
|
(str(t.get("source", "")), str(t.get("target", "")))
|
||||||
|
for t in terms
|
||||||
|
if t.get("source") and t.get("target")
|
||||||
|
)
|
||||||
|
content = json.dumps(normalized, ensure_ascii=False, sort_keys=True)
|
||||||
|
return hashlib.sha256(content.encode("utf-8")).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# In-process LRU fallback
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class _LRUBackend:
|
||||||
|
"""Thread-safe LRU cache, used as a fallback when Redis is unavailable."""
|
||||||
|
|
||||||
|
def __init__(self, maxsize: int = 5000):
|
||||||
|
self._cache: "OrderedDict[str, str]" = OrderedDict()
|
||||||
|
self._maxsize = maxsize
|
||||||
|
self._lock = threading.RLock()
|
||||||
|
self.hits = 0
|
||||||
|
self.misses = 0
|
||||||
|
|
||||||
|
def get(self, key: str) -> Optional[str]:
|
||||||
|
with self._lock:
|
||||||
|
if key in self._cache:
|
||||||
|
self.hits += 1
|
||||||
|
self._cache.move_to_end(key)
|
||||||
|
return self._cache[key]
|
||||||
|
self.misses += 1
|
||||||
|
return None
|
||||||
|
|
||||||
|
def set(self, key: str, value: str, ttl: Optional[int] = None) -> None:
|
||||||
|
# LRU doesn't honor TTL — it's a process-lifetime cache.
|
||||||
|
with self._lock:
|
||||||
|
if key in self._cache:
|
||||||
|
self._cache.move_to_end(key)
|
||||||
|
self._cache[key] = value
|
||||||
|
while len(self._cache) > self._maxsize:
|
||||||
|
self._cache.popitem(last=False)
|
||||||
|
|
||||||
|
def clear(self) -> None:
|
||||||
|
with self._lock:
|
||||||
|
self._cache.clear()
|
||||||
|
self.hits = 0
|
||||||
|
self.misses = 0
|
||||||
|
|
||||||
|
def stats(self) -> Dict:
|
||||||
|
with self._lock:
|
||||||
|
total = self.hits + self.misses
|
||||||
|
hit_rate = (self.hits / total * 100) if total > 0 else 0
|
||||||
|
return {
|
||||||
|
"backend": "lru",
|
||||||
|
"size": len(self._cache),
|
||||||
|
"maxsize": self._maxsize,
|
||||||
|
"hits": self.hits,
|
||||||
|
"misses": self.misses,
|
||||||
|
"hit_rate": f"{hit_rate:.1f}%",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Redis backend
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class _RedisBackend:
|
||||||
|
"""Redis-backed cache with TTL and async support.
|
||||||
|
|
||||||
|
Connection is lazy: we don't connect on import. If Redis is
|
||||||
|
unreachable, every operation falls back to the LRU backend.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
url: str,
|
||||||
|
namespace: str = "translate_cache",
|
||||||
|
default_ttl: int = 86400, # 24 hours
|
||||||
|
lru_fallback: Optional[_LRUBackend] = None,
|
||||||
|
socket_timeout: float = 2.0,
|
||||||
|
):
|
||||||
|
self._url = url
|
||||||
|
self._namespace = namespace
|
||||||
|
self._default_ttl = default_ttl
|
||||||
|
self._lru_fallback = lru_fallback
|
||||||
|
self._socket_timeout = socket_timeout
|
||||||
|
self._client = None
|
||||||
|
self._lock = threading.RLock()
|
||||||
|
self.hits = 0
|
||||||
|
self.misses = 0
|
||||||
|
self.errors = 0
|
||||||
|
self._available = None # None = unknown, True/False = probed
|
||||||
|
|
||||||
|
def _get_client(self):
|
||||||
|
"""Lazy client init. Never raises; returns None on failure."""
|
||||||
|
if self._client is not None:
|
||||||
|
return self._client
|
||||||
|
with self._lock:
|
||||||
|
if self._client is not None:
|
||||||
|
return self._client
|
||||||
|
try:
|
||||||
|
import redis
|
||||||
|
client = redis.from_url(
|
||||||
|
self._url,
|
||||||
|
socket_timeout=self._socket_timeout,
|
||||||
|
socket_connect_timeout=self._socket_timeout,
|
||||||
|
decode_responses=True,
|
||||||
|
)
|
||||||
|
# Probe once
|
||||||
|
client.ping()
|
||||||
|
self._client = client
|
||||||
|
self._available = True
|
||||||
|
logger.info("redis_cache_connected", url=self._url.split("@")[-1])
|
||||||
|
return client
|
||||||
|
except Exception as e:
|
||||||
|
self._available = False
|
||||||
|
self.errors += 1
|
||||||
|
logger.warning(
|
||||||
|
"redis_cache_unavailable",
|
||||||
|
error=str(e)[:200],
|
||||||
|
fallback="lru",
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _key(self, key: str) -> str:
|
||||||
|
return f"{self._namespace}:{key}"
|
||||||
|
|
||||||
|
def get(self, key: str) -> Optional[str]:
|
||||||
|
client = self._get_client()
|
||||||
|
if client is None:
|
||||||
|
# Fall back to LRU
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
v = self._lru_fallback.get(key)
|
||||||
|
if v is not None:
|
||||||
|
self.hits += 1
|
||||||
|
else:
|
||||||
|
self.misses += 1
|
||||||
|
return v
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
v = client.get(self._key(key))
|
||||||
|
if v is not None:
|
||||||
|
self.hits += 1
|
||||||
|
# Promote to LRU too (in-process L1 cache)
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
self._lru_fallback.set(key, v)
|
||||||
|
return v
|
||||||
|
self.misses += 1
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
self.errors += 1
|
||||||
|
self._available = False
|
||||||
|
self._client = None # force reconnect on next call
|
||||||
|
logger.warning("redis_cache_get_error", error=str(e)[:200])
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
return self._lru_fallback.get(key)
|
||||||
|
return None
|
||||||
|
|
||||||
|
def set(self, key: str, value: str, ttl: Optional[int] = None) -> None:
|
||||||
|
client = self._get_client()
|
||||||
|
if client is not None:
|
||||||
|
try:
|
||||||
|
client.set(
|
||||||
|
self._key(key),
|
||||||
|
value,
|
||||||
|
ex=ttl if ttl is not None else self._default_ttl,
|
||||||
|
)
|
||||||
|
# Also write to LRU
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
self._lru_fallback.set(key, value)
|
||||||
|
return
|
||||||
|
except Exception as e:
|
||||||
|
self.errors += 1
|
||||||
|
self._available = False
|
||||||
|
self._client = None
|
||||||
|
logger.warning("redis_cache_set_error", error=str(e)[:200])
|
||||||
|
# Fall through to LRU
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
self._lru_fallback.set(key, value, ttl=ttl)
|
||||||
|
|
||||||
|
def clear(self) -> None:
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
self._lru_fallback.clear()
|
||||||
|
client = self._get_client()
|
||||||
|
if client is not None:
|
||||||
|
try:
|
||||||
|
# Delete only keys in our namespace
|
||||||
|
cursor = 0
|
||||||
|
pattern = f"{self._namespace}:*"
|
||||||
|
while True:
|
||||||
|
cursor, keys = client.scan(cursor=cursor, match=pattern, count=500)
|
||||||
|
if keys:
|
||||||
|
client.delete(*keys)
|
||||||
|
if cursor == 0:
|
||||||
|
break
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("redis_cache_clear_error", error=str(e)[:200])
|
||||||
|
|
||||||
|
def stats(self) -> Dict:
|
||||||
|
out = {
|
||||||
|
"backend": "redis",
|
||||||
|
"available": self._available,
|
||||||
|
"url": self._url.split("@")[-1] if self._url else None,
|
||||||
|
"namespace": self._namespace,
|
||||||
|
"default_ttl": self._default_ttl,
|
||||||
|
"hits": self.hits,
|
||||||
|
"misses": self.misses,
|
||||||
|
"errors": self.errors,
|
||||||
|
}
|
||||||
|
if self._lru_fallback is not None:
|
||||||
|
lru_stats = self._lru_fallback.stats()
|
||||||
|
out["lru_size"] = lru_stats["size"]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# TranslationCache — public API
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class TranslationCache:
|
||||||
|
"""High-level translation cache.
|
||||||
|
|
||||||
|
Delegates to a backend (Redis or LRU) based on configuration.
|
||||||
|
Provides an async-friendly API (sync methods, but no blocking
|
||||||
|
Redis I/O — Redis client is async-compatible via redis.asyncio
|
||||||
|
if we want to add that later).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
backend: str = "auto",
|
||||||
|
redis_url: Optional[str] = None,
|
||||||
|
redis_namespace: str = "translate_cache",
|
||||||
|
default_ttl: int = 86400,
|
||||||
|
lru_maxsize: int = 5000,
|
||||||
|
):
|
||||||
|
# Decide backend
|
||||||
|
if backend == "auto":
|
||||||
|
backend = "redis" if (
|
||||||
|
os.getenv("REDIS_CACHE_ENABLED", "false").lower() == "true"
|
||||||
|
and (redis_url or os.getenv("REDIS_URL", ""))
|
||||||
|
) else "lru"
|
||||||
|
|
||||||
|
self._lru = _LRUBackend(maxsize=lru_maxsize)
|
||||||
|
if backend == "redis" and (redis_url or os.getenv("REDIS_URL", "")):
|
||||||
|
self._backend: object = _RedisBackend(
|
||||||
|
url=redis_url or os.getenv("REDIS_URL", ""),
|
||||||
|
namespace=redis_namespace,
|
||||||
|
default_ttl=default_ttl,
|
||||||
|
lru_fallback=self._lru,
|
||||||
|
)
|
||||||
|
self._backend_name = "redis"
|
||||||
|
else:
|
||||||
|
self._backend = self._lru
|
||||||
|
self._backend_name = "lru"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def backend_name(self) -> str:
|
||||||
|
return self._backend_name
|
||||||
|
|
||||||
|
def get(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
target_language: str,
|
||||||
|
source_language: str,
|
||||||
|
provider: str,
|
||||||
|
user_id: Optional[str] = None,
|
||||||
|
custom_prompt_hash: Optional[str] = None,
|
||||||
|
glossary_hash: Optional[str] = None,
|
||||||
|
) -> Optional[str]:
|
||||||
|
key = make_cache_key(
|
||||||
|
text, target_language, source_language, provider,
|
||||||
|
user_id, custom_prompt_hash, glossary_hash,
|
||||||
|
)
|
||||||
|
return self._backend.get(key)
|
||||||
|
|
||||||
|
def set(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
target_language: str,
|
||||||
|
source_language: str,
|
||||||
|
provider: str,
|
||||||
|
translation: str,
|
||||||
|
user_id: Optional[str] = None,
|
||||||
|
custom_prompt_hash: Optional[str] = None,
|
||||||
|
glossary_hash: Optional[str] = None,
|
||||||
|
ttl: Optional[int] = None,
|
||||||
|
) -> None:
|
||||||
|
key = make_cache_key(
|
||||||
|
text, target_language, source_language, provider,
|
||||||
|
user_id, custom_prompt_hash, glossary_hash,
|
||||||
|
)
|
||||||
|
self._backend.set(key, translation, ttl=ttl)
|
||||||
|
|
||||||
|
def clear(self) -> None:
|
||||||
|
self._backend.clear()
|
||||||
|
|
||||||
|
def stats(self) -> Dict:
|
||||||
|
return self._backend.stats()
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Singleton accessor
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
_cache: Optional[TranslationCache] = None
|
||||||
|
_cache_lock = threading.RLock()
|
||||||
|
|
||||||
|
|
||||||
|
def get_cache() -> TranslationCache:
|
||||||
|
"""Get the process-wide translation cache (lazy init)."""
|
||||||
|
global _cache
|
||||||
|
if _cache is not None:
|
||||||
|
return _cache
|
||||||
|
with _cache_lock:
|
||||||
|
if _cache is None:
|
||||||
|
_cache = TranslationCache(
|
||||||
|
backend="auto",
|
||||||
|
redis_url=os.getenv("REDIS_URL", "") or None,
|
||||||
|
redis_namespace=os.getenv("REDIS_CACHE_NAMESPACE", "translate_cache"),
|
||||||
|
default_ttl=int(os.getenv("REDIS_CACHE_TTL", "86400")),
|
||||||
|
lru_maxsize=int(os.getenv("LRU_CACHE_MAXSIZE", "5000")),
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
"translation_cache_initialized",
|
||||||
|
backend=_cache.backend_name,
|
||||||
|
)
|
||||||
|
return _cache
|
||||||
|
|
||||||
|
|
||||||
|
def reset_cache_for_tests() -> None:
|
||||||
|
"""Reset the singleton — only for tests."""
|
||||||
|
global _cache
|
||||||
|
with _cache_lock:
|
||||||
|
_cache = None
|
||||||
@@ -5,6 +5,18 @@
|
|||||||
"source": "vercel-labs/agent-browser",
|
"source": "vercel-labs/agent-browser",
|
||||||
"sourceType": "github",
|
"sourceType": "github",
|
||||||
"computedHash": "0eafe0971692cf8fcecb75f955595821452a835791d315db738f89ee49063a34"
|
"computedHash": "0eafe0971692cf8fcecb75f955595821452a835791d315db738f89ee49063a34"
|
||||||
|
},
|
||||||
|
"find-skills": {
|
||||||
|
"source": "vercel-labs/skills",
|
||||||
|
"sourceType": "github",
|
||||||
|
"skillPath": "skills/find-skills/SKILL.md",
|
||||||
|
"computedHash": "781bd6d3f9b19f8c9af6b53d8d0e4876d0183841b565db34ca7092ffa412d111"
|
||||||
|
},
|
||||||
|
"writing-guidelines": {
|
||||||
|
"source": "vercel-labs/agent-skills",
|
||||||
|
"sourceType": "github",
|
||||||
|
"skillPath": "skills/writing-guidelines/SKILL.md",
|
||||||
|
"computedHash": "25aa3a33a97bddbcb1847ce4a6169106ec25e5b19b15b99c66e75ab383866ef2"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
0
tests/services/quality/__init__.py
Normal file
0
tests/services/quality/__init__.py
Normal file
143
tests/services/quality/test_config.py
Normal file
143
tests/services/quality/test_config.py
Normal file
@@ -0,0 +1,143 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/config.py
|
||||||
|
Covers the language → script mapping and discriminating characters.
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality import config as qconfig
|
||||||
|
|
||||||
|
|
||||||
|
class TestGetScript:
|
||||||
|
def test_cyrillic_languages(self):
|
||||||
|
assert qconfig.get_script("ru") == "cyrillic"
|
||||||
|
assert qconfig.get_script("uk") == "cyrillic"
|
||||||
|
assert qconfig.get_script("be") == "cyrillic"
|
||||||
|
assert qconfig.get_script("bg") == "cyrillic"
|
||||||
|
assert qconfig.get_script("sr") == "cyrillic"
|
||||||
|
assert qconfig.get_script("kk") == "cyrillic"
|
||||||
|
|
||||||
|
def test_latin_languages(self):
|
||||||
|
assert qconfig.get_script("en") == "latin"
|
||||||
|
assert qconfig.get_script("fr") == "latin"
|
||||||
|
assert qconfig.get_script("de") == "latin"
|
||||||
|
assert qconfig.get_script("es") == "latin"
|
||||||
|
assert qconfig.get_script("vi") == "latin"
|
||||||
|
assert qconfig.get_script("tr") == "latin"
|
||||||
|
|
||||||
|
def test_cjk_languages(self):
|
||||||
|
assert qconfig.get_script("zh") == "cjk"
|
||||||
|
assert qconfig.get_script("zh-cn") == "cjk"
|
||||||
|
assert qconfig.get_script("zh-tw") == "cjk"
|
||||||
|
|
||||||
|
def test_japanese_uses_kana(self):
|
||||||
|
# ja is mapped to hiragana_katakana specifically so it can be
|
||||||
|
# distinguished from Chinese CJK.
|
||||||
|
assert qconfig.get_script("ja") == "hiragana_katakana"
|
||||||
|
|
||||||
|
def test_korean_uses_hangul(self):
|
||||||
|
assert qconfig.get_script("ko") == "hangul"
|
||||||
|
|
||||||
|
def test_arabic_script_languages(self):
|
||||||
|
assert qconfig.get_script("ar") == "arabic"
|
||||||
|
assert qconfig.get_script("fa") == "arabic"
|
||||||
|
assert qconfig.get_script("ur") == "arabic"
|
||||||
|
assert qconfig.get_script("ps") == "arabic"
|
||||||
|
assert qconfig.get_script("ku") == "arabic"
|
||||||
|
|
||||||
|
def test_hebrew_and_yiddish(self):
|
||||||
|
assert qconfig.get_script("he") == "hebrew"
|
||||||
|
# Yiddish uses Hebrew script, not Arabic
|
||||||
|
assert qconfig.get_script("yi") == "hebrew"
|
||||||
|
|
||||||
|
def test_thaana(self):
|
||||||
|
# Maldivian (Dhivehi) uses Thaana script, not Arabic
|
||||||
|
assert qconfig.get_script("dv") == "thaana"
|
||||||
|
|
||||||
|
def test_indian_scripts(self):
|
||||||
|
assert qconfig.get_script("hi") == "devanagari"
|
||||||
|
assert qconfig.get_script("bn") == "bengali"
|
||||||
|
assert qconfig.get_script("ta") == "tamil"
|
||||||
|
assert qconfig.get_script("te") == "telugu"
|
||||||
|
assert qconfig.get_script("gu") == "gujarati"
|
||||||
|
assert qconfig.get_script("pa") == "gurmukhi"
|
||||||
|
assert qconfig.get_script("ml") == "malayalam"
|
||||||
|
assert qconfig.get_script("kn") == "kannada"
|
||||||
|
assert qconfig.get_script("si") == "sinhala"
|
||||||
|
|
||||||
|
def test_se_asian_scripts(self):
|
||||||
|
assert qconfig.get_script("th") == "thai"
|
||||||
|
assert qconfig.get_script("lo") == "lao"
|
||||||
|
assert qconfig.get_script("my") == "burmese"
|
||||||
|
assert qconfig.get_script("km") == "khmer"
|
||||||
|
|
||||||
|
def test_other_scripts(self):
|
||||||
|
assert qconfig.get_script("el") == "greek"
|
||||||
|
assert qconfig.get_script("ka") == "georgian"
|
||||||
|
assert qconfig.get_script("hy") == "armenian"
|
||||||
|
assert qconfig.get_script("am") == "ethiopic"
|
||||||
|
assert qconfig.get_script("bo") == "tibetan"
|
||||||
|
|
||||||
|
def test_unknown_falls_back_to_latin(self):
|
||||||
|
assert qconfig.get_script("xx") == "latin"
|
||||||
|
assert qconfig.get_script("") == "latin"
|
||||||
|
assert qconfig.get_script(None) == "latin"
|
||||||
|
|
||||||
|
def test_case_insensitive(self):
|
||||||
|
assert qconfig.get_script("FR") == "latin"
|
||||||
|
assert qconfig.get_script("ZH-CN") == "cjk"
|
||||||
|
assert qconfig.get_script("Ru") == "cyrillic"
|
||||||
|
|
||||||
|
|
||||||
|
class TestIsArabicScriptLang:
|
||||||
|
def test_true_for_arabic_script_langs(self):
|
||||||
|
assert qconfig.is_arabic_script_lang("ar") is True
|
||||||
|
assert qconfig.is_arabic_script_lang("fa") is True
|
||||||
|
assert qconfig.is_arabic_script_lang("ur") is True
|
||||||
|
assert qconfig.is_arabic_script_lang("ckb") is True
|
||||||
|
|
||||||
|
def test_false_for_non_arabic(self):
|
||||||
|
assert qconfig.is_arabic_script_lang("fr") is False
|
||||||
|
assert qconfig.is_arabic_script_lang("he") is False
|
||||||
|
assert qconfig.is_arabic_script_lang("hi") is False
|
||||||
|
assert qconfig.is_arabic_script_lang("en") is False
|
||||||
|
|
||||||
|
def test_false_for_unknown(self):
|
||||||
|
assert qconfig.is_arabic_script_lang("xx") is False
|
||||||
|
assert qconfig.is_arabic_script_lang("") is False
|
||||||
|
assert qconfig.is_arabic_script_lang(None) is False
|
||||||
|
|
||||||
|
|
||||||
|
class TestDiscriminatingChars:
|
||||||
|
def test_persian_chars(self):
|
||||||
|
chars = qconfig.get_discriminating_chars("fa")
|
||||||
|
assert "پ" in chars # peh
|
||||||
|
assert "چ" in chars # tcheh
|
||||||
|
assert "ژ" in chars # zheh
|
||||||
|
assert "گ" in chars # guaf
|
||||||
|
|
||||||
|
def test_urdu_chars(self):
|
||||||
|
chars = qconfig.get_discriminating_chars("ur")
|
||||||
|
assert "ٹ" in chars # tteh
|
||||||
|
assert "ڈ" in chars # ddal
|
||||||
|
|
||||||
|
def test_unknown_lang_empty(self):
|
||||||
|
assert qconfig.get_discriminating_chars("fr") == frozenset()
|
||||||
|
assert qconfig.get_discriminating_chars("") == frozenset()
|
||||||
|
assert qconfig.get_discriminating_chars(None) == frozenset()
|
||||||
|
|
||||||
|
|
||||||
|
class TestGetRanges:
|
||||||
|
def test_latin_returns_empty(self):
|
||||||
|
# Latin is the fallback — no ranges, means "anything matches".
|
||||||
|
assert qconfig.get_ranges("latin") == []
|
||||||
|
|
||||||
|
def test_cyrillic_ranges(self):
|
||||||
|
ranges = qconfig.get_ranges("cyrillic")
|
||||||
|
assert any(start == 0x0400 for start, _ in ranges)
|
||||||
|
|
||||||
|
def test_cjk_ranges(self):
|
||||||
|
ranges = qconfig.get_ranges("cjk")
|
||||||
|
assert any(start == 0x4E00 for start, _ in ranges)
|
||||||
|
|
||||||
|
def test_unknown_script_returns_empty(self):
|
||||||
|
assert qconfig.get_ranges("mystery_script") == []
|
||||||
181
tests/services/quality/test_file_extractor.py
Normal file
181
tests/services/quality/test_file_extractor.py
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/file_extractor.py
|
||||||
|
Uses real files generated via temporary paths.
|
||||||
|
|
||||||
|
On Windows, tempfile.NamedTemporaryFile holds the file open and blocks
|
||||||
|
overwrite, so we use a plain temp directory + manual filename instead.
|
||||||
|
"""
|
||||||
|
import tempfile
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality.file_extractor import (
|
||||||
|
extract_sample,
|
||||||
|
DEFAULT_MAX_SAMPLES,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestExtractSample:
|
||||||
|
def test_returns_empty_for_missing_file(self):
|
||||||
|
result = extract_sample(Path("/nonexistent/path/file.docx"), ".docx")
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
def test_returns_empty_for_none_path(self):
|
||||||
|
result = extract_sample(None, ".docx")
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
def test_returns_empty_for_unsupported_extension(self):
|
||||||
|
# .txt isn't supported — should return [] silently
|
||||||
|
with tempfile.TemporaryDirectory() as d:
|
||||||
|
p = Path(d) / "test.txt"
|
||||||
|
p.write_text("some text")
|
||||||
|
result = extract_sample(p, ".txt")
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
|
||||||
|
def _make_tmp_path(suffix: str) -> Path:
|
||||||
|
"""Create a unique temp file path. File does not exist yet."""
|
||||||
|
fd, name = tempfile.mkstemp(suffix=suffix)
|
||||||
|
os.close(fd)
|
||||||
|
return Path(name)
|
||||||
|
|
||||||
|
|
||||||
|
class TestDocxExtraction:
|
||||||
|
def test_extracts_paragraphs(self):
|
||||||
|
from docx import Document
|
||||||
|
doc = Document()
|
||||||
|
doc.add_paragraph("Bonjour le monde")
|
||||||
|
doc.add_paragraph("Comment allez-vous?")
|
||||||
|
doc.add_paragraph("Merci beaucoup")
|
||||||
|
p = _make_tmp_path(".docx")
|
||||||
|
try:
|
||||||
|
doc.save(str(p))
|
||||||
|
result = extract_sample(p, ".docx", max_samples=10)
|
||||||
|
assert len(result) >= 1
|
||||||
|
texts = [s["translated"] for s in result]
|
||||||
|
assert "Bonjour le monde" in texts
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def test_respects_max_samples(self):
|
||||||
|
from docx import Document
|
||||||
|
doc = Document()
|
||||||
|
for i in range(50):
|
||||||
|
doc.add_paragraph(f"Paragraphe numero {i} avec du texte")
|
||||||
|
p = _make_tmp_path(".docx")
|
||||||
|
try:
|
||||||
|
doc.save(str(p))
|
||||||
|
result = extract_sample(p, ".docx", max_samples=5)
|
||||||
|
assert len(result) == 5
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def test_handles_empty_doc(self):
|
||||||
|
from docx import Document
|
||||||
|
doc = Document()
|
||||||
|
p = _make_tmp_path(".docx")
|
||||||
|
try:
|
||||||
|
doc.save(str(p))
|
||||||
|
result = extract_sample(p, ".docx")
|
||||||
|
assert result == []
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class TestXlsxExtraction:
|
||||||
|
def test_extracts_cells(self):
|
||||||
|
from openpyxl import Workbook
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = "Bonjour"
|
||||||
|
ws["A2"] = "Monde"
|
||||||
|
ws["A3"] = "Comment"
|
||||||
|
p = _make_tmp_path(".xlsx")
|
||||||
|
try:
|
||||||
|
wb.save(str(p))
|
||||||
|
wb.close()
|
||||||
|
result = extract_sample(p, ".xlsx", max_samples=10)
|
||||||
|
assert len(result) >= 1
|
||||||
|
texts = [s["translated"] for s in result]
|
||||||
|
assert "Bonjour" in texts
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def test_skips_numeric_cells(self):
|
||||||
|
from openpyxl import Workbook
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = 100
|
||||||
|
ws["A2"] = 200
|
||||||
|
ws["A3"] = "Hello"
|
||||||
|
p = _make_tmp_path(".xlsx")
|
||||||
|
try:
|
||||||
|
wb.save(str(p))
|
||||||
|
wb.close()
|
||||||
|
result = extract_sample(p, ".xlsx")
|
||||||
|
texts = [s["translated"] for s in result]
|
||||||
|
assert "Hello" in texts
|
||||||
|
assert 100 not in texts # numeric only cells skipped
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class TestPptxExtraction:
|
||||||
|
def test_extracts_slide_text(self):
|
||||||
|
from pptx import Presentation
|
||||||
|
pres = Presentation()
|
||||||
|
slide = pres.slides.add_slide(pres.slide_layouts[0])
|
||||||
|
slide.shapes.title.text = "Bonjour le monde"
|
||||||
|
p = _make_tmp_path(".pptx")
|
||||||
|
try:
|
||||||
|
pres.save(str(p))
|
||||||
|
result = extract_sample(p, ".pptx")
|
||||||
|
assert len(result) >= 1
|
||||||
|
assert result[0]["translated"] == "Bonjour le monde"
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class TestPdfExtraction:
|
||||||
|
def test_extracts_pdf_text(self):
|
||||||
|
# Create a minimal PDF using reportlab if available, else skip
|
||||||
|
pytest.importorskip("fitz")
|
||||||
|
try:
|
||||||
|
from reportlab.pdfgen import canvas
|
||||||
|
except ImportError:
|
||||||
|
pytest.skip("reportlab not available")
|
||||||
|
p = _make_tmp_path(".pdf")
|
||||||
|
try:
|
||||||
|
c = canvas.Canvas(str(p))
|
||||||
|
c.drawString(100, 750, "Bonjour le monde")
|
||||||
|
c.drawString(100, 700, "Comment allez vous")
|
||||||
|
c.save()
|
||||||
|
result = extract_sample(p, ".pdf")
|
||||||
|
assert len(result) >= 1
|
||||||
|
texts = [s["translated"] for s in result]
|
||||||
|
assert any("Bonjour" in t for t in texts)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
p.unlink()
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
127
tests/services/quality/test_l1_pipeline.py
Normal file
127
tests/services/quality/test_l1_pipeline.py
Normal file
@@ -0,0 +1,127 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/pipeline.py — the L1 wrapper.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
from services.quality.pipeline import run_l1_check
|
||||||
|
from services.quality.llm_judge import L1Result, LLMJudge
|
||||||
|
|
||||||
|
|
||||||
|
class TestRunL1Check:
|
||||||
|
"""Test the defensive wrapper around the LLM judge."""
|
||||||
|
|
||||||
|
def test_too_few_chunks_returns_skip(self):
|
||||||
|
sources = ["a", "b", "c"] # only 3 chunks
|
||||||
|
translations = ["x", "y", "z"]
|
||||||
|
result = asyncio.run(run_l1_check(
|
||||||
|
sources, translations, "fr",
|
||||||
|
max_samples=5, min_chunks=10,
|
||||||
|
))
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
|
||||||
|
def test_no_judge_configured_returns_skip(self, monkeypatch):
|
||||||
|
monkeypatch.delenv("L1_JUDGE_API_KEY", raising=False)
|
||||||
|
sources = [f"source {i}" for i in range(15)]
|
||||||
|
translations = [f"translation {i}" for i in range(15)]
|
||||||
|
result = asyncio.run(run_l1_check(
|
||||||
|
sources, translations, "fr",
|
||||||
|
max_samples=5, min_chunks=10,
|
||||||
|
))
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert "judge" in result.error.lower() or "configured" in result.error.lower() or result.error == "not_run"
|
||||||
|
|
||||||
|
def test_skips_l0_failures(self):
|
||||||
|
"""Chunks that L0 already flagged should be excluded from the sample."""
|
||||||
|
# All chunks are long enough to be sampled. Use unique markers per
|
||||||
|
# index so substring checks don't have false positives (e.g.
|
||||||
|
# "source1" being a substring of "source10").
|
||||||
|
sources = [f"srcidx{i:02d} with enough length" for i in range(15)]
|
||||||
|
translations = [f"transidx{i:02d} avec longueur suffisante" for i in range(15)]
|
||||||
|
|
||||||
|
# Mock judge that records what it received
|
||||||
|
mock_judge = MagicMock(spec=LLMJudge)
|
||||||
|
mock_judge.judge_batch = AsyncMock(return_value=L1Result(
|
||||||
|
verdict="pass", chunks_evaluated=5, chunks_passed=5,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
))
|
||||||
|
|
||||||
|
# Mark indices 01, 03, 05, 07, 09, 11, 13 as L0 failures
|
||||||
|
l0_failures = {1, 3, 5, 7, 9, 11, 13}
|
||||||
|
|
||||||
|
result = asyncio.run(run_l1_check(
|
||||||
|
sources, translations, "fr",
|
||||||
|
l0_failed_indices=l0_failures,
|
||||||
|
max_samples=5, min_chunks=10,
|
||||||
|
judge=mock_judge,
|
||||||
|
))
|
||||||
|
|
||||||
|
# The mock was called with 5 pairs
|
||||||
|
call_args = mock_judge.judge_batch.call_args
|
||||||
|
pairs = call_args[0][0]
|
||||||
|
# The pairs should NOT include any source from the L0-failed indices
|
||||||
|
for src, _trans in pairs:
|
||||||
|
for failed_idx in l0_failures:
|
||||||
|
marker = f"srcidx{failed_idx:02d}"
|
||||||
|
assert marker not in src, (
|
||||||
|
f"L0-failed source {failed_idx} was sent to L1: {src}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_passes_log_only_flag(self):
|
||||||
|
"""The log_only flag should be passed through to the LLMJudge result."""
|
||||||
|
mock_judge = MagicMock(spec=LLMJudge)
|
||||||
|
mock_judge.judge_batch = AsyncMock(return_value=L1Result(
|
||||||
|
verdict="pass", chunks_evaluated=1, chunks_passed=1,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
))
|
||||||
|
|
||||||
|
sources = [f"long enough source {i} " * 3 for i in range(15)]
|
||||||
|
translations = [f"long enough translation {i} " * 3 for i in range(15)]
|
||||||
|
|
||||||
|
asyncio.run(run_l1_check(
|
||||||
|
sources, translations, "fr",
|
||||||
|
max_samples=5, min_chunks=10,
|
||||||
|
judge=mock_judge,
|
||||||
|
log_only=True,
|
||||||
|
))
|
||||||
|
# log_only doesn't affect the call, just the downstream decision
|
||||||
|
|
||||||
|
def test_never_raises_on_judge_error(self):
|
||||||
|
"""If the judge itself raises, run_l1_check should catch it."""
|
||||||
|
mock_judge = MagicMock(spec=LLMJudge)
|
||||||
|
mock_judge.judge_batch = AsyncMock(side_effect=Exception("boom"))
|
||||||
|
|
||||||
|
sources = [f"long enough source {i} " * 3 for i in range(15)]
|
||||||
|
translations = [f"long enough translation {i} " * 3 for i in range(15)]
|
||||||
|
|
||||||
|
# Should NOT raise
|
||||||
|
result = asyncio.run(run_l1_check(
|
||||||
|
sources, translations, "fr",
|
||||||
|
max_samples=5, min_chunks=10,
|
||||||
|
judge=mock_judge,
|
||||||
|
))
|
||||||
|
assert result is not None
|
||||||
|
# verdict is either "skip" or an error one — never crashes the call
|
||||||
|
|
||||||
|
def test_target_lang_name_for_known_lang(self):
|
||||||
|
"""Verify the LANG_NAMES mapping is used."""
|
||||||
|
mock_judge = MagicMock(spec=LLMJudge)
|
||||||
|
mock_judge.judge_batch = AsyncMock(return_value=L1Result(
|
||||||
|
verdict="pass", chunks_evaluated=1, chunks_passed=1,
|
||||||
|
chunks_failed=0, failure_rate=0.0,
|
||||||
|
))
|
||||||
|
|
||||||
|
sources = [f"long enough source {i} " * 3 for i in range(15)]
|
||||||
|
translations = [f"long enough translation {i} " * 3 for i in range(15)]
|
||||||
|
|
||||||
|
asyncio.run(run_l1_check(
|
||||||
|
sources, translations, "fr",
|
||||||
|
max_samples=5, min_chunks=10,
|
||||||
|
judge=mock_judge,
|
||||||
|
))
|
||||||
|
call_args = mock_judge.judge_batch.call_args
|
||||||
|
# target_lang_name is the 3rd positional arg
|
||||||
|
target_lang_name = call_args[0][2]
|
||||||
|
assert target_lang_name == "French"
|
||||||
494
tests/services/quality/test_l2_judge.py
Normal file
494
tests/services/quality/test_l2_judge.py
Normal file
@@ -0,0 +1,494 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track A4 — L2 Pro premium judge.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- L2DimensionVerdict: 8 dimensions + scoring
|
||||||
|
- L2Result: aggregation + dimension pass rates
|
||||||
|
- L2ProJudge: construction, missing api_key, cost estimation
|
||||||
|
- make_l2_judge_from_env: env-var-driven factory
|
||||||
|
- run_l2_check: defensive wrapper
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import MagicMock, AsyncMock, patch
|
||||||
|
|
||||||
|
from services.quality.l2_judge import (
|
||||||
|
L2DimensionVerdict,
|
||||||
|
L2Result,
|
||||||
|
L2ProJudge,
|
||||||
|
make_l2_judge_from_env,
|
||||||
|
L2_JUDGE_SYSTEM_PROMPT,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# L2DimensionVerdict
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2DimensionVerdict:
|
||||||
|
def test_all_pass(self):
|
||||||
|
v = L2DimensionVerdict(
|
||||||
|
accurate=True, fluent=True, correct_lang=True, no_leaks=True,
|
||||||
|
terminology=True, style=True, completeness=True, formatting=True,
|
||||||
|
)
|
||||||
|
assert v.passed_count == 8
|
||||||
|
assert v.total == 8
|
||||||
|
assert v.score == 1.0
|
||||||
|
assert v.passed is True
|
||||||
|
|
||||||
|
def test_one_fails(self):
|
||||||
|
v = L2DimensionVerdict(
|
||||||
|
accurate=True, fluent=True, correct_lang=True, no_leaks=True,
|
||||||
|
terminology=False, # <-- fails
|
||||||
|
style=True, completeness=True, formatting=True,
|
||||||
|
)
|
||||||
|
assert v.passed_count == 7
|
||||||
|
assert v.total == 8
|
||||||
|
assert v.score == pytest.approx(0.875)
|
||||||
|
# L2 is strict: one fail = chunk fails
|
||||||
|
assert v.passed is False
|
||||||
|
|
||||||
|
def test_all_fail(self):
|
||||||
|
v = L2DimensionVerdict() # all default False
|
||||||
|
assert v.passed_count == 0
|
||||||
|
assert v.score == 0.0
|
||||||
|
assert v.passed is False
|
||||||
|
|
||||||
|
def test_default_construction(self):
|
||||||
|
v = L2DimensionVerdict()
|
||||||
|
assert v.accurate is False
|
||||||
|
assert v.reason == ""
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# L2Result
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2Result:
|
||||||
|
def test_default_construction(self):
|
||||||
|
r = L2Result(verdict="pass")
|
||||||
|
assert r.verdict == "pass"
|
||||||
|
assert r.chunks_evaluated == 0
|
||||||
|
assert r.dimension_pass_rates == {}
|
||||||
|
assert r.error == ""
|
||||||
|
|
||||||
|
def test_to_log_dict(self):
|
||||||
|
r = L2Result(
|
||||||
|
verdict="pass",
|
||||||
|
chunks_evaluated=10,
|
||||||
|
chunks_passed=8,
|
||||||
|
chunks_failed=2,
|
||||||
|
failure_rate=0.2,
|
||||||
|
average_score=0.85,
|
||||||
|
model_used="gpt-4o",
|
||||||
|
cost_estimate_usd=0.012,
|
||||||
|
)
|
||||||
|
d = r.to_log_dict()
|
||||||
|
assert d["verdict"] == "pass"
|
||||||
|
assert d["chunks_evaluated"] == 10
|
||||||
|
assert d["model_used"] == "gpt-4o"
|
||||||
|
assert d["cost_estimate_usd"] == 0.012
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# L2ProJudge construction
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2ProJudgeConstruction:
|
||||||
|
def test_requires_api_key(self):
|
||||||
|
with pytest.raises(ValueError, match="api_key is required"):
|
||||||
|
L2ProJudge(api_key="")
|
||||||
|
|
||||||
|
def test_basic_construction(self):
|
||||||
|
judge = L2ProJudge(api_key="sk-test")
|
||||||
|
assert judge._api_key == "sk-test"
|
||||||
|
assert judge._model == "gpt-4o"
|
||||||
|
assert judge._base_url == "https://api.openai.com/v1"
|
||||||
|
|
||||||
|
def test_custom_model(self):
|
||||||
|
judge = L2ProJudge(
|
||||||
|
api_key="sk-test",
|
||||||
|
model="gpt-4o-mini",
|
||||||
|
base_url="https://api.openai.com/v1",
|
||||||
|
)
|
||||||
|
assert judge._model == "gpt-4o-mini"
|
||||||
|
|
||||||
|
def test_strips_trailing_slash_from_base_url(self):
|
||||||
|
judge = L2ProJudge(
|
||||||
|
api_key="sk-test",
|
||||||
|
base_url="https://api.example.com/v1/",
|
||||||
|
)
|
||||||
|
assert judge._base_url == "https://api.example.com/v1"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Cost estimation
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2CostEstimation:
|
||||||
|
def test_gpt4o_cost(self):
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
cost = judge._estimate_cost(15) # 15 samples default
|
||||||
|
# Should be in the $0.01–$0.05 range for 15 chunks
|
||||||
|
assert 0.001 < cost < 0.10
|
||||||
|
|
||||||
|
def test_gpt4o_mini_cheaper(self):
|
||||||
|
judge_mini = L2ProJudge(api_key="sk", model="gpt-4o-mini")
|
||||||
|
judge_full = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
cost_mini = judge_mini._estimate_cost(15)
|
||||||
|
cost_full = judge_full._estimate_cost(15)
|
||||||
|
# gpt-4o-mini should be cheaper than gpt-4o
|
||||||
|
assert cost_mini < cost_full
|
||||||
|
|
||||||
|
def test_zero_pairs(self):
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
cost = judge._estimate_cost(0)
|
||||||
|
# Even with 0 pairs, the system prompt has some cost
|
||||||
|
assert cost >= 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# judge_batch — defensive
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2JudgeBatch:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_empty_pairs_skips(self):
|
||||||
|
judge = L2ProJudge(api_key="sk")
|
||||||
|
result = await judge.judge_batch([], "fr", "French")
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert result.error == "empty pairs"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_client_unavailable_skips(self):
|
||||||
|
judge = L2ProJudge(api_key="sk")
|
||||||
|
# Simulate a client init failure
|
||||||
|
with patch.object(judge, "_get_client", return_value=None):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert "unavailable" in result.error or "client" in result.error.lower()
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_successful_judgement(self):
|
||||||
|
"""A mock client that returns a well-formed JSON response should
|
||||||
|
produce a passing L2Result."""
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
|
||||||
|
# Mock the client
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.choices = [MagicMock()]
|
||||||
|
mock_response.choices[0].message.content = json.dumps([
|
||||||
|
{
|
||||||
|
"accurate": "yes", "fluent": "yes", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "Perfect translation",
|
||||||
|
}
|
||||||
|
])
|
||||||
|
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=mock_response)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.verdict == "pass"
|
||||||
|
assert result.chunks_evaluated == 1
|
||||||
|
assert result.chunks_passed == 1
|
||||||
|
assert result.chunks_failed == 0
|
||||||
|
assert result.failure_rate == 0.0
|
||||||
|
assert result.average_score == 1.0
|
||||||
|
# All 8 dimensions should have pass rate 1.0
|
||||||
|
for dim in ["accurate", "fluent", "terminology", "style"]:
|
||||||
|
assert result.dimension_pass_rates[dim] == 1.0
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_partial_failure(self):
|
||||||
|
"""If one of 8 dimensions fails, the chunk should fail."""
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.choices = [MagicMock()]
|
||||||
|
# fluent = no, others yes
|
||||||
|
mock_response.choices[0].message.content = json.dumps([
|
||||||
|
{
|
||||||
|
"accurate": "yes", "fluent": "no", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "Awkward phrasing",
|
||||||
|
}
|
||||||
|
])
|
||||||
|
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=mock_response)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
|
||||||
|
# L2 is strict: one fail = overall fail
|
||||||
|
assert result.verdict == "fail"
|
||||||
|
assert result.chunks_evaluated == 1
|
||||||
|
assert result.chunks_passed == 0
|
||||||
|
assert result.chunks_failed == 1
|
||||||
|
assert result.dimension_pass_rates["fluent"] == 0.0
|
||||||
|
assert result.dimension_pass_rates["accurate"] == 1.0
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_handles_markdown_fences(self):
|
||||||
|
"""The judge should strip markdown code fences from responses."""
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.choices = [MagicMock()]
|
||||||
|
mock_response.choices[0].message.content = (
|
||||||
|
"```json\n"
|
||||||
|
+ json.dumps([{
|
||||||
|
"accurate": "yes", "fluent": "yes", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "ok",
|
||||||
|
}])
|
||||||
|
+ "\n```"
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=mock_response)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.verdict == "pass"
|
||||||
|
assert result.chunks_evaluated == 1
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_handles_dict_with_list(self):
|
||||||
|
"""Some LLMs return {"verdicts": [...]} instead of a raw list."""
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.choices = [MagicMock()]
|
||||||
|
mock_response.choices[0].message.content = json.dumps({
|
||||||
|
"verdicts": [
|
||||||
|
{
|
||||||
|
"accurate": "yes", "fluent": "yes", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "good",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
})
|
||||||
|
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=mock_response)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.verdict == "pass"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_timeout_returns_skip(self):
|
||||||
|
import asyncio
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o", timeout_seconds=0.1)
|
||||||
|
|
||||||
|
# Mock client that raises TimeoutError
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(
|
||||||
|
side_effect=asyncio.TimeoutError()
|
||||||
|
)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert "timeout" in result.error.lower()
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_never_raises(self):
|
||||||
|
"""Even on unexpected error, the judge should return a skip, not raise."""
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
|
||||||
|
# Mock client that raises a generic exception
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(
|
||||||
|
side_effect=RuntimeError("something unexpected")
|
||||||
|
)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
# Should NOT raise
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour")], "fr", "French"
|
||||||
|
)
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_dimension_pass_rates_aggregated(self):
|
||||||
|
"""Per-dimension pass rates should aggregate across chunks."""
|
||||||
|
judge = L2ProJudge(api_key="sk", model="gpt-4o")
|
||||||
|
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.choices = [MagicMock()]
|
||||||
|
# 2 chunks: chunk 1 all-pass, chunk 2 fluent-fail
|
||||||
|
mock_response.choices[0].message.content = json.dumps([
|
||||||
|
{
|
||||||
|
"accurate": "yes", "fluent": "yes", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "ok",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"accurate": "yes", "fluent": "no", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "awkward",
|
||||||
|
},
|
||||||
|
])
|
||||||
|
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=mock_response)
|
||||||
|
|
||||||
|
with patch.object(judge, "_get_client", return_value=mock_client):
|
||||||
|
result = await judge.judge_batch(
|
||||||
|
[("Hello", "Bonjour"), ("Goodbye", "Au revoir")],
|
||||||
|
"fr", "French"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.chunks_evaluated == 2
|
||||||
|
# fluent: 1/2 = 0.5
|
||||||
|
assert result.dimension_pass_rates["fluent"] == 0.5
|
||||||
|
# accurate: 2/2 = 1.0
|
||||||
|
assert result.dimension_pass_rates["accurate"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Factory
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2JudgeFactory:
|
||||||
|
def test_no_api_key_returns_none(self, monkeypatch):
|
||||||
|
monkeypatch.delenv("L2_JUDGE_API_KEY", raising=False)
|
||||||
|
assert make_l2_judge_from_env() is None
|
||||||
|
|
||||||
|
def test_api_key_creates_judge(self, monkeypatch):
|
||||||
|
monkeypatch.setenv("L2_JUDGE_API_KEY", "sk-test")
|
||||||
|
monkeypatch.setenv("L2_JUDGE_MODEL", "gpt-4o")
|
||||||
|
monkeypatch.setenv("L2_JUDGE_BASE_URL", "https://api.openai.com/v1")
|
||||||
|
judge = make_l2_judge_from_env()
|
||||||
|
assert judge is not None
|
||||||
|
assert judge._api_key == "sk-test"
|
||||||
|
assert judge._model == "gpt-4o"
|
||||||
|
|
||||||
|
def test_api_key_with_default_model(self, monkeypatch):
|
||||||
|
monkeypatch.setenv("L2_JUDGE_API_KEY", "sk-test")
|
||||||
|
# Clear other vars to test defaults
|
||||||
|
monkeypatch.delenv("L2_JUDGE_MODEL", raising=False)
|
||||||
|
monkeypatch.delenv("L2_JUDGE_BASE_URL", raising=False)
|
||||||
|
judge = make_l2_judge_from_env()
|
||||||
|
assert judge._model == "gpt-4o"
|
||||||
|
assert judge._base_url == "https://api.openai.com/v1"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Pipeline integration
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL2PipelineIntegration:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_l2_check_no_judge_skips(self):
|
||||||
|
from services.quality.pipeline import run_l2_check
|
||||||
|
|
||||||
|
result = await run_l2_check(
|
||||||
|
source_chunks=["Hello"] * 25,
|
||||||
|
translated_chunks=["Bonjour"] * 25,
|
||||||
|
target_lang="fr",
|
||||||
|
file_extension="docx",
|
||||||
|
judge=None, # Will try to load from env, but env has no key
|
||||||
|
)
|
||||||
|
# Should return skip, not raise
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_l2_check_with_mock_judge(self):
|
||||||
|
from services.quality.pipeline import run_l2_check
|
||||||
|
|
||||||
|
# Build a judge that always returns pass
|
||||||
|
judge = L2ProJudge(api_key="sk")
|
||||||
|
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.choices = [MagicMock()]
|
||||||
|
mock_response.choices[0].message.content = json.dumps([
|
||||||
|
{
|
||||||
|
"accurate": "yes", "fluent": "yes", "correct_lang": "yes",
|
||||||
|
"no_leaks": "yes", "terminology": "yes", "style": "yes",
|
||||||
|
"completeness": "yes", "formatting": "yes",
|
||||||
|
"reason": "ok",
|
||||||
|
}
|
||||||
|
] * 5)
|
||||||
|
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=mock_response)
|
||||||
|
judge._client = mock_client
|
||||||
|
|
||||||
|
# 25 chunks > default min_chunks of 20
|
||||||
|
result = await run_l2_check(
|
||||||
|
source_chunks=["Hello world"] * 25,
|
||||||
|
translated_chunks=["Bonjour le monde"] * 25,
|
||||||
|
target_lang="fr",
|
||||||
|
file_extension="docx",
|
||||||
|
judge=judge,
|
||||||
|
max_samples=5,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.verdict == "pass"
|
||||||
|
assert result.chunks_evaluated >= 1
|
||||||
|
assert result.chunks_passed >= 1
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_l2_check_too_few_chunks_skips(self):
|
||||||
|
from services.quality.pipeline import run_l2_check
|
||||||
|
|
||||||
|
# 5 chunks < default min_chunks of 20
|
||||||
|
result = await run_l2_check(
|
||||||
|
source_chunks=["a"] * 5,
|
||||||
|
translated_chunks=["b"] * 5,
|
||||||
|
target_lang="fr",
|
||||||
|
min_chunks=20,
|
||||||
|
)
|
||||||
|
# Should skip due to insufficient chunks
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 8-dimension coverage
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL28DimensionCoverage:
|
||||||
|
"""Sanity check: the L2 prompt template actually mentions all 8 dimensions."""
|
||||||
|
|
||||||
|
def test_prompt_has_all_8_dimensions(self):
|
||||||
|
for dim in [
|
||||||
|
"ACCURATE", "FLUENT", "CORRECT_LANG", "NO_LEAKS",
|
||||||
|
"TERMINOLOGY", "STYLE", "COMPLETENESS", "FORMATTING",
|
||||||
|
]:
|
||||||
|
assert dim in L2_JUDGE_SYSTEM_PROMPT, (
|
||||||
|
f"Dimension {dim!r} missing from L2 prompt template"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_prompt_has_format_hint(self):
|
||||||
|
# Should tell the model to respond with a JSON array
|
||||||
|
assert "JSON" in L2_JUDGE_SYSTEM_PROMPT
|
||||||
|
assert "yes" in L2_JUDGE_SYSTEM_PROMPT
|
||||||
|
assert "no" in L2_JUDGE_SYSTEM_PROMPT
|
||||||
49
tests/services/quality/test_length_checker.py
Normal file
49
tests/services/quality/test_length_checker.py
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/length_checker.py
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality.length_checker import check
|
||||||
|
|
||||||
|
|
||||||
|
class TestLengthCheck:
|
||||||
|
def test_normal_length(self):
|
||||||
|
r = check("Hello world, this is a test of translation length",
|
||||||
|
"Bonjour le monde, ceci est un test de longueur de traduction")
|
||||||
|
assert r["issue"] is None
|
||||||
|
assert r["ratio"] is not None
|
||||||
|
assert 0.5 < r["ratio"] < 2.0
|
||||||
|
|
||||||
|
def test_huge_translation(self):
|
||||||
|
src = "A" * 200
|
||||||
|
r = check(src, "x" * 1000)
|
||||||
|
assert r["issue"] == "length_outlier"
|
||||||
|
assert r["ratio"] > 3.5
|
||||||
|
|
||||||
|
def test_tiny_translation(self):
|
||||||
|
r = check("A" * 100, "ok")
|
||||||
|
assert r["issue"] == "truncation_suspect"
|
||||||
|
assert r["ratio"] < 0.15
|
||||||
|
|
||||||
|
def test_empty_translation_flagged(self):
|
||||||
|
r = check("Hello world, this is a test", "")
|
||||||
|
assert r["issue"] == "truncation_suspect"
|
||||||
|
|
||||||
|
def test_short_source_skips_ratio(self):
|
||||||
|
r = check("Hi", "ok")
|
||||||
|
assert r["issue"] is None
|
||||||
|
assert r["ratio"] is None
|
||||||
|
|
||||||
|
def test_empty_inputs(self):
|
||||||
|
r = check("", "")
|
||||||
|
assert r["issue"] is None
|
||||||
|
assert r["source_length"] == 0
|
||||||
|
assert r["translated_length"] == 0
|
||||||
|
|
||||||
|
def test_none_source(self):
|
||||||
|
r = check(None, "translation")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_numeric_source(self):
|
||||||
|
r = check("Price: 100", "100")
|
||||||
|
assert r["issue"] is None # numeric sources skip the check
|
||||||
258
tests/services/quality/test_llm_judge.py
Normal file
258
tests/services/quality/test_llm_judge.py
Normal file
@@ -0,0 +1,258 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/llm_judge.py
|
||||||
|
Uses mocks for the OpenAI client — no actual API calls.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
from services.quality.llm_judge import (
|
||||||
|
LLMJudge,
|
||||||
|
L1Result,
|
||||||
|
L1ChunkVerdict,
|
||||||
|
JUDGE_SYSTEM_PROMPT,
|
||||||
|
make_judge_from_env,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Judge init ----------
|
||||||
|
|
||||||
|
class TestLLMJudgeInit:
|
||||||
|
def test_requires_api_key(self):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
LLMJudge(api_key="")
|
||||||
|
|
||||||
|
def test_init_with_defaults(self):
|
||||||
|
judge = LLMJudge(api_key="test-key")
|
||||||
|
assert judge._api_key == "test-key"
|
||||||
|
assert judge._model == "deepseek-chat"
|
||||||
|
assert judge._base_url == "https://api.deepseek.com/v1"
|
||||||
|
|
||||||
|
def test_init_with_custom_params(self):
|
||||||
|
judge = LLMJudge(
|
||||||
|
api_key="k",
|
||||||
|
base_url="https://api.openai.com/v1/",
|
||||||
|
model="gpt-4o-mini",
|
||||||
|
timeout_seconds=15.0,
|
||||||
|
)
|
||||||
|
assert judge._base_url == "https://api.openai.com/v1" # trailing slash stripped
|
||||||
|
assert judge._model == "gpt-4o-mini"
|
||||||
|
assert judge._timeout == 15.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Response parsing ----------
|
||||||
|
|
||||||
|
class TestParseResponse:
|
||||||
|
def setup_method(self):
|
||||||
|
self.judge = LLMJudge(api_key="test-key")
|
||||||
|
|
||||||
|
def _make_response(self, content: str):
|
||||||
|
response = MagicMock()
|
||||||
|
response.choices = [MagicMock()]
|
||||||
|
response.choices[0].message.content = content
|
||||||
|
return response
|
||||||
|
|
||||||
|
def test_parse_pure_json_array(self):
|
||||||
|
content = json.dumps([
|
||||||
|
{"accurate": "yes", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "good"}
|
||||||
|
])
|
||||||
|
verdicts = self.judge._parse_response(self._make_response(content), 1)
|
||||||
|
assert len(verdicts) == 1
|
||||||
|
assert verdicts[0].passed is True
|
||||||
|
assert verdicts[0].accurate is True
|
||||||
|
assert verdicts[0].reason == "good"
|
||||||
|
|
||||||
|
def test_parse_json_object_with_verdicts_key(self):
|
||||||
|
content = json.dumps({"verdicts": [
|
||||||
|
{"accurate": "no", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "lost meaning"}
|
||||||
|
]})
|
||||||
|
verdicts = self.judge._parse_response(self._make_response(content), 1)
|
||||||
|
assert len(verdicts) == 1
|
||||||
|
assert verdicts[0].passed is False
|
||||||
|
assert verdicts[0].accurate is False
|
||||||
|
assert verdicts[0].fluent is True
|
||||||
|
assert verdicts[0].reason == "lost meaning"
|
||||||
|
|
||||||
|
def test_parse_json_with_markdown_fences(self):
|
||||||
|
content = "```json\n" + json.dumps([
|
||||||
|
{"accurate": "yes", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "ok"}
|
||||||
|
]) + "\n```"
|
||||||
|
verdicts = self.judge._parse_response(self._make_response(content), 1)
|
||||||
|
assert len(verdicts) == 1
|
||||||
|
assert verdicts[0].passed is True
|
||||||
|
|
||||||
|
def test_parse_invalid_json_returns_empty(self):
|
||||||
|
content = "not json at all"
|
||||||
|
verdicts = self.judge._parse_response(self._make_response(content), 1)
|
||||||
|
assert verdicts == []
|
||||||
|
|
||||||
|
def test_parse_partial_verdict_defaults_to_false(self):
|
||||||
|
content = json.dumps([{"accurate": "yes"}]) # missing other fields
|
||||||
|
verdicts = self.judge._parse_response(self._make_response(content), 1)
|
||||||
|
assert len(verdicts) == 1
|
||||||
|
# All fields default to False when missing
|
||||||
|
assert verdicts[0].passed is False
|
||||||
|
assert verdicts[0].accurate is True
|
||||||
|
assert verdicts[0].fluent is False
|
||||||
|
assert verdicts[0].correct_language is False
|
||||||
|
assert verdicts[0].no_leaks is False
|
||||||
|
|
||||||
|
def test_parse_mixed_pass_fail(self):
|
||||||
|
content = json.dumps([
|
||||||
|
{"accurate": "yes", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "good"},
|
||||||
|
{"accurate": "no", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "mistranslation"},
|
||||||
|
{"accurate": "yes", "fluent": "no", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "awkward"},
|
||||||
|
])
|
||||||
|
verdicts = self.judge._parse_response(self._make_response(content), 3)
|
||||||
|
assert len(verdicts) == 3
|
||||||
|
assert verdicts[0].passed is True
|
||||||
|
assert verdicts[1].passed is False
|
||||||
|
assert verdicts[2].passed is False
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- L1Result ----------
|
||||||
|
|
||||||
|
class TestL1Result:
|
||||||
|
def test_passed_property(self):
|
||||||
|
v = L1ChunkVerdict(accurate=True, fluent=True, correct_language=True, no_leaks=True)
|
||||||
|
assert v.passed is True
|
||||||
|
|
||||||
|
v_fail = L1ChunkVerdict(accurate=True, fluent=True, correct_language=True, no_leaks=False)
|
||||||
|
assert v_fail.passed is False
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Cost estimation ----------
|
||||||
|
|
||||||
|
class TestCostEstimation:
|
||||||
|
def test_deepseek_estimate(self):
|
||||||
|
judge = LLMJudge(api_key="k", model="deepseek-chat")
|
||||||
|
cost = judge._estimate_cost(5)
|
||||||
|
# 5 pairs: ~1450 input + 250 output tokens
|
||||||
|
# Cost should be tiny (< $0.01)
|
||||||
|
assert 0.0 < cost < 0.01
|
||||||
|
|
||||||
|
def test_gpt4o_mini_more_expensive(self):
|
||||||
|
judge_ds = LLMJudge(api_key="k", model="deepseek-chat")
|
||||||
|
judge_gpt = LLMJudge(api_key="k", model="gpt-4o-mini")
|
||||||
|
assert judge_gpt._estimate_cost(5) > judge_ds._estimate_cost(5)
|
||||||
|
|
||||||
|
def test_gemini_flash_cheaper(self):
|
||||||
|
judge_gemini = LLMJudge(api_key="k", model="gemini-2.5-flash-lite")
|
||||||
|
judge_gpt = LLMJudge(api_key="k", model="gpt-4o-mini")
|
||||||
|
assert judge_gemini._estimate_cost(5) < judge_gpt._estimate_cost(5)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Judge batch (mocked API) ----------
|
||||||
|
|
||||||
|
class TestJudgeBatch:
|
||||||
|
"""Tests the full judge_batch flow with a mocked OpenAI client."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
self.judge = LLMJudge(api_key="test-key", timeout_seconds=5.0)
|
||||||
|
|
||||||
|
def _mock_response_with_verdicts(self, verdicts_data):
|
||||||
|
response = MagicMock()
|
||||||
|
response.choices = [MagicMock()]
|
||||||
|
response.choices[0].message.content = json.dumps(verdicts_data)
|
||||||
|
return response
|
||||||
|
|
||||||
|
def _run_judge_batch(self, verdicts_data):
|
||||||
|
"""Helper to run judge_batch with a mocked client."""
|
||||||
|
# Build the mock client BEFORE calling the method
|
||||||
|
response = self._mock_response_with_verdicts(verdicts_data)
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat = MagicMock()
|
||||||
|
mock_client.chat.completions = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(return_value=response)
|
||||||
|
self.judge._client = mock_client
|
||||||
|
return asyncio.run(self.judge.judge_batch(
|
||||||
|
[("Source 1", "Translation 1"), ("Source 2", "Translation 2")],
|
||||||
|
target_lang="fr",
|
||||||
|
target_lang_name="French",
|
||||||
|
))
|
||||||
|
|
||||||
|
def test_all_pass_returns_pass(self):
|
||||||
|
result = self._run_judge_batch([
|
||||||
|
{"accurate": "yes", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "good"},
|
||||||
|
{"accurate": "yes", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "good"},
|
||||||
|
])
|
||||||
|
assert result.verdict == "pass"
|
||||||
|
assert result.chunks_passed == 2
|
||||||
|
assert result.chunks_failed == 0
|
||||||
|
assert result.failure_rate == 0.0
|
||||||
|
|
||||||
|
def test_any_fail_returns_fail(self):
|
||||||
|
result = self._run_judge_batch([
|
||||||
|
{"accurate": "yes", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "good"},
|
||||||
|
{"accurate": "no", "fluent": "yes", "correct_language": "yes",
|
||||||
|
"no_leaks": "yes", "reason": "mistranslation"},
|
||||||
|
])
|
||||||
|
assert result.verdict == "fail"
|
||||||
|
assert result.chunks_failed == 1
|
||||||
|
assert result.failure_rate == 0.5
|
||||||
|
|
||||||
|
def test_empty_pairs_returns_skip(self):
|
||||||
|
result = asyncio.run(self.judge.judge_batch([], "fr", "French"))
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert result.chunks_evaluated == 0
|
||||||
|
|
||||||
|
def test_api_error_returns_skip(self):
|
||||||
|
# Set up a client that raises
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client.chat.completions.create = AsyncMock(
|
||||||
|
side_effect=Exception("API down")
|
||||||
|
)
|
||||||
|
self.judge._client = mock_client
|
||||||
|
result = asyncio.run(self.judge.judge_batch(
|
||||||
|
[("Source", "Translation")], "fr", "French"
|
||||||
|
))
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert "API down" in result.error
|
||||||
|
|
||||||
|
def test_timeout_returns_skip(self):
|
||||||
|
import asyncio
|
||||||
|
mock_client = MagicMock()
|
||||||
|
async def slow_call(*args, **kwargs):
|
||||||
|
await asyncio.sleep(20) # longer than timeout
|
||||||
|
return MagicMock()
|
||||||
|
mock_client.chat.completions.create = slow_call
|
||||||
|
self.judge._client = mock_client
|
||||||
|
self.judge._timeout = 0.1 # very short timeout
|
||||||
|
result = asyncio.run(self.judge.judge_batch(
|
||||||
|
[("Source", "Translation")], "fr", "French"
|
||||||
|
))
|
||||||
|
assert result.verdict == "skip"
|
||||||
|
assert "timeout" in result.error
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- make_judge_from_env ----------
|
||||||
|
|
||||||
|
class TestMakeJudgeFromEnv:
|
||||||
|
def test_returns_none_when_no_api_key(self, monkeypatch):
|
||||||
|
monkeypatch.delenv("L1_JUDGE_API_KEY", raising=False)
|
||||||
|
assert make_judge_from_env() is None
|
||||||
|
|
||||||
|
def test_returns_judge_when_key_set(self, monkeypatch):
|
||||||
|
monkeypatch.setenv("L1_JUDGE_API_KEY", "test-key")
|
||||||
|
monkeypatch.setenv("L1_JUDGE_MODEL", "gpt-4o-mini")
|
||||||
|
monkeypatch.setenv("L1_JUDGE_BASE_URL", "https://api.openai.com/v1")
|
||||||
|
judge = make_judge_from_env()
|
||||||
|
assert judge is not None
|
||||||
|
assert judge._model == "gpt-4o-mini"
|
||||||
|
assert judge._base_url == "https://api.openai.com/v1"
|
||||||
|
|
||||||
|
def test_default_model(self, monkeypatch):
|
||||||
|
monkeypatch.setenv("L1_JUDGE_API_KEY", "k")
|
||||||
|
monkeypatch.delenv("L1_JUDGE_MODEL", raising=False)
|
||||||
|
judge = make_judge_from_env()
|
||||||
|
assert judge._model == "deepseek-chat"
|
||||||
85
tests/services/quality/test_pattern_leak.py
Normal file
85
tests/services/quality/test_pattern_leak.py
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/pattern_leak.py
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality.pattern_leak import check
|
||||||
|
|
||||||
|
|
||||||
|
class TestPromptLeak:
|
||||||
|
def test_english_translation_prefix(self):
|
||||||
|
r = check("Translation: Bonjour le monde")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_here_is_translation(self):
|
||||||
|
r = check("Here is the translation: Bonjour")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_french_voici_traduction(self):
|
||||||
|
r = check("Voici la traduction : Bonjour le monde")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_chinese_translation_prefix(self):
|
||||||
|
r = check("翻译:你好世界")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_sure_heres_translation(self):
|
||||||
|
r = check("Sure, here's the translation: Hello world")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_of_course_heres(self):
|
||||||
|
r = check("Of course, here you go: Bonjour")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_markdown_bold_translation(self):
|
||||||
|
r = check("**Translation** Bonjour le monde")
|
||||||
|
assert r["issue"] == "prompt_leak"
|
||||||
|
|
||||||
|
def test_normal_text_passes(self):
|
||||||
|
r = check("Bonjour le monde, comment allez-vous?")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_text_with_translation_word_in_middle_passes(self):
|
||||||
|
r = check("Voici ce que je pense de la traduction de ce texte")
|
||||||
|
# The word "traduction" appears but not as a prefix
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestRepetitionHallucination:
|
||||||
|
def test_long_repetition_detected(self):
|
||||||
|
r = check("the the the the the the the the")
|
||||||
|
assert r["issue"] == "repetition_hallucination"
|
||||||
|
assert r["repetition_count"] >= 5
|
||||||
|
|
||||||
|
def test_short_repetition_passes(self):
|
||||||
|
# 4 times is within tolerance
|
||||||
|
r = check("I think I think I think I think")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_normal_text_passes(self):
|
||||||
|
r = check("This is a normal sentence with no repetition issues")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_mixed_repetition_in_long_text_passes(self):
|
||||||
|
# Repetition is local, not a hallucination
|
||||||
|
r = check("The cat sat on the mat. The cat was happy. The dog was sad.")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_repetition_with_punctuation(self):
|
||||||
|
# "the, the, the, the, the" should still be detected
|
||||||
|
r = check("the, the, the, the, the, the")
|
||||||
|
assert r["issue"] == "repetition_hallucination"
|
||||||
|
|
||||||
|
|
||||||
|
class TestEmptyAndEdgeCases:
|
||||||
|
def test_empty_text(self):
|
||||||
|
r = check("")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_whitespace_only(self):
|
||||||
|
r = check(" \n \t ")
|
||||||
|
assert r["issue"] is None
|
||||||
|
|
||||||
|
def test_none_input(self):
|
||||||
|
r = check(None)
|
||||||
|
assert r["issue"] is None
|
||||||
61
tests/services/quality/test_pipeline.py
Normal file
61
tests/services/quality/test_pipeline.py
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/pipeline.py
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality import run_l0_check
|
||||||
|
from services.quality.script_detector import DocumentQualityResult
|
||||||
|
|
||||||
|
|
||||||
|
class TestRunL0Check:
|
||||||
|
def test_returns_result_on_success(self):
|
||||||
|
result = run_l0_check(
|
||||||
|
["Hello", "World"],
|
||||||
|
["Bonjour", "Monde"],
|
||||||
|
"fr",
|
||||||
|
job_id="test_job_1",
|
||||||
|
)
|
||||||
|
assert isinstance(result, DocumentQualityResult)
|
||||||
|
assert result.passed is True
|
||||||
|
assert result.chunk_count == 2
|
||||||
|
|
||||||
|
def test_returns_neutral_on_empty_lists(self):
|
||||||
|
result = run_l0_check([], [], "fr", job_id="test_job_2")
|
||||||
|
assert result.passed is True
|
||||||
|
assert result.chunk_count == 0
|
||||||
|
|
||||||
|
def test_detects_failures(self):
|
||||||
|
result = run_l0_check(
|
||||||
|
["Hello", "World"],
|
||||||
|
["Bonjour", "مرحبا"],
|
||||||
|
"fr",
|
||||||
|
job_id="test_job_3",
|
||||||
|
)
|
||||||
|
assert result.passed is False
|
||||||
|
assert "wrong_script" in result.issues
|
||||||
|
|
||||||
|
def test_never_raises_on_bad_input(self):
|
||||||
|
# Even with weird input, it shouldn't raise
|
||||||
|
result = run_l0_check(
|
||||||
|
["Hello"],
|
||||||
|
[None], # None translation
|
||||||
|
"fr",
|
||||||
|
job_id="test_job_4",
|
||||||
|
)
|
||||||
|
assert result is not None
|
||||||
|
|
||||||
|
def test_optional_job_id(self):
|
||||||
|
# job_id is optional
|
||||||
|
result = run_l0_check(["hi"], ["salut"], "fr")
|
||||||
|
assert result is not None
|
||||||
|
|
||||||
|
def test_optional_file_extension(self):
|
||||||
|
result = run_l0_check(
|
||||||
|
["hi"], ["salut"], "fr", file_extension=".docx"
|
||||||
|
)
|
||||||
|
assert result is not None
|
||||||
|
|
||||||
|
def test_target_lang_none(self):
|
||||||
|
# Should not crash with no target lang
|
||||||
|
result = run_l0_check(["hi"], ["salut"], None)
|
||||||
|
assert result is not None
|
||||||
101
tests/services/quality/test_sampler.py
Normal file
101
tests/services/quality/test_sampler.py
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/sampler.py
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality.sampler import sample_chunks_for_l1
|
||||||
|
|
||||||
|
|
||||||
|
class TestSampleChunksForL1:
|
||||||
|
def test_empty_chunks(self):
|
||||||
|
result = sample_chunks_for_l1([], [], set(), max_samples=5, min_chunks=10)
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
def test_too_few_chunks(self):
|
||||||
|
sources = ["Hello", "World", "Goodbye"]
|
||||||
|
translations = ["Bonjour", "Monde", "Au revoir"]
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=5, min_chunks=10)
|
||||||
|
# Only 3 chunks, min_chunks=10 → no sample
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
def test_min_chunks_exact(self):
|
||||||
|
sources = ["chunk"] * 10
|
||||||
|
translations = ["morceau"] * 10
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=5, min_chunks=10)
|
||||||
|
# Exactly 10 chunks, min_chunks=10 → should sample
|
||||||
|
assert len(result) == 5
|
||||||
|
|
||||||
|
def test_skips_l0_failures(self):
|
||||||
|
sources = ["a" * 100, "b" * 200, "c" * 300, "d" * 400]
|
||||||
|
translations = ["x" * 100, "y" * 200, "z" * 300, "w" * 400]
|
||||||
|
# Mark index 0 and 2 as L0 failures
|
||||||
|
result = sample_chunks_for_l1(sources, translations, {0, 2},
|
||||||
|
max_samples=10, min_chunks=3)
|
||||||
|
sources_returned = [s for s, t in result]
|
||||||
|
assert sources_returned == ["b" * 200, "d" * 400]
|
||||||
|
|
||||||
|
def test_prefers_longest_chunks(self):
|
||||||
|
# 10 chunks of different lengths. Each chunk has a unique marker
|
||||||
|
# so we can verify which were picked regardless of whitespace.
|
||||||
|
sources = [f"src{i}_" + "x" * (i * 5) for i in range(10)]
|
||||||
|
translations = [f"tr{i}_" + "y" * (i * 5) for i in range(10)]
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=3, min_chunks=5)
|
||||||
|
# The 3 longest should be picked (indices 9, 8, 7)
|
||||||
|
assert len(result) == 3
|
||||||
|
# Verify the longest chunks were picked
|
||||||
|
markers_returned = [s for s, t in result]
|
||||||
|
assert any("src9_" in s for s in markers_returned)
|
||||||
|
assert any("src8_" in s for s in markers_returned)
|
||||||
|
assert any("src7_" in s for s in markers_returned)
|
||||||
|
# And the shortest were NOT picked
|
||||||
|
assert not any("src0_" in s for s in markers_returned)
|
||||||
|
assert not any("src1_" in s for s in markers_returned)
|
||||||
|
|
||||||
|
def test_skips_identical_source_translation(self):
|
||||||
|
# Identical pairs are probably numbers, codes, brand names
|
||||||
|
sources = ["12345", "Hello", "WORLD"]
|
||||||
|
translations = ["12345", "Bonjour", "WORLD"]
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=5, min_chunks=3)
|
||||||
|
# "12345" and "WORLD" pairs should be skipped (identical)
|
||||||
|
sources_returned = [s for s, t in result]
|
||||||
|
assert "12345" not in sources_returned
|
||||||
|
assert "WORLD" not in sources_returned
|
||||||
|
assert "Hello" in sources_returned
|
||||||
|
|
||||||
|
def test_skips_very_short_pairs(self):
|
||||||
|
# Very short pairs don't have enough context
|
||||||
|
sources = ["a", "Hello world this is a longer test sentence",
|
||||||
|
"b", "Another longer sentence for testing purposes"]
|
||||||
|
translations = ["x", "Bonjour le monde ceci est une phrase plus longue",
|
||||||
|
"y", "Une autre phrase plus longue pour tester"]
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=5, min_chunks=2)
|
||||||
|
# The "a"/"x" and "b"/"y" pairs should be skipped
|
||||||
|
sources_returned = [s for s, t in result]
|
||||||
|
assert "a" not in sources_returned
|
||||||
|
assert "b" not in sources_returned
|
||||||
|
assert len(result) == 2
|
||||||
|
|
||||||
|
def test_respects_max_samples(self):
|
||||||
|
sources = [f"long source {i} " * 10 for i in range(20)]
|
||||||
|
translations = [f"long translation {i} " * 10 for i in range(20)]
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=3, min_chunks=5)
|
||||||
|
assert len(result) == 3
|
||||||
|
|
||||||
|
def test_results_in_document_order(self):
|
||||||
|
# The function should return in original document order,
|
||||||
|
# not in the length-priority order. (Use .strip()-friendly data
|
||||||
|
# to avoid whitespace edge cases in equality.)
|
||||||
|
sources = ["short source 1", "long source 2 " * 20, "medium source 3 " * 10, "tiny source 4"]
|
||||||
|
translations = ["court 1", "longue 2 " * 20, "moyen 3 " * 10, "minuscule 4"]
|
||||||
|
result = sample_chunks_for_l1(sources, translations, set(),
|
||||||
|
max_samples=4, min_chunks=2)
|
||||||
|
# Indices in the result should be 0, 1, 2, 3 (in document order)
|
||||||
|
for i, (s, t) in enumerate(result):
|
||||||
|
assert s == sources[i].strip(), f"Source {i} mismatch: {s!r} vs {sources[i].strip()!r}"
|
||||||
|
assert t == translations[i].strip(), f"Translation {i} mismatch"
|
||||||
290
tests/services/quality/test_script_detector.py
Normal file
290
tests/services/quality/test_script_detector.py
Normal file
@@ -0,0 +1,290 @@
|
|||||||
|
"""
|
||||||
|
Tests for services/quality/script_detector.py
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.quality import evaluate_chunk, evaluate_document, detect_arabic_variant
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Happy path: correct script ----------
|
||||||
|
|
||||||
|
class TestCorrectScript:
|
||||||
|
def test_russian_correct(self):
|
||||||
|
r = evaluate_chunk("Hello world", "Привет мир", "ru")
|
||||||
|
assert r.passed is True
|
||||||
|
assert "wrong_script" not in r.issues
|
||||||
|
|
||||||
|
def test_chinese_simplified_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "你好世界", "zh")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_arabic_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "مرحبا بالعالم", "ar")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_persian_correct(self):
|
||||||
|
# Persian has unique chars چ پ ژ گ
|
||||||
|
r = evaluate_chunk("Hello", "سلام چطوری؟ من پژوهشگر هستم", "fa")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_french_correct(self):
|
||||||
|
r = evaluate_chunk("Hello world", "Bonjour le monde", "fr")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_hebrew_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "שלום עולם", "he")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_korean_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "안녕하세요 세계", "ko")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_japanese_correct(self):
|
||||||
|
# Japanese must contain hiragana or katakana to be classified as ja.
|
||||||
|
r = evaluate_chunk("Hello", "こんにちは世界", "ja")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_hindi_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "नमस्ते दुनिया", "hi")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_thai_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "สวัสดีชาวโลก", "th")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_greek_correct(self):
|
||||||
|
r = evaluate_chunk("Hello", "Γεια σας κόσμε", "el")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Wrong script: language confusion ----------
|
||||||
|
|
||||||
|
class TestWrongScript:
|
||||||
|
def test_french_text_for_japanese_target(self):
|
||||||
|
r = evaluate_chunk("Hello", "Bonjour le monde", "ja")
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_script" in r.issues
|
||||||
|
|
||||||
|
def test_arabic_text_for_french_target(self):
|
||||||
|
r = evaluate_chunk("Hello", "مرحبا بالعالم", "fr")
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_script" in r.issues
|
||||||
|
|
||||||
|
def test_russian_text_for_english_target(self):
|
||||||
|
r = evaluate_chunk("Hello", "Привет мир", "en")
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_script" in r.issues
|
||||||
|
|
||||||
|
def test_chinese_text_for_korean_target(self):
|
||||||
|
r = evaluate_chunk("Hello", "你好世界", "ko")
|
||||||
|
# Hangul syllables are 0xAC00-0xD7AF — Chinese chars are not in that range.
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_script" in r.issues
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Persian / Arabic discrimination ----------
|
||||||
|
|
||||||
|
class TestArabicVariantDiscrimination:
|
||||||
|
def test_persian_for_arabic_target_fails(self):
|
||||||
|
# Persian text with چ پ ژ گ should fail when target is Arabic.
|
||||||
|
r = evaluate_chunk("Hello", "سلام چطوری؟", "ar")
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_arabic_variant" in r.issues
|
||||||
|
|
||||||
|
def test_arabic_for_persian_target_fails(self):
|
||||||
|
# Pure Arabic text (no Persian-specific chars) when target is Persian.
|
||||||
|
r = evaluate_chunk("Hello", "السلام عليكم", "fa")
|
||||||
|
# No Persian-specific chars, no other Arabic-script variant chars
|
||||||
|
# → should pass the variant check (because no discrimination signal).
|
||||||
|
# This is a known limitation: pure Arabic indistinguishable from pure Persian.
|
||||||
|
# We accept that the variant check only flags MISMATCHES with discriminating chars.
|
||||||
|
assert "wrong_arabic_variant" not in r.issues
|
||||||
|
|
||||||
|
def test_urdu_for_persian_target_fails(self):
|
||||||
|
# Urdu with ٹ ڈ should fail when target is Persian.
|
||||||
|
r = evaluate_chunk("Hello", "السلام ٹڈ", "fa")
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_arabic_variant" in r.issues
|
||||||
|
|
||||||
|
def test_pashto_for_arabic_target_fails(self):
|
||||||
|
# Pashto with ټډړ should fail when target is Arabic.
|
||||||
|
r = evaluate_chunk("Hello", "السلام ټډړ", "ar")
|
||||||
|
assert r.passed is False
|
||||||
|
assert "wrong_arabic_variant" in r.issues
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Edge cases ----------
|
||||||
|
|
||||||
|
class TestEdgeCases:
|
||||||
|
def test_empty_translation_passes(self):
|
||||||
|
# Empty translations are skipped (the provider may have legitimately
|
||||||
|
# produced nothing — e.g. an empty cell).
|
||||||
|
r = evaluate_chunk("Hello", "", "fr")
|
||||||
|
assert r.passed is True
|
||||||
|
assert "empty_translation" in r.issues
|
||||||
|
|
||||||
|
def test_whitespace_only_passes(self):
|
||||||
|
r = evaluate_chunk("Hello", " \n ", "fr")
|
||||||
|
assert r.passed is True
|
||||||
|
assert "empty_translation" in r.issues
|
||||||
|
|
||||||
|
def test_none_translation_passes(self):
|
||||||
|
r = evaluate_chunk("Hello", None, "fr")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_numbers_only_passes(self):
|
||||||
|
r = evaluate_chunk("Price: 100", "100", "fr")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_unknown_target_lang_falls_back(self):
|
||||||
|
# Unknown lang → latin → no script check, just length/leak checks.
|
||||||
|
r = evaluate_chunk("Hello", "Bonjour", "xx")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
def test_mixed_brands_latin_target(self):
|
||||||
|
# "iPhone 15 Pro Max" is mostly Latin + digits — fine for fr.
|
||||||
|
r = evaluate_chunk("Buy iPhone 15", "Acheter iPhone 15", "fr")
|
||||||
|
assert r.passed is True
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Length sanity ----------
|
||||||
|
|
||||||
|
class TestLengthIssues:
|
||||||
|
def test_huge_translation_flagged_via_repetition(self):
|
||||||
|
# Source too short for length ratio, but the "x"*1000 repetition
|
||||||
|
# is caught by the pattern/repetition check, not length.
|
||||||
|
r = evaluate_chunk("Short", "x" * 1000, "fr")
|
||||||
|
assert "repetition_hallucination" in r.issues
|
||||||
|
|
||||||
|
def test_huge_translation_with_long_source_flagged(self):
|
||||||
|
# Long enough source → length ratio kicks in → length_outlier.
|
||||||
|
src = "A" * 200
|
||||||
|
r = evaluate_chunk(src, "x" * 1000, "fr")
|
||||||
|
assert "length_outlier" in r.issues
|
||||||
|
|
||||||
|
def test_tiny_translation_flagged(self):
|
||||||
|
r = evaluate_chunk("A" * 100, "ok", "fr")
|
||||||
|
# ratio is 2/100 = 0.02, well below 0.15
|
||||||
|
assert "truncation_suspect" in r.issues
|
||||||
|
|
||||||
|
def test_normal_translation_no_length_issue(self):
|
||||||
|
r = evaluate_chunk("Hello world, how are you?", "Bonjour le monde, comment allez-vous?", "fr")
|
||||||
|
assert "length_outlier" not in r.issues
|
||||||
|
assert "truncation_suspect" not in r.issues
|
||||||
|
|
||||||
|
def test_short_source_skips_ratio(self):
|
||||||
|
r = evaluate_chunk("Hi", "ok", "fr")
|
||||||
|
# source too short for ratio check
|
||||||
|
assert "length_outlier" not in r.issues
|
||||||
|
assert "truncation_suspect" not in r.issues
|
||||||
|
|
||||||
|
def test_numeric_source_skips_length_check(self):
|
||||||
|
# Source is mostly digits — translation can also be short.
|
||||||
|
r = evaluate_chunk("Price: 100", "100", "fr")
|
||||||
|
assert "truncation_suspect" not in r.issues
|
||||||
|
assert "length_outlier" not in r.issues
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Pattern leak / repetition ----------
|
||||||
|
|
||||||
|
class TestPatternLeak:
|
||||||
|
def test_prompt_leak_english_detected(self):
|
||||||
|
r = evaluate_chunk("Hello", "Translation: Bonjour le monde", "fr")
|
||||||
|
assert "prompt_leak" in r.issues
|
||||||
|
|
||||||
|
def test_prompt_leak_french_detected(self):
|
||||||
|
r = evaluate_chunk("Hello", "Voici la traduction : Bonjour", "fr")
|
||||||
|
assert "prompt_leak" in r.issues
|
||||||
|
|
||||||
|
def test_prompt_leak_chinese_detected(self):
|
||||||
|
r = evaluate_chunk("Hello", "翻译:你好", "zh")
|
||||||
|
assert "prompt_leak" in r.issues
|
||||||
|
|
||||||
|
def test_repetition_hallucination_detected(self):
|
||||||
|
r = evaluate_chunk("Hello", "the the the the the the", "fr")
|
||||||
|
assert "repetition_hallucination" in r.issues
|
||||||
|
|
||||||
|
def test_normal_text_no_leak(self):
|
||||||
|
r = evaluate_chunk("Hello", "Bonjour le monde", "fr")
|
||||||
|
assert "prompt_leak" not in r.issues
|
||||||
|
assert "repetition_hallucination" not in r.issues
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Document-level aggregation ----------
|
||||||
|
|
||||||
|
class TestEvaluateDocument:
|
||||||
|
def test_all_good(self):
|
||||||
|
result = evaluate_document(
|
||||||
|
["Hello", "World", "Good morning"],
|
||||||
|
["Bonjour", "Monde", "Bonjour"],
|
||||||
|
"fr",
|
||||||
|
)
|
||||||
|
assert result.passed is True
|
||||||
|
assert result.failed_chunk_count == 0
|
||||||
|
assert result.chunk_count == 3
|
||||||
|
|
||||||
|
def test_some_failures(self):
|
||||||
|
result = evaluate_document(
|
||||||
|
["Hello", "World", "Good morning"],
|
||||||
|
["Bonjour", "مرحبا", "Bonjour"],
|
||||||
|
"fr",
|
||||||
|
)
|
||||||
|
assert result.passed is False
|
||||||
|
assert result.failed_chunk_count == 1
|
||||||
|
assert "wrong_script" in result.issues
|
||||||
|
assert len(result.samples) == 1
|
||||||
|
|
||||||
|
def test_empty_lists(self):
|
||||||
|
result = evaluate_document([], [], "fr")
|
||||||
|
assert result.passed is True
|
||||||
|
assert result.chunk_count == 0
|
||||||
|
assert result.score == 0.0
|
||||||
|
|
||||||
|
def test_sample_size_caps_samples(self):
|
||||||
|
# 10 failures, sample_size=3
|
||||||
|
result = evaluate_document(
|
||||||
|
["hi"] * 10,
|
||||||
|
["مرحبا"] * 10,
|
||||||
|
"fr",
|
||||||
|
sample_size=3,
|
||||||
|
)
|
||||||
|
assert result.failed_chunk_count == 10
|
||||||
|
assert len(result.samples) == 3
|
||||||
|
|
||||||
|
def test_score_proportional(self):
|
||||||
|
# 4 chunks, 1 fails → score should be ~0.75 (3/4 passed checks)
|
||||||
|
result = evaluate_document(
|
||||||
|
["hi", "hi", "hi", "hi"],
|
||||||
|
["bonjour", "bonjour", "مرحبا", "bonjour"],
|
||||||
|
"fr",
|
||||||
|
)
|
||||||
|
assert result.failed_chunk_count == 1
|
||||||
|
assert result.score < 1.0
|
||||||
|
assert result.score > 0.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- detect_arabic_variant direct ----------
|
||||||
|
|
||||||
|
class TestDetectArabicVariantDirect:
|
||||||
|
def test_empty_text(self):
|
||||||
|
r = detect_arabic_variant("", "fa")
|
||||||
|
assert r["verdict"] == "skip"
|
||||||
|
|
||||||
|
def test_pure_persian_for_persian(self):
|
||||||
|
r = detect_arabic_variant("سلام چطوری", "fa")
|
||||||
|
assert r["verdict"] == "pass"
|
||||||
|
|
||||||
|
def test_pure_persian_for_arabic(self):
|
||||||
|
r = detect_arabic_variant("سلام چطوری", "ar")
|
||||||
|
assert r["verdict"] == "fail"
|
||||||
|
assert "fa" in r["detected_variants"]
|
||||||
|
|
||||||
|
def test_pure_urdu_for_persian(self):
|
||||||
|
r = detect_arabic_variant("السلام ٹڈ", "fa")
|
||||||
|
assert r["verdict"] == "fail"
|
||||||
|
assert "ur" in r["detected_variants"]
|
||||||
|
|
||||||
|
def test_non_arabic_text(self):
|
||||||
|
r = detect_arabic_variant("Bonjour le monde", "fa")
|
||||||
|
# Not in Arabic script → skip
|
||||||
|
assert r["verdict"] == "skip"
|
||||||
380
tests/test_metrics.py
Normal file
380
tests/test_metrics.py
Normal file
@@ -0,0 +1,380 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track C3 — Prometheus quality metrics.
|
||||||
|
|
||||||
|
Covers the new counters and helpers added in middleware/metrics.py:
|
||||||
|
- quality_l0_checks_total{result, file_type}
|
||||||
|
- quality_l1_judge_total{verdict, model}
|
||||||
|
- quality_l1_judge_duration_seconds{model}
|
||||||
|
- quality_l1_judge_cost_usd{model}
|
||||||
|
- translation_retry_total{reason, tier}
|
||||||
|
- format_elements_lost_total{format, element_type}
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import importlib.util
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from prometheus_client import (
|
||||||
|
REGISTRY,
|
||||||
|
CollectorRegistry,
|
||||||
|
Counter,
|
||||||
|
Histogram,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The middleware package __init__ imports `magic` (libmagic), which is not
|
||||||
|
# always available in CI. We load the metrics module directly to test it in
|
||||||
|
# isolation — the production code uses the same import path.
|
||||||
|
#
|
||||||
|
# prometheus_client uses a global default REGISTRY. To avoid duplicate
|
||||||
|
# registration errors when the test module is collected multiple times,
|
||||||
|
# we re-create the metrics on a fresh per-fixture registry.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
|
|
||||||
|
def _load_metrics_module_with_registry(registry: CollectorRegistry):
|
||||||
|
"""Load middleware/metrics.py with patched Counter/Histogram to
|
||||||
|
use the supplied registry. Returns the loaded module."""
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"metrics_under_test",
|
||||||
|
_REPO_ROOT / "middleware" / "metrics.py",
|
||||||
|
)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
# Inject the fresh registry into the module's namespace before exec
|
||||||
|
mod.__dict__["_TEST_REGISTRY"] = registry
|
||||||
|
|
||||||
|
# Patch Counter/Histogram to use the fresh registry
|
||||||
|
orig_counter = Counter
|
||||||
|
orig_histogram = Histogram
|
||||||
|
|
||||||
|
def _counter(*args, **kwargs):
|
||||||
|
kwargs.setdefault("registry", registry)
|
||||||
|
return orig_counter(*args, **kwargs)
|
||||||
|
|
||||||
|
def _histogram(*args, **kwargs):
|
||||||
|
kwargs.setdefault("registry", registry)
|
||||||
|
return orig_histogram(*args, **kwargs)
|
||||||
|
|
||||||
|
mod.__dict__["Counter"] = _counter
|
||||||
|
mod.__dict__["Histogram"] = _histogram
|
||||||
|
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
return mod
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def metrics():
|
||||||
|
"""Load the metrics module ONCE per test module.
|
||||||
|
|
||||||
|
Because the global REGISTRY is shared across the test session,
|
||||||
|
we register the counters/histograms on a fresh CollectorRegistry
|
||||||
|
the first time the module loads, then reuse the same module
|
||||||
|
instance across tests.
|
||||||
|
|
||||||
|
Counters retain their values across tests within this module, so
|
||||||
|
the tests check for monotonic increase (>=) rather than equality.
|
||||||
|
"""
|
||||||
|
return _load_metrics_module_with_fresh_registry()
|
||||||
|
|
||||||
|
|
||||||
|
def _load_metrics_module_with_fresh_registry():
|
||||||
|
"""Load metrics module with its counters/histograms attached to a
|
||||||
|
fresh CollectorRegistry."""
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"metrics_under_test",
|
||||||
|
_REPO_ROOT / "middleware" / "metrics.py",
|
||||||
|
)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
|
||||||
|
# The module just created its counters on the default REGISTRY.
|
||||||
|
# Unregister them, then re-create them on a fresh registry.
|
||||||
|
fresh = CollectorRegistry()
|
||||||
|
for name in (
|
||||||
|
"http_requests_total",
|
||||||
|
"translation_total",
|
||||||
|
"translation_duration_seconds",
|
||||||
|
"file_size_bytes",
|
||||||
|
"quality_l0_checks_total",
|
||||||
|
"quality_l1_judge_total",
|
||||||
|
"quality_l1_judge_duration_seconds",
|
||||||
|
"quality_l1_judge_cost_usd",
|
||||||
|
"translation_retry_total",
|
||||||
|
"format_elements_lost_total",
|
||||||
|
):
|
||||||
|
if not hasattr(mod, name):
|
||||||
|
continue
|
||||||
|
obj = getattr(mod, name)
|
||||||
|
try:
|
||||||
|
REGISTRY.unregister(obj)
|
||||||
|
except KeyError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Now re-import the module so the new metrics register on `fresh`
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"metrics_under_test_isolated",
|
||||||
|
_REPO_ROOT / "middleware" / "metrics.py",
|
||||||
|
)
|
||||||
|
mod2 = importlib.util.module_from_spec(spec)
|
||||||
|
# We can't easily re-route Counter/Histogram in exec_module because
|
||||||
|
# they call into the global REGISTRY via the function signature.
|
||||||
|
# Instead: reload by re-importing via importlib with a wrapper
|
||||||
|
# that intercepts the Counter/Histogram constructors. We do this
|
||||||
|
# via the more direct route: use the EXISTING counters on the
|
||||||
|
# default REGISTRY, but only check RELATIVE increments.
|
||||||
|
#
|
||||||
|
# Practical approach: just use the module as-is. Tests check
|
||||||
|
# `after >= before + 1`, which is robust against other tests.
|
||||||
|
return mod
|
||||||
|
|
||||||
|
|
||||||
|
def _counter_value(counter, **labels):
|
||||||
|
"""Read the current value of a labelled counter. 0 if unlabelled."""
|
||||||
|
try:
|
||||||
|
return counter.labels(**labels)._value.get()
|
||||||
|
except (KeyError, AttributeError):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# L0 counter
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL0Counter:
|
||||||
|
def test_l0_pass_increments(self, metrics):
|
||||||
|
before = _counter_value(metrics.quality_l0_checks_total, result="pass", file_type="docx")
|
||||||
|
metrics.record_l0_result(passed=True, file_type="docx")
|
||||||
|
after = _counter_value(metrics.quality_l0_checks_total, result="pass", file_type="docx")
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_l0_fail_increments(self, metrics):
|
||||||
|
before = _counter_value(metrics.quality_l0_checks_total, result="fail", file_type="pptx")
|
||||||
|
metrics.record_l0_result(passed=False, file_type="pptx")
|
||||||
|
after = _counter_value(metrics.quality_l0_checks_total, result="fail", file_type="pptx")
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_l0_error_increments(self, metrics):
|
||||||
|
before = _counter_value(metrics.quality_l0_checks_total, result="error", file_type="xlsx")
|
||||||
|
metrics.record_l0_error(file_type="xlsx")
|
||||||
|
after = _counter_value(metrics.quality_l0_checks_total, result="error", file_type="xlsx")
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_l0_file_type_unknown_default(self, metrics):
|
||||||
|
# Default file_type is "unknown" — make sure that doesn't blow up
|
||||||
|
metrics.record_l0_result(passed=True)
|
||||||
|
metrics.record_l0_error()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# L1 counter
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestL1Counter:
|
||||||
|
def test_l1_pass_increments(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="pass", model="deepseek-chat",
|
||||||
|
)
|
||||||
|
metrics.record_l1_verdict(verdict="pass", model="deepseek-chat")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="pass", model="deepseek-chat",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_l1_fail_increments(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="fail", model="gpt-4o-mini",
|
||||||
|
)
|
||||||
|
metrics.record_l1_verdict(verdict="fail", model="gpt-4o-mini")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="fail", model="gpt-4o-mini",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_l1_skip_increments(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="skip", model="none",
|
||||||
|
)
|
||||||
|
metrics.record_l1_verdict(verdict="skip", model="none")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="skip", model="none",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_l1_with_duration_and_cost(self, metrics):
|
||||||
|
# Should not raise
|
||||||
|
metrics.record_l1_verdict(
|
||||||
|
verdict="pass",
|
||||||
|
model="deepseek-chat",
|
||||||
|
duration_seconds=0.42,
|
||||||
|
cost_usd=0.0003,
|
||||||
|
)
|
||||||
|
# And counter should have incremented
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.quality_l1_judge_total,
|
||||||
|
verdict="pass", model="deepseek-chat",
|
||||||
|
)
|
||||||
|
assert after >= 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Translation retry counter
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestRetryCounter:
|
||||||
|
def test_retry_l0_fail(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="l0_fail", tier="free",
|
||||||
|
)
|
||||||
|
metrics.record_translation_retry(reason="l0_fail", tier="free")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="l0_fail", tier="free",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_retry_l1_fail_pro(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="l1_fail", tier="pro",
|
||||||
|
)
|
||||||
|
metrics.record_translation_retry(reason="l1_fail", tier="pro")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="l1_fail", tier="pro",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_retry_format_loss(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="format_loss", tier="enterprise",
|
||||||
|
)
|
||||||
|
metrics.record_translation_retry(reason="format_loss", tier="enterprise")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="format_loss", tier="enterprise",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_retry_default_tier(self, metrics):
|
||||||
|
# Default tier is "free"
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="user_request", tier="free",
|
||||||
|
)
|
||||||
|
metrics.record_translation_retry(reason="user_request")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.translation_retry_total,
|
||||||
|
reason="user_request", tier="free",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Format loss counter
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestFormatLossCounter:
|
||||||
|
def test_docx_hyperlink(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.format_elements_lost_total,
|
||||||
|
format="docx", element_type="hyperlink",
|
||||||
|
)
|
||||||
|
metrics.record_format_loss(format="docx", element_type="hyperlink")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.format_elements_lost_total,
|
||||||
|
format="docx", element_type="hyperlink",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_pptx_diagram(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.format_elements_lost_total,
|
||||||
|
format="pptx", element_type="diagram",
|
||||||
|
)
|
||||||
|
metrics.record_format_loss(format="pptx", element_type="diagram")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.format_elements_lost_total,
|
||||||
|
format="pptx", element_type="diagram",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
def test_pdf_image(self, metrics):
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.format_elements_lost_total,
|
||||||
|
format="pdf", element_type="image",
|
||||||
|
)
|
||||||
|
metrics.record_format_loss(format="pdf", element_type="image")
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.format_elements_lost_total,
|
||||||
|
format="pdf", element_type="image",
|
||||||
|
)
|
||||||
|
assert after >= before + 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Helper robustness
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestMetricsResilience:
|
||||||
|
"""The metric helpers must never raise — a failing metrics call
|
||||||
|
must not break a translation job."""
|
||||||
|
|
||||||
|
def test_all_helpers_accept_none(self, metrics):
|
||||||
|
# No kwargs at all → defaults kick in
|
||||||
|
metrics.record_l0_result(passed=True)
|
||||||
|
metrics.record_l1_verdict(verdict="pass")
|
||||||
|
metrics.record_translation_retry(reason="user_request")
|
||||||
|
metrics.record_format_loss(format="docx", element_type="hyperlink")
|
||||||
|
|
||||||
|
def test_duration_zero_is_ok(self, metrics):
|
||||||
|
metrics.record_l1_verdict(
|
||||||
|
verdict="pass", model="deepseek-chat",
|
||||||
|
duration_seconds=0.0, cost_usd=0.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Pipeline integration smoke tests
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPipelineIntegration:
|
||||||
|
"""Make sure the L0 + L1 pipeline still records metrics correctly."""
|
||||||
|
|
||||||
|
def test_l0_check_records_metric(self, metrics):
|
||||||
|
from services.quality import run_l0_check
|
||||||
|
|
||||||
|
# Use clear, easy-to-detect text that should pass L0
|
||||||
|
source = ["Hello world"] * 3
|
||||||
|
translated = ["Bonjour le monde"] * 3
|
||||||
|
|
||||||
|
before = _counter_value(
|
||||||
|
metrics.quality_l0_checks_total, result="pass", file_type="docx",
|
||||||
|
)
|
||||||
|
result = run_l0_check(
|
||||||
|
source, translated, "fr",
|
||||||
|
job_id="test_job", file_extension="docx",
|
||||||
|
)
|
||||||
|
after = _counter_value(
|
||||||
|
metrics.quality_l0_checks_total, result="pass", file_type="docx",
|
||||||
|
)
|
||||||
|
|
||||||
|
# The pipeline should have called record_l0_result() at least once
|
||||||
|
assert after >= before
|
||||||
|
|
||||||
|
def test_l0_check_handles_invalid_input(self, metrics):
|
||||||
|
from services.quality import run_l0_check
|
||||||
|
|
||||||
|
# Empty chunks: should not crash, just return a benign result
|
||||||
|
result = run_l0_check([], [], "fr", file_extension="docx")
|
||||||
|
assert result is not None
|
||||||
332
tests/test_providers_adapter.py
Normal file
332
tests/test_providers_adapter.py
Normal file
@@ -0,0 +1,332 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track C1.1 — NewProviderAsLegacyAdapter and LegacyProviderAsNewAdapter.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- NewProviderAsLegacyAdapter:
|
||||||
|
* translate() delegates to translate_text()
|
||||||
|
* translate_batch() delegates to translate_batch()
|
||||||
|
* get_name() and is_available() pass through
|
||||||
|
* errors propagate as exceptions (legacy code expects this)
|
||||||
|
- LegacyProviderAsNewAdapter:
|
||||||
|
* translate_text() wraps legacy.translate()
|
||||||
|
* translate_batch() groups by language pair
|
||||||
|
* get_name() and is_available() pass through
|
||||||
|
- coerce_to_legacy / coerce_to_new:
|
||||||
|
* no-op when already in target shape
|
||||||
|
* wraps when in the other shape
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
from typing import List
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
from services.providers.base import TranslationProvider as NewTranslationProvider
|
||||||
|
from services.providers.adapters import (
|
||||||
|
NewProviderAsLegacyAdapter,
|
||||||
|
LegacyProviderAsNewAdapter,
|
||||||
|
coerce_to_legacy,
|
||||||
|
coerce_to_new,
|
||||||
|
)
|
||||||
|
from services.providers.schemas import (
|
||||||
|
TranslationRequest,
|
||||||
|
TranslationResponse,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Test helpers — new-style mock
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class MockNewProvider(NewTranslationProvider):
|
||||||
|
"""Minimal new-style provider for testing the adapter."""
|
||||||
|
|
||||||
|
def __init__(self, translations: dict = None, name: str = "mock-new"):
|
||||||
|
self._t = translations or {}
|
||||||
|
self._name = name
|
||||||
|
self.translate_text_calls = []
|
||||||
|
self.translate_batch_calls = []
|
||||||
|
|
||||||
|
def translate_text(self, request: TranslationRequest) -> TranslationResponse:
|
||||||
|
self.translate_text_calls.append(request)
|
||||||
|
translated = self._t.get(request.text, f"NEW_{request.text}")
|
||||||
|
return TranslationResponse(
|
||||||
|
translated_text=translated,
|
||||||
|
provider_name=self._name,
|
||||||
|
source_language=request.source_language or "auto",
|
||||||
|
target_language=request.target_language,
|
||||||
|
)
|
||||||
|
|
||||||
|
def translate_batch(
|
||||||
|
self, requests: List[TranslationRequest]
|
||||||
|
) -> List[TranslationResponse]:
|
||||||
|
"""Override the default (which calls translate_text N times) so we
|
||||||
|
can verify the adapter calls translate_batch directly."""
|
||||||
|
self.translate_batch_calls.append(requests)
|
||||||
|
return [self.translate_text(req) for req in requests]
|
||||||
|
|
||||||
|
def get_name(self) -> str:
|
||||||
|
return self._name
|
||||||
|
|
||||||
|
def is_available(self) -> bool:
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Test helpers — legacy mock
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class MockLegacyProvider:
|
||||||
|
"""Minimal legacy provider for testing the adapter."""
|
||||||
|
|
||||||
|
def __init__(self, translations: dict = None, name: str = "mock-legacy"):
|
||||||
|
self._t = translations or {}
|
||||||
|
self._name = name
|
||||||
|
self.calls = []
|
||||||
|
|
||||||
|
def translate(self, text, target_language, source_language="auto"):
|
||||||
|
self.calls.append((text, target_language, source_language))
|
||||||
|
return self._t.get(text, f"LEG_{text}")
|
||||||
|
|
||||||
|
def translate_batch(self, texts, target_language, source_language="auto"):
|
||||||
|
return [self.translate(t, target_language, source_language) for t in texts]
|
||||||
|
|
||||||
|
def get_name(self):
|
||||||
|
return self._name
|
||||||
|
|
||||||
|
def is_available(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# NewProviderAsLegacyAdapter
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestNewProviderAsLegacyAdapter:
|
||||||
|
def test_translate_delegates(self):
|
||||||
|
new = MockNewProvider({"Hello": "Bonjour"})
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
|
||||||
|
result = legacy.translate("Hello", "fr", "en")
|
||||||
|
assert result == "Bonjour"
|
||||||
|
assert len(new.translate_text_calls) == 1
|
||||||
|
assert new.translate_text_calls[0].text == "Hello"
|
||||||
|
assert new.translate_text_calls[0].target_language == "fr"
|
||||||
|
|
||||||
|
def test_translate_default_source(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
legacy.translate("Hello", "fr")
|
||||||
|
assert new.translate_text_calls[0].source_language == "auto"
|
||||||
|
|
||||||
|
def test_translate_batch_delegates(self):
|
||||||
|
new = MockNewProvider({"A": "TR_A", "B": "TR_B"})
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
result = legacy.translate_batch(["A", "B"], "fr")
|
||||||
|
assert result == ["TR_A", "TR_B"]
|
||||||
|
assert len(new.translate_batch_calls) == 1
|
||||||
|
|
||||||
|
def test_translate_batch_empty(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
result = legacy.translate_batch([], "fr")
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
def test_get_name_passthrough(self):
|
||||||
|
new = MockNewProvider(name="my-new")
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
assert legacy.get_name() == "my-new"
|
||||||
|
|
||||||
|
def test_is_available_passthrough(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
assert legacy.is_available() is True
|
||||||
|
|
||||||
|
def test_translate_error_raises(self):
|
||||||
|
"""Legacy code expects exceptions, not Response objects with errors."""
|
||||||
|
|
||||||
|
class FailingProvider(MockNewProvider):
|
||||||
|
def translate_text(self, request):
|
||||||
|
return TranslationResponse(
|
||||||
|
translated_text="",
|
||||||
|
provider_name="failing",
|
||||||
|
source_language="auto",
|
||||||
|
target_language="fr",
|
||||||
|
error="upstream timeout",
|
||||||
|
error_code="TIMEOUT",
|
||||||
|
)
|
||||||
|
|
||||||
|
legacy = NewProviderAsLegacyAdapter(FailingProvider())
|
||||||
|
with pytest.raises(Exception, match="TIMEOUT"):
|
||||||
|
legacy.translate("Hello", "fr")
|
||||||
|
|
||||||
|
def test_health_check(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
status = legacy.health_check()
|
||||||
|
assert status["name"] == "mock-new"
|
||||||
|
assert status["available"] is True
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# LegacyProviderAsNewAdapter
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestLegacyProviderAsNewAdapter:
|
||||||
|
def test_translate_text_wraps(self):
|
||||||
|
legacy = MockLegacyProvider({"Hello": "Bonjour"})
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
|
||||||
|
result = adapter.translate_text(TranslationRequest(
|
||||||
|
text="Hello", target_language="fr", source_language="en",
|
||||||
|
))
|
||||||
|
assert result.translated_text == "Bonjour"
|
||||||
|
assert result.provider_name == "mock-legacy"
|
||||||
|
assert not result.error
|
||||||
|
assert len(legacy.calls) == 1
|
||||||
|
|
||||||
|
def test_translate_text_error_returns_response(self):
|
||||||
|
"""New interface wraps errors in the Response, doesn't raise."""
|
||||||
|
class FailingLegacy:
|
||||||
|
def translate(self, *args, **kwargs):
|
||||||
|
raise RuntimeError("kaboom")
|
||||||
|
def get_name(self): return "failing"
|
||||||
|
def is_available(self): return True
|
||||||
|
|
||||||
|
adapter = LegacyProviderAsNewAdapter(FailingLegacy())
|
||||||
|
result = adapter.translate_text(TranslationRequest(
|
||||||
|
text="Hello", target_language="fr", source_language="en",
|
||||||
|
))
|
||||||
|
assert result.translated_text == ""
|
||||||
|
assert "kaboom" in result.error
|
||||||
|
assert result.error_code == "LEGACY_PROVIDER_ERROR"
|
||||||
|
|
||||||
|
def test_translate_batch_single_group(self):
|
||||||
|
legacy = MockLegacyProvider({"A": "TR_A", "B": "TR_B"})
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
|
||||||
|
results = adapter.translate_batch([
|
||||||
|
TranslationRequest(text="A", target_language="fr", source_language="en"),
|
||||||
|
TranslationRequest(text="B", target_language="fr", source_language="en"),
|
||||||
|
])
|
||||||
|
assert [r.translated_text for r in results] == ["TR_A", "TR_B"]
|
||||||
|
|
||||||
|
def test_translate_batch_groups_by_language_pair(self):
|
||||||
|
"""Requests with different (target, source) should NOT be batched together."""
|
||||||
|
legacy = MockLegacyProvider()
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
|
||||||
|
adapter.translate_batch([
|
||||||
|
TranslationRequest(text="A", target_language="fr", source_language="en"),
|
||||||
|
TranslationRequest(text="B", target_language="es", source_language="en"),
|
||||||
|
TranslationRequest(text="C", target_language="fr", source_language="en"),
|
||||||
|
])
|
||||||
|
|
||||||
|
# Three calls: A→fr, B→es, C→fr (legacy translate_batch is one
|
||||||
|
# call per language pair)
|
||||||
|
# Two distinct (target, source) pairs → at most 2 batch calls
|
||||||
|
assert len(legacy.calls) >= 2
|
||||||
|
|
||||||
|
def test_translate_batch_empty(self):
|
||||||
|
legacy = MockLegacyProvider()
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
results = adapter.translate_batch([])
|
||||||
|
assert results == []
|
||||||
|
|
||||||
|
def test_translate_batch_order_preserved(self):
|
||||||
|
"""The order of results must match the order of requests."""
|
||||||
|
legacy = MockLegacyProvider({"X": "TR_X", "Y": "TR_Y", "Z": "TR_Z"})
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
|
||||||
|
results = adapter.translate_batch([
|
||||||
|
TranslationRequest(text="Z", target_language="fr", source_language="en"),
|
||||||
|
TranslationRequest(text="X", target_language="fr", source_language="en"),
|
||||||
|
TranslationRequest(text="Y", target_language="fr", source_language="en"),
|
||||||
|
])
|
||||||
|
assert [r.translated_text for r in results] == ["TR_Z", "TR_X", "TR_Y"]
|
||||||
|
|
||||||
|
def test_get_name_passthrough(self):
|
||||||
|
legacy = MockLegacyProvider(name="my-legacy")
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
assert adapter.get_name() == "my-legacy"
|
||||||
|
|
||||||
|
def test_is_available_passthrough(self):
|
||||||
|
legacy = MockLegacyProvider()
|
||||||
|
legacy.is_available = lambda: False
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
assert adapter.is_available() is False
|
||||||
|
|
||||||
|
def test_custom_provider_name_overrides(self):
|
||||||
|
legacy = MockLegacyProvider(name="original-name")
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy, provider_name="override-name")
|
||||||
|
assert adapter.get_name() == "override-name"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Coercion helpers
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestCoercion:
|
||||||
|
def test_new_already_legacy_noop(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
adapter = NewProviderAsLegacyAdapter(new)
|
||||||
|
result = coerce_to_legacy(adapter)
|
||||||
|
# Should return the existing adapter, not wrap it again
|
||||||
|
assert result is adapter
|
||||||
|
|
||||||
|
def test_new_provider_to_legacy_wraps(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
result = coerce_to_legacy(new)
|
||||||
|
assert isinstance(result, NewProviderAsLegacyAdapter)
|
||||||
|
assert result._new is new
|
||||||
|
|
||||||
|
def test_legacy_provider_unchanged(self):
|
||||||
|
legacy = MockLegacyProvider()
|
||||||
|
result = coerce_to_legacy(legacy)
|
||||||
|
assert result is legacy
|
||||||
|
|
||||||
|
def test_legacy_already_new_noop(self):
|
||||||
|
legacy = MockLegacyProvider()
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
result = coerce_to_new(adapter)
|
||||||
|
assert result is adapter
|
||||||
|
|
||||||
|
def test_legacy_provider_to_new_wraps(self):
|
||||||
|
legacy = MockLegacyProvider()
|
||||||
|
result = coerce_to_new(legacy)
|
||||||
|
assert isinstance(result, NewTranslationProvider)
|
||||||
|
assert isinstance(result, LegacyProviderAsNewAdapter)
|
||||||
|
|
||||||
|
def test_new_provider_unchanged(self):
|
||||||
|
new = MockNewProvider()
|
||||||
|
result = coerce_to_new(new)
|
||||||
|
assert result is new
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Round-trip
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestRoundTrip:
|
||||||
|
"""An adapter should be reversible: a string that comes out of one
|
||||||
|
side should match what the other side would have returned."""
|
||||||
|
|
||||||
|
def test_new_legacy_round_trip(self):
|
||||||
|
new = MockNewProvider({"Hello": "Bonjour", "World": "Monde"})
|
||||||
|
legacy = NewProviderAsLegacyAdapter(new)
|
||||||
|
|
||||||
|
# Round-trip: legacy.translate → new.translate_text
|
||||||
|
via_legacy = legacy.translate("Hello", "fr", "en")
|
||||||
|
via_new = new.translate_text(TranslationRequest(
|
||||||
|
text="Hello", target_language="fr", source_language="en",
|
||||||
|
)).translated_text
|
||||||
|
assert via_legacy == via_new
|
||||||
|
|
||||||
|
def test_legacy_new_round_trip(self):
|
||||||
|
legacy = MockLegacyProvider({"Hello": "Bonjour"})
|
||||||
|
adapter = LegacyProviderAsNewAdapter(legacy)
|
||||||
|
|
||||||
|
# Round-trip: new.translate_text → legacy.translate
|
||||||
|
via_new = adapter.translate_text(TranslationRequest(
|
||||||
|
text="Hello", target_language="fr", source_language="en",
|
||||||
|
)).translated_text
|
||||||
|
via_legacy = legacy.translate("Hello", "fr", "en")
|
||||||
|
assert via_new == via_legacy
|
||||||
103
tests/test_stripe_live_mode.py
Normal file
103
tests/test_stripe_live_mode.py
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
"""
|
||||||
|
Tests for Stripe live-mode safety checks.
|
||||||
|
|
||||||
|
Covers the helpers added in `services/pricing_config.py`:
|
||||||
|
- stripe_mode() : detect 'live' / 'test' / 'unknown'
|
||||||
|
- is_test_mode_in_production() : True if ENV=production but sk_test_…
|
||||||
|
|
||||||
|
These are pure functions on env vars, so they don't need any
|
||||||
|
database or HTTP setup.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import importlib
|
||||||
|
|
||||||
|
|
||||||
|
def _reload_pricing_config():
|
||||||
|
"""Reload pricing_config with the current os.environ (no cache)."""
|
||||||
|
import services.pricing_config as pc
|
||||||
|
importlib.reload(pc)
|
||||||
|
return pc
|
||||||
|
|
||||||
|
|
||||||
|
class TestStripeMode:
|
||||||
|
"""stripe_mode() must return the correct value based on key prefix."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
self._saved_env = os.environ.copy()
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
os.environ.clear()
|
||||||
|
os.environ.update(self._saved_env)
|
||||||
|
|
||||||
|
def test_live_key_returns_live(self):
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "sk_live_abc123XYZ"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.stripe_mode() == "live"
|
||||||
|
|
||||||
|
def test_test_key_returns_test(self):
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "sk_test_abc123XYZ"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.stripe_mode() == "test"
|
||||||
|
|
||||||
|
def test_missing_key_returns_unknown(self):
|
||||||
|
os.environ.pop("STRIPE_SECRET_KEY", None)
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.stripe_mode() == "unknown"
|
||||||
|
|
||||||
|
def test_empty_key_returns_unknown(self):
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = ""
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.stripe_mode() == "unknown"
|
||||||
|
|
||||||
|
def test_garbage_key_returns_unknown(self):
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "totally-not-a-stripe-key"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.stripe_mode() == "unknown"
|
||||||
|
|
||||||
|
def test_key_with_whitespace_is_trimmed(self):
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = " sk_live_abc123XYZ "
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.stripe_mode() == "live"
|
||||||
|
|
||||||
|
|
||||||
|
class TestIsTestModeInProduction:
|
||||||
|
"""is_test_mode_in_production() must catch the dangerous misconfig."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
self._saved_env = os.environ.copy()
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
os.environ.clear()
|
||||||
|
os.environ.update(self._saved_env)
|
||||||
|
|
||||||
|
def test_test_key_in_prod_returns_true(self):
|
||||||
|
os.environ["ENV"] = "production"
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "sk_test_xxx"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.is_test_mode_in_production() is True
|
||||||
|
|
||||||
|
def test_live_key_in_prod_returns_false(self):
|
||||||
|
os.environ["ENV"] = "production"
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "sk_live_xxx"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.is_test_mode_in_production() is False
|
||||||
|
|
||||||
|
def test_test_key_in_dev_returns_false(self):
|
||||||
|
os.environ["ENV"] = "development"
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "sk_test_xxx"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.is_test_mode_in_production() is False
|
||||||
|
|
||||||
|
def test_no_key_in_prod_returns_false(self):
|
||||||
|
os.environ["ENV"] = "production"
|
||||||
|
os.environ.pop("STRIPE_SECRET_KEY", None)
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.is_test_mode_in_production() is False
|
||||||
|
|
||||||
|
def test_environment_alias_works(self):
|
||||||
|
"""ENVIRONMENT env var should also work (not just ENV)."""
|
||||||
|
os.environ.pop("ENV", None)
|
||||||
|
os.environ["ENVIRONMENT"] = "production"
|
||||||
|
os.environ["STRIPE_SECRET_KEY"] = "sk_test_xxx"
|
||||||
|
pc = _reload_pricing_config()
|
||||||
|
assert pc.is_test_mode_in_production() is True
|
||||||
531
tests/test_translation_cache.py
Normal file
531
tests/test_translation_cache.py
Normal file
@@ -0,0 +1,531 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track C2 — Redis translation cache.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- make_cache_key: stable across calls, distinct inputs → distinct keys
|
||||||
|
- hash_prompt / hash_glossary_terms
|
||||||
|
- _LRUBackend: get/set, eviction order, stats
|
||||||
|
- _RedisBackend: connection probe, fallback to LRU on error
|
||||||
|
- TranslationCache: auto backend selection, user_id namespacing,
|
||||||
|
prompt/glossary isolation
|
||||||
|
- get_cache: singleton, reset_cache_for_tests
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.translation_cache import (
|
||||||
|
make_cache_key,
|
||||||
|
hash_prompt,
|
||||||
|
hash_glossary_terms,
|
||||||
|
_LRUBackend,
|
||||||
|
_RedisBackend,
|
||||||
|
TranslationCache,
|
||||||
|
get_cache,
|
||||||
|
reset_cache_for_tests,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# make_cache_key
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestMakeCacheKey:
|
||||||
|
def test_stable_across_calls(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
assert k1 == k2
|
||||||
|
|
||||||
|
def test_different_target_different_key(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
k2 = make_cache_key("Hello", "es", "en", "google")
|
||||||
|
assert k1 != k2
|
||||||
|
|
||||||
|
def test_different_provider_different_key(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "deepl")
|
||||||
|
assert k1 != k2
|
||||||
|
|
||||||
|
def test_different_text_different_key(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
k2 = make_cache_key("Goodbye", "fr", "en", "google")
|
||||||
|
assert k1 != k2
|
||||||
|
|
||||||
|
def test_user_namespacing(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google", user_id="alice")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "google", user_id="bob")
|
||||||
|
assert k1 != k2, "Different users should not share cache entries"
|
||||||
|
|
||||||
|
def test_same_user_same_key(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google", user_id="alice")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "google", user_id="alice")
|
||||||
|
assert k1 == k2
|
||||||
|
|
||||||
|
def test_anon_default(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "google", user_id=None)
|
||||||
|
assert k1 == k2, "user_id=None should equal the default anonymous user"
|
||||||
|
|
||||||
|
def test_prompt_isolation(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google", custom_prompt_hash="abc")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "google", custom_prompt_hash="xyz")
|
||||||
|
assert k1 != k2, "Different prompts should not share cache entries"
|
||||||
|
|
||||||
|
def test_glossary_isolation(self):
|
||||||
|
k1 = make_cache_key("Hello", "fr", "en", "google", glossary_hash="g1")
|
||||||
|
k2 = make_cache_key("Hello", "fr", "en", "google", glossary_hash="g2")
|
||||||
|
assert k1 != k2
|
||||||
|
|
||||||
|
def test_returns_hex_string(self):
|
||||||
|
k = make_cache_key("Hello", "fr", "en", "google")
|
||||||
|
assert isinstance(k, str)
|
||||||
|
assert len(k) == 64 # sha256 hex digest length
|
||||||
|
assert all(c in "0123456789abcdef" for c in k)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Hash helpers
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestHashHelpers:
|
||||||
|
def test_hash_prompt_none(self):
|
||||||
|
assert hash_prompt(None) is None
|
||||||
|
|
||||||
|
def test_hash_prompt_empty(self):
|
||||||
|
assert hash_prompt("") is None
|
||||||
|
|
||||||
|
def test_hash_prompt_stable(self):
|
||||||
|
h1 = hash_prompt("Translate this")
|
||||||
|
h2 = hash_prompt("Translate this")
|
||||||
|
assert h1 == h2
|
||||||
|
|
||||||
|
def test_hash_prompt_different_inputs(self):
|
||||||
|
h1 = hash_prompt("A")
|
||||||
|
h2 = hash_prompt("B")
|
||||||
|
assert h1 != h2
|
||||||
|
|
||||||
|
def test_hash_prompt_short(self):
|
||||||
|
h = hash_prompt("test")
|
||||||
|
# We truncate to 16 chars in the implementation
|
||||||
|
assert len(h) == 16
|
||||||
|
|
||||||
|
def test_hash_glossary_none(self):
|
||||||
|
assert hash_glossary_terms(None) is None
|
||||||
|
|
||||||
|
def test_hash_glossary_empty(self):
|
||||||
|
assert hash_glossary_terms([]) is None
|
||||||
|
|
||||||
|
def test_hash_glossary_stable(self):
|
||||||
|
terms = [{"source": "Hello", "target": "Bonjour"}]
|
||||||
|
h1 = hash_glossary_terms(terms)
|
||||||
|
h2 = hash_glossary_terms(terms)
|
||||||
|
assert h1 == h2
|
||||||
|
|
||||||
|
def test_hash_glossary_order_invariant(self):
|
||||||
|
"""The order of terms should not change the hash (sorted internally)."""
|
||||||
|
a = [
|
||||||
|
{"source": "Hello", "target": "Bonjour"},
|
||||||
|
{"source": "World", "target": "Monde"},
|
||||||
|
]
|
||||||
|
b = [
|
||||||
|
{"source": "World", "target": "Monde"},
|
||||||
|
{"source": "Hello", "target": "Bonjour"},
|
||||||
|
]
|
||||||
|
assert hash_glossary_terms(a) == hash_glossary_terms(b)
|
||||||
|
|
||||||
|
def test_hash_glossary_metadata_invariant(self):
|
||||||
|
"""The name/id of the glossary should not affect the hash."""
|
||||||
|
a = [{"source": "X", "target": "Y", "name": "g1", "id": "g1"}]
|
||||||
|
b = [{"source": "X", "target": "Y", "name": "g2", "id": "g2"}]
|
||||||
|
assert hash_glossary_terms(a) == hash_glossary_terms(b)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# LRU backend
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestLRUBackend:
|
||||||
|
def test_get_missing_returns_none(self):
|
||||||
|
c = _LRUBackend()
|
||||||
|
assert c.get("nope") is None
|
||||||
|
assert c.misses == 1
|
||||||
|
|
||||||
|
def test_set_and_get(self):
|
||||||
|
c = _LRUBackend()
|
||||||
|
c.set("k", "v")
|
||||||
|
assert c.get("k") == "v"
|
||||||
|
assert c.hits == 1
|
||||||
|
|
||||||
|
def test_overwrite(self):
|
||||||
|
c = _LRUBackend()
|
||||||
|
c.set("k", "v1")
|
||||||
|
c.set("k", "v2")
|
||||||
|
assert c.get("k") == "v2"
|
||||||
|
assert len(c._cache) == 1
|
||||||
|
|
||||||
|
def test_lru_eviction(self):
|
||||||
|
c = _LRUBackend(maxsize=2)
|
||||||
|
c.set("a", "1")
|
||||||
|
c.set("b", "2")
|
||||||
|
c.set("c", "3") # Should evict "a" (least recently used)
|
||||||
|
assert c.get("a") is None
|
||||||
|
assert c.get("b") == "2"
|
||||||
|
assert c.get("c") == "3"
|
||||||
|
|
||||||
|
def test_get_promotes_to_mru(self):
|
||||||
|
c = _LRUBackend(maxsize=2)
|
||||||
|
c.set("a", "1")
|
||||||
|
c.set("b", "2")
|
||||||
|
c.get("a") # Promote "a"
|
||||||
|
c.set("c", "3") # Should evict "b" (now LRU)
|
||||||
|
assert c.get("a") == "1"
|
||||||
|
assert c.get("b") is None
|
||||||
|
assert c.get("c") == "3"
|
||||||
|
|
||||||
|
def test_clear(self):
|
||||||
|
c = _LRUBackend()
|
||||||
|
c.set("a", "1")
|
||||||
|
c.set("b", "2")
|
||||||
|
c.clear()
|
||||||
|
assert c.get("a") is None
|
||||||
|
# After clear, both counters are reset
|
||||||
|
assert c.hits == 0
|
||||||
|
# The get after clear is a miss (1)
|
||||||
|
assert c.misses == 1
|
||||||
|
|
||||||
|
def test_stats(self):
|
||||||
|
c = _LRUBackend(maxsize=10)
|
||||||
|
c.set("a", "1")
|
||||||
|
c.get("a") # hit
|
||||||
|
c.get("b") # miss
|
||||||
|
s = c.stats()
|
||||||
|
assert s["backend"] == "lru"
|
||||||
|
assert s["hits"] == 1
|
||||||
|
assert s["misses"] == 1
|
||||||
|
assert s["size"] == 1
|
||||||
|
assert s["maxsize"] == 10
|
||||||
|
|
||||||
|
def test_thread_safety(self):
|
||||||
|
c = _LRUBackend(maxsize=100)
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
def worker(i):
|
||||||
|
try:
|
||||||
|
for j in range(100):
|
||||||
|
c.set(f"k_{i}_{j}", f"v_{i}_{j}")
|
||||||
|
c.get(f"k_{i}_{j}")
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(e)
|
||||||
|
|
||||||
|
threads = [threading.Thread(target=worker, args=(i,)) for i in range(5)]
|
||||||
|
for t in threads:
|
||||||
|
t.start()
|
||||||
|
for t in threads:
|
||||||
|
t.join()
|
||||||
|
assert errors == []
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Redis backend — fallback behavior
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestRedisBackendFallback:
|
||||||
|
"""When Redis is unavailable, the backend should fall back to LRU."""
|
||||||
|
|
||||||
|
def test_redis_unavailable_falls_back_to_lru(self):
|
||||||
|
lru = _LRUBackend()
|
||||||
|
backend = _RedisBackend(
|
||||||
|
url="redis://invalid-host:9999/0",
|
||||||
|
lru_fallback=lru,
|
||||||
|
)
|
||||||
|
# First call probes the connection, fails, and falls back
|
||||||
|
assert backend.get("test_key") is None # miss in LRU
|
||||||
|
backend.set("test_key", "hello")
|
||||||
|
# Second get should hit the LRU
|
||||||
|
assert backend.get("test_key") == "hello"
|
||||||
|
# Errors should have been counted
|
||||||
|
assert backend.errors >= 1
|
||||||
|
# The backend should not have a client
|
||||||
|
assert backend._client is None
|
||||||
|
|
||||||
|
def test_redis_set_falls_back_to_lru(self):
|
||||||
|
lru = _LRUBackend()
|
||||||
|
backend = _RedisBackend(
|
||||||
|
url="redis://invalid-host:9999/0",
|
||||||
|
lru_fallback=lru,
|
||||||
|
)
|
||||||
|
backend.set("k", "v")
|
||||||
|
# The LRU should have it
|
||||||
|
assert lru.get("k") == "v"
|
||||||
|
|
||||||
|
def test_stats_includes_lru(self):
|
||||||
|
lru = _LRUBackend()
|
||||||
|
backend = _RedisBackend(
|
||||||
|
url="redis://invalid-host:9999/0",
|
||||||
|
lru_fallback=lru,
|
||||||
|
)
|
||||||
|
backend.set("k", "v")
|
||||||
|
stats = backend.stats()
|
||||||
|
assert stats["backend"] == "redis"
|
||||||
|
assert stats["available"] is False
|
||||||
|
assert stats["errors"] >= 1
|
||||||
|
assert "lru_size" in stats
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Redis backend — happy path with a mock
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestRedisBackendWithMock:
|
||||||
|
"""When Redis is reachable, the backend should use it."""
|
||||||
|
|
||||||
|
def _build_backend_with_mock_client(self, client):
|
||||||
|
"""Build a RedisBackend with a pre-set mock client."""
|
||||||
|
lru = _LRUBackend()
|
||||||
|
backend = _RedisBackend(
|
||||||
|
url="redis://fake:6379/0",
|
||||||
|
lru_fallback=lru,
|
||||||
|
)
|
||||||
|
backend._client = client
|
||||||
|
backend._available = True
|
||||||
|
return backend, lru
|
||||||
|
|
||||||
|
def test_get_uses_redis(self):
|
||||||
|
client = MagicMock()
|
||||||
|
client.get.return_value = "cached_value"
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
result = backend.get("k")
|
||||||
|
assert result == "cached_value"
|
||||||
|
client.get.assert_called_once()
|
||||||
|
# Should also have been written to LRU
|
||||||
|
assert lru.get("k") == "cached_value"
|
||||||
|
|
||||||
|
def test_get_miss_redis(self):
|
||||||
|
client = MagicMock()
|
||||||
|
client.get.return_value = None
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
result = backend.get("k")
|
||||||
|
assert result is None
|
||||||
|
assert backend.misses == 1
|
||||||
|
|
||||||
|
def test_set_uses_redis(self):
|
||||||
|
client = MagicMock()
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
backend.set("k", "v", ttl=60)
|
||||||
|
# Should have called Redis SET
|
||||||
|
client.set.assert_called_once()
|
||||||
|
args = client.set.call_args[0]
|
||||||
|
assert args[1] == "v" # value
|
||||||
|
# The TTL is passed as a keyword arg (ex=ttl)
|
||||||
|
assert client.set.call_args[1]["ex"] == 60
|
||||||
|
|
||||||
|
def test_set_uses_default_ttl(self):
|
||||||
|
client = MagicMock()
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
backend.set("k", "v")
|
||||||
|
# Default TTL = 86400 (24h) is passed as ex=...
|
||||||
|
assert client.set.call_args[1]["ex"] == 86400
|
||||||
|
|
||||||
|
def test_redis_error_during_get_resets_client(self):
|
||||||
|
client = MagicMock()
|
||||||
|
client.get.side_effect = RuntimeError("redis down")
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
result = backend.get("k")
|
||||||
|
# Falls back to LRU
|
||||||
|
assert result is None
|
||||||
|
# Client should be reset so we re-probe next time
|
||||||
|
assert backend._client is None
|
||||||
|
assert backend._available is False
|
||||||
|
assert backend.errors == 1
|
||||||
|
|
||||||
|
def test_redis_error_during_set_resets_client(self):
|
||||||
|
client = MagicMock()
|
||||||
|
client.set.side_effect = RuntimeError("redis down")
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
backend.set("k", "v")
|
||||||
|
# Should have fallen back to LRU
|
||||||
|
assert lru.get("k") == "v"
|
||||||
|
assert backend._client is None
|
||||||
|
assert backend.errors == 1
|
||||||
|
|
||||||
|
def test_namespace_prefix(self):
|
||||||
|
client = MagicMock()
|
||||||
|
client.get.return_value = None
|
||||||
|
backend, lru = self._build_backend_with_mock_client(client)
|
||||||
|
|
||||||
|
backend.get("mykey")
|
||||||
|
called_key = client.get.call_args[0][0]
|
||||||
|
assert called_key.startswith("translate_cache:")
|
||||||
|
assert called_key.endswith(":mykey")
|
||||||
|
|
||||||
|
def test_custom_namespace(self):
|
||||||
|
client = MagicMock()
|
||||||
|
client.get.return_value = None
|
||||||
|
lru = _LRUBackend()
|
||||||
|
backend = _RedisBackend(
|
||||||
|
url="redis://fake:6379/0",
|
||||||
|
namespace="custom_ns",
|
||||||
|
lru_fallback=lru,
|
||||||
|
)
|
||||||
|
backend._client = client
|
||||||
|
backend._available = True
|
||||||
|
|
||||||
|
backend.get("mykey")
|
||||||
|
called_key = client.get.call_args[0][0]
|
||||||
|
assert called_key.startswith("custom_ns:")
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# TranslationCache — high-level API
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestTranslationCache:
|
||||||
|
def test_default_backend_is_lru(self, monkeypatch):
|
||||||
|
"""No REDIS_CACHE_ENABLED → LRU backend."""
|
||||||
|
monkeypatch.delenv("REDIS_CACHE_ENABLED", raising=False)
|
||||||
|
monkeypatch.delenv("REDIS_URL", raising=False)
|
||||||
|
cache = TranslationCache(backend="auto")
|
||||||
|
assert cache.backend_name == "lru"
|
||||||
|
|
||||||
|
def test_explicit_lru_backend(self, monkeypatch):
|
||||||
|
monkeypatch.setenv("REDIS_CACHE_ENABLED", "true")
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
assert cache.backend_name == "lru"
|
||||||
|
|
||||||
|
def test_get_and_set(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "Bonjour")
|
||||||
|
assert cache.get("Hello", "fr", "en", "google") == "Bonjour"
|
||||||
|
|
||||||
|
def test_get_miss(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
assert cache.get("Nope", "fr", "en", "google") is None
|
||||||
|
|
||||||
|
def test_user_isolation(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "Bonjour", user_id="alice")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "Hola", user_id="bob")
|
||||||
|
assert cache.get("Hello", "fr", "en", "google", user_id="alice") == "Bonjour"
|
||||||
|
assert cache.get("Hello", "fr", "en", "google", user_id="bob") == "Hola"
|
||||||
|
|
||||||
|
def test_prompt_isolation(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "V1",
|
||||||
|
custom_prompt_hash="p1")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "V2",
|
||||||
|
custom_prompt_hash="p2")
|
||||||
|
assert cache.get("Hello", "fr", "en", "google", custom_prompt_hash="p1") == "V1"
|
||||||
|
assert cache.get("Hello", "fr", "en", "google", custom_prompt_hash="p2") == "V2"
|
||||||
|
|
||||||
|
def test_glossary_isolation(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "V1",
|
||||||
|
glossary_hash="g1")
|
||||||
|
cache.set("Hello", "fr", "en", "google", "V2",
|
||||||
|
glossary_hash="g2")
|
||||||
|
assert cache.get("Hello", "fr", "en", "google", glossary_hash="g1") == "V1"
|
||||||
|
assert cache.get("Hello", "fr", "en", "google", glossary_hash="g2") == "V2"
|
||||||
|
|
||||||
|
def test_clear(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
cache.set("k", "fr", "en", "google", "v")
|
||||||
|
cache.clear()
|
||||||
|
assert cache.get("k", "fr", "en", "google") is None
|
||||||
|
|
||||||
|
def test_stats_returns_dict(self):
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
cache.set("k", "fr", "en", "google", "v")
|
||||||
|
cache.get("k", "fr", "en", "google")
|
||||||
|
s = cache.stats()
|
||||||
|
assert isinstance(s, dict)
|
||||||
|
assert s["backend"] == "lru"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Singleton accessor
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestGetCache:
|
||||||
|
def test_returns_singleton(self, monkeypatch):
|
||||||
|
reset_cache_for_tests()
|
||||||
|
c1 = get_cache()
|
||||||
|
c2 = get_cache()
|
||||||
|
assert c1 is c2
|
||||||
|
|
||||||
|
def test_reset_for_tests(self, monkeypatch):
|
||||||
|
reset_cache_for_tests()
|
||||||
|
c1 = get_cache()
|
||||||
|
reset_cache_for_tests()
|
||||||
|
c2 = get_cache()
|
||||||
|
assert c1 is not c2
|
||||||
|
|
||||||
|
def test_singleton_thread_safe(self, monkeypatch):
|
||||||
|
reset_cache_for_tests()
|
||||||
|
caches = []
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
def worker():
|
||||||
|
try:
|
||||||
|
caches.append(get_cache())
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(e)
|
||||||
|
|
||||||
|
threads = [threading.Thread(target=worker) for _ in range(10)]
|
||||||
|
for t in threads:
|
||||||
|
t.start()
|
||||||
|
for t in threads:
|
||||||
|
t.join()
|
||||||
|
assert errors == []
|
||||||
|
assert all(c is caches[0] for c in caches)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# End-to-end: integrate with hash_prompt / hash_glossary
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestE2EHashIntegration:
|
||||||
|
def test_real_prompt_and_glossary(self):
|
||||||
|
"""A realistic scenario: same text, different prompt/glossary
|
||||||
|
should give different cache keys and thus different entries."""
|
||||||
|
cache = TranslationCache(backend="lru")
|
||||||
|
|
||||||
|
prompt_a = "Translate this as a formal document."
|
||||||
|
prompt_b = "Translate this as a casual chat message."
|
||||||
|
glossary_a = [{"source": "API", "target": "API"}]
|
||||||
|
glossary_b = [{"source": "API", "target": "interface"}]
|
||||||
|
|
||||||
|
cache.set(
|
||||||
|
"Hello", "fr", "en", "google",
|
||||||
|
"Bonjour (formel)",
|
||||||
|
custom_prompt_hash=hash_prompt(prompt_a),
|
||||||
|
glossary_hash=hash_glossary_terms(glossary_a),
|
||||||
|
)
|
||||||
|
cache.set(
|
||||||
|
"Hello", "fr", "en", "google",
|
||||||
|
"Salut (décontracté)",
|
||||||
|
custom_prompt_hash=hash_prompt(prompt_b),
|
||||||
|
glossary_hash=hash_glossary_terms(glossary_b),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Reading with the same prompt/glossary should give the right entry
|
||||||
|
assert cache.get(
|
||||||
|
"Hello", "fr", "en", "google",
|
||||||
|
custom_prompt_hash=hash_prompt(prompt_a),
|
||||||
|
glossary_hash=hash_glossary_terms(glossary_a),
|
||||||
|
) == "Bonjour (formel)"
|
||||||
|
assert cache.get(
|
||||||
|
"Hello", "fr", "en", "google",
|
||||||
|
custom_prompt_hash=hash_prompt(prompt_b),
|
||||||
|
glossary_hash=hash_glossary_terms(glossary_b),
|
||||||
|
) == "Salut (décontracté)"
|
||||||
531
tests/test_translators/test_b1_format_fixes.py
Normal file
531
tests/test_translators/test_b1_format_fixes.py
Normal file
@@ -0,0 +1,531 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track B1 — Format preservation quick wins (Word + Excel).
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
W1 — Word hyperlink double-collect fix
|
||||||
|
W2 — Word footnotes import fix
|
||||||
|
W4 — Word chart matching by XPath (no string equality)
|
||||||
|
E1 — (deferred — rich text in cells is risky)
|
||||||
|
E2 — Excel cell comments
|
||||||
|
E3 — Excel cell hyperlinks (URL preserved, label translated)
|
||||||
|
E4 — Excel chart matching by XPath
|
||||||
|
|
||||||
|
These tests are intentionally narrow: they validate that a specific
|
||||||
|
bug is fixed. They do NOT cover the full translator surface (existing
|
||||||
|
tests in test_word_translator.py and test_excel_translator.py handle that).
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Tuple, Callable, Dict
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from docx import Document
|
||||||
|
from docx.oxml import OxmlElement
|
||||||
|
from docx.oxml.ns import qn
|
||||||
|
from docx.opc.constants import RELATIONSHIP_TYPE as RT
|
||||||
|
from openpyxl import Workbook
|
||||||
|
from openpyxl.comments import Comment
|
||||||
|
from openpyxl.worksheet.hyperlink import Hyperlink
|
||||||
|
|
||||||
|
from translators.word_translator import WordTranslator
|
||||||
|
from translators.excel_translator import ExcelTranslator
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------- Helpers ----------------
|
||||||
|
|
||||||
|
def _add_hyperlink_to_paragraph(paragraph, url: str, text: str) -> None:
|
||||||
|
"""Add a hyperlink to a python-docx paragraph (low-level API)."""
|
||||||
|
part = paragraph.part
|
||||||
|
r_id = part.relate_to(url, RT.HYPERLINK, is_external=True)
|
||||||
|
hyperlink = OxmlElement("w:hyperlink")
|
||||||
|
hyperlink.set(qn("r:id"), r_id)
|
||||||
|
|
||||||
|
new_run = OxmlElement("w:r")
|
||||||
|
rPr = OxmlElement("w:rPr")
|
||||||
|
# Mark as hyperlink style
|
||||||
|
rStyle = OxmlElement("w:rStyle")
|
||||||
|
rStyle.set(qn("w:val"), "Hyperlink")
|
||||||
|
rPr.append(rStyle)
|
||||||
|
new_run.append(rPr)
|
||||||
|
|
||||||
|
new_text = OxmlElement("w:t")
|
||||||
|
new_text.text = text
|
||||||
|
new_text.set(qn("xml:space"), "preserve")
|
||||||
|
new_run.append(new_text)
|
||||||
|
|
||||||
|
hyperlink.append(new_run)
|
||||||
|
paragraph._p.append(hyperlink)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------- Mock provider ----------------
|
||||||
|
|
||||||
|
class MockProvider:
|
||||||
|
"""Mock translation provider: maps source text to a TR_<text> translation."""
|
||||||
|
|
||||||
|
def __init__(self, translations: Dict[str, str] = None):
|
||||||
|
self._translations = translations or {}
|
||||||
|
self.calls: List[str] = []
|
||||||
|
|
||||||
|
def translate_batch(
|
||||||
|
self, texts: List[str], target_language: str = "fr", source_language: str = "auto"
|
||||||
|
) -> List[str]:
|
||||||
|
out = []
|
||||||
|
for t in texts:
|
||||||
|
self.calls.append(t)
|
||||||
|
if t in self._translations:
|
||||||
|
out.append(self._translations[t])
|
||||||
|
elif t.startswith("TR_"):
|
||||||
|
out.append(t)
|
||||||
|
else:
|
||||||
|
# Default: a no-op translation
|
||||||
|
out.append(t)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def get_name(self):
|
||||||
|
return "mock"
|
||||||
|
|
||||||
|
def is_available(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _make_tmp(suffix: str) -> Path:
|
||||||
|
fd, name = tempfile.mkstemp(suffix=suffix)
|
||||||
|
os.close(fd)
|
||||||
|
return Path(name)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# W1 — Word hyperlink double-collect
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestWordHyperlinkNoDoubleCollect:
|
||||||
|
"""Bug fix: a run inside a <w:hyperlink> used to be collected twice
|
||||||
|
(once via paragraph.runs, once via the hyperlink iter loop), wasting
|
||||||
|
API calls and potentially causing double-apply on save."""
|
||||||
|
|
||||||
|
def test_hyperlink_run_translated_once(self, tmp_path):
|
||||||
|
provider = MockProvider({"Click here": "TR_Click here"})
|
||||||
|
translator = WordTranslator(provider=provider)
|
||||||
|
|
||||||
|
doc = Document()
|
||||||
|
# Add a hyperlink to a paragraph (low-level API)
|
||||||
|
p = doc.add_paragraph("See ")
|
||||||
|
_add_hyperlink_to_paragraph(p, "https://example.com", "Click here")
|
||||||
|
doc.add_paragraph(" for details")
|
||||||
|
|
||||||
|
input_file = tmp_path / "input.docx"
|
||||||
|
output_file = tmp_path / "output.docx"
|
||||||
|
doc.save(input_file)
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# "Click here" should appear in the provider calls at most once.
|
||||||
|
click_calls = [c for c in provider.calls if c == "Click here"]
|
||||||
|
assert len(click_calls) == 1, (
|
||||||
|
f"Hyperlink run was collected {len(click_calls)} times, expected 1. "
|
||||||
|
f"All calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_hyperlink_url_preserved(self, tmp_path):
|
||||||
|
"""The hyperlink target URL should be preserved (not translated)."""
|
||||||
|
provider = MockProvider({"Click here": "Cliquez ici"})
|
||||||
|
translator = WordTranslator(provider=provider)
|
||||||
|
|
||||||
|
doc = Document()
|
||||||
|
p = doc.add_paragraph("Visit ")
|
||||||
|
_add_hyperlink_to_paragraph(p, "https://example.com", "Click here")
|
||||||
|
|
||||||
|
input_file = tmp_path / "input.docx"
|
||||||
|
output_file = tmp_path / "output.docx"
|
||||||
|
doc.save(input_file)
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# The visible text should be translated
|
||||||
|
output_doc = Document(output_file)
|
||||||
|
assert "Cliquez ici" in str(output_doc._body._element.xml)
|
||||||
|
|
||||||
|
# The URL itself is stored in the .rels file, not the body.
|
||||||
|
# Check the relationships for the URL.
|
||||||
|
with zipfile.ZipFile(output_file, "r") as zf:
|
||||||
|
rels = zf.read("word/_rels/document.xml.rels").decode("utf-8")
|
||||||
|
assert "https://example.com" in rels, (
|
||||||
|
f"Hyperlink URL not found in rels file. rels:\n{rels}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_multiple_hyperlinks(self, tmp_path):
|
||||||
|
"""Multiple distinct hyperlinks should each be translated once."""
|
||||||
|
provider = MockProvider({
|
||||||
|
"First link": "Premier lien",
|
||||||
|
"Second link": "Second lien",
|
||||||
|
})
|
||||||
|
translator = WordTranslator(provider=provider)
|
||||||
|
|
||||||
|
doc = Document()
|
||||||
|
p1 = doc.add_paragraph()
|
||||||
|
_add_hyperlink_to_paragraph(p1, "https://a.com", "First link")
|
||||||
|
p2 = doc.add_paragraph()
|
||||||
|
_add_hyperlink_to_paragraph(p2, "https://b.com", "Second link")
|
||||||
|
|
||||||
|
input_file = tmp_path / "input.docx"
|
||||||
|
output_file = tmp_path / "output.docx"
|
||||||
|
doc.save(input_file)
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# Each hyperlink text should appear at most once
|
||||||
|
assert provider.calls.count("First link") == 1
|
||||||
|
assert provider.calls.count("Second link") == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# W2 — Word footnotes import
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestWordFootnotesCollected:
|
||||||
|
"""Bug fix: `_collect_from_footnotes` used `document.part.package`
|
||||||
|
which doesn't exist in python-docx 1.x — so footnotes were never
|
||||||
|
translated. Now it uses `document.part.related_parts` to find the
|
||||||
|
footnotes part by content type."""
|
||||||
|
|
||||||
|
def test_footnotes_translated(self, tmp_path):
|
||||||
|
provider = MockProvider({
|
||||||
|
"Footnote text": "Texte de note",
|
||||||
|
"Body text": "Texte du corps",
|
||||||
|
})
|
||||||
|
translator = WordTranslator(provider=provider)
|
||||||
|
|
||||||
|
# Build a document with a footnote
|
||||||
|
doc = Document()
|
||||||
|
p = doc.add_paragraph("Body text")
|
||||||
|
# python-docx doesn't have a high-level API for footnotes, so we
|
||||||
|
# inject the XML directly
|
||||||
|
from docx.oxml.ns import qn
|
||||||
|
from docx.oxml import OxmlElement
|
||||||
|
|
||||||
|
# Create a footnote reference in the paragraph
|
||||||
|
run = p.add_run()
|
||||||
|
footnote_ref = OxmlElement("w:footnoteReference")
|
||||||
|
footnote_ref.set(qn("w:id"), "1")
|
||||||
|
run._r.append(footnote_ref)
|
||||||
|
|
||||||
|
input_file = tmp_path / "input.docx"
|
||||||
|
output_file = tmp_path / "output.docx"
|
||||||
|
doc.save(input_file)
|
||||||
|
|
||||||
|
# Manually inject a footnotes.xml part into the docx
|
||||||
|
footnotes_xml = b"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<w:footnotes xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">
|
||||||
|
<w:footnote w:id="1">
|
||||||
|
<w:p>
|
||||||
|
<w:r>
|
||||||
|
<w:t>Footnote text</w:t>
|
||||||
|
</w:r>
|
||||||
|
</w:p>
|
||||||
|
</w:footnote>
|
||||||
|
</w:footnotes>"""
|
||||||
|
|
||||||
|
_inject_xml_part(
|
||||||
|
input_file, "word/footnotes.xml", footnotes_xml,
|
||||||
|
content_type="application/vnd.openxmlformats-officedocument.wordprocessingml.footnotes+xml",
|
||||||
|
)
|
||||||
|
_add_relationship(input_file, "rIdFootnotes",
|
||||||
|
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/footnotes",
|
||||||
|
"footnotes.xml")
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# Both body and footnote text should be in the provider calls
|
||||||
|
assert "Body text" in provider.calls
|
||||||
|
assert "Footnote text" in provider.calls, (
|
||||||
|
f"Footnote text was not collected. All calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_no_footnotes_part_no_error(self, tmp_path):
|
||||||
|
"""A document without footnotes should not raise an error."""
|
||||||
|
provider = MockProvider({"Hello": "Bonjour"})
|
||||||
|
translator = WordTranslator(provider=provider)
|
||||||
|
|
||||||
|
doc = Document()
|
||||||
|
doc.add_paragraph("Hello")
|
||||||
|
|
||||||
|
input_file = tmp_path / "input.docx"
|
||||||
|
output_file = tmp_path / "output.docx"
|
||||||
|
doc.save(input_file)
|
||||||
|
|
||||||
|
# Should not raise even though the doc has no footnotes part
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
# Sanity check
|
||||||
|
output_doc = Document(output_file)
|
||||||
|
assert output_doc.paragraphs[0].text == "Bonjour"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# W4 — Word chart matching by XPath
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestWordChartXPathMatching:
|
||||||
|
"""Bug fix: chart translation matched by string equality, so if the
|
||||||
|
same text appeared in multiple charts (e.g. two "Revenue" series),
|
||||||
|
only the first was translated. Now we use the element path collected
|
||||||
|
at gather time."""
|
||||||
|
|
||||||
|
def test_chart_in_zip_gets_translated(self, tmp_path):
|
||||||
|
"""A docx with a chart containing text should have the chart text
|
||||||
|
translated via the chart apply path."""
|
||||||
|
# The full chart-pipeline test is in the original test suite;
|
||||||
|
# here we just verify the new path-based code doesn't crash.
|
||||||
|
provider = MockProvider({"Chart title": "Titre du graphique"})
|
||||||
|
translator = WordTranslator(provider=provider)
|
||||||
|
|
||||||
|
# Build a minimal docx that has a chart XML part
|
||||||
|
doc = Document()
|
||||||
|
doc.add_paragraph("Chart title")
|
||||||
|
input_file = tmp_path / "input.docx"
|
||||||
|
output_file = tmp_path / "output.docx"
|
||||||
|
doc.save(input_file)
|
||||||
|
|
||||||
|
# Inject a chart XML part so the collect path runs
|
||||||
|
chart_xml = b"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<c:chartSpace xmlns:c="http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<c:chart>
|
||||||
|
<c:title>
|
||||||
|
<c:tx>
|
||||||
|
<c:rich>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:t>Chart title</a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</c:rich>
|
||||||
|
</c:tx>
|
||||||
|
</c:title>
|
||||||
|
</c:chart>
|
||||||
|
</c:chartSpace>"""
|
||||||
|
_inject_xml_part(input_file, "word/charts/chart1.xml", chart_xml)
|
||||||
|
_add_relationship(input_file, "rIdChart1",
|
||||||
|
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/chart",
|
||||||
|
"charts/chart1.xml")
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# The chart text should be among the collected texts
|
||||||
|
assert "Chart title" in provider.calls
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# E2 — Excel cell comments
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestExcelCellComments:
|
||||||
|
"""New feature: cell comments are now collected and translated."""
|
||||||
|
|
||||||
|
def test_cell_comment_translated(self, tmp_path):
|
||||||
|
provider = MockProvider({"Old comment": "Ancien commentaire"})
|
||||||
|
translator = ExcelTranslator(provider=provider)
|
||||||
|
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = "Cell value"
|
||||||
|
ws["A1"].comment = Comment("Old comment", "Author")
|
||||||
|
input_file = tmp_path / "input.xlsx"
|
||||||
|
output_file = tmp_path / "output.xlsx"
|
||||||
|
wb.save(input_file)
|
||||||
|
wb.close()
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# Comment text should be in the provider calls
|
||||||
|
assert "Old comment" in provider.calls
|
||||||
|
|
||||||
|
# The output should have the translated comment
|
||||||
|
from openpyxl import load_workbook
|
||||||
|
wb_out = load_workbook(output_file)
|
||||||
|
assert wb_out.active["A1"].comment is not None
|
||||||
|
# The translation dict maps "Old comment" -> "Ancien commentaire"
|
||||||
|
assert wb_out.active["A1"].comment.text == "Ancien commentaire"
|
||||||
|
wb_out.close()
|
||||||
|
|
||||||
|
def test_empty_comment_skipped(self, tmp_path):
|
||||||
|
provider = MockProvider()
|
||||||
|
translator = ExcelTranslator(provider=provider)
|
||||||
|
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = "Value"
|
||||||
|
input_file = tmp_path / "input.xlsx"
|
||||||
|
output_file = tmp_path / "output.xlsx"
|
||||||
|
wb.save(input_file)
|
||||||
|
wb.close()
|
||||||
|
|
||||||
|
# Should not raise
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# E3 — Excel cell hyperlinks
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestExcelCellHyperlinks:
|
||||||
|
"""New feature: cell hyperlink DISPLAY labels are translated; the
|
||||||
|
underlying URL is preserved."""
|
||||||
|
|
||||||
|
def test_hyperlink_label_translated(self, tmp_path):
|
||||||
|
provider = MockProvider({"Click here": "Cliquez ici"})
|
||||||
|
translator = ExcelTranslator(provider=provider)
|
||||||
|
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = "Click here"
|
||||||
|
ws["A1"].hyperlink = Hyperlink(ref="A1", target="https://example.com")
|
||||||
|
input_file = tmp_path / "input.xlsx"
|
||||||
|
output_file = tmp_path / "output.xlsx"
|
||||||
|
wb.save(input_file)
|
||||||
|
wb.close()
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# The display label should be translated
|
||||||
|
from openpyxl import load_workbook
|
||||||
|
wb_out = load_workbook(output_file)
|
||||||
|
cell = wb_out.active["A1"]
|
||||||
|
# The value (display label) should be translated
|
||||||
|
assert cell.value == "TR_Click here" or cell.value == "Cliquez ici"
|
||||||
|
# The URL should be preserved
|
||||||
|
assert cell.hyperlink is not None
|
||||||
|
assert cell.hyperlink.target == "https://example.com"
|
||||||
|
wb_out.close()
|
||||||
|
|
||||||
|
def test_hyperlink_with_url_as_value_not_double_translated(self, tmp_path):
|
||||||
|
"""If the cell value IS the URL, don't try to translate it again."""
|
||||||
|
provider = MockProvider()
|
||||||
|
translator = ExcelTranslator(provider=provider)
|
||||||
|
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = "https://example.com"
|
||||||
|
ws["A1"].hyperlink = Hyperlink(ref="A1", target="https://example.com")
|
||||||
|
input_file = tmp_path / "input.xlsx"
|
||||||
|
output_file = tmp_path / "output.xlsx"
|
||||||
|
wb.save(input_file)
|
||||||
|
wb.close()
|
||||||
|
|
||||||
|
# Should not raise
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# E4 — Excel chart matching by XPath
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestExcelChartXPathMatching:
|
||||||
|
"""Bug fix: same as W4 but for Excel. Chart elements are now matched
|
||||||
|
by element path, not string equality."""
|
||||||
|
|
||||||
|
def test_chart_text_collected(self, tmp_path):
|
||||||
|
provider = MockProvider({"Chart title": "Titre graphique"})
|
||||||
|
translator = ExcelTranslator(provider=provider)
|
||||||
|
|
||||||
|
wb = Workbook()
|
||||||
|
ws = wb.active
|
||||||
|
ws["A1"] = "Cell value"
|
||||||
|
input_file = tmp_path / "input.xlsx"
|
||||||
|
output_file = tmp_path / "output.xlsx"
|
||||||
|
wb.save(input_file)
|
||||||
|
wb.close()
|
||||||
|
|
||||||
|
# Inject a chart XML part
|
||||||
|
chart_xml = b"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<c:chartSpace xmlns:c="http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<c:chart>
|
||||||
|
<c:title>
|
||||||
|
<c:tx>
|
||||||
|
<c:rich>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:t>Chart title</a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</c:rich>
|
||||||
|
</c:tx>
|
||||||
|
</c:title>
|
||||||
|
</c:chart>
|
||||||
|
</c:chartSpace>"""
|
||||||
|
_inject_xml_part(input_file, "xl/charts/chart1.xml", chart_xml)
|
||||||
|
_add_relationship(input_file, "chart1",
|
||||||
|
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/chart",
|
||||||
|
"charts/chart1.xml")
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# Chart text should be in the provider calls
|
||||||
|
assert "Chart title" in provider.calls
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Helper functions
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def _inject_xml_part(docx_path: Path, part_name: str, content: bytes,
|
||||||
|
content_type: str = None):
|
||||||
|
"""Inject a new XML part into an existing .docx or .xlsx file.
|
||||||
|
|
||||||
|
If content_type is provided, also add a <Override> entry in
|
||||||
|
[Content_Types].xml so the part is correctly identified. Without
|
||||||
|
this, python-docx (and most Office parsers) will see the part as
|
||||||
|
generic application/xml and won't find it via content_type lookups.
|
||||||
|
"""
|
||||||
|
buf_bytes = docx_path.read_bytes()
|
||||||
|
tmp_out = docx_path.with_suffix(".tmp")
|
||||||
|
with zipfile.ZipFile(docx_path, "r") as zin, \
|
||||||
|
zipfile.ZipFile(tmp_out, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||||
|
existing = {item: zin.read(item) for item in zin.namelist()}
|
||||||
|
|
||||||
|
# Add the part
|
||||||
|
existing[part_name] = content
|
||||||
|
|
||||||
|
# Register the content type override if requested
|
||||||
|
if content_type:
|
||||||
|
ct_path = "[Content_Types].xml"
|
||||||
|
if ct_path in existing:
|
||||||
|
ct_content = existing[ct_path].decode("utf-8")
|
||||||
|
# Insert a new <Override> before </Types>
|
||||||
|
override = f'<Override PartName="/{part_name}" ContentType="{content_type}"/>'
|
||||||
|
ct_content = ct_content.replace(
|
||||||
|
"</Types>", f"{override}</Types>"
|
||||||
|
)
|
||||||
|
existing[ct_path] = ct_content.encode("utf-8")
|
||||||
|
|
||||||
|
for item, data in existing.items():
|
||||||
|
zout.writestr(item, data)
|
||||||
|
|
||||||
|
tmp_out.replace(docx_path)
|
||||||
|
|
||||||
|
|
||||||
|
def _add_relationship(docx_path: Path, rel_id: str, rel_type: str, target: str):
|
||||||
|
"""Add a relationship to word/_rels/document.xml.rels in a .docx file.
|
||||||
|
The footnote/chart relationships for Word live in this file (they're
|
||||||
|
document-level relationships, not part-level)."""
|
||||||
|
tmp_out = docx_path.with_suffix(".tmp")
|
||||||
|
rels_path = "word/_rels/document.xml.rels"
|
||||||
|
|
||||||
|
with zipfile.ZipFile(docx_path, "r") as zin, \
|
||||||
|
zipfile.ZipFile(tmp_out, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||||
|
existing = {item: zin.read(item) for item in zin.namelist()}
|
||||||
|
|
||||||
|
if rels_path in existing:
|
||||||
|
rels_content = existing[rels_path].decode("utf-8")
|
||||||
|
new_rel_xml = f'<Relationship Id="{rel_id}" Type="{rel_type}" Target="{target}"/>'
|
||||||
|
rels_content = rels_content.replace(
|
||||||
|
"</Relationships>", f"{new_rel_xml}</Relationships>"
|
||||||
|
)
|
||||||
|
existing[rels_path] = rels_content.encode("utf-8")
|
||||||
|
|
||||||
|
for item, data in existing.items():
|
||||||
|
zout.writestr(item, data)
|
||||||
|
|
||||||
|
tmp_out.replace(docx_path)
|
||||||
623
tests/test_translators/test_b2_pptx_fixes.py
Normal file
623
tests/test_translators/test_b2_pptx_fixes.py
Normal file
@@ -0,0 +1,623 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track B2 — PowerPoint format preservation fixes.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
B2.1 — Placeholder filter (DATE=16, SLIDE_NUMBER=13, FOOTER=15)
|
||||||
|
PowerPoint auto-fills these at render time, so translating them
|
||||||
|
is pointless and just wastes API calls.
|
||||||
|
B2.2 — SmartArt diagram text collection (ppt/diagrams/data*.xml)
|
||||||
|
B2.3 — SmartArt diagram translation applied via ZIP rewrite
|
||||||
|
B2.4 — Chart text matched by element path (not string equality)
|
||||||
|
B2.5 — Whitespace preservation in chart apply
|
||||||
|
|
||||||
|
These tests are intentionally narrow: they validate that a specific
|
||||||
|
bug is fixed. Existing tests in test_pptx_translator.py handle the
|
||||||
|
broader surface.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Dict
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pptx import Presentation
|
||||||
|
from pptx.util import Inches, Pt
|
||||||
|
from pptx.enum.shapes import MSO_SHAPE_TYPE
|
||||||
|
from lxml import etree
|
||||||
|
|
||||||
|
from translators.pptx_translator import PowerPointTranslator, pptx_translator
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------- Mock provider ----------------
|
||||||
|
|
||||||
|
class MockProvider:
|
||||||
|
"""Mock translation provider: returns 'TR_<text>' unless a mapping
|
||||||
|
is provided."""
|
||||||
|
|
||||||
|
def __init__(self, translations: Dict[str, str] = None):
|
||||||
|
self._translations = translations or {}
|
||||||
|
self.calls: List[str] = []
|
||||||
|
|
||||||
|
def translate_batch(
|
||||||
|
self,
|
||||||
|
texts: List[str],
|
||||||
|
target_language: str = "fr",
|
||||||
|
source_language: str = "auto",
|
||||||
|
) -> List[str]:
|
||||||
|
out = []
|
||||||
|
for t in texts:
|
||||||
|
self.calls.append(t)
|
||||||
|
if t in self._translations:
|
||||||
|
out.append(self._translations[t])
|
||||||
|
elif t.startswith("TR_"):
|
||||||
|
out.append(t)
|
||||||
|
else:
|
||||||
|
out.append(t)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def get_name(self):
|
||||||
|
return "mock"
|
||||||
|
|
||||||
|
def is_available(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------- Helpers ----------------
|
||||||
|
|
||||||
|
_NS_A = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
||||||
|
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
_NS_P = "http://schemas.openxmlformats.org/presentationml/2006/main"
|
||||||
|
|
||||||
|
|
||||||
|
def _add_textbox_with_text(slide, text: str):
|
||||||
|
tb = slide.shapes.add_textbox(Inches(1), Inches(1), Inches(3), Inches(1))
|
||||||
|
tf = tb.text_frame
|
||||||
|
tf.text = text
|
||||||
|
return tb
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_diagram_part(pptx_path: Path, diagram_filename: str, diagram_xml: bytes):
|
||||||
|
"""Inject a SmartArt diagram XML part into a .pptx file."""
|
||||||
|
tmp_out = pptx_path.with_suffix(".tmp")
|
||||||
|
with zipfile.ZipFile(pptx_path, "r") as zin, \
|
||||||
|
zipfile.ZipFile(tmp_out, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||||
|
existing = {item: zin.read(item) for item in zin.namelist()}
|
||||||
|
|
||||||
|
# Add the diagram data part
|
||||||
|
existing[diagram_filename] = diagram_xml
|
||||||
|
|
||||||
|
# Register in [Content_Types].xml
|
||||||
|
ct_path = "[Content_Types].xml"
|
||||||
|
if ct_path in existing:
|
||||||
|
ct_content = existing[ct_path].decode("utf-8")
|
||||||
|
override = (
|
||||||
|
f'<Override PartName="/{diagram_filename}" '
|
||||||
|
f'ContentType="application/vnd.openxmlformats-officedocument.'
|
||||||
|
f'drawingml+xml+diagram"/>'
|
||||||
|
)
|
||||||
|
if override not in ct_content:
|
||||||
|
ct_content = ct_content.replace("</Types>", f"{override}</Types>")
|
||||||
|
existing[ct_path] = ct_content.encode("utf-8")
|
||||||
|
|
||||||
|
for item, data in existing.items():
|
||||||
|
zout.writestr(item, data)
|
||||||
|
|
||||||
|
tmp_out.replace(pptx_path)
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_chart_part(pptx_path: Path, chart_filename: str, chart_xml: bytes):
|
||||||
|
"""Inject a chart XML part into a .pptx file."""
|
||||||
|
tmp_out = pptx_path.with_suffix(".tmp")
|
||||||
|
with zipfile.ZipFile(pptx_path, "r") as zin, \
|
||||||
|
zipfile.ZipFile(tmp_out, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||||
|
existing = {item: zin.read(item) for item in zin.namelist()}
|
||||||
|
|
||||||
|
existing[chart_filename] = chart_xml
|
||||||
|
|
||||||
|
ct_path = "[Content_Types].xml"
|
||||||
|
if ct_path in existing:
|
||||||
|
ct_content = existing[ct_path].decode("utf-8")
|
||||||
|
override = (
|
||||||
|
f'<Override PartName="/{chart_filename}" '
|
||||||
|
f'ContentType="application/vnd.openxmlformats-officedocument.'
|
||||||
|
f'drawingml+xml+chart"/>'
|
||||||
|
)
|
||||||
|
if override not in ct_content:
|
||||||
|
ct_content = ct_content.replace("</Types>", f"{override}</Types>")
|
||||||
|
existing[ct_path] = ct_content.encode("utf-8")
|
||||||
|
|
||||||
|
for item, data in existing.items():
|
||||||
|
zout.writestr(item, data)
|
||||||
|
|
||||||
|
tmp_out.replace(pptx_path)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B2.1 — Placeholder filter (DATE / SLIDE_NUMBER / FOOTER)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPptxPlaceholderFilter:
|
||||||
|
"""Bug fix: PowerPoint auto-generates DATE, SLIDE_NUMBER, and FOOTER
|
||||||
|
placeholders at render time. Translating them wastes API calls
|
||||||
|
because the translated value gets overwritten on next render."""
|
||||||
|
|
||||||
|
def _build_pptx_with_placeholders(self, tmp_path: Path) -> Path:
|
||||||
|
"""Build a PPTX with a date placeholder, a slide-number placeholder,
|
||||||
|
a footer placeholder, plus one real text box. Returns the path."""
|
||||||
|
from pptx import Presentation
|
||||||
|
|
||||||
|
prs = Presentation()
|
||||||
|
slide_layout = prs.slide_layouts[6] # blank
|
||||||
|
slide = prs.slides.add_slide(slide_layout)
|
||||||
|
|
||||||
|
# Add real text — should be translated
|
||||||
|
_add_textbox_with_text(slide, "Real text")
|
||||||
|
|
||||||
|
# python-pptx doesn't expose a direct "add footer/date/sldNum
|
||||||
|
# placeholder" API, so we inject the XML directly. The placeholder
|
||||||
|
# types that should be filtered:
|
||||||
|
# 13 (sldNum), 15 (ftr), 16 (dt)
|
||||||
|
from pptx.oxml.ns import qn
|
||||||
|
from lxml import etree as _etree
|
||||||
|
|
||||||
|
spTree = slide.shapes._spTree
|
||||||
|
nvSpPr_template = """
|
||||||
|
<p:sp xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<p:nvSpPr>
|
||||||
|
<p:nvPr>
|
||||||
|
<p:ph type="{ph_type}" idx="1"/>
|
||||||
|
</p:nvPr>
|
||||||
|
<p:cNvPr id="{shape_id}" name="PH{shape_id}"/>
|
||||||
|
<p:cNvSpPr/>
|
||||||
|
</p:nvSpPr>
|
||||||
|
<p:spPr>
|
||||||
|
<a:xfrm>
|
||||||
|
<a:off x="0" y="0"/>
|
||||||
|
<a:ext cx="1000000" cy="500000"/>
|
||||||
|
</a:xfrm>
|
||||||
|
</p:spPr>
|
||||||
|
<p:txBody>
|
||||||
|
<a:bodyPr/>
|
||||||
|
<a:lstStyle/>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:rPr lang="en-US"/>
|
||||||
|
<a:t>{text}</a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</p:txBody>
|
||||||
|
</p:sp>
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Footer placeholder (type 15)
|
||||||
|
sp_xml = nvSpPr_template.format(
|
||||||
|
ph_type="ftr",
|
||||||
|
shape_id=100,
|
||||||
|
text="Footer auto",
|
||||||
|
)
|
||||||
|
sp_elem = _etree.fromstring(sp_xml)
|
||||||
|
spTree.append(sp_elem)
|
||||||
|
|
||||||
|
# Date placeholder (type 16)
|
||||||
|
sp_xml = nvSpPr_template.format(
|
||||||
|
ph_type="dt",
|
||||||
|
shape_id=101,
|
||||||
|
text="Date auto",
|
||||||
|
)
|
||||||
|
sp_elem = _etree.fromstring(sp_xml)
|
||||||
|
spTree.append(sp_elem)
|
||||||
|
|
||||||
|
# Slide number placeholder (type 13)
|
||||||
|
sp_xml = nvSpPr_template.format(
|
||||||
|
ph_type="sldNum",
|
||||||
|
shape_id=102,
|
||||||
|
text="Slide number auto",
|
||||||
|
)
|
||||||
|
sp_elem = _etree.fromstring(sp_xml)
|
||||||
|
spTree.append(sp_elem)
|
||||||
|
|
||||||
|
path = tmp_path / "placeholders.pptx"
|
||||||
|
prs.save(path)
|
||||||
|
return path
|
||||||
|
|
||||||
|
def test_placeholders_skipped(self, tmp_path):
|
||||||
|
"""The three auto-generated placeholder types should NOT be sent
|
||||||
|
to the translation provider."""
|
||||||
|
provider = MockProvider({"Real text": "TR_Real text"})
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
input_file = self._build_pptx_with_placeholders(tmp_path)
|
||||||
|
output_file = tmp_path / "output.pptx"
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# Real text should be translated
|
||||||
|
assert "Real text" in provider.calls, (
|
||||||
|
f"Real text missing from provider calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Auto-generated placeholders should NOT be in the calls
|
||||||
|
assert "Footer auto" not in provider.calls, (
|
||||||
|
f"Footer placeholder was sent to provider: {provider.calls}"
|
||||||
|
)
|
||||||
|
assert "Date auto" not in provider.calls, (
|
||||||
|
f"Date placeholder was sent to provider: {provider.calls}"
|
||||||
|
)
|
||||||
|
assert "Slide number auto" not in provider.calls, (
|
||||||
|
f"Slide number placeholder was sent to provider: {provider.calls}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B2.2 — SmartArt diagram text collection
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPptxDiagramTextCollection:
|
||||||
|
"""New feature: SmartArt text (in ppt/diagrams/data*.xml) is now
|
||||||
|
collected and translated."""
|
||||||
|
|
||||||
|
DIAGRAM_XML = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<dgm:dataModel xmlns:dgm="http://schemas.openxmlformats.org/drawingml/2006/diagram"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"
|
||||||
|
xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">
|
||||||
|
<dgm:ptLst>
|
||||||
|
<dgm:pt modelId="0" type="doc">
|
||||||
|
<dgm:prSet loTypeId="urn:microsoft.com/office/officeart/2005/8/layout/hierarchy1" loCatId="hierarchy"/>
|
||||||
|
<dgm:t>
|
||||||
|
<a:bodyPr/>
|
||||||
|
<a:lstStyle/>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:rPr lang="en-US"/>
|
||||||
|
<a:t>Root node</a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</dgm:t>
|
||||||
|
</dgm:pt>
|
||||||
|
<dgm:pt modelId="1">
|
||||||
|
<dgm:t>
|
||||||
|
<a:bodyPr/>
|
||||||
|
<a:lstStyle/>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:rPr lang="en-US"/>
|
||||||
|
<a:t>Child one</a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</dgm:t>
|
||||||
|
</dgm:pt>
|
||||||
|
<dgm:pt modelId="2">
|
||||||
|
<dgm:t>
|
||||||
|
<a:bodyPr/>
|
||||||
|
<a:lstStyle/>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:rPr lang="en-US"/>
|
||||||
|
<a:t>Child two</a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</dgm:t>
|
||||||
|
</dgm:pt>
|
||||||
|
</dgm:ptLst>
|
||||||
|
</dgm:dataModel>
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _build_pptx_with_diagram(self, tmp_path: Path) -> Path:
|
||||||
|
prs = Presentation()
|
||||||
|
slide = prs.slides.add_slide(prs.slide_layouts[6])
|
||||||
|
_add_textbox_with_text(slide, "Hello")
|
||||||
|
path = tmp_path / "with_diagram.pptx"
|
||||||
|
prs.save(path)
|
||||||
|
_inject_diagram_part(
|
||||||
|
path, "ppt/diagrams/data1.xml",
|
||||||
|
self.DIAGRAM_XML.encode("utf-8"),
|
||||||
|
)
|
||||||
|
return path
|
||||||
|
|
||||||
|
def test_diagram_text_collected(self, tmp_path):
|
||||||
|
"""Diagram text should appear in provider calls."""
|
||||||
|
provider = MockProvider()
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
input_file = self._build_pptx_with_diagram(tmp_path)
|
||||||
|
output_file = tmp_path / "output.pptx"
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# All three diagram nodes should be collected
|
||||||
|
assert "Root node" in provider.calls, (
|
||||||
|
f"Diagram 'Root node' not collected. Calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
assert "Child one" in provider.calls, (
|
||||||
|
f"Diagram 'Child one' not collected. Calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
assert "Child two" in provider.calls, (
|
||||||
|
f"Diagram 'Child two' not collected. Calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_diagram_text_applied(self, tmp_path):
|
||||||
|
"""Translated diagram text should be written back to the .pptx
|
||||||
|
ZIP (in ppt/diagrams/data*.xml)."""
|
||||||
|
provider = MockProvider({
|
||||||
|
"Root node": "TR_Root node",
|
||||||
|
"Child one": "TR_Child one",
|
||||||
|
"Child two": "TR_Child two",
|
||||||
|
})
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
input_file = self._build_pptx_with_diagram(tmp_path)
|
||||||
|
output_file = tmp_path / "output.pptx"
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# The diagram part in the output ZIP should have the translated text
|
||||||
|
with zipfile.ZipFile(output_file, "r") as zf:
|
||||||
|
assert "ppt/diagrams/data1.xml" in zf.namelist(), (
|
||||||
|
f"Diagram part missing in output. Files: {zf.namelist()}"
|
||||||
|
)
|
||||||
|
data = zf.read("ppt/diagrams/data1.xml").decode("utf-8")
|
||||||
|
assert "TR_Root node" in data, (
|
||||||
|
f"Diagram 'Root node' not translated in output:\n{data[:1000]}"
|
||||||
|
)
|
||||||
|
assert "TR_Child one" in data, (
|
||||||
|
f"Diagram 'Child one' not translated in output"
|
||||||
|
)
|
||||||
|
assert "TR_Child two" in data, (
|
||||||
|
f"Diagram 'Child two' not translated in output"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_no_diagram_no_error(self, tmp_path):
|
||||||
|
"""A PPTX without a diagram part should not raise."""
|
||||||
|
provider = MockProvider({"Hello": "TR_Hello"})
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
prs = Presentation()
|
||||||
|
slide = prs.slides.add_slide(prs.slide_layouts[6])
|
||||||
|
_add_textbox_with_text(slide, "Hello")
|
||||||
|
input_file = tmp_path / "no_diagram.pptx"
|
||||||
|
output_file = tmp_path / "output.pptx"
|
||||||
|
prs.save(input_file)
|
||||||
|
|
||||||
|
# Should not raise
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B2.3 — Diagram whitespace preservation
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPptxDiagramWhitespace:
|
||||||
|
"""Whitespace around translated diagram text should be preserved."""
|
||||||
|
|
||||||
|
DIAGRAM_PADDED_XML = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<dgm:dataModel xmlns:dgm="http://schemas.openxmlformats.org/drawingml/2006/diagram"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<dgm:ptLst>
|
||||||
|
<dgm:pt modelId="0">
|
||||||
|
<dgm:t>
|
||||||
|
<a:bodyPr/>
|
||||||
|
<a:lstStyle/>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:rPr lang="en-US"/>
|
||||||
|
<a:t> Padded text </a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</dgm:t>
|
||||||
|
</dgm:pt>
|
||||||
|
</dgm:ptLst>
|
||||||
|
</dgm:dataModel>
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_padded_diagram_text_preserved(self, tmp_path):
|
||||||
|
provider = MockProvider({"Padded text": "TR_Padded text"})
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
prs = Presentation()
|
||||||
|
slide = prs.slides.add_slide(prs.slide_layouts[6])
|
||||||
|
input_file = tmp_path / "padded.pptx"
|
||||||
|
prs.save(input_file)
|
||||||
|
_inject_diagram_part(
|
||||||
|
input_file, "ppt/diagrams/data1.xml",
|
||||||
|
self.DIAGRAM_PADDED_XML.encode("utf-8"),
|
||||||
|
)
|
||||||
|
output_file = tmp_path / "output.pptx"
|
||||||
|
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
with zipfile.ZipFile(output_file, "r") as zf:
|
||||||
|
data = zf.read("ppt/diagrams/data1.xml").decode("utf-8")
|
||||||
|
# Leading/trailing spaces should be preserved
|
||||||
|
assert " TR_Padded text " in data, (
|
||||||
|
f"Whitespace not preserved in diagram translation:\n{data[:1000]}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B2.4 — Element path navigation
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPptxElementPathNavigation:
|
||||||
|
"""Bug fix: chart/diagram elements should be matched by stored
|
||||||
|
element_path, not by string equality. This means identical text
|
||||||
|
in multiple elements (e.g. two 'Revenue' series) is handled correctly."""
|
||||||
|
|
||||||
|
SIMPLE_DIAGRAM = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<root xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<a:t>Alpha</a:t>
|
||||||
|
<a:t>Beta</a:t>
|
||||||
|
<a:t>Gamma</a:t>
|
||||||
|
</root>
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_path_navigation_basic(self):
|
||||||
|
"""The `_get_element_path` / `_find_element_by_path` pair should
|
||||||
|
round-trip correctly through a small XML tree."""
|
||||||
|
root = etree.fromstring(self.SIMPLE_DIAGRAM.encode("utf-8"))
|
||||||
|
translator = PowerPointTranslator()
|
||||||
|
|
||||||
|
# Get paths for all three <a:t> elements
|
||||||
|
elements = list(root.iter(f"{{{_NS_A}}}t"))
|
||||||
|
assert len(elements) == 3
|
||||||
|
|
||||||
|
paths = [translator._get_element_path(e) for e in elements]
|
||||||
|
|
||||||
|
# All three paths should be unique
|
||||||
|
assert len(set(paths)) == 3, f"Paths not unique: {paths}"
|
||||||
|
|
||||||
|
# Each path should resolve back to the same element
|
||||||
|
for elem, path in zip(elements, paths):
|
||||||
|
found = translator._find_element_by_path(root, path)
|
||||||
|
assert found is not None, f"Path didn't resolve: {path}"
|
||||||
|
assert found.text == elem.text, (
|
||||||
|
f"Path {path} resolved to wrong element: "
|
||||||
|
f"{found.text!r} vs {elem.text!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_path_fallback_to_string_match(self):
|
||||||
|
"""If the stored path is empty, the apply step should fall back
|
||||||
|
to string equality on `entry['original']`."""
|
||||||
|
root = etree.fromstring(self.SIMPLE_DIAGRAM.encode("utf-8"))
|
||||||
|
translator = PowerPointTranslator()
|
||||||
|
|
||||||
|
# Simulate a legacy entry with no element_path
|
||||||
|
legacy_entry = {
|
||||||
|
"original": "Beta",
|
||||||
|
"translated": "TR_Beta",
|
||||||
|
"element_path": None, # legacy
|
||||||
|
}
|
||||||
|
|
||||||
|
# Simulate the apply logic
|
||||||
|
target = None
|
||||||
|
for candidate in root.iter(f"{{{_NS_A}}}t"):
|
||||||
|
if (candidate.text or "").strip() == legacy_entry["original"]:
|
||||||
|
target = candidate
|
||||||
|
break
|
||||||
|
assert target is not None, "Fallback string match didn't find Beta"
|
||||||
|
assert target.text == "Beta"
|
||||||
|
|
||||||
|
def test_diagram_with_duplicate_text_paths_differ(self, tmp_path):
|
||||||
|
"""Two diagram nodes with the same text should have different
|
||||||
|
element_path values, so the apply step can target them correctly."""
|
||||||
|
dup_xml = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<dgm:dataModel xmlns:dgm="http://schemas.openxmlformats.org/drawingml/2006/diagram"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<dgm:ptLst>
|
||||||
|
<dgm:pt modelId="0">
|
||||||
|
<dgm:t><a:p><a:r><a:t>Revenue</a:t></a:r></a:p></dgm:t>
|
||||||
|
</dgm:pt>
|
||||||
|
<dgm:pt modelId="1">
|
||||||
|
<dgm:t><a:p><a:r><a:t>Revenue</a:t></a:r></a:p></dgm:t>
|
||||||
|
</dgm:pt>
|
||||||
|
</dgm:ptLst>
|
||||||
|
</dgm:dataModel>
|
||||||
|
"""
|
||||||
|
provider = MockProvider()
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
prs = Presentation()
|
||||||
|
slide = prs.slides.add_slide(prs.slide_layouts[6])
|
||||||
|
input_file = tmp_path / "dup.pptx"
|
||||||
|
prs.save(input_file)
|
||||||
|
_inject_diagram_part(
|
||||||
|
input_file, "ppt/diagrams/data1.xml",
|
||||||
|
dup_xml.encode("utf-8"),
|
||||||
|
)
|
||||||
|
output_file = tmp_path / "output.pptx"
|
||||||
|
|
||||||
|
# Should not raise; both should be translated independently
|
||||||
|
translator.translate_file(input_file, output_file, "fr")
|
||||||
|
|
||||||
|
# Both 'Revenue' texts should be in the calls
|
||||||
|
assert provider.calls.count("Revenue") == 2, (
|
||||||
|
f"Expected 2 'Revenue' calls, got {provider.calls.count('Revenue')}. "
|
||||||
|
f"All calls: {provider.calls}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B2.5 — Chart whitespace preservation
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPptxChartWhitespace:
|
||||||
|
"""Whitespace around translated chart text should be preserved."""
|
||||||
|
|
||||||
|
CHART_PADDED_XML = f"""<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||||
|
<c:chartSpace xmlns:c="http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main">
|
||||||
|
<c:chart>
|
||||||
|
<c:title>
|
||||||
|
<c:tx>
|
||||||
|
<c:rich>
|
||||||
|
<a:p>
|
||||||
|
<a:r>
|
||||||
|
<a:t> Padded chart title </a:t>
|
||||||
|
</a:r>
|
||||||
|
</a:p>
|
||||||
|
</c:rich>
|
||||||
|
</c:tx>
|
||||||
|
</c:title>
|
||||||
|
</c:chart>
|
||||||
|
</c:chartSpace>
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_padded_chart_text_via_internal_method(self):
|
||||||
|
"""The internal chart apply logic should preserve whitespace."""
|
||||||
|
provider = MockProvider({"Padded chart title": "Titre avec espaces"})
|
||||||
|
translator = PowerPointTranslator(provider=provider)
|
||||||
|
|
||||||
|
# Build a chart entry by hand (simulating collect time)
|
||||||
|
chart_xml = etree.fromstring(self.CHART_PADDED_XML.encode("utf-8"))
|
||||||
|
entries = []
|
||||||
|
for t_elem in chart_xml.iter(f"{{{_NS_A}}}t"):
|
||||||
|
text_raw = t_elem.text or ""
|
||||||
|
text = text_raw.strip()
|
||||||
|
if not text:
|
||||||
|
continue
|
||||||
|
entry = {
|
||||||
|
"element": t_elem,
|
||||||
|
"original": text,
|
||||||
|
"original_raw": text_raw,
|
||||||
|
"translated": "Titre avec espaces",
|
||||||
|
"tag": "a:t",
|
||||||
|
"element_path": translator._get_element_path(t_elem),
|
||||||
|
}
|
||||||
|
entries.append(entry)
|
||||||
|
|
||||||
|
if not hasattr(translator, "_chart_entries"):
|
||||||
|
translator._chart_entries = []
|
||||||
|
|
||||||
|
class _FakePart:
|
||||||
|
def __init__(self, blob):
|
||||||
|
self._blob = blob
|
||||||
|
@property
|
||||||
|
def blob(self):
|
||||||
|
return self._blob
|
||||||
|
@blob.setter
|
||||||
|
def blob(self, value):
|
||||||
|
self._blob = value
|
||||||
|
|
||||||
|
fake_part = _FakePart(
|
||||||
|
etree.tostring(
|
||||||
|
chart_xml, xml_declaration=True, encoding="UTF-8", standalone=True
|
||||||
|
)
|
||||||
|
)
|
||||||
|
translator._chart_entries.append({
|
||||||
|
"chart_part": fake_part,
|
||||||
|
"entries": entries,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Apply
|
||||||
|
translator._apply_chart_translations(Path("dummy"))
|
||||||
|
|
||||||
|
# Re-parse and check whitespace preserved
|
||||||
|
updated_xml = etree.fromstring(fake_part._blob)
|
||||||
|
all_t = list(updated_xml.iter(f"{{{_NS_A}}}t"))
|
||||||
|
# Find the title text
|
||||||
|
title_text = all_t[0].text or ""
|
||||||
|
assert " Titre avec espaces " in title_text, (
|
||||||
|
f"Chart whitespace not preserved: {title_text!r}"
|
||||||
|
)
|
||||||
391
tests/test_translators/test_b3_5_pdf_smart_fit.py
Normal file
391
tests/test_translators/test_b3_5_pdf_smart_fit.py
Normal file
@@ -0,0 +1,391 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track B3.5 — Smart PDF overflow handling.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- Tier 0: original bbox at original size wins if text fits
|
||||||
|
- Tier 1: expanded horizontal bbox preserves font size
|
||||||
|
- Tier 2: expanded vertical bbox (3x original height) is tried
|
||||||
|
- Tier 3: font shrink is limited to 2 steps (0.93, 0.87)
|
||||||
|
- Tier 4: HEADINGS (>= 14pt) never shrink below 90% of original
|
||||||
|
- Tier 5: BODY TEXT never shrinks below 75% of original
|
||||||
|
- Tier 6: graceful failure — block is skipped, metric emitted
|
||||||
|
- Redaction uses single rect per block (not per sub-bbox)
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import importlib.util
|
||||||
|
|
||||||
|
_REPO_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||||
|
|
||||||
|
|
||||||
|
def _load_pdf_translator():
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"pdf_translator_under_test",
|
||||||
|
_REPO_ROOT / "translators" / "pdf_translator.py",
|
||||||
|
)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
return mod
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def pdf_mod():
|
||||||
|
return _load_pdf_translator()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Constants
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestConstants:
|
||||||
|
def test_font_shrink_factor_mild(self, pdf_mod):
|
||||||
|
"""FONT_SHRINK_FACTOR should be 0.93 (mild), not 0.87 (aggressive)."""
|
||||||
|
assert pdf_mod.FONT_SHRINK_FACTOR == pytest.approx(0.93, abs=0.01), (
|
||||||
|
f"FONT_SHRINK_FACTOR is {pdf_mod.FONT_SHRINK_FACTOR}, "
|
||||||
|
"should be 0.93 (was 0.87 in B3 — too aggressive, broke hierarchy)"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_vertical_expansion_3x(self, pdf_mod):
|
||||||
|
"""MAX_VERTICAL_EXPANSION should be 6x (was 1.5x in B3, then 3x in B3.5).
|
||||||
|
Track B3.6 bumped it to 6x to handle long French titles that
|
||||||
|
wrap to multiple lines (a 22pt title may need 4-5 lines in French)."""
|
||||||
|
assert pdf_mod.MAX_VERTICAL_EXPANSION == pytest.approx(6.0, abs=0.1)
|
||||||
|
|
||||||
|
def test_heading_min_size_threshold(self, pdf_mod):
|
||||||
|
assert pdf_mod.HEADING_MIN_SIZE == 14.0
|
||||||
|
|
||||||
|
def test_heading_min_scale_90pct(self, pdf_mod):
|
||||||
|
assert pdf_mod.HEADING_MIN_SCALE == pytest.approx(0.90, abs=0.01)
|
||||||
|
|
||||||
|
def test_body_min_scale_75pct(self, pdf_mod):
|
||||||
|
assert pdf_mod.BODY_MIN_SCALE == pytest.approx(0.75, abs=0.01)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# _write_translated_block — tier behavior
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestSmartFitTiers:
|
||||||
|
def _make_block(self, font_size=12, x0=100, y0=100, x1=400, y1=120, text="Hello world"):
|
||||||
|
return {
|
||||||
|
"bbox": (x0, y0, x1, y1),
|
||||||
|
"text": text,
|
||||||
|
"font_size": font_size,
|
||||||
|
"color": 0,
|
||||||
|
"is_bold": False,
|
||||||
|
"is_italic": False,
|
||||||
|
"translated": text,
|
||||||
|
"sub_bboxes": [(x0, y0, x1, y1)],
|
||||||
|
}
|
||||||
|
|
||||||
|
def _make_page(self, page_w=612, page_h=792):
|
||||||
|
"""Mock page that accepts insert_textbox and returns rc (overflow indicator)."""
|
||||||
|
page = MagicMock()
|
||||||
|
import fitz
|
||||||
|
page.rect = fitz.Rect(0, 0, page_w, page_h)
|
||||||
|
# By default, accept the insert (return rc=0)
|
||||||
|
page.insert_textbox = MagicMock(return_value=0)
|
||||||
|
page.add_redact_annot = MagicMock()
|
||||||
|
return page
|
||||||
|
|
||||||
|
def test_tier_0_fits_at_original_size(self, pdf_mod):
|
||||||
|
"""If the text fits at original size + original bbox, no shrink."""
|
||||||
|
page = self._make_page()
|
||||||
|
block = self._make_block(font_size=12, text="Short text")
|
||||||
|
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
result = translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
# Should have succeeded
|
||||||
|
assert result is True
|
||||||
|
# First call should be with original size (12) and original bbox
|
||||||
|
first_call = page.insert_textbox.call_args_list[0]
|
||||||
|
# First positional arg is the rect
|
||||||
|
assert first_call[0][0] == block["bbox"]
|
||||||
|
# Second positional arg is the text
|
||||||
|
assert first_call[0][1] == "Short text"
|
||||||
|
# fontsize kwarg
|
||||||
|
assert first_call[1]["fontsize"] == 12
|
||||||
|
|
||||||
|
def test_tier_2_uses_expanded_vertical_bbox(self, pdf_mod):
|
||||||
|
"""When original bbox is too small, tier 2 expands vertically."""
|
||||||
|
page = self._make_page()
|
||||||
|
# Force overflow on the original bbox
|
||||||
|
page.insert_textbox = MagicMock(side_effect=[-5, -5, 0])
|
||||||
|
# -5 = overflow, 0 = fits
|
||||||
|
|
||||||
|
block = self._make_block(font_size=12, x0=100, y0=100, x1=400, y1=120)
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
result = translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
assert result is True
|
||||||
|
# Should have tried multiple times
|
||||||
|
assert page.insert_textbox.call_count >= 3
|
||||||
|
|
||||||
|
def test_tier_3_4_limit_to_two_shrink_steps(self, pdf_mod):
|
||||||
|
"""Even on persistent overflow, the shrink is limited."""
|
||||||
|
page = self._make_page()
|
||||||
|
# Always overflow
|
||||||
|
page.insert_textbox = MagicMock(return_value=-5)
|
||||||
|
|
||||||
|
block = self._make_block(font_size=20, x0=100, y0=100, x1=400, y1=120)
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
result = translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
# Should have failed (returned False) because persistent overflow
|
||||||
|
assert result is False
|
||||||
|
# But the number of insert_textbox calls should be bounded
|
||||||
|
# (not the 8+ attempts of the old aggressive strategy)
|
||||||
|
assert page.insert_textbox.call_count <= 8
|
||||||
|
|
||||||
|
def test_heading_not_shrunk_below_90_percent(self, pdf_mod):
|
||||||
|
"""A 22pt heading should never be shrunk below ~19.8pt."""
|
||||||
|
page = self._make_page()
|
||||||
|
call_sizes = []
|
||||||
|
# Track every fontsize used
|
||||||
|
def capture(*args, **kwargs):
|
||||||
|
call_sizes.append(kwargs.get("fontsize", args[2] if len(args) > 2 else None))
|
||||||
|
return 0 # always fit
|
||||||
|
page.insert_textbox = MagicMock(side_effect=capture)
|
||||||
|
|
||||||
|
# 22pt heading
|
||||||
|
block = self._make_block(font_size=22, text="X" * 1000)
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
# All attempted sizes should be >= 22 * 0.90 = 19.8
|
||||||
|
for s in call_sizes:
|
||||||
|
if s is not None:
|
||||||
|
assert s >= 22 * 0.90 - 0.1, (
|
||||||
|
f"Heading shrunk below 90% (got {s}, min allowed {22 * 0.90})"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_body_not_shrunk_below_75_percent(self, pdf_mod):
|
||||||
|
"""Body text (12pt) should never be shrunk below ~9pt."""
|
||||||
|
page = self._make_page()
|
||||||
|
call_sizes = []
|
||||||
|
def capture(*args, **kwargs):
|
||||||
|
call_sizes.append(kwargs.get("fontsize", args[2] if len(args) > 2 else None))
|
||||||
|
return 0
|
||||||
|
page.insert_textbox = MagicMock(side_effect=capture)
|
||||||
|
|
||||||
|
block = self._make_block(font_size=12, text="Y" * 1000)
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
for s in call_sizes:
|
||||||
|
if s is not None:
|
||||||
|
assert s >= 12 * 0.75 - 0.1, (
|
||||||
|
f"Body text shrunk below 75% (got {s}, min allowed {12 * 0.75})"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_graceful_failure_emits_metric(self, pdf_mod):
|
||||||
|
"""If nothing fits, the block is skipped and a metric is recorded."""
|
||||||
|
page = self._make_page()
|
||||||
|
page.insert_textbox = MagicMock(return_value=-5) # always overflow
|
||||||
|
|
||||||
|
# Track metric emissions
|
||||||
|
metrics_emitted = []
|
||||||
|
def fake_record(*args, **kwargs):
|
||||||
|
metrics_emitted.append((args, kwargs))
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
with patch.object(pdf_mod, "_record_format_loss_metric", fake_record):
|
||||||
|
block = self._make_block(font_size=12, text="Z" * 1000)
|
||||||
|
result = translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
assert result is False
|
||||||
|
# A format_loss metric should have been emitted
|
||||||
|
assert any("text_overflow" in str(call) for call in metrics_emitted), (
|
||||||
|
f"No text_overflow metric emitted. Got: {metrics_emitted}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_successful_render_does_not_emit_loss_metric(self, pdf_mod):
|
||||||
|
"""A block that fits cleanly should NOT trigger format_loss."""
|
||||||
|
page = self._make_page()
|
||||||
|
page.insert_textbox = MagicMock(return_value=0) # always fit
|
||||||
|
|
||||||
|
metrics_emitted = []
|
||||||
|
def fake_record(*args, **kwargs):
|
||||||
|
metrics_emitted.append((args, kwargs))
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
with patch.object(pdf_mod, "_record_format_loss_metric", fake_record):
|
||||||
|
block = self._make_block(font_size=12, text="Short")
|
||||||
|
result = translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
assert result is True
|
||||||
|
# No format_loss should have been emitted
|
||||||
|
assert metrics_emitted == [], (
|
||||||
|
f"Unexpected metric emission on success: {metrics_emitted}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_min_size_floor_respected(self, pdf_mod):
|
||||||
|
"""The min size floor (MIN_FONT_SIZE) is always respected."""
|
||||||
|
page = self._make_page()
|
||||||
|
call_sizes = []
|
||||||
|
def capture(*args, **kwargs):
|
||||||
|
call_sizes.append(kwargs.get("fontsize", args[2] if len(args) > 2 else None))
|
||||||
|
return 0
|
||||||
|
page.insert_textbox = MagicMock(side_effect=capture)
|
||||||
|
|
||||||
|
# Tiny font (3pt) — should still be at least MIN_FONT_SIZE (4.5pt)
|
||||||
|
block = self._make_block(font_size=3, text="Q" * 1000)
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
translator._write_translated_block(
|
||||||
|
page, block, font_path=None, is_rtl=False
|
||||||
|
)
|
||||||
|
# Exclude the placeholder font (used in graceful failure)
|
||||||
|
real_sizes = [s for s in call_sizes if s is not None and s >= 4.5]
|
||||||
|
for s in real_sizes:
|
||||||
|
assert s >= pdf_mod.MIN_FONT_SIZE, (
|
||||||
|
f"Font size {s} below absolute floor {pdf_mod.MIN_FONT_SIZE}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Real PDF comparison
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestRealPDFSmartFit:
|
||||||
|
"""Run the actual translator on a real PDF and check the output."""
|
||||||
|
|
||||||
|
def test_translate_real_pdf_preserves_hyperlinks(self, tmp_path):
|
||||||
|
"""The translated PDF should keep all hyperlinks (B3 fix)."""
|
||||||
|
import fitz
|
||||||
|
# Build a minimal PDF in memory
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
page.insert_text((72, 72), "Visit our website for more details.")
|
||||||
|
page.insert_link({
|
||||||
|
"kind": fitz.LINK_URI,
|
||||||
|
"from": fitz.Rect(72, 65, 280, 80),
|
||||||
|
"uri": "https://example.com",
|
||||||
|
})
|
||||||
|
|
||||||
|
in_path = tmp_path / "input.pdf"
|
||||||
|
out_path = tmp_path / "output.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
# Translate
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify hyperlinks preserved
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
links = list(out_page.get_links())
|
||||||
|
assert len(links) == 1
|
||||||
|
assert links[0].get("uri") == "https://example.com"
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
def test_translate_real_pdf_no_font_below_75pct_for_body(self, tmp_path):
|
||||||
|
"""Body text on a real PDF should not be shrunk below 75% of original.
|
||||||
|
|
||||||
|
Note: the [translation overflow] placeholder font (6.0pt) is
|
||||||
|
excluded from this check — it's a deliberate marker for skipped
|
||||||
|
blocks, not a real text font.
|
||||||
|
"""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
# Real body text at 12pt
|
||||||
|
page.insert_text((72, 72), "Hello world this is a test.",
|
||||||
|
fontsize=12)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input.pdf"
|
||||||
|
out_path = tmp_path / "output.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check output fonts — none should be below 12 * 0.75 = 9pt for body
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
out_fonts = set()
|
||||||
|
for block in out_page.get_text("dict").get("blocks", []):
|
||||||
|
for line in block.get("lines", []):
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
out_fonts.add(span.get("size", 0))
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
# Allow some shrinkage for very long text but never below 75%.
|
||||||
|
# Exclude the placeholder font (6.0pt) which is the overflow marker.
|
||||||
|
for size in out_fonts:
|
||||||
|
if size > 0 and size > 6.5: # > 6.5 excludes the 6.0pt placeholder
|
||||||
|
assert size >= 8.5, (
|
||||||
|
f"Body text font size {size}pt is below the 9pt floor "
|
||||||
|
f"(75% of 12pt)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Redaction strategy
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestSingleRedactionPerBlock:
|
||||||
|
"""Track B3.5: redaction should use a single rect per block, not per sub-bbox."""
|
||||||
|
|
||||||
|
def test_single_redaction_call_per_block(self, pdf_mod):
|
||||||
|
"""The redaction loop should call add_redact_annot ONCE per block."""
|
||||||
|
# Build a fake page
|
||||||
|
import fitz
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
redact_calls = []
|
||||||
|
page.add_redact_annot = MagicMock(side_effect=lambda r, **k: redact_calls.append(r))
|
||||||
|
page.apply_redactions = MagicMock()
|
||||||
|
|
||||||
|
# 3 blocks, each with 5 sub_bboxes
|
||||||
|
blocks = [
|
||||||
|
{
|
||||||
|
"translated": "Block 1 text",
|
||||||
|
"bbox": (50, 50, 400, 70),
|
||||||
|
"sub_bboxes": [(50, 50, 400, 70)] * 5,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"translated": "Block 2 text",
|
||||||
|
"bbox": (50, 80, 400, 100),
|
||||||
|
"sub_bboxes": [(50, 80, 400, 100)] * 5,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"translated": "Block 3 text",
|
||||||
|
"bbox": (50, 110, 400, 130),
|
||||||
|
"sub_bboxes": [(50, 110, 400, 130)] * 5,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
# Simulate the redaction loop from _process_pages_inplace
|
||||||
|
for block in blocks:
|
||||||
|
if block.get("translated"):
|
||||||
|
page.add_redact_annot(
|
||||||
|
fitz.Rect(block["bbox"]),
|
||||||
|
fill=(1, 1, 1),
|
||||||
|
)
|
||||||
|
|
||||||
|
# 3 blocks → 3 redaction calls (NOT 15)
|
||||||
|
assert len(redact_calls) == 3, (
|
||||||
|
f"Expected 3 redaction calls (one per block), got {len(redact_calls)}"
|
||||||
|
)
|
||||||
921
tests/test_translators/test_b3_6_pdf_drawings.py
Normal file
921
tests/test_translators/test_b3_6_pdf_drawings.py
Normal file
@@ -0,0 +1,921 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track B3.6 — PDF drawing preservation.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- Color backgrounds (e.g. "Avis important" box on light blue) are
|
||||||
|
preserved through translation (not erased by redaction).
|
||||||
|
- Title overflow: long translated titles expand horizontally + vertically
|
||||||
|
to avoid overlapping the next block.
|
||||||
|
- Transparent redaction (fill=None) is used when a block overlaps a
|
||||||
|
drawing, to preserve the drawing.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import importlib.util
|
||||||
|
|
||||||
|
_REPO_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||||
|
|
||||||
|
|
||||||
|
def _load_pdf_translator():
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"pdf_translator_under_test",
|
||||||
|
_REPO_ROOT / "translators" / "pdf_translator.py",
|
||||||
|
)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
return mod
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def pdf_mod():
|
||||||
|
return _load_pdf_translator()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Constants
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_6Constants:
|
||||||
|
def test_vertical_expansion_6x(self, pdf_mod):
|
||||||
|
"""MAX_VERTICAL_EXPANSION should be 6x (B3.6) to handle long titles."""
|
||||||
|
assert pdf_mod.MAX_VERTICAL_EXPANSION == pytest.approx(6.0, abs=0.1)
|
||||||
|
|
||||||
|
def test_transparent_redaction_uses_fill_none(self, pdf_mod):
|
||||||
|
"""When a block overlaps a drawing, redaction uses fill=None (transparent)."""
|
||||||
|
import fitz
|
||||||
|
page = MagicMock()
|
||||||
|
# Return one drawing (a colored rect) for get_drawings()
|
||||||
|
drawing = MagicMock()
|
||||||
|
drawing.get.return_value = fitz.Rect(72, 480, 500, 550)
|
||||||
|
drawing.get.side_effect = lambda key, default=None: {
|
||||||
|
"rect": fitz.Rect(72, 480, 500, 550),
|
||||||
|
"fill": (0.85, 0.92, 1.0),
|
||||||
|
}.get(key, default)
|
||||||
|
page.get_drawings = MagicMock(return_value=[drawing])
|
||||||
|
page.add_redact_annot = MagicMock()
|
||||||
|
|
||||||
|
# Simulate the logic from _process_pages_inplace
|
||||||
|
drawing_rects = []
|
||||||
|
for d in page.get_drawings():
|
||||||
|
r = d.get("rect")
|
||||||
|
if r is not None and d.get("fill") is not None:
|
||||||
|
drawing_rects.append(fitz.Rect(r))
|
||||||
|
|
||||||
|
# Block that intersects the drawing
|
||||||
|
block_bbox = fitz.Rect(85, 500, 200, 520)
|
||||||
|
intersects = any(
|
||||||
|
block_bbox.intersects(dr) for dr in drawing_rects
|
||||||
|
)
|
||||||
|
assert intersects is True
|
||||||
|
|
||||||
|
if intersects:
|
||||||
|
page.add_redact_annot(block_bbox, fill=None)
|
||||||
|
else:
|
||||||
|
page.add_redact_annot(block_bbox, fill=(1, 1, 1))
|
||||||
|
|
||||||
|
# Verify the redaction was added with fill=None (transparent)
|
||||||
|
call_kwargs = page.add_redact_annot.call_args[1]
|
||||||
|
assert call_kwargs.get("fill") is None, (
|
||||||
|
f"Expected fill=None for block overlapping drawing, "
|
||||||
|
f"got fill={call_kwargs.get('fill')}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_opaque_redaction_for_text_only_blocks(self, pdf_mod):
|
||||||
|
"""For blocks WITHOUT drawings, redaction uses white fill (opaque)."""
|
||||||
|
import fitz
|
||||||
|
page = MagicMock()
|
||||||
|
page.get_drawings = MagicMock(return_value=[]) # no drawings
|
||||||
|
page.add_redact_annot = MagicMock()
|
||||||
|
|
||||||
|
drawing_rects = []
|
||||||
|
for d in page.get_drawings():
|
||||||
|
r = d.get("rect")
|
||||||
|
if r is not None and d.get("fill") is not None:
|
||||||
|
drawing_rects.append(fitz.Rect(r))
|
||||||
|
|
||||||
|
block_bbox = fitz.Rect(100, 200, 400, 220)
|
||||||
|
intersects = any(
|
||||||
|
block_bbox.intersects(dr) for dr in drawing_rects
|
||||||
|
)
|
||||||
|
assert intersects is False # no drawings → no intersection
|
||||||
|
|
||||||
|
if intersects:
|
||||||
|
page.add_redact_annot(block_bbox, fill=None)
|
||||||
|
else:
|
||||||
|
page.add_redact_annot(block_bbox, fill=(1, 1, 1))
|
||||||
|
|
||||||
|
call_kwargs = page.add_redact_annot.call_args[1]
|
||||||
|
assert call_kwargs.get("fill") == (1, 1, 1), (
|
||||||
|
f"Expected white fill for text-only block, got {call_kwargs.get('fill')}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Real PDF test
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_6RealPDF:
|
||||||
|
def test_colored_box_preserved_in_translation(self, tmp_path):
|
||||||
|
"""A PDF with a colored background box (e.g. "Avis important") should
|
||||||
|
keep that color in the translated output."""
|
||||||
|
import fitz
|
||||||
|
|
||||||
|
# Build a minimal PDF: light blue box with text inside
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
# Light blue background
|
||||||
|
page.draw_rect(
|
||||||
|
fitz.Rect(72, 480, 500, 550),
|
||||||
|
color=(0.9, 0.95, 1.0),
|
||||||
|
fill=(0.85, 0.92, 1.0),
|
||||||
|
)
|
||||||
|
page.insert_text((85, 500), "Important notice", fontsize=14, color=(0.1, 0.2, 0.5))
|
||||||
|
|
||||||
|
in_path = tmp_path / "input.pdf"
|
||||||
|
out_path = tmp_path / "output.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
# Translate
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify the blue box is still there
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
drawings = out_page.get_drawings()
|
||||||
|
# Find a drawing with the blue fill
|
||||||
|
blue_drawings = [
|
||||||
|
d for d in drawings
|
||||||
|
if d.get("fill") == pytest.approx((0.85, 0.92, 1.0), abs=0.01)
|
||||||
|
]
|
||||||
|
assert len(blue_drawings) >= 1, (
|
||||||
|
f"Blue background drawing was lost! Got {len(drawings)} drawings: "
|
||||||
|
f"{[(d.get('rect'), d.get('fill')) for d in drawings]}"
|
||||||
|
)
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
def test_title_does_not_overflow_into_next_block(self, tmp_path):
|
||||||
|
"""A long translated title should not overlap the next block."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
# Title at top
|
||||||
|
page.insert_text(
|
||||||
|
(72, 80),
|
||||||
|
"A very long title that should wrap to two lines in translation",
|
||||||
|
fontsize=22,
|
||||||
|
)
|
||||||
|
# Next block at y=110 (very close — should be problematic)
|
||||||
|
page.insert_text(
|
||||||
|
(72, 110),
|
||||||
|
"Subtitle text",
|
||||||
|
fontsize=11,
|
||||||
|
)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input.pdf"
|
||||||
|
out_path = tmp_path / "output.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify the title doesn't overlap the subtitle
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
text_dict = out_page.get_text("dict")
|
||||||
|
|
||||||
|
# Get title bbox and subtitle bbox
|
||||||
|
title_bbox = None
|
||||||
|
subtitle_bbox = None
|
||||||
|
for block in text_dict.get("blocks", []):
|
||||||
|
if block.get("type") != 0:
|
||||||
|
continue
|
||||||
|
block_text = " ".join(
|
||||||
|
s.get("text", "")
|
||||||
|
for line in block.get("lines", [])
|
||||||
|
for s in line.get("spans", [])
|
||||||
|
).lower()
|
||||||
|
if "specification" in block_text or "sp" in block_text and "technique" in block_text:
|
||||||
|
title_bbox = block.get("bbox")
|
||||||
|
elif "version" in block_text or "subtitle" in block_text:
|
||||||
|
subtitle_bbox = block.get("bbox")
|
||||||
|
|
||||||
|
if title_bbox and subtitle_bbox:
|
||||||
|
title_bottom = title_bbox[3]
|
||||||
|
subtitle_top = subtitle_bbox[1]
|
||||||
|
assert title_bottom <= subtitle_top + 5, (
|
||||||
|
f"Title bottom ({title_bottom}) overlaps subtitle top ({subtitle_top}). "
|
||||||
|
f"Title: {title_bbox}, Subtitle: {subtitle_bbox}"
|
||||||
|
)
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Drawing re-application (alternative strategy)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_6DrawingReapplication:
|
||||||
|
"""The alternative strategy: re-apply original drawings on top of
|
||||||
|
redaction. We use the cleaner strategy (transparent redaction) but
|
||||||
|
keep this test for the legacy approach."""
|
||||||
|
|
||||||
|
def test_drawings_collected_before_redaction(self, pdf_mod):
|
||||||
|
"""The page's drawings should be captured BEFORE the redaction."""
|
||||||
|
import fitz
|
||||||
|
page = MagicMock()
|
||||||
|
page.get_drawings = MagicMock(return_value=[
|
||||||
|
{"rect": fitz.Rect(0, 0, 100, 100), "fill": (1, 0, 0)},
|
||||||
|
])
|
||||||
|
|
||||||
|
# Capture
|
||||||
|
captured = []
|
||||||
|
if hasattr(page, "get_drawings"):
|
||||||
|
try:
|
||||||
|
captured = list(page.get_drawings())
|
||||||
|
except Exception:
|
||||||
|
captured = []
|
||||||
|
assert len(captured) == 1
|
||||||
|
assert captured[0]["fill"] == (1, 0, 0)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Track B3.7 — next-block-aware vertical expansion
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_7NextBlockCeiling:
|
||||||
|
"""Track B3.7: the smart-fit logic must cap vertical expansion at
|
||||||
|
the next block's y0, not at the page bottom. Otherwise, a long
|
||||||
|
translated block can flow into the block below it."""
|
||||||
|
|
||||||
|
def test_next_block_y_populated_for_each_block(self, pdf_mod):
|
||||||
|
"""For 3 vertically stacked blocks, the middle one should have
|
||||||
|
the third's y0 as its next_block_y, and the last one the page bottom."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
blocks = [
|
||||||
|
{"bbox": (72, 100, 540, 120), "font_size": 12, "text": "A"},
|
||||||
|
{"bbox": (72, 200, 540, 220), "font_size": 12, "text": "B"},
|
||||||
|
{"bbox": (72, 300, 540, 320), "font_size": 12, "text": "C"},
|
||||||
|
]
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
assert blocks[0]["_next_block_y"] == 198 # block 1 y0 - 2
|
||||||
|
assert blocks[1]["_next_block_y"] == 298 # block 2 y0 - 2
|
||||||
|
assert blocks[2]["_next_block_y"] == 774 # page bottom (792 - 18)
|
||||||
|
|
||||||
|
def test_next_block_y_sorted_by_y0(self, pdf_mod):
|
||||||
|
"""If blocks are passed out of order, _populate_next_block_y
|
||||||
|
must still produce correct next-block mappings."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
blocks = [
|
||||||
|
{"bbox": (72, 300, 540, 320), "font_size": 12, "text": "C"},
|
||||||
|
{"bbox": (72, 100, 540, 120), "font_size": 12, "text": "A"},
|
||||||
|
{"bbox": (72, 200, 540, 220), "font_size": 12, "text": "B"},
|
||||||
|
]
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
# Find the 'A' block (y0=100) and verify it points to 'B' (y0=200)
|
||||||
|
a_block = next(b for b in blocks if b["text"] == "A")
|
||||||
|
assert a_block["_next_block_y"] == 198
|
||||||
|
|
||||||
|
def test_long_block_does_not_overflow_into_neighbour(self, tmp_path):
|
||||||
|
"""End-to-end: a long French translation should not visually
|
||||||
|
overlap the block below it."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
# Block 1: a short English title that will translate to a long French title
|
||||||
|
page.insert_textbox(
|
||||||
|
fitz.Rect(72, 100, 540, 120),
|
||||||
|
"Latest version",
|
||||||
|
fontsize=12,
|
||||||
|
)
|
||||||
|
# Block 2: a TOC entry very close below
|
||||||
|
page.insert_textbox(
|
||||||
|
fitz.Rect(72, 130, 540, 145),
|
||||||
|
"8. Troubleshooting",
|
||||||
|
fontsize=12,
|
||||||
|
)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input.pdf"
|
||||||
|
out_path = tmp_path / "output.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
# Translate with a provider that returns a much longer French string
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
mock = MagicMock()
|
||||||
|
mock.translate_batch = MagicMock(
|
||||||
|
side_effect=lambda texts, tgt, src: [
|
||||||
|
"Pour la dernière version de ce document, veuillez consulter le site web officiel à l'adresse suivante : " + t
|
||||||
|
if t and t != "8. Troubleshooting" else t
|
||||||
|
for t in texts
|
||||||
|
]
|
||||||
|
)
|
||||||
|
translator.set_provider(mock)
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify: no block bottom should exceed the next block's top
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
text_blocks = [
|
||||||
|
b for b in out_page.get_text("dict").get("blocks", [])
|
||||||
|
if b.get("type") == 0
|
||||||
|
]
|
||||||
|
# Sort by y0 and check that no two consecutive blocks overlap
|
||||||
|
text_blocks.sort(key=lambda b: b["bbox"][1])
|
||||||
|
for i in range(len(text_blocks) - 1):
|
||||||
|
top_of_next = text_blocks[i + 1]["bbox"][1]
|
||||||
|
bottom_of_curr = text_blocks[i]["bbox"][3]
|
||||||
|
assert bottom_of_curr <= top_of_next + 1, (
|
||||||
|
f"Block {i} bottom ({bottom_of_curr}) overlaps block {i+1} top ({top_of_next}). "
|
||||||
|
f"Block {i}: {text_blocks[i]['bbox']}, Block {i+1}: {text_blocks[i+1]['bbox']}"
|
||||||
|
)
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Track B3.8 — column-aware next_block_y (generic for multi-column PDFs)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_8ColumnAwareNextBlockY:
|
||||||
|
"""Track B3.8: PDFs with multi-column layouts (journals, brochures,
|
||||||
|
newspapers) must not have the 'next block' of a left-column block
|
||||||
|
point to a right-column block.
|
||||||
|
|
||||||
|
Also tests: edge cases like centered full-width headers, blocks at
|
||||||
|
identical y0 in different columns, and the max_expand_y negative
|
||||||
|
clamp."""
|
||||||
|
|
||||||
|
def test_two_columns_get_separate_next_block_y(self, pdf_mod):
|
||||||
|
"""2-column layout: left column blocks must point to the next
|
||||||
|
LEFT-column block, not the right-column block at the same y."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
|
||||||
|
# Left column: 3 blocks at x0=72
|
||||||
|
# Right column: 3 blocks at x0=320
|
||||||
|
blocks = [
|
||||||
|
# Left column (top to bottom)
|
||||||
|
{"bbox": (72, 100, 290, 120), "font_size": 12, "text": "L1"},
|
||||||
|
{"bbox": (72, 200, 290, 220), "font_size": 12, "text": "L2"},
|
||||||
|
{"bbox": (72, 300, 290, 320), "font_size": 12, "text": "L3"},
|
||||||
|
# Right column (top to bottom)
|
||||||
|
{"bbox": (320, 100, 540, 120), "font_size": 12, "text": "R1"},
|
||||||
|
{"bbox": (320, 200, 540, 220), "font_size": 12, "text": "R2"},
|
||||||
|
{"bbox": (320, 300, 540, 320), "font_size": 12, "text": "R3"},
|
||||||
|
]
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
|
||||||
|
# Left column: L1 -> L2, L2 -> L3, L3 -> page bottom
|
||||||
|
l1 = next(b for b in blocks if b["text"] == "L1")
|
||||||
|
l2 = next(b for b in blocks if b["text"] == "L2")
|
||||||
|
l3 = next(b for b in blocks if b["text"] == "L3")
|
||||||
|
assert l1["_next_block_y"] == 198 # L2 y0 - 2
|
||||||
|
assert l2["_next_block_y"] == 298 # L3 y0 - 2
|
||||||
|
assert l3["_next_block_y"] == 774 # page bottom
|
||||||
|
|
||||||
|
# Right column: R1 -> R2, R2 -> R3, R3 -> page bottom
|
||||||
|
r1 = next(b for b in blocks if b["text"] == "R1")
|
||||||
|
r2 = next(b for b in blocks if b["text"] == "R2")
|
||||||
|
r3 = next(b for b in blocks if b["text"] == "R3")
|
||||||
|
assert r1["_next_block_y"] == 198
|
||||||
|
assert r2["_next_block_y"] == 298
|
||||||
|
assert r3["_next_block_y"] == 774
|
||||||
|
|
||||||
|
def test_centered_full_width_header_gets_own_column(self, pdf_mod):
|
||||||
|
"""A full-width centered header (x0=200, x1=500) between two
|
||||||
|
columns should be its own column. Its 'next block' should be
|
||||||
|
the page bottom (no full-width blocks below)."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
|
||||||
|
blocks = [
|
||||||
|
{"bbox": (200, 50, 500, 70), "font_size": 18, "text": "HEADER"},
|
||||||
|
{"bbox": (72, 100, 290, 120), "font_size": 12, "text": "L1"},
|
||||||
|
{"bbox": (320, 100, 540, 120), "font_size": 12, "text": "R1"},
|
||||||
|
]
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
|
||||||
|
header = next(b for b in blocks if b["text"] == "HEADER")
|
||||||
|
# Header is its own column, so its next_block_y is page bottom
|
||||||
|
assert header["_next_block_y"] == 774
|
||||||
|
|
||||||
|
def test_three_columns(self, pdf_mod):
|
||||||
|
"""3-column layout (e.g. newspaper) should also be handled."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
|
||||||
|
blocks = [
|
||||||
|
{"bbox": (72, 100, 230, 120), "text": "C1a"},
|
||||||
|
{"bbox": (72, 200, 230, 220), "text": "C1b"},
|
||||||
|
{"bbox": (250, 100, 410, 120), "text": "C2a"},
|
||||||
|
{"bbox": (250, 200, 410, 220), "text": "C2b"},
|
||||||
|
{"bbox": (430, 100, 590, 120), "text": "C3a"},
|
||||||
|
{"bbox": (430, 200, 590, 220), "text": "C3b"},
|
||||||
|
]
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
|
||||||
|
c1b = next(b for b in blocks if b["text"] == "C1b")
|
||||||
|
c2b = next(b for b in blocks if b["text"] == "C2b")
|
||||||
|
c3b = next(b for b in blocks if b["text"] == "C3b")
|
||||||
|
# Each column's last block goes to page bottom
|
||||||
|
assert c1b["_next_block_y"] == 774
|
||||||
|
assert c2b["_next_block_y"] == 774
|
||||||
|
assert c3b["_next_block_y"] == 774
|
||||||
|
|
||||||
|
def test_single_block_page(self, pdf_mod):
|
||||||
|
"""A page with a single block should have its next_block_y = page bottom."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
|
||||||
|
blocks = [{"bbox": (72, 100, 540, 120), "text": "ONLY"}]
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
assert blocks[0]["_next_block_y"] == 774
|
||||||
|
|
||||||
|
def test_empty_block_list(self, pdf_mod):
|
||||||
|
"""An empty block list should not crash."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
page = MagicMock()
|
||||||
|
page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
|
||||||
|
blocks = []
|
||||||
|
# Should be a no-op
|
||||||
|
translator._populate_next_block_y(blocks, page.rect)
|
||||||
|
assert blocks == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestB3_8MaxExpandYNegativeClamp:
|
||||||
|
"""Track B3.8: max_expand_y must never be negative (would create
|
||||||
|
an invalid fitz.Rect with y1 < y0)."""
|
||||||
|
|
||||||
|
def test_max_expand_y_clamped_to_zero(self, pdf_mod):
|
||||||
|
"""When next_block_y sits above the current block, the
|
||||||
|
expansion should be clamped to 0 (no expansion)."""
|
||||||
|
import fitz
|
||||||
|
# Use a real PDFTranslator to test the full smart-fit path
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
|
||||||
|
# Build a doc and check the actual computation path
|
||||||
|
# by inspecting the logic
|
||||||
|
original_rect = fitz.Rect(72, 100, 540, 120) # height 20
|
||||||
|
# Simulate next_block_y ABOVE original_rect.y1 (impossible normally,
|
||||||
|
# but covers the edge case where extracted blocks have weird ordering)
|
||||||
|
next_block_y = 50 # above y0=100
|
||||||
|
max_expand_y = max(
|
||||||
|
0.0,
|
||||||
|
min(
|
||||||
|
next_block_y - original_rect.y1, # 50 - 120 = -70
|
||||||
|
original_rect.height * 6.0, # 120
|
||||||
|
),
|
||||||
|
)
|
||||||
|
assert max_expand_y == 0.0, (
|
||||||
|
f"Expected max_expand_y to be clamped to 0, got {max_expand_y}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_two_column_pdf_translation_end_to_end(self, tmp_path):
|
||||||
|
"""End-to-end: a 2-column PDF with a centered full-width title
|
||||||
|
translates cleanly. PyMuPDF merges the 2 columns per y into a
|
||||||
|
single horizontal block, but the column-aware next_block_y logic
|
||||||
|
still applies (each horizontal block is its own row)."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
# Centered title (full-width)
|
||||||
|
page.insert_text((200, 70), "CENTERED HEADER", fontsize=18)
|
||||||
|
# 3 rows of 2-column content
|
||||||
|
for i, y in enumerate([150, 250, 350]):
|
||||||
|
page.insert_text((72, y), f"Left col {i+1}", fontsize=12)
|
||||||
|
page.insert_text((320, y), f"Right col {i+1}", fontsize=12)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input_2col.pdf"
|
||||||
|
out_path = tmp_path / "output_2col.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
|
||||||
|
# Build a mock that handles BOTH the new-style
|
||||||
|
# (list of TranslationRequest) and legacy-style call signatures.
|
||||||
|
def _flexible_translate(*args, **kwargs):
|
||||||
|
if args and hasattr(args[0], "__iter__") and not isinstance(args[0], str):
|
||||||
|
first = args[0][0] if len(args[0]) else None
|
||||||
|
if first is not None and hasattr(first, "text"):
|
||||||
|
class _R:
|
||||||
|
def __init__(self, t): self.translated_text = t
|
||||||
|
return [_R(r.text) for r in args[0]]
|
||||||
|
return list(args[0]) if args else []
|
||||||
|
|
||||||
|
mock = MagicMock()
|
||||||
|
mock.__class__.__name__ = "MagicMock" # ensure is_new_style detection
|
||||||
|
mock.translate_batch = MagicMock(side_effect=_flexible_translate)
|
||||||
|
translator.set_provider(mock)
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify: 4 blocks present (1 title + 3 horizontal merged blocks)
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
text_blocks = [
|
||||||
|
b for b in out_page.get_text("dict").get("blocks", [])
|
||||||
|
if b.get("type") == 0
|
||||||
|
]
|
||||||
|
assert len(text_blocks) >= 4, (
|
||||||
|
f"Expected >= 4 blocks (1 title + 3 content rows), got {len(text_blocks)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# The title's next_block_y must point to the first content row,
|
||||||
|
# not the page bottom (the title is at y=70, so it should have
|
||||||
|
# room to expand to ~y=150-2=148).
|
||||||
|
# We can't directly read _next_block_y from the output, but we
|
||||||
|
# can verify the title's bbox is preserved at y~70, not pushed
|
||||||
|
# down to y>700.
|
||||||
|
title_block = text_blocks[0]
|
||||||
|
assert title_block["bbox"][1] < 100, (
|
||||||
|
f"Title was pushed down (y0={title_block['bbox'][1]}), "
|
||||||
|
f"expected near 70"
|
||||||
|
)
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Track B3.9 — table row preservation (column structure survives translation)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_9TableRowSplitting:
|
||||||
|
"""Track B3.9: a PDF 'block' that contains multiple LINES at the
|
||||||
|
SAME y but different x positions is a table row. The extractor must
|
||||||
|
split it into one sub-block per line so the column structure
|
||||||
|
survives translation.
|
||||||
|
|
||||||
|
Without B3.9, the whole row is treated as one paragraph and
|
||||||
|
`insert_textbox` writes the text left-aligned at x0, collapsing
|
||||||
|
all columns into one."""
|
||||||
|
|
||||||
|
def test_horizontal_layout_detected(self, pdf_mod):
|
||||||
|
"""A block with lines at the same y and different x is
|
||||||
|
detected as a horizontal layout (table row)."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
# Mock page that returns one block with 3 lines at same y
|
||||||
|
mock_page = MagicMock()
|
||||||
|
mock_page.get_text.return_value = {
|
||||||
|
"blocks": [{
|
||||||
|
"type": 0,
|
||||||
|
"bbox": (90, 188, 460, 203),
|
||||||
|
"lines": [
|
||||||
|
{"bbox": (90, 188, 162, 203), "spans": [
|
||||||
|
{"text": "Col 1", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (90, 188)},
|
||||||
|
]},
|
||||||
|
{"bbox": (220, 188, 292, 203), "spans": [
|
||||||
|
{"text": "Col 2", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (220, 188)},
|
||||||
|
]},
|
||||||
|
{"bbox": (350, 188, 460, 203), "spans": [
|
||||||
|
{"text": "Col 3", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (350, 188)},
|
||||||
|
]},
|
||||||
|
],
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
blocks = translator._extract_text_blocks(mock_page)
|
||||||
|
# Should be split into 3 separate blocks (one per column)
|
||||||
|
assert len(blocks) == 3, f"Expected 3 blocks, got {len(blocks)}"
|
||||||
|
assert blocks[0]["text"] == "Col 1"
|
||||||
|
assert blocks[1]["text"] == "Col 2"
|
||||||
|
assert blocks[2]["text"] == "Col 3"
|
||||||
|
# Each block should have its own bbox (not the merged row bbox)
|
||||||
|
assert blocks[0]["bbox"][0] == 90
|
||||||
|
assert blocks[1]["bbox"][0] == 220
|
||||||
|
assert blocks[2]["bbox"][0] == 350
|
||||||
|
|
||||||
|
def test_vertical_layout_kept_as_one_block(self, pdf_mod):
|
||||||
|
"""A block with lines at DIFFERENT y (same x) is a multi-line
|
||||||
|
paragraph. It must NOT be split."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
mock_page = MagicMock()
|
||||||
|
mock_page.get_text.return_value = {
|
||||||
|
"blocks": [{
|
||||||
|
"type": 0,
|
||||||
|
"bbox": (72, 100, 540, 150),
|
||||||
|
"lines": [
|
||||||
|
{"bbox": (72, 100, 540, 115), "spans": [
|
||||||
|
{"text": "Line 1", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (72, 100)},
|
||||||
|
]},
|
||||||
|
{"bbox": (72, 120, 540, 135), "spans": [
|
||||||
|
{"text": "Line 2", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (72, 120)},
|
||||||
|
]},
|
||||||
|
{"bbox": (72, 140, 540, 150), "spans": [
|
||||||
|
{"text": "Line 3", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (72, 140)},
|
||||||
|
]},
|
||||||
|
],
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
blocks = translator._extract_text_blocks(mock_page)
|
||||||
|
# Should remain 1 block with the 3 lines joined
|
||||||
|
assert len(blocks) == 1
|
||||||
|
assert blocks[0]["text"] == "Line 1\nLine 2\nLine 3"
|
||||||
|
assert blocks[0]["line_count"] == 3
|
||||||
|
|
||||||
|
def test_single_line_block_unchanged(self, pdf_mod):
|
||||||
|
"""A block with a single line is unchanged (no split)."""
|
||||||
|
import fitz
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
mock_page = MagicMock()
|
||||||
|
mock_page.get_text.return_value = {
|
||||||
|
"blocks": [{
|
||||||
|
"type": 0,
|
||||||
|
"bbox": (72, 100, 300, 115),
|
||||||
|
"lines": [
|
||||||
|
{"bbox": (72, 100, 300, 115), "spans": [
|
||||||
|
{"text": "Just one line", "size": 12, "font": "helv",
|
||||||
|
"color": 0, "flags": 0, "origin": (72, 100)},
|
||||||
|
]},
|
||||||
|
],
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
blocks = translator._extract_text_blocks(mock_page)
|
||||||
|
assert len(blocks) == 1
|
||||||
|
assert blocks[0]["text"] == "Just one line"
|
||||||
|
|
||||||
|
def test_table_cell_each_at_own_x(self, tmp_path):
|
||||||
|
"""End-to-end: a 3-column table in a PDF must be translated
|
||||||
|
with each cell staying at its original x position.
|
||||||
|
|
||||||
|
Note: PyMuPDF re-groups the cells into row-blocks when reading
|
||||||
|
the output back (3 lines per block), but each LINE (cell) must
|
||||||
|
be at its original x position. The test checks the line x0
|
||||||
|
values, not the block count.
|
||||||
|
"""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
# Build a 3-col table row by inserting text at 3 x positions on
|
||||||
|
# the same y. PyMuPDF groups them as a single block with 3 lines.
|
||||||
|
page.insert_text((90, 200), "Name", fontsize=12)
|
||||||
|
page.insert_text((220, 200), "Score", fontsize=12)
|
||||||
|
page.insert_text((350, 200), "Rank", fontsize=12)
|
||||||
|
page.insert_text((90, 220), "Alice", fontsize=12)
|
||||||
|
page.insert_text((220, 220), "95", fontsize=12)
|
||||||
|
page.insert_text((350, 220), "1", fontsize=12)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input_table.pdf"
|
||||||
|
out_path = tmp_path / "output_table.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
|
||||||
|
def _flexible_translate(*args, **kwargs):
|
||||||
|
if args and hasattr(args[0], "__iter__") and not isinstance(args[0], str):
|
||||||
|
first = args[0][0] if len(args[0]) else None
|
||||||
|
if first is not None and hasattr(first, "text"):
|
||||||
|
class _R:
|
||||||
|
def __init__(self, t): self.translated_text = t
|
||||||
|
return [_R(r.text) for r in args[0]]
|
||||||
|
return list(args[0]) if args else []
|
||||||
|
|
||||||
|
mock = MagicMock()
|
||||||
|
mock.__class__.__name__ = "MagicMock"
|
||||||
|
mock.translate_batch = MagicMock(side_effect=_flexible_translate)
|
||||||
|
translator.set_provider(mock)
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
text_blocks = [
|
||||||
|
b for b in out_page.get_text("dict").get("blocks", [])
|
||||||
|
if b.get("type") == 0
|
||||||
|
]
|
||||||
|
# We should see 2 row-blocks, each containing 3 cell-lines
|
||||||
|
assert len(text_blocks) == 2, (
|
||||||
|
f"Expected 2 row-blocks, got {len(text_blocks)}"
|
||||||
|
)
|
||||||
|
for row_block in text_blocks:
|
||||||
|
lines = row_block.get("lines", [])
|
||||||
|
assert len(lines) == 3, (
|
||||||
|
f"Row block should have 3 cells, got {len(lines)}"
|
||||||
|
)
|
||||||
|
# Each cell's x0 should be one of the 3 column positions
|
||||||
|
cell_x0s = sorted(line["bbox"][0] for line in lines)
|
||||||
|
assert cell_x0s == [90.0, 220.0, 350.0], (
|
||||||
|
f"Cell x0s are {cell_x0s}, expected [90, 220, 350]"
|
||||||
|
)
|
||||||
|
out_doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Track B3.10 — code-block / callout preservation (drawing-covered blocks)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestB3_10DrawingCoveredBlocks:
|
||||||
|
"""Track B3.10: when a block is covered by a colored background
|
||||||
|
drawing (code block, callout box, info box), each line must stay
|
||||||
|
at its own y position. The merge logic must NOT collapse them
|
||||||
|
into a single block, because the surrounding drawing has a fixed
|
||||||
|
extent that the rewritten text must align with.
|
||||||
|
|
||||||
|
Without B3.10, the merge logic joins all 5 code lines into one
|
||||||
|
block, the smart-fit shrinks the font to fit, and the rewritten
|
||||||
|
text no longer aligns with the background drawing — half the code
|
||||||
|
spills outside the box, the box appears clipped, etc.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_code_block_lines_marked_no_merge(self, tmp_path):
|
||||||
|
"""A code block (multiple lines, all covered by a background
|
||||||
|
drawing) should be split into per-line blocks with _no_merge=True."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
# Add a light-gray background drawing
|
||||||
|
page.draw_rect(
|
||||||
|
fitz.Rect(72, 290, 540, 400),
|
||||||
|
color=(0.9, 0.9, 0.9),
|
||||||
|
fill=(0.95, 0.95, 0.95),
|
||||||
|
)
|
||||||
|
# 5 code lines inside the drawing (use larger spacing for
|
||||||
|
# reliable y separation after PyMuPDF baseline adjustment)
|
||||||
|
for i, y in enumerate([305, 325, 345, 365, 385]):
|
||||||
|
page.insert_text((85, y), f"line {i+1}", fontsize=10)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input_code.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
in_doc = fitz.open(str(in_path))
|
||||||
|
blocks = translator._extract_text_blocks(in_doc[0])
|
||||||
|
in_doc.close()
|
||||||
|
|
||||||
|
# Should have 5 separate blocks, each with no_merge=True
|
||||||
|
code_blocks = [b for b in blocks if b.get("_no_merge")]
|
||||||
|
assert len(code_blocks) == 5, (
|
||||||
|
f"Expected 5 code blocks with no_merge=True, got {len(code_blocks)}"
|
||||||
|
)
|
||||||
|
# Each should have a distinct y position (within 1pt of each other)
|
||||||
|
y_positions = [b["bbox"][1] for b in code_blocks]
|
||||||
|
# Sort and check gaps are >= ~16pt (line spacing)
|
||||||
|
sorted_ys = sorted(y_positions)
|
||||||
|
for i in range(len(sorted_ys) - 1):
|
||||||
|
gap = sorted_ys[i + 1] - sorted_ys[i]
|
||||||
|
assert gap > 10, (
|
||||||
|
f"y positions too close: {sorted_ys}, gap={gap}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_paragraph_not_marked_no_merge(self, tmp_path):
|
||||||
|
"""A regular paragraph (no background drawing) should NOT
|
||||||
|
be marked as no_merge. The merge logic should still combine
|
||||||
|
consecutive lines into a single block."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
# 3 lines, no background drawing
|
||||||
|
page.insert_text((72, 100), "Line 1", fontsize=12)
|
||||||
|
page.insert_text((72, 120), "Line 2", fontsize=12)
|
||||||
|
page.insert_text((72, 140), "Line 3", fontsize=12)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input_para.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
in_doc = fitz.open(str(in_path))
|
||||||
|
blocks = translator._extract_text_blocks(in_doc[0])
|
||||||
|
merged = translator._merge_adjacent_blocks(blocks, in_doc[0].rect)
|
||||||
|
in_doc.close()
|
||||||
|
|
||||||
|
# All 3 lines should be merged into 1 block
|
||||||
|
assert len(merged) == 1, (
|
||||||
|
f"Expected 1 merged paragraph, got {len(merged)}"
|
||||||
|
)
|
||||||
|
assert "Line 1" in merged[0]["text"]
|
||||||
|
assert "Line 2" in merged[0]["text"]
|
||||||
|
assert "Line 3" in merged[0]["text"]
|
||||||
|
|
||||||
|
def test_code_block_end_to_end_preserves_lines(self, tmp_path):
|
||||||
|
"""End-to-end: a code block translated must keep each line
|
||||||
|
at its original y position, inside the background drawing."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
# Background drawing
|
||||||
|
page.draw_rect(
|
||||||
|
fitz.Rect(72, 290, 540, 410),
|
||||||
|
color=(0.9, 0.9, 0.9),
|
||||||
|
fill=(0.95, 0.95, 0.95),
|
||||||
|
)
|
||||||
|
# 5 code lines (use larger spacing for reliable separation)
|
||||||
|
code_lines = [
|
||||||
|
"$ npm install",
|
||||||
|
"$ npm start",
|
||||||
|
"$ curl http://localhost:3000",
|
||||||
|
"$ echo done",
|
||||||
|
"$ exit",
|
||||||
|
]
|
||||||
|
for i, line in enumerate(code_lines):
|
||||||
|
page.insert_text((85, 305 + i * 20), line, fontsize=10)
|
||||||
|
|
||||||
|
in_path = tmp_path / "input_code_e2e.pdf"
|
||||||
|
out_path = tmp_path / "output_code_e2e.pdf"
|
||||||
|
doc.save(str(in_path))
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
from translators.pdf_translator import PDFTranslator
|
||||||
|
translator = PDFTranslator()
|
||||||
|
|
||||||
|
def _flex(*args, **kwargs):
|
||||||
|
if args and hasattr(args[0], "__iter__") and not isinstance(args[0], str):
|
||||||
|
first = args[0][0] if len(args[0]) else None
|
||||||
|
if first is not None and hasattr(first, "text"):
|
||||||
|
class _R:
|
||||||
|
def __init__(self, t): self.translated_text = t
|
||||||
|
return [_R(r.text) for r in args[0]]
|
||||||
|
return list(args[0]) if args else []
|
||||||
|
|
||||||
|
mock = MagicMock()
|
||||||
|
mock.__class__.__name__ = "MagicMock"
|
||||||
|
mock.translate_batch = MagicMock(side_effect=_flex)
|
||||||
|
translator.set_provider(mock)
|
||||||
|
translator.translate_file(
|
||||||
|
in_path, out_path,
|
||||||
|
target_language="fr",
|
||||||
|
source_language="en",
|
||||||
|
)
|
||||||
|
|
||||||
|
out_doc = fitz.open(str(out_path))
|
||||||
|
out_page = out_doc[0]
|
||||||
|
text_blocks = [
|
||||||
|
b for b in out_page.get_text("dict").get("blocks", [])
|
||||||
|
if b.get("type") == 0
|
||||||
|
]
|
||||||
|
# Should have 5 separate code-line blocks, each at its own y
|
||||||
|
code_blocks = [
|
||||||
|
b for b in text_blocks
|
||||||
|
if b["bbox"][0] == 85 and b["bbox"][1] >= 290
|
||||||
|
]
|
||||||
|
assert len(code_blocks) == 5, (
|
||||||
|
f"Expected 5 separate code lines in output, got {len(code_blocks)}"
|
||||||
|
)
|
||||||
|
# Each line should be at a unique y position (no merging)
|
||||||
|
y_positions = [b["bbox"][1] for b in code_blocks]
|
||||||
|
assert len(set(y_positions)) == 5, (
|
||||||
|
f"Code lines were collapsed to fewer y positions: {y_positions}"
|
||||||
|
)
|
||||||
|
# All code lines should be inside the background drawing
|
||||||
|
for b in code_blocks:
|
||||||
|
y = b["bbox"][1]
|
||||||
|
assert 290 <= y <= 400, (
|
||||||
|
f"Code line at y={y} is outside the background drawing "
|
||||||
|
f"(y range 290-410)"
|
||||||
|
)
|
||||||
|
out_doc.close()
|
||||||
354
tests/test_translators/test_b3_pdf_fixes.py
Normal file
354
tests/test_translators/test_b3_pdf_fixes.py
Normal file
@@ -0,0 +1,354 @@
|
|||||||
|
"""
|
||||||
|
Tests for Track B3 — PDF format preservation fixes.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
B3.1 — Hyperlink preservation: links inside redacted text areas are
|
||||||
|
re-inserted after redaction so the page remains navigable.
|
||||||
|
B3.2 — Safe redaction: PDF_REDACT_IMAGE_NONE is used so images
|
||||||
|
intersecting the redaction area are NOT rasterized away.
|
||||||
|
B3.3 — Block merge: paragraph-spanning blocks are merged into one
|
||||||
|
before translation (better context, less font shrinking).
|
||||||
|
B3.4 — LibreOffice absence is logged with hint, not just a warning.
|
||||||
|
B3.5 — _record_format_loss_metric and _libreoffice_available helpers
|
||||||
|
never raise.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import tempfile
|
||||||
|
import importlib.util
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load pdf_translator directly to avoid pulling the heavy app imports
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_REPO_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||||
|
|
||||||
|
|
||||||
|
def _load_pdf_translator():
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"pdf_translator_under_test",
|
||||||
|
_REPO_ROOT / "translators" / "pdf_translator.py",
|
||||||
|
)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
return mod
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def pdf_mod():
|
||||||
|
return _load_pdf_translator()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B3.1 — Hyperlink preservation
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPdfHyperlinkPreservation:
|
||||||
|
"""A PDF page with a hyperlink should keep the link after the
|
||||||
|
in-place text replacement."""
|
||||||
|
|
||||||
|
def test_hyperlink_captured_before_redaction(self, pdf_mod):
|
||||||
|
"""The pipeline must read links BEFORE applying redactions."""
|
||||||
|
# Build a fake page
|
||||||
|
fake_page = MagicMock()
|
||||||
|
fake_page.get_links.return_value = [
|
||||||
|
{"uri": "https://example.com", "from": MagicMock()}
|
||||||
|
]
|
||||||
|
# The from_rect needs to be a real fitz.Rect for the page_rect.intersects() check
|
||||||
|
import fitz
|
||||||
|
fake_page.get_links.return_value = [
|
||||||
|
{"uri": "https://example.com", "from": fitz.Rect(0, 0, 100, 20)}
|
||||||
|
]
|
||||||
|
fake_page.rect = fitz.Rect(0, 0, 612, 792)
|
||||||
|
|
||||||
|
links = list(fake_page.get_links())
|
||||||
|
assert len(links) == 1
|
||||||
|
assert links[0]["uri"] == "https://example.com"
|
||||||
|
|
||||||
|
def test_internal_goto_link_recognized(self, pdf_mod):
|
||||||
|
"""An internal goto link (page number) should be re-insertable."""
|
||||||
|
import fitz
|
||||||
|
link = {
|
||||||
|
"kind": fitz.LINK_GOTO,
|
||||||
|
"from": fitz.Rect(10, 10, 80, 30),
|
||||||
|
"page": 5,
|
||||||
|
"to": fitz.Point(0, 0),
|
||||||
|
}
|
||||||
|
# The branching logic in the PDF translator should recognize
|
||||||
|
# `link.get("page") is not None` as a goto link
|
||||||
|
assert link.get("page") is not None
|
||||||
|
assert "uri" not in link
|
||||||
|
|
||||||
|
def test_external_uri_link_recognized(self, pdf_mod):
|
||||||
|
"""An external URI link should be re-insertable."""
|
||||||
|
import fitz
|
||||||
|
link = {
|
||||||
|
"kind": fitz.LINK_URI,
|
||||||
|
"from": fitz.Rect(10, 10, 80, 30),
|
||||||
|
"uri": "https://example.com",
|
||||||
|
}
|
||||||
|
assert link.get("uri") is not None
|
||||||
|
|
||||||
|
def test_link_without_uri_or_page_is_lost(self, pdf_mod):
|
||||||
|
"""A link with no URI and no target page can't be re-inserted —
|
||||||
|
it should be counted as lost, not crash the pipeline."""
|
||||||
|
link = {"from": MagicMock()}
|
||||||
|
# The branching should fall through to the `lost += 1` branch
|
||||||
|
assert "uri" not in link
|
||||||
|
assert link.get("page") is None
|
||||||
|
|
||||||
|
def test_page_insert_link_uri(self, tmp_path):
|
||||||
|
"""Smoke test: insert_link accepts the URI shape we build."""
|
||||||
|
import fitz
|
||||||
|
# Create an empty PDF in memory
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
rect = fitz.Rect(50, 50, 200, 70)
|
||||||
|
# The actual insertion
|
||||||
|
page.insert_link({
|
||||||
|
"kind": fitz.LINK_URI,
|
||||||
|
"from": rect,
|
||||||
|
"uri": "https://example.com",
|
||||||
|
})
|
||||||
|
# Save and re-open to verify persistence (PyMuPDF only shows
|
||||||
|
# links after save in some cases).
|
||||||
|
out = tmp_path / "test_uri.pdf"
|
||||||
|
doc.save(str(out))
|
||||||
|
doc.close()
|
||||||
|
# Re-open and check
|
||||||
|
doc2 = fitz.open(str(out))
|
||||||
|
links = list(doc2[0].get_links())
|
||||||
|
doc2.close()
|
||||||
|
assert len(links) == 1
|
||||||
|
assert links[0].get("uri") == "https://example.com"
|
||||||
|
|
||||||
|
def test_page_insert_link_goto(self, tmp_path):
|
||||||
|
"""Smoke test: insert_link accepts the goto shape we build."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
# Create at least 2 pages first
|
||||||
|
doc.new_page()
|
||||||
|
page2 = doc.new_page()
|
||||||
|
# Now insert the link on page 2 (last created page is the
|
||||||
|
# only valid reference; the first page reference is invalid
|
||||||
|
# because PyMuPDF invalidates earlier page refs when new
|
||||||
|
# pages are added)
|
||||||
|
page2.insert_link({
|
||||||
|
"kind": fitz.LINK_GOTO,
|
||||||
|
"from": fitz.Rect(50, 50, 200, 70),
|
||||||
|
"page": 0,
|
||||||
|
"to": fitz.Point(0, 0),
|
||||||
|
})
|
||||||
|
out = tmp_path / "test_goto.pdf"
|
||||||
|
doc.save(str(out))
|
||||||
|
doc.close()
|
||||||
|
doc2 = fitz.open(str(out))
|
||||||
|
links = list(doc2[1].get_links()) # page 2 (index 1)
|
||||||
|
doc2.close()
|
||||||
|
assert len(links) == 1
|
||||||
|
assert links[0].get("page") == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B3.2 — Safe redaction (PDF_REDACT_IMAGE_NONE)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestSafeRedaction:
|
||||||
|
def test_redact_image_none_used(self):
|
||||||
|
"""The code MUST use PDF_REDACT_IMAGE_NONE so images intersecting
|
||||||
|
the redaction area are not rasterized away."""
|
||||||
|
import fitz
|
||||||
|
# The flag exists in PyMuPDF
|
||||||
|
assert hasattr(fitz, "PDF_REDACT_IMAGE_NONE")
|
||||||
|
# PDF_REDACT_IMAGE_NONE = 0 means "do not touch images in the
|
||||||
|
# redaction area". This is the SAFE choice — images, vector
|
||||||
|
# graphics, etc. stay intact.
|
||||||
|
# PDF_REDACT_IMAGE_OTHER (1) or PDF_REDACT_IMAGE_ALL (2) would
|
||||||
|
# rasterize the affected area and remove the image.
|
||||||
|
assert fitz.PDF_REDACT_IMAGE_NONE == 0
|
||||||
|
# The PDF translator must use the NONE flag.
|
||||||
|
import inspect
|
||||||
|
src = inspect.getsource(_load_pdf_translator())
|
||||||
|
assert "PDF_REDACT_IMAGE_NONE" in src, (
|
||||||
|
"pdf_translator.py should use PDF_REDACT_IMAGE_NONE for safe "
|
||||||
|
"redaction of image content."
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_redact_with_no_image_param(self):
|
||||||
|
"""The pipeline calls apply_redactions with images=PDF_REDACT_IMAGE_NONE.
|
||||||
|
We can verify by mocking and inspecting the call."""
|
||||||
|
import fitz
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page()
|
||||||
|
# Insert some text and add a redaction
|
||||||
|
page.insert_text((50, 50), "Hello world")
|
||||||
|
page.add_redact_annot(fitz.Rect(0, 0, 200, 100), fill=(1, 1, 1))
|
||||||
|
|
||||||
|
# Spy on apply_redactions
|
||||||
|
original = page.apply_redactions
|
||||||
|
called_with = {}
|
||||||
|
|
||||||
|
def spy(images=None, **kwargs):
|
||||||
|
called_with["images"] = images
|
||||||
|
called_with["kwargs"] = kwargs
|
||||||
|
return original(images=images, **kwargs)
|
||||||
|
|
||||||
|
with patch.object(page, "apply_redactions", side_effect=spy):
|
||||||
|
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
||||||
|
|
||||||
|
assert called_with["images"] == fitz.PDF_REDACT_IMAGE_NONE
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B3.3 — Block merge
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestBlockMerge:
|
||||||
|
def test_should_merge_same_paragraph(self, pdf_mod):
|
||||||
|
"""Two blocks with the same font, vertically close, and same x
|
||||||
|
should be merged."""
|
||||||
|
a = {
|
||||||
|
"bbox": (50, 100, 200, 120),
|
||||||
|
"font_size": 12,
|
||||||
|
"line_count": 1,
|
||||||
|
}
|
||||||
|
b = {
|
||||||
|
"bbox": (50, 125, 200, 145),
|
||||||
|
"font_size": 12,
|
||||||
|
"line_count": 1,
|
||||||
|
}
|
||||||
|
assert pdf_mod.PDFTranslator._should_merge_blocks(None, a, b) is True
|
||||||
|
|
||||||
|
def test_should_not_merge_different_size(self, pdf_mod):
|
||||||
|
"""Big font-size gap → don't merge."""
|
||||||
|
a = {"bbox": (50, 100, 200, 120), "font_size": 12, "line_count": 1}
|
||||||
|
b = {"bbox": (50, 125, 200, 200), "font_size": 24, "line_count": 1}
|
||||||
|
assert pdf_mod.PDFTranslator._should_merge_blocks(None, a, b) is False
|
||||||
|
|
||||||
|
def test_should_not_merge_different_x_origin(self, pdf_mod):
|
||||||
|
"""Far-apart x origins → don't merge (different columns)."""
|
||||||
|
a = {"bbox": (50, 100, 200, 120), "font_size": 12, "line_count": 1}
|
||||||
|
b = {"bbox": (300, 125, 450, 145), "font_size": 12, "line_count": 1}
|
||||||
|
assert pdf_mod.PDFTranslator._should_merge_blocks(None, a, b) is False
|
||||||
|
|
||||||
|
def test_should_not_merge_too_far_apart(self, pdf_mod):
|
||||||
|
"""Vertical gap > 1.5x line height → don't merge."""
|
||||||
|
a = {"bbox": (50, 100, 200, 120), "font_size": 12, "line_count": 1}
|
||||||
|
b = {"bbox": (50, 200, 200, 220), "font_size": 12, "line_count": 1}
|
||||||
|
# vertical_gap = 200 - 120 = 80; line_height = 12 * 1.4 = 16.8
|
||||||
|
# 80 > 16.8 * 1.5 = 25.2
|
||||||
|
assert pdf_mod.PDFTranslator._should_merge_blocks(None, a, b) is False
|
||||||
|
|
||||||
|
def test_merge_two_blocks(self, pdf_mod):
|
||||||
|
"""End-to-end merge: text combined, bbox expanded."""
|
||||||
|
blocks = [
|
||||||
|
{"bbox": (50, 100, 200, 120), "font_size": 12, "line_count": 1,
|
||||||
|
"text": "Line 1", "sub_bboxes": [(50, 100, 200, 120)]},
|
||||||
|
{"bbox": (50, 125, 200, 145), "font_size": 12, "line_count": 1,
|
||||||
|
"text": "Line 2", "sub_bboxes": [(50, 125, 200, 145)]},
|
||||||
|
]
|
||||||
|
translator = pdf_mod.PDFTranslator()
|
||||||
|
merged = translator._merge_adjacent_blocks(blocks, page_rect=None)
|
||||||
|
assert len(merged) == 1
|
||||||
|
assert "Line 1" in merged[0]["text"]
|
||||||
|
assert "Line 2" in merged[0]["text"]
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B3.4 — LibreOffice absence log
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestLibreOfficeAbsence:
|
||||||
|
def test_libreoffice_available_returns_bool(self, pdf_mod):
|
||||||
|
result = pdf_mod._libreoffice_available()
|
||||||
|
assert isinstance(result, bool)
|
||||||
|
|
||||||
|
def test_libreoffice_available_caches_result(self, pdf_mod):
|
||||||
|
"""Calling twice should NOT spawn two subprocesses (cached)."""
|
||||||
|
# Reset cache
|
||||||
|
pdf_mod._libreoffice_available._cache = None
|
||||||
|
with patch("subprocess.run") as mock_run:
|
||||||
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
|
r1 = pdf_mod._libreoffice_available()
|
||||||
|
r2 = pdf_mod._libreoffice_available()
|
||||||
|
# Only one subprocess call should have happened
|
||||||
|
assert mock_run.call_count == 1
|
||||||
|
assert r1 == r2 == True
|
||||||
|
|
||||||
|
def test_libreoffice_available_handles_missing(self, pdf_mod):
|
||||||
|
"""FileNotFoundError → False."""
|
||||||
|
pdf_mod._libreoffice_available._cache = None
|
||||||
|
with patch("subprocess.run", side_effect=FileNotFoundError):
|
||||||
|
result = pdf_mod._libreoffice_available()
|
||||||
|
assert result is False
|
||||||
|
|
||||||
|
def test_libreoffice_available_handles_timeout(self, pdf_mod):
|
||||||
|
"""TimeoutExpired → False (don't block the job)."""
|
||||||
|
import subprocess
|
||||||
|
pdf_mod._libreoffice_available._cache = None
|
||||||
|
with patch("subprocess.run", side_effect=subprocess.TimeoutExpired("lo", 5)):
|
||||||
|
result = pdf_mod._libreoffice_available()
|
||||||
|
assert result is False
|
||||||
|
|
||||||
|
def test_libreoffice_not_found_logs_hint(self, pdf_mod):
|
||||||
|
"""When LibreOffice is missing, the log should include a hint."""
|
||||||
|
# Patch the module-level logger's `warning` method to capture calls
|
||||||
|
captured = []
|
||||||
|
original_warning = pdf_mod.logger.warning
|
||||||
|
|
||||||
|
def spy_warning(*args, **kwargs):
|
||||||
|
captured.append((args, kwargs))
|
||||||
|
return original_warning(*args, **kwargs)
|
||||||
|
|
||||||
|
with patch.object(pdf_mod.logger, "warning", side_effect=spy_warning):
|
||||||
|
with patch.object(pdf_mod, "_libreoffice_available", return_value=False):
|
||||||
|
result = pdf_mod.PDFTranslator()._convert_docx_to_pdf(
|
||||||
|
Path("dummy.docx"), Path("dummy.pdf")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Returns None (no conversion happened)
|
||||||
|
assert result is None
|
||||||
|
# Check the log message includes the install hint
|
||||||
|
all_args = [str(a) + str(kwargs) for args, kwargs in captured for a in args]
|
||||||
|
all_args += [str(kwargs) for args, kwargs in captured]
|
||||||
|
flat = " ".join(all_args)
|
||||||
|
assert "install" in flat.lower() or "Install" in flat, (
|
||||||
|
f"No install hint in logs. Captured: {captured}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# B3.5 — Helper resilience
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestHelperResilience:
|
||||||
|
def test_record_format_loss_zero_count_is_noop(self, pdf_mod):
|
||||||
|
"""count=0 should be a no-op (no metric emitted)."""
|
||||||
|
# Should not raise
|
||||||
|
pdf_mod._record_format_loss_metric("hyperlink", count=0)
|
||||||
|
|
||||||
|
def test_record_format_loss_negative_count_is_noop(self, pdf_mod):
|
||||||
|
"""Negative count should be ignored."""
|
||||||
|
pdf_mod._record_format_loss_metric("hyperlink", count=-5)
|
||||||
|
|
||||||
|
def test_record_format_loss_with_broken_metrics(self, pdf_mod):
|
||||||
|
"""If middleware.metrics fails to import, the helper must NOT raise."""
|
||||||
|
with patch.dict(sys.modules, {"middleware.metrics": None}):
|
||||||
|
# Should swallow the ImportError
|
||||||
|
pdf_mod._record_format_loss_metric("hyperlink", count=1)
|
||||||
|
|
||||||
|
def test_libreoffice_available_cache_expires(self, pdf_mod):
|
||||||
|
"""After 5 minutes, the cache should be considered stale."""
|
||||||
|
pdf_mod._libreoffice_available._cache = (time.time() - 301, True)
|
||||||
|
with patch("subprocess.run") as mock_run:
|
||||||
|
mock_run.return_value = MagicMock(returncode=1)
|
||||||
|
result = pdf_mod._libreoffice_available()
|
||||||
|
# Cache was stale → fresh subprocess call happened
|
||||||
|
assert mock_run.call_count == 1
|
||||||
@@ -479,11 +479,75 @@ class ExcelTranslator:
|
|||||||
worksheet: Worksheet,
|
worksheet: Worksheet,
|
||||||
text_elements: List[Tuple[str, Callable[[str], None]]],
|
text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Collect all translatable text from worksheet cells."""
|
"""Collect all translatable text from worksheet cells, cell comments,
|
||||||
|
and cell hyperlinks (URLs are preserved as-is, but their display
|
||||||
|
labels are translated)."""
|
||||||
for row in worksheet.iter_rows():
|
for row in worksheet.iter_rows():
|
||||||
for cell in row:
|
for cell in row:
|
||||||
if cell.value is not None:
|
if cell.value is not None:
|
||||||
self._collect_from_cell(cell, text_elements)
|
self._collect_from_cell(cell, text_elements)
|
||||||
|
# Cell comments: visible to users when hovering the cell.
|
||||||
|
if cell.comment is not None and cell.comment.text:
|
||||||
|
self._collect_from_cell_comment(cell, text_elements)
|
||||||
|
# Cell hyperlinks: the URL is preserved; the display label
|
||||||
|
# (when different from the URL) is translatable.
|
||||||
|
if cell.hyperlink is not None:
|
||||||
|
self._collect_from_cell_hyperlink(cell, text_elements)
|
||||||
|
|
||||||
|
def _collect_from_cell_comment(
|
||||||
|
self, cell, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||||
|
) -> None:
|
||||||
|
"""Collect text from a cell comment."""
|
||||||
|
if cell.comment is None or not cell.comment.text:
|
||||||
|
return
|
||||||
|
original = cell.comment.text
|
||||||
|
if not original.strip():
|
||||||
|
return
|
||||||
|
|
||||||
|
def make_comment_setter(c):
|
||||||
|
def setter(text: str) -> None:
|
||||||
|
# The comment object may have been replaced if openpyxl
|
||||||
|
# rebuilt it. Replace the text in place.
|
||||||
|
try:
|
||||||
|
c.comment.text = text
|
||||||
|
except Exception:
|
||||||
|
# Re-create the comment if direct assignment fails
|
||||||
|
from openpyxl.comments import Comment
|
||||||
|
c.comment = Comment(text, c.comment.author or "Translator")
|
||||||
|
return setter
|
||||||
|
|
||||||
|
text_elements.append((original, make_comment_setter(cell)))
|
||||||
|
|
||||||
|
def _collect_from_cell_hyperlink(
|
||||||
|
self, cell, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
|
) -> None:
|
||||||
|
"""Collect the DISPLAY LABEL of a cell hyperlink (not the URL).
|
||||||
|
|
||||||
|
The URL itself is preserved as-is — we only translate the visible
|
||||||
|
label, which is typically the cell's value when it differs from the
|
||||||
|
URL (e.g. a clickable "Click here" pointing to https://example.com).
|
||||||
|
"""
|
||||||
|
if cell.hyperlink is None:
|
||||||
|
return
|
||||||
|
# The display target is what shows up in the cell when read.
|
||||||
|
# openpyxl exposes `hyperlink.display` or `hyperlink.target`.
|
||||||
|
display = getattr(cell.hyperlink, "display", None) or getattr(cell.hyperlink, "target", None)
|
||||||
|
if not display or not str(display).strip():
|
||||||
|
return
|
||||||
|
# If the display matches the cell value, the value setter already
|
||||||
|
# handles it — don't double-translate.
|
||||||
|
if str(display).strip() == str(cell.value or "").strip():
|
||||||
|
return
|
||||||
|
|
||||||
|
def make_hyperlink_setter(hl):
|
||||||
|
def setter(text: str) -> None:
|
||||||
|
try:
|
||||||
|
hl.display = text
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return setter
|
||||||
|
|
||||||
|
text_elements.append((str(display), make_hyperlink_setter(cell.hyperlink)))
|
||||||
|
|
||||||
def _collect_from_cell(
|
def _collect_from_cell(
|
||||||
self, cell: Cell, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, cell: Cell, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||||
@@ -568,7 +632,13 @@ class ExcelTranslator:
|
|||||||
self, input_path: Path, text_elements: List[Tuple[str, Callable[[str], None]]],
|
self, input_path: Path, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
chart_translations: List[Dict[str, Any]]
|
chart_translations: List[Dict[str, Any]]
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Parse chart XML from the .xlsx ZIP and collect translatable text."""
|
"""Parse chart XML from the .xlsx ZIP and collect translatable text.
|
||||||
|
|
||||||
|
Each translatable element is tracked by its `element_path` (an
|
||||||
|
XPath-like path computed at collect time) so the apply step can
|
||||||
|
target it precisely even when the same text appears multiple times
|
||||||
|
in the same chart.
|
||||||
|
"""
|
||||||
_NS_A = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
_NS_A = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
||||||
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
|
||||||
@@ -579,16 +649,16 @@ class ExcelTranslator:
|
|||||||
for chart_file in chart_files:
|
for chart_file in chart_files:
|
||||||
try:
|
try:
|
||||||
chart_xml = etree.fromstring(zf.read(chart_file))
|
chart_xml = etree.fromstring(zf.read(chart_file))
|
||||||
seen_texts: set = set()
|
|
||||||
|
|
||||||
# Collect from <a:t> elements (titles, axis labels, legend text)
|
# Collect <a:t> elements (titles, axis labels, legend text)
|
||||||
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
||||||
if t_elem.text and t_elem.text.strip() and t_elem.text.strip() not in seen_texts:
|
if t_elem.text and t_elem.text.strip():
|
||||||
seen_texts.add(t_elem.text.strip())
|
|
||||||
entry = {
|
entry = {
|
||||||
'chart_file': chart_file,
|
'chart_file': chart_file,
|
||||||
'original': t_elem.text.strip(),
|
'original': t_elem.text.strip(),
|
||||||
'translated': None,
|
'translated': None,
|
||||||
|
'tag': 'a:t',
|
||||||
|
'element_path': self._get_chart_element_path(chart_xml, t_elem),
|
||||||
}
|
}
|
||||||
chart_translations.append(entry)
|
chart_translations.append(entry)
|
||||||
|
|
||||||
@@ -601,16 +671,16 @@ class ExcelTranslator:
|
|||||||
(t_elem.text.strip(), make_chart_setter(chart_translations, len(chart_translations) - 1))
|
(t_elem.text.strip(), make_chart_setter(chart_translations, len(chart_translations) - 1))
|
||||||
)
|
)
|
||||||
|
|
||||||
# Collect from <c:v> elements (category names, series names)
|
# Collect <c:v> elements (category names, series names)
|
||||||
for v_elem in chart_xml.iter(f'{{{_NS_C}}}v'):
|
for v_elem in chart_xml.iter(f'{{{_NS_C}}}v'):
|
||||||
text = v_elem.text
|
text = v_elem.text
|
||||||
if text and text.strip() and not text.strip().replace('.', '').replace('-', '').replace(',', '').isdigit():
|
if text and text.strip() and not text.strip().replace('.', '').replace('-', '').replace(',', '').isdigit():
|
||||||
if text.strip() not in seen_texts:
|
|
||||||
seen_texts.add(text.strip())
|
|
||||||
entry = {
|
entry = {
|
||||||
'chart_file': chart_file,
|
'chart_file': chart_file,
|
||||||
'original': text.strip(),
|
'original': text.strip(),
|
||||||
'translated': None,
|
'translated': None,
|
||||||
|
'tag': 'c:v',
|
||||||
|
'element_path': self._get_chart_element_path(chart_xml, v_elem),
|
||||||
}
|
}
|
||||||
chart_translations.append(entry)
|
chart_translations.append(entry)
|
||||||
|
|
||||||
@@ -629,8 +699,53 @@ class ExcelTranslator:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
_log_error("excel_charts_zip_error", error=str(e))
|
_log_error("excel_charts_zip_error", error=str(e))
|
||||||
|
|
||||||
|
def _get_chart_element_path(self, root, target_element) -> str:
|
||||||
|
"""Compute an XPath-like path that uniquely identifies target_element within root."""
|
||||||
|
path_parts: List[str] = []
|
||||||
|
current = target_element
|
||||||
|
while current is not None:
|
||||||
|
parent = current.getparent()
|
||||||
|
if parent is None:
|
||||||
|
break
|
||||||
|
idx = list(parent).index(current)
|
||||||
|
tag = current.tag.split('}')[-1] if '}' in current.tag else current.tag
|
||||||
|
path_parts.append(f"{tag}[{idx}]")
|
||||||
|
current = parent
|
||||||
|
return '/'.join(reversed(path_parts))
|
||||||
|
|
||||||
|
def _find_chart_element_by_path(self, root, path: str):
|
||||||
|
"""Navigate the XML tree using the path produced by _get_chart_element_path."""
|
||||||
|
if not path:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
current = root
|
||||||
|
for segment in path.split('/'):
|
||||||
|
if '[' not in segment or not segment.endswith(']'):
|
||||||
|
return None
|
||||||
|
tag_name, idx_str = segment[:-1].split('[', 1)
|
||||||
|
try:
|
||||||
|
idx = int(idx_str)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
children = list(current)
|
||||||
|
if idx < 0 or idx >= len(children):
|
||||||
|
return None
|
||||||
|
candidate = children[idx]
|
||||||
|
candidate_tag = candidate.tag.split('}')[-1] if '}' in candidate.tag else candidate.tag
|
||||||
|
if candidate_tag != tag_name:
|
||||||
|
return None
|
||||||
|
current = candidate
|
||||||
|
return current
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
def _apply_chart_translations(self, output_path: Path, chart_translations: List[Dict[str, Any]]) -> None:
|
def _apply_chart_translations(self, output_path: Path, chart_translations: List[Dict[str, Any]]) -> None:
|
||||||
"""Re-inject chart translations into the .xlsx ZIP."""
|
"""Re-inject chart translations into the .xlsx ZIP.
|
||||||
|
|
||||||
|
Uses the `element_path` collected during `_collect_charts_from_zip`
|
||||||
|
to target each element precisely. Falls back to string matching
|
||||||
|
for entries that predate the path-based collection.
|
||||||
|
"""
|
||||||
if not chart_translations:
|
if not chart_translations:
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -660,6 +775,11 @@ class ExcelTranslator:
|
|||||||
try:
|
try:
|
||||||
chart_xml = etree.fromstring(data)
|
chart_xml = etree.fromstring(data)
|
||||||
for entry in chart_files_to_update[item]:
|
for entry in chart_files_to_update[item]:
|
||||||
|
target = self._find_chart_element_by_path(chart_xml, entry.get('element_path', ''))
|
||||||
|
if target is not None:
|
||||||
|
target.text = entry['translated']
|
||||||
|
else:
|
||||||
|
# Fallback for legacy entries without path
|
||||||
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
||||||
if t_elem.text and t_elem.text.strip() == entry['original']:
|
if t_elem.text and t_elem.text.strip() == entry['original']:
|
||||||
t_elem.text = entry['translated']
|
t_elem.text = entry['translated']
|
||||||
|
|||||||
@@ -37,12 +37,71 @@ logger = get_logger(__name__)
|
|||||||
MIN_FONT_SIZE = 4.5
|
MIN_FONT_SIZE = 4.5
|
||||||
|
|
||||||
# Font size reduction factor when text overflows its bounding box
|
# Font size reduction factor when text overflows its bounding box
|
||||||
FONT_SHRINK_FACTOR = 0.87
|
FONT_SHRINK_FACTOR = 0.93 # Track B3.5: was 0.87 (too aggressive, broke hierarchy)
|
||||||
|
|
||||||
|
# Maximum vertical expansion factor (relative to original bbox height)
|
||||||
|
# Track B3.5: was 1.5x; bumped to 3x so longer translations can flow down
|
||||||
|
# Track B3.6: bumped to 6x to handle long French titles that wrap to
|
||||||
|
# multiple lines (a 22pt title may need 4-5 lines in French).
|
||||||
|
MAX_VERTICAL_EXPANSION = 6.0
|
||||||
|
|
||||||
|
# Maximum font shrink for headings (font >= HEADING_MIN_SIZE points)
|
||||||
|
# Track B3.5: never shrink a heading below 90% of its original size
|
||||||
|
HEADING_MIN_SIZE = 14.0
|
||||||
|
HEADING_MIN_SCALE = 0.90
|
||||||
|
|
||||||
|
# Maximum font shrink for body text
|
||||||
|
BODY_MIN_SCALE = 0.75
|
||||||
|
|
||||||
# RTL language codes
|
# RTL language codes
|
||||||
RTL_LANGUAGES = frozenset({"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"})
|
RTL_LANGUAGES = frozenset({"ar", "he", "fa", "ur", "ku", "ps", "ug", "sd", "yi", "dv", "ckb"})
|
||||||
|
|
||||||
|
|
||||||
|
def _record_format_loss_metric(element_type: str, count: int = 1) -> None:
|
||||||
|
"""Best-effort emission of a format-loss metric (never raises).
|
||||||
|
|
||||||
|
Used by the PDF translator to track hyperlinks/annotations lost
|
||||||
|
during in-place text replacement.
|
||||||
|
"""
|
||||||
|
if count <= 0:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
from middleware.metrics import record_format_loss
|
||||||
|
for _ in range(count):
|
||||||
|
record_format_loss(format="pdf", element_type=element_type)
|
||||||
|
except Exception:
|
||||||
|
# Metrics must never break a job
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _libreoffice_available() -> bool:
|
||||||
|
"""Check if the LibreOffice CLI is available on this host.
|
||||||
|
|
||||||
|
Result is cached for 5 minutes to avoid spawning a subprocess
|
||||||
|
on every PDF translation.
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
now = time.time()
|
||||||
|
cached = _libreoffice_available._cache
|
||||||
|
if cached and (now - cached[0]) < 300:
|
||||||
|
return cached[1]
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["libreoffice", "--version"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=5,
|
||||||
|
)
|
||||||
|
ok = result.returncode == 0
|
||||||
|
except (FileNotFoundError, subprocess.TimeoutExpired, Exception):
|
||||||
|
ok = False
|
||||||
|
_libreoffice_available._cache = (now, ok)
|
||||||
|
return ok
|
||||||
|
|
||||||
|
|
||||||
|
_libreoffice_available._cache = None # type: ignore[attr-defined]
|
||||||
|
|
||||||
|
|
||||||
class PDFTranslator:
|
class PDFTranslator:
|
||||||
"""Translates PDF files with layout preservation using PyMuPDF."""
|
"""Translates PDF files with layout preservation using PyMuPDF."""
|
||||||
|
|
||||||
@@ -217,6 +276,13 @@ class PDFTranslator:
|
|||||||
blocks = self._merge_adjacent_blocks(raw_blocks, page.rect)
|
blocks = self._merge_adjacent_blocks(raw_blocks, page.rect)
|
||||||
total_blocks += len(blocks)
|
total_blocks += len(blocks)
|
||||||
|
|
||||||
|
# Track B3.7: compute each block's "next block" y0 so the
|
||||||
|
# smart-fit logic never expands a block into its neighbour.
|
||||||
|
# Without this, a long translated block can flow into the
|
||||||
|
# block below it (e.g. "Pour la dernière version" overlapping
|
||||||
|
# "8. Résolution des problèmes").
|
||||||
|
self._populate_next_block_y(blocks, page.rect)
|
||||||
|
|
||||||
# Phase 1: translate all blocks on this page
|
# Phase 1: translate all blocks on this page
|
||||||
for block in blocks:
|
for block in blocks:
|
||||||
original = block["text"]
|
original = block["text"]
|
||||||
@@ -244,13 +310,121 @@ class PDFTranslator:
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Phase 2: redact original text areas
|
# Phase 2: redact original text areas
|
||||||
|
#
|
||||||
|
# CRITICAL (Track B3.6): For blocks that contain a vector
|
||||||
|
# drawing (e.g. a colored background box), we need to use a
|
||||||
|
# TRANSPARENT redaction (fill=None) so the original drawing
|
||||||
|
# isn't erased. For all other blocks, we use the standard
|
||||||
|
# white redaction to fully erase the text.
|
||||||
|
#
|
||||||
|
# Before redaction, we also capture the page's hyperlinks
|
||||||
|
# (PDF links are not stored in text blocks; they're attached
|
||||||
|
# to page annotations). After redaction, we re-insert the
|
||||||
|
# links so the translated page remains navigable.
|
||||||
|
#
|
||||||
|
# PDF_REDACT_IMAGE_NONE is critical: it tells PyMuPDF NOT to
|
||||||
|
# rasterize image content that intersects redaction areas.
|
||||||
|
# Without this flag, the redaction would replace the text
|
||||||
|
# AND the underlying image with a blank rectangle — wiping
|
||||||
|
# out any diagrams, charts, or photos in the same area.
|
||||||
|
page_links_before: list = []
|
||||||
|
if hasattr(page, "get_links"):
|
||||||
|
try:
|
||||||
|
page_links_before = list(page.get_links())
|
||||||
|
except Exception:
|
||||||
|
page_links_before = []
|
||||||
|
|
||||||
|
# Find the rectangles of all vector drawings on the page.
|
||||||
|
# If a block's bbox intersects a drawing rect, we use
|
||||||
|
# transparent redaction to preserve the drawing.
|
||||||
|
drawing_rects: list = []
|
||||||
|
if hasattr(page, "get_drawings"):
|
||||||
|
try:
|
||||||
|
for d in page.get_drawings():
|
||||||
|
r = d.get("rect")
|
||||||
|
if r is not None and d.get("fill") is not None:
|
||||||
|
drawing_rects.append(fitz.Rect(r))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _block_intersects_drawing(block_bbox):
|
||||||
|
"""True if the block's bbox intersects any drawing rect."""
|
||||||
|
if not drawing_rects:
|
||||||
|
return False
|
||||||
|
b = fitz.Rect(block_bbox)
|
||||||
|
for dr in drawing_rects:
|
||||||
|
if b.intersects(dr):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
for block in blocks:
|
for block in blocks:
|
||||||
if block.get("translated"):
|
if block.get("translated"):
|
||||||
for sub_rect in block["sub_bboxes"]:
|
# Track B3.5: single redaction per block, not per sub-bbox.
|
||||||
page.add_redact_annot(fitz.Rect(sub_rect), fill=(1, 1, 1))
|
# Track B3.6: if the block contains a drawing, use
|
||||||
|
# transparent redaction (fill=None) so the drawing
|
||||||
|
# underneath isn't erased. Otherwise use white fill
|
||||||
|
# to fully remove the original text.
|
||||||
|
block_bbox = fitz.Rect(block["bbox"])
|
||||||
|
if _block_intersects_drawing(block_bbox):
|
||||||
|
# Transparent redaction: removes text but keeps
|
||||||
|
# the underlying drawing intact.
|
||||||
|
page.add_redact_annot(block_bbox, fill=None)
|
||||||
|
else:
|
||||||
|
page.add_redact_annot(block_bbox, fill=(1, 1, 1))
|
||||||
|
|
||||||
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
|
||||||
|
|
||||||
|
# Re-insert hyperlinks that were inside the redacted area.
|
||||||
|
# We use the original link geometry (it's unaffected by the
|
||||||
|
# redaction). URIs are preserved verbatim; only the visible
|
||||||
|
# text changes.
|
||||||
|
if page_links_before:
|
||||||
|
reinserted = 0
|
||||||
|
lost = 0
|
||||||
|
page_rect = page.rect
|
||||||
|
for link in page_links_before:
|
||||||
|
try:
|
||||||
|
from_rect = link.get("from")
|
||||||
|
if from_rect is None:
|
||||||
|
lost += 1
|
||||||
|
continue
|
||||||
|
# Clamp the rect to the page (in case redaction
|
||||||
|
# shrunk it)
|
||||||
|
if not page_rect.intersects(from_rect):
|
||||||
|
lost += 1
|
||||||
|
continue
|
||||||
|
# Build the link insertion kwargs based on kind
|
||||||
|
if link.get("uri"):
|
||||||
|
# External URI link
|
||||||
|
page.insert_link({
|
||||||
|
"kind": fitz.LINK_URI,
|
||||||
|
"from": from_rect,
|
||||||
|
"uri": link["uri"],
|
||||||
|
})
|
||||||
|
reinserted += 1
|
||||||
|
elif link.get("page") is not None:
|
||||||
|
# Internal goto link
|
||||||
|
page.insert_link({
|
||||||
|
"kind": fitz.LINK_GOTO,
|
||||||
|
"from": from_rect,
|
||||||
|
"page": link["page"],
|
||||||
|
"to": link.get("to", fitz.Point(0, 0)),
|
||||||
|
})
|
||||||
|
reinserted += 1
|
||||||
|
else:
|
||||||
|
lost += 1
|
||||||
|
except Exception:
|
||||||
|
lost += 1
|
||||||
|
if reinserted > 0:
|
||||||
|
logger.info(
|
||||||
|
"pdf_hyperlinks_reinserted",
|
||||||
|
page=page_num + 1,
|
||||||
|
reinserted=reinserted,
|
||||||
|
lost=lost,
|
||||||
|
)
|
||||||
|
if lost > 0:
|
||||||
|
_record_format_loss_metric("hyperlink", count=lost)
|
||||||
|
|
||||||
# Phase 3: write translated text
|
# Phase 3: write translated text
|
||||||
for block in blocks:
|
for block in blocks:
|
||||||
if block.get("translated"):
|
if block.get("translated"):
|
||||||
@@ -280,12 +454,76 @@ class PDFTranslator:
|
|||||||
return output_path
|
return output_path
|
||||||
|
|
||||||
def _extract_text_blocks(self, page) -> List[Dict]:
|
def _extract_text_blocks(self, page) -> List[Dict]:
|
||||||
"""Extract text blocks with position, font, and color information."""
|
"""Extract text blocks with position, font, and color information.
|
||||||
|
|
||||||
|
Track B3.9: when a PDF "block" contains multiple LINES at the
|
||||||
|
same y (typical of table rows where each cell is a separate line
|
||||||
|
in the block), split it into one sub-block per line. This
|
||||||
|
preserves the column structure so a translated table row stays
|
||||||
|
a table row instead of collapsing into a single left-aligned
|
||||||
|
column.
|
||||||
|
|
||||||
|
Track B3.10: when a block is COVERED by a drawing (code block
|
||||||
|
with a background rect, callout box, etc.), split it into
|
||||||
|
per-line blocks and mark them as ``no_merge``. Without this,
|
||||||
|
the merge logic collapses 5 code lines into one block, and the
|
||||||
|
smart-fit rewrites them all at the block's bbox, which
|
||||||
|
destroys the line-by-line layout and the surrounding
|
||||||
|
background drawing's apparent extent.
|
||||||
|
|
||||||
|
The detection is: if a block has >= 2 lines whose y0 is within
|
||||||
|
``SAME_ROW_Y_TOLERANCE`` of each other, treat it as a
|
||||||
|
multi-column row and emit one sub-block per line. Blocks whose
|
||||||
|
lines are vertically stacked (multi-line paragraphs) keep the
|
||||||
|
old behavior, UNLESS they are covered by a drawing — in which
|
||||||
|
case we split per-line and set ``no_merge`` to preserve the
|
||||||
|
code-block / callout layout.
|
||||||
|
"""
|
||||||
import fitz
|
import fitz
|
||||||
|
|
||||||
blocks = []
|
blocks = []
|
||||||
data = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)
|
data = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)
|
||||||
|
|
||||||
|
SAME_ROW_Y_TOLERANCE = 3.0 # points
|
||||||
|
|
||||||
|
# B3.10: pre-compute the set of page drawings with a fill
|
||||||
|
# (colored rects, callout boxes, etc.) so we can detect
|
||||||
|
# "code block" or "callout" patterns and avoid merging their
|
||||||
|
# constituent lines.
|
||||||
|
drawing_rects: list = []
|
||||||
|
if hasattr(page, "get_drawings"):
|
||||||
|
try:
|
||||||
|
for d in page.get_drawings():
|
||||||
|
r = d.get("rect")
|
||||||
|
if r is not None and d.get("fill") is not None:
|
||||||
|
drawing_rects.append(fitz.Rect(r))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _block_covered_by_drawing(block_bbox_tuple) -> bool:
|
||||||
|
"""True if the block's bbox is covered by a filled drawing.
|
||||||
|
|
||||||
|
We use "covered" loosely — a substantial intersection
|
||||||
|
(>= 50% of the block's area) qualifies. This avoids
|
||||||
|
false positives from drawings that merely touch a corner
|
||||||
|
of the block.
|
||||||
|
"""
|
||||||
|
if not drawing_rects:
|
||||||
|
return False
|
||||||
|
b = fitz.Rect(block_bbox_tuple)
|
||||||
|
block_area = b.width * b.height
|
||||||
|
if block_area <= 0:
|
||||||
|
return False
|
||||||
|
for dr in drawing_rects:
|
||||||
|
if not b.intersects(dr):
|
||||||
|
continue
|
||||||
|
# Compute intersection area
|
||||||
|
inter = b & dr
|
||||||
|
inter_area = inter.width * inter.height
|
||||||
|
if inter_area / block_area >= 0.5:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
for block in data.get("blocks", []):
|
for block in data.get("blocks", []):
|
||||||
if block.get("type") != 0:
|
if block.get("type") != 0:
|
||||||
continue
|
continue
|
||||||
@@ -294,6 +532,76 @@ class PDFTranslator:
|
|||||||
if not lines:
|
if not lines:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
# B3.9: detect table-row pattern. A row is multiple lines
|
||||||
|
# at the same y (different x). Multi-line paragraphs have
|
||||||
|
# lines at different y (same x).
|
||||||
|
line_y0s = [line["bbox"][1] for line in lines]
|
||||||
|
same_y = all(
|
||||||
|
abs(y - line_y0s[0]) <= SAME_ROW_Y_TOLERANCE
|
||||||
|
for y in line_y0s
|
||||||
|
)
|
||||||
|
has_horizontal_layout = (
|
||||||
|
same_y and len(lines) > 1
|
||||||
|
and any(
|
||||||
|
abs(lines[i]["bbox"][0] - lines[0]["bbox"][0]) > 5
|
||||||
|
for i in range(1, len(lines))
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# B3.10: detect code-block / callout pattern — a block
|
||||||
|
# whose lines are covered by a colored background drawing.
|
||||||
|
# Note: each code line is its own PDF block, so we check
|
||||||
|
# the block bbox (which is the line's bbox) against the
|
||||||
|
# drawing, not "len(lines) > 1".
|
||||||
|
covered_by_drawing = (
|
||||||
|
not has_horizontal_layout
|
||||||
|
and _block_covered_by_drawing(block["bbox"])
|
||||||
|
)
|
||||||
|
|
||||||
|
if has_horizontal_layout or covered_by_drawing:
|
||||||
|
# Multi-column row OR code block: emit one block per line.
|
||||||
|
for line in lines:
|
||||||
|
span_parts = []
|
||||||
|
spans_info = []
|
||||||
|
for span in line.get("spans", []):
|
||||||
|
text = span.get("text", "")
|
||||||
|
if text:
|
||||||
|
span_parts.append(text)
|
||||||
|
spans_info.append({
|
||||||
|
"size": span.get("size", 12),
|
||||||
|
"font": span.get("font", "Helvetica"),
|
||||||
|
"color": span.get("color", 0),
|
||||||
|
"flags": span.get("flags", 0),
|
||||||
|
"origin": span.get("origin", (0, 0)),
|
||||||
|
})
|
||||||
|
if not span_parts:
|
||||||
|
continue
|
||||||
|
line_text = "".join(span_parts).strip()
|
||||||
|
if not line_text:
|
||||||
|
continue
|
||||||
|
avg_size = (
|
||||||
|
sum(s["size"] for s in spans_info) / len(spans_info)
|
||||||
|
if spans_info else 12.0
|
||||||
|
)
|
||||||
|
first_color = spans_info[0]["color"] if spans_info else 0
|
||||||
|
is_bold = any(s["flags"] & 16 for s in spans_info)
|
||||||
|
is_italic = any(s["flags"] & 2 for s in spans_info)
|
||||||
|
blocks.append({
|
||||||
|
"bbox": tuple(line["bbox"]),
|
||||||
|
"text": line_text,
|
||||||
|
"font_size": round(avg_size, 1),
|
||||||
|
"color": first_color,
|
||||||
|
"is_bold": is_bold,
|
||||||
|
"is_italic": is_italic,
|
||||||
|
"line_count": 1,
|
||||||
|
"translated": None,
|
||||||
|
"sub_bboxes": [tuple(line["bbox"])],
|
||||||
|
"_is_table_cell": has_horizontal_layout,
|
||||||
|
"_no_merge": covered_by_drawing,
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Standard multi-line paragraph: join lines with \n.
|
||||||
line_parts = []
|
line_parts = []
|
||||||
spans_info = []
|
spans_info = []
|
||||||
|
|
||||||
@@ -383,6 +691,15 @@ class PDFTranslator:
|
|||||||
a_bbox = a["bbox"]
|
a_bbox = a["bbox"]
|
||||||
b_bbox = b["bbox"]
|
b_bbox = b["bbox"]
|
||||||
|
|
||||||
|
# Track B3.10: never merge blocks marked as no_merge.
|
||||||
|
# These are lines inside a code block, callout box, or any
|
||||||
|
# other structure that needs to preserve its line-by-line
|
||||||
|
# layout (e.g. each line of a curl command must stay at its
|
||||||
|
# own y position to remain aligned with the background
|
||||||
|
# drawing that covers them).
|
||||||
|
if a.get("_no_merge") or b.get("_no_merge"):
|
||||||
|
return False
|
||||||
|
|
||||||
# Must have similar font size (within 20%)
|
# Must have similar font size (within 20%)
|
||||||
if abs(a["font_size"] - b["font_size"]) > max(a["font_size"], b["font_size"]) * 0.2:
|
if abs(a["font_size"] - b["font_size"]) > max(a["font_size"], b["font_size"]) * 0.2:
|
||||||
return False
|
return False
|
||||||
@@ -405,21 +722,84 @@ class PDFTranslator:
|
|||||||
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
def _populate_next_block_y(self, blocks: List[Dict], page_rect) -> None:
|
||||||
|
"""For each block, set ``block['_next_block_y']`` to the y0 of the
|
||||||
|
nearest block below it in the SAME COLUMN (or the page bottom
|
||||||
|
margin if none).
|
||||||
|
|
||||||
|
Track B3.7: this lets the smart-fit logic use the next block as
|
||||||
|
a ceiling for vertical expansion, so a long translated block
|
||||||
|
never overlaps its neighbour.
|
||||||
|
|
||||||
|
Track B3.8: COLUMN-AWARE. PDFs with multi-column layouts
|
||||||
|
(e.g. journal articles, brochures) would otherwise get
|
||||||
|
the wrong "next block" — sorting globally by y0 would map
|
||||||
|
a left-column block to a right-column block as its "next".
|
||||||
|
We group blocks into columns by x0 proximity (15pt tolerance),
|
||||||
|
then sort each column by y0. Blocks that don't fit any
|
||||||
|
column (e.g. full-width centered headers) are treated as
|
||||||
|
their own single-block column.
|
||||||
|
"""
|
||||||
|
margin = 18
|
||||||
|
page_bottom = page_rect.y1 - margin
|
||||||
|
COLUMN_TOLERANCE = 15.0 # points — blocks within this x0 are "same column"
|
||||||
|
|
||||||
|
if not blocks:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Group blocks into columns by x0 proximity. We iterate in
|
||||||
|
# x0-sorted order and assign each block to the first column
|
||||||
|
# whose reference x0 is within tolerance, or start a new column.
|
||||||
|
# This gives us columns ordered left-to-right.
|
||||||
|
columns: List[List[Dict]] = []
|
||||||
|
for block in sorted(blocks, key=lambda b: b["bbox"][0]):
|
||||||
|
assigned = False
|
||||||
|
for column in columns:
|
||||||
|
col_ref_x0 = column[0]["bbox"][0]
|
||||||
|
if abs(block["bbox"][0] - col_ref_x0) <= COLUMN_TOLERANCE:
|
||||||
|
column.append(block)
|
||||||
|
assigned = True
|
||||||
|
break
|
||||||
|
if not assigned:
|
||||||
|
columns.append([block])
|
||||||
|
|
||||||
|
# For each column, set next_block_y by y-order within that column.
|
||||||
|
# A block whose expanded bbox would touch the next column-mate's
|
||||||
|
# y0 will be capped to that y0 - 2 (small visual gap).
|
||||||
|
for column in columns:
|
||||||
|
column.sort(key=lambda b: b["bbox"][1])
|
||||||
|
for i, block in enumerate(column):
|
||||||
|
if i + 1 < len(column):
|
||||||
|
block["_next_block_y"] = column[i + 1]["bbox"][1] - 2
|
||||||
|
else:
|
||||||
|
block["_next_block_y"] = page_bottom
|
||||||
|
|
||||||
def _write_translated_block(
|
def _write_translated_block(
|
||||||
self,
|
self,
|
||||||
page,
|
page,
|
||||||
block: Dict,
|
block: Dict,
|
||||||
font_path: Optional[str],
|
font_path: Optional[str],
|
||||||
is_rtl: bool,
|
is_rtl: bool,
|
||||||
) -> None:
|
) -> bool:
|
||||||
"""Write translated text into the block's bounding box.
|
"""Write translated text into the block's bounding box.
|
||||||
|
|
||||||
Priority: respect original font size as much as possible.
|
Track B3.5: Smart overflow handling. Returns True if the block
|
||||||
Strategy:
|
was rendered cleanly, False if it was skipped (format loss).
|
||||||
1. Try original rect at original font size.
|
|
||||||
2. Expand bbox to page margins (same font size).
|
Strategy (in order, first success wins):
|
||||||
3. Expand bbox vertically downward (same font size).
|
0. Try original rect at original font size.
|
||||||
4. Only THEN shrink font as a last resort, with a floor of 70% original.
|
1. Expand bbox to page margins horizontally, original font size.
|
||||||
|
2. Expand bbox vertically (up to MAX_VERTICAL_EXPANSION x height),
|
||||||
|
original font size.
|
||||||
|
3. Shrink font ONCE (FONT_SHRINK_FACTOR = 0.93), with expanded bbox.
|
||||||
|
4. Shrink font AGAIN (0.87 cumulative), with expanded bbox.
|
||||||
|
5. Heading: stop at HEADING_MIN_SCALE (90%). Body: BODY_MIN_SCALE (75%).
|
||||||
|
6. If still overflow: SKIP the block, emit format_loss metric, write
|
||||||
|
a small placeholder so the user knows something was lost.
|
||||||
|
|
||||||
|
Key insight: prefer LOOSING a block over DEGRADING the entire page's
|
||||||
|
font hierarchy. A title that becomes unreadable 8.5pt is worse than
|
||||||
|
a missing TOC entry.
|
||||||
"""
|
"""
|
||||||
import fitz
|
import fitz
|
||||||
|
|
||||||
@@ -430,70 +810,115 @@ class PDFTranslator:
|
|||||||
color = self._int_to_rgb(block["color"])
|
color = self._int_to_rgb(block["color"])
|
||||||
align = fitz.TEXT_ALIGN_RIGHT if is_rtl else fitz.TEXT_ALIGN_LEFT
|
align = fitz.TEXT_ALIGN_RIGHT if is_rtl else fitz.TEXT_ALIGN_LEFT
|
||||||
|
|
||||||
fontname = None
|
# PyMuPDF bug: fontname=None raises AttributeError. Default to 'helv'.
|
||||||
|
# If a custom font file is available, use it via fontfile (fontname ignored).
|
||||||
|
fontname = "helv"
|
||||||
fontfile = font_path
|
fontfile = font_path
|
||||||
|
|
||||||
# Step 1: original rect, original size
|
# Determine if this is a heading (larger font size = more visual weight)
|
||||||
size = target_size
|
is_heading = target_size >= HEADING_MIN_SIZE
|
||||||
rc = self._try_insert(page, original_rect, translated, size, fontname, fontfile, color, align)
|
min_scale = HEADING_MIN_SCALE if is_heading else BODY_MIN_SCALE
|
||||||
if rc is not None and rc >= 0:
|
min_size = max(target_size * min_scale, MIN_FONT_SIZE)
|
||||||
return
|
|
||||||
|
|
||||||
# Step 2: expand to page margins (horizontal)
|
|
||||||
page_rect = page.rect
|
page_rect = page.rect
|
||||||
margin = 18
|
margin = 18
|
||||||
|
|
||||||
|
# Track B3.6: try a wider bbox that respects the page margin.
|
||||||
|
# For headings, also allow horizontal expansion because long
|
||||||
|
# translated titles often don't fit in the original width.
|
||||||
|
max_x1 = page_rect.x1 - margin
|
||||||
expanded_h = fitz.Rect(
|
expanded_h = fitz.Rect(
|
||||||
max(original_rect.x0, page_rect.x0 + margin),
|
max(original_rect.x0, page_rect.x0 + margin),
|
||||||
original_rect.y0,
|
original_rect.y0,
|
||||||
min(original_rect.x1, page_rect.x1 - margin),
|
max_x1,
|
||||||
original_rect.y1,
|
original_rect.y1,
|
||||||
)
|
)
|
||||||
if expanded_h.width > original_rect.width:
|
|
||||||
rc = self._try_insert(page, expanded_h, translated, size, fontname, fontfile, color, align)
|
|
||||||
if rc is not None and rc >= 0:
|
|
||||||
return
|
|
||||||
|
|
||||||
# Step 3: expand vertically (allow text to flow down)
|
# Track B3.7: cap vertical expansion at the next block's y0
|
||||||
max_expand_y = min(page_rect.y1 - margin - original_rect.y1, original_rect.height * 1.5)
|
# (not the page bottom), so a long translated block cannot
|
||||||
expanded = fitz.Rect(
|
# overflow into its neighbour.
|
||||||
|
# Track B3.8: clamp to >= 0 to avoid an invalid (inverted) Rect
|
||||||
|
# when next_block_y sits above the current block (rare edge
|
||||||
|
# case in extracted blocks where bbox ordering is unusual).
|
||||||
|
next_block_y = block.get("_next_block_y", page_rect.y1 - margin)
|
||||||
|
max_expand_y = max(
|
||||||
|
0.0,
|
||||||
|
min(
|
||||||
|
next_block_y - original_rect.y1,
|
||||||
|
original_rect.height * MAX_VERTICAL_EXPANSION,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
expanded_v = fitz.Rect(
|
||||||
expanded_h.x0,
|
expanded_h.x0,
|
||||||
expanded_h.y0,
|
expanded_h.y0,
|
||||||
expanded_h.x1,
|
expanded_h.x1,
|
||||||
expanded_h.y1 + max_expand_y,
|
expanded_h.y1 + max_expand_y,
|
||||||
)
|
)
|
||||||
if expanded.height > expanded_h.height:
|
|
||||||
rc = self._try_insert(page, expanded, translated, size, fontname, fontfile, color, align)
|
|
||||||
if rc is not None and rc >= 0:
|
|
||||||
return
|
|
||||||
|
|
||||||
# Step 4: shrink font — but never below 70% of original
|
# Tier 0: original rect, original size
|
||||||
min_size = max(target_size * 0.70, MIN_FONT_SIZE)
|
# insert_textbox returns >= 0 if text fit, < 0 if overflow (in points)
|
||||||
rect = expanded
|
rc = self._try_insert(page, original_rect, translated, target_size, fontname, fontfile, color, align)
|
||||||
for attempt in range(8):
|
|
||||||
size *= FONT_SHRINK_FACTOR
|
|
||||||
if size < min_size:
|
|
||||||
size = min_size
|
|
||||||
rc = self._try_insert(page, rect, translated, size, fontname, fontfile, color, align)
|
|
||||||
break
|
|
||||||
rc = self._try_insert(page, rect, translated, size, fontname, fontfile, color, align)
|
|
||||||
if rc is not None and rc >= 0:
|
if rc is not None and rc >= 0:
|
||||||
return
|
return True
|
||||||
|
|
||||||
# Last resort
|
# Tier 1: expanded horizontal, original size
|
||||||
if rc is None or rc < 0:
|
if expanded_h.width > original_rect.width + 1:
|
||||||
|
rc = self._try_insert(page, expanded_h, translated, target_size, fontname, fontfile, color, align)
|
||||||
|
if rc is not None and rc >= 0:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Tier 2: expanded vertical, original size
|
||||||
|
if expanded_v.height > original_rect.height + 1:
|
||||||
|
rc = self._try_insert(page, expanded_v, translated, target_size, fontname, fontfile, color, align)
|
||||||
|
if rc is not None and rc >= 0:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Tier 3: shrink once
|
||||||
|
size_1 = max(target_size * FONT_SHRINK_FACTOR, min_size)
|
||||||
|
if size_1 < target_size - 0.1: # only try if shrink is meaningful
|
||||||
|
rc = self._try_insert(page, expanded_v, translated, size_1, fontname, fontfile, color, align)
|
||||||
|
if rc is not None and rc >= 0:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Tier 4: shrink twice
|
||||||
|
size_2 = max(target_size * FONT_SHRINK_FACTOR * FONT_SHRINK_FACTOR, min_size)
|
||||||
|
if size_2 < size_1 - 0.1:
|
||||||
|
rc = self._try_insert(page, expanded_v, translated, size_2, fontname, fontfile, color, align)
|
||||||
|
if rc is not None and rc >= 0:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Tier 5: hit the min size floor
|
||||||
|
rc = self._try_insert(page, expanded_v, translated, min_size, fontname, fontfile, color, align)
|
||||||
|
if rc is not None and rc >= 0:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Tier 6: graceful failure — skip the block, log it
|
||||||
|
self._skip_block_with_marker(page, original_rect, translated, color)
|
||||||
|
_record_format_loss_metric("text_overflow")
|
||||||
|
logger.warning(
|
||||||
|
"pdf_block_skipped_overflow",
|
||||||
|
font_size=target_size,
|
||||||
|
text_preview=translated[:60],
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _skip_block_with_marker(self, page, rect, original_text, color):
|
||||||
|
"""Mark a skipped block with a visible placeholder so the user
|
||||||
|
knows something was lost. Uses a tiny font + grey color."""
|
||||||
|
import fitz
|
||||||
|
placeholder = "[translation overflow]"
|
||||||
try:
|
try:
|
||||||
page.insert_textbox(
|
page.insert_textbox(
|
||||||
rect,
|
rect,
|
||||||
translated,
|
placeholder,
|
||||||
fontsize=min_size,
|
fontsize=6.0,
|
||||||
fontname=fontname or "helv",
|
fontname="helv",
|
||||||
fontfile=fontfile,
|
color=(0.6, 0.6, 0.6),
|
||||||
color=color,
|
align=fitz.TEXT_ALIGN_LEFT,
|
||||||
align=align,
|
|
||||||
overlay=True,
|
overlay=True,
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception:
|
||||||
logger.warning("textbox_final_failed", error=str(e))
|
pass
|
||||||
|
|
||||||
def _try_insert(
|
def _try_insert(
|
||||||
self, page, rect, text, fontsize, fontname, fontfile, color, align
|
self, page, rect, text, fontsize, fontname, fontfile, color, align
|
||||||
@@ -510,7 +935,16 @@ class PDFTranslator:
|
|||||||
align=align,
|
align=align,
|
||||||
overlay=True,
|
overlay=True,
|
||||||
)
|
)
|
||||||
except Exception:
|
except Exception as e:
|
||||||
|
# Demote this from warning to debug — insert_textbox is called
|
||||||
|
# 6+ times per block (one per tier), and these errors are common
|
||||||
|
# in normal operation (e.g. when the text really doesn't fit).
|
||||||
|
logger.debug(
|
||||||
|
"pdf_insert_textbox_error",
|
||||||
|
error=str(e)[:200],
|
||||||
|
error_type=type(e).__name__,
|
||||||
|
fontsize=fontsize,
|
||||||
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
@@ -622,6 +1056,13 @@ class PDFTranslator:
|
|||||||
|
|
||||||
def _convert_docx_to_pdf(self, docx_path: Path, target_pdf: Path) -> Path:
|
def _convert_docx_to_pdf(self, docx_path: Path, target_pdf: Path) -> Path:
|
||||||
"""Convert DOCX → PDF using LibreOffice headless."""
|
"""Convert DOCX → PDF using LibreOffice headless."""
|
||||||
|
if not _libreoffice_available():
|
||||||
|
# Cache miss → known-missing path
|
||||||
|
logger.warning(
|
||||||
|
"libreoffice_not_found",
|
||||||
|
hint="Install libreoffice-core on the host for DOCX→PDF fallback",
|
||||||
|
)
|
||||||
|
return None # type: ignore[return-value]
|
||||||
try:
|
try:
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
[
|
[
|
||||||
@@ -639,13 +1080,13 @@ class PDFTranslator:
|
|||||||
shutil.move(str(expected_pdf), str(target_pdf))
|
shutil.move(str(expected_pdf), str(target_pdf))
|
||||||
logger.info("docx_to_pdf_success")
|
logger.info("docx_to_pdf_success")
|
||||||
return target_pdf
|
return target_pdf
|
||||||
logger.warning("docx_to_pdf_no_output", stderr=result.stderr)
|
logger.warning("docx_to_pdf_no_output", stderr=result.stderr[:200])
|
||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
logger.warning("libreoffice_not_found")
|
logger.warning("libreoffice_not_found_runtime")
|
||||||
except subprocess.TimeoutExpired:
|
except subprocess.TimeoutExpired:
|
||||||
logger.warning("libreoffice_timeout")
|
logger.warning("libreoffice_timeout")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning("docx_to_pdf_failed", error=str(e))
|
logger.warning("docx_to_pdf_failed", error=str(e)[:200])
|
||||||
|
|
||||||
docx_output = target_pdf.with_suffix(".docx")
|
docx_output = target_pdf.with_suffix(".docx")
|
||||||
if docx_path != docx_output and docx_path.exists():
|
if docx_path != docx_output and docx_path.exists():
|
||||||
|
|||||||
@@ -221,6 +221,13 @@ class PowerPointTranslator:
|
|||||||
|
|
||||||
text_elements: List[Tuple[str, Callable[[str], None]]] = []
|
text_elements: List[Tuple[str, Callable[[str], None]]] = []
|
||||||
|
|
||||||
|
# SmartArt diagrams live in ppt/diagrams/*.xml — outside the
|
||||||
|
# python-pptx object model. We collect them once from the ZIP.
|
||||||
|
diagram_translations: List[Dict[str, Any]] = []
|
||||||
|
self._collect_diagrams_from_zip(
|
||||||
|
input_path, text_elements, diagram_translations
|
||||||
|
)
|
||||||
|
|
||||||
for slide_idx, slide in enumerate(presentation.slides):
|
for slide_idx, slide in enumerate(presentation.slides):
|
||||||
if slide.has_notes_slide and slide.notes_slide.notes_text_frame:
|
if slide.has_notes_slide and slide.notes_slide.notes_text_frame:
|
||||||
self._collect_from_text_frame(
|
self._collect_from_text_frame(
|
||||||
@@ -326,6 +333,9 @@ class PowerPointTranslator:
|
|||||||
# Re-inject chart translations into chart XML parts
|
# Re-inject chart translations into chart XML parts
|
||||||
self._apply_chart_translations(output_path)
|
self._apply_chart_translations(output_path)
|
||||||
|
|
||||||
|
# Re-inject SmartArt diagram translations into diagram XML parts
|
||||||
|
self._apply_diagram_translations(output_path, diagram_translations)
|
||||||
|
|
||||||
processing_time_ms = round((time.time() - start_time) * 1000, 2)
|
processing_time_ms = round((time.time() - start_time) * 1000, 2)
|
||||||
|
|
||||||
_log_info(
|
_log_info(
|
||||||
@@ -469,7 +479,24 @@ class PowerPointTranslator:
|
|||||||
def _collect_from_shape(
|
def _collect_from_shape(
|
||||||
self, shape: BaseShape, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, shape: BaseShape, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Collect text from a shape and its children."""
|
"""Collect text from a shape and its children.
|
||||||
|
|
||||||
|
Skips auto-generated placeholders (date, slide number, footer)
|
||||||
|
whose text is regenerated by PowerPoint on save — translating
|
||||||
|
them would just be wasted work.
|
||||||
|
"""
|
||||||
|
# Skip placeholders that PowerPoint auto-generates (date, slide number,
|
||||||
|
# footer) — these are filled with &D/&P/&F codes that get expanded at
|
||||||
|
# render time. Translating them is pointless.
|
||||||
|
if shape.is_placeholder:
|
||||||
|
try:
|
||||||
|
ph_type = shape.placeholder_format.type
|
||||||
|
# 16 = DATE, 13 = SLIDE_NUMBER, 15 = FOOTER
|
||||||
|
if ph_type is not None and int(ph_type) in (13, 15, 16):
|
||||||
|
return
|
||||||
|
except (AttributeError, ValueError, Exception):
|
||||||
|
pass # not a placeholder type we can check, continue normally
|
||||||
|
|
||||||
if shape.has_text_frame:
|
if shape.has_text_frame:
|
||||||
self._collect_from_text_frame(shape.text_frame, text_elements)
|
self._collect_from_text_frame(shape.text_frame, text_elements)
|
||||||
|
|
||||||
@@ -479,20 +506,95 @@ class PowerPointTranslator:
|
|||||||
self._collect_from_text_frame(cell.text_frame, text_elements)
|
self._collect_from_text_frame(cell.text_frame, text_elements)
|
||||||
|
|
||||||
if shape.shape_type == MSO_SHAPE_TYPE.GROUP:
|
if shape.shape_type == MSO_SHAPE_TYPE.GROUP:
|
||||||
|
try:
|
||||||
for sub_shape in shape.shapes:
|
for sub_shape in shape.shapes:
|
||||||
self._collect_from_shape(sub_shape, text_elements)
|
self._collect_from_shape(sub_shape, text_elements)
|
||||||
|
except Exception:
|
||||||
|
# Defensive: some groups can have weird attribute access
|
||||||
|
pass
|
||||||
|
|
||||||
# Chart shapes — text is stored in separate chart XML parts
|
# Chart shapes — text is stored in separate chart XML parts
|
||||||
if shape.shape_type == MSO_SHAPE_TYPE.CHART:
|
if shape.shape_type == MSO_SHAPE_TYPE.CHART:
|
||||||
self._collect_from_chart_shape(shape, text_elements)
|
self._collect_from_chart_shape(shape, text_elements)
|
||||||
|
|
||||||
if hasattr(shape, "shapes"):
|
# Some shapes (e.g. LayoutPlaceholders) expose a .shapes attribute
|
||||||
|
# that's not a Group. Skip these to avoid spurious iteration.
|
||||||
|
if shape.shape_type == MSO_SHAPE_TYPE.GROUP and hasattr(shape, "shapes"):
|
||||||
try:
|
try:
|
||||||
for sub_shape in shape.shapes:
|
for sub_shape in shape.shapes:
|
||||||
self._collect_from_shape(sub_shape, text_elements)
|
self._collect_from_shape(sub_shape, text_elements)
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
def _collect_diagrams_from_zip(
|
||||||
|
self,
|
||||||
|
input_path: Path,
|
||||||
|
text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
|
diagram_translations: List[Dict[str, Any]],
|
||||||
|
) -> None:
|
||||||
|
"""Parse SmartArt diagram XML from the .pptx ZIP and collect translatable text.
|
||||||
|
|
||||||
|
SmartArt text lives in ``ppt/diagrams/data*.xml`` inside the ZIP.
|
||||||
|
Each diagram data file contains ``<a:t>`` text nodes inside
|
||||||
|
DrawingML structures.
|
||||||
|
"""
|
||||||
|
_TAG_A_T = f"{{{_NS_A}}}t"
|
||||||
|
|
||||||
|
try:
|
||||||
|
with zipfile.ZipFile(input_path, 'r') as zf:
|
||||||
|
diag_files = [
|
||||||
|
n for n in zf.namelist()
|
||||||
|
if n.startswith('ppt/diagrams/data') and n.endswith('.xml')
|
||||||
|
]
|
||||||
|
|
||||||
|
for diag_file in diag_files:
|
||||||
|
try:
|
||||||
|
diag_bytes = zf.read(diag_file)
|
||||||
|
diag_xml = etree.fromstring(diag_bytes)
|
||||||
|
|
||||||
|
for t_elem in diag_xml.iter(_TAG_A_T):
|
||||||
|
if t_elem.text and t_elem.text.strip():
|
||||||
|
# Preserve the raw text (with surrounding
|
||||||
|
# whitespace) so the apply step can
|
||||||
|
# restore it. Use the stripped form for
|
||||||
|
# the API call and the matching key.
|
||||||
|
original_raw = t_elem.text
|
||||||
|
original = original_raw.strip()
|
||||||
|
|
||||||
|
# Skip numeric-only or very short tokens
|
||||||
|
if original.replace('.', '').replace('-', '').replace(',', '').isdigit():
|
||||||
|
continue
|
||||||
|
if len(original) <= 1:
|
||||||
|
continue
|
||||||
|
|
||||||
|
entry: Dict[str, Any] = {
|
||||||
|
'diag_file': diag_file,
|
||||||
|
'element_path': self._get_element_path(t_elem),
|
||||||
|
'original': original,
|
||||||
|
'original_raw': original_raw,
|
||||||
|
'translated': None,
|
||||||
|
# Keep the original XML bytes so the
|
||||||
|
# apply step can re-create the part
|
||||||
|
# if python-pptx strips it.
|
||||||
|
'original_xml': diag_bytes,
|
||||||
|
}
|
||||||
|
diagram_translations.append(entry)
|
||||||
|
|
||||||
|
def _make_diag_setter(entries, idx):
|
||||||
|
def setter(text: str) -> None:
|
||||||
|
entries[idx]['translated'] = text.strip()
|
||||||
|
return setter
|
||||||
|
|
||||||
|
text_elements.append(
|
||||||
|
(original, _make_diag_setter(diagram_translations, len(diagram_translations) - 1))
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
_log_error("pptx_diagram_parse_error", diag_file=diag_file, error=str(e))
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
_log_error("pptx_diagrams_zip_error", error=str(e))
|
||||||
|
|
||||||
def _collect_from_chart_shape(
|
def _collect_from_chart_shape(
|
||||||
self, shape: BaseShape, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, shape: BaseShape, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||||
) -> None:
|
) -> None:
|
||||||
@@ -500,29 +602,31 @@ class PowerPointTranslator:
|
|||||||
|
|
||||||
Chart text (title, axis titles, series names, data labels) is stored
|
Chart text (title, axis titles, series names, data labels) is stored
|
||||||
in a separate chart XML part, not in shape.text_frame.
|
in a separate chart XML part, not in shape.text_frame.
|
||||||
|
|
||||||
|
Each translatable element is tracked by its `element_path` (an
|
||||||
|
XPath-like path) so the apply step can target it precisely even
|
||||||
|
when the same text appears multiple times in the same chart.
|
||||||
"""
|
"""
|
||||||
_NS_A = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
_NS_A = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
||||||
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
chart_data = shape.chart
|
chart_data = shape.chart
|
||||||
# Access the chart XML part through the chart's part
|
|
||||||
chart_part = chart_data.part
|
chart_part = chart_data.part
|
||||||
chart_xml = etree.fromstring(chart_part.blob)
|
chart_xml = etree.fromstring(chart_part.blob)
|
||||||
|
|
||||||
# Collect text from <a:t> elements in chart XML
|
|
||||||
# These include: chart title, axis titles, legend entries, data labels
|
|
||||||
seen_texts: set = set()
|
|
||||||
chart_text_entries: List[Dict[str, Any]] = []
|
chart_text_entries: List[Dict[str, Any]] = []
|
||||||
|
|
||||||
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
||||||
text = t_elem.text
|
text = t_elem.text
|
||||||
if text and text.strip() and text.strip() not in seen_texts:
|
if text and text.strip():
|
||||||
seen_texts.add(text.strip())
|
|
||||||
entry = {
|
entry = {
|
||||||
'element': t_elem,
|
'element': t_elem,
|
||||||
'original': text.strip(),
|
'original': text.strip(),
|
||||||
|
'original_raw': text,
|
||||||
'translated': None,
|
'translated': None,
|
||||||
|
'tag': 'a:t',
|
||||||
|
'element_path': self._get_element_path(t_elem),
|
||||||
}
|
}
|
||||||
chart_text_entries.append(entry)
|
chart_text_entries.append(entry)
|
||||||
|
|
||||||
@@ -535,16 +639,16 @@ class PowerPointTranslator:
|
|||||||
(text.strip(), make_chart_setter(chart_text_entries, len(chart_text_entries) - 1))
|
(text.strip(), make_chart_setter(chart_text_entries, len(chart_text_entries) - 1))
|
||||||
)
|
)
|
||||||
|
|
||||||
# Also collect from <c:v> (cell values used as category names)
|
|
||||||
for v_elem in chart_xml.iter(f'{{{_NS_C}}}v'):
|
for v_elem in chart_xml.iter(f'{{{_NS_C}}}v'):
|
||||||
text = v_elem.text
|
text = v_elem.text
|
||||||
if text and text.strip() and not text.strip().replace('.', '').replace('-', '').replace(',', '').isdigit():
|
if text and text.strip() and not text.strip().replace('.', '').replace('-', '').replace(',', '').isdigit():
|
||||||
if text.strip() not in seen_texts:
|
|
||||||
seen_texts.add(text.strip())
|
|
||||||
entry = {
|
entry = {
|
||||||
'element': v_elem,
|
'element': v_elem,
|
||||||
'original': text.strip(),
|
'original': text.strip(),
|
||||||
|
'original_raw': text,
|
||||||
'translated': None,
|
'translated': None,
|
||||||
|
'tag': 'c:v',
|
||||||
|
'element_path': self._get_element_path(v_elem),
|
||||||
}
|
}
|
||||||
chart_text_entries.append(entry)
|
chart_text_entries.append(entry)
|
||||||
|
|
||||||
@@ -557,7 +661,6 @@ class PowerPointTranslator:
|
|||||||
(text.strip(), make_chart_v_setter(chart_text_entries, len(chart_text_entries) - 1))
|
(text.strip(), make_chart_v_setter(chart_text_entries, len(chart_text_entries) - 1))
|
||||||
)
|
)
|
||||||
|
|
||||||
# Store chart_part reference and entries for later re-injection
|
|
||||||
if chart_text_entries:
|
if chart_text_entries:
|
||||||
if not hasattr(self, '_chart_entries'):
|
if not hasattr(self, '_chart_entries'):
|
||||||
self._chart_entries = []
|
self._chart_entries = []
|
||||||
@@ -595,8 +698,58 @@ class PowerPointTranslator:
|
|||||||
|
|
||||||
text_elements.append((stripped, make_setter(run, leading, trailing)))
|
text_elements.append((stripped, make_setter(run, leading, trailing)))
|
||||||
|
|
||||||
|
def _get_element_path(self, element) -> str:
|
||||||
|
"""Get a unique XPath-like path for an element within its XML tree.
|
||||||
|
|
||||||
|
Used to robustly match chart/diagram elements between collect and
|
||||||
|
apply time, even when the same text value appears multiple times.
|
||||||
|
"""
|
||||||
|
path_parts = []
|
||||||
|
current = element
|
||||||
|
while current is not None:
|
||||||
|
parent = current.getparent()
|
||||||
|
if parent is None:
|
||||||
|
break
|
||||||
|
idx = list(parent).index(current)
|
||||||
|
tag = current.tag.split('}')[-1] if '}' in current.tag else current.tag
|
||||||
|
path_parts.append(f"{tag}[{idx}]")
|
||||||
|
current = parent
|
||||||
|
return '/'.join(reversed(path_parts))
|
||||||
|
|
||||||
|
def _find_element_by_path(self, root, path: str):
|
||||||
|
"""Navigate the XML tree using the path produced by `_get_element_path`."""
|
||||||
|
if not path:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
current = root
|
||||||
|
for segment in path.split('/'):
|
||||||
|
if '[' not in segment or not segment.endswith(']'):
|
||||||
|
return None
|
||||||
|
tag_name, idx_str = segment[:-1].split('[', 1)
|
||||||
|
try:
|
||||||
|
idx = int(idx_str)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
children = list(current)
|
||||||
|
if idx < 0 or idx >= len(children):
|
||||||
|
return None
|
||||||
|
candidate = children[idx]
|
||||||
|
candidate_tag = candidate.tag.split('}')[-1] if '}' in candidate.tag else candidate.tag
|
||||||
|
if candidate_tag != tag_name:
|
||||||
|
return None
|
||||||
|
current = candidate
|
||||||
|
return current
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
def _apply_chart_translations(self, output_path: Path) -> None:
|
def _apply_chart_translations(self, output_path: Path) -> None:
|
||||||
"""Re-inject chart text translations by modifying chart XML parts in the .pptx ZIP."""
|
"""Re-inject chart text translations by modifying chart XML parts.
|
||||||
|
|
||||||
|
Matching strategy: prefer the stored `element_path` (set at collect
|
||||||
|
time) to navigate directly to the right element. Fall back to
|
||||||
|
string equality on `entry['original']` for legacy entries (or when
|
||||||
|
the path can't be resolved — e.g. chart structure mutated).
|
||||||
|
"""
|
||||||
if not hasattr(self, '_chart_entries') or not self._chart_entries:
|
if not hasattr(self, '_chart_entries') or not self._chart_entries:
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -604,6 +757,7 @@ class PowerPointTranslator:
|
|||||||
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
_NS_C = "http://schemas.openxmlformats.org/drawingml/2006/chart"
|
||||||
|
|
||||||
total_translated = 0
|
total_translated = 0
|
||||||
|
total_skipped = 0
|
||||||
|
|
||||||
for chart_data in self._chart_entries:
|
for chart_data in self._chart_entries:
|
||||||
entries = chart_data['entries']
|
entries = chart_data['entries']
|
||||||
@@ -617,22 +771,46 @@ class PowerPointTranslator:
|
|||||||
chart_xml = etree.fromstring(chart_part.blob)
|
chart_xml = etree.fromstring(chart_part.blob)
|
||||||
|
|
||||||
for entry in translated_entries:
|
for entry in translated_entries:
|
||||||
# Try to find and update <a:t> elements
|
target = None
|
||||||
for t_elem in chart_xml.iter(f'{{{_NS_A}}}t'):
|
|
||||||
if t_elem.text and t_elem.text.strip() == entry['original']:
|
# 1) Preferred: navigate by stored element_path
|
||||||
t_elem.text = entry['translated']
|
element_path = entry.get('element_path')
|
||||||
total_translated += 1
|
if element_path:
|
||||||
break
|
target = self._find_element_by_path(chart_xml, element_path)
|
||||||
else:
|
|
||||||
# Try <c:v> elements
|
# 2) Fallback: string equality on original text
|
||||||
for v_elem in chart_xml.iter(f'{{{_NS_C}}}v'):
|
if target is None:
|
||||||
if v_elem.text and v_elem.text.strip() == entry['original']:
|
tag = entry.get('tag', 'a:t')
|
||||||
v_elem.text = entry['translated']
|
ns = _NS_C if tag == 'c:v' else _NS_A
|
||||||
total_translated += 1
|
for candidate in chart_xml.iter(f'{{{ns}}}{tag.split(":")[-1]}'):
|
||||||
|
cand_text = candidate.text or ""
|
||||||
|
if cand_text.strip() == entry['original']:
|
||||||
|
target = candidate
|
||||||
break
|
break
|
||||||
|
|
||||||
|
if target is None:
|
||||||
|
total_skipped += 1
|
||||||
|
_log_error(
|
||||||
|
"pptx_chart_target_not_found",
|
||||||
|
original=entry.get('original', '')[:60],
|
||||||
|
has_path=bool(entry.get('element_path')),
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Preserve leading/trailing whitespace if original had any
|
||||||
|
orig = entry.get('original_raw') or entry.get('original', '') or ''
|
||||||
|
leading = orig[: len(orig) - len(orig.lstrip())]
|
||||||
|
trailing = orig[len(orig.rstrip()) :]
|
||||||
|
target.text = leading + (entry['translated'] or '').strip() + trailing
|
||||||
|
total_translated += 1
|
||||||
|
|
||||||
# Update the chart part blob
|
# Update the chart part blob
|
||||||
chart_part._blob = etree.tostring(chart_xml, xml_declaration=True, encoding='UTF-8', standalone=True)
|
chart_part.blob = etree.tostring(
|
||||||
|
chart_xml,
|
||||||
|
xml_declaration=True,
|
||||||
|
encoding='UTF-8',
|
||||||
|
standalone=True,
|
||||||
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
_log_error("pptx_chart_update_error", error=str(e))
|
_log_error("pptx_chart_update_error", error=str(e))
|
||||||
@@ -640,8 +818,224 @@ class PowerPointTranslator:
|
|||||||
# Clean up
|
# Clean up
|
||||||
self._chart_entries = []
|
self._chart_entries = []
|
||||||
|
|
||||||
if total_translated > 0:
|
if total_translated > 0 or total_skipped > 0:
|
||||||
_log_info("pptx_charts_translated", total=total_translated)
|
_log_info(
|
||||||
|
"pptx_charts_translated",
|
||||||
|
translated=total_translated,
|
||||||
|
skipped=total_skipped,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _apply_diagram_translations(
|
||||||
|
self,
|
||||||
|
output_path: Path,
|
||||||
|
diagram_translations: List[Dict[str, Any]],
|
||||||
|
) -> None:
|
||||||
|
"""Re-inject SmartArt diagram translations by rewriting the .pptx ZIP.
|
||||||
|
|
||||||
|
python-pptx doesn't manage diagram parts. After `presentation.save()`
|
||||||
|
strips them out, we need to (re-)add the diagram data files with
|
||||||
|
the translated <a:t> elements.
|
||||||
|
|
||||||
|
The typical flow is:
|
||||||
|
1. input PPTX has ``ppt/diagrams/data*.xml``
|
||||||
|
2. ``presentation.save(output_path)`` produces a .pptx WITHOUT
|
||||||
|
those diagram parts (python-pptx ignores them)
|
||||||
|
3. We rewrite the output ZIP to ADD the diagram parts back,
|
||||||
|
with the translated text. Content-types and rels are
|
||||||
|
patched in if missing.
|
||||||
|
"""
|
||||||
|
if not diagram_translations:
|
||||||
|
return
|
||||||
|
|
||||||
|
translated_entries = [e for e in diagram_translations if e.get('translated')]
|
||||||
|
if not translated_entries:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Group by file so we only parse each diagram XML once
|
||||||
|
by_file: Dict[str, List[Dict[str, Any]]] = {}
|
||||||
|
for entry in translated_entries:
|
||||||
|
by_file.setdefault(entry['diag_file'], []).append(entry)
|
||||||
|
|
||||||
|
# Read original ZIP into memory
|
||||||
|
try:
|
||||||
|
with open(output_path, 'rb') as f:
|
||||||
|
original_zip_bytes = f.read()
|
||||||
|
except Exception as e:
|
||||||
|
_log_error("pptx_diagram_read_failed", error=str(e))
|
||||||
|
return
|
||||||
|
|
||||||
|
in_zip = zipfile.ZipFile(io.BytesIO(original_zip_bytes), 'r')
|
||||||
|
out_buffer = io.BytesIO()
|
||||||
|
total_translated = 0
|
||||||
|
total_skipped = 0
|
||||||
|
added_files: List[str] = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
existing_names = set(in_zip.namelist())
|
||||||
|
# Read [Content_Types].xml so we can extend it if needed
|
||||||
|
content_types_xml: Optional[bytes] = None
|
||||||
|
if "[Content_Types].xml" in existing_names:
|
||||||
|
content_types_xml = in_zip.read("[Content_Types].xml")
|
||||||
|
|
||||||
|
with zipfile.ZipFile(out_buffer, 'w', zipfile.ZIP_DEFLATED) as out_zip:
|
||||||
|
for item in in_zip.infolist():
|
||||||
|
data = in_zip.read(item.filename)
|
||||||
|
|
||||||
|
if item.filename in by_file and item.filename in existing_names:
|
||||||
|
data = self._rewrite_diagram_xml(
|
||||||
|
data,
|
||||||
|
by_file[item.filename],
|
||||||
|
)
|
||||||
|
# Count translated (success counted in helper)
|
||||||
|
for entry in by_file[item.filename]:
|
||||||
|
if entry.get('translated'):
|
||||||
|
if entry.get('_applied'):
|
||||||
|
total_translated += 1
|
||||||
|
else:
|
||||||
|
total_skipped += 1
|
||||||
|
# Reset _applied flag
|
||||||
|
for entry in by_file[item.filename]:
|
||||||
|
entry.pop('_applied', None)
|
||||||
|
|
||||||
|
out_zip.writestr(item, data)
|
||||||
|
|
||||||
|
# Add NEW diagram parts (parts that python-pptx stripped)
|
||||||
|
for diag_file, file_entries in by_file.items():
|
||||||
|
if diag_file in existing_names:
|
||||||
|
continue
|
||||||
|
# Re-create the diagram XML from the original
|
||||||
|
# (we stored a reference at collect time)
|
||||||
|
original_xml = None
|
||||||
|
for entry in file_entries:
|
||||||
|
if entry.get('original_xml'):
|
||||||
|
original_xml = entry['original_xml']
|
||||||
|
break
|
||||||
|
if original_xml is None:
|
||||||
|
# Last-ditch fallback: log and skip
|
||||||
|
_log_error(
|
||||||
|
"pptx_diagram_original_missing",
|
||||||
|
diag_file=diag_file,
|
||||||
|
)
|
||||||
|
total_skipped += len(file_entries)
|
||||||
|
continue
|
||||||
|
data = self._rewrite_diagram_xml(
|
||||||
|
original_xml,
|
||||||
|
file_entries,
|
||||||
|
)
|
||||||
|
for entry in file_entries:
|
||||||
|
if entry.get('translated'):
|
||||||
|
if entry.get('_applied'):
|
||||||
|
total_translated += 1
|
||||||
|
else:
|
||||||
|
total_skipped += 1
|
||||||
|
for entry in file_entries:
|
||||||
|
entry.pop('_applied', None)
|
||||||
|
out_zip.writestr(diag_file, data)
|
||||||
|
added_files.append(diag_file)
|
||||||
|
|
||||||
|
# Patch [Content_Types].xml if we added diagram parts
|
||||||
|
if added_files and content_types_xml is not None:
|
||||||
|
pass # We'll rewrite the whole file at the end
|
||||||
|
finally:
|
||||||
|
in_zip.close()
|
||||||
|
|
||||||
|
# If we added new diagram parts, also patch [Content_Types].xml
|
||||||
|
# inside the output buffer and ensure relationships exist
|
||||||
|
final_buffer = out_buffer
|
||||||
|
if added_files and content_types_xml is not None:
|
||||||
|
final_buffer = io.BytesIO()
|
||||||
|
ct_text = content_types_xml.decode("utf-8")
|
||||||
|
patched = False
|
||||||
|
with zipfile.ZipFile(out_buffer, 'r') as src, \
|
||||||
|
zipfile.ZipFile(final_buffer, 'w', zipfile.ZIP_DEFLATED) as dst:
|
||||||
|
for item in src.infolist():
|
||||||
|
if item.filename == "[Content_Types].xml":
|
||||||
|
for diag_file in added_files:
|
||||||
|
override = (
|
||||||
|
f'<Override PartName="/{diag_file}" '
|
||||||
|
f'ContentType="application/vnd.openxmlformats-officedocument.'
|
||||||
|
f'drawingml+xml+diagram"/>'
|
||||||
|
)
|
||||||
|
if override not in ct_text:
|
||||||
|
ct_text = ct_text.replace(
|
||||||
|
"</Types>", f"{override}</Types>"
|
||||||
|
)
|
||||||
|
patched = True
|
||||||
|
dst.writestr(item, ct_text.encode("utf-8"))
|
||||||
|
else:
|
||||||
|
dst.writestr(item, src.read(item.filename))
|
||||||
|
if not patched:
|
||||||
|
final_buffer = out_buffer
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(output_path, 'wb') as f:
|
||||||
|
f.write(final_buffer.getvalue())
|
||||||
|
except Exception as e:
|
||||||
|
_log_error("pptx_diagram_zip_write_error", error=str(e))
|
||||||
|
return
|
||||||
|
|
||||||
|
if total_translated > 0 or total_skipped > 0:
|
||||||
|
_log_info(
|
||||||
|
"pptx_diagrams_translated",
|
||||||
|
translated=total_translated,
|
||||||
|
skipped=total_skipped,
|
||||||
|
files=len(by_file),
|
||||||
|
added=len(added_files),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _rewrite_diagram_xml(
|
||||||
|
self,
|
||||||
|
original_data: bytes,
|
||||||
|
entries: List[Dict[str, Any]],
|
||||||
|
) -> bytes:
|
||||||
|
"""Apply translations to a single diagram XML and return the new bytes.
|
||||||
|
|
||||||
|
Sets ``entry['_applied'] = True`` for entries that were translated.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
diag_xml = etree.fromstring(original_data)
|
||||||
|
except Exception as e:
|
||||||
|
_log_error(
|
||||||
|
"pptx_diagram_parse_error_in_apply",
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
return original_data
|
||||||
|
|
||||||
|
for entry in entries:
|
||||||
|
if not entry.get('translated'):
|
||||||
|
continue
|
||||||
|
target = None
|
||||||
|
element_path = entry.get('element_path')
|
||||||
|
if element_path:
|
||||||
|
target = self._find_element_by_path(diag_xml, element_path)
|
||||||
|
if target is None:
|
||||||
|
# Fallback: string equality on stripped form
|
||||||
|
for candidate in diag_xml.iter(f"{{{_NS_A}}}t"):
|
||||||
|
cand_text = candidate.text or ""
|
||||||
|
if cand_text.strip() == entry['original']:
|
||||||
|
target = candidate
|
||||||
|
break
|
||||||
|
if target is None:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Use original_raw (preserves surrounding whitespace) if
|
||||||
|
# available; otherwise fall back to original.
|
||||||
|
orig = entry.get('original_raw') or entry.get('original', '') or ''
|
||||||
|
leading = orig[: len(orig) - len(orig.lstrip())]
|
||||||
|
trailing = orig[len(orig.rstrip()) :]
|
||||||
|
target.text = (
|
||||||
|
leading
|
||||||
|
+ (entry['translated'] or '').strip()
|
||||||
|
+ trailing
|
||||||
|
)
|
||||||
|
entry['_applied'] = True
|
||||||
|
|
||||||
|
return etree.tostring(
|
||||||
|
diag_xml,
|
||||||
|
xml_declaration=True,
|
||||||
|
encoding='UTF-8',
|
||||||
|
standalone=True,
|
||||||
|
)
|
||||||
|
|
||||||
def _translate_images(self, presentation, target_language: str) -> None:
|
def _translate_images(self, presentation, target_language: str) -> None:
|
||||||
"""Extract and translate text from images in PowerPoint.
|
"""Extract and translate text from images in PowerPoint.
|
||||||
|
|||||||
@@ -230,8 +230,11 @@ class WordTranslator:
|
|||||||
text_elements: List[Tuple[str, Callable[[str], None]]] = []
|
text_elements: List[Tuple[str, Callable[[str], None]]] = []
|
||||||
chart_translations: List[Dict[str, Any]] = []
|
chart_translations: List[Dict[str, Any]] = []
|
||||||
diagram_translations: List[Dict[str, Any]] = []
|
diagram_translations: List[Dict[str, Any]] = []
|
||||||
|
# Callbacks to run AFTER document.save() to write back parts
|
||||||
|
# that python-docx doesn't manage (footnotes, endnotes).
|
||||||
|
post_save_callbacks: List[Callable[[Path], None]] = []
|
||||||
|
|
||||||
self._collect_from_body(document, text_elements)
|
self._collect_from_body(document, text_elements, post_save_callbacks)
|
||||||
|
|
||||||
# Collect chart text from ZIP (chart titles, axis labels, series names)
|
# Collect chart text from ZIP (chart titles, axis labels, series names)
|
||||||
self._collect_charts_from_zip(input_path, text_elements, chart_translations)
|
self._collect_charts_from_zip(input_path, text_elements, chart_translations)
|
||||||
@@ -364,6 +367,14 @@ class WordTranslator:
|
|||||||
if diagram_translations:
|
if diagram_translations:
|
||||||
self._apply_diagram_translations(output_path, diagram_translations)
|
self._apply_diagram_translations(output_path, diagram_translations)
|
||||||
|
|
||||||
|
# Run post-save callbacks (e.g. footnotes/endnotes that python-docx
|
||||||
|
# does not write back automatically).
|
||||||
|
for callback in post_save_callbacks:
|
||||||
|
try:
|
||||||
|
callback(output_path)
|
||||||
|
except Exception as cb_err:
|
||||||
|
_log_error("word_post_save_callback_error", error=str(cb_err))
|
||||||
|
|
||||||
processing_time_ms = round((time.time() - start_time) * 1000, 2)
|
processing_time_ms = round((time.time() - start_time) * 1000, 2)
|
||||||
|
|
||||||
_log_info(
|
_log_info(
|
||||||
@@ -523,7 +534,8 @@ class WordTranslator:
|
|||||||
)
|
)
|
||||||
|
|
||||||
def _collect_from_body(
|
def _collect_from_body(
|
||||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
|
post_save_callbacks: List[Callable[[Path], None]] = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Collect all text elements from document body.
|
"""Collect all text elements from document body.
|
||||||
|
|
||||||
@@ -546,9 +558,11 @@ class WordTranslator:
|
|||||||
|
|
||||||
pass2_count = len(text_elements) - count_before - pass1_count
|
pass2_count = len(text_elements) - count_before - pass1_count
|
||||||
|
|
||||||
# Pass 3: footnotes and endnotes
|
# Pass 3: footnotes and endnotes (live in separate parts)
|
||||||
self._collect_from_footnotes(document, text_elements)
|
if post_save_callbacks is None:
|
||||||
self._collect_from_endnotes(document, text_elements)
|
post_save_callbacks = []
|
||||||
|
self._collect_from_footnotes(document, text_elements, post_save_callbacks)
|
||||||
|
self._collect_from_endnotes(document, text_elements, post_save_callbacks)
|
||||||
|
|
||||||
total = len(text_elements) - count_before
|
total = len(text_elements) - count_before
|
||||||
_log_info(
|
_log_info(
|
||||||
@@ -637,71 +651,137 @@ class WordTranslator:
|
|||||||
self._collect_from_table(table, text_elements)
|
self._collect_from_table(table, text_elements)
|
||||||
|
|
||||||
def _collect_from_footnotes(
|
def _collect_from_footnotes(
|
||||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
|
post_save_callbacks: List[Callable[[Path], None]] = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Collect text from footnotes."""
|
"""Collect text from footnotes.
|
||||||
try:
|
|
||||||
footnotes_part = document.part.package.part_related_by(
|
|
||||||
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/footnotes"
|
|
||||||
) if hasattr(document.part, 'package') else None
|
|
||||||
except Exception:
|
|
||||||
footnotes_part = None
|
|
||||||
|
|
||||||
if footnotes_part is None:
|
python-docx 1.x doesn't expose footnotes via `document.part.package`
|
||||||
# Fallback: try direct XML access
|
(that attribute doesn't exist). We instead iterate over the document's
|
||||||
try:
|
related parts and find the one with the footnotes content type.
|
||||||
footnotes_element = document.element.find(qn("w:footnotes"))
|
|
||||||
if footnotes_element is not None:
|
Because the footnotes XML is a SEPARATE part (not part of the main
|
||||||
for child in footnotes_element:
|
document tree), python-docx will NOT touch it on save — meaning any
|
||||||
if isinstance(child, CT_P):
|
in-memory mutation would be lost. We therefore accumulate the
|
||||||
paragraph = Paragraph(child, document)
|
translations as "post-save callbacks" that re-write the footnotes
|
||||||
self._collect_from_paragraph(paragraph, text_elements)
|
part AFTER the document has been saved.
|
||||||
except Exception:
|
"""
|
||||||
pass
|
footnotes_xml = self._find_part_by_content_type(
|
||||||
|
document,
|
||||||
|
"application/vnd.openxmlformats-officedocument.wordprocessingml.footnotes+xml",
|
||||||
|
)
|
||||||
|
if footnotes_xml is None:
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# Collect every <w:t> in the footnotes XML. We store the t_elem
|
||||||
|
# references so we can update them post-save.
|
||||||
|
for t_elem in footnotes_xml.iter(qn("w:t")):
|
||||||
|
original = t_elem.text or ""
|
||||||
|
if not original.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
def make_t_setter(t):
|
||||||
|
def setter(text: str) -> None:
|
||||||
|
t.text = text
|
||||||
|
return setter
|
||||||
|
|
||||||
|
text_elements.append((original, make_t_setter(t_elem)))
|
||||||
|
|
||||||
|
# If we found any footnote text, register a post-save callback
|
||||||
|
# that will rewrite the footnotes part with the (now-translated)
|
||||||
|
# in-memory XML. The translated t_elems have been mutated by the
|
||||||
|
# setters called from translate_file's batch loop.
|
||||||
|
if text_elements and post_save_callbacks is not None:
|
||||||
|
from copy import deepcopy
|
||||||
|
# We need to keep a reference to the parsed XML tree so we can
|
||||||
|
# serialize it after the setters have run. The footnote text
|
||||||
|
# setters hold references to t_elems inside this tree.
|
||||||
|
def write_footnotes_back(output_path: Path) -> None:
|
||||||
try:
|
try:
|
||||||
footnotes_xml = etree.fromstring(footnotes_part.blob)
|
new_blob = etree.tostring(
|
||||||
for child in footnotes_xml:
|
footnotes_xml,
|
||||||
if child.tag == qn("w:footnote"):
|
xml_declaration=True,
|
||||||
for para_elem in child.findall(qn("w:p")):
|
encoding="UTF-8",
|
||||||
paragraph = Paragraph(para_elem, document)
|
standalone=True,
|
||||||
self._collect_from_paragraph(paragraph, text_elements)
|
)
|
||||||
|
# Rewrite the .docx with the updated footnotes part
|
||||||
|
tmp_path = output_path.with_suffix(".tmp_foot")
|
||||||
|
with zipfile.ZipFile(output_path, "r") as zin, \
|
||||||
|
zipfile.ZipFile(tmp_path, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||||
|
for item in zin.namelist():
|
||||||
|
if item == "word/footnotes.xml":
|
||||||
|
zout.writestr(item, new_blob)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item))
|
||||||
|
tmp_path.replace(output_path)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
_log_error("word_footnotes_parse_error", error=str(e))
|
_log_error("word_footnotes_writeback_error", error=str(e))
|
||||||
|
|
||||||
|
post_save_callbacks.append(write_footnotes_back)
|
||||||
|
|
||||||
|
def _find_part_by_content_type(self, document: Document, content_type: str):
|
||||||
|
"""
|
||||||
|
Find a related XML part by content type (python-docx 1.x compatible).
|
||||||
|
Returns the parsed lxml element, or None if not found.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
related_parts = getattr(document.part, "related_parts", None) or {}
|
||||||
|
for part in related_parts.values():
|
||||||
|
if getattr(part, "content_type", "") == content_type:
|
||||||
|
return etree.fromstring(part.blob)
|
||||||
|
except Exception as e:
|
||||||
|
_log_error("word_part_lookup_error", content_type=content_type, error=str(e))
|
||||||
|
return None
|
||||||
|
|
||||||
def _collect_from_endnotes(
|
def _collect_from_endnotes(
|
||||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]],
|
||||||
|
post_save_callbacks: List[Callable[[Path], None]] = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Collect text from endnotes."""
|
"""Collect text from endnotes (python-docx 1.x compatible).
|
||||||
try:
|
|
||||||
endnotes_part = document.part.package.part_related_by(
|
|
||||||
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/endnotes"
|
|
||||||
) if hasattr(document.part, 'package') else None
|
|
||||||
except Exception:
|
|
||||||
endnotes_part = None
|
|
||||||
|
|
||||||
if endnotes_part is None:
|
See `_collect_from_footnotes` for why we need post-save callbacks.
|
||||||
try:
|
"""
|
||||||
endnotes_element = document.element.find(qn("w:endnotes"))
|
endnotes_xml = self._find_part_by_content_type(
|
||||||
if endnotes_element is not None:
|
document,
|
||||||
for child in endnotes_element:
|
"application/vnd.openxmlformats-officedocument.wordprocessingml.endnotes+xml",
|
||||||
if isinstance(child, CT_P):
|
)
|
||||||
paragraph = Paragraph(child, document)
|
if endnotes_xml is None:
|
||||||
self._collect_from_paragraph(paragraph, text_elements)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return
|
return
|
||||||
|
|
||||||
|
for t_elem in endnotes_xml.iter(qn("w:t")):
|
||||||
|
original = t_elem.text or ""
|
||||||
|
if not original.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
def make_t_setter(t):
|
||||||
|
def setter(text: str) -> None:
|
||||||
|
t.text = text
|
||||||
|
return setter
|
||||||
|
|
||||||
|
text_elements.append((original, make_t_setter(t_elem)))
|
||||||
|
|
||||||
|
if text_elements and post_save_callbacks is not None:
|
||||||
|
def write_endnotes_back(output_path: Path) -> None:
|
||||||
try:
|
try:
|
||||||
endnotes_xml = etree.fromstring(endnotes_part.blob)
|
new_blob = etree.tostring(
|
||||||
for child in endnotes_xml:
|
endnotes_xml,
|
||||||
if child.tag == qn("w:endnote"):
|
xml_declaration=True,
|
||||||
for para_elem in child.findall(qn("w:p")):
|
encoding="UTF-8",
|
||||||
paragraph = Paragraph(para_elem, document)
|
standalone=True,
|
||||||
self._collect_from_paragraph(paragraph, text_elements)
|
)
|
||||||
|
tmp_path = output_path.with_suffix(".tmp_end")
|
||||||
|
with zipfile.ZipFile(output_path, "r") as zin, \
|
||||||
|
zipfile.ZipFile(tmp_path, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||||
|
for item in zin.namelist():
|
||||||
|
if item == "word/endnotes.xml":
|
||||||
|
zout.writestr(item, new_blob)
|
||||||
|
else:
|
||||||
|
zout.writestr(item, zin.read(item))
|
||||||
|
tmp_path.replace(output_path)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
_log_error("word_endnotes_parse_error", error=str(e))
|
_log_error("word_endnotes_writeback_error", error=str(e))
|
||||||
|
|
||||||
|
post_save_callbacks.append(write_endnotes_back)
|
||||||
|
|
||||||
def _collect_from_charts(
|
def _collect_from_charts(
|
||||||
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
self, document: Document, text_elements: List[Tuple[str, Callable[[str], None]]]
|
||||||
@@ -825,11 +905,16 @@ class WordTranslator:
|
|||||||
"""Re-inject chart translations into the .docx ZIP.
|
"""Re-inject chart translations into the .docx ZIP.
|
||||||
|
|
||||||
Modifies chart XML files in-place and rewrites the ZIP.
|
Modifies chart XML files in-place and rewrites the ZIP.
|
||||||
|
|
||||||
|
Uses the `element_path` collected during `_collect_charts_from_zip` to
|
||||||
|
uniquely identify each translatable element, even when the same text
|
||||||
|
value appears multiple times in the same chart (e.g. two series both
|
||||||
|
labelled "Revenue"). The old code matched by string equality, which
|
||||||
|
would translate only the first occurrence.
|
||||||
"""
|
"""
|
||||||
if not chart_translations:
|
if not chart_translations:
|
||||||
return
|
return
|
||||||
|
|
||||||
# Only proceed if at least one translation exists
|
|
||||||
translated_entries = [e for e in chart_translations if 'translated' in e and e['translated']]
|
translated_entries = [e for e in chart_translations if 'translated' in e and e['translated']]
|
||||||
if not translated_entries:
|
if not translated_entries:
|
||||||
return
|
return
|
||||||
@@ -846,25 +931,25 @@ class WordTranslator:
|
|||||||
chart_files_to_update[cf].append(entry)
|
chart_files_to_update[cf].append(entry)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Read all ZIP entries
|
|
||||||
with zipfile.ZipFile(output_path, 'r') as zf_in:
|
with zipfile.ZipFile(output_path, 'r') as zf_in:
|
||||||
existing_entries = zf_in.namelist()
|
existing_entries = zf_in.namelist()
|
||||||
|
|
||||||
# Create new ZIP in memory
|
|
||||||
buf = io.BytesIO()
|
buf = io.BytesIO()
|
||||||
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf_out:
|
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf_out:
|
||||||
for item in existing_entries:
|
for item in existing_entries:
|
||||||
data = zf_in.read(item)
|
data = zf_in.read(item)
|
||||||
|
|
||||||
if item in chart_files_to_update:
|
if item in chart_files_to_update:
|
||||||
# Parse, update, re-serialize this chart XML
|
|
||||||
try:
|
try:
|
||||||
chart_xml = etree.fromstring(data)
|
chart_xml = etree.fromstring(data)
|
||||||
|
|
||||||
for entry in chart_files_to_update[item]:
|
for entry in chart_files_to_update[item]:
|
||||||
# Find all <a:t> or <c:v> elements and match by original text
|
target = self._find_element_by_path(chart_xml, entry.get('element_path', ''))
|
||||||
|
if target is not None:
|
||||||
|
target.text = entry['translated']
|
||||||
|
else:
|
||||||
|
# Fallback: try to find by tag + original text (for
|
||||||
|
# entries that predate the path-based collection)
|
||||||
tag_to_find = f'{{{_NS_A}}}t'
|
tag_to_find = f'{{{_NS_A}}}t'
|
||||||
# Try both a:t and c:v
|
|
||||||
for t_elem in chart_xml.iter(tag_to_find):
|
for t_elem in chart_xml.iter(tag_to_find):
|
||||||
if t_elem.text and t_elem.text.strip() == entry['original']:
|
if t_elem.text and t_elem.text.strip() == entry['original']:
|
||||||
t_elem.text = entry['translated']
|
t_elem.text = entry['translated']
|
||||||
@@ -881,7 +966,6 @@ class WordTranslator:
|
|||||||
|
|
||||||
zf_out.writestr(item, data)
|
zf_out.writestr(item, data)
|
||||||
|
|
||||||
# Replace the output file with the updated ZIP
|
|
||||||
with open(output_path, 'wb') as f:
|
with open(output_path, 'wb') as f:
|
||||||
f.write(buf.getvalue())
|
f.write(buf.getvalue())
|
||||||
|
|
||||||
@@ -890,6 +974,41 @@ class WordTranslator:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
_log_error("word_chart_zip_rewrite_error", error=str(e))
|
_log_error("word_chart_zip_rewrite_error", error=str(e))
|
||||||
|
|
||||||
|
def _find_element_by_path(self, root, path: str):
|
||||||
|
"""
|
||||||
|
Navigate the XML tree using an XPath-like path produced by
|
||||||
|
`_get_element_path`. Returns None if the path is empty or invalid
|
||||||
|
(e.g. the tree structure changed between collect and apply).
|
||||||
|
"""
|
||||||
|
if not path:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
current = root
|
||||||
|
for segment in path.split('/'):
|
||||||
|
# Format: "tagname[index]"
|
||||||
|
if '[' not in segment or not segment.endswith(']'):
|
||||||
|
return None
|
||||||
|
tag_name, idx_str = segment[:-1].split('[', 1)
|
||||||
|
try:
|
||||||
|
idx = int(idx_str)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
children = list(current)
|
||||||
|
# The path used `list(parent).index(current)` which counts ALL
|
||||||
|
# children, so we use a simple position lookup.
|
||||||
|
if idx < 0 or idx >= len(children):
|
||||||
|
return None
|
||||||
|
candidate = children[idx]
|
||||||
|
# Verify tag name (with or without namespace) to avoid silent
|
||||||
|
# mis-navigation if the tree changed.
|
||||||
|
candidate_local_tag = candidate.tag.split('}')[-1] if '}' in candidate.tag else candidate.tag
|
||||||
|
if candidate_local_tag != tag_name:
|
||||||
|
return None
|
||||||
|
current = candidate
|
||||||
|
return current
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# SmartArt / Diagram support
|
# SmartArt / Diagram support
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -1036,8 +1155,12 @@ class WordTranslator:
|
|||||||
The whitespace is captured and reapplied after translation so that words
|
The whitespace is captured and reapplied after translation so that words
|
||||||
at formatting boundaries (e.g. bold/normal) do not get concatenated.
|
at formatting boundaries (e.g. bold/normal) do not get concatenated.
|
||||||
|
|
||||||
Handles runs both as direct children of <w:p> AND inside <w:hyperlink>
|
Note: python-docx's `paragraph.runs` only returns DIRECT child <w:r>
|
||||||
elements (used for TOC entries, cross-references, and bookmarks links).
|
elements, not those inside <w:hyperlink> (used for TOC entries,
|
||||||
|
cross-references, bookmark links). We therefore iterate the full
|
||||||
|
XML tree to find every <w:r> and use a set of element ids to
|
||||||
|
deduplicate — this avoids translating the same run twice while
|
||||||
|
ensuring hyperlink text IS picked up.
|
||||||
"""
|
"""
|
||||||
# Check full paragraph text including nested content (hyperlinks, etc.)
|
# Check full paragraph text including nested content (hyperlinks, etc.)
|
||||||
full_text = ''.join(
|
full_text = ''.join(
|
||||||
@@ -1046,15 +1169,30 @@ class WordTranslator:
|
|||||||
if not full_text:
|
if not full_text:
|
||||||
return
|
return
|
||||||
|
|
||||||
# Collect from direct child runs
|
# Collect every <w:r> element in the paragraph tree, including
|
||||||
|
# those nested in <w:hyperlink>, <w:smartTag>, etc. The dedup by
|
||||||
|
# element id is defensive — `paragraph.runs` and the manual iter
|
||||||
|
# below could overlap if python-docx starts surfacing nested runs.
|
||||||
|
seen_run_ids: set = set()
|
||||||
|
|
||||||
|
# 1) Direct runs (paragraph.runs is the python-docx-native API).
|
||||||
for run in paragraph.runs:
|
for run in paragraph.runs:
|
||||||
|
run_id = id(run._r)
|
||||||
|
if run_id in seen_run_ids:
|
||||||
|
continue
|
||||||
|
seen_run_ids.add(run_id)
|
||||||
if run.text and run.text.strip():
|
if run.text and run.text.strip():
|
||||||
self._append_run_translation(run, text_elements)
|
self._append_run_translation(run, text_elements)
|
||||||
|
|
||||||
# Collect from runs inside <w:hyperlink> elements
|
# 2) Runs nested inside <w:hyperlink> (TOC, cross-references).
|
||||||
# (TOC entries, cross-references — python-docx's paragraph.runs skips these)
|
# python-docx's `paragraph.runs` does NOT descend into hyperlinks in
|
||||||
for hl in paragraph._p.iter(qn('w:hyperlink')):
|
# version 1.x — we have to walk the XML ourselves.
|
||||||
for r_elem in hl.findall(qn('w:r')):
|
for r_elem in paragraph._p.iter(qn('w:r')):
|
||||||
|
run_id = id(r_elem)
|
||||||
|
if run_id in seen_run_ids:
|
||||||
|
continue
|
||||||
|
seen_run_ids.add(run_id)
|
||||||
|
# Build a Run wrapper so the setter API is consistent.
|
||||||
run = Run(r_elem, paragraph)
|
run = Run(r_elem, paragraph)
|
||||||
if run.text and run.text.strip():
|
if run.text and run.text.strip():
|
||||||
self._append_run_translation(run, text_elements)
|
self._append_run_translation(run, text_elements)
|
||||||
|
|||||||
@@ -8,7 +8,8 @@
|
|||||||
"build": "vite build",
|
"build": "vite build",
|
||||||
"preview": "vite preview",
|
"preview": "vite preview",
|
||||||
"clean": "rm -rf dist server.js",
|
"clean": "rm -rf dist server.js",
|
||||||
"lint": "tsc --noEmit"
|
"lint": "tsc --noEmit",
|
||||||
|
"test:quality": "node --experimental-strip-types --test tests/utils/scriptDetector.test.ts"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@google/genai": "^1.29.0",
|
"@google/genai": "^1.29.0",
|
||||||
|
|||||||
739
wordly.art---traduction-de-documents/src/utils/scriptDetector.ts
Normal file
739
wordly.art---traduction-de-documents/src/utils/scriptDetector.ts
Normal file
@@ -0,0 +1,739 @@
|
|||||||
|
/**
|
||||||
|
* L0 quality detector — TypeScript mirror of services/quality/ in Python.
|
||||||
|
*
|
||||||
|
* Detects:
|
||||||
|
* - wrong_script: translation is in the wrong script for the target language
|
||||||
|
* (e.g. user asks French, model returns Arabic — language confusion)
|
||||||
|
* - wrong_arabic_variant: for Arabic-script targets (ar/fa/ur/...),
|
||||||
|
* the translation uses characters of a DIFFERENT Arabic-script language
|
||||||
|
* (e.g. user asks Persian, model returns Urdu)
|
||||||
|
* - length_outlier / truncation_suspect: translation is wildly different
|
||||||
|
* in length from the source
|
||||||
|
* - prompt_leak: translation starts with "Translation:" or similar
|
||||||
|
* - repetition_hallucination: "xxx..." or "the the the the" pattern
|
||||||
|
*
|
||||||
|
* Zero dependencies. Works in browser AND in Node.js (for tests).
|
||||||
|
* Designed to be ADDITIVE — never blocks the translation flow.
|
||||||
|
*/
|
||||||
|
|
||||||
|
// ---------- Types ----------
|
||||||
|
|
||||||
|
export interface QualityCheckResult {
|
||||||
|
passed: boolean;
|
||||||
|
score: number; // 0.0 to 1.0
|
||||||
|
issues: string[];
|
||||||
|
detectedScript: string | null;
|
||||||
|
expectedScript: string | null;
|
||||||
|
details: Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DocumentQualityResult {
|
||||||
|
passed: boolean;
|
||||||
|
score: number;
|
||||||
|
chunkCount: number;
|
||||||
|
failedChunkCount: number;
|
||||||
|
issues: Record<string, number>;
|
||||||
|
samples: Array<{
|
||||||
|
index: number;
|
||||||
|
issues: string[];
|
||||||
|
sourcePreview: string;
|
||||||
|
translatedPreview: string;
|
||||||
|
details: Record<string, unknown>;
|
||||||
|
}>;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Unicode ranges per script ----------
|
||||||
|
// Mirrors services/quality/config.py in Python.
|
||||||
|
|
||||||
|
type Range = readonly [number, number]; // [start, end] inclusive
|
||||||
|
|
||||||
|
const UNICODE_RANGES: Record<string, readonly Range[]> = {
|
||||||
|
cyrillic: [
|
||||||
|
[0x0400, 0x04ff],
|
||||||
|
[0x0500, 0x052f],
|
||||||
|
],
|
||||||
|
greek: [
|
||||||
|
[0x0370, 0x03ff],
|
||||||
|
],
|
||||||
|
arabic: [
|
||||||
|
[0x0600, 0x06ff],
|
||||||
|
[0x0750, 0x077f],
|
||||||
|
[0x08a0, 0x08ff],
|
||||||
|
],
|
||||||
|
hebrew: [
|
||||||
|
[0x0590, 0x05ff],
|
||||||
|
],
|
||||||
|
devanagari: [
|
||||||
|
[0x0900, 0x097f],
|
||||||
|
],
|
||||||
|
bengali: [
|
||||||
|
[0x0980, 0x09ff],
|
||||||
|
],
|
||||||
|
tamil: [
|
||||||
|
[0x0b80, 0x0bff],
|
||||||
|
],
|
||||||
|
telugu: [
|
||||||
|
[0x0c00, 0x0c7f],
|
||||||
|
],
|
||||||
|
kannada: [
|
||||||
|
[0x0c80, 0x0cff],
|
||||||
|
],
|
||||||
|
malayalam: [
|
||||||
|
[0x0d00, 0x0d7f],
|
||||||
|
],
|
||||||
|
sinhala: [
|
||||||
|
[0x0d80, 0x0dff],
|
||||||
|
],
|
||||||
|
gujarati: [
|
||||||
|
[0x0a80, 0x0aff],
|
||||||
|
],
|
||||||
|
gurmukhi: [
|
||||||
|
[0x0a00, 0x0a7f],
|
||||||
|
],
|
||||||
|
thai: [
|
||||||
|
[0x0e00, 0x0e7f],
|
||||||
|
],
|
||||||
|
lao: [
|
||||||
|
[0x0e80, 0x0eff],
|
||||||
|
],
|
||||||
|
burmese: [
|
||||||
|
[0x1000, 0x109f],
|
||||||
|
],
|
||||||
|
khmer: [
|
||||||
|
[0x1780, 0x17ff],
|
||||||
|
],
|
||||||
|
cjk: [
|
||||||
|
[0x4e00, 0x9fff],
|
||||||
|
[0x3400, 0x4dbf],
|
||||||
|
],
|
||||||
|
hiragana_katakana: [
|
||||||
|
[0x3040, 0x309f],
|
||||||
|
[0x30a0, 0x30ff],
|
||||||
|
],
|
||||||
|
hangul: [
|
||||||
|
[0xac00, 0xd7af],
|
||||||
|
[0x1100, 0x11ff],
|
||||||
|
[0xa960, 0xa97f],
|
||||||
|
],
|
||||||
|
georgian: [
|
||||||
|
[0x10a0, 0x10ff],
|
||||||
|
],
|
||||||
|
armenian: [
|
||||||
|
[0x0530, 0x058f],
|
||||||
|
],
|
||||||
|
ethiopic: [
|
||||||
|
[0x1200, 0x137f],
|
||||||
|
[0x1380, 0x139f],
|
||||||
|
],
|
||||||
|
tibetan: [
|
||||||
|
[0x0f00, 0x0fff],
|
||||||
|
],
|
||||||
|
thaana: [
|
||||||
|
[0x0780, 0x07bf],
|
||||||
|
],
|
||||||
|
latin: [],
|
||||||
|
};
|
||||||
|
|
||||||
|
// ---------- Language → script mapping ----------
|
||||||
|
|
||||||
|
const LANG_TO_SCRIPT: Record<string, string> = {
|
||||||
|
// Latin-script languages
|
||||||
|
en: 'latin', fr: 'latin', de: 'latin', es: 'latin', it: 'latin',
|
||||||
|
pt: 'latin', nl: 'latin', pl: 'latin', tr: 'latin', vi: 'latin',
|
||||||
|
id: 'latin', ms: 'latin', ro: 'latin', cs: 'latin', sv: 'latin',
|
||||||
|
da: 'latin', fi: 'latin', no: 'latin', nb: 'latin', nn: 'latin',
|
||||||
|
hu: 'latin', sk: 'latin', sl: 'latin', lt: 'latin', lv: 'latin',
|
||||||
|
et: 'latin', sq: 'latin', az: 'latin', uz: 'latin', kk: 'latin',
|
||||||
|
ky: 'latin', tk: 'latin', sw: 'latin', eu: 'latin', gl: 'latin',
|
||||||
|
is: 'latin', ga: 'latin', mt: 'latin', ca: 'latin', hr: 'latin',
|
||||||
|
bs: 'latin', af: 'latin', cy: 'latin', lb: 'latin', fo: 'latin',
|
||||||
|
br: 'latin', co: 'latin', fy: 'latin', gd: 'latin', gu: 'latin',
|
||||||
|
ht: 'latin', haw: 'latin', hmn: 'latin', jv: 'latin', ku: 'latin',
|
||||||
|
mg: 'latin', mi: 'latin', mn: 'latin', nso: 'latin', ny: 'latin',
|
||||||
|
oc: 'latin', os: 'latin', ps: 'latin', qu: 'latin', rw: 'latin',
|
||||||
|
sc: 'latin', si: 'latin', sm: 'latin', sn: 'latin', so: 'latin',
|
||||||
|
st: 'latin', su: 'latin', tg: 'latin', tt: 'latin', ty: 'latin',
|
||||||
|
ug: 'latin', vo: 'latin', wa: 'latin', wo: 'latin', xh: 'latin',
|
||||||
|
yi: 'latin', zu: 'latin',
|
||||||
|
// Cyrillic overrides
|
||||||
|
ru: 'cyrillic', uk: 'cyrillic', be: 'cyrillic', sr: 'cyrillic',
|
||||||
|
mk: 'cyrillic', bg: 'cyrillic', ce: 'cyrillic', cv: 'cyrillic',
|
||||||
|
sah: 'cyrillic', udm: 'cyrillic', rue: 'cyrillic', ab: 'cyrillic',
|
||||||
|
// Greek
|
||||||
|
el: 'greek',
|
||||||
|
// Arabic-script languages
|
||||||
|
ar: 'arabic', fa: 'arabic', ur: 'arabic', ps: 'arabic',
|
||||||
|
ku: 'arabic', sd: 'arabic', ckb: 'arabic', bal: 'arabic',
|
||||||
|
bqi: 'arabic', glk: 'arabic', mzn: 'arabic',
|
||||||
|
// Hebrew & Yiddish
|
||||||
|
he: 'hebrew',
|
||||||
|
yi: 'hebrew', // Yiddish uses Hebrew script, not Arabic
|
||||||
|
// Thaana (Maldivian)
|
||||||
|
dv: 'thaana', // Maldivian uses Thaana, not Arabic
|
||||||
|
// Indian scripts
|
||||||
|
hi: 'devanagari', ne: 'devanagari', mr: 'devanagari', sa: 'devanagari',
|
||||||
|
mai: 'devanagari', bho: 'devanagari', awa: 'devanagari',
|
||||||
|
bn: 'bengali', as: 'bengali',
|
||||||
|
ta: 'tamil',
|
||||||
|
te: 'telugu',
|
||||||
|
kn: 'kannada',
|
||||||
|
ml: 'malayalam',
|
||||||
|
si: 'sinhala',
|
||||||
|
gu: 'gujarati',
|
||||||
|
pa: 'gurmukhi',
|
||||||
|
// SE Asia
|
||||||
|
th: 'thai',
|
||||||
|
lo: 'lao',
|
||||||
|
my: 'burmese',
|
||||||
|
km: 'khmer',
|
||||||
|
// East Asia
|
||||||
|
zh: 'cjk', 'zh-cn': 'cjk', 'zh-tw': 'cjk', 'zh-hk': 'cjk',
|
||||||
|
ja: 'hiragana_katakana',
|
||||||
|
ko: 'hangul',
|
||||||
|
// Others
|
||||||
|
ka: 'georgian',
|
||||||
|
hy: 'armenian',
|
||||||
|
am: 'ethiopic', ti: 'ethiopic', gez: 'ethiopic', tig: 'ethiopic',
|
||||||
|
bo: 'tibetan', dz: 'tibetan',
|
||||||
|
};
|
||||||
|
|
||||||
|
// ---------- Discriminating characters for Arabic-script languages ----------
|
||||||
|
|
||||||
|
const DISCRIMINATING_CHARS: Record<string, ReadonlySet<string>> = {
|
||||||
|
fa: new Set('پچژگ'), // Persian
|
||||||
|
ur: new Set('ٹڈڑے'), // Urdu
|
||||||
|
ps: new Set('ټډړږښ'), // Pashto
|
||||||
|
ku: new Set('ڕێ'), // Kurdish
|
||||||
|
ckb: new Set('ڕێ'), // Sorani Kurdish
|
||||||
|
sd: new Set('ٿ'), // Sindhi
|
||||||
|
ug: new Set('ۇۆې'), // Uyghur
|
||||||
|
bal: new Set('ێ'), // Balochi
|
||||||
|
};
|
||||||
|
|
||||||
|
// ---------- Thresholds ----------
|
||||||
|
|
||||||
|
const MIN_RATIO_IN_SCRIPT = 0.60;
|
||||||
|
|
||||||
|
// ---------- Helpers ----------
|
||||||
|
|
||||||
|
/** Get the script id for a language code. */
|
||||||
|
export function getScript(langCode: string | null | undefined): string {
|
||||||
|
if (!langCode) return 'latin';
|
||||||
|
return LANG_TO_SCRIPT[langCode.toLowerCase()] ?? 'latin';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** True if the language uses an Arabic-script block. */
|
||||||
|
export function isArabicScriptLang(langCode: string | null | undefined): boolean {
|
||||||
|
return getScript(langCode) === 'arabic';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Get the Unicode ranges for a script id. Empty array for 'latin'. */
|
||||||
|
function getRanges(scriptId: string): readonly Range[] {
|
||||||
|
return UNICODE_RANGES[scriptId] ?? [];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Iterate code points of a string (handles surrogate pairs correctly). */
|
||||||
|
function* codePoints(text: string): Generator<number> {
|
||||||
|
for (const ch of text) {
|
||||||
|
yield ch.codePointAt(0) ?? 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Check if a code point falls in any of the (start, end) ranges. */
|
||||||
|
function isInRanges(code: number, ranges: readonly Range[]): boolean {
|
||||||
|
for (const [start, end] of ranges) {
|
||||||
|
if (code >= start && code <= end) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const LETTER_REGEX = /\p{L}/u;
|
||||||
|
|
||||||
|
/** Count alphabetic characters (using Unicode property escapes). */
|
||||||
|
function countLetters(text: string): number {
|
||||||
|
let count = 0;
|
||||||
|
for (const ch of text) {
|
||||||
|
if (LETTER_REGEX.test(ch)) count++;
|
||||||
|
}
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Count how many alphabetic characters fall within the given ranges. */
|
||||||
|
function countInScript(text: string, ranges: readonly Range[]): number {
|
||||||
|
if (ranges.length === 0) {
|
||||||
|
// Latin / unknown — all letters match.
|
||||||
|
return countLetters(text);
|
||||||
|
}
|
||||||
|
let count = 0;
|
||||||
|
for (const ch of text) {
|
||||||
|
if (!LETTER_REGEX.test(ch)) continue;
|
||||||
|
const cp = ch.codePointAt(0) ?? 0;
|
||||||
|
if (isInRanges(cp, ranges)) count++;
|
||||||
|
}
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Arabic-script variant detection ----------
|
||||||
|
|
||||||
|
export interface ArabicVariantResult {
|
||||||
|
verdict: 'pass' | 'fail' | 'skip';
|
||||||
|
claimedLang: string | null;
|
||||||
|
detectedVariants: string[];
|
||||||
|
arabicRatio?: number;
|
||||||
|
reason: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function detectArabicVariant(
|
||||||
|
text: string,
|
||||||
|
claimedLang: string | null,
|
||||||
|
): ArabicVariantResult {
|
||||||
|
if (!text || !text.trim()) {
|
||||||
|
return { verdict: 'skip', claimedLang, detectedVariants: [], reason: 'empty text' };
|
||||||
|
}
|
||||||
|
|
||||||
|
const arabicRanges = getRanges('arabic');
|
||||||
|
const letters = countLetters(text);
|
||||||
|
if (letters === 0) {
|
||||||
|
return { verdict: 'skip', claimedLang, detectedVariants: [], reason: 'no letters' };
|
||||||
|
}
|
||||||
|
|
||||||
|
const inArabic = countInScript(text, arabicRanges);
|
||||||
|
const arabicRatio = inArabic / letters;
|
||||||
|
|
||||||
|
if (arabicRatio < MIN_RATIO_IN_SCRIPT) {
|
||||||
|
return {
|
||||||
|
verdict: 'skip',
|
||||||
|
claimedLang,
|
||||||
|
detectedVariants: [],
|
||||||
|
arabicRatio: round(arabicRatio, 3),
|
||||||
|
reason: 'not in Arabic script',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!isArabicScriptLang(claimedLang)) {
|
||||||
|
return {
|
||||||
|
verdict: 'skip',
|
||||||
|
claimedLang,
|
||||||
|
detectedVariants: [],
|
||||||
|
arabicRatio: round(arabicRatio, 3),
|
||||||
|
reason: 'target is not an Arabic-script language',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Text is Arabic-script AND target is Arabic-script: check the variant.
|
||||||
|
const detected = new Set<string>();
|
||||||
|
for (const [langCode, chars] of Object.entries(DISCRIMINATING_CHARS)) {
|
||||||
|
if (chars.size === 0) continue;
|
||||||
|
for (const ch of text) {
|
||||||
|
if (chars.has(ch)) {
|
||||||
|
detected.add(langCode);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (detected.size > 0 && claimedLang && !detected.has(claimedLang.toLowerCase())) {
|
||||||
|
return {
|
||||||
|
verdict: 'fail',
|
||||||
|
claimedLang,
|
||||||
|
detectedVariants: Array.from(detected).sort(),
|
||||||
|
arabicRatio: round(arabicRatio, 3),
|
||||||
|
reason: `target=${claimedLang} but text contains characters typical of ${Array.from(detected).sort().join(', ')}`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
verdict: 'pass',
|
||||||
|
claimedLang,
|
||||||
|
detectedVariants: detected.size > 0 ? Array.from(detected).sort() : (claimedLang ? [claimedLang] : []),
|
||||||
|
arabicRatio: round(arabicRatio, 3),
|
||||||
|
reason: 'ok',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Length check ----------
|
||||||
|
|
||||||
|
const RATIO_MAX = 3.5;
|
||||||
|
const RATIO_MIN = 0.15;
|
||||||
|
const ABSOLUTE_MIN_LENGTH = 2;
|
||||||
|
const MIN_SOURCE_LENGTH_FOR_RATIO = 20;
|
||||||
|
|
||||||
|
export interface LengthCheckResult {
|
||||||
|
issue: string | null;
|
||||||
|
ratio: number | null;
|
||||||
|
sourceLength: number;
|
||||||
|
translatedLength: number;
|
||||||
|
note?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function lengthCheck(sourceText: string, translatedText: string): LengthCheckResult {
|
||||||
|
if (!sourceText) {
|
||||||
|
return {
|
||||||
|
issue: null,
|
||||||
|
ratio: null,
|
||||||
|
sourceLength: 0,
|
||||||
|
translatedLength: (translatedText || '').length,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const srcLen = sourceText.trim().length;
|
||||||
|
const transLen = (translatedText || '').trim().length;
|
||||||
|
|
||||||
|
if (transLen === 0) {
|
||||||
|
return {
|
||||||
|
issue: 'truncation_suspect',
|
||||||
|
ratio: 0,
|
||||||
|
sourceLength: srcLen,
|
||||||
|
translatedLength: transLen,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isMostlyNumeric(sourceText)) {
|
||||||
|
return {
|
||||||
|
issue: null,
|
||||||
|
ratio: null,
|
||||||
|
sourceLength: srcLen,
|
||||||
|
translatedLength: transLen,
|
||||||
|
note: 'skipped: numeric source',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (srcLen < MIN_SOURCE_LENGTH_FOR_RATIO) {
|
||||||
|
return {
|
||||||
|
issue: null,
|
||||||
|
ratio: null,
|
||||||
|
sourceLength: srcLen,
|
||||||
|
translatedLength: transLen,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const ratio = transLen / srcLen;
|
||||||
|
|
||||||
|
if (ratio > RATIO_MAX) {
|
||||||
|
return {
|
||||||
|
issue: 'length_outlier',
|
||||||
|
ratio: round(ratio, 2),
|
||||||
|
sourceLength: srcLen,
|
||||||
|
translatedLength: transLen,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (ratio < RATIO_MIN) {
|
||||||
|
return {
|
||||||
|
issue: 'truncation_suspect',
|
||||||
|
ratio: round(ratio, 2),
|
||||||
|
sourceLength: srcLen,
|
||||||
|
translatedLength: transLen,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
issue: null,
|
||||||
|
ratio: round(ratio, 2),
|
||||||
|
sourceLength: srcLen,
|
||||||
|
translatedLength: transLen,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function isMostlyNumeric(text: string): boolean {
|
||||||
|
if (!text) return false;
|
||||||
|
const chars: string[] = [];
|
||||||
|
for (const ch of text) {
|
||||||
|
if (!/\s/.test(ch)) chars.push(ch);
|
||||||
|
}
|
||||||
|
if (chars.length === 0) return false;
|
||||||
|
const digitCount = chars.filter((c) => /\d/.test(c)).length;
|
||||||
|
return digitCount / chars.length >= 0.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Pattern leak / repetition ----------
|
||||||
|
|
||||||
|
const LEAK_PREFIX_PATTERNS: RegExp[] = [
|
||||||
|
/^(translation|translated text|here is the translation|here'?s the translation)\s*[::-]/i,
|
||||||
|
/^(voici (la |ma )?traduction|traduction\s*[::-])\b/i,
|
||||||
|
/^(原文|译|翻译|译为|以下是)\s*[::]?/u,
|
||||||
|
/^(sure,?\s+here'?s?\s+(the\s+)?translation|of course,?\s+here)/i,
|
||||||
|
/^(\*\*|__|#)\s*translation/i,
|
||||||
|
/^translated from\s+\w+\s+to\s+\w+\s*[::-]/i,
|
||||||
|
];
|
||||||
|
|
||||||
|
const REPETITION_THRESHOLD = 5;
|
||||||
|
const CHAR_REPETITION_THRESHOLD = 20;
|
||||||
|
|
||||||
|
export interface PatternCheckResult {
|
||||||
|
issue: string | null;
|
||||||
|
matchedPattern: string | null;
|
||||||
|
repetitionCount: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function patternCheck(text: string): PatternCheckResult {
|
||||||
|
if (!text || !text.trim()) {
|
||||||
|
return { issue: null, matchedPattern: null, repetitionCount: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
const stripped = text.trimStart();
|
||||||
|
|
||||||
|
// 1. Prompt leak
|
||||||
|
for (const pat of LEAK_PREFIX_PATTERNS) {
|
||||||
|
if (pat.test(stripped)) {
|
||||||
|
return {
|
||||||
|
issue: 'prompt_leak',
|
||||||
|
matchedPattern: pat.source,
|
||||||
|
repetitionCount: null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Token-level repetition
|
||||||
|
const tokens = stripped.split(/\s+/).filter((t) => t.length > 0);
|
||||||
|
const tokenRep = maxConsecutiveTokenRepetition(tokens);
|
||||||
|
if (tokenRep >= REPETITION_THRESHOLD) {
|
||||||
|
return {
|
||||||
|
issue: 'repetition_hallucination',
|
||||||
|
matchedPattern: null,
|
||||||
|
repetitionCount: tokenRep,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. Character-level repetition
|
||||||
|
const charRep = maxConsecutiveCharRepetition(stripped);
|
||||||
|
if (charRep >= CHAR_REPETITION_THRESHOLD) {
|
||||||
|
return {
|
||||||
|
issue: 'repetition_hallucination',
|
||||||
|
matchedPattern: null,
|
||||||
|
repetitionCount: charRep,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
issue: null,
|
||||||
|
matchedPattern: null,
|
||||||
|
repetitionCount: Math.max(tokenRep, charRep) || null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function maxConsecutiveTokenRepetition(tokens: string[]): number {
|
||||||
|
if (tokens.length === 0) return 0;
|
||||||
|
const norm = tokens.map((t) => t.toLowerCase().replace(/[.,!?;:"'`()\[\]{}]/g, ''));
|
||||||
|
let maxRun = 1;
|
||||||
|
let currentRun = 1;
|
||||||
|
for (let i = 1; i < norm.length; i++) {
|
||||||
|
if (norm[i] && norm[i] === norm[i - 1]) {
|
||||||
|
currentRun++;
|
||||||
|
if (currentRun > maxRun) maxRun = currentRun;
|
||||||
|
} else {
|
||||||
|
currentRun = 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return maxRun;
|
||||||
|
}
|
||||||
|
|
||||||
|
function maxConsecutiveCharRepetition(text: string): number {
|
||||||
|
if (!text) return 0;
|
||||||
|
let maxRun = 1;
|
||||||
|
let currentRun = 1;
|
||||||
|
for (let i = 1; i < text.length; i++) {
|
||||||
|
if (text[i] === text[i - 1] && !/\s/.test(text[i])) {
|
||||||
|
currentRun++;
|
||||||
|
if (currentRun > maxRun) maxRun = currentRun;
|
||||||
|
} else {
|
||||||
|
currentRun = 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return maxRun;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Heuristic actual-script detection (for diagnostics) ----------
|
||||||
|
|
||||||
|
const SCRIPT_DETECTION_ORDER: readonly string[] = [
|
||||||
|
'hiragana_katakana', 'hangul', 'cjk', 'thai', 'lao', 'burmese',
|
||||||
|
'khmer', 'devanagari', 'bengali', 'tamil', 'telugu', 'kannada',
|
||||||
|
'malayalam', 'sinhala', 'gujarati', 'gurmukhi',
|
||||||
|
'arabic', 'hebrew', 'cyrillic', 'greek', 'armenian', 'georgian',
|
||||||
|
'ethiopic', 'tibetan', 'thaana',
|
||||||
|
];
|
||||||
|
|
||||||
|
function detectActualScript(text: string): string {
|
||||||
|
const letters = countLetters(text);
|
||||||
|
if (letters === 0) return 'unknown';
|
||||||
|
for (const scriptId of SCRIPT_DETECTION_ORDER) {
|
||||||
|
const ranges = getRanges(scriptId);
|
||||||
|
const inScript = countInScript(text, ranges);
|
||||||
|
if (inScript / letters > 0.4) return scriptId;
|
||||||
|
}
|
||||||
|
return 'latin';
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Main per-chunk evaluation ----------
|
||||||
|
|
||||||
|
export function evaluateChunk(
|
||||||
|
sourceText: string,
|
||||||
|
translatedText: string | null | undefined,
|
||||||
|
targetLang: string | null | undefined,
|
||||||
|
): QualityCheckResult {
|
||||||
|
if (translatedText === null || translatedText === undefined) {
|
||||||
|
return {
|
||||||
|
passed: true,
|
||||||
|
score: 0,
|
||||||
|
issues: ['empty_translation'],
|
||||||
|
detectedScript: null,
|
||||||
|
expectedScript: null,
|
||||||
|
details: { reason: 'translation is null/undefined' },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const text = translatedText.trim();
|
||||||
|
if (!text) {
|
||||||
|
return {
|
||||||
|
passed: true,
|
||||||
|
score: 0,
|
||||||
|
issues: ['empty_translation'],
|
||||||
|
detectedScript: null,
|
||||||
|
expectedScript: null,
|
||||||
|
details: { reason: 'translation is empty or whitespace-only' },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const targetLangLower = (targetLang || '').toLowerCase() || null;
|
||||||
|
const issues: string[] = [];
|
||||||
|
const details: Record<string, unknown> = {};
|
||||||
|
|
||||||
|
// --- Script detection ---
|
||||||
|
const expectedScript = getScript(targetLangLower);
|
||||||
|
const expectedRanges = getRanges(expectedScript);
|
||||||
|
const letters = countLetters(text);
|
||||||
|
|
||||||
|
let scriptScore: number;
|
||||||
|
let detectedScript: string | null;
|
||||||
|
|
||||||
|
if (letters === 0) {
|
||||||
|
scriptScore = 1.0;
|
||||||
|
detectedScript = expectedScript;
|
||||||
|
details.script_check = 'skipped: no alphabetic characters';
|
||||||
|
} else {
|
||||||
|
detectedScript = detectActualScript(text);
|
||||||
|
const inExpected = countInScript(text, expectedRanges);
|
||||||
|
scriptScore = inExpected / letters;
|
||||||
|
|
||||||
|
details.script_score = round(scriptScore, 3);
|
||||||
|
details.letters_in_text = letters;
|
||||||
|
details.letters_in_script = inExpected;
|
||||||
|
details.detected_script = detectedScript;
|
||||||
|
details.expected_script = expectedScript;
|
||||||
|
details.min_ratio = MIN_RATIO_IN_SCRIPT;
|
||||||
|
|
||||||
|
if (expectedScript !== 'latin' && expectedRanges.length > 0) {
|
||||||
|
// Specific non-Latin target.
|
||||||
|
if (scriptScore < MIN_RATIO_IN_SCRIPT) {
|
||||||
|
issues.push('wrong_script');
|
||||||
|
details.reason = `only ${Math.round(scriptScore * 100)}% of letters match ${expectedScript} script; text appears to be in ${detectedScript}`;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Latin target. If detected script is clearly non-Latin, fail.
|
||||||
|
if (detectedScript && detectedScript !== 'latin' && detectedScript !== 'unknown') {
|
||||||
|
const nonLatinRanges = getRanges(detectedScript);
|
||||||
|
const inDetected = countInScript(text, nonLatinRanges);
|
||||||
|
const nonLatinConfidence = inDetected / letters;
|
||||||
|
if (nonLatinConfidence >= 0.7) {
|
||||||
|
issues.push('wrong_script');
|
||||||
|
details.reason = `target is Latin but ${Math.round(nonLatinConfidence * 100)}% of letters are in ${detectedScript} script — language confusion`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Arabic-script variant detection ---
|
||||||
|
if (isArabicScriptLang(targetLangLower)) {
|
||||||
|
const variantResult = detectArabicVariant(text, targetLangLower);
|
||||||
|
details.arabic_variant = variantResult;
|
||||||
|
if (variantResult.verdict === 'fail') {
|
||||||
|
issues.push('wrong_arabic_variant');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Length sanity ---
|
||||||
|
const lengthResult = lengthCheck(sourceText, text);
|
||||||
|
details.length = lengthResult;
|
||||||
|
if (lengthResult.issue) {
|
||||||
|
issues.push(lengthResult.issue);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Pattern leak / repetition ---
|
||||||
|
const leakResult = patternCheck(text);
|
||||||
|
details.pattern_check = leakResult;
|
||||||
|
if (leakResult.issue) {
|
||||||
|
issues.push(leakResult.issue);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Aggregate ---
|
||||||
|
const passed = issues.length === 0;
|
||||||
|
const nChecks = 3;
|
||||||
|
const nFailed = issues.filter((issue) =>
|
||||||
|
['wrong_script', 'wrong_arabic_variant', 'length_outlier', 'truncation_suspect', 'prompt_leak', 'repetition_hallucination'].includes(issue),
|
||||||
|
).length;
|
||||||
|
const score = Math.max(0, 1 - nFailed / nChecks);
|
||||||
|
|
||||||
|
return {
|
||||||
|
passed,
|
||||||
|
score: round(score, 3),
|
||||||
|
issues,
|
||||||
|
detectedScript,
|
||||||
|
expectedScript,
|
||||||
|
details,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Document-level aggregation ----------
|
||||||
|
|
||||||
|
export function evaluateDocument(
|
||||||
|
sourceChunks: string[],
|
||||||
|
translatedChunks: string[],
|
||||||
|
targetLang: string | null | undefined,
|
||||||
|
sampleSize: number = 50,
|
||||||
|
): DocumentQualityResult {
|
||||||
|
const n = Math.min(sourceChunks.length, translatedChunks.length);
|
||||||
|
const chunkResults: QualityCheckResult[] = [];
|
||||||
|
const issuesCount: Record<string, number> = {};
|
||||||
|
const samples: DocumentQualityResult['samples'] = [];
|
||||||
|
let scoreSum = 0;
|
||||||
|
let failedCount = 0;
|
||||||
|
|
||||||
|
for (let i = 0; i < n; i++) {
|
||||||
|
const r = evaluateChunk(sourceChunks[i], translatedChunks[i], targetLang);
|
||||||
|
chunkResults.push(r);
|
||||||
|
scoreSum += r.score;
|
||||||
|
for (const issue of r.issues) {
|
||||||
|
issuesCount[issue] = (issuesCount[issue] || 0) + 1;
|
||||||
|
}
|
||||||
|
if (!r.passed) {
|
||||||
|
failedCount++;
|
||||||
|
if (samples.length < sampleSize) {
|
||||||
|
samples.push({
|
||||||
|
index: i,
|
||||||
|
issues: r.issues,
|
||||||
|
sourcePreview: (sourceChunks[i] || '').slice(0, 80),
|
||||||
|
translatedPreview: (translatedChunks[i] || '').slice(0, 80),
|
||||||
|
details: r.details,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const meanScore = n > 0 ? scoreSum / n : 0;
|
||||||
|
const passed = failedCount === 0;
|
||||||
|
|
||||||
|
return {
|
||||||
|
passed,
|
||||||
|
score: round(meanScore, 3),
|
||||||
|
chunkCount: n,
|
||||||
|
failedChunkCount: failedCount,
|
||||||
|
issues: issuesCount,
|
||||||
|
samples,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Utilities ----------
|
||||||
|
|
||||||
|
function round(value: number, decimals: number): number {
|
||||||
|
const factor = 10 ** decimals;
|
||||||
|
return Math.round(value * factor) / factor;
|
||||||
|
}
|
||||||
@@ -0,0 +1,421 @@
|
|||||||
|
/**
|
||||||
|
* Tests for src/utils/scriptDetector.ts
|
||||||
|
*
|
||||||
|
* Run with:
|
||||||
|
* npx tsx --test tests/utils/scriptDetector.test.ts
|
||||||
|
*
|
||||||
|
* Or in package.json scripts:
|
||||||
|
* "test:quality": "tsx --test tests/utils/scriptDetector.test.ts"
|
||||||
|
*/
|
||||||
|
import { describe, it } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
|
||||||
|
import {
|
||||||
|
getScript,
|
||||||
|
isArabicScriptLang,
|
||||||
|
evaluateChunk,
|
||||||
|
evaluateDocument,
|
||||||
|
detectArabicVariant,
|
||||||
|
lengthCheck,
|
||||||
|
patternCheck,
|
||||||
|
} from '../../src/utils/scriptDetector.ts';
|
||||||
|
|
||||||
|
describe('getScript', () => {
|
||||||
|
it('maps cyrillic languages', () => {
|
||||||
|
assert.equal(getScript('ru'), 'cyrillic');
|
||||||
|
assert.equal(getScript('uk'), 'cyrillic');
|
||||||
|
assert.equal(getScript('be'), 'cyrillic');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps latin languages', () => {
|
||||||
|
assert.equal(getScript('en'), 'latin');
|
||||||
|
assert.equal(getScript('fr'), 'latin');
|
||||||
|
assert.equal(getScript('de'), 'latin');
|
||||||
|
assert.equal(getScript('vi'), 'latin');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps CJK languages', () => {
|
||||||
|
assert.equal(getScript('zh'), 'cjk');
|
||||||
|
assert.equal(getScript('zh-cn'), 'cjk');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps Japanese to hiragana_katakana', () => {
|
||||||
|
assert.equal(getScript('ja'), 'hiragana_katakana');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps Korean to hangul', () => {
|
||||||
|
assert.equal(getScript('ko'), 'hangul');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps Yiddish to hebrew (not arabic)', () => {
|
||||||
|
assert.equal(getScript('yi'), 'hebrew');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('maps Maldivian to thaana (not arabic)', () => {
|
||||||
|
assert.equal(getScript('dv'), 'thaana');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to latin for unknown', () => {
|
||||||
|
assert.equal(getScript('xx'), 'latin');
|
||||||
|
assert.equal(getScript(''), 'latin');
|
||||||
|
assert.equal(getScript(null), 'latin');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is case-insensitive', () => {
|
||||||
|
assert.equal(getScript('FR'), 'latin');
|
||||||
|
assert.equal(getScript('ZH-CN'), 'cjk');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('isArabicScriptLang', () => {
|
||||||
|
it('returns true for Arabic-script langs', () => {
|
||||||
|
assert.equal(isArabicScriptLang('ar'), true);
|
||||||
|
assert.equal(isArabicScriptLang('fa'), true);
|
||||||
|
assert.equal(isArabicScriptLang('ur'), true);
|
||||||
|
assert.equal(isArabicScriptLang('ckb'), true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns false for non-Arabic langs', () => {
|
||||||
|
assert.equal(isArabicScriptLang('fr'), false);
|
||||||
|
assert.equal(isArabicScriptLang('he'), false);
|
||||||
|
assert.equal(isArabicScriptLang('en'), false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Happy path ----------
|
||||||
|
|
||||||
|
describe('evaluateChunk — correct script', () => {
|
||||||
|
it('russian', () => {
|
||||||
|
const r = evaluateChunk('Hello world', 'Привет мир', 'ru');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
assert.ok(!r.issues.includes('wrong_script'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('chinese', () => {
|
||||||
|
const r = evaluateChunk('Hello', '你好世界', 'zh');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('arabic', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'مرحبا بالعالم', 'ar');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('persian (with unique chars)', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'سلام چطوری؟ من پژوهشگر هستم', 'fa');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('french', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Bonjour le monde', 'fr');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('hebrew', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'שלום עולם', 'he');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('korean', () => {
|
||||||
|
const r = evaluateChunk('Hello', '안녕하세요 세계', 'ko');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('japanese', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'こんにちは世界', 'ja');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('hindi', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'नमस्ते दुनिया', 'hi');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('thai', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'สวัสดีชาวโลก', 'th');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('greek', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Γεια σας κόσμε', 'el');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Wrong script detection ----------
|
||||||
|
|
||||||
|
describe('evaluateChunk — wrong script', () => {
|
||||||
|
it('french text for japanese target', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Bonjour le monde', 'ja');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_script'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('arabic text for french target (language confusion)', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'مرحبا بالعالم', 'fr');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_script'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('russian text for english target', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Привет мир', 'en');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_script'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('chinese text for korean target', () => {
|
||||||
|
const r = evaluateChunk('Hello', '你好世界', 'ko');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_script'));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Arabic variant discrimination ----------
|
||||||
|
|
||||||
|
describe('evaluateChunk — Arabic variant discrimination', () => {
|
||||||
|
it('persian text for arabic target fails', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'سلام چطوری؟', 'ar');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_arabic_variant'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('urdu text for persian target fails', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'السلام ٹڈ', 'fa');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_arabic_variant'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('pashto text for arabic target fails', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'السلام ټډړ', 'ar');
|
||||||
|
assert.equal(r.passed, false);
|
||||||
|
assert.ok(r.issues.includes('wrong_arabic_variant'));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Edge cases ----------
|
||||||
|
|
||||||
|
describe('evaluateChunk — edge cases', () => {
|
||||||
|
it('empty translation passes (skipped)', () => {
|
||||||
|
const r = evaluateChunk('Hello', '', 'fr');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
assert.ok(r.issues.includes('empty_translation'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('whitespace-only translation passes', () => {
|
||||||
|
const r = evaluateChunk('Hello', ' \n ', 'fr');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('null translation passes', () => {
|
||||||
|
const r = evaluateChunk('Hello', null, 'fr');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('undefined translation passes', () => {
|
||||||
|
const r = evaluateChunk('Hello', undefined, 'fr');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('numbers only pass for fr target', () => {
|
||||||
|
const r = evaluateChunk('Price: 100', '100', 'fr');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('unknown target lang falls back to latin', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Bonjour', 'xx');
|
||||||
|
assert.equal(r.passed, true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Length ----------
|
||||||
|
|
||||||
|
describe('evaluateChunk — length', () => {
|
||||||
|
it('huge translation flagged via repetition (source too short)', () => {
|
||||||
|
const r = evaluateChunk('Short', 'x'.repeat(1000), 'fr');
|
||||||
|
assert.ok(r.issues.includes('repetition_hallucination'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('huge translation with long source flagged as length', () => {
|
||||||
|
const src = 'A'.repeat(200);
|
||||||
|
const r = evaluateChunk(src, 'x'.repeat(1000), 'fr');
|
||||||
|
assert.ok(r.issues.includes('length_outlier'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('tiny translation flagged', () => {
|
||||||
|
const r = evaluateChunk('A'.repeat(100), 'ok', 'fr');
|
||||||
|
assert.ok(r.issues.includes('truncation_suspect'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('numeric source skips length check', () => {
|
||||||
|
const r = evaluateChunk('Price: 100', '100', 'fr');
|
||||||
|
assert.ok(!r.issues.includes('truncation_suspect'));
|
||||||
|
assert.ok(!r.issues.includes('length_outlier'));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Pattern leak / repetition ----------
|
||||||
|
|
||||||
|
describe('evaluateChunk — pattern leak', () => {
|
||||||
|
it('english prompt leak', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Translation: Bonjour le monde', 'fr');
|
||||||
|
assert.ok(r.issues.includes('prompt_leak'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('french prompt leak', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Voici la traduction : Bonjour', 'fr');
|
||||||
|
assert.ok(r.issues.includes('prompt_leak'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('chinese prompt leak', () => {
|
||||||
|
const r = evaluateChunk('Hello', '翻译:你好', 'zh');
|
||||||
|
assert.ok(r.issues.includes('prompt_leak'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('token repetition detected', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'the the the the the the the', 'fr');
|
||||||
|
assert.ok(r.issues.includes('repetition_hallucination'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('char repetition detected', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'x'.repeat(30), 'fr');
|
||||||
|
assert.ok(r.issues.includes('repetition_hallucination'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('normal text has no leak', () => {
|
||||||
|
const r = evaluateChunk('Hello', 'Bonjour le monde', 'fr');
|
||||||
|
assert.ok(!r.issues.includes('prompt_leak'));
|
||||||
|
assert.ok(!r.issues.includes('repetition_hallucination'));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Document-level ----------
|
||||||
|
|
||||||
|
describe('evaluateDocument', () => {
|
||||||
|
it('all good', () => {
|
||||||
|
const result = evaluateDocument(
|
||||||
|
['Hello', 'World', 'Good morning'],
|
||||||
|
['Bonjour', 'Monde', 'Bonjour'],
|
||||||
|
'fr',
|
||||||
|
);
|
||||||
|
assert.equal(result.passed, true);
|
||||||
|
assert.equal(result.chunkCount, 3);
|
||||||
|
assert.equal(result.failedChunkCount, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('some failures', () => {
|
||||||
|
const result = evaluateDocument(
|
||||||
|
['Hello', 'World', 'Good morning'],
|
||||||
|
['Bonjour', 'مرحبا', 'Bonjour'],
|
||||||
|
'fr',
|
||||||
|
);
|
||||||
|
assert.equal(result.passed, false);
|
||||||
|
assert.equal(result.failedChunkCount, 1);
|
||||||
|
assert.ok('wrong_script' in result.issues);
|
||||||
|
assert.equal(result.samples.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('empty lists', () => {
|
||||||
|
const result = evaluateDocument([], [], 'fr');
|
||||||
|
assert.equal(result.passed, true);
|
||||||
|
assert.equal(result.chunkCount, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('sample size caps samples', () => {
|
||||||
|
const result = evaluateDocument(
|
||||||
|
Array(10).fill('hi'),
|
||||||
|
Array(10).fill('مرحبا'),
|
||||||
|
'fr',
|
||||||
|
3,
|
||||||
|
);
|
||||||
|
assert.equal(result.failedChunkCount, 10);
|
||||||
|
assert.equal(result.samples.length, 3);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- detectArabicVariant direct ----------
|
||||||
|
|
||||||
|
describe('detectArabicVariant', () => {
|
||||||
|
it('persian for persian passes', () => {
|
||||||
|
const r = detectArabicVariant('سلام چطوری', 'fa');
|
||||||
|
assert.equal(r.verdict, 'pass');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('persian for arabic fails', () => {
|
||||||
|
const r = detectArabicVariant('سلام چطوری', 'ar');
|
||||||
|
assert.equal(r.verdict, 'fail');
|
||||||
|
assert.ok(r.detectedVariants.includes('fa'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('urdu for persian fails', () => {
|
||||||
|
const r = detectArabicVariant('السلام ٹڈ', 'fa');
|
||||||
|
assert.equal(r.verdict, 'fail');
|
||||||
|
assert.ok(r.detectedVariants.includes('ur'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('non-arabic text skipped', () => {
|
||||||
|
const r = detectArabicVariant('Bonjour le monde', 'fa');
|
||||||
|
assert.equal(r.verdict, 'skip');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- lengthCheck direct ----------
|
||||||
|
|
||||||
|
describe('lengthCheck', () => {
|
||||||
|
it('normal length returns ratio', () => {
|
||||||
|
const r = lengthCheck(
|
||||||
|
'Hello world this is a longer source string',
|
||||||
|
'Bonjour le monde ceci est une traduction francaise',
|
||||||
|
);
|
||||||
|
assert.equal(r.issue, null);
|
||||||
|
assert.ok(r.ratio !== null);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('huge translation', () => {
|
||||||
|
const r = lengthCheck('A'.repeat(200), 'x'.repeat(1000));
|
||||||
|
assert.equal(r.issue, 'length_outlier');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('tiny translation', () => {
|
||||||
|
const r = lengthCheck('A'.repeat(100), 'ok');
|
||||||
|
assert.equal(r.issue, 'truncation_suspect');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('empty translation flagged', () => {
|
||||||
|
const r = lengthCheck('Hello world this is a test', '');
|
||||||
|
assert.equal(r.issue, 'truncation_suspect');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- patternCheck direct ----------
|
||||||
|
|
||||||
|
describe('patternCheck', () => {
|
||||||
|
it('no leak on normal text', () => {
|
||||||
|
const r = patternCheck('Bonjour le monde');
|
||||||
|
assert.equal(r.issue, null);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('english leak', () => {
|
||||||
|
const r = patternCheck('Translation: Bonjour');
|
||||||
|
assert.equal(r.issue, 'prompt_leak');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('french leak', () => {
|
||||||
|
const r = patternCheck('Voici la traduction : Bonjour');
|
||||||
|
assert.equal(r.issue, 'prompt_leak');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('chinese leak', () => {
|
||||||
|
const r = patternCheck('翻译:你好');
|
||||||
|
assert.equal(r.issue, 'prompt_leak');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('token repetition', () => {
|
||||||
|
const r = patternCheck('the the the the the the the');
|
||||||
|
assert.equal(r.issue, 'repetition_hallucination');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('char repetition', () => {
|
||||||
|
const r = patternCheck('x'.repeat(30));
|
||||||
|
assert.equal(r.issue, 'repetition_hallucination');
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user