perf+security: fix build, secure downloads, dedupe translations, refactor i18n
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m49s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m49s
Frontend:
- Fix Framer Motion / motion-dom build error by pinning framer-motion to
11.18.2 (compatible with React 19 and Next.js 16).
- Add cross-env and build:local script to bypass standalone symlink errors
on Windows without Developer Mode.
- Allow NEXT_OUTPUT=default to disable standalone output for local builds.
- Refactor i18n: split 14,177-line src/lib/i18n.tsx into per-locale,
per-namespace JSON files under src/lib/i18n/messages/.
- Load English synchronously; other locales loaded on demand via dynamic
imports (reduces initial bundle, improves maintainability).
- Remove unused next-intl message files src/messages/en.json and fr.json.
Backend:
- Remove insecure legacy /api/v1/download/{filename} and /api/v1/cleanup/{filename}
endpoints. The job-based /api/v1/download/{job_id} already enforces ownership.
- Deduplicate texts in TranslationService.translate_batch before sending them
to the provider, reducing API calls for repeated strings.
- Pin httpx to <0.28 to fix TestClient incompatibility with starlette 0.35.1.
- Add pytest-cov and ruff dev dependencies/config.
DevOps:
- Remove hardcoded Grafana password from docker-compose.yml and
docker-compose.monitoring.yml; use GRAFANA_PASSWORD env var.
- Change default TRANSLATION_SERVICE from ollama to google in
docker-compose.yml (Ollama is an optional profile).
- Add GRAFANA_PASSWORD to .env.example.
- Add .coverage and frontend/pnpm-workspace.yaml to .gitignore.
Tests:
- Update API versioning tests for removed legacy endpoints.
- Add tests/test_translation_service.py for deduplication behavior.
Verified:
- pnpm run build:local passes.
- uv run pytest tests/test_providers/* tests/test_translation_service.py
tests/test_story_3_5_api_versioning.py tests/test_download_endpoint.py
tests/test_translators/test_excel_translator.py: provider/translator tests
pass; one pre-existing French error-message test still fails (message is
returned in English, unrelated to this change).
This commit is contained in:
@@ -1207,28 +1207,55 @@ class TranslationService:
|
||||
"""
|
||||
Translate multiple text strings efficiently using batch processing.
|
||||
|
||||
Deduplicates non-empty texts before sending them to the provider so that
|
||||
repeated strings (common in documents with headers, formulas, tables)
|
||||
are translated only once. This reduces API calls, cost and latency.
|
||||
|
||||
Args:
|
||||
texts: List of texts to translate
|
||||
target_language: Target language code
|
||||
source_language: Source language code (default: 'auto')
|
||||
|
||||
Returns:
|
||||
List of translated texts
|
||||
List of translated texts in the same order as input
|
||||
"""
|
||||
if not texts:
|
||||
return []
|
||||
|
||||
# Use provider's batch method if available
|
||||
if hasattr(self.provider, "translate_batch"):
|
||||
return self.provider.translate_batch(
|
||||
texts, target_language, source_language
|
||||
)
|
||||
# Preserve original positions and filter out empty/whitespace-only texts
|
||||
results: list[str] = [""] * len(texts)
|
||||
unique_texts: list[str] = []
|
||||
text_to_positions: dict[str, list[int]] = {}
|
||||
|
||||
# Fallback to individual translations
|
||||
return [
|
||||
self.translate_text(text, target_language, source_language)
|
||||
for text in texts
|
||||
]
|
||||
for idx, text in enumerate(texts):
|
||||
if not text or not text.strip():
|
||||
results[idx] = text if text else ""
|
||||
continue
|
||||
if text not in text_to_positions:
|
||||
text_to_positions[text] = []
|
||||
unique_texts.append(text)
|
||||
text_to_positions[text].append(idx)
|
||||
|
||||
if not unique_texts:
|
||||
return results
|
||||
|
||||
# Translate unique texts only
|
||||
if hasattr(self.provider, "translate_batch"):
|
||||
translated_unique = self.provider.translate_batch(
|
||||
unique_texts, target_language, source_language
|
||||
)
|
||||
else:
|
||||
translated_unique = [
|
||||
self.translate_text(text, target_language, source_language)
|
||||
for text in unique_texts
|
||||
]
|
||||
|
||||
# Map translated texts back to their original positions
|
||||
for original, translated in zip(unique_texts, translated_unique):
|
||||
for idx in text_to_positions[original]:
|
||||
results[idx] = translated if translated else original
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# Global translation service instance
|
||||
|
||||
Reference in New Issue
Block a user