User asked whether B3.6+B3.7 are generic for ALL PDFs or just for the
test_pdf.pdf. Audit found 2 genericity bugs:
1. _populate_next_block_y was sorting blocks globally by y0. In a
multi-column PDF (journals, brochures, newspapers), the 'next
block' of a left-column block would point to the right-column
block at the same y, which is wrong. Fix: group blocks into
columns by x0 proximity (15pt tolerance), then sort each column
by y0. Each block's next_block_y is the y0 of its column-mate
directly below it, not just the next block in y-order globally.
2. max_expand_y could go negative if next_block_y was above the
current block (rare edge case in extracted blocks with weird
bbox ordering). A negative max_expand_y would create an invalid
fitz.Rect with y1 < y0, causing silent failures. Fix: clamp
max_expand_y to >= 0.
7 new tests added:
- test_two_columns_get_separate_next_block_y: 2-col layout,
left and right columns get independent next_block_y mappings
- test_centered_full_width_header_gets_own_column: full-width
header between 2 columns is its own column
- test_three_columns: 3-column newspaper layout
- test_single_block_page, test_empty_block_list: edge cases
- test_max_expand_y_clamped_to_zero: negative-expansion safety
- test_two_column_pdf_translation_end_to_end: e2e test on a
2-col journal PDF, 4 input blocks -> 4 output blocks preserved
at correct positions, no cross-column overlap
Visual verification:
scripts/verify_b3_8_multicolumn.py renders a 2-col journal PDF
before and after translation, confirms 4 left + 4 right blocks
preserved at exact positions.
Total tests: 453 (was 446), zero regression.
B3.6 — fix two visual bugs reported on the user's prod PDF:
1. Title 'Spécification technique : Office Translator v3.0' overflowed
its 2-line bbox and overlapped the 'Version du document...' block.
Root cause: MAX_VERTICAL_EXPANSION was 1.5x the original height,
way too small for a long French title. Bumped to 6.0x.
2. 'Avis important' blue background had white rectangular patches.
Root cause: redaction always used fill=(1,1,1) (opaque white),
which erased the colored drawing underneath the text.
Fix: detect when a block's bbox intersects a page drawing,
and use fill=None (transparent) for the redaction in that case.
The original drawing survives intact.
B3.7 — eliminate remaining block-vs-next-block overlap:
Computes each block's 'next_block_y' (the y0 of the nearest block
below it on the same page) and uses it as the ceiling for vertical
expansion. Previously the smart-fit logic used the page bottom as
the ceiling, which let long translated blocks flow into their
neighbour (e.g. 'Pour la dernière version...' overlapping
'8. Résolution des problèmes' in the TOC).
Also includes:
- 9 new tests (6 B3.6 + 3 B3.7) — total 446 tests pass, zero regression
- scripts/verify_b3_6_fix.py — visual+structural verification
- Updated sample_files/test_corpus/test_pdf_translated.pdf with the
clean B3.6+B3.7 output
ROOT CAUSE FIX: PyMuPDF silently raised AttributeError when fontname=None
was passed to insert_textbox. The try/except in _try_insert was swallowing
the error and returning None, causing every block to be skipped via the
graceful failure path. Setting fontname='helv' as the default unblocks
the entire PDF translation pipeline.
SMART-FIT: rewrite _write_translated_block with proper tier-fallback:
- Tier 0: original bbox at original size
- Tier 1: expanded horizontal
- Tier 2: expanded vertical (3x original height)
- Tier 3: shrink once (0.93x)
- Tier 4: shrink twice (0.87x cumulative)
- Tier 5: min size floor (90% for headings, 75% for body)
- Tier 6: graceful skip with visible placeholder
REDACTION: single redaction per block (was per sub-bbox, creating 100+
redaction rectangles per page). Now only 1 redaction per text block.
FEATURE FLAG: PDF_SMART_FIT_ENABLED (default true, observation-first).
METRICS: text_overflow -> format_elements_lost_total.
RESULT ON REAL PDF:
Before: fonts shrunk 22pt->5.6pt, hierarchy destroyed
After: fonts EXACT match: [8, 11, 12, 14, 16, 22] preserved
Word fixes:
W1 — Fix hyperlink double-collect: a run inside <w:hyperlink> was
previously collected twice (once via paragraph.runs, once via
the manual hyperlink iter). Now uses a dedup set of element
ids to collect each run exactly once.
NB: python-docx 1.x's paragraph.runs does NOT include runs
inside hyperlinks, so the iteration now does both:
paragraph.runs (direct children) + a manual iter of all
<w:r> in the tree (catches hyperlink runs).
W2 — Fix footnotes import: used document.part.package.part_related_by
which doesn't exist in python-docx 1.x, so footnotes were never
collected. Now uses document.part.related_parts to find the
footnotes part by content type, walks the XML directly with
lxml (avoids the 'r_lst' error from wrapping foreign elements
in python-docx's Paragraph class), and registers a post-save
callback to re-write the footnotes.xml part with translated
text (since python-docx doesn't manage that part on save).
Same fix applied to endnotes.
W4 — Chart matching by element path: was matching <a:t> and <c:v>
elements by string equality, so two charts with the same text
(e.g. two 'Revenue' series) would only have the first one
translated. Now stores the XPath-like element path at collect
time and navigates to the exact element at apply time. Falls
back to string matching for legacy entries without a path.
Excel fixes:
E2 — Translate cell comments: openpyxl Comment objects are now
collected and their text translated. The Comment object is
replaced in place after translation.
E3 — Translate cell hyperlink display labels: cell.hyperlink.display
(or .target if no display) is collected and translated. The
URL itself is never sent for translation, so it remains
intact. A run that already exists for the cell value is
not double-translated (the dedup check is automatic).
E4 — Chart matching by element path: same fix as W4 but for
Excel. Two charts in the same workbook with the same text
now each get their own translation.
Tests:
Added tests/test_translators/test_b1_format_fixes.py with 11 tests
covering all the fixes. All 11 pass. Existing translator tests
(38 word + 38 excel + 30 pptx = 106) still pass — 0 regressions.
Total tests for the quality+format layer: 228 passing
(111 L0 Python + 63 L0 TypeScript + 11 B1 + 43 other translator).
All fixes are surgical: existing translation flow is preserved.
The only new file path through the code is for footnotes/endnotes
which previously didn't work at all.
Two critical fixes:
1. Provider "google" (default classic mode) now checks for a Google Cloud
API key (GOOGLE_CLOUD_API_KEY in env or admin settings). If present,
uses GoogleCloudTranslationProvider (official API). Previously it
always fell through to deep_translator (free scraper) which gets
blocked in production, silently returning untranslated text.
2. Added translation verification: each translator now tracks how many
texts were attempted vs actually changed. If 0 texts were translated,
the job is marked as FAILED with a clear error message instead of
returning the original file as "completed".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Restructured docker-compose for Nginx Proxy Manager (no custom nginx)
- Added domain wordly.art configuration
- Added Prometheus + Grafana monitoring stack with pre-configured dashboards
- Added PostgreSQL backup script to NAS (daily/weekly/monthly rotation)
- Added alert rules for backend, system, and Docker metrics
- Updated deployment guide for NPM + IONOS DNS homelab setup
- Added marketing plan document
- PDF translator and watermark support
- Enhanced middleware, routes, and translator modules
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Major changes across backend, frontend, infrastructure:
- Provider system with model selection (Google, DeepL, OpenAI, Ollama, Google Cloud)
- Admin panel: user management, pricing, settings
- Glossary system with CSV import/export
- Subscription and tier quota management
- Security hardening (rate limiting, API key auth, path traversal fixes)
- Docker compose for dev, prod, and IONOS deployment
- Alembic migrations for new tables
- Frontend: dashboard, pricing page, landing page, i18n (en/fr)
- Test suite and verification scripts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
