User reported that the page 6 table on the test PDF ('6. Performance
and Scaling' page) was completely broken: the 3-column table
('Document size | Avg latency (s) | Throughput (docs/min)' with
5 data rows) was rendered as a vertical list of label/value pairs
instead of as a proper table.
Root cause: a PDF 'block' that contains multiple LINES at the SAME
y but different x positions is a table row (3 cells side-by-side).
The extractor was treating the whole row as one paragraph, joining
all cell texts with newline. When the smart-fit logic wrote the
text back, it used the row's full-width bbox and \insert_textbox\
wrote everything left-aligned, collapsing all columns into one.
Fix: at extraction time, detect horizontal-layout blocks (lines at
the same y, different x within 5pt tolerance) and split them into
one sub-block per line. Each cell gets its own bbox, so the
translator writes each cell at its original x position, preserving
the column structure.
Detection heuristic:
- Block has >= 2 lines
- All lines have y0 within 3pt of each other (SAME_ROW_Y_TOLERANCE)
- At least 2 lines have different x0 (within > 5pt)
If all three hold, it's a table row. Otherwise, keep the old
multi-line-paragraph behavior.
Note: PyMuPDF re-groups cells into row-blocks when reading the
output back (so 'len(blocks)' looks unchanged), but the LINES
within each block are at their correct x positions. Tests check
the line x0 values, not the block count.
Visual proof: page 7 of sample_files/test_corpus/test_pdf_translated.pdf
now shows the table with proper 3-column structure (Taille du document
| Latence moyenne (s) | Débit (docs/min)) instead of an '[translation
overflow]' placeholder.
4 new tests added:
- test_horizontal_layout_detected: 3 lines at same y -> 3 blocks
- test_vertical_layout_kept_as_one_block: 3 lines at different y -> 1 block
- test_single_line_block_unchanged: 1 line -> 1 block
- test_table_cell_each_at_own_x: e2e table translation, cells at
correct x positions
Total: 457 tests pass (was 453), zero regression.
User asked whether B3.6+B3.7 are generic for ALL PDFs or just for the
test_pdf.pdf. Audit found 2 genericity bugs:
1. _populate_next_block_y was sorting blocks globally by y0. In a
multi-column PDF (journals, brochures, newspapers), the 'next
block' of a left-column block would point to the right-column
block at the same y, which is wrong. Fix: group blocks into
columns by x0 proximity (15pt tolerance), then sort each column
by y0. Each block's next_block_y is the y0 of its column-mate
directly below it, not just the next block in y-order globally.
2. max_expand_y could go negative if next_block_y was above the
current block (rare edge case in extracted blocks with weird
bbox ordering). A negative max_expand_y would create an invalid
fitz.Rect with y1 < y0, causing silent failures. Fix: clamp
max_expand_y to >= 0.
7 new tests added:
- test_two_columns_get_separate_next_block_y: 2-col layout,
left and right columns get independent next_block_y mappings
- test_centered_full_width_header_gets_own_column: full-width
header between 2 columns is its own column
- test_three_columns: 3-column newspaper layout
- test_single_block_page, test_empty_block_list: edge cases
- test_max_expand_y_clamped_to_zero: negative-expansion safety
- test_two_column_pdf_translation_end_to_end: e2e test on a
2-col journal PDF, 4 input blocks -> 4 output blocks preserved
at correct positions, no cross-column overlap
Visual verification:
scripts/verify_b3_8_multicolumn.py renders a 2-col journal PDF
before and after translation, confirms 4 left + 4 right blocks
preserved at exact positions.
Total tests: 453 (was 446), zero regression.
B3.6 — fix two visual bugs reported on the user's prod PDF:
1. Title 'Spécification technique : Office Translator v3.0' overflowed
its 2-line bbox and overlapped the 'Version du document...' block.
Root cause: MAX_VERTICAL_EXPANSION was 1.5x the original height,
way too small for a long French title. Bumped to 6.0x.
2. 'Avis important' blue background had white rectangular patches.
Root cause: redaction always used fill=(1,1,1) (opaque white),
which erased the colored drawing underneath the text.
Fix: detect when a block's bbox intersects a page drawing,
and use fill=None (transparent) for the redaction in that case.
The original drawing survives intact.
B3.7 — eliminate remaining block-vs-next-block overlap:
Computes each block's 'next_block_y' (the y0 of the nearest block
below it on the same page) and uses it as the ceiling for vertical
expansion. Previously the smart-fit logic used the page bottom as
the ceiling, which let long translated blocks flow into their
neighbour (e.g. 'Pour la dernière version...' overlapping
'8. Résolution des problèmes' in the TOC).
Also includes:
- 9 new tests (6 B3.6 + 3 B3.7) — total 446 tests pass, zero regression
- scripts/verify_b3_6_fix.py — visual+structural verification
- Updated sample_files/test_corpus/test_pdf_translated.pdf with the
clean B3.6+B3.7 output
ROOT CAUSE FIX: PyMuPDF silently raised AttributeError when fontname=None
was passed to insert_textbox. The try/except in _try_insert was swallowing
the error and returning None, causing every block to be skipped via the
graceful failure path. Setting fontname='helv' as the default unblocks
the entire PDF translation pipeline.
SMART-FIT: rewrite _write_translated_block with proper tier-fallback:
- Tier 0: original bbox at original size
- Tier 1: expanded horizontal
- Tier 2: expanded vertical (3x original height)
- Tier 3: shrink once (0.93x)
- Tier 4: shrink twice (0.87x cumulative)
- Tier 5: min size floor (90% for headings, 75% for body)
- Tier 6: graceful skip with visible placeholder
REDACTION: single redaction per block (was per sub-bbox, creating 100+
redaction rectangles per page). Now only 1 redaction per text block.
FEATURE FLAG: PDF_SMART_FIT_ENABLED (default true, observation-first).
METRICS: text_overflow -> format_elements_lost_total.
RESULT ON REAL PDF:
Before: fonts shrunk 22pt->5.6pt, hierarchy destroyed
After: fonts EXACT match: [8, 11, 12, 14, 16, 22] preserved
Two critical fixes:
1. Provider "google" (default classic mode) now checks for a Google Cloud
API key (GOOGLE_CLOUD_API_KEY in env or admin settings). If present,
uses GoogleCloudTranslationProvider (official API). Previously it
always fell through to deep_translator (free scraper) which gets
blocked in production, silently returning untranslated text.
2. Added translation verification: each translator now tracks how many
texts were attempted vs actually changed. If 0 texts were translated,
the job is marked as FAILED with a clear error message instead of
returning the original file as "completed".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Restructured docker-compose for Nginx Proxy Manager (no custom nginx)
- Added domain wordly.art configuration
- Added Prometheus + Grafana monitoring stack with pre-configured dashboards
- Added PostgreSQL backup script to NAS (daily/weekly/monthly rotation)
- Added alert rules for backend, system, and Docker metrics
- Updated deployment guide for NPM + IONOS DNS homelab setup
- Added marketing plan document
- PDF translator and watermark support
- Enhanced middleware, routes, and translator modules
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
