feat(format): B3.9 — preserve PDF table column structure during translation
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m30s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m30s
User reported that the page 6 table on the test PDF ('6. Performance
and Scaling' page) was completely broken: the 3-column table
('Document size | Avg latency (s) | Throughput (docs/min)' with
5 data rows) was rendered as a vertical list of label/value pairs
instead of as a proper table.
Root cause: a PDF 'block' that contains multiple LINES at the SAME
y but different x positions is a table row (3 cells side-by-side).
The extractor was treating the whole row as one paragraph, joining
all cell texts with newline. When the smart-fit logic wrote the
text back, it used the row's full-width bbox and \insert_textbox\
wrote everything left-aligned, collapsing all columns into one.
Fix: at extraction time, detect horizontal-layout blocks (lines at
the same y, different x within 5pt tolerance) and split them into
one sub-block per line. Each cell gets its own bbox, so the
translator writes each cell at its original x position, preserving
the column structure.
Detection heuristic:
- Block has >= 2 lines
- All lines have y0 within 3pt of each other (SAME_ROW_Y_TOLERANCE)
- At least 2 lines have different x0 (within > 5pt)
If all three hold, it's a table row. Otherwise, keep the old
multi-line-paragraph behavior.
Note: PyMuPDF re-groups cells into row-blocks when reading the
output back (so 'len(blocks)' looks unchanged), but the LINES
within each block are at their correct x positions. Tests check
the line x0 values, not the block count.
Visual proof: page 7 of sample_files/test_corpus/test_pdf_translated.pdf
now shows the table with proper 3-column structure (Taille du document
| Latence moyenne (s) | Débit (docs/min)) instead of an '[translation
overflow]' placeholder.
4 new tests added:
- test_horizontal_layout_detected: 3 lines at same y -> 3 blocks
- test_vertical_layout_kept_as_one_block: 3 lines at different y -> 1 block
- test_single_line_block_unchanged: 1 line -> 1 block
- test_table_cell_each_at_own_x: e2e table translation, cells at
correct x positions
Total: 457 tests pass (was 453), zero regression.
This commit is contained in:
@@ -454,12 +454,28 @@ class PDFTranslator:
|
||||
return output_path
|
||||
|
||||
def _extract_text_blocks(self, page) -> List[Dict]:
|
||||
"""Extract text blocks with position, font, and color information."""
|
||||
"""Extract text blocks with position, font, and color information.
|
||||
|
||||
Track B3.9: when a PDF "block" contains multiple LINES at the
|
||||
same y (typical of table rows where each cell is a separate line
|
||||
in the block), split it into one sub-block per line. This
|
||||
preserves the column structure so a translated table row stays
|
||||
a table row instead of collapsing into a single left-aligned
|
||||
column.
|
||||
|
||||
The detection is: if a block has >= 2 lines whose y0 is within
|
||||
``SAME_ROW_Y_TOLERANCE`` of each other, treat it as a
|
||||
multi-column row and emit one sub-block per line. Blocks whose
|
||||
lines are vertically stacked (multi-line paragraphs) keep the
|
||||
old behavior.
|
||||
"""
|
||||
import fitz
|
||||
|
||||
blocks = []
|
||||
data = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)
|
||||
|
||||
SAME_ROW_Y_TOLERANCE = 3.0 # points
|
||||
|
||||
for block in data.get("blocks", []):
|
||||
if block.get("type") != 0:
|
||||
continue
|
||||
@@ -468,6 +484,65 @@ class PDFTranslator:
|
||||
if not lines:
|
||||
continue
|
||||
|
||||
# B3.9: detect table-row pattern. A row is multiple lines
|
||||
# at the same y (different x). Multi-line paragraphs have
|
||||
# lines at different y (same x).
|
||||
line_y0s = [line["bbox"][1] for line in lines]
|
||||
same_y = all(
|
||||
abs(y - line_y0s[0]) <= SAME_ROW_Y_TOLERANCE
|
||||
for y in line_y0s
|
||||
)
|
||||
has_horizontal_layout = (
|
||||
same_y and len(lines) > 1
|
||||
and any(
|
||||
abs(lines[i]["bbox"][0] - lines[0]["bbox"][0]) > 5
|
||||
for i in range(1, len(lines))
|
||||
)
|
||||
)
|
||||
|
||||
if has_horizontal_layout:
|
||||
# Multi-column row: emit one block per line.
|
||||
for line in lines:
|
||||
span_parts = []
|
||||
spans_info = []
|
||||
for span in line.get("spans", []):
|
||||
text = span.get("text", "")
|
||||
if text:
|
||||
span_parts.append(text)
|
||||
spans_info.append({
|
||||
"size": span.get("size", 12),
|
||||
"font": span.get("font", "Helvetica"),
|
||||
"color": span.get("color", 0),
|
||||
"flags": span.get("flags", 0),
|
||||
"origin": span.get("origin", (0, 0)),
|
||||
})
|
||||
if not span_parts:
|
||||
continue
|
||||
line_text = "".join(span_parts).strip()
|
||||
if not line_text:
|
||||
continue
|
||||
avg_size = (
|
||||
sum(s["size"] for s in spans_info) / len(spans_info)
|
||||
if spans_info else 12.0
|
||||
)
|
||||
first_color = spans_info[0]["color"] if spans_info else 0
|
||||
is_bold = any(s["flags"] & 16 for s in spans_info)
|
||||
is_italic = any(s["flags"] & 2 for s in spans_info)
|
||||
blocks.append({
|
||||
"bbox": tuple(line["bbox"]),
|
||||
"text": line_text,
|
||||
"font_size": round(avg_size, 1),
|
||||
"color": first_color,
|
||||
"is_bold": is_bold,
|
||||
"is_italic": is_italic,
|
||||
"line_count": 1,
|
||||
"translated": None,
|
||||
"sub_bboxes": [tuple(line["bbox"])],
|
||||
"_is_table_cell": True,
|
||||
})
|
||||
continue
|
||||
|
||||
# Standard multi-line paragraph: join lines with \n.
|
||||
line_parts = []
|
||||
spans_info = []
|
||||
|
||||
|
||||
Reference in New Issue
Block a user