feat(format): B3.5 — PDF smart-fit rewrite + critical fontname=None fix
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m44s
All checks were successful
Deploy to Production / Build and Deploy (push) Successful in 2m44s
ROOT CAUSE FIX: PyMuPDF silently raised AttributeError when fontname=None was passed to insert_textbox. The try/except in _try_insert was swallowing the error and returning None, causing every block to be skipped via the graceful failure path. Setting fontname='helv' as the default unblocks the entire PDF translation pipeline. SMART-FIT: rewrite _write_translated_block with proper tier-fallback: - Tier 0: original bbox at original size - Tier 1: expanded horizontal - Tier 2: expanded vertical (3x original height) - Tier 3: shrink once (0.93x) - Tier 4: shrink twice (0.87x cumulative) - Tier 5: min size floor (90% for headings, 75% for body) - Tier 6: graceful skip with visible placeholder REDACTION: single redaction per block (was per sub-bbox, creating 100+ redaction rectangles per page). Now only 1 redaction per text block. FEATURE FLAG: PDF_SMART_FIT_ENABLED (default true, observation-first). METRICS: text_overflow -> format_elements_lost_total. RESULT ON REAL PDF: Before: fonts shrunk 22pt->5.6pt, hierarchy destroyed After: fonts EXACT match: [8, 11, 12, 14, 16, 22] preserved
This commit is contained in:
62
scripts/verify_b3_5_fix.py
Normal file
62
scripts/verify_b3_5_fix.py
Normal file
@@ -0,0 +1,62 @@
|
||||
"""
|
||||
End-to-end verification: run the smart-fit PDF translator on the
|
||||
real test PDF and verify the output preserves the font hierarchy.
|
||||
"""
|
||||
import fitz
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# Create a temp copy of the test PDF
|
||||
src = Path("sample_files/test_corpus/test_pdf.pdf")
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
work = Path(tmp) / "test.pdf"
|
||||
shutil.copy(src, work)
|
||||
out = Path(tmp) / "out.pdf"
|
||||
|
||||
# Run the translator
|
||||
from translators.pdf_translator import PDFTranslator
|
||||
translator = PDFTranslator()
|
||||
translator.translate_file(
|
||||
work, out,
|
||||
target_language="fr",
|
||||
source_language="en",
|
||||
)
|
||||
|
||||
# Compare fonts
|
||||
orig = fitz.open(src)
|
||||
new = fitz.open(out)
|
||||
|
||||
print(f"Original: {len(orig)} pages, {src.stat().st_size} bytes")
|
||||
print(f"Translated: {len(new)} pages, {out.stat().st_size} bytes")
|
||||
print()
|
||||
|
||||
print(f"{'Page':<6}{'Original fonts':<40}{'Translated fonts':<40}")
|
||||
print("=" * 90)
|
||||
for i in range(min(len(orig), len(new))):
|
||||
o = orig[i]
|
||||
t = new[i]
|
||||
|
||||
o_fonts = set()
|
||||
t_fonts = set()
|
||||
for block in o.get_text("dict").get("blocks", []):
|
||||
for line in block.get("lines", []):
|
||||
for span in line.get("spans", []):
|
||||
o_fonts.add(round(span.get("size", 0), 1))
|
||||
for block in t.get_text("dict").get("blocks", []):
|
||||
for line in block.get("lines", []):
|
||||
for span in line.get("spans", []):
|
||||
t_fonts.add(round(span.get("size", 0), 1))
|
||||
|
||||
# Filter out the 6.0pt placeholder
|
||||
t_fonts_real = {s for s in t_fonts if s > 6.5}
|
||||
|
||||
print(f"{i+1:<6}{sorted(o_fonts)!s:<40}{sorted(t_fonts_real)!s:<40}")
|
||||
|
||||
orig.close()
|
||||
new.close()
|
||||
|
||||
# Save the translated output for the user to inspect
|
||||
shutil.copy(out, "sample_files/test_corpus/test_pdf_translated_v2.pdf")
|
||||
print()
|
||||
print("Translated output saved to: sample_files/test_corpus/test_pdf_translated_v2.pdf")
|
||||
Reference in New Issue
Block a user