Many teams struggle to turn heterogeneous sources – HTML pages, PDFs, scanned images, or plain text – into uniform image tiles that can be fed to vision‑language models. The main pain points are: inconsistent page sizes, lots of blank or decorative regions, near‑duplicate headers or footers that waste storage and compute, and the need for a reliable fallback when headless browsers fail. A practical workflow solves these issues in three steps.
First, normalize the input width to a fixed tile width (e.g., 1024 px) while preserving aspect ratio, then slide a vertical window with overlap (tile height minus overlap) to guarantee that no sentence or table row is split. Overlap keeps context intact and dramatically improves recall compared with non‑overlapping slicing.
Second, before any tile is sent to the GPU, run a cheap informativeness check: compute the standard deviation of pixel intensities and discard tiles whose variance falls below a threshold. This removes blank, solid‑colour, or mostly whitespace sections, cutting downstream processing by up to 40 %.
Third, apply a lightweight duplicate filter using an average hash and Hamming distance. Only tiles whose hash differs enough from previously seen ones are kept, eliminating repeated banners, navigation bars, or cookie notices.
If a headless browser (Playwright) cannot render a URL – due to network restrictions, missing dependencies, or timeouts – the pipeline automatically falls back to a text‑only renderer: strip HTML, render the remaining text onto a white canvas with a monospaced font, then tile that image using the same overlap logic. This guarantees that every source yields at least some tiles.
Finally, the same slicing routine works for PDF pages rendered at a chosen DPI, ensuring a single code path handles HTML, PDF, and PDF pages are processed DPI and only tiles are produced and is then of non‑blank, near‑unique tiles ready for embedding and retrieval, with clear logging at each stage for debugging.
#AI #Product #MachineLearning #DataEngineering #Automation #RAG