Multimodal RAG: Query Images and Tables Alongside Text

What Multimodal RAG Opens Up

Standard RAG operates on text. You embed sentences, retrieve paragraphs, and generate answers from prose. But the real world is not all text. Your product manuals include architecture diagrams. Your financial reports are full of charts. Your research papers contain tables and graphs that hold data not repeated anywhere in the prose. If your RAG system cannot read images and tables, it is missing a significant portion of your knowledge base.

Multimodal RAG solves this by embedding and retrieving non-text content alongside text. A user can ask "show me the architecture diagram for the authentication service" or "what does the sales chart for Q2 show?" and receive an accurate, grounded answer based on the actual visual content in your documents, not a guess or a fallback to surrounding text.

How Multimodal Embeddings Work

Text embeddings convert a sentence into a vector. Multimodal embeddings do the same for images and text together, projecting both into a shared vector space where semantically related text and images land near each other. A model trained on image-caption pairs learns that a photo of a system diagram and the sentence "microservice architecture with an API gateway" should have similar vectors.

For document retrieval, the key insight is that you want to match a text query against both text chunks and images. This means your image embeddings need to be in the same vector space as your text embeddings, so a query like "quarterly revenue chart" retrieves the right bar chart even if the chart contains no embedded text. Models like CLIP and its descendants make this cross-modal retrieval possible.

ColPali: Document Retrieval Without OCR

ColPali is a document retrieval model that treats every page of a document as an image and embeds it directly using a vision-language model. Instead of running OCR to extract text and then embedding that text, ColPali embeds the raw page image. This means it can retrieve pages based on visual layout, charts, diagrams, and tables that standard OCR-based pipelines would mangle or miss entirely.

The trade-off is storage: you store page-level image patches rather than text tokens. For most document corpora, this is a reasonable trade-off because retrieval quality on visually complex documents is dramatically better than OCR-then-embed approaches. Pages with complex multi-column layouts, mixed content, and annotated diagrams benefit the most.

Building a Multimodal Index with ColPali

The code below shows how to index a directory of PDF documents using ColPali and query the index with a plain text question. The byaldi library provides a clean interface to the ColPali model family. Each page is stored as an image embedding that your text query retrieves directly.

pip install byaldi transformers pillow

from byaldi import RAGMultiModalModel

# Load the ColPali model
model = RAGMultiModalModel.from_pretrained("vidore/colpali-v1.2")

# Index a directory of PDFs (each page is stored as an image embedding)
model.index(
    input_path="./documents",
    index_name="product_docs",
    store_collection_with_index=True,
    overwrite=True,
)

# Query with a plain text question
query = "Show me the system architecture diagram"
results = model.search(query, k=3)

for i, result in enumerate(results):
    print("Result {}: doc={}, page={}, score={:.3f}".format(
        i + 1,
        result.doc_id,
        result.page_num,
        result.score,
    ))

# Pass retrieved page images to a vision LLM to generate the final answer
# result.base64 contains the page image ready for an image-capable model

Once you have retrieved the relevant pages, pass them as images to a vision-capable language model to generate the final answer. The model receives the actual rendered page images, not extracted text, so it can reason over charts, diagrams, and tables directly. This two-stage approach, ColPali for retrieval and a vision LLM for generation, is currently the most robust pipeline for visually complex document corpora.