RAG in Production: What Engineering Teams Get Wrong (and How to Fix It)

RAG is standard. Production-quality RAG is not.

Most engineering teams can ship a RAG proof-of-concept in a day. The hard part is everything that comes after: chunking strategies that don't destroy context, retrieval quality that doesn't degrade as the corpus grows, evals that catch regressions before users do.

The four most common RAG failures we see

1. Fixed-size chunking that breaks semantic coherence

Splitting documents every 500 tokens without regard for structure means your retrieval system routinely returns fragments that make no sense in isolation. Switch to structure-aware chunking (markdown headers, sentence boundaries, semantic sections) and retrieval quality improves immediately.

2. No reranking step

Embedding similarity is a good first filter, not a final answer. Adding a cross-encoder reranker (Cohere Rerank, BGE Reranker, or a custom fine-tune) between retrieval and generation cuts hallucination rates significantly on knowledge-intensive queries.

3. No evals pipeline

If you can't measure retrieval precision and answer faithfulness, you can't improve them. Teams that build evals early — even simple ones using Claude as a judge — catch regression before it reaches users. Teams that skip evals find out about degradation through support tickets.

4. Single-vector search with no hybrid fallback

Dense vector search alone misses exact-match queries (product SKUs, named entities, specific codes). Hybrid search (BM25 + vector + rerank) covers the full query distribution your users actually produce.

The production RAG stack we recommend

• Ingestion: LlamaIndex or custom pipeline with structure-aware chunking
• Vector store: Pinecone, pgvector, or Weaviate depending on your infra
• Hybrid search: BM25 + dense vector, merged with RRF
• Reranking: Cohere Rerank or BGE-M3
• Evals: Ragas or custom Claude-as-judge pipeline
• Observability: LangSmith or Langfuse for tracing and regression detection