What I learned building production RAG systems

I spent a good part of the last two years building the ingestion and retrieval behind Predli Studio, a multi-tenant enterprise RAG product. The lessons that held up in production weren't about model choice. They were about everything that happens to a document before a model ever sees it.

Chunking decided more than the model did

We ran the obvious experiment early: swap the generation model, measure answer quality. The lift was marginal. Then we rewrote chunking to respect document structure, section headers, paragraph boundaries, table boundaries, and quality moved in a way no model swap had. Same embeddings, same model, better units of retrieval.

On the Enterprise RAG platform we measured it directly: file-type-aware chunking with TOC-based hierarchical context landed about 40% better retrieval relevance than a uniform fixed-window baseline. The reason is mundane. A fixed 512-token window splits a key paragraph in half, the retriever returns two fragments that each make partial sense, and the model fills the gap with something plausible and wrong.

Different file types are different problems

The first mistake was treating every document the same: extract text, chunk, embed. It works for prose and falls apart on everything else.

PDFs carry layout that naive extraction flattens, so tables arrive as word salad. Slide decks have a hierarchy that matters, where a title on slide 12 governs the bullets beneath it and not the footer. Spreadsheets are tables, and linearizing them into paragraphs destroys the structure that made them answerable in the first place. We ended up with file-type-specific parsers feeding chunking strategies tuned per format: layout-aware extraction for PDFs, slide-level chunking with title metadata for decks, structured representations with preserved headers for spreadsheets. Each path was its own small project, and each one paid for itself.

Retrieval needs metadata, not just vectors

Pure semantic search is a starting point. In a real product you almost always filter or boost before similarity even enters the picture.

An analyst asking for the latest revenue guidance does not want semantically similar text from a 2019 earnings call. They want the most recent disclosure, from a specific company. Date, source, document type, and tenant aren't optional metadata, they're the constraints that decide whether a retrieval is useful at all. So we ran semantic intent against the vector store, structured constraints as filters, and merged the two. On Studio that also meant persona-aware retrieval: the same corpus surfaces differently depending on who is asking.

Idempotency at the boundary

The least visible decision aged the best. Studio runs ingestion on Dagster with asset-based orchestration instead of a task scheduler, and we hash content at the raw-document boundary. A connector that retries, or re-sends the same file, cannot write a duplicate into the vector store. It reads like plumbing. It is the difference between a knowledge base people trust and one that quietly fills with near-duplicates nobody can account for.

Where the budget should go

If you're spending most of your RAG effort on model selection, you're optimizing the cheap part. Put it into parsing, chunking, metadata, and the ingestion guarantees that keep the index clean. The model is close to the least interesting component in a production RAG system, and it tends to stay that way.