Your RAG Demo Works on Clean PDFs. Mine Had to Read a Decade of SharePoint.

Over the past few months, I had the privilege of working with the AI & Digitalisierung team at Maisel, on some genuinely exciting projects. The one that posed the greatest challenge to me was an internal chatbot: a RAG system operating over that entire document estate. Here's what building at that scale actually taught me.

1. Trending architectures and tutorials don't automatically fit your use case. There's no shortage of RAG techniques that genuinely help someone, somewhere: HyDE, query decomposition, agentic chunking, hybrid retrieval. Whether any of them help you isn't something a tutorial or a benchmark leaderboard can answer, because those numbers are measured on someone else's corpus, not yours. It only shows up once you run it against a real set of questions your users actually ask and compare what comes back to what came back before. An AI assistant can help you implement a technique fast. It can't tell you, for your specific documents and your specific users, whether it actually moved the needle. That judgment still has to come from you, checking real output against real queries. My advice: pick one solid structure, build an evaluation set from real questions, and test against it before you reach for the next technique.

2. Understanding your data and where it's stored. A RAG system's retrieval is only as good as the data underneath it, and in an enterprise environment that data usually lives scattered across dozens of libraries, shared drives, and storage tiers, not one clean corpus. Before you write a line of retrieval code, you need an actual inventory: which sites and subsites exist, which storage layer each document set sits on, and which permission boundaries apply to each one. That inventory is what lets you build a data pipeline where every document is indexed the way you intend, not however the crawler happened to stumble across it. It's also what makes role-based access control possible at the retrieval layer: if you don't know where a document lives and who's allowed to see it, you can't reliably keep the system from surfacing it to someone who shouldn't have access. And it's the only way to make sure your crawler is actually finding every subsite, not just the ones it was pointed at directly.

3. Evaluate every stage, and test with real user questions, not synthetic ones. A correct answer depends on several stages going right in sequence: query augmentation, embedding, retrieval, re-ranking, and generation. If you only measure the final answer, you can't tell which stage actually failed. Instrumenting each stage separately gives you a real benchmark to improve against, and it's the only way to see where your latency and your LLM API cost are actually going, which is often not where you'd assume. In our case, generation was a rounding error on the timeline; re-ranking was the actual bottleneck, and I only found that by measuring each stage on its own instead of the pipeline end to end.

4. Some of the questions coming through weren't semantic at all: an identifier, a role, a value sitting in a structured field with exactly one correct answer. Sending those through the full pipeline (embedding, ANN search, cross-encoder re-ranking) anyway didn't make them more findable, it just added five failure-prone stages to a problem that needed one exact-match lookup against a labeled field. I added a lightweight query classifier ahead of retrieval: identifier-shaped and field-value questions route to a direct, deterministic match against the structured source of truth; everything else goes through embeddings as before. That cut the failure surface for single-answer queries down to the classifier and the lookup itself, instead of the full seven-stage path, and it kept the vector index doing what dense retrieval is actually suited for: semantically fuzzy, open-ended questions where there isn't one correct string to match.

The takeaway I keep coming back to: in an enterprise environment, most RAG failures are document estate failures wearing an AI costume. The retrieval algorithm, the re-ranker, the prompt: all of that assumes you already have clean, complete, discoverable content to work with. Getting to that starting point was most of the project.

With this chapter coming to an end, I'm excited to be joining the Digital Solutions & AI team at CANCOM next.

Many thanks to Johannes Köber and the rest of the team at Maisel for an amazing experience.