I Built a RAG System That Skips the LLM Whenever It Can. Here's Why That Was the Right Call

If you have built a retrieval-augmented generation (RAG) system before, you know the standard recipe: embed the query, search a vector store, stuff the results into a prompt, call an LLM, and wait. It works. It is also, for a huge fraction of real queries, massive overkill.

That realization is what shaped SpeaklarRAG, a production-oriented, low-latency Bangla RAG system I built for conversational catalog search and question answering. The question that drove almost every design decision was simple. If a user asks "মিল্কভিটা নুডুলস ৫০০ গ্রাম দাম কত?" (how much is Milk Vita noodles, 500g?), do I really need to embed that query, run a vector search, fuse results, and pay for an LLM call just to look up a number I already have in a table?

The answer, obviously, is no. Building the system around that "no" is what makes SpeaklarRAG interesting.

The problem with treating every query as an LLM problem

Most catalog and support chatbots funnel every single message through the same expensive pipeline, embed, retrieve, generate, regardless of how simple the question actually is. For a Bangla grocery catalog, the overwhelming majority of real user queries are not open-ended or ambiguous. They are things like:

• "নুডুলস বিক্রি করেন?" (Do you sell noodles?)
• "দাম কত?" (How much?)
• "মিল্কভিটা দুধ কত লিটার?" (What size is the Milk Vita milk?)

These are lookups, not reasoning tasks. Routing them through an embedder, a FAISS index, a BM25 index, rank fusion, and an LLM call is like hiring a research analyst to answer "what's 2+2." It is slow, it is expensive, and, because LLMs hallucinate, it is occasionally wrong about a fact that was sitting right there in a structured table the whole time.

So I designed SpeaklarRAG around a two-speed architecture.

Every query enters through a context resolver, which checks whether it is a short follow-up (more on that below) and rewrites it if so. From there, it hits a fork.

1. The deterministic hot path. Exact catalog lookups, brand, size and price questions, group existence checks, price-range queries, and context-resolved follow-ups get answered directly from an in-memory catalog structure. No embedding model, no vector search, no BM25, no LLM. This path typically finishes in under 10 milliseconds.
2. The retrieval and LLM fallback. Open-ended, ambiguous, or subjective queries ("ভালো নুডুলস কোনটা?", "which noodles are good?") fall through to a full hybrid RAG pipeline. FAISS vector search and BM25 lexical search run in parallel, their results are merged with Reciprocal Rank Fusion (RRF), and only then does a prompt get built and sent to an LLM.

The deterministic responder even gets a second chance after retrieval. If the fused documents contain a clean, renderable answer, the pipeline still short-circuits before paying for a generation call. LLM calls are the last resort, not the default.

Getting the ordering right: a lesson in async pipelines

One detail I want to highlight, because it is the kind of thing that is easy to get subtly wrong in early versions of a RAG pipeline, is the ordering of side effects relative to the response path.

An earlier, naive version of the pipeline looked like this:

context_resolve -> append_history -> cache_lookup -> embed -> retrieve -> LLM

The problem: writing to session history, a Redis call, was sitting directly in the critical path, before the response was even computed. Every request paid for a network round-trip that had nothing to do with answering the question.

The corrected version:

context_resolve -> cache_lookup -> embed -> retrieve -> LLM
                                     -> [fire and forget: cache.set + append_history]

Two changes matter here. First, the semantic cache lookup moved before embedding, or more precisely, the cache check now happens as early as structurally possible so that a cache hit skips retrieval and generation entirely. Second, and more importantly, session history writes and cache writes were moved to the end and dispatched as fire-and-forget async tasks (asyncio.create_task(...)) rather than being awaited. Redis is simply never allowed to sit on the critical path of a user-facing response. If Redis is slow or briefly unavailable, the user still gets their answer on time and the bookkeeping catches up a few milliseconds later.

It is a small structural change, but it is the difference between a pipeline that is usually fast and one that has a hard latency budget it actually respects.

Teaching a system to handle a question with no explicit subject

Here is a scenario that trips up a lot of naive chatbots.

Q1. আপনাদের কোম্পানি কি নুডুলস বিক্রি করে? ("Do you sell noodles?")

A1. হ্যাঁ, নুডুলস বিক্রি করা হয়। ("Yes, we sell them.")

Q2. দাম কত টাকা? ("How much does it cost?")

A2. নুডুলসের দাম ৫৬ টাকা থেকে ৪১৬ টাকা পর্যন্ত। Resolved from Q1's context. No embedding, retrieval, or LLM call.

Q2 is a three-word fragment with no explicit subject. A generic RAG pipeline would either embed "দাম কত টাকা?" in isolation, and retrieve close to nothing useful, or need an LLM to infer what "it" refers to from chat history, burning a generation call just to resolve a pronoun.

SpeaklarRAG handles this with a dedicated BanglaContextResolver that detects short follow-up turns and rewrites them using entities stored from the previous turn, paired with session/store.py, which keeps Redis-backed conversation history and a structured "last target context", meaning the last item or group the user was actually asking about. Once the follow-up is rewritten into something like "নুডুলসের দাম কত?", it is a completely ordinary query that the deterministic path can answer directly.

The entity extraction that makes this possible is deliberately unglamorous. context/ner.py is a pure regex and lookup-based Bangla recognizer, with no ML model at all. It is organized as a dictionary of categories (grains, oils, noodles and pasta, vegetables, fruits, proteins, fish, dairy and eggs, spices, sweeteners, bakery, beverages, packaged foods, and more), each mapped to a regex pattern of Bangla terms. That sounds almost too simple for an NLP pipeline until you remember the latency budget. A transformer-based NER model would blow past the single-digit-millisecond budget this component needs to hit. Regex, tuned against a real vocabulary, is not a shortcut here. It is the correct engineering choice for the constraint.

Why the semantic cache threshold is set almost paranoidly high

Semantic caching, caching LLM and retrieval responses keyed by embedding similarity rather than exact string match, is a great way to cut redundant work. SpeaklarRAG has a two-tier SemanticCache: an in-process cosine-similarity layer for speed, backed by a Redis exact-match layer for durability across restarts.

But there is a sharp edge specific to this domain. Bangla price queries for different items share nearly identical surface structure. "মিল্কভিটা দুধ দাম কত?" and "রাধুনী নুডুলস দাম কত?" are structurally almost the same sentence with the item swapped out, which means their embeddings can end up suspiciously close together. A cache tuned the way you would tune a generic FAQ cache, with a similarity threshold around 0.85 to 0.90, would happily return the price of Milk Vita milk when someone asked about Radhuni noodles.

The fix was to set CACHE_SIMILARITY_THRESHOLD deliberately high, 0.98 by default, explicitly documented in the codebase as a defense against this exact failure mode. It is a good reminder that semantic similarity thresholds are not a universal constant. They need to be calibrated against how your specific query distribution actually clusters, and in a narrow, repetitive domain, "similar-looking" is not the same as "same intent."

The retrieval stack, when it is needed

For everything that does not resolve deterministically, SpeaklarRAG falls back to a genuinely hybrid retrieval pipeline.

• retrieval/embedder.py is a singleton wrapper around intfloat/multilingual-e5-small (384 dimensions), loaded via ONNX Runtime with PyTorch as fallback, capped at 64 tokens since these queries are short. Crucially, this model is loaded once, at application startup, and injected into the pipeline. The request-handling code is explicitly forbidden from calling the loader again, because doing so would reintroduce a multi-second cold-load delay into a hot path that is supposed to run in tens of milliseconds.
• retrieval/faiss_store.py does cosine-similarity vector search with FAISS disk persistence.
• retrieval/bm25_store.py does BM25 lexical search tuned for Bangla text, also persisted to disk.
• retrieval/fusion.py implements Reciprocal Rank Fusion, which merges the FAISS and BM25 result lists into one ranked list without needing to normalize incomparable similarity scores across the two methods.

Running FAISS and BM25 in parallel via asyncio.gather, rather than sequentially, is a small thing that matters. Dense and sparse retrieval are catching different failure modes, semantic gaps versus exact keyword and brand matches, and there is no reason to pay their latencies additively when they do not depend on each other.

A resolution policy that refuses to guess

One of the more design-flavored decisions in the system is how it handles queries that are almost exact but not quite. If someone asks "নুডুলসের দাম কত?" ("What's the price of noodles?") without specifying brand or size, the catalog might have a dozen matching entries ranging from ৫৬ to ৪১৬ টাকা. A naive system might just return the top vector-search hit and confidently state its price, which would be actively misleading.

SpeaklarRAG's DeterministicResponder instead distinguishes between query specificity levels.

This is a small design choice with an outsized effect on trustworthiness. The system is honest about ambiguity instead of silently picking one entry and stating its price as if it were the only answer. Getting this distinction right required decent metadata extraction (utils/product_metadata.py handles brand, category and pack-size enrichment from semi-structured names), which the project's own documentation flags as a genuine limitation: deterministic accuracy is bounded by how cleanly those attributes can be pulled out of catalog text in the first place.

What is actually under the hood

For the engineering-minded, the stack is a fairly conventional but well-instrumented Python service.

• A FastAPI app (api/main.py) with lifespan-managed startup. It connects Redis, preloads the embedder, loads or builds the FAISS and BM25 indexes, and initializes the cache and LLM generator. This is all done before the app starts accepting traffic, so nothing expensive happens inside a request.
• api/pipeline.py. The RAGPipeline orchestrator is instrumented at every stage (context_ms, exact_lookup_ms, embed_ms, cache_ms, retrieval_ms, fusion_ms, template_ms, llm_ms, total_ms) and those numbers are returned to the client in the response payload, so latency is not just a target, it is observable per request.
• generation/generator.py. An LLMGenerator providing a unified async interface across Groq, OpenAI and Gemini, with primary and fallback routing and per-provider timeouts, so a slow or down provider degrades gracefully instead of taking the whole system with it.
• Redis. Used for session state, the exact-match cache tier, and rate limiting, with the codebase explicitly designed so that Redis being unavailable degrades to in-memory behavior rather than causing a hard failure.
• Prometheus and Grafana. Wired up via docker-compose, alongside structured JSON logging that includes request IDs, session IDs, per-stage latency breakdowns, intent classification, and whether coreference resolution fired, which makes debugging a weird response days later actually tractable.
• A multi-stage Dockerfile. It pre-bakes the ONNX-exported embedding model into the image at build time, eliminating a roughly 70-second export delay that would otherwise happen on every cold container start. It also runs as a non-root user and pins --workers 1 deliberately: scaling this service horizontally, with more containers rather than more workers per container, keeps the embedding model's memory footprint predictable.

A single POST /query endpoint carries most of the weight, but the API also exposes /health, /readiness (which specifically confirms the FAISS and BM25 indexes are loaded, not just that the process is alive), /metrics and /metrics/json, a DELETE /session/{id}, and a /ws WebSocket route for streaming. That last one is worth noting honestly in the docs as streaming the final response word by word rather than true provider-native token streaming, which is a real distinction and one I did not want to fudge in the README.

The honest list of limitations

It is tempting, writing about your own project, to only talk about what works. But part of what makes this system production-oriented rather than a demo is being upfront about where it is incomplete.

• Group-level deterministic answers are implemented for existence and price-range questions specifically, not every conceivable attribute combination a user might ask about.
• The WebSocket endpoint streams a completed response word by word, not true token-level streaming from the LLM provider.
• Deterministic accuracy is only as good as the metadata extraction from semi-structured names. Messy or inconsistent catalog data will degrade the hot path's accuracy before anything else does.
• LLM fallback quality and latency are, unavoidably, dependent on whichever provider is configured and the network conditions at request time.

The bigger takeaway

If there is one idea worth taking away from SpeaklarRAG, it is this: RAG does not have to mean "always call an LLM." For domains with a lot of structured, factual, high-frequency queries, catalogs, FAQs, internal documentation with clear ground truth, the highest-leverage move is not a better prompt or a bigger model. It is recognizing which queries do not need a model at all, building a fast, honest, deterministic path for them, and reserving retrieval-augmented generation for the genuinely open-ended cases it is actually good at.

The result, in this case, is a system where the common case, a price lookup, an existence check, a natural three-word follow-up, resolves in single-digit milliseconds with zero LLM cost, and the hard case still gets a properly retrieved, properly grounded answer when it needs one.

SpeaklarRAG is open source. The full code, architecture docs, and benchmark scripts are available on GitHub.

References

1. Roy, R. SpeaklarRAG: Constraint-Conditioned Bangla Retrieval-Augmented Generation API. GitHub repository. github.com/Rhythm05Roy/speaklar_rag
2. Wang, L., Yang, N., Huang, X., et al. Multilingual E5 Text Embeddings: A Technical Report. Model: intfloat/multilingual-e5-small. huggingface.co
3. Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009.
4. Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.
5. Johnson, J., Douze, M., & Jégou, H. FAISS: A Library for Efficient Similarity Search. Facebook AI Research. github.com/facebookresearch/faiss
6. ONNX Runtime, Microsoft. High-Performance Inference Engine for ONNX Models. onnxruntime.ai
7. FastAPI. Modern, High-Performance Web Framework for Building APIs with Python. fastapi.tiangolo.com
8. Redis. In-Memory Data Structure Store. redis.io
9. Prometheus. Monitoring System and Time Series Database. prometheus.io
10. Groq, OpenAI, and Google Gemini API documentation. LLM providers integrated as primary and fallback generation backends in this project.

---

Want to share your own experience? Every member can write here: reach out and we'll help you publish your first post.