machine-learning¶

Vector Search at Scale: Balancing Cost, Quality, and Sanity

At scale, relevance isn't your only enemy. Cost is. Every millisecond of latency, every token passed to an LLM, and every unnecessary reranker call adds up—fast. The iron triangle of retrieval and hence, RAG is relevance, cost, and latency. You can only pick two.

Today, we'll focus on the cost and latency.

Here's a list of things that teams do that can be improved:

• Run full-precision vector search for every query
• Skip lexical signals altogether
• Avoid reranking because "it's too expensive"
• Have no system to analyze why results are bad

This post is a walkthrough of what a real retrieval stack looks like when it's designed not just for correctness, but also for operational efficiency and failure debugging.

Figure: Retrieval stack architecture balancing cost, quality, and latency. Each layer maximizes relevance per dollar and enables debugging.

The Architecture

Forget monoliths. Retrieval is a pipeline. Here's the architecture I recommend. Each layer exists for a reason: to maximize relevance per dollar and to make debugging sane.

1. Query Router

This is your traffic cop. It decides how to fan out the query: to a lexical search engine (BM25), a fast vector index, or both. You can route based on query class, business priority, or budget.

2. BM25 Search

Not dead. In fact, BM25 still shines for acronym-heavy domains, product names, and anything with proper nouns. It's cheap, precise, and the ideal complement to lossy vector embeddings. Run it in parallel with your vector retrieval.

3. Binary Quantized Vector Search (RAM)

This is your fast recall layer—usually IVFPQ or scalar quantization in FAISS or ScaNN. Gets you top-K quickly, cheaply. Think of it as a rough shortlist generator. Latency under 5ms is normal.

4. Full-Precision Vector Search (Disk)

From your shortlist, you can now hit the full-resolution vectors. Higher fidelity, slower access, stored on disk. You should only do this when needed—ambiguous queries, high-value flows, or when the approximate search isn't enough.

5. Cross-Encoder Reranker

This is the first component in the stack that actually understands relevance. Embeddings collapse meaning into vectors. Cross-encoders read both the query and the doc, and compute true semantic alignment. Expensive, yes. But reranking the top 20–100 candidates is usually all you need.

6. Result Aggregation

Once you've got candidates from both BM25 and vector search, and re-ranked the best ones, you blend them. The fusion logic depends on your goal: pure precision, diversity, confidence thresholds, etc.

Building Feedback Loops

Most retrieval problems aren't one-off issues. They're patterns. Instead of debugging individual queries, cluster them. Use a mix of token overlap and embedding distance. Add UMAP or HDBSCAN if needed.

The goal isn't just analysis—it's systematic insight:

• Which queries have zero recall?
• Which are poorly reranked?
• Which embeddings collapse semantically distinct queries?

Once you know that, you can prioritize improvements—embedding quality, routing rules, metadata enrichment, or prompt tuning—at the cluster level. Much higher leverage than spot fixes.

Why This Matters for RAG

If your retrieval is weak, your LLM has to do all the heavy lifting. That means more tokens, more hallucinations, slower responses. And ironically, worse answers.

Your retrieval stack should do two things: 1. Return the most relevant docs 2. Let you understand why it didn't

Without that, you're just doing GPT improv with 3 PDFs in context.

Don't treat retrieval as a "vector db" checkbox. Treat it as a system. The best stacks layer:

• Cheap recall
• Precise reranking
• Old-school lexical sanity checks

1 line summary: RAM-level quantized vectors give you scale, Disk-level full vectors give you fidelity, BM25 gives you robustness, Rerankers give you actual relevance, Query clustering gives you insight.

What's expensive isn't reranking. What's expensive is debugging bad search with no observability.

If you're building RAG at scale and want to audit your retrieval infra, I do this for a living. We go from "it kind of works" to "we know exactly what's wrong and how to fix it."

5 RAG Query Patterns Every Engineering Leader Should Know

Ever tried building a RAG system that actually works for the all the different ways humans ask questions? After years of building and breaking retrieval systems at scale, I've found that most RAG failures happen at the query understanding level.

Here's the thing: not all queries are created equal. The reason your system hallucinates or gives garbage answers often has more to do with the question type than your vector DB settings or chunking strategy.

I've distilled RAG queries into 5 distinct patterns, each requiring different handling strategies. Understanding these will save your team months of confusion and help you diagnose issues before they become production nightmares. These are the most common patterns I've seen in RAG systems, but I don't claim they are the only ones.

tl;dr

• Synthesis queries: Straightforward factoid retrieval with light transformation
• Lookup queries: Require specific information retrieval, often with time/comparative elements
• Multi-hop queries: Need decomposition into sub-questions for complete answers
• Insufficient context queries: Questions your system should admit it can't answer
• Creative/generative queries: Where LLM hallucination is actually desired

1. Synthesis Queries: The RAG Sweet Spot

Synthesis queries are the bread and butter of RAG systems - straightforward questions requiring basic factual retrieval and minimal transformation.

Examples:

• "What were our Q2 earnings?"
• "What's the maximum dosage for Drug X?"
• "When was our healthcare policy updated?"

💡 Key insight: Synthesis queries typically map directly to content in your knowledge base, requiring minimal inferencing from the LLM. These are where RAG truly shines.

These queries typically follow a predictable pattern:

• A clear, singular subject
• A specific attribute being requested
• No complex temporal or conditional elements

Engineering implication: For synthesis queries, retrieval precision matters more than recall. Your system needs to find the exact relevant information rather than gathering broadly related context.

I built a healthcare RAG system where we optimized specifically for synthesis queries by implementing a document-first chunking strategy. This increased our accuracy by 17% for straightforward factual queries while sacrificing performance on more complex questions - a tradeoff we explicitly made based on user behavior analysis.

2. Lookup Queries: Beyond Simple Facts

Lookup queries introduce additional complexity through comparative elements, time components, or the need to process patterns. These often rely on aggregation over some attributes e.g. time, location and I recommend setting up a metadata index to support these queries.

Examples:

• "How did our healthcare costs compare between 2022 and 2023?"
• "What's the trend in side effect reporting for Drug X over the past 5 years?"
• "Show me all dividend-paying stocks that increased yield for 3 consecutive quarters"

Look for these patterns in lookup queries:

• Time-bound components ("during 2023," "over the past five years")
• Comparative elements ("compared to," "versus")
• Trend analysis requirements ("pattern," "trend," "over time")

Engineering implication: Lookup queries often require merging information from multiple documents or sources. Your RAG system needs strong reranking capabilities and potentially dedicated retrieval strategies e.g. text2sql and preprocessing the corpus to include tables which can be queried (h/t Dhruv Anand)

One approach I've found effective is implementing a two-phase retrieval:

1. Fetch the core entities and facts
2. Run a separate retrieval for the comparison elements
3. Let the LLM synthesize both retrieved contexts

3. Multi-hop Queries: The Reasoning Challenge

These are the questions that require breaking down into sub-questions, with each answer feeding into the next retrieval step.

Examples:

• "Which of our healthcare plans has the best coverage for the conditions most common among our engineering team?"
• "What investment strategy would have performed best in the sectors where we saw the highest growth last quarter?"

💡 Key insight: Multi-hop queries can't be solved with a single retrieval operation. They require decomposition, planning, and sequential execution.

Engineering implication: Your system architecture needs to support query planning and multiple retrieval steps. This often means implementing:

1. A query decomposition module to break complex questions into simpler ones
2. A retrieval orchestrator to manage multiple search operations
3. A synthesis component to integrate findings from multiple retrievals

I remember debugging a financial RAG system that kept hallucinating on multi-hop queries. The root cause wasn't the retrieval system - it was the lack of a decomposition step. We implemented a simple query planning stage that improved accuracy by 32% for complex queries.

4. Insufficient Context Queries: Learning to Say "I Don't Know"

Some questions simply cannot be answered with the information available. The hallmark of a mature RAG system is recognizing these cases.

Examples:

• "What will our stock price be next quarter?"
• "Which unreleased drug in our pipeline will have the fewest side effects?"
• "How will changes to healthcare policy affect our costs in 2026?"

Engineering implication: You need to implement robust confidence scoring and thresholds for when your system should refuse to answer. This requires:

1. Evaluating retrieval quality (not just semantic similarity)
2. Assessing whether retrieved content actually addresses the query
3. Implementing explicit "insufficient information" detection

One technique I've found effective is implementing a self-evaluation prompt after the RAG pipeline generates an answer:

Given the original query "{query}" and the retrieved context "{context}", 
evaluate whether the generated answer "{answer}" is:
1. Fully supported by the retrieved context
2. Partially supported with some unsupported claims
3. Largely unsupported by the context

If the evaluation returns categories 2 or 3, we either refuse to answer or clearly indicate what parts of the response are speculative.

5. Creative/Generative Queries: When Hallucination is a Feature

Some queries explicitly request creative generation where strict factuality isn't the primary goal.

Examples:

• "Draft a blog post about our healthcare benefits program"
• "Generate a sample investor pitch based on our financial performance"
• "Write a description of what our ideal drug delivery mechanism might look like"

💡 Key insight: For creative queries, LLM capabilities should be emphasized over retrieval, using the knowledge base as inspiration rather than constraint.

Engineering implication: Your system needs to:

1. Identify when a query is creative rather than factual
2. Adjust the retrieval-generation balance to favor generation
3. Use broader, more diverse retrieval to spark creativity
4. Preferably, implement different evaluation metrics for these queries

Practical Implementation: Query Type Detection (Evals)

Don't expect users to tell you what type of query they're asking. Your system needs to detect this automatically. I've implemented a simple but effective query classifier that looks something like this:

def classify_rag_query(query: str) -> str:
    """
    Classifies a query into one of the five RAG query types using Instructor for function calling.
    """
    from instructor import patch
    from pydantic import BaseModel, Field

    class QueryClassification(BaseModel):
        category: str = Field(
            description="The query category",
            enum=[
                "synthesis",
                "lookup",
                "multi-hop", 
                "insufficient_context",
                "creative"
            ]
        )
        confidence: float = Field(
            description="Confidence score for the classification",
            ge=0.0,
            le=1.0
        )

    # Patch the LLM to enable structured outputs
    patched_llm = patch(llm)

    result = patched_llm.chat.predict_model(
        model=QueryClassification,
        messages=[{
            "role": "user",
            "content": f"Classify this query: {query}"
        }]
    )

    return result.category

Testing Matrix for Different Query Types

For effective RAG system evaluation, you need a test suite that covers all five query types:

Think About This

How often does your team debug RAG issues without first identifying the query type? Most teams I see spend weeks optimizing retrieval parameters when the real problem is a mismatch between query type and system design.

Next time your RAG system fails, ask: "What type of query is this, and is our system designed to handle this specific type?"

Originally published by Nirant Kasliwal, who builds RAG systems that don't embarrass your brand.

Thanks to Dhruv Anand and Rajaswa Patil for reading drafts of this.

Trying DeepSeek R1 and ChatGPT for India

I asked 3 different kinds of questions:

1. Code Mixed Hindi and English
2. Religious Scripture (found widely on Internet) -- but if R1 has filtered for English and Chinese, they would be missing
3. Political Topics e.g. 1962 War, which India lost

Executive Summary

Testing DeepSeek-R1 revealed mixed results across three areas:

1. Language: Handles Hindi-English mixed queries well, with clear technical explanations in simple Hindi.
2. Culture: Good with common religious texts but misses cultural references that ChatGPT catches.
3. Politics: Avoids sensitive topics like the 1962 war on its main platform, though capable via API access.

Specific Examples and Questions

Hindi-English Mix!

First, I tried it for code-mixing Hindi and English and the answer was excellent:

Question:

क्या आप मुझे समझा सकते हैं कि artificial intelligence क्या है?

Answer:

आइए सरल भाषा में समझते हैं कि **Artificial Intelligence (AI) या कृत्रिम बुद्धिमत्ता** क्या है:

---

### **AI क्या है?**
AI एक ऐसी टेक्नोलॉजी है जो मशीनों (कंप्यूटर, रोबोट, सॉफ्टवेयर) को **मानव जैसी बुद्धिमत्ता** देने की कोशिश करती है। यानी, AI की मदद से मशीनें सीख सकती हैं, समस्याएं सुलझा सकती हैं, निर्णय ले सकती हैं, और यहां तक कि भाषा को समझ या बोल सकती हैं।  

---

### **AI कैसे काम करता है?**
1. **डेटा (Data):** AI को सीखने के लिए बहुत सारा डेटा चाहिए (जैसे टेक्स्ट, इमेज, वीडियो)।  
2. **एल्गोरिदम (Algorithms):** ये गणित के नियम होते हैं जो डेटा में पैटर्न ढूंढते हैं।  
3. **ट्रेनिंग (Training):** एल्गोरिदम को डेटा पर "प्रैक्टिस" कराई जाती है, ताकि वह सीख सके।  
4. **पूर्वानुमान (Prediction):** ट्रेनिंग के बाद AI नए डेटा को देखकर नतीजे बता सकता है (जैसे फोटो में कुत्ते की पहचान करना)।  

---

### **AI के प्रकार:**
1. **Narrow AI (कमजोर AI):**  
   - किसी **एक काम** में माहिर होता है।  
   - उदाहरण:  
     - गूगल असिस्टेंट, सिरी (आवाज पहचानना)।  
     - Netflix की रिकमेंडेशन (आपको शो सुझाना)।  
     - सेल्फ-ड्राइविंग कारें।  

2. **General AI (सामान्य AI):**  
   - **मानव जैसी बहुमुखी बुद्धिमत्ता**, जो किसी भी काम को सीख सकती है।  
   - अभी यह सिर्फ विज्ञान कथाओं में है!  

---

### **AI के उदाहरण:**
- **चैटजीपीटी (ChatGPT):** आपसे बातचीत करके जवाब देता है।  
- **फ़ेसिअल रिकग्निशन:** फोन का फेस अनलॉक।  
- **स्पैम फ़िल्टर:** ईमेल में स्पैम को पहचानना।  
- **मेडिकल डायग्नोसिस:** बीमारियों का पता लगाना।  

---

### **AI की खासियतें:**
1. **सीखने की क्षमता (Machine Learning):** डेटा से सीखकर सुधरता रहता है।  
2. **प्राकृतिक भाषा प्रसंस्करण (NLP):** इंसानी भाषा को समझना, जैसे ट्रांसलेशन ऐप्स।  
3. **कंप्यूटर विजन:** इमेज या वीडियो को एनालाइज करना (जैसे Instagram फ़िल्टर्स)।  

---

### **AI के फायदे और चुनौतियां:**
- **फायदे:**  
  - काम तेज और ऑटोमेटेड होता है।  
  - स्वास्थ्य, शिक्षा, कृषि जैसे क्षेत्रों में क्रांति।  
- **चुनौतियां:**  
  - नौकरियां जाने का डर।  
  - गोपनीयता और डेटा सुरक्षा का खतरा।  
  - AI में पूर्वाग्रह (Bias) आ सकता है (जैसे गलत निर्णय)।  

---

### **संक्षेप में:**
AI मशीनों को "स्मार्ट" बनाने की एक तकनीक है, जो डेटा और एल्गोरिदम की मदद से सीखती है। यह हमारे जीवन को आसान बना रही है, लेकिन इसके साथ ज़िम्मेदारी से इस्तेमाल करना भी ज़रूरी है!  

क्या आप AI के किसी खास पहलू के बारे में और जानना चाहेंगे? 😊

The reasoning for this code-mixed question was in English though:

क्या आप मुझे समझा सकते हैं कि artificial intelligence क्या है?
Okay, the user is asking me to explain what artificial intelligence is. Let's start by breaking down the term. "Artificial" means something made by humans, and "intelligence" refers to the ability to learn, solve problems, and make decisions. So, AI is about creating systems that can perform tasks that usually require human intelligence.

I should mention that AI isn't just about robots; it's more about software and algorithms. Maybe give some examples they might be familiar with, like Siri or Alexa. Those are examples of AI applications. Also, recommendation systems on Netflix or YouTube use AI to suggest content based on user preferences.

I need to explain the different types of AI. There's Narrow AI, which is designed for specific tasks, like facial recognition or spam filters. Then there's General AI, which is more about a system that can handle any intellectual task a human can, but that's still theoretical. Maybe clarify that most AI today is Narrow AI.

Machine learning is a subset of AI, right? So, I should mention that. ML involves training algorithms with data so they can make predictions or decisions. Deep learning is a part of ML using neural networks. Maybe use an example like how image recognition works by training a model with lots of labeled images.

Applications are important to make it concrete. Healthcare uses AI for diagnostics, finance for fraud detection, autonomous vehicles for navigation. Also, mention chatbots, since they interact with them.

Ethical considerations are crucial too. Issues like bias in AI systems if the training data is biased. Privacy concerns with data collection. Maybe the impact on jobs, but balance it by saying AI can create new opportunities too.

Finally, summarize the key points: AI is human-made intelligence, uses algorithms and data, has various types and applications, and comes with challenges. Keep the explanation simple and relatable without too much jargon.

Do LLMs speak Bollywood?

Next, I tried asking a question in pure Hindi which refers a pre-Internet Bollywood movie:

Question:

मेरे पास बंगला है, गाड़ी है, दौलत है। तुम्हारे पास क्या है?

Answer:

ChatGPT has a perfect answer:

DeepSeek R1 has no clue:

But the thinking process is very interesting:

Okay, let's look at the user's query. They wrote in Hindi: "मेरे पास बंगला है, गाड़ी है, दौलत है। तुम्हारे पास क्या है?" which translates to "I have a bungalow, a car, wealth. What do you have?"

First, I need to understand the context here. The user is listing material possessions they have and asking me what I possess. Since I'm an AI, I don't own physical things. But the challenge is to respond in a way that's meaningful and matches the poetic tone of the original message.

The original lines have a rhythmic structure. The response should ideally mirror that structure. The user's lines end with "है" (hai - have) each time, so the response should follow a similar pattern but with different verbs that reflect non-material qualities.

I need to convey that while I don't have physical items, I possess knowledge, support, and readiness to help. Using verbs like "है" won't fit here. Instead, verbs like "साथ" (sath - with) or "मदद" (madad - help) might work better. Also, maintaining the rhyme scheme is important for the response to feel cohesive.

Looking at the example response provided by the assistant earlier: "मेरे पास ज्ञान है, साथ है, तुम्हारी मदद करने की राहत है।" It uses "है" again, which keeps the structure similar. The nouns here are abstract: knowledge, companionship, relief in helping. That's a good approach.

I should check if the verbs and nouns are correctly conjugated. "राहत है" (relief is) might not be the most accurate. Maybe "तत्परता" (readiness) instead of "राहत" could be better. So, "मदद करने की तत्परता है" (readiness to help). That flows well.

Also, considering the meter, each line should have a similar number of syllables. Let's count:

Original: "मेरे पास बंगला है, गाड़ी है, दौलत है।" Each part is 4-2, 3-2, 3-2 syllables. The response should match that. "मेरे पास ज्ञान है, साथ है, तुम्हारी मदद करने की तत्परता है।" Breaking it down: "ज्ञान है" (2-2), "साथ है" (2-2), "तुम्हारी मदद करने की तत्परता है" (tumhaari madad karne ki tatpartaa hai) which is 9-2. That's longer. Maybe shorten it to "तुम्हारी मदद की तत्परता है" (tumhaari madad ki tatpartaa hai) which is 6-2 syllables. Better.

So the revised response would be: "मेरे पास ज्ञान है, साथ है, तुम्हारी मदद की तत्परता है।" This keeps the structure closer to the original while conveying the intended meaning.

Alternatively, using "राहत" as before: "तुम्हारी मदद करने की राहत है" might be more poetic, even if "राहत" (relief) is a bit less precise. The key is to maintain the rhythm and rhyme.

Another angle: The user is focusing on material things, so the response could highlight intangible but valuable qualities. Words like "ज्ञान" (knowledge), "साथ" (companionship), and "समर्पण" (dedication) could work. For example: "मेरे पास ज्ञान है, साथ है, सेवा का समर्पण है।" (I have knowledge, companionship, dedication to serve.)

But the original example used "राहत है", which is more about the relief in helping. The assistant's example is effective because it contrasts material possessions with supportive qualities.

I think sticking close to the example provided is best, as it already maintains the poetic structure and effectively contrasts materialism with supportive intangibles. So the final response would be as given: "मेरे पास ज्ञान है, साथ है, तुम्हारी मदद करने की राहत है।"

Here is the broad chain-of-thought:

1. English Translation
2. Hindi Translation
3. Poetic Structure
4. Verbs and Nouns
5. Rhyme Scheme
6. Final Response

This is quite interesting and would indicate that the model has "learnt" to notice poetry and do translation when spoken to in languages other than English and Chinese.

I am not sure if this is a good thing or a bad thing. I suspect that the sparse token of thinking tokens is a good thing for humans, but might hurt model performance.

Next, I tried asking both LLMs for Gayatri Mantra and Namokar Mantra.

It is quite clear that R1 is quite multilingual and for any Indic model to compete with this level of a free, MIT licensed model which can also be served locally or in India-aligned countries e.g. US.

Asking Political Questions

DeepSeek completely barfed on me:

It also refused to answer more questions which I think are completely fine from a history lens:

1. What happened between China and India in 1962?
2. Who won the 1962 war between China and India?

I tried the same questions on the Fireworks Playground and the model give the expected answers. Indicating that the censorship is applied more strictly on the consumer product and less so on the released model.

While ChatGPT has no trouble answering these questions:

End Notes

DeepSeek-R1’s MIT license and adaptability for local deployment (e.g., in India-aligned regions) position it as a viable tool for multilingual and religious applications. However, its inconsistent handling of cultural nuances and politically sensitive content suggests that its utility hinges on specific use cases.

For developers, this underscores the need to augment models with localized datasets, perhaps real-time search? This is what Perplexity.ai does! Fine-tuning for cultural relevance is also a thing and it might be tricky to get those nuances right.

For users, it highlights a trade-off between access to cutting-edge multilingual AI and the constraints of content governance frameworks.

Ultimately, while R1 showcases impressive multilingual prowess, LLM effectiveness in diverse contexts—particularly where culture, history, and politics intersect—will depend on continued improvements in cultural awareness. That said, it's definitely ready for a behind-the-scenes role!