HyDE (Hypythetical Document Embedding)

HyDE (Hypothetical Document Embedding) is a technique used in Retrieval-Augmented Generation (RAG) systems to improve document retrieval by generating a hypothetical document based on the user's query.
Instead of directly using the user query for retrieval, HyDE leverages an LLM to create a synthetic document that represents the potential answer, and then uses this document's embedding for similarity search within the knowledge base.

Let’s get some insights with daigram :

Ek example se samjhata hoon abhi clear ho jayega ye concept :

🔍 Scene:

Imagine you’re building a RAG (Retrieval-Augmented Generation) system. A user asks a question like:

"Ashoka ne Buddhism kyun apnaya?"

Now normally, RAG goes to your document database, pulls in some passages that seem relevant, and then answers.

But what if your database doesn’t have an exact match?
Matlab — jo dhoondhna tha, woh mila hi nahi. 😕
Bas kuch vague vague results aa rahe hain.

---

💡 Enters: HyDE – Hypothetical Document Embeddings

Instead of directly searching, you ask the AI:

"Bhai, agar aisi document hoti jisme iska jawab hota, toh woh kya bolti?"

So the AI imagines an answer — ek hypothetical document banata hai — based on its knowledge.

Phir us imagined content ko embed karke vector search karta hai — taaki jo actual documents hain database mein, unse achhe se match kar sake.

---

📦 Simple Analogy:

Maan lo tu kisi padosi se poochhta hai:

"Dilli ke Lajpat Nagar mein best chole bhature kahan milte hain?"

Lekin padosi Dilli gaya hi nahi kabhi.

Toh woh kya karega?
Sochke bolega:

"Agar main gaya hota na, toh mujhe lagta ki bade purane area mein, local dukaan hogi jiska bada naam hoga, line lagi hoti logon ki..."

Ab woh dimaag laga ke ek imaginary dukaan ka idea deta hai.
Aur us idea se tu Google ya kisi dost se real dukaan dhoondhta hai.

Wahi HyDE karta hai — ek sochi samjhi (hypothetical) document banata hai, uski embedding karta hai, aur fir search mein use karta hai.

---

🧠 Step-by-step :

1. User asks:
   "Ashoka ne Buddhism kyun apnaya?"

2. AI thinks:
   "Agar mujhe is pe ek document likhni hoti, toh main kya likhta?"

3. AI writes something like:
   "After the Kalinga war, Ashoka felt deep remorse and adopted Buddhism to seek peace and spiritual growth..."

4. Ye hypothetical paragraph ko vector banake, database mein search karta hai.

5. Matches actual historical texts related to Ashoka, Kalinga war, and Buddhism — jo shayad directly query se nahi milte.

---

🎯 Summary:

HyDE = "Mann hi mann mein soch le, agar aisa likha hota, toh kya hota?"
Aur fir ussi imagination se asli cheezein dhoondh le."

Isse search aur relevant content milne ke chances bahut better ho jaate hain, especially jab query direct match na kare.

Ab actual code dekhte hain :

from openai import OpenAI
import requests
from dotenv import load_dotenv
import json
import os
from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from pathlib import Path
from collections import Counter
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from dotenv import load_dotenv
from openai import OpenAI
import os
import ast

load_dotenv()


# Load environment variables from .env file
load_dotenv()
apikey = os.environ["OPENAI_API_KEY"]

# Initialize OpenAI client
client = OpenAI(api_key=apikey)

# 1. Load and split PDF
pdf_path = Path(__file__).parent / "node_js_sample.pdf"
loader = PyPDFLoader(pdf_path)
docs = loader.load()

# Split document into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
split_docs = text_splitter.split_documents(documents=docs)

# 2. Create an embedder
embedder = OpenAIEmbeddings(
    model="text-embedding-3-large",
    api_key=apikey
)

# Only run the below once to insert data into Qdrant
# vector_store = QdrantVectorStore.from_documents(
#     documents=split_docs,
#     embedding=embedder,
#     url="http://localhost:6333",
#     collection_name="learning_node_js",
# )
# vector_store.add_documents(documents=split_docs)

# Connect to existing Qdrant vector store

retriever = QdrantVectorStore.from_existing_collection(
    url="http://localhost:6333",
    collection_name="learning_node_js",
    embedding=embedder,
)

print("📄 PDF Ingestion Complete!\n")

user_input = input("Please enter your question on node js : >  ")

system_prompt=f"""
you are a helpful assistant that provides relevant answer of user query

now give answer to the question

user input : ${user_input}
"""

print("\n🧠 LLM Thinking \n")

llm_answer = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"system","content":system_prompt}]
)

print("\n🧠 LLM generated answer\n")
print(llm_answer.choices[0].message.content.replace("*","").replace("`",""))

print("\n🧠 Preparing final answer : \n")

hypothetical_answer=llm_answer.choices[0].message.content.replace("*","").replace("`","")
query_embedding = embedder.embed_query(hypothetical_answer)

# Step 4: Use embedding to do similarity search
docs = retriever.similarity_search_by_vector(query_embedding, k=5)

# Step 5 (Optional): Final answer from LLM using context + question
context_text = "\n\n".join([doc.page_content for doc in docs])

final_prompt = f"""
You are a Node.js expert. Answer the user's question using the context below.

Context:
{context_text}

User question: {user_input}
"""

final_response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": final_prompt}
    ]
)

print("\n✅ Final Answer:\n")
print(final_response.choices[0].message.content.replace("*","").replace("`",""))

OUTPUT :

So that’s all about HyDE! 🧠✨
Umeed karta hoon ki aapko sab kuch achhe se samajh aaya hoga 🙌😊

Milte hain next blog/post/video mein! 📚🚀
Tab tak ke liye —
👉 Keep Learning 📖
👉 Keep Exploring 🔍
👉 Keep Growing 🌱💡

Take care and happy prompting! 🤖💬💛

#ChaiCode

#GENAI