Hypothetical Document Embedding : Query Transformation Technique for Advance RAG

🌅Introduction

This article is part of Advance RAG Series, an article series where the tenets and features of Advance RAG Systems are explained with diagrams and code. Hypothetical Document Embedding popularly known by its acronym HyDE, is a Query Transformation/Translation Technique which optimizes the RAG system by creating an overview document of the user’s query based on its pre-trained knowledge and based on that overview document performs the Semantic Search in the Vector Store and gives the relevant data chunks for the Generation Process.

🕵️When & Where does HyDE come into picture?

A RAG system works on three fundamental steps : Indexing, Retrieval & Generation. To see in detail how a RAG works, read this article here.

Now HyDE is used in the Retrieval Process where as the user’s query is received, an overview document for the main topic/highlight of the user’s query is generated based on the pre-trained data of the LLM. This hypothetical document’s vector embeddings are then used for similarity search in the Vector Store’s data (stored during the Indexing Process). The output is passed as context to the LLM (the Generating Process now begins) to respond to the user’s query.

✨Effect of Using HyDE Technique

The Hypothetical Document generated using the HyDE technique provides additional foundational and overarching information related to the user's main query. This enriched content leads to the semantic search of a greater number of relevant document chunks. When these chunks are incorporated into the language model’s context along with the original user query, the result is a more optimized, accurate, and contextually precise response.

📊💻Step By Step Working Through Diagram & Code

1. From user’s query, a hypothetical document is created.

2. Then after finding the vector embeddings of the hypothetical document semantic search is performed on them to get relevant context.

3. The LLM uses that context and the original user query to generate more accurate response.

🌉Hypothetical Document Embedding Output

🔗Important Links

• Hypothetical Document Embedding Code File

• Advance RAG Article Series

• Advance RAG Repository

🎯Conclusion

Through this article you saw how to implement HyDE Technique in your RAG and make the response more accurate, efficient and optimised.