Improving Information Retrieval with Knowledge Graphs: Comparing VectorDB RAG vs Graph-Powered RAG on AWS

Introduction

Large Language Models (LLMs) have become ubiquitous, but their tendency to “hallucinate” non-factual details remains a challenge. Retrieval Augmented Generation (RAG) was introduced to ground LLM responses in actual data by retrieving relevant facts and incorporating them into prompts. However, when your data is highly interconnected—such as in codebases or scientific documents which call and cite other parts of the data respectively —traditional vector-based search can miss key relationships.

In this post, we’ll explore how integrating Knowledge Graphs into the RAG pipeline can boost the relevance and factual accuracy of the output. We’ll compare a conventional VectorDB RAG (using Amazon Bedrock with OpenSearch) with a Graph-Powered RAG (using Amazon Bedrock alongside a graph database like Amazon Neptune or Neo4j).

Summary YouTube Video:

• Will be Published Soon!

Problem Statement

While RAG helps mitigate LLM hallucinations by providing factual context, its typical implementation using vector similarity search has a critical flaw: It looks for similar chunks of document to add to the context and “similar” does not always mean “relevant”! For instance, when asking, “What are the possible causes of obesity?” a vector search might return descriptive text about obesity rather than pinpointing factors like “hypothyroidism” or “bad diet”—the actual causes.

This issue was tried to solve using various techniques, most notably ReRanking of the RAG result using another LLM based on relevancy. But one solution we are interested in is using Knowledge Graphs. We wrote a series of blog posts on Knowledge Graphs: Our Blog Series. Here we want to see if adding them to the LLM pipeline can help the performance of the generation.

Fig 1. Knowledge Graph-Powered RAG vs Vector RAG

Use Case Example: Code Refactoring

Consider a codebase, here the Python Requests Library, where functions are scattered across multiple files and are highly interconnected. Suppose you need to refactor a function (say, utils.urldefragauth), but you’re unsure which other functions depend on it. A simple vector search might return the function definition only. However, a Knowledge Graph that maps function dependencies can retrieve all related functions—giving you the necessary context to ensure a safe refactor.

Knowledge Graph-Powered RAG: How Does It Work?

Traditional RAG uses vector embeddings to find similar text chunks. In contrast, Knowledge Graph-Powered RAG leverages a Knowledge Graph to capture explicit relationships among data points (e.g., function calls in code). This approach uses a query language—such as Cypher—to ask precise questions about how entities are connected.

For example, a valid Cypher query to find which functions call a particular function might be:

MATCH (caller:Function)-[:CALLS]->(target:Function {name: 'utils.urldefragauth'})

RETURN caller.name AS CallingFunction, caller.code AS CallingFunctionCode

LIMIT 5;

Here I used the LlamaIndex package which provides some functionality including the TextToCypherRetriever which does the heavy job of turning a text query into a Cypher query compatible with your knowledge graph. What happens underneath it though is basically begging an LLM to turn the query and knowledge graph schema into a valid Cypher query!

How to Prepare the Code Graph?

Code repositories are a perfect example of highly interconnected data. Modules import functions from various other modules and are themselves called by various other functions.

A crucial step in our workflow is preparing a fact-based Knowledge Graph—in this case, a graph of functions and their relationships, specifically which function calls which. Here, I used the Python ast library to parse the Python files of the popular package requests.

Here are some code snippets:

def parse_python_file(file_path):

    """Parse a Python file to extract functions, classes, dependencies, imports, and code."""

    with open(file_path, "r") as f:

        source = f.read()

        source_lines = source.split('\n')

        tree = ast.parse(source, filename=file_path)


    visitor = FunctionVisitor(source_lines)

    visitor.visit(tree)

    return visitor.functions, visitor.imports, visitor.classes

Below is a part of the final graph (stored in Neo4j AuraDB):

Graph-Powered RAG vs Vector Database RAG: Example

To show the difference between a vector database approach and a knowledge graph approach, I set up an AWS Bedrock Knowledge Base, which uses OpenSearch Service for vector search. So I will compare its results with the Knowledge Graph hosted on a Neo4j database (alternatively, I could use an Amazon Neptune database for Graph with some small changes in the code!).

I asked both of these workflows (or you can call them AI Agents!) what happens if I change a function? This question clearly shows the difference in the graph approach, since to answer, you need to know which other functions call this function, which can be in any other files!

Here is the input to both of these agents:

I want to refactor the utils.urldefragauth function. What other functions call it and might get affected? Show the CODE to those function.

Graph-Powered RAG Answer (Generate using Amazon Bedrock):

Based on the provided information, the function that calls utils.urldefragauth is adapters.HTTPAdapter.request_url. This function might be affected if you refactor utils.urldefragauth. Here's the code for adapters.HTTPAdapter.request_url:

def request_url(self, request, proxies):

    """Obtain the url to use when making the final request.
    If the message is being sent through a HTTP proxy, the full URL has to
    be used. Otherwise, we should only use the path portion of the URL.
    This should not be called from user code, and is only exposed for use
    when subclassing the
    :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
    :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
    :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
    :rtype: str
    """
    proxy = select_proxy(request.url, proxies)
    scheme = urlparse(request.url).scheme
    is_proxied_http_request = proxy and scheme != "https"
    using_socks_proxy = False

    if proxy:
        proxy_scheme = urlparse(proxy).scheme.lower()
        using_socks_proxy = proxy_scheme.startswith("socks")

    url = request.path_url
    if url.startswith("//"):  # Don't confuse urllib3
        url = f"/{url.lstrip('/')}"

    if is_proxied_http_request and not using_socks_proxy:
        url = urldefragauth(request.url)

    return url

This function uses urldefragauth when dealing with proxied HTTP requests that are not using SOCKS proxy. Any changes to the behavior or signature of utils.urldefragauth could potentially impact this function, so you'll need to ensure that your refactoring doesn't break the existing functionality or interface expected by adapters.HTTPAdapter.request_url .

Vector Database RAG Answer (Generate using Amazon Bedrock):

I apologize, but I cannot provide an answer to your question based on the given context information. The provided code snippet does not contain any reference to a utils.urldefragauth function or any functions that call it. The code appears to be a collection of utility functions related to networking, proxies, and HTTP requests, but it doesn't include the specific function you're asking about or any functions that call it. Without more relevant context about the utils.urldefragauth function, I'm unable to identify the functions that might be affected by refactoring it or show you their code.

Comparison:

As you can see since the Vector database is unable to see the called functions relationship it was unable to answer the question unlike the Knowledge Graph approach which successfully found the relationship and answered the questions!

Table Summary:

Conclusion

For applications where factual accuracy and interconnected data are critical, a Graph-Powered RAG approach shows considerable promise. Although it demands a higher upfront investment in terms of graph preparation and system complexity, the improved contextual accuracy can be invaluable—especially for domains like code refactoring or multi-entity document retrieval.

For those interested in experimenting further, the full code repository is available here:
GitHub Repository: CodeGraphRAG