If you are developing an AI agent or a simple rag app. You must have faced the issue where the LLM spits out random made-up stuff that is not true and not relevant to the context that you provided
This is called the infamous “Hallucination” Trait of the powerful LLMs where they start acting super weird.

They will hallucinate even when you have set the temperature as 0 in the model config.
So how do we solve this issue, how to tame the LLMs so that they don’t go crazy?

Well, there are many solutions out there but the one that worked best for me and is mostly followed in the industry is called Faithfulness Check.

RAG

So let’s take an example of a RAG application. It mainly has 3 components

1. A Vector Storage

2. Retriever

3. LLM

The vector storage holds our documents, the Retriever does the semantic search on our query and fetches the relevant documents based on the semantic score,e, and the documents are passed as a context to the LLM to generate the answer. That’s the basic idea behind the RAG

So when an answer is generated it is generated based on these 3 things being passed to an LLM

1. System Prompt

2. Context

3. Question

Faithfulness Check

We will add one more step in this flow called the Faithfulness check. Basically, we are going to use another LLM model to check if the answer generated is relevant to the context that was given to the LLM or not.
Based on that we will tell the LLM to give a score which will determine if the answer is hallucinating or not.
This helps us check if the answer generated is Grounded in actual truth or not.
You can use a prompt like this for the Judge LLM:

PROMPT = """
Given the following QUESTION, DOCUMENT and ANSWER you must analyze the provided answer and determine whether it is faithful to the contents of the DOCUMENT. The ANSWER must not offer new information beyond the context provided in the DOCUMENT. The ANSWER also must not contradict information provided in the DOCUMENT. Output your final verdict by strictly following this format: "PASS" if the answer is faithful to the DOCUMENT and "FAIL" if the answer is not faithful to the DOCUMENT. Show your reasoning.

--
QUESTION (THIS DOES NOT COUNT AS BACKGROUND INFORMATION):
{question}

--
DOCUMENT:
{context}

--
ANSWER:
{answer}

--

Your output should be in JSON FORMAT with the keys "REASONING" and "SCORE":
{{"REASONING": <your reasoning as bullet points>, "SCORE": <your final score>}}
"""

This score will help you determine the accuracy of the answer. This can be paired with tracing solutions like langsmith to determine what’s actually causing this in more depth.
Based on this you can tweak your pipelines to get better outputs and reduce hallucinations.