RAGAS v.0.2.14 Arbitrary File Read Vulnerability

The Arbitrary File Read in RAGAS was introduced in multimodal eval support feature of release v0.2.3. This vulnerability affects all the versions from v0.2.3 to v0.2.14 (latest)

The vulnerability arises because URL provided in retrieved_contexts is improperly handled and sanitized. The Vulnerability can be exploited in implementations where URL is accepted from untrusted source, opening up potential vulnerabilities that attackers can exploit. Malicious actors may use this opportunity to perform arbitrary file reads, internal port scans and accessing sensitive cloud metadata that should remain secure and private. Additionally, attackers might employ side channels to exfiltrate data, bypassing traditional security measures. These side channels can be subtle and difficult to detect, allowing attackers to quietly siphon off data without raising immediate alarms.

To successfully perform an Arbitrary File Read from the server, we need to achieve two goals.

1. Bypass existing image validations

2. Access the arbitrary internal files

Vulnerability

class ImageTextPromptValue(PromptValue):
    items: t.List[str]

    def to_messages(self) -> t.List[BaseMessage]:
        messages = []
        for item in self.items:
            if self.is_image(item):
                messages.append(self.get_image(item))
            else:
                messages.append(self.get_text(item))
        return [HumanMessage(content=messages)]

This class enables multi-modal prompting in RAGAS by providing an abstraction layer that handles both text and images in a unified prompt structure. It serves as a bridge between raw multi-modal content and language model interfaces, allowing RAGAS to evaluate retrieval-augmented generation systems that work with diverse content types beyond just text.

Our primary goal is to make is_image(item) return true, so the code will fetch the image using get_image(item)

Payload

item = "file://localhost/etc/passwd#payload.jpg"

1. is_image(url) check:

        def is_image(self, item):
            if self.is_base64(item):
                return True
            elif self.is_valid_url(item):
                mime_type, _ = mimetypes.guess_type(item)
                return mime_type and mime_type.startswith("image")
            elif isinstance(item, str):
                mime_type, _ = mimetypes.guess_type(item)
                return mime_type and mime_type.startswith("image")
            return False
   
        def is_valid_url(self, url):
            try:
                result = urlparse(url)
                return all([result.scheme, result.netloc])
            except ValueError:
                return False
   

   • is_base64(url): False.

   • is_valid_url("file://localhost/etc/passwd#payload.jpg"):

     • urlparse result: scheme='file', netloc='localhost', path='/etc/passwd', fragment='payload.jpg'.

     • Both scheme and netloc are non-empty. Returns True.

   • The elif self.is_valid_url(url): block is entered.

   • mimetypes.guess_type("file://localhost/etc/passwd#payload.jpg"): Python's mimetypes looks at the entire string. It sees .jpg near the end (even though it's in the fragment) and returns ('image/jpeg', None).

   • mime_type and mime_type.startswith("image"): True.

   • Goal 1 accomplished ✅

2. get_image(url) execution:

    def get_image(self, item):
            if self.is_base64(item):
                encoded_image = item
            elif self.is_valid_url(item):
                encoded_image = self.download_and_encode_image(item)
            else:
                encoded_image = self.encode_image_to_base64(item)
   
            return {
                "type": "image_url",
                "image_url": {"url": f"data:image/jpeg;base64,{encoded_image}"},
            }
   

   • is_base64(item): False.

   • is_valid_url(item): True.

   • The elif self.is_valid_url(item): block is entered again inside get_image.

   • Calls self.download_and_encode_image("file://localhost/etc/passwd#payload.jpg").

3. download_and_encode_image(url) execution:

    def download_and_encode_image(self, url):
            with urllib.request.urlopen(url) as response:
                return base64.b64encode(response.read()).decode("utf-8")
   

   • Calls urllib.request.urlopen("file://localhost/etc/passwd#payload.jpg").

   • urlopen correctly processes the file:// scheme.

   • urlopen standard behavior ignores the fragment (#payload.jpg) when resolving the resource path.

   • It requests the local file at the path /etc/passwd.

   • If the application has permission, it opens /etc/passwd.

   • The content is read (response.read()) and base64 encoded.

   • The encoded content of /etc/passwd is returned.

   • Goal 2 accomplished ✅

   • The file contents can now be leeched to any LLM or side-channel of attacker’s choice

---

Real-World Attack Scenarios

This vulnerability becomes critically dangerous in real-world implementations where untrusted input flows into the RAG evaluation pipeline — particularly through the retrieved_contexts field used in multimodal prompt construction. Below are key attack scenarios where this flaw can be weaponized:

1. Poisoned Knowledge Base or Evaluation Dataset

In many RAGAS implementations, users evaluate LLM performance based on documents indexed in a vector store or supplied as retrieval-augmented evaluation datasets. If these sources accept external or user-submitted data, attackers can embed malicious URLs directly into the knowledge graph or evaluation corpus.

Once the retriever fetches this poisoned data, the malicious URL (e.g., file://localhost/etc/passwd#img.jpg or http://169.254.169.254/latest/meta-data/iam/role-name.jpg) becomes part of the retrieved_contexts. RAGAS then treats it as an image due to improper MIME-type checks and attempts to fetch and encode the target — inadvertently leaking sensitive files or triggering SSRF-based access to cloud metadata and internal services.

This can happen silently during automated LLM evaluation, making it a stealthy and persistent attack vector in any system that evaluates generation based on user-controllable inputs.

2. Compromised or Malicious Evaluation Datasets

RAGAS is often used as part of automated QA pipelines where evaluation datasets are sourced from external contributors, CI/CD pipelines, or remote storage (e.g., GitHub, S3). If a dataset includes malicious entries in retrieved_contexts, the attack can be triggered without direct user interaction.

For instance, a data contributor (or attacker posing as one) inserts entries containing URLs that appear to be images but actually point to local files or internal endpoints. When RAGAS processes these during a batch evaluation run, arbitrary internal files can be read, encoded, and returned as part of the LLM prompt context or logs — effectively enabling data exfiltration through side channels.

3. Multi-Tenant Applications with RAG Evaluation APIs

SaaS platforms offering RAG evaluation as a service — especially in multi-tenant settings — are particularly exposed. A malicious user can submit crafted retrieved_contexts to probe the backend infrastructure, access cloud metadata, or escalate privileges by reading environment files (.env, AWS keys, etc.).

In these contexts, the vulnerability enables cross-tenant data leakage or server-side reconnaissance — even without explicit file upload permissions — simply by manipulating prompt input values.

4. Exposed Prototypes and Internal Tooling

Many developers run RAGAS locally or expose it through tools like Streamlit, Gradio, or FastAPI — often tunneled via ngrok or made available for demos. In these setups, attackers can deliver payloads through exposed UIs or APIs and abuse file URLs to read local secrets or scan internal networks.

---

Alternate Attack Vectors

Targeting cloud metadata: http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name.jpg – The AWS EC2 metadata service IP, with a fake .jpg suffix. The server (if running on AWS) will request the URL, receive sensitive credentials in response (the metadata service doesn’t care about the extension), and then base64-encode and likely return it to the attacker. This can leak AWS keys or instance data.

Targeting internal APIs: http://localhost:8000/admin/config.json.png – The server might reach an internal admin API on localhost. Even if the path is .json data, the added .png makes the URL end in .png. The service’s response (e.g. JSON with secrets) gets captured and encoded as if it were an image.

Port scanning or rogue requests: The attacker can supply URLs to internal IPs (e.g. http://192.168.1.100:22/ssh_banner.jpg) to scan open ports or interact with services using HTTP or even other protocols (if the code supports schemes like ftp:// or others, those could be used too.