#9 - Tracing in AI

INTRODUCTION

In traditional web applications, engineers rely on tools like Loki for logging, Prometheus for monitoring server utilization, and Grafana for visualization in order to bring observability into their application to get more metrics. However, these tools aren't specifically designed for AI applications, which have unique observability challenges. AI systems need specialized monitoring to track various data, particularly tracing user journeys, capture detailed API usage metrics, identify potential bugs and debug application issues. In order to address the tracing challenges, we can either adapt existing tools with custom configurations and plugins or explore new tools specifically designed for AI, offering tailored features for tracking and analyzing AI-specific metrics. We will explore new AI tracing tools in this article.

LANGSMITH

Langsmith is a closed-source tool developed by the Langchain company. It offers a free tier and is particularly effective when used with Langchain products. To get started with Langsmith, you can use a free-tier API key, import the library, and create a wrapper for the OpenAI client. By marking functions with a ‘@traceable’ decorator, you can easily trace and monitor their usage. Langsmith also provides an option to gather feedback from users, and you can organize logs using ‘threads’ for better clarity. Langsmith works with both python and javascript.

How to use Langsmith:

1. Install Dependencies
   “pip install -U langsmith openai”

2. Get an API key [from their website]

3. Set the API key in the environment variables

4. Trace OpenAI calls
   Wrap the function call which creates the OpenAI client with ‘wrap_openai()’ method like this -

   

   Now all queries are tracked which can be viewed in the Langsmith dashboard like this:

   

5. Trace the entire application with ‘@tracable’ decorator [add it on top of every function declaration]
   like this:

   

   As we can see from the screenshot below, the ‘rag’ function (as declared above) call is traced and is presented in the dashboard.

   

Now, one concern with Langsmith is that the data is being sent to the Langsmith server to process and visualize. Wouldn’t it be better if there is an open source version alternative to Langsmith? There is one such tool and it is called - LangFuse.

LANGFUSE

LangFuse is an open-source tracing tool for monitoring AI usage. It allows self-hosting, which benefits organizations focused on data privacy. However, certain advanced features like RBAC, customer support, etc comes at an extra cost. LangFuse operates asynchronously, processing data in the background to avoid affecting app performance. It uses the 'clickhouse' database and ‘minio’ (open source version of AWS S3) in the background to work on the captured data efficiently.

How to use Langfuse:

1. Install dependencies
   “pip install langfuse”

2. Setup environment variables
   [langfuse secret-key, public-key & host and OpenAI api key as suggested by langfuse]

3. Instead of OpenAI, import the OpenAI client from the langfuse library

4. Trace with ‘@observe’ decorator

   

5. See the data coming in the dashboard of langfuse UI

   

There are a few other options for tracing like - Helicone, Arize Phoenix, etc. Regardless, Langsmith and Langfuse are the most prominent tracer applications in AI which are commonly used by organizations.

CONCLUSION

In this article, we took a brief look at the concept of observability in AI applications and saw the usage of two tracer tools - langsmith and langfuse. We’ve seen how to set them up to start gauging an AI application and how it works.

---

REFERENCE

1. GenAI cohort by Hitesh Chaudhary and Piyush Garg

2. Langsmith documentation

3. Langfuse documentation