Data Flywheel in agents

Data flywheels are very important if you are building agents in production. If you have live users using your agent having a proper data flywheel in place is really crucial.

What is a data flywheel?

Data flywheel is a self-reinforcing loop that uses the data that you have gathered via logs or user feedback to continuously improve and adapt the product for better usability and accuracy.

Its basically a feedback loop that you implement in your product development lifecycle. It is mostly used in live products. Its a important facet that you have to consider to keep the product error free and easy to use for the users.

Data Flywheels in AI Agents

AI agent is just classical piece of software paired with token predicting language models called LLM’s.
Though mostly the ideas and concepts used while building the agents are similar to the traditional softwares, there are some major differences, the classical software is mostly Deterministic on the other hand software powered by AI is non-deterministic.
We need data flywheels for these agents too to keep them error free, and as easy as possible to use for the users with lesser friction.
For any data flywheel to provide us value, the most important thing we need is data.

Data points to use

In agents there few sources that you can utilize to collect this data for your flywheel, here some examples:

1. Tracing Logs:

   If you are using something like langsmith or langfuse to trace your llm calls and chain , data at each step is captured on these platforms. You can use this data to better understand whats going under the hood and how do different models perform for different nodes or use cases. It also allows you to look at the time it took for each call.

2. Failed Chats:

   Sometimes the agent might fail at serving some user query and might cause your agent to break , in such case that request is failed. You can look at the traces and logs to understand what happened at each step , each llm call and each parsing that happened.

3. User feedback in the chatbot:

   Its always better to have some kind of a feedback mechanism inside the UI of your agent to capture if the response by the agent was good or not. This is a valuable marker for use that is organic and true with the user.

4. User patterns:

   You might observer some patterns that user are following with your agent , maybe you need to see what tasks users are performing frequently and see how we can improve that task or that capability of the agent.

5. Direct user feedback:

   The most valuable data point that you would ever get is live feedback from the user, be it on online meetings or some meetups. You can have a genuine conversation with user and get important feedback.

How do they help you

By observing the tracing logs you can save the cost of each interaction with the agent by choosing the right LLM to use at each step. You can also see what parsing techniques are performing better with XML or JSON, also seeing the tool calls traces you will be able to understand if you have written the tools in way that is comprehensible to the smart LLM’s.
Failed chats allows you to find the loop holes in the agent, maybe some queries are causing recursion and eventually crash the agent. Also by referencing and analyzing the chats where users gave good feedback , you can extract good prompts for the LLM’s and some user approved response formats with relevant examples in the prompt.

Better Agents

After seeing all of this data , your beliefs might shatter , the feature that you thought was so important maybe isn’t, the users want something else, maybe the core reason why people are using the agent is totally different from what you intended it to. Facing the harsh truth and seeing the loopholes and failed chats you will be informed with the holes that need cementing. By prioritizing these issues and fixing them will make your agents flow smoothly through each node and solve real user problems.

P.S

This was a very high level overview and short article on the topic I’am thinking of writing on this topic in more detail. I hope you enjoyed it , if you have any feedback let me know ,Peace, GG!