Client Feedback Processing with AI Agents Using CrewAI

There are a number of blog posts out there on how to use crewAI to create personal assistants like blog post creating AIs. My motivation to write another one was to show how ai agents can be implemented to improve existing data processing software.
In this made-up scenario, we are running a big e-learning platform and users are constantly submitting messages through our feedback form. Each message needs to be reviewed, and a decision must be made about what action should be taken. The possible actions include:/

• Create a ticket in the ticketing system if the issue needs attention but isn’t urgent.

• Immediately escalate the issue to a human if it’s urgent or delaying a response could cause harm.

• Create a general feedback note if no immediate action is necessary.

As a task that requires understanding of human language and simple decision making skills, this seems to be an obvious choice for AI!

What’s An AI Agent And What’s CrewAI?

An AI agent is a piece of software built around a large language model (LLM) that can act autonomously - make decisions, perform tasks, and use tools. Essentially, it's like a mini version of ChatGPT that you can integrate directly into your code, giving it access to a variety of tools. These tools can scrape the web, read from databases, parse Excel files, or perform virtually any function you can think of. By combining decision making capabilities with external tools, these agents can automate a wide range of tasks.

CrewAI is a python framework build on top of langchain that makes it very simple to define those agents, their tasks and tools and have them work together in a crew. A crew can consist of a single agent processing a single task or of multiple agents working together and even delegating work to each other.

Why CrewAI?

Today, OpenAI and others offer APIs to make requests to their LLM, have it return specific results, and even use tools. So why should we use crewAI, especially if we are only using a single agent to solve a single task?

CrewAI makes it easy to define agents and tools in a way that is reusable and very easy to understand. It provides tools for logging and monitoring, and - as we will see later - it is not tied to a specific LLM. Although crewAI uses OpenAI's GTP models by default, its agents can be driven by a large number of models, including local LLMs, so that sensitive data is kept in our system.

CrewAI is one of the most popular frameworks right now, but it’s worth mentioning that the same can be achieved with a number of other frameworks like haystack or ell.

Defining A CrewAI agent

Let’s get to it! You can check out the full code repository on github.

First, we define an agent. CrewAI makes this very straight-forward, we just have to provide a role, a goal and a short backstory. This will be used in the LLMs system prompt to let the language model know how it should act.

Agent(
    role="Customer Support Representative / Customer Service Agent",
    goal="Understand user feedback, decide what action to take and apply the tool to take the action.",
    backstory="You are an experienced customer support agent.",
    verbose=True,
)

Setting verbose=True will make the agent print it’s thoughts and decision making to the terminal, a very useful feature when it comes to debugging.

Creating The Tools

Next, we define some tools for the agents to use. Remember, we want the agent to decide how to handle a user message, whether to create a ticket in the ticketing system, escalate the issue to a human, or create a feedback note.

@tool("Save Task Tool")
def create_ticket_tool(ticket_description: str, user_id: Optional[str] = None):
    """Create a ticket in the ticket system that will be reviewed and solved later by a team member"""
    tasks_db.insert({"task": ticket_description, "user_id": user_id})
    return "Task saved"


@tool("Escalate issue tool")
def escalate_issue_tool(issue_description: str, user_id: Optional[str] = None):
    """Immediately escalate an urgent issue to a human"""
    escalations_db.insert({"issue": issue_description, "user_id": user_id})
    return "A human has been informed!"


@tool("Create Feedback Note Tool")
def create_feedback_note_tool(feedback_description: str, user_id: Optional[str] = None):
    """Create a general feedback note for the team to review"""
    feedback_db.insert({"feedback": feedback_description, "user_id": user_id})
    return "Feedback note saved"

Let’s break it down:

• The @tool() decorator marks our python function as a crewAI tool (and provides a tool name for the agent.

• The python docstring serves as a tool description and will also be handed to the agent.

• The agent will then receive feedback from the return statement.

• In this example, we simply use tinyDB to store these items.

You may be thinking: How can this work reliably? How will the agent know what parameters to provide? - And you wouldn't be wrong. For example, the agent gets a clue about the parameter type from its name: ticket_description suggests that the function expects a string. In the past, simply calling the ticket parameter would confuse the agent into trying to pass a dictionary with title and description attributes. The definitions above work for our simple case, but crewAI also provides a way to implement tools as classes and use Pydantic to thoroughly define their parameters and their types.

Defining A Task

We can now define a precise task for our agent to complete:

def create_message_processing_task(agent: Agent):
    return Task(
        description=(
            """
            Your task is:
            1. Read the user's feedback.
            2. Decide what action to take.
            3. Use the tools provided to take the action.

            These are the decisions you can make and their corresponding actions:
            1. If the user requires immediate help to prevent further damage, escalate the issue to a human.
            2. If the user is reporting on something that needs to be fixed or looked into at some point, create a task.
            3. If the user is providing general feedback, create a feedback note.
            4. If you think no action is required, you can ignore the feedback.

            You are done when you have taken the action.

            The user's user_id is: {user_id}

            This is the user's feedback:

            {feedback}
        """
        ),
        expected_output="Explain what action you took and why.",
        human_input=False,
        tools=[
            create_task_tool,
            create_feedback_note_tool,
            escalate_issue_tool
        ],
        agent=agent,
    )

Traditionally, a task can be defined much shorter and doesn't need to have tools attached. In our case, we don’t rely on task outputs directly, because we use tools to handle that. A task description could be as simple as: "Improve the wording in this text: {text}" , with the expected output being used directly from the task. Variables like {user_id} and {feedback} are provided and replaced dynamically once we get our crew to work.

Experience shows that there is a trade-off between task definition and performance and reliability. A broader task definition might give the agent more room to be "creative" and come up with a solution to more unforeseen problems. But in tests, it also significantly increased the likelihood that the agent would go around in circles or get stuck. A solution might be to be precise in the task definition, but allow the agent to delegate the work as a new task to another agent if complications arise.

Putting Together The Crew

It proved most effective to have a dedicated crew work on a specific task or set of related tasks with a corresponding agent or group of agents. Reusing crews, agents, or task instances often led to unwanted side effects in their behavior, so we now instantiate everything fresh for each incoming feedback item.

for message in feedback_messages:
        crew = Crew(
            agents=[agent],
            tasks=[task],
        )
        crew.kickoff(
            {"feedback": item["message"], "user_id": item["user_id"]},
        )

You can view the full source code on github.

Results

So far, the set-up agent works very reliably. Urgent issues where users cannot upload their assignments are immediately escalated, while the system formulates tickets for bug reports and forwards general feedback.

Using A Local LLM For Privacy And Data Security

Per default, crewAI will require you to have an OpenAI API key set and use the gpt4-o-mini model for agents (by the way, if you don’t have an OpenAPI API key, you should get one, their API is ridiculously cheap to use for development).
Im many scenarios, especially with EU clients, privacy and data security are big concerns, so businesses might be very reluctant to use remote models.

Using a local LLM in crewAI is as easy as installing Ollama, pulling the model and configuring the agent with something like llm=”ollama/llama3”.

Local LLMs have made incredible progress in the past: They can be powerful alternatives and they come in many shapes and sizes for many different purposes. Although a local LLM might produce comparable results, it will likely be a lot slower and changes in the agents behavior have to be expected. For example, an agents interpolation of tool parameter types from parameter names might be different.

Final Thoughts

Talking with my clients and colleagues, I found it surprisingly difficult to identify real-world scenarios where an actual team of AI agents could be used to make complex decisions and delegate tasks. These scenarios often touch on the core operations of a business - areas where clients generate the most value and prefer that humans make the key decisions. Some might argue that this is exactly the reason why these humans will be obsolete in the future. However, after working with agents for some time, I can confidently say that we’re not even close to a point where I’d trust current-generation language models to take on such responsibilities. That said, it's October 2024, and things could evolve faster than we expect.

However, we came across many examples like this one, where a single AI agent may be implemented to make easier decisions and derive actions. In these cases, some control flow might still remain in human hands or be managed through the predictability of traditional structures like foreach-, if-, and else-statements. Meanwhile, projects like crewAI offer new ways to define control flows where AI agents are actively integrated into the process