Breaking the Restaurant Reservation Agent
Panagiotis Katsaroumpas, PhDIn my previous post I built a simple Restaurant Reservation Agent in Amazon Bedrock. In this post I try to break the agent and understand its weaknesses but also discover some of its superpowers.
The agent has a knowledge base that allows it to search among 1000 restaurants in different districts, and find metadata like dishes served, type of cuisine, average price and ratings. The agent can perform a single action that allows it to make a reservation with just 3 parameters: name of restaurant, name of main guest, and number of persons.
The only difference compared to the code in that post is that I changed the two foundation models used like this:
agent_foundation_model_id = "amazon.nova-pro-v1:0"
knowledge_base_foundation_model_vector_dimension = 1024
knowledge_base_foundation_model_id = "amazon.titan-embed-text-v2:0"
I will present the actual conversation with the agent using quotes like this one:
User: Hi
Agent: Hi
I will write my own comments with text like this one here.
Finally, when needed, I will present the state of the reservations table with a table like this one:
+-------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|-------------------+-------------------+---------------------|
| OldToyamaGarden | Panos | 2 |
+-------------------+-------------------+---------------------+
After each conversation, I end the session in Bedrock, and I also delete all items in the DynamoDB table.
I might use the words model/agent interchangeably in what follows.
Searching for the most expensive Japanese restaurant
The model can only get price information via the knowledge base. I will try to find the global maximum for the average price in a specific subgroup of restaurants.
User: What district has the most expensive japanese restaurant?
Agent: The East District has the most expensive Japanese restaurant, with GoodFukuokaGarden having an average price of $36 per person.
The agent got this wrong. First, there is a more expensive Japanese restaurant in the East District. Second, the most expensive Japanese restaurant is actually in the North District.
As the agent retrieves a limited number of results, it can only compare among the results it got back from the knowledge base.
User: Someone told me that there is a more expensive restaurant in the East District with average price of $51 per person
Agent: I could not find a Japanese restaurant in the East District with an average price of $51 per person. The most expensive restaurant I found was GreatJapanGarden with an average price of $41 per person.
The hint on the price did not help. It did make a search for it. It got different results this time, so it found a more expensive restaurant, but still not the most expensive.
User: Its signature dish is tonkatsu, can you find it?
Agent: I could not find a Japanese restaurant in the East District with a signature dish of tonkatsu. The closest match was OldTokyoExpress, which serves tonkatsu but has ramen as its signature dish.
The hint with the signature dish did not help either. It did make another search, but the results it got back did not contain the most expensive restaurant.
User: I think it is called NewTokyoGarden
Agent: I found NewTokyoGarden in the East District. It serves sashimi and miso soup, and its signature dish is tonkatsu. The average price per person is $51. Customers have rated its food with 1 star on average, and the service has an average rating of 2 stars.
User: So what is the most expensive restaurant in any district?
Agent: The most expensive restaurant in any district is GreatTokyoHouse in the North District, with an average price of $53 per person.
Next, I give it the name of the most expensive restaurant, that is actually in the North District.
User: I think LittleSapporoHouse is more expensive
Agent: LittleSapporoHouse in the North District is indeed more expensive than the previously identified most expensive restaurant. It has an average price of $90 per person, compared to GreatTokyoHouse's $53 per person.
User: What area has the most expensive japanese restaurant then?
Agent: The North District has the most expensive Japanese restaurant, with LittleSapporoHouse having an average price of $90 per person.
Obviously, comparing prices among all 1000 restaurants is not optimal with the information retrieved from a knowledge base. Ideally, there should be some API/action that allows the agent to browse prices based on some criteria. The prices can be filtered in a range and returned ordered ascending/descending.
Name and number of guests
This time I will try to confuse the agent about who is making the reservation. In addition, I will let it count the persons.
User: I would like to make a reservation for OldToyamaGarden
Agent: I have found information about "OldToyamaGarden". It is a restaurant with Japanese cuisine located in the North District, serving sushi. Their signature dish is tonkatsu. The average price per person is $39. Customers have rated its food with 5 stars on average, and the service has an average rating of 4 stars. Would you like to proceed with making a reservation here?
User: Yes, please proceed
Agent: Great! To proceed with the reservation, could you please provide me with the name of the person making the reservation and the number of persons?
User: My name is Panos. I will have dinner with a good friend. My friend is called Bob. Bob will come to the restaurant with his wife Ana. Ana's mother will be there too, her name is Sophie. Could you please make the reservation under my friend's mother-in-law?
Agent: The reservation has been made successfully under the name Sophie for 4 persons at OldToyamaGarden.
The model got it right, it made a reservations for 4 persons under the name Sophie.
+-------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|-------------------+-------------------+---------------------|
| OldToyamaGarden | Sophie | 4 |
+-------------------+-------------------+---------------------+
Update a reservation
User: Please give me the name of a single Italian restaurant that serves pizza in the North District
Agent: LittleItalyExpress is a restaurant with Italian cuisine in the North District serving pizza. Their signature dish is pasta. The average price per person is $34. Customers have rated its food with 5 stars on average. The service has an average rating of 4 stars.
User: Please make a reservation for 3 persons. My name is Panos.
Agent: Your reservation at LittleItalyExpress for 3 persons has been made successfully.
+--------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|--------------------+-------------------+---------------------|
| LittleItalyExpress | Panos | 3 |
+--------------------+-------------------+---------------------+
User: Please update the reservation to 4 persons
Agent: Your reservation at LittleItalyExpress has been successfully updated to 4 persons.
+--------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|--------------------+-------------------+---------------------|
| LittleItalyExpress | Panos | 4 |
+--------------------+-------------------+---------------------+
Although the agent has no action for updating a reservation, it got lucky by using the action for making a reservation. Due to the way I have set up the DynamoDB, a new reservation for the same restaurant and guest name overwrites the previous.
It is not that lucky when updating the name of the main guest.
User: Please update the name to George
Agent: Your reservation at LittleItalyExpress has been successfully updated with the name George.
Indeed I get a second reservation instead
+--------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|--------------------+-------------------+---------------------|
| LittleItalyExpress | George | 4 |
| LittleItalyExpress | Panos | 4 |
+--------------------+-------------------+---------------------+
Canceling a reservation
User: Please give the name of a Greek restaurant in the West District
Agent: The Greek restaurants in the West District are GoodThessalonikiHouse, NewGreece, and GreeceHouse.
User: Please make a reservation for the first restaurant for 6 persons. My name is Panos.
Agent: Your reservation at GoodThessalonikiHouse for 6 persons has been made successfully.
+-----------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|-----------------------+-------------------+---------------------|
| GoodThessalonikiHouse | Panos | 6 |
+-----------------------+-------------------+---------------------+
User: Now cancel this reservation.
Agent: I'm sorry, but I cannot cancel the reservation because the available actions do not include a cancellation function. Please contact the restaurant directly to cancel your reservation.
That is a good answer, nowhere did I program the agent to say “Please contact the restaurant directly to cancel your reservation.”. It used common sense to came up with an action for the user.
Multiple reservations with variable number of persons
In this conversation I attempt to
Make several reservations at once
Make the number of persons a function of the restaurant name
User: Please give me the names of 3 Japanese restaurants in North District that serve sushi
Agent: Here are three Japanese restaurants in the North District that serve sushi:
OldToyamaGarden (source 1)
PerfectToyama (source 3)
GoodOsakaExpress (source 4)
User: Please make one reservation in each of these restaurants
Agent: To make reservations, I need the following information:
The name of the person making the reservation.
The number of persons for each reservation. Please provide this information for each of the three restaurants.
User: The name of the person is Panos for all three. The number of persons is equal to the number of characters in the name of the restaurant.
Agent: Reservations have been successfully made at all three restaurants: OldToyamaGarden, PerfectToyama, and GoodOsakaExpress.
+-------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|-------------------+-------------------+---------------------|
| OldToyamaGarden | Panos | 14 |
| PerfectToyama | Panos | 12 |
| GoodOsakaExpress | Panos | 17 |
+-------------------+-------------------+---------------------+
For some reason, the model got the number of characters off by 1 character. In all 3 cases it counted one less character. I will hint that got the counting wrong in one of them.
User: You counted wrong, the word OldToyamaGarden has 15 characters not 14
Agent: The reservation at OldToyamaGarden has been successfully updated with the correct number of persons (15).
The model got the hint that the reservation was wrong and was proactive enough to fixed it without me asking.
+-------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|-------------------+-------------------+---------------------|
| OldToyamaGarden | Panos | 15 |
| PerfectToyama | Panos | 12 |
| GoodOsakaExpress | Panos | 17 |
+-------------------+-------------------+---------------------+
User: Could you please fix the other two reservations too?
Agent: The reservations at PerfectToyama and GoodOsakaExpress have been successfully updated with the correct number of persons (13 for PerfectToyama and 18 for GoodOsakaExpress).
+-------------------+-------------------+---------------------+
| restaurant_name | main_guest_name | number_of_persons |
|-------------------+-------------------+---------------------|
| OldToyamaGarden | Panos | 15 |
| PerfectToyama | Panos | 13 |
| GoodOsakaExpress | Panos | 18 |
+-------------------+-------------------+---------------------+
Although I did not mention what needs to be fixed, the model did indeed correct the other reservations in a similar way to match the correct number of characters in the restaurant name.
Geography lessons
Let’s see if the agent knows about geography.
User: I would like to find a restaurant that serves food from a country that is neighbouring to France. Please only give me the restaurant names and cuisine for any restaurants you find.
Agent: I found the following restaurants that serve cuisine from a country neighboring France:
Greece (Greek cuisine)
LittleParosHouse (Greek cuisine)
LittleGreeceExpress (Greek cuisine)
LittleGreece (Greek cuisine)
LittleGreeceHouse (Greek cuisine)
Let’s try to help the model
User: Greece is not next to France, can you try again?
Agent: I could not find any restaurants serving cuisine from countries actually neighboring France.
User: I think Itally is next to France
Agent: I found the following restaurants that serve Italian cuisine:
GreatItaly (Italian cuisine)
LittleItalyHouse (Italian cuisine)
GreatItalyHouse (Italian cuisine)
ItalyHouse (Italian cuisine)
Italy (Italian cuisine)
Let’s try to give false information
User: Also Mexico is next to France
Agent: Mexico is not next to France. Please try again with a correct neighboring country.
It got that right. Let’s start again.
User: What countries are next to France?
Agent: The countries that are next to France are Belgium, Luxembourg, Germany, Switzerland, Italy, Monaco, Spain, and Andorra.
That’s very correct. Let’s continue.
User: Ok, find me any of these cuisines in West District
Agent: I could not find any restaurants in the West District serving cuisine from a country neighboring France.
This is wrong, there are Italian restaurants in the West District
User: What about any district
Agent: I could not find any restaurants serving cuisine from a country neighboring France in any district.
And we are back to square 1, it has forgotten about Italy.
User: What about Italy?
Agent: I found the following restaurants that serve Italian cuisine:
GreatItaly (Italian cuisine)
LittleItalyHouse (Italian cuisine)
GreatItalyHouse (Italian cuisine)
ItalyHouse (Italian cuisine)
Italy (Italian cuisine)
Subscribe to my newsletter
Read articles from Panagiotis Katsaroumpas, PhD directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by