Building An AI Assistant For a Smart Helmet

Problem Statement

With the devlopment of various quick commerce app, delivery app in India especially,as a delivery rider checking your phones have become more and more frequent which can cause accidents. So basically the solution was to provide a smart helmet with crash detection features, and also an ai assistant which can be used to give handsfree control over device.

PS: This was a hackthon project idea, code is very just a bunch of quick fixes haha..

Link to the Repo

---

System Design for the AI chatbot

This is basically a glorified natural language to api request convertor.

Explaining the Working of this:

1. First the user send the natural query to the main NEXT JS server I have created.

2. Then the query is stitched to the prompt I have written.

const prompt = 
`You are an intelligent voice assistant router that determines how to handle user queries.

Available tools and their actions:
${toolDescriptions}

Based on the user's query: "${params.query}"

Please analyze if this is:
1. A GENERAL_QUESTION that requires factual information, conversation, 
or advice that doesn't need external tools
2. A TOOL_REQUEST that requires using one of the available tools listed 
above,also process the raw data carefully for example weather is sunnny or not 

If this is a GENERAL_QUESTION:
- Provide a direct, conversational answer
- Keep responses concise and natural for voice delivery

If this is a TOOL_REQUEST:
- Determine which tool and action should handle this request
- Specify what parameters should be passed to that action

Respond with a JSON object in this format:
{
  "queryType": "GENERAL_QUESTION" or "TOOL_REQUEST",
  "answer": "Direct answer if this is a general question" (only for GENERAL_QUESTION),
  "tool": "toolName" (only for TOOL_REQUEST),
  "action": "actionName" (only for TOOL_REQUEST),
  "params": {} (only for TOOL_REQUEST),
  "note": "Additional information if needed"
}`;

So basically I have a tool list called tools description basically all the api request I could write for e.g

Get Weather, Fake Zomato Api’s for product displaying just a simple get order and pickup orders ,etc;

So the gemini receives all the information and if it is a task to be performed it returns the tool name and action.

  if (parsedResponse.queryType === "TOOL_REQUEST") {
        // Validate tool and action exist
        if (!parsedResponse.tool || !parsedResponse.action) {
          return {
            queryType: "TOOL_REQUEST",
            execution_plan: parsedResponse,
            result: null,
            note: parsedResponse.note || 'Tool or action not specified'
          };
        }

        if (!tools[parsedResponse.tool] || !tools[parsedResponse.tool][parsedResponse.action]) {
          return {
            queryType: "TOOL_REQUEST",
            execution_plan: parsedResponse,
            result: null,
            note: `Tool or action not found: ${parsedResponse.tool}.${parsedResponse.action}`
          };
        }

        // Execute the tool
        //tools is module exported from tool manager can look in the github code for better explaination   
     const result = await tools[parsedResponse.tool][parsedResponse.action](parsedResponse.params || {});

Then this information is passed to the main server where the response is modified for bertter human understanding and shortened as well .

So basically if a rider say ,”Hey list all orders”

then the rider would receive response such as “Order from Place 1 , Place 2 , Place 3“,,

then the rider can say “Hey pickup order from Place 1 with order ID 1‘

and would receive response “Order picked up “

Again this was just coded in 24hr hackathon so there is no indexing done with data and not a really effiecient solution , Would’ve worked more on this but we didn’t win so the project is scraped.

---

Thanks For Reading