Chatbot Not Responding: Colab FastAPI + Flask + Ngrok Setup

UserUser
4 min read

I'm setting up a chatbot where the language model runs on Google Colab (FastAPI exposed via Ngrok), and a local Flask app serves the HTML frontend, acting as a proxy. The Ngrok tunnel for the Colab FastAPI appears to be running, but the chatbot isn't providing responses.

I suspect there's a disconnect in how the components are communicating.

Current Setup Details

1. Colab Code (FastAPI + Ngrok)

!pip install fastapi uvicorn nest_asyncio pyngrok transformers torch
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import uvicorn
import nest_asyncio
import threading
from pyngrok import ngrok
import time # Added for keeping Colab alive

# Apply the fix for Colab event loop
nest_asyncio.apply()

model_path = "/content/drive/MyDrive/mistral-finetuned-alpaca"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")

# Define FastAPI app
app = FastAPI()

class RequestModel(BaseModel):
    message: str

@app.post("/predict")
def predict(data: RequestModel):
    input_text = data.message
    # Ensure inputs are moved to the correct device (e.g., "cuda" if available)
    inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
    output = model.generate(**inputs, max_length=100)
    return {"response": tokenizer.decode(output[0], skip_special_tokens=True)}

# Function to run FastAPI
def run_fastapi():
    uvicorn.run(app, host="0.0.0.0", port=8000)

# Run FastAPI in a separate thread
fastapi_thread = threading.Thread(target=run_fastapi)
fastapi_thread.start()

ngrok.set_auth_token("###################") # (Placeholder, actual token used)


colab_public_url = ngrok.connect(addr="8000", proto="http")
print("Colab FastAPI Public URL:", colab_public_url)

# Keep the Colab environment alive
while True:
    time.sleep(60)

2. Flask Code (Local Proxy)

This Flask app runs on my local machine, serves chat.html, and proxies requests to the Colab FastAPI.

from flask import Flask, render_template, request, jsonify
import requests

app = Flask(__name__)


COLAB_API_BASE_URL = "https://2b19-24-105-72-44.ngrok-free.app" # Example URL from Colab

@app.route("/", methods=["GET"])
def home():
    return render_template("chat.html")

@app.route("/chat", methods=["POST"])
def chat():
    user_input = request.json.get("message")

    try:
        response = requests.post(f"{COLAB_API_BASE_URL}/predict", json={"message": user_input})

        if response.status_code == 200:
            bot_reply = response.json()["response"]
            return jsonify({"answer": bot_reply})
        else:
            print(f"Error from Colab API: {response.status_code} - {response.text}")
            return jsonify({"answer": "Sorry, the bot is offline or unavailable. (Colab API error)"}), 500
    except requests.exceptions.RequestException as e:
        print(f"Request to Colab API failed: {e}")
        return jsonify({"answer": "Sorry, the bot is offline or unavailable. (Network error)"}), 500

if __name__ == "__main__":
    app.run(debug=True)

3. HTML Frontend (chat.html)

<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Mistral Chatbot</title>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" rel="stylesheet">
    <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css" rel="stylesheet">
    <style type="text/css">
        /* (CSS styles omitted for brevity, identical to original) */
    </style>
</head>
<body>
    <div class="container background-color: rgb(255,0,255);">
        <br /><br /><br />
        <div class="row">
            <div class="col-md-8 col-md-offset-2">
                <div id="chatPanel" class="panel panel-info">
                    <div class="panel-heading">
                        <strong><span class="glyphicon glyphicon-globe"></span> Welcome To Mistral Chatbot !!!
                            (You: Green / Bot: White) </strong>
                    </div>
                    <div class="panel-body fixed-panel">
                        <ul class="media-list"></ul>
                    </div>
                    <div class="panel-footer">
                        <form method="post" id="chatbot-form">
                            <div class="input-group">
                                <input type="text" class="form-control" placeholder="Enter Message" name="messageText"
                                    id="messageText" autofocus />
                                <span class="input-group-btn">
                                    <button class="btn btn-info" type="button" id="chatbot-form-btn">Send</button>
                                </span>
                            </div>
                        </form>
                    </div>
                </div>
            </div>
        </div>
    </div>
    <script src="https://code.jquery.com/jquery-1.12.4.min.js"></script>
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
    <script>
        var exports = {};
    </script>
    <script>
        $(function () {
            $('#chatbot-form-btn').click(function (e) {
                e.preventDefault();
                $('#chatbot-form').submit();
            });

            $('#chatbot-form').submit(function (e) {
                e.preventDefault();
                var message = $('#messageText').val();
                if (message.trim() === "") {
                    return;
                }
                $(".media-list").append(
                    '<li class="media"><div class="media-body"><div class="media"><div style = "text-align:right; color : #2EFE2E" class="media-body">' +
                    message + '<hr/></div></div></div></li>');

                $.ajax({
                    type: "POST",
                    url: "/chat", // Correctly points to local Flask app's /chat endpoint
                    data: JSON.stringify({ message: message }),
                    contentType: "application/json",
                    success: function (response) {
                        $('#messageText').val('');
                        var answer = response.answer;
                        $(".media-list").append(
                            '<li class="media"><div class="media-body"><div class="media"><div style = "color : white" class="media-body">' +
                            answer + '<hr/></div></div></div></li>');
                        $(".fixed-panel").stop().animate({
                            scrollTop: $(".fixed-panel")[0].scrollHeight
                        }, 1000);
                    },
                    error: function (error) {
                        console.error("Error sending message to Flask:", error);
                        $(".media-list").append(
                            '<li class="media"><div class="media-body"><div class="media"><div style = "color : red" class="media-body">Error: Could not get a response from the bot.</div></div></div></li>');
                    }
                });
            });
        });
    </script>
</body>
</html>

Problem and Troubleshooting Performed

The chatbot interface loads correctly when I navigate to http://127.0.0.1:5000/. When I type a message and hit "Send", nothing appears in the chat history from the bot.

Here's what I've tried so far for debugging:

  1. Colab Ngrok Tunnel Confirmation: The Colab notebook outputs the Ngrok URL (e.g., https://2b19-34-125-72-30.ngrok-free.app), indicating the tunnel is established.

  2. Direct FastAPI Test (from PowerShell): I attempted to test the FastAPI endpoint directly using curl.exe in PowerShell:

$jsonData = "{""message"": ""Hello, what is the capital of France?""}"
$ngrokUrl = "https://2b19-34-125-72-30.ngrok-free.app/predict" 
curl.exe -X POST -H "Content-Type: application/json" -d $jsonData $ngrokUrl

This command succeeded and I received a valid JSON response from the FastAPI server running on Colab. This suggests the Colab backend and its Ngrok tunnel are working correctly.

What I'm Looking For

Since the direct call to the Colab FastAPI works, the issue likely lies in the communication between my local Flask app and the Colab FastAPI, or how the Flask app handles the response.

My questions are:

  1. What specific steps should I take to debug the communication flow from the Flask app to the Colab FastAPI? What logs or console outputs should I be checking most carefully?

  2. Are there any common Flask or requests library pitfalls when calling an external API like this that could lead to the bot not responding even if the direct curl test works?

  3. Given the provided code, are there any subtle errors I might have overlooked in the Flask app or chat.html that could cause the response not to be displayed, even if the data is being sent correctly?

0
Subscribe to my newsletter

Read articles from User directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Help NeededPythonFlaskFastAPIwebdevMachine Learningngrokcolab

Written by

User
User