Introduction

In today's AI-driven world, accessing Large Language Models (LLMs) often requires a stable internet connection and subscription fees. However, there are many scenarios where a local, offline AI assistant is essential:

Remote areas with limited or no internet connectivity
Budget-conscious individuals looking to avoid subscription costs
Projects requiring strict data privacy and security
Educational environments where students need consistent AI access
Developers working with sensitive intellectual property
Students or individuals on low data budgets due to high data tariffs in certain regions

This guide will walk you through setting up a powerful offline AI assistant using Ollama and Open WebUI entirely on your local machine. By the end, you’ll have an AI assistant that works without an internet connection, safeguards your privacy, and eliminates ongoing costs.

Prerequisites

Before you begin, ensure you have the following:

A computer with at least 8GB RAM (16GB recommended)
At least 10GB of free storage space (more for additional models)
Basic familiarity with command-line operations
One-time internet access to download software and models

What is Ollama and Open WebUI?

In this guide, you’ll use two powerful tools: Ollama and Open WebUI.

Ollama is a lightweight framework for running Large Language Models (LLMs) locally on your machine. It provides a simple API for downloading and using models without an internet connection. Ollama supports various pre-built models, which can be used for tasks like chatting, summarization, and more.

Open WebUI is a self-hosted platform that provides a user-friendly interface to interact with AI models. It works with Ollama and other LLM runners to offer an easy-to-use web interface for AI interactions. By running Open WebUI locally, you can have a more seamless and customizable experience for managing your AI assistant.

Together, Ollama and Open WebUI offer a complete offline AI solution for anyone looking to use large models without relying on the internet or cloud-based services.

Choosing the Right AI Model

LLMs (Large Language Models) are advanced AI models capable of understanding and generating human-like text. Selecting the right model ensures optimal performance based on your hardware capabilities and the type of tasks you want to perform. Larger models provide more accurate results but require more system resources like RAM and storage.

Small Models (1B–7B parameters): Best for basic chat, instructions, knowledge retrieval, and summarization. Ideal for low-resource systems. (e.g., Gemma3 1B, Llama 3.2 1B, TinyLlama 1.1B)
Medium Models (13B–30B parameters): Balanced for multimodal, general conversations and creative writing. These require more RAM and processing power. (e.g., DeepSeek-R1 14B, Gemma 3 27B, Llama2 13B)
Large Models (65B+ parameters): Best for complex tasks and deep contextual understanding but require high-end hardware. (e.g., Llama3.3 70B, DeepSeek-R1 70B).

For most users, small to medium models offer the best balance between performance and resource efficiency. If your system has limited resources, 1B models are the best choice, as they typically require under 1GB of storage and minimal computational power.

Getting Started

Installing Ollama

Follow these steps to install Ollama on your system:

Step 1: Download Ollama

To download Ollama, click the link below for your operating system. This guide will focus on Windows installation: 🔗 Download Ollama

Step 2: Install Ollama on Windows

After downloading, locate the OllamaSetup.exe file in your download folder.
Double-click the file and follow the on-screen installation prompts.

Step 3: Verify the Installation

Once the installation is complete, open a terminal (Command Prompt, PowerShell, or your preferred CLI) and run the following command to confirm that Ollama is installed correctly:

ollama --version

If the installation is successful, you should see the installed version of Ollama displayed in the terminal.

Run Your First Model

Now that you have installed Ollama, it's time to run your first AI model.

Step 1: Download and Run a Model

To test your installation, use the Llama3.2 1B model. This model is small, efficient, and ideal for basic AI tasks.

Open your terminal (Command Prompt or PowerShell).
Run the following command:

ollama run llama3.2:1b

This command downloads and runs the Llama3.2 1B model.

Step 2: Interact with the AI

Once the model loads, you can start typing messages. The AI will respond in real-time. Try asking a question or giving it a prompt, such as:

What is the capital of France?

The AI should respond in real-time.

Step 3: Verify Successful Execution

If the AI responds correctly, you’ve successfully set up your first AI model! Now you can explore other models and integrate them into your workflow.

For more models, visit the Ollama Model Library.

Install and Set Up Open WebUI

Now that you’ve run your first AI model, the next step is to install Open WebUI for a more user-friendly interface.

Step 1: Install Docker Desktop

Open WebUI runs inside a Docker container. Docker is a tool that helps run applications in isolated environments, making it easier to manage and deploy them. Here’s how to install Docker Desktop:

Download Docker Desktop from the official website: Download Docker Desktop
Locate the downloaded Docker Desktop Installer file in your Downloads folder.
Double-click the file and follow the installation prompts.
Once installed, restart your computer to apply changes.

Step 2: Deploy Open WebUI using Docker

Open your terminal (Command Prompt or PowerShell) and run the following docker command to deploy ollama-webui docker container on your local machine.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

This command:

Creates a container named "open-webui"
Maps port 3000 on your computer to port 8080 in the container
Sets up the container to connect to Ollama running on your host computer
Creates a persistent volume to store your conversations and settings
Configures the container to restart automatically when your computer restarts

Step 3: Access open WebUI interface

Once Open WebUI is running, open your web browser and go to:

http://localhost:3000

The first time you visit, click the Sign Up link to register for an account.

Troubleshooting Ollama

If you encounter issues running Ollama, try the following:

Check if Ollama is running: Open your browser and visit http://127.0.0.1:11434/. If Ollama is running, you should see a message like Ollama is running displayed in your browser. This confirms that Ollama is active and ready to use.

List installed models: Run the following command in your terminal to check if the models are installed correctly:

ollama list

This command will display a list of all the models installed on your system. If no models are listed, it means you need to download the models first using the ollama run <model_name> command.

Restart Ollama: If the models don’t load properly, restart your machine and try again.

Optimizing for Low-Resource Environments: If you're working with limited hardware:

Use the smallest models possible (1B-2B parameter models)
Close other applications when using your AI assistant
Restart Ollama periodically to free up memory

Conclusion

In this guide, we covered how to set up an offline AI assistant using Ollama and Open WebUI. You have successfully:

Run LLMs locally without an internet connection.
Enjoy a privacy-focused, cost-free AI assistant.

With this setup, you now have a powerful AI assistant that works completely offline, ensuring privacy, cost savings, and reliability in any environment.

Next Steps

Explore more LLMs from the Ollama Model Library.
Customize Open WebUI to fit your needs (check out the WebUI documentation).
Consider fine-tuning models for specialized tasks (Learn more about fine-tuning).

Set Up Your Own Offline AI Assistant: A Step-by-Step Guide Using Ollama & Open WebUI