Unless you’ve been living under a rock, you’ve likely heard about AI & LLMs more than once.

LLM stands for Large Language Model. It's a type of artificial intelligence (AI) model that's designed to process and understand human language at a massive scale. Large Language Models are trained on vast amounts of text data, which enables them to learn patterns, relationships, and structures within language. This training allows the models to generate human-like text, respond to questions, and even engage in conversations.

And to be fully transparent, an LLM (Llama 3.2) wrote that last paragraph. In fact, AI generates the SEO summaries for all of my blog posts, as it’s a feature of the Hashnode blogging platform. Ya, it’s everywhere.

So why not integrate it with our local development platform?

In this post, we’ll walk through the steps to download & install an LLM - Llama 3.2 - to our local machine, fire it up and integrate it with APEX in our Oracle 23ai container.

Installing Ollama & LLM

The first thing that we need to do is download a program called Ollama. Simply put, Ollama is an application that enables users to install and run LLMs on their local machines. It obfuscates all of the hard parts, and in fact, the next few steps will only take a few minutes to complete.

Download & Install Ollama

To download Ollama, follow the steps below:

1. Navigate https://ollama.com/

2. Click Download.

3. Select your operating system and the download will start.

4. Once it’s downloaded, unzip the file and drag the Ollama icon to your main Applications folder.

5. Double-click the Ollama icon to start the installation. You should see the following window:

6. Click Next.

7. Ollama will prompt to install the command portion. You will need to enter an Administrator password to pass this step after you click Install.

8. Next, copy the command ollama run llama3.2 from the window by clicking on the copy icon.

9. Click Finish to dismiss the window.

10. Open a new terminal window.

11. Paste in the command - ollama run llama3.2 - to install the llama3.2 model and hit return. The model will begin to download. Depending on your network speed, this may take a few minutes, as the size of the download is roughly 2GB.

12. From the >>> prompt, enter any question that you want to ask the model. For example: what does llm stand for?

Ollama is now up and running with the Llama 3.2 LLM. This is the same exact model that you may have interacted with online, but it’s running locally on your machine and is completely free!

Feel free to play around. When you want to exit the interactive mode, type /bye and hit return.

Managing Ollama

Since we “ran” a model, Ollama will check to see if it has the latest model, pull it down if not, and then run the model. You can check to see if it’s running with the following command: ollama ps

If we wanted to stop this model, we can use the following command:
ollama stop llama3.2

This will stop the model, but Ollama will technically still be running. To completely quit Ollama, select the Ollama icon in the menubar and select Quit Ollama.

This will stop all models, as well as any of the Ollama background processes.

Ollama as a Service

While this is all well and good, it’s really not much different from interacting with a LLM via a web site. In fact, if we wanted to go that direction, we could install Open WebUI and get a very familiar looking interface.

But that’s not what we’re after. We want to be able to use this LLM in APEX to drive the APEX Assistant as well as the APEX_AI API. To do that, it may make more sense to run the LLM as a service.

1. Open a new terminal window and run the following command: ollama serve

2. Open another new terminal window and copy & paste the following command:

curl --request POST \
  --url http://localhost:11434/api/chat \
  --header 'content-type: application/json' \
  --data '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "who was the first president?" }
  ],
  "stream": false
}'

The results should look similar to the following:

{
  "model": "llama3.2",
  "created_at": "2024-12-05T04:03:00.706654Z",
  "message": {
    "role": "assistant",
    "content": "The first President of the United States was 
                George Washington. He served two terms in office 
                from April 30, 1789, to March 4, 1797."
  },
  "done_reason": "stop",
  "done": true,
  "total_duration": 1633274834,
  "load_duration": 18144875,
  "prompt_eval_count": 31,
  "prompt_eval_duration": 667000000,
  "eval_count": 36,
  "eval_duration": 946000000
}

Note that the content line contains the answer from the LLM. Thus, the Ollama service is up and running.

Given that we want to reach out to this service from a Podman container, we can’t use localhost, as that would translate to the container itself. Thus, we need to refer to the service on the IP address of the local machine - i.e. your local IP address.

To find out your local IP on Mac:

3. Open System Settings > WiFi > Details.

4. From there, either scroll down or select TCP/IP to see the IP address of your local machine. Note this address, as we will need it in the next step.

5. Next, copy the below command, replacing [your IP address] with you local machine’s IP address and run it:

curl --request POST \
  --url http://[your IP address]:11434/api/chat \
  --header 'content-type: application/json' \
  --data '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "who was the first president?" }
  ],
  "stream": false
}'

You should get the same or very similar result as you did before.

If all is well, then we’re ready to hook up the LLM to APEX.

Troubleshooting

In some cases, you may need to run the following command on the Mac, and then stop & start Ollama:

launchctl setenv OLLAMA_HOST "0.0.0.0"

This essentially “unbinds” the Ollama service from 127.0.0.1, allowing you to use your local IP address to access it.

Configuring APEX

Integrating Ollama with APEX is done pretty much the same way as a hosted LLM, with a couple of small differences. Let’s walk through the steps.

Creating an ACE Entry

First things first, we need to create an additional ACE entry to allow the database to talk to the Ollama service.

1. Open a new terminal window.

2. Connect to the database via SQLcl as the SYS user.

3. Ensure that you’re connected to the correct PDB. In this case, it should be freepdb1.

4. Copy the below command, replacing [your IP address] with you local machine’s IP address and run it:

begin
  dbms_network_acl_admin.append_host_ace (
    host       => '[your IP address]',
    lower_port => 11434,
    upper_port => 11434,
    ace        => xs$ace_type(privilege_list => xs$name_list('connect'),
                              principal_name => 'APEX_240100',
                              principal_type => xs_acl.ptype_db));
end;
/

If the IP address of your local machine changes, you will need to create another ACE entry for each new IP address. Thus, it may be easier to use * as the hostname, as that will apply to any and all IP address and/or hostname.

Again, this idea is fine for a local development server, but not for production.

We can quickly test this via SQL Workshop:

5. Login to any APEX workspace (except INTERNAL).

6. Click on SQL Workshop.

7. Click on SQL Commands.

8. Copy the below command, replacing [your IP address] with you local machine’s IP address and run it:

declare
  l_clob clob;
begin

l_clob := apex_web_service.make_rest_request
  (
    p_url         => 'http://[your IP address]:11434/api/chat'
   ,p_http_method => 'POST'
   ,p_body        => '{ "model": "llama3.2", "messages": [ { "role": "user", "content": "who was the first president" }], "stream": false}'
  );

htp.prn(l_clob);

end;
/

If all is working, you should see something similar to the following:

Again, the results may differ slightly, but as long as there’s a JSON document that looks similar to the one above, you’re good to go.

Troubleshooting

If there’s an error here, it almost always has to do with the ACE being improperly configured or the IP address being wrong. Verify that both of these components are correct and try again.

Integrating with App Builder

Integrating our model with App Builder is done pretty much the same way as you would any other hosted LLM. The only real difference is that we’re using our local machine’s IP and the value of the API Key really doesn’t matter, but can’t be null.

Let’s get this thing integrated!

Workspace Integration

Let’s start by adding our LLM to the workspace.

1. Click the App Builder icon.

2. Click the Workspace Utilities icon.

3. Click Generative AI.

4. Click Create.

5. Set or enter the following values, replacing [your IP address] with you local machine’s IP address:

Note: The value of API Key can be anything, as it’s not needed when using local LLMs, but cannot be null.

6. Click Create.

Let’s ensure that it’s working.

7. Navigate to SQL Workshop > SQL Commands.

8. Click on the APEX Assistant button.

9. Since we may not have any tables in our schema, simply type hello and hit enter. You should see a similar response:

While this is less than helpful, it does prove that APEX is able to talk to Ollama.

Application Integration - Chat

Rather than re-invent the wheel, there’s a decent amount of content on the internet about how to build applications with APEX & AI - using chat, generation and the APEX_AI API. Now that we have our local LLM integrated, there’s nothing different that needs to happen when building APEX application that interact with AI - either via a dynamic action or API call.

Here’s a few links to get started:

• Including Generative AI in Applications

• Build AI Powered Apps with Oracle APEX

• How the new APEX AI features work

Conclusion

AI offers an unparalleled set of features that can be used as a platform to build the next generation of application on. APEX provides an ideal place to start to quickly hash out, prototype and build these solutions. The ability to pack the power of AI onto your local machine can accelerate development and save cost at the same time.

Title Photo by Chris on Unsplash