Many people think AI is magic. But actually it's not!

AI is like a very smart word guesser. It reads your words and tries to guess what word should come next. That's it!

But how does it get so good at guessing? The answer is lots and lots of training and practice.

The Beginning - Google Translate

The story starts with Google. Google wanted to make their translate app better.

In 2017, Google's team made something called a "Transformer." [their research paper] This was like a new brain for computers.

This new brain was much better at understanding language. Other companies saw this and said "Wow! We want to use this too!"

From that small start, we now have ChatGPT, Gemini, and many other AI helpers.

Ofcouse, it didn’t happened overnight. but you get the idea, right?

Instead of going more deep into, how google build that Transformer, we will only understand the surface level part, which we actually need.

Step 1: Breaking Words into Pieces (Tokens)

When you type "Hello, how are you?" the computer doesn't see it like you do.

First, it breaks your sentence into small pieces called "tokens."

Think of tokens like puzzle pieces. Each piece might be:

A whole word like "hello"
Part of a word like "ing"
A space or comma

Different AI models cut up sentences in different ways. Some make big pieces. Some make tiny pieces.

Step 2: Turning Tokens into Numbers (Vector Embeddings)

Computers love numbers. They hate words.

So we need to turn each token into numbers. We call these number lists "vector embeddings." In a very layman terms, its an array of numbers. [0.2, 1.3, …..].

But why do we do this? Because we need to teach the computer what words mean and how they relate to each other in a real world.

The Problem: How Do You Explain Word Meanings?

Think about these examples:

Krishna and Radha are related (they are divine lovers)
Krishna and Mahadev are related (they are both gods)
A king and a queen are related (they both rule)

How do you tell a computer that these words are connected?

Smart computer scientists had a clever idea.

They created a special space with many directions. Not just up-down, left-right, forward-back. But hundreds or thousands of directions!

In this space, each word gets a special arrow pointing from the center. We call this arrow a "vector."

Words that mean similar things get arrows pointing in similar directions.

Context Changes Everything

Here's where it gets tricky. Some words mean different things in different sentences:

"I went to the bank" (money bank)
"I sat by the river bank" (land near water)

The word "bank" is the same. But the meaning is totally different!

The computer needs to understand this. So it looks at all the other words around "bank" to figure out which meaning is right. so, now we also understand that, more the library of words (tokens) a model have, more efficiently its able to generate or predict next words.

More Dimensions = Smarter Understanding

Remember that special space we talked about? The number of directions in that space matters a lot.

100 directions = computer understands a little
1,000 directions = computer understands better
10,000 directions = computer understands much better

More directions mean the computer can store more details about each word.

The Two Big Steps

So here's what happens when you talk to an AI:

Step 1: Your sentence → Tokens (sentences break down to words)

Step 2: Tokens → Vector embeddings (number arrows in n-dimentional space)

Different AI models do these steps differently. Some are faster. Some are more accurate. but sequence is always going to be same.

Choosing the Right Model

There are many llm models to pick from:

Some are small and fast
Some are big and smart
Some are good at one language
Some know many languages

The model you choose decides:

How your words get cut into tokens
How many directions are in the special space
How well the AI understands context

The Magic is Really Math

At the end of the day, AI is just very good engineering and mathematics.

It takes your words, turns them into numbers, does lots of calculations, and guesses the best next word.

The "magic" comes from:

Huge amounts of training data
Very powerful computers
Years of smart people making it better

But remember - it's still just predicting the next word. A very, very good word predictor! What's Next?

AI will keep getting better. Due to increase usage of AI models, this learning speed is also increased quite significantly.

Scientists are working on:

Faster models
Smarter models
Models that use less power
Models that understand images and sounds too

The future is exciting! But now you know the secret - it's all about turning words into numbers and making really good guesses.

AI is Not Magic | How LLMs Works - A Simple Guide

Table of contents