A Beginner’s Guide to AI Jargons

AI models like Chat GPT, Google Gemini , Anthropic Claude and other LLMs are just neural networks predicting next token or word based on some data they have been trained on . One of its type is Transformers.

Transformer →

Transformer is a neural network architecture that has fundamentally changed the approach to Artificial Intelligence. Transformer was first introduced in the seminal paper Attention is All You Need in 2017 and has since become the go-to architecture for deep learning models, powering text-generative models like OpenAI's GPT, Meta's Llama, and Google's Gemini.

Transformer Architecture :

Transformer is a special type of neural network architecture that uses attention mechanism to pay attention to many parts of a text at once, rather than just reading from left to right.

To know more about this : Check this out : Transformer Explainer

Now Let’s deep dive into Transformer working !

Tokenisation :

Just like all of us understand languages such as Hindi, English etc. Transformers and other AI models understand numbers, which are called vectors. Tokenization is simply the process of converting our language into a form that AI can understand — it's as simple as that.

Tokenization is the process of turning text into a sequence of tokens, which can be words, subwords, or characters.. LLM can understand hindi, english, hinglish in reality what happen all ai model have there own tokenization mechanism like (gemini,gpt-4o,deepseek)

For example:

gpt-4o has approx 200k tokens

import tiktoken
encoder  =  tiktoken.encoding_for_model('gpt-4o')
print('Vocab size: ',encoder.n_vocab)

Here vocab size is the total number of words that model understands .

To visualise this check this out → Tiktokenizer (Enter text and select model then visualise)

Two terminologies under this :

• Encoder → This is used to convert text to meaningful numbers(vectors) so that a model can understand this.

    text  = 'The cat sat on the mat'
    print('Tokens: ', encoder.encode(text)) 
    # output -> Tokens:  [976, 9059, 10139, 402, 290, 2450]
  

  Decoder → This is used to get meaningful text from vectors.

    myTokens = [976, 9059, 10139, 402, 290, 2450]
    print('Tokens: ', encoder.decode(myTokens))
    # output -> Tokens:  The cat sat on the mat
  

Vector Embeddings →

This is used to give semantic meaning to an word also Embeddings are like mapping words to meaningful coordinates.

Let’s understand this with an example .

If I say King → queen

father → mother

boyfriend → ??

Your answer might be girlfriend . Because our brain makes some connections or relations.

You can also understand mathematically.

from openai import OpenAI
from dotenv import load_dotenv
import os 
client = OpenAI(
    api_key=os.getenv("GEMINI_API_KEY") ,
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)
text = "You are understanding navneet's blog"
response = client.embeddings.create(
    input=text,
    model="gemini-embedding-exp-03-07"
)
print("Vector embeddings: ", response.data[0].embedding)

If you want to visualize, you can use this link: https://projector.tensorflow.org/

Till now we have understand split text to words converting of text to numbers and giving text a semantic meaning. But now we have a “ Big problem ” .

Let’s say I have (The cat sat on the mat ) and (The mat sat on the cat ) both of them have same tokens also (The river bank) and (The ICICI bank)

Now this is the place where transformers shine - they use mechanism like “ Positional encoding “ and

“ Self attention ” . Let’s understand them one by one .

Positional Encoding →

Transformers don’t have memory of order by default . They see the entire sentence at once — as a set.

Positional Encoding provides the information about the order of words in a sentence ensuring that the model understands the sequence and maintains the correct context.

Token, embedding batata hai "ye kya word hai"

But positional embedding batata hai "ye word kahan aaya hai"

And transformer ko dono chahiye — "kya" + "kahan" — tabhi toh context samjhega!

Self Attention →

Each token is chatting with all other tokens — asking, “Are you relevant to me?”

In self-attention, tokens interact with each other and update the meaning of each token based on the surrounding words so that it can change its meaning according to context .

e.x → “ Riya slapped Tina because she was angry” .

Now who was angry Riya or tina. Now Self attention comes into picture

• 🫣 Glances at “slapped” — definitely some heat involved

• 👀 Focuses on “Riya” — 80% chance it was her.

Multi-head attention → If self attention wan’t enough then it comes into picture focusing on different different aspects of tokens .

Softmax →

Softmax is a function (used in machine learning) to convert the output of a model into probabilities . It helps the model decide how confident it is about each possible answer. It is bascically playing

For example: if INPUT: How are you ?

Possible OUTPUTS:

Temperature - if you use AI Model API Keys, you must be familiar with this term.

if we inc. Randomness/Temprature → Creativeness will decrease . You can try in Gemini studio . What is difference in Outputs after changing this.

Knowledge Cutoff :-

Knowledge Cutoff is the last date an AI model was trained on real-world data.

If you ask gpt or some other model what is current weather it might not have the answers to those type of questions .

But Try this in chatgpt it will tell you . Why??

Because It is gpt+ agent which uses real-time tools like web search or specific plugins. Traditional models can’t tell this .

Ok ji Aaj ke liye itna hi. Milte hain agle blog mein, tab tak ke liye khayal rakhein, muskurate rahein!!!!! Agar blog acha lage to like share kare .

LinkedIn: https://www.linkedin.com/in/navneetsharmame/

Twitter: https://x.com/navnneeet

Github : https://github.com/Navneet0094