The word Generative AI means to generate something using of AI. It refers to an algorithm that can create new content based on its pre trained data.

Its key technologies are Transformer and LLMs.

Transformers (engine of LLMS)

Transformers model is made by google researchers in “Attention is all you need” paper which is currently available on google. This is made for the purpose of Google Translate. But now it is basic foundations of open ai and GPT.

Working of Transformers

Google translate uses transformer to get the output of given input in another language. It uses self attention to relate each word with every other word.

GPT transformer take input , tokenizes it , use masked self attention to look at only past token(word) it predict next token append it and repeats the process till get the answer.

Token

Token is a piece of text such as a word , space , symbols etc. that GPT can understand and process

LLMs

LLMs are AI models trained on huge amounts of data to understand and generate human language.

They are built using Transformer.

Trained on billions of words.

Open-Ai and GPT

Open ai is a company which creates GPT.

GPT is a technology .It is specific series of LLMs. It stands for Generative pre-trained transformers

Generative → It can generate new content .

Pre-trained → It's trained on a huge dataset.

Transformer→ It uses the Transformer architecture .

Steps In LLMs

Tokenization

It is a process in which inputs are broken into tokens.

And those tokens are then converted into there specific ids .In open ai model for tokenization Tiktoken are used.

Tokenizing in python using GPT-4 tokenizer

pip install tiktoken

import tiktoken
tokenizer = tiktoken.get_encoding("cl100k_base") #gpt-4 tokenizer
text = "My name is sanskar"
tokens = tokenizer.encode(text)
print (tokens)
#output
[5159, 836, 374, 15878, 29234]

Vector Embeddings

In simple words vector embedding is large 3d space where the relation between objects are defined.

Lets take an example of real world (plane - fuel - petrol ) , (train -fuel - coal)

in this case the distance between plane and petrol is same as train and coal in 3d space and the distance between plane and train is same as petrol and coal. in this way relations are defined in vector embedding. Through this we can compare text by meaning not by exact word.

pip install google-generativeai

import google.generativeai as genai

genai.configure(api_key="Your genai api key")

response = genai.embed_content(
    model="models/embedding-001",
    content="cat drinks milk"
)

print(response['embedding'][:10]) #print first 10 values 
#output 
[0.0008019432, 0.006092885, -0.046081897, -0.06915646, -0.01316108, -0.013176519, -0.01969769, -0.04082793, 0.01923299, 0.03155357]

Positional Encoding

Positional encoding refers to the transfer of positions of words to understand its sequence

Example-

→ man loves dog.

→ dog loves man.

Both sentences have same words but huge difference in their meaning .This is the reason behind positional encoding to relate its original meaning .

Self Attention & Multi-Head Attention

Self attention means to look at the other words of the same sentence to understand the context .

Example

→ The river Bank , Here self attention refers to river to understand the context of Bank.

→the ICICI bank , here similar with ICICI.

In Multi-Head attention multiple self attention pointers runs on different direction to understand the context of sentence .Here each pointer runs parallelly and helps to understand context more clearly.

Output

Prediction of next word(token).