Decoding AI

“Your dost and your career ghost”

🤖 Hi, I’m your AI

Don’t be afraid of me i am like your personal Doraemon😼,I'm here to help you, just like Doraemon helps Nobita, i am not a job replacer or a automation threat on this planet. My mission is to: “Zindagi sawar doon Ik nayi bahar doon Duniya hi badal du main To pyara sa chamatkaar hoon 🌬️“ (Make your life better, bring a new spring, change the world, and be a lovely miracle).
But, fair warning, I might take your job if you don’t practice what you learn 😤.

So in this blog we will learn how a cute blue raccoon cat (yes, that’s me!) understand what you type in my textbox and how i generate you output. So, let’s dive in!

➡️ This is me, this is what happens internally in me when you give me Prompt

A complex process unfolds behind the scenes when you provide me with a prompt. Here, I break down the behind-the-scenes (BTS) magic and explain what happens internally:

1. INPUT PROCESSING (TOKENIZATION)

   😸 Your prompt is broken down into smaller units called tokens. These can be words, letters, punctuation marks, or other elements.

   😸 Every AI (LLMs like ChatGPT, Grok, DeepSeek, Claude etc) uses a tokenizer to convert your text into numbers (or you can say assigns a unique ID to each token).

   😽 Example:

   

2. VECTOR EMBEDDING (Contextual Understanding)

   😸 After tokenization, each tokenized unit is converted into embeddings.

   😸 Embeddings are numerical vectors (list of numbers) that captures the meaning and context of each token.

   😸 Embeddings helps AI to understand relationship between words and their context in the Prompt.

   😽 Example: “Doraemon” and “Dorami” share a semantic relationship and have similar embeddings.

   ➡️ Semantic Relationship:- “A semantic relationship is the connection between words, phrases, or concepts based on their meaning”.

   😽 Examples:
   🩵🩵 "Big" and "large"(Similar Meaning)
   🩵🩵 "Hot" and "cold" (Opposite Meaning)
   🩵🩵 "Wheel" and "car" (A part-whole relationship)
   🩵🩵 "Bank" as a financial institution or river edge (context-dependent meaning)

   ➡️Positional Encoding: This is a method to help me understand the order of words in a sentence. Since I process all words simultaneously, I don’t naturally know which word comes first, second, or later. Positional encoding assigns each word a special “position tag” to indicate its place in the sequence.

   😽 Example: “The dog chased the cat” versus “The cat chased the dog” have different meanings due to word order, which positional encoding helps me distinguish.

3. NEURAL NETWORK PROCESSING (Transformer Magic)

   😸 After embedding process done, embeddings will go into my body known as transformer architecture, the core of modern language. Here i “thinks” about your prompt.

   ➡️➡️ How my core architecture works internally?? Lets see:-)

   ⚙️ Attention Mechanism: I am using self-attention mechanism to understand the importance of each token in relation to others. Here i allowed every token to talk to each other to understand and learn each others meaning in respect to their context.

   ⚙️ Layers of refinement: The input passes through many layers and each layer refines your prompt to its best possible outcome.

   ⚙️ Context Window: Its like my memory which i use to understand and respond to you. It’s the amount of text (words, sentences, etc.) I can "look at" at one time to keep track of our conversation or process what you send me.

4. GENERATING A RESPONSE

   😸 After processing the prompt, i predicts the next best tokens to form a response.

   ➡️➡️ How i predicts the best tokens(i am not a magician or god)???

   ⚙️ I have a massive dictionary and i used it for the next token.

   ⚙️ I select a token and repeats the process to build the response token by token.

   ⚙️ The output tokens are converted back into human-readable text.

   🤖 My Limitations

   🥞 No True Understanding: I don’t “understand” like humans do; I generate responses based on patterns.

   🥞 Context Limits: My context window is large but finite, so very long conversations might lose earlier details.

   🥞 Bias and Errors: My responses reflect patterns in my training data, which can occasionally lead to inaccuracies or biases.