tokens and tokenisation
DHIRENDER CHOUDHARY1 min read
Jab main NLP (Natural Language Processing) shuru kiya tha, har jagah ek word sunai deta tha — Tokenization.
Pehle laga koi heavy technical cheez hogi, par asal me kaafi simple hai.
Tokenization ka matlab hota hai text ko chhote tukdo me todna — jinhe hum tokens bolte hain.
Example: text = I LOVE JAIPUR
tokens = I,LOVE,JAIPUR
AI ko text samajhne me easy hota hai
Search, translation, chatbot sab me use hota hai
0
Subscribe to my newsletter
Read articles from DHIRENDER CHOUDHARY directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by