Large language models (LLMs) write, translate, summarise and code. Underneath, they do one thing remarkably well: predict the next piece of text. This chapter explains how text becomes numbers, how "attention" lets a model understand context, how raw prediction is shaped into a helpful assistant, and where the limits still are.
By the end of this chapter you can
- Explain tokens, embeddings and why Indian-language text can cost more to process.
- Describe what attention does, using a simple example sentence.
- Walk through the LLM training pipeline: pre-training, instruction tuning and alignment.
- Choose between prompting, retrieval and fine-tuning, and explain why LLMs hallucinate.
Step 1: text becomes tokens
A model cannot read letters. Text is first split into tokens: common words, pieces of words or single characters. "Irrigation" might be one token; a rarer word such as "agroforestry" might become "agro", "forest", "ry". Each token is then given an ID number.
Tokenisers are built from the text they were trained on. If that text was mostly English, words in Hindi, Tamil or Kashmiri are often broken into many more pieces. More tokens means more cost, slower answers and less room in the model's memory. This is one reason India is investing in models trained for Indian languages.
Step 2: tokens become meaning (embeddings)
Each token ID is turned into an embedding: a long list of numbers (often thousands) that places the token as a point in a "meaning space". During training, tokens used in similar ways drift close together. Nobody tells the model that "wheat" and "rice" are related; it discovers it from how the words are used.
Swipe sideways to see the full diagram.
Step 3: attention, the idea that changed everything
Words change meaning with context. In "The farmer sold the crop because it was ripe", the word "it" refers to the crop, not the farmer. In "The farmer sold the crop because he needed money", the pronoun points to the farmer. To understand a sentence, each word needs to look at the others and decide which ones matter.
That is exactly what self-attention does. For every token, the model computes how relevant every other token is, then blends their information together in proportion to that relevance. The 2017 research paper that introduced the Transformer architecture built an entire model from layers of attention, and almost every modern LLM descends from it.
Swipe sideways to see the full diagram.
Attention has a practical advantage too: unlike older recurrent networks that read one word at a time, a Transformer processes all tokens in parallel. That made it possible to train on enormous amounts of text using thousands of GPUs, which is how today's large models became possible.
From next-word predictor to helpful assistant
An LLM is built in stages. Each stage changes what the model is good at, and each one involves human decisions that matter for governance.
Swipe sideways to see the full diagram.
- Pre-training is the expensive part. The model reads a huge collection of text and repeatedly guesses the next token. To get good at this it has to absorb grammar, facts, reasoning patterns, and, unfortunately, the biases and errors present in that text.
- Instruction tuning (supervised fine-tuning) teaches the model to respond to requests in a useful format instead of simply continuing text.
- Alignment uses human (or AI-assisted) feedback: people compare answers and the model is trained to prefer the helpful, honest and harmless ones. A common method is reinforcement learning from human feedback (RLHF).
- Deployment adds a system prompt, safety classifiers, rate limits and monitoring around the model.
How an LLM actually writes an answer
When you ask a question, the model does not look up an answer. It calculates a probability for every possible next token, picks one, adds it to the text, and repeats, token by token. A setting called temperature controls how adventurous that choice is: low temperature picks the most likely token (consistent, good for facts); higher temperature allows less likely tokens (more varied, good for brainstorming).
Everything the model can "see" at once, including your question, the conversation so far and any documents you paste in, must fit inside its context window, measured in tokens. Newer models have large windows, but very long inputs still increase cost and can make the model overlook details in the middle.
Some recent models are trained to "think" before answering: they generate a hidden chain of intermediate reasoning steps, spending more compute on harder problems. This improves performance on maths, coding and planning, but it does not remove the need to check their answers.
Prompting, retrieval or fine-tuning?
When you build a product on top of an LLM, you have three main ways to adapt it to your problem:
| Approach | What it changes | Best when | Watch out for |
|---|---|---|---|
| Prompt engineering | The instructions and examples you send | Quick prototypes; clear tasks | Fragile to wording; limited by context window |
| Retrieval (RAG) | What information the model sees at answer time | Answers must be current, specific or cite sources | Quality depends on your documents and search (Chapter 3) |
| Fine-tuning | The model's weights, using your own examples | A consistent style, format or specialised skill | Needs good data; harder to update; can introduce new bias |
Most strong products start with prompting, add retrieval for knowledge, and only fine-tune when there is a clear, measured need.
The limits you must design around
Hallucination
The model is trained to produce plausible text, not verified truth, so it can state false facts or invent sources with full confidence.
Knowledge cut-off
A model only knows what was in its training data up to a certain date, unless you give it fresh information.
Inherited bias
Stereotypes and gaps in the training text can appear in outputs, especially for under-represented languages and groups.
Prompt injection
Hidden instructions inside a web page, email or document can trick the model into ignoring its rules or leaking data.
Open-weight vs closed models, and why it matters for IndiaGo deeper
Closed models are used only through a company's service; you never see the weights. Open-weight models publish their weights so anyone can download, run and fine-tune them on their own hardware. Open-weight models give more control over data location, cost and customisation, which matters for privacy and public services; closed models are often the most capable and come with managed safety features.
Under the IndiaAI Mission, the government is funding shared GPU compute and supporting Indian teams to build foundation models trained for Indian languages and contexts. For a challenge idea, the right question is not "which model is best?" but "which model can I use responsibly, affordably and in the languages my users speak?"
Key takeaways
- Text is split into tokens and turned into embeddings, numbers that capture meaning by usage.
- Self-attention lets every token weigh every other token; Transformers stack many attention layers and train in parallel at huge scale.
- LLMs are built through pre-training, instruction tuning, alignment and guarded deployment, and each stage involves human choices.
- Start with prompting, add retrieval for facts, fine-tune only with a clear need, and always design around hallucination, bias and prompt injection.