On its own, a language model only knows what it saw during training and can only produce text. The most useful AI products today wrap the model in a system: they feed it trusted information, let it use tools, and connect it to images and speech. This chapter explains the three building blocks behind that shift: retrieval-augmented generation, AI agents and multimodal AI.
By the end of this chapter you can
- Draw the two halves of a RAG pipeline and name the design choices that decide its quality.
- Explain the agent loop and how tools and protocols connect models to real systems.
- Set sensible levels of autonomy and guardrails for an agent.
- Describe how vision, speech and image-generation models extend what AI can do.
Retrieval-augmented generation (RAG)
Suppose you want an assistant that answers farmers' questions about government schemes. A general LLM might give an outdated or invented answer. With RAG, the system first searches a trusted collection of documents (the latest scheme guidelines), then gives the most relevant passages to the model along with the question, and asks it to answer using those passages and cite them.
Swipe sideways to see the full diagram.
RAG is popular because it is cheaper than retraining, easy to update (replace a document and the next answer changes), and lets users check the source. But a RAG system is only as good as its design choices:
How you split documents
Chunks that are too small lose context; too large and the relevant sentence gets buried. Keep headings and tables intact.
Hybrid search and reranking
Combine meaning-based search with keyword search (great for scheme names and IDs), then rerank the top results.
Citations and refusals
Instruct the model to answer only from retrieved passages, cite them, and say "I don't know" when the answer is not there.
Permissions
Retrieval must respect who is asking. An employee should never receive passages from files they are not allowed to open.
AI agents: from answering to acting
An AI agent is a system in which a model can take actions toward a goal, not just reply. It is given tools (search, a calculator, a database query, a booking system, an email sender) and works in a loop: decide what to do, call a tool, look at the result, and decide again, until the goal is met or it needs to ask a human.
Swipe sideways to see the full diagram.
Models call tools through function calling: the developer describes each tool (its name, purpose and inputs), and the model replies with a structured request such as search_labs(district="Budgam") that the software then runs. Open standards such as the Model Context Protocol (MCP) make it easier to connect one model to many tools and data sources in a consistent, permissioned way.
Agents can also work in teams: a "planner" agent breaks a job into steps and hands them to specialised agents for research, writing or checking. This is powerful, and it multiplies the ways things can go wrong.
How much autonomy is safe?
The more an agent can do without asking, the more value it creates, and the higher the stakes when it errs. Good teams decide the autonomy level per action, based on impact and reversibility:
| Level | What the agent may do | Suitable for |
|---|---|---|
| 0 · Suggest | Recommend; a human does everything | Medical, legal or credit decisions |
| 1 · Act with approval | Prepare the action; a human confirms each one | Sending emails, payments, publishing |
| 2 · Act within limits | Act alone inside strict boundaries (budget, scope, allow-list) | Scheduling, data look-ups, drafting |
| 3 · Fully autonomous | Act and only report afterwards | Low-risk, easily reversible, well-tested tasks |
Agent guardrails checklist
Least privilege (give only the tools and data the task needs) · human approval for irreversible or high-impact actions · sandboxing for code · spending and rate limits · treat retrieved web pages and emails as untrusted (they may contain prompt injections) · full logging of every tool call so actions can be audited and undone.
Multimodal AI: seeing, hearing and speaking
Modern models increasingly work across several kinds of data, or modalities:
Understanding images
Read a photo of a leaf, an X-ray, a form or a meter reading, and answer questions about it.
Voice in, voice out
Speech recognition and text-to-speech let people use AI by talking, in their own language, without typing.
Creating media
Diffusion models start from random noise and remove it step by step to create images, audio or video from a description.
For India, voice is often the key to inclusion. Platforms such as the government's Bhashini initiative provide speech and translation services across Indian languages, so a solution can serve someone who has never used a keyboard. Generative media also raises new governance questions, such as labelling synthetic content and preventing deepfakes, which you will meet in Chapter 7.
Putting it together
A voice assistant for farmers
- A farmer asks a question by voice in Kashmiri. Speech recognition turns it into text.
- RAG retrieves the relevant crop advisory and scheme rules from verified sources.
- An agent checks the local weather and soil-lab availability through tools.
- A vision model analyses a photo of the diseased leaf.
- The answer is spoken back in Kashmiri, with the source named, and any booking waits for the farmer's "yes".
Key takeaways
- RAG grounds a model in trusted, current documents; chunking, hybrid search, citations and permissions decide its quality.
- Agents plan, act through tools, observe and repeat; function calling and open protocols such as MCP connect them to real systems.
- Set autonomy per action by impact and reversibility, and add least privilege, approvals, limits and logging.
- Multimodal AI adds vision, speech and generation, and voice in local languages is a powerful route to inclusion.