In the Foundations course you learned that deep learning uses "neural networks". This chapter opens the box. You will see what a single artificial neuron computes, how millions of them are stacked into layers, and how a network improves itself one tiny step at a time, and why that same process can quietly go wrong.
By the end of this chapter you can
- Explain what a neuron, a weight, a bias and an activation function do.
- Describe the training loop: forward pass, loss, backpropagation and gradient descent.
- Recognise overfitting and name four ways engineers prevent it.
- Match common architectures (CNN, RNN, Transformer, diffusion) to the problems they solve.
The artificial neuron: a tiny weighted vote
An artificial neuron is much simpler than a brain cell. It takes several numbers as input, multiplies each one by a weight (how important that input is), adds them up together with a bias (a baseline nudge), and passes the total through an activation function that decides how strongly the neuron "fires".
Swipe sideways to see the full diagram.
The activation function matters more than it looks. Without it, stacking neurons would only ever produce a straight-line relationship, however many layers you added. Functions such as ReLU (output the input if positive, otherwise zero) and sigmoid (squash any number into 0–1) add the "bends" that let a network model curved, messy, real-world patterns.
Layers: from pixels to meaning
Neurons are arranged in layers. The input layer receives raw data, one or more hidden layers transform it, and the output layer produces the answer. A network is "deep" when it has many hidden layers.
Each layer learns to detect patterns built from the layer before it. In an image model, early layers respond to edges, middle layers combine edges into shapes such as a leaf outline, and later layers combine shapes into concepts such as "leaf with rust disease". Nobody programs these detectors; they emerge during training. This is called representation learning, and it is the main reason deep learning beat older methods on images, speech and language.
A parameter is simply one learned number: a weight or a bias. More parameters let a model capture more complex patterns, but they also need far more data, compute and energy to train, which is why model size is a cost and sustainability question as well as a technical one.
How a network learns: the training loop
A new network starts with random weights, so its first answers are nonsense. Training improves it by repeating four steps millions of times:
Swipe sideways to see the full diagram.
- Forward pass: a batch of examples flows through the network and it makes predictions.
- Loss: a loss function turns "how wrong" into a single number. For a yes/no question this is often cross-entropy; for predicting a quantity such as crop yield it might be mean squared error.
- Backpropagation: using calculus (the chain rule), the network works backwards from the loss and calculates, for every weight, which direction would reduce the error. This list of directions is called the gradient.
- Gradient descent: each weight moves a small step in the helpful direction. The size of that step is the learning rate: too large and training overshoots and becomes unstable; too small and it takes forever.
Intuition
Imagine standing on a foggy hillside at night and trying to reach the lowest valley. You cannot see the whole landscape, but you can feel which way the ground slopes under your feet. Gradient descent takes a small step downhill, feels the slope again, and repeats. One full pass through the training data is called an epoch; large models train for many epochs on very large datasets.
Overfitting: when a model memorises instead of learning
A model's real job is to perform well on data it has never seen. That ability is called generalisation. To measure it, engineers split data into three parts: a training set (to learn from), a validation set (to tune choices during development) and a test set (kept locked away for a final, honest score).
Swipe sideways to see the full diagram.
A model that scores 99% on training data but 70% on new data has overfitted. It has learned the noise and quirks of its examples, the way a student who memorises last year's answer key fails a new exam. The opposite problem, underfitting, happens when the model is too simple to capture the pattern at all.
Engineers fight overfitting with a toolbox of techniques:
Data & augmentation
Collect more varied examples, or create variations (rotated, cropped, recoloured photos) so the model cannot memorise.
Early stopping
Watch the validation loss and stop training at the point where it is lowest (Figure 1.3).
Dropout
Randomly switch off some neurons during training so the network cannot rely on any single path.
Regularisation
Penalise very large weights (weight decay) so the model prefers simpler explanations.
Architectures: different shapes for different problems
All deep networks share the same learning loop, but their wiring, or architecture, is designed for the structure of the data.
| Architecture | Built for | Example use |
|---|---|---|
| Feed-forward (MLP) | Tables of numbers | Predicting loan repayment from applicant features |
| Convolutional (CNN) | Images and grids; scans small windows for local patterns | Detecting diabetic retinopathy in eye scans |
| Recurrent (RNN, LSTM) | Sequences read one step at a time | Earlier speech recognition and sensor time series |
| Transformer | Sequences processed all at once using attention | Large language models, translation, and now vision too |
| Diffusion model | Generating images, audio or video by removing noise step by step | Text-to-image tools |
Why did "deep" networks only take off after 2010?Go deeper
The core ideas behind neural networks and backpropagation are decades old. Three things changed. First, the internet created huge labelled datasets such as ImageNet. Second, graphics processors (GPUs), designed for video games, turned out to be ideal for the matrix multiplications networks need. Third, practical fixes made deep networks trainable: ReLU activations and residual connections (shortcuts that let the gradient skip layers) solved the vanishing gradient problem, where the learning signal shrank to almost nothing before it reached the early layers.
The lesson for innovators: breakthroughs often come from combining an old idea with new data, new hardware or a small engineering fix.
Governance link
Because a network learns everything from its training data, choosing the data is a governance decision, not just a technical one. If your training photos only show crops from one state, or your speech data only one accent, the model will quietly work worse for everyone else. You will see in Part 2 how frameworks require you to document and test exactly this.
Key takeaways
- A neuron computes a weighted sum plus a bias, then applies an activation function; layers of neurons learn increasingly abstract patterns.
- Training repeats forward pass → loss → backpropagation → gradient descent, guided by the learning rate.
- Generalisation is the real goal: watch validation loss, and use more data, early stopping, dropout and regularisation to prevent overfitting.
- Pick the architecture that fits the data: CNNs for images, Transformers for language and sequences, diffusion for generation.