Where is the magic? What exactly is a neural network and how does it work?

The futuristic sounding ‘multi-layer perceptron’ is a term sometimes used for a neural network, or artificial neural network to be more precise. They are everywhere today, eager to tell you which turn to take at the next junction, keen to confirm your face is indeed your face when you log in to your bank account, and duty bound in suggesting which Netflix series you should enjoy next.

A neural network is a mathematical equation that uses a visual representation. Imagine a single neurone as a node or simple unit, often represented as a small circle in schematic diagrams. The neurone takes a number as an input, and performs a relatively simple calculation on that number to deliver another number as an output.

Neural networks

To construct a neural network, a number of these neurones are brought together and wired together sequentially. So the output of a neurone becomes the input for the next neurone. This neurone then performs its own little computation on that input, and then its output becomes the input for the next neurone. And so on, until a final neurone or group of neurones delivers our answer.

The real magic here is that nobody ever told the neural network to look for those features. It just figured it out.

The neurones are grouped into layers. A group of neurones representing the input is called the input layer, and a group that will output the answer for our task is called the output layer. In between is a number of groups called hidden layers. The input layer neurones pass their values to all the neurones in the first hidden layer, which all perform their simple computation. And then all these outputs are passed to each of the neurones in the next layer. And so on until we reach the output layer.

All connections are assigned a weight, which is a numeric value for the ‘strength’ of the connection to a neurone.

But how does this network learn?

Assume we have a neural network that we want to use to visually identify a car. If our image is 1,000 x 1,000 pixels, then we have 1,000,000 pixel intensity values that define our image. Each of these values becomes a neurone, and all of them together constitute the input layer.

The first hidden layer will begin with randomised weights and compute a weighted sum for each neurone, and then apply a simple non-linearity function referred to as an activation function. And the same happens for each layer, until we reach our output layer of one neurone, which tells us the probability that a car is identified.

The network learns by modifying the weights and a few other parameters via a larger number of iterative passes through the network. During this training process the system is fed thousands of different images, both with and without cars. These images will be labelled as car or not a car. So the neural network is told the answer for thousands of training examples, and during this training phase its job is to tune the weights to fit the answers.

So it knows the answers; it just has to keep iterating and fit all the weights to get the training examples identified correctly. That done, then the neural network is trained and we can provide images it’s not seen before. It will then happily identify whether there’s a car or not.

Hidden layers

The magic happens in the hidden layers. A neural network learns to identify which features it decides are important for the task. In our example, the first hidden layer will optimise its weights to identify short segments of shapes in the images. The next layer or so will learn to identify groups of these short segments that organise together to form doors, windows, and wheels. And the next layers will specialise to identify more composite forms of these, as they group to represent whole vehicles.

The real magic here is that nobody ever told the neural network to look for those features. It just figured it out. If we asked the same network to identify cats from dogs, then it would start to learn about the shapes of pointed ears and fluffy tails. The neural network just learns these things from scratch as it is trained.

On top of this, nobody fully understands how all of this works. We just know that it can and does work remarkably well. The so-called large language models (ChatGPT, Bard, etc) are beginning to use trillions of parameters. Once these networks are trained, at this scale, it’s becoming increasingly impractical to look inside to see what’s going on. The exponential growth in this ‘black box’ intelligence is fascinating, if somewhat daunting.

Dr James Henderson is a freelance writer based in the UK.