### Introduction to Neural Networks

So far, we've considered two examples of artificial intelligence: game-playing and computer vision.

We've also seen the human brain automatically performs tasks that don't break easily into algorithmic rules for a computer. This approach often devolves into a maze of caveats, minutiae, and special cases.

Intelligence, natural or artificial, is flexible. The human brain, unlike a computer, doesn’t follow a prescriptive program to make decisions. If we were able to create a program that works more like the human brain, what would it look like? What could it do? What would its limits be?

# Neural Networks

The brain contains more than $80$ billion cells, called neurons, which influence one another via small pulses of electricity. A particular neuron doesn't talk directly with every other neuron; they are connected into structures that perform specialized functions. These structures are biological neural networks.

Nedivi et al., PLoS Biology 2005

The human brain is too complex to discuss in much detail here. Even today, neuroscientists do not fully understand how it works. We don't need to know all the details to land on its most important characteristic: brains learn from experience.

Each time you learn a new game, a dance, or a mathematical skill, neurons strengthen their lines of communication with some neurons and prune their connections with others. The structure of neural networks evolves as you gain new abilities.

# Neural Networks

Artificial neurons, the basic units of an ANN, behave more or less like biological neurons. An artificial neuron responds to information in signals it receives, and sends out its own signal to other neurons.

The single artificial neuron below has $4$ inputs that can each be in two states: on or off. You can click on an input to change its state.

The response of the neuron is indicated by its color. When the neuron turns completely black, it means it's fully activated by the inputs. When the inputs completely inhibit its activity, it is white. It is partially filled when its output is between the maximum and minimum.

Some of the inputs drive the neuron toward its active state, and others inhibit its activity. Which color inputs drive the neuron toward its active state?

# Neural Networks

The artificial neuron on the previous page responds to different inputs in a predictable fashion, but it has no purpose as far as we can tell.

Now here's a neuron with a real purpose:

You're a volunteer chaperone on a bus full of hungry kids. It's a long drive, so, to preserve your sanity, you program an artificial neuron to control a snack dispenser.

It asks, "What snack would you like?" The two options are a cookie and some celery. Each kid makes a selection and if the neuron is activated, a snack is dispensed.

A selection is made by switching the corresponding input from off to on. What happens if they select both?

# Neural Networks

It might not seem like much, but the neuron in the previous question is replicating an important logical function: $\mathbf{OR}.$ It also kept the bus from descending into madness — any hungry kid received the snack they requested.

We'll explore how neurons are configured to reproduce logical functions in greater depth in the next chapter. But for now there are more pressing matters: the cookies and celery are running low, putting the snack-dispensing artificial neuron on an unsustainable path.

To avoid disaster, you make a slight change to the internal configuration of the neuron, hopeful that its new logic will better ration the remaining supplies.

The snack dispenser now asks the children, "Which of these snacks have you had today?"

Under what conditions will you be given a snack with the new logic? (Note that the connection between the inputs and the neuron are pink now.)

# Neural Networks

A few principles have emerged from our first encounter with an artificial neuron:

1. an artificial neuron follows rules mechanically as it goes from inputs to output, and
2. an artificial neuron can have a different response to the same inputs if we make slight changes in its internal configuration.

We'll spend almost the entire next chapter exploring how single neurons work, but the real power of neurons comes from wiring them together into a network or, as they say, a neural network.

# Neural Networks

What this means in practice is feeding the output of one artificial neuron into the input of another.

By stacking neurons in layers, we create the potential to make increasingly complex predictions... as long as we can find the right connections between the layers.

For example, here's an artificial neural network that can recognize the digits $0$ through $9.$ Its inputs are a $20\times 20$ field of pixels which are fed through two layers of $25$ neurons, after which the network makes a guess in the form of activated neurons in the top layer.

By carefully setting the connections among neurons, we can give it the ability to classify a decent fraction of hand-drawn digits. This is a much more interesting result than the snack dispenser

Try it out for yourself. How does it do? Where does it go wrong?

Note: The images in the database used to train this network are centered with the total height of the digit nearly equal to the full $20$ pixels. This network doesn't automatically rescale and center the digits you draw, so if you want to give this artificial neural network a chance to classify your inputs correctly, you need to center them.

# Neural Networks

As capable as it is, this neural network isn't "thinking" — it's a mathematical waterfall. Inputs cascade through layers of calculation to end up at an output.

Here's the same network, but with the connections between the layers randomized:

If you ask it to predict what digit you've drawn, its predictions are wildly off.

Nobody can tell what these connections need to be ahead of time, and stripped of the correct ones, the ANN's predictions are no better than random guesses.

Finding these connections is the crux of neural networks. As we'll see later, the magic of ANNs is that they can learn them through experience.

# Neural Networks

But how does it learn?

During training, the artificial neural network (ANN, from this point on) is fed examples of digits. For each input, the ANN makes its best guess about what digit is represented, and its guess is compared to the correct answer (the identity of the digit it was shown).

If its guess is correct, then nothing happens.

But if its guess is incorrect, then the computational machinery within the ANN is updated so that it’s more likely to be correct the next time.

# Neural Networks

To get a sense for how ANNs use guesses to learn, let's play a game many of us are familiar with, called "Where The Hell Are My Keys?"

You thought they were in your pocket, but as you go to lock your door, you realize you have no idea — they could be anywhere in your apartment — so you start checking every shelf, jacket, drawer, and cereal box to find them. Because this is such a joyous experience, we've created a simulation of it for you below.

Each gray tile represents a place in your apartment where your keys could be hiding. To make a guess, click a tile. If the key is there, it will show itself, and if it's not, the tile will turn gray.

If there are $64$ places in your apartment, about how long will it take to find your keys?

# Neural Networks

On the last page, you searched for a key using trial and error. This can take a long time, especially if the space of possibilities is large. Guessing wrong on one tile gives you no information on how to adjust your future guesses.

However, a simple change makes the search a lot faster: feedback.

So, one day you rig your key with a transmitter that tells your phone when you're getting closer. When you lose your key, you use the feedback on your phone to figure out where you left your keys that day.

Below, you can look for your keys again. But this time, we've colored the spaces under the tiles. If the tile turns a shade of blue, you are far from the key. And as you get closer, the tiles get redder.

# Neural Networks

With meaningful feedback to tell you how wrong you are, you can improve your guesses with time.

Although "Where The Hell Are My Keys?" is an extremely simple game, it contains two features of learning that take place in biological brains and ANNs:

• Adjusting wrong guesses: Learning requires trying different actions to complete a task. Trying the same wrong action doesn't lead to learning.
• Feedback: Learning is faster when feedback not only tells you when you're wrong but also how wrong you are.

# Neural Networks

It's amazing how much improvement can be made with feedback.

A decent strategy can find the keys in about 10 guesses. But, with some ingenuity, you should be able to get down to 6 or even less.

What's the lowest average you can achieve?

# Neural Networks

If you play Key Search long enough, you may notice that it feels less like you're thinking and more like you're matching patterns.

Despite having no concept of deductive reasoning, ANNs can do things like finding cats in pictures purely through pattern matching. This has brought enormous power to tasks like diagnosing tumors, playing defined games like chess, and driving cars.

There's a lot more to say about how ANNs work, and in the rest of the course we'll dive into the details. Each chapter will answer a different question related to ANNs:

• Chapter 2: How does an artificial neuron process data?
• Chapter 3: How can we design ANNs to perform arbitrarily complex tasks?
• Chapter 4: How do ANNs learn?

Along the way, you'll be able to interact with artificial neurons and ANNs as you start with the basics up and climb toward modern concepts, such as convolutional nets and backpropagation. All of the computation is wrapped up in our interactive ANNs, so you don't get tangled in technicalities.

Put the neural networks inside your skull to good use, and let's get started!

# Neural Networks

×