 Artificial neural networks are computing systems inspired by the biological neural networks of the brain. Such systems can progressively improve their ability to do tasks and recognize patterns by learning from experience. Artificial neural networks are in their essence computational networks that can perform certain specific tasks like clustering, classification or pattern recognition. They do this by representing patterns in data as structures of connections between nodes on the network. They then learn by altering the strength of the connections between the nodes to create new network structures that can represent new patterns. For example, neural nets are now widely used for image recognition, where they learn to identify images that contain, say a house by analyzing example images that have been manually labeled as such and using the results to identify houses in other images. As the name implies, they're directly inspired and modeled on the workings of the brain. Thus to understand neural networks, it is of value to understand a little how the brain works to represent and process information. The brain is composed of neurons and connections between them called axons, which have synapses where the different neurons meet. Neurons generate electrical signals that travel along their axons. Any given axon has a number of inputs from other neurons. If those inputs are above a given threshold, then it is activated and fires. If a neuron is activated, it then sends out signals to other neurons that it's connected to. The synapses change in their chemical composition as one learns in order to create stronger connections between networks of neurons. In such a way, the cognitive system can adapt and change over time to form new patterns of neural networks. If two neurons are turned on when a pattern is stimulated, then the synaptic connection between them becomes stronger. The brain is physically built as a neural network and cognition happens in patterns. Networks of interconnected neurons form a pattern which corresponds to an idea or memory. An artificial neural network, then, is based on this very same architecture as collections of connected nodes that form a network. Each connection between nodes can transmit a signal to another node. The receiving node can process the signal and then signal to downstream nodes connected to it. They're all connected to each other going down through the layers. The nodes mimic neurons in that they're little triggers. Each neuron has something of a threshold where it makes a decision. Nodes take inputs from connected nodes and have some internal function to determine when or if they'll fire to send a signal downstream. The nodes and connections may also have a weight which can increase or decrease the strength of the signal that is sent downstream which itself varies as learning proceeds. Typically, weights represent the strength of the connections between neurons inside the neural network. As such, artificial neural networks can be viewed as weighted, directed graphs in which artificial neurons are nodes and directed edges with weights are connections between neuron outputs and neuron inputs. Typically, neurons are organized in layers. Different layers may form different kinds of transformations on their inputs. Signals travel from the first input to the last output layer, possibly traversing multiple layers in between. The input layer contains those units of artificial neurons which receive inputs from the outside world on which the network will learn or process. The output layer contains units that respond to information that the network has learned to represent or process. Those units between the input and output layers are termed the hidden layers. A neural network with many layers in between is called the deep learning neural network. When there are no layers in between, it's simply a neural network. Because there are very few practical applications for a two-layered neural network, virtually all useful ones are going to be of the deep learning form. As of 2018, neural networks typically have a few thousand to a few million units and millions of connections. These different layers of the network can be used to represent the different levels of abstraction during classification identification, as we'll discuss further in the coming module in deep learning. The key innovation required to actually get a functioning neural network is what we call backpropagation. This involves the network learning through an itchage of process where you send all of the information back through the network and the network adjusts its weights so as to learn to match the output better. The basic idea in backpropagation is quite intuitive. It's simply looking for the difference between the network's actual output and the desired output. The error in the network's output is used to determine how much the network adjusts and changes on the next iteration. So if we put in a graphic of a circle and ask the system to identify it, and it outputs an estimation that it was 0.8 likely to be a circle, we know that there is an error of 0.2. We can then adjust the edges and nodes up and down by a very little amount to see how the error changes. The amount that they're adjusted is determined by how big the error is. With a large error, they're adjusted a lot. With a small error, just a small bit. With no error, they're not adjusted at all. We're doing this alteration to try to go down the gradient to where the error is the minimal. We keep making these adjustments to the weight of the nodes and connections all the way back to the input, and this is why it's called backpropagation, because what we're doing is propagating backwards the error and updating the weights to make the error as small as possible. This backpropagation method is what is at the heart of deep learning, and these days it's used in just about everything you hear about in machine learning. Very early on, people used it to do things like predict the stock market. These days it's used for search, automatically adding color to black and white images, for video recognition, for automatic handwriting generation, for generating captions on images, for speech recognition, for simultaneous translation, social network filtering, playing board games and video games, medical diagnostics and many other applications. The idea of neural networks has been around since the 60s, but at that time computers didn't have enough processing power to effectively handle the work required by large neural networks. Neural network research slowed until computers achieved far-grating processing power. Support vector machines and other much simpler methods such as linear classifiers gradually overtook neural networks in machine learning popularity. It's only very recently that this has changed as available computing power increased through the use of GPUs parallel and distributed computing. Neural networks are now starting to be deployed on a large scale throughout industry. They have found best usage in applications difficult to express with traditional computer algorithms using rule-based programming. Neural networks are the basis of deep learning that has become highly popular in the past years and we'll take a look at this topic in the next module.