Fundamental Concepts of Neural Nets: Perceptron

Neural Networks are extremely efficient tools for analyzing past and present data. They are made such that they reflect a biological network of neurons at a very fundamental level. However, even this fundamental image of our mind has done wonders with the help of computation resources like high processing power and voluminous storage systems; and with each passing day, we are getting nearer to building closer representations of the mind. The amalgamation of the human mind and external machines has the capability to not only change the world we live in, but also create new ones. Decades or even centuries of research goes into such initiatives, but here also comes the question of mass extinction before the achievement of something so close to a miracle.

Coming back to basics, the study of neural networks is called Deep Learning and it expands into several interesting sub-topics.

The origins of the neural network can be credited to the initial design of the Perceptron in the year 1957 by Frank Rosenblatt. Understanding the functioning of a Perceptron is the most basic step that must be taken to grasp the advanced concepts of deep learning. It is the simplest neural network model which very closely follows the model of Logistic Regression, only with a slight difference in the loss function. In fact, a Perceptron can be treated as a singled layered neural net.

The Intuition of a Perceptron

Image Courtesy: Wikipedia

The Perceptron is a system which takes in a set of inputs (x₁ to x_n) and then applies respective weights on each incoming input (w₁ to w_n). Thereafter, it uses a simple summation function on the weighted inputs and passes the value through an activation function. In simple terms, it is just like a calculator with only three functions:

• -          Multiplication (Weights * Inputs)
• -          Summation (Adding the weighted Inputs)
• -          Activation Function (Could be any depending upon the requirement)

The activation function can be any function which processes the summed value and provides a certain outcome. For instance, a perceptron might use the Rectified Linear Unit or ReLU function. It is one of the most used functions in neural networks due to its simplicity, efficiency and ease of computation. It outputs the input directly if it is positive and otherwise it outputs zero.

The entire model of a perceptron was loosely inspired by a single biological neuron that forms the base of the complex network of the human mind.

To make it simpler, one can compare the perceptron with a logistic regression model. They are essentially the same as already mentioned, except for the loss function.

Y = sigmoid(wx + b)

This is the equation represents a logistic regression model where x is the known variable, w is the assigned weight and b is the constant. The sigmoid function is used to weigh the probability outputs either towards 0 or a 1. This is a complex function due to computations involving inverse of logit functions.

On the other hand, the Perceptron follows the exact equation, but finally, uses a very simple activation function. It is often the ReLU function which helps to speed up any neural network manifold.

Image Courtesy: towardsdatascience

Multilayer Perceptron

Researchers started wondering if it is also possible to replicate the network-like structure of the human brain. Thus, they formulated the idea that if multiple layers of neurons are inserted in a perceptron, it might start giving improved results. The concept of multi-layer perceptron was thereby created.

Image Courtesy: Wikipedia

A tremendous amount of research work went into transforming this basic idea into a success story, and a most celebrated scientist and major believer in the concept, Jeff Hinton, took the lead.

In the year 1986, Jeff Hinton and his team came up with the milestone idea of Back Propagation which turned out to be the backbone of major neural network structures that we use today.

Backpropagation is the simple idea of chain rule of differentiation, but it served as a major founding ground for robust Artificial Intelligence initiatives.

For over twenty years, the research under Jeff Hinton and his team continued to build what most of the world had little hope from. The idea that machines can replicate the human mind and perform miraculous tasks. Even during the AI Winter when the funding for neural network research dried up since there were computational limitations and people moved over to more feasible techniques like SVMs and Gradient Boosting, Hinton kept the lead on ample research work in the field of neural networks.

In 2006, he finally came up with an impactful paper which talked of deep neural networks with a lot of hidden layers. It was eminent that multiple layers improved results significantly (also demonstrated by the ImageNet competition in 2012 where Deep Neural Nets beat other results by a huge margin). This encouraged powerful companies in the technical sector to invest heavily in neural network research since they had the volume of data necessary and sufficient computation resources.

Today, neural networks have taken the field of artificial intelligence to unimaginable heights and there is still so much to do!

[ Watch this space for a detailed discussion on multi-layered perceptron. ]