Artificial Neural Network: The Neuron

I am by no means an expert in machine learning or artificial intelligence, but just an avid learner and curious individual. Please use the information provided here as a supplement to further your knowledge and research.

I am currently building an artificial neural network from scratch (more specifically a multi-layer perceptron neural network. To help clear my head and make sure I understand everything correctly (or still remember from my machine learning subjects at university) I will be writing several blog posts about machine learning and artificial neural networks. To start the list of blog posts off I am going to write about one of the fundamental building blocks of the neural network, the neuron.

I will be skipping the background about the neuron and how it relates to the biological neuron. Instead I will cover how it is constructed and used in an artificial neural network.

Inputs

The neuron will process inputs from the previous artificial neural network layer. The previous layer could be:

  1. The input layer of the artificial neural network. In this case then the values are taken directly from the training data, test data, or user input when using the trained artificial neural network.
  2. A hidden layer of the artificial neural network. In this case the values would be outputs from other neurons.

Note: There is a special input which is called the “bias” which always has a value of +1. The role and purpose of the bias is so that there is a trainable constant and allows the activation function (will talk about that a little later) to give us the best possible chance of an optimal model. You can read some more about the bias purpose and use here and here.

Weights

For every input to the neuron there is an associated weight. Initially the weights are generally given a random value between the range 0.1 to 0.9. As the artificial neural network is trained the weights are adjusted as to better match the desired output.

Operations

Summation

The first operation to obtain the neuron output is to perform a summation of the multiplication of the associated weight and input. Mathematically this is represented as follows:

 

Artificial Neural Network Summation

 

where:

  • y is the output.
  • n is the number of inputs to the neuron.
  • i is the index of the input to the neuron.
  • w is a weight.
  • x is an input.

You may be asking, Chris why do we have the zero indexed weight and input outside the actual summation when you just add it anyway? Good question reader. Generally the zero indexed input is the bias which then makes the associate weight the zero indexed weight.

Activation

After the summation operation is performed there needs to be an activation function used. The use of the activation function allows for the enhancement or simplification of the neural network. Generally the activation function is non-linear and there are a variety of functions that can be used. Two of the most common activation functions are either sigmoid or rectifier functions. More information about the sigmoid function can be found here. More information about the rectifier function can be found here. The final value of the activation is the output of the neuron.

The next blog post would most likely be about the various layers of the neural network and how they are all interconnected. So stay tuned.

Learning Patience From Teaching

I would consider myself a fairly calm, level headed and patient individual; never can I recall a time where I have gone off the deep end and blown a gasket. There is nothing beneficial in becoming angry, frustrated and furious at an individual or team, or questioning whether someone has really been paying attention in a demeaning fashion. I have seen first hand what happens when someone does blow up and starts angrily blaming and non-constructively criticize someone. As I worked with a variety of different individuals from a number different backgrounds, beliefs, understanding and competence, I have learnt to be patient and try to understand why and where they are coming from.

The reason why I am writing this blog post is because this weekend I was asked by my parents to construct an specific Microsoft Excel spreadsheet that can be easily modified aka “Idiot Proof” as they put it. I am not an Excel wizard or master, but I do know my way around a spreadsheet and I can do what was necessary in Excel for them. As I finished the spreadsheet and was showing them, they seemed fairly happy with the result which was good. But as I was showing them how to update and use it to meet their needs, that is when my patience was being tested. What I originally thought was going to be several minutes explaining how to use the spreadsheet turned into an hour worth of explaining.

My dad is an electrical engineer and as far back as I can remember he has used a computer. He knows how to build them and use a number of various software programs effectively. But his knowledge is limited when it comes to the Microsoft Office suite other than how to send an email in Outlook. My mum on the other hand is what you would call  unfortunately, computer illiterate. She knows the very (and I mean very) basic computer functions. Browsing the Internet (safely), and updating the computer is all she really knows how to do. Anything else would require me to either write it down in simple easy to understand steps or show her continuously until it becomes second nature. She has not used computers as much as my dad or me, so I don’t expect her computer literacy level to be as high as either of us, and that is perfectly fine. She is more than willing to learn which is fantastic.

Back to the Excel spreadsheet. It was constructed in a way that would auto-populate nearly all of the necessary cells with only a couple left to be filled in, and rows in a table would be used to auto-populate a number of other cells. I showed them once how to add a new row to a table in the spreadsheet which allowed the cells to be auto-populated. The actions and process was extremely simple. I laid it out to them in the following way:

  1. Left mouse click in the table cell to select the table.
  2. Right mouse click in the table cell to bring up the pop-up.
  3. Select Insert from the pop-up.
  4. Select Insert Row Below from the pop-up.

I really didn’t think it could get any simpler or easier. However it took them a number of times to understand that you cannot just enter values below the current row and have the necessary cells auto-populated. Knowing that for them this is fairly foreign, I needed to be patient and try to answer their questions as best as possible, in the most simplest way too. If I didn’t have the patience to explain something as simple as adding a new row to a table in Excel then it was a perfect time to learn to be more patient.

After showing them a couple of other necessary actions to perform some other tasks I asked them are there any other follow up questions they had. My parents wanted to see if they could do the whole process from start to finish by themselves with me watching to make sure the right actions were performed. The very first task was adding a new row to the table, and already like an old bad habit they just entered the values below the row and not in the table. I calmly explained to them again that they needed to add a new row to the table otherwise the value will not auto-populate. So I proceeded to show them again how to do that. Again, if I didn’t have patience or was hot headed then I most likely would have said something like “I just showed you how to add a new row to the table like 5 minutes ago, how did you forget? Or were you just not paying attention?”.

As a project technical lead and a software engineer who has been with my current company a number of years, I also provide support to any new software engineers that join our teams. Once again I get to interact with individuals ranging from software engineering interns who are still studying at university to software engineers who have been around the block. With the vast difference in experience that someone has, I need to be able to be more forgiving, understanding and patience with someone who has never rebased a Git branch or resolved merge conflicts before pushing to their origin. I get the opportunity to teach someone something new while also learn to be more patient, understanding and a better leader. It is a win win for everyone 🙂