Artificial Neural Networks: The biological model

The first computational model of an artificial neuron was proposed by McCulloch & Pitts all the way back in 1943, and it still serves as the basis for almost all modern day artificial neural networks (ANN). It was a purely biological approach to learning, and it strove to mimic the functionality of a biological neuron [1]. Extremely simplified, the biological neuron consists of four main components:

• Dendrite: branched extensions of a nerve cell which receives signals from other neurons in form of chemically induced potentials.
• Soma: the cell body processing the received information.
• Axon: nerve fiber which transmits the output of the soma.
• Synapse: the point of connection to other neurons.

Generally, but somewhat biologically inaccurate, the process for the neurons is that the dendrites receive output from another neuron or stimuli, the soma then processes it and transmits its calculated response to other neurons if it is activated. Of course, this process occurs in billions of neurons in parallel, and potentially in a hierarchical manner. This ensures that there is a division of labor so to say, i.e. that each neuron only activates when its inherent criteria is met; for example an amount of chemical potential being received [2].

This event is called action potential (firing), and explains the chemical process within the membrane of the neuron that causes it to spike in potential (activate). Specifically this occurs when the potential of the membrane rapidly rises and falls. The depolarization causes the spike to travel long the axon, and when reaching a synaptic terminal, it contributes to post-synaptic potentials which excites the neuron (moves it away from resting state) [3]. The below figure illustrates the membrane potential changing over time in the neuron during action potential.

Reconnecting to the computational model proposed by McCulloch & Pitts, the biological neuron explained previously can be simplified to the artificial neuron. Here, the dendrites are represented as input values to the soma, for example a vector of scalar values. The axon represent output values, produced by the soma based on the synaptic inputs. Mathematically, we can define this behavior using two main components:

$$ g = \Sigma_i x_i \cdot w_i $$

and

$$ \hat{y} = \sigma(g) $$

where g is an aggregate of the input values x that represent excitatory and inhibitory post-synaptic potentials, and σ is an activation function applied to the aggregate response and in the biological model often represented by a threshold function (related to action potential).  In the case of the biological neuron, we could view it as a binary classifier; either it is activated due to sufficient excitation, or it remains in a resting state. The mathematical model introduces the weights w that are applied to each input value, and represent how well connected each dendrite is to its corresponding synapse [1]. We represent the output as ŷ and call it the predicted response; because both in the biological brain and in the mathematical models, we want to allow the neuron to adapt whenever it fires inappropriately, i.e. it excites when it should have remained in a resting state.

This mathematical, artificial, neuron can conceptually be reconnected to the biological counterpart by the following figure:

Hopefully this was an interesting read to you and granted you some newfound insight in to the origin of modern day ANNs. In my next blog post I will go over how we can use the artificial neuron to perform binary classification with the so-called perceptron. Stay tuned and take care!

References

[1] Warren S. McCulloch and Walter Pitts. ‘A logical calculus of the
ideas immanent in nervous activity’. In: The bulletin of mathematical
biophysics 5.4 (Dec. 1943), pp. 115–133. issn: 1522-9602. doi: 10.1
007/BF02478259.

[2] Pawel Herman, Erik Fransén. DD2437 Artifical Neural Networks and Deep Architectures: Lecture 1. KTH Royal Institute of Technology (Aug. 2020), pp. 30-38.

[3] Kress GJ, Mennerick S. Action potential initiation and propagation: upstream influences on neurotransmission. Neuroscience. 2009 Jan 12;158(1):211-22. doi: 10.1016/j.neuroscience.2008.03.021. Epub 2008 Mar 19. PMID: 18472347; PMCID: PMC2661755.