Appedix D: Artificial Neural Network

Artificial neural network (ANN) is a computational tool inspired by the network of

neurons in biological nervous system. It is a network consisting of arrays of artificial
neurons linked together with different weights of connection. The states of the neurons
as well as the weights of connections among them evolve according to certain learning
rules. Practically speaking, neural networks are nonlinear statistical modeling tools
which can be used to find the relationship between input and output or to find patterns in
vast database. ANN has been applied in statistical model development, adaptive control
system, pattern recognition in data mining, and decision making under uncertainty.

D.1 Classical Hebb’s Rule

Hebb’s rule is a postulate proposed by Donald Hebb in 1949 [1]. It is a learning rule that
describes how the neuronal activities influence the connection between neurons, i.e., the
synaptic plasticity. It provides an algorithm to update weight of neuronal connection
within neural network. Hebb’s rule provides a simplistic physiology-based model to
mimic the activity dependent features of synaptic plasticity and has been widely used in
the area of artificial neural network. Different versions of the rule have been proposed to
make the updating rule more realistic.

The weight of connection between neurons is a function of the neuronal activity. The
classical Hebb’s rule indicates “neurons that fire together, wire together”. In the simplest
form of Hebb’s rule, Eq. (A.1), wij stands for the weight of the connection from neuron j
to neuron i

wij = xi x j (A.1)

A B

1
 Figure D.1-1 The plasticity within neural network. (A) The single connection between
neuron i and neuron j. (B) A network of neurons connecting to neuron i.

The perceptron is type of artificial neural network. It can be seen as the simple
feedforward network acting as the binary classifier.

D.2 Hopfield Model

Hopfield model is an associative memory model using the Hebb’s rule for all possible
pairs ij with binary units. The state variable xi of the neuron i takes on either on of the
two possible values: 1 or -1, which corresponds to the firing state or not firing state,
respectively.

⎛ ⎞
Si (t + 1) := sgn ⎜ ∑ wij S j (t ) − μi ⎟ (A.2)
⎝ j ⎠

Where sgn is the sign function defined as follows:

⎧ 1 if x ≥ 0
sgn( x) = ⎨ (A.3)
⎩-1 if x < 0

A B

Figure D.2-2 The schematic diagram of neural network. (A) Hopfield neuron. The unit
fires if the weighted sum Σj wijSj of the inputs reaches or exceeds the threshold μi [2]. (B)
An example of three layer perceptron.

The example of one pattern:

D.3 The Drawbacks of Hebb’s Rule

2
Though being simple, the classical Hebb’s rule has some disadvantages. Depending the
application area of neural network, some drawbacks are tolerable but some need
improvement. Generally speaking, for the purpose of data processing and statistical
analysis, the speed and power of computation are valued more than the resemblance
between the model and the physiological realism. On the other hand, for unraveling the
memory formation mechanism and harvesting the emerging properties of biological
network, the models need to be built upon a certain degree of biophysical basis.

D.3.1 Stability of Hebbian network

For classical Hebb’s rule, there is no algorithm for synapses to get weaker and no upper
bound that limits how strong the connectivity can get [3]. Therefore, it is intrinsically
unstable. To overcome the stability problem, Bienenstock, Cooper, and Munro proposed
an omega shaped learning rule called BCM rule. The general BCM improves Hebb’s
rule and takes the following form:

dwij
= ϕ ( xi ) ⋅ x j − kw ⋅ wij (A.4)
dt

There are two main differences between these two rules: the decay of synaptic weight
with a rate constant kw and the nonlinear dependence of synaptic weight with respect to
postsynaptic activities. The nonlinearity was described by the postsynaptic activation
function, φ(x). As shown in Figure D.3.1-3, the activation function is negative for xi
under threshold value θw but grows positive once the postsynaptic activity becomes larger
than θw.

Figure D.3.1-3 The activation function of synaptic strength with respect to postsynaptic
activity. Hebb’s rule poses linear dependence of synaptic strength on postsynaptic
activity while BCM rule sets nonlinear omega-shaped dependence. The x-axis

3
corresponds to postsynaptic activity, xi and the y-axis represents the activation function,
φ(x) in Eq. (A.4).

D.3.2 The simplification of neuronal activities

Under external stimulation, neurons fire and emit a series of pulses instead of a simple
output level. In Hebb’s rule, only a single value xi is assigned to the neuron i to represent
its activity. Yet the simplification results into the loss of information such as the
threshold and tendency of firing as well as the detailed change in the short-term and the
long-term plasticity.

The variety and emerging properties may be the results of differentiated neurons. The
network of these specialized neurons with individual attributes gives rise to bountiful
functionality.

D.3.3 The inability to model spike timing dependence

It has been experimentally shown that the change in synaptic plasticity depends upon the
relative spike timing between presynaptic and postsynaptic neurons. Spike timing
dependence has become an important experimental protocol in eliciting change in
synaptic plasticity since its discovery. Although the role of temporal order was suggested
in the original statement of Hebb’s rule, the time window requirement is not incorporated
in the equations.

There are various subtypes of spike timing dependent plasticity (STDP); details are
provided in Chapter 6: Modeling Spike Timing Dependent Plasticity. Some synapses
demonstrate anti-Hebbian behavior in which presynaptic spiking preceding postsynaptic
stimulation results into long term depression instead of potentiation. Still other synapses
demonstrate only spike timing dependent potentiation or depression. Therefore,
modifying classical Hebb’s rule can lead to more versatile behavior of neural network
which potentially may

1. Hebb D (1949) The Organization of Behavior. A Neuropsychological Theory.

Wiley, New York, NY
2. Hertz J, Krogh A, Palmer RG (1991) Introduction to the theory of neural
computation. Addison-Wesley Publishing Company, Redwood City, CA
3. Leon NC (2000) Memories and memory: a physicist's approach to the brain.
International Journal of Modern Physics A 15:4069-4082