3391ch18.pdf
(
407 KB
)
Pobierz
Microelectronics, Second Edition
18
Neural Networks and
Fuzzy Systems
18.1 Neural Networks and Fuzzy Systems
..................
18
-
1
18.2 Neuron cell
..........................................
18
-1
18.3 Feedforward Neural Networks
........................
18
-3
18.4 Learning Algorithms for Neural Networks
............
18
-
5
Hebbian Learning Rule
•
Correlation Learning Rule
•
Instar
Learning Rule
•
Winner Takes All (WTA)
•
Outstar Learning
Rule
•
Widrow-Hoff LMS Learning Rule
•
Linear Regression
•
Delta Learning Rule
•
Error Backpropagation Learning
18.5 Special Feedforward Networks
........................
18
-
11
Functional Link Network
•
Cascade Correlation Architecture
18.6 Recurrent Neural Networks
...........................
18
-16
Hopfield Network
•
Autoassociative Memory
•
Bidirectional
Associative Memories (BAM)
18.7 Fuzzy Systems
........................................
18
-18
Fuzzification
Rule Evaluation
•
Defuzzification
18.8 Design Example
......................................
18
-21
18.9 Genetic Algorithms
..................................
18
-
21
Coding and Initialization
•
Selection and Reproduction
•
Reproduction
•
Mutation
Bogdan M. Wilamowski
18.1 Neural Networks and Fuzzy Systems
New and better electronic devices have inspired researchers to build intelligent machines operating in a
fashion similar to the human nervous system. Fascination with this goal started when McCulloch and
Pitts (1943) developed their model of an elementary computing neuron and when Hebb (1949) intro-
duced his
learning rules
. A decade latter Rosenblatt (1958) introduced the
perceptron
concept. In the early
1960s Widrow and Holf (1960, 1962) developed intelligent systems such as ADALINE and MADALINE.
Nillson (1965) in his book
Learning Machines
summarized many developments of that time. The pub-
lication of the Mynsky and Paper (1969) book, with some discouraging results, stopped for sometime
the fascination with artificial neural networks, and achievements in the mathematical foundation of the
backpropagation
algorithm by Werbos (1974) went unnoticed. The current rapid growth in the area of
neural networks started with the Hopfield (1982, 1984) recurrent network, Kohonen (1982) unsupervised
training algorithms, and a description of the backpropagation algorithm by Rumelhart et al. (1986).
18.2 Neuron cell
A biological neuron is a complicated structure, which receives trains of pulses on hundreds of
excitatory
and
inhibitory
inputs. Those incoming pulses are summed with different weights (averaged) during the
time period of
latent summation
. If the summed value is higher than a threshold, then the neuron itself is
18
-
1
© 2006 by Taylor & Francis Group, LLC
•
18
-2
Microelectronics
MEMORY
A
+
1
OR
A
+
1
AND
NOT
+
1
1
+
1
A
+
B
+
C
1
+
1
ABC
−
1
NOT
A
WRITE 1
WRITE 0
−
2
+
1
B
C
T
=
0.5
B
C
T
=
2.5
A
T
= −
0.5
T
=
0.5
(a)
(b)
(c)
(d)
FIGURE 18.1
OR, AND, NOT, and MEMORY operations using networks with McCulloch–Pitts neuron model.
generating a pulse, which is sent to neighboring neurons. Because incoming pulses are summed with time,
the neuron generates a pulse train with a higher frequency for higher positive excitation. In other words,
if the value of the summed weighted inputs is higher, the neuron generates pulses more frequently. At the
same time, each neuron is characterized by the nonexcitability for a certain time after the firing pulse. This
so-called
refractory period
can be more accurately described as a phenomenon where after excitation the
threshold value increases to a very high value and then decreases gradually with a certain time constant.
The refractory period sets soft upper limits on the frequency of the output pulse train. In the biological
neuron, information is sent in the form of frequency modulated pulse trains.
This description of neuron action leads to a very complex neuron model, which is not practical. Mc-
Culloch and Pitts (1943) show that even with a very simple neuron model, it is possible to build logic and
memory circuits. Furthermore, these simple neurons with thresholds are usually more powerful than typi-
cal logic gates used in computers. The McCulloch-Pitts neuron model assumes that incoming and outgoing
signals may have only binary values 0 and 1. If incoming signals summed through positive or negative
weights have a value larger than threshold, then the neuron output is set to 1. Otherwise, it is set to 0.
1f
net
T
=
≥
T
(18.1)
0f
net
<
T
where
T
is the threshold and
net
value is the weighted sum of all incoming signals
n
net
=
w
i
x
i
(18.2)
i
=
1
Examples of McCulloch-Pitts neurons realizing OR, AND, NOT, and MEMORY operations are shown in
Fig. 18.1. Note that the structure of OR and AND gates can be identical. With the same structure, other
logic functions can be realized, as Fig. 18.2 shows.
The perceptron model has a similar structure. Its input signals, the weights, and the thresholds could
have any positive or negative values. Usually, instead of using variable threshold, one additional constant
input with a negative or positive weight can added to each neuron, as
Fig. 18.3
shows. In this case, the
threshold is always set to be zero and the net value is calculated as
n
net
=
w
i
x
i
+
w
n
+
1
(18.3)
i
=
1
where
w
n
+
1
has the same value as the required threshold and the opposite sign. Single-layer percep-
trons were successfully used to solve many pattern classification problems. The hard threshold activation
functions are given by
1f
net
o
=
f
(
net
)
=
sgn(
net
)
+
1
=
≥
0
(18.4)
2
0if
net
<
0
A
+
1
+
1
+
A
+
1
+
1
+
AB
+
BC
+
CA
AB
+
C
B
C
T
=
1.5
B
C
T
=
1.5
1
2
(a)
(b)
FIGURE 18.2
Other logic functions realized with McCulloch–Pitts neuron model.
© 2006 by Taylor & Francis Group, LLC
+
+
Neural Networks and Fuzzy Systems
18
-3
x
1
x
2
x
3
x
4
x
n
x
1
x
2
x
3
x
4
x
n
T
=
0
T
=
t
w
n
+
1
= −
T
(a)
(b)
+
1
FIGURE 18.3
Threshold implementation with an additional weight and constant input with
+
1 value: (a) Neuron
with threshold
T
, (b) modified neuron with threshold
T
=
0 and additional weight equal to
−
T
.
λ
λ
(a)
(b)
(c)
(d)
FIGURE 18.4
Typical activation functions: (a) hard threshold unipolar, (b) hard threshold bipolar, (c) continuous
unipolar, (d) continuous bipolar.
for
unipolar
neurons and
o
=
f
(
net
)
=
sgn(
net
)
=
1f
net
≥
0
(18.5)
−
1f
net
<
0
for
bipolar
neurons. For these types of neurons, most of the known training algorithms are able to adjust
weights only in single-layer networks.
Multilayer neural networks usually use continuous activation functions, either unipolar
o
=
f
(
net
)
=
1
(18.6)
1
+
exp(
−
λ
net
)
or bipolar
o
=
f
(
net
)
=
tanh(0
.
5
λ
net
)
=
2
net
)
−
1
(18.7)
1
+
exp(
−
λ
parameter can be eliminated from Eqs. (18.6) and (18.7) and the
steepness of the neuron response can be controlled by the weight scaling only. Therefore, there is no real
need to use neurons with variable gains.
Note, that even neuron models with continuous activation functions are far from an actual biological
neuron, which operates with frequency modulated pulse trains.
λ
18.3 Feedforward Neural Networks
Feedforwardneuralnetworks
allow only one directional signal flow. Furthermore, most feedforward
neural networks are organized in layers. An example of the three-layer feedforward neural network is
shown in
Fig. 18.5
.
This network consists of input nodes, two
hidden layers
, and an output layer.
A single neuron is capable of separating input patterns into two categories, and this separation is linear.
For example, for the patterns shown in
Fig. 18.6
,
the separation line is crossing
x
1
and
x
2
axes at points
x
10
and
x
20
. This separation can be achieved with a neuron having the following weights:
w
1
=
1
/
x
10
,
© 2006 by Taylor & Francis Group, LLC
These continuous activation functions allow for the gradient-based training of multilayer networks.
Typical activation functions are shown in Fig. 18.4. In the case when neurons with additional threshold
input are used (Fig. 18.3(b)), the
18
-4
Microelectronics
HIDDEN
LAYER #1
HIDDEN
LAYER #2
OUTPUT
LAYER
+
1
+
1
+
1
FIGURE 18.5
An example of the three layer feedforward neural network, which is sometimes known also as the
backpropagation network.
w
2
=
1
/
x
20
, and
w
3
=−
1. In general for
n
dimensions, the weights are
w
i
=
1
x
i
0
for
w
n
+
1
=−
1
(18.8)
1 neurons should be used. If more input clusters are to be selected, then the number
of neurons in the input (hidden) layer should be properly multiplied. If the number of neurons in the input
(hidden) layer is not limited, then all classification problems can be solved using the three-layer network.
An example of such a neural network, classifying three clusters in the two-dimensional space, is shown in
Fig. 18.7
.
Neurons in the first hidden layer create the separation lines between input clusters. Neurons in
the second hidden layer perform the AND operation, as shown in
Fig. 18.1(b)
. Output neurons perform
the OR operation as shown in Fig. 18.1(a), for each category. The linear separation property of neurons
makes some problems specially difficult for neural networks, such as exclusive OR, parity computation
for several bits, or to separate patterns laying on two neighboring spirals.
+
x
1
x
10
x
1
x
2
w
1
w
2
w
3
+
1
x
20
x
2
w
1
=
1
x
10
1
x
20
w
3
= −
1
FIGURE 18.6
Illustration of the property of linear separation of patterns in the two-dimensional space by a single
neuron.
© 2006 by Taylor & Francis Group, LLC
One neuron can divide only linearly separated patterns. To select just one region in
n
-dimensional input
space, more than
n
w
2
=
Neural Networks and Fuzzy Systems
18
-5
HIDDEN
LAYER #1
HIDDEN
LAYER #2
AND
OR
INPUTS
OUTPUT
+
1
+
1
+
1
FIGURE 18.7
An example of the three layer neural network with two inputs for classification of three different clusters
into one category. This network can be generalized and can be used for solution of all classification problems.
The feedforward neural network is also used for nonlinear transformation (mapping) of a multidimen-
sional input variable into another multidimensional variable in the output. In theory, any input-output
mapping should be possible if the neural network has enough neurons in hidden layers (size of output
layer is set by the number of outputs required). In practice, this is not an easy task. Presently, there is no
satisfactory method to define how many neurons should be used in hidden layers. Usually, this is found
by the trial-and-error method. In general, it is known that if more neurons are used, more complicated
shapes can be mapped. On the other hand, networks with large numbers of neurons lose their ability for
generalization, and it is more likely that such networks will also try to map noise supplied to the input.
18.4 Learning Algorithms for Neural Networks
Similarly to the biological neurons, the weights in artificial neurons are adjusted during a training pro-
cedure. Various learning algorithms were developed, and only a few are suitable for multilayer neuron
networks. Some use only local signals in the neurons, others require information from outputs; some
require a supervisor who knows what outputs should be for the given patterns, and other unsupervised
algorithms need no such information. Common learning rules are described in the following sections.
Hebbian Learning Rule
The Hebb (1949) learning rule is based on the assumption that if two neighbor neurons must be activated
and deactivated at the same time, then the weight connecting these neurons should increase. For neurons
operating in the opposite phase, the weight between them should decrease. If there is no signal correlation,
the weight should remain unchanged. This assumption can be described by the formula
w
ij
=
cx
i
o
j
(18.9)
where
w
ij
=
weight from
i
th to
j
th neuron
c
=
learning constant
x
i
=
signal on the
i
th input
o
j
=
output signal
The training process starts usually with values of all weights set to zero. This learning rule can be used for
both soft and hard threshold neurons. Since desired responses of neurons is not used in the learning pro-
cedure, this is the
unsupervised learning
rule. The absolute values of the weights are usually proportional
to the learning time, which is undesired.
Correlation Learning Rule
The correlation learning rule is based on a similar principle as the Hebbian learning rule. It assumes
that weights between simultaneously responding neurons should be largely positive, and weights between
neurons with opposite reaction should be largely negative.
© 2006 by Taylor & Francis Group, LLC
Plik z chomika:
Januszek66
Inne pliki z tego folderu:
Back Seat_ A Mumbai Tale - Aditya Kripalani.mobi
(755 KB)
Brief Wondrous Life of Oscar Wao, The - Junot Diaz.opf
(3 KB)
Don't Make Me Think, Revisited_ - Steve Krug.mobi
(9256 KB)
M. T. Anderson - Norumbegan 03 - The Empire of Gut and Bone # (v5.0).epub
(2209 KB)
M. T. Anderson - Norumbegan 02 - The Suburb Beyond the Stars # (v5.0).epub
(2105 KB)
Inne foldery tego chomika:
Dokumenty
Galeria
LUDLUM ROBERT
Midi - Kar
Mszał Rzymski PL
Zgłoś jeśli
naruszono regulamin