Neural network models: an analysis

Lecture Notes in Control and Information Sciences Editor: M. Thoma

210

Philippe De Wilde

Neural Network Models An Analysis

~ Springer

Series Advisory Board A. Bensoussan • M.]. Grimble • P. Kokotovic • H. Kwakernaak J.L. Massey • Y.Z. Tsypkin

Author Philippe De Wilde, Dr. Department of Electrical and Electronic Engineering, Imperial College of Science, Technology and Medicine, Exhibition Road, London SW7 2BT, UK

ISBN 3-540-19995-0 Springer-Verlag Berlin HeidelbergNew York British Library Cataloguing in Publication Data Wilde, Philippe De Neural Network Models: Analysis. (Lecture Notes in Control & Information Sciences; Vol.210) I. Title. IL Series 006.3 ISBN 3-540-19995-0 Library of Congress Cataloging-in-Publication Data De Wilde, Philippe, 1958Neural network models : an analysis / Philippe De Wilde p. cm. - - (Lecture notes in control and information sciences : zlo) Includes index. 1SBN 3-540-19995-0 (pbk. : a l l paper) L Neural networks (computer science) I. Title. II. Series. QA76.87.D45 1995 006.3 - - dc20

9541716 CIP

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 19S8, this publication may only be reproduced, stored or transmitted, in any form o r b y any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. © Springer-Verlag London Limited 1996 Printed in Great Britain The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Typesetting: Camera ready by author Printed and bound at the Athenaeum Press Ltd, Gateshead 69/3830-543210 Printed on acid-free paper

Contents Key concepts in neural networks 1.1 1.2 1.3

1.4

1.5 1.6

Keywords and a Key Concept . . . . . . . . . . . . . . . A Guide to the L i t e r a t u r e . . . . . . . . . . . . . . . . . T h e Operation of a Basic Network . . . . . . . . . . . . . 1.3.1 Static Description of the Network . . . . . . . . . 1.3.2 Evolution in T i m e . . . . . . . . . . . . . . . . . 1.3.3 Construction of the U p d a t e Rule . . . . . . . . . 1.3.4 Trajectories in State Space . . . . . . . . . . . . . 1.3.5 C a p a c i t y . . . . . . . . . . . . . . . . . . . . . . . 1.3.6 Auto-associativity . . . . . . . . . . . . . . . . . . O t h e r Notions . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Hetero-associativity (Associative Memory) . . . . 1.4.2 Layered Networks . . . . . . . . . . . . . . . . . . 1.4.3 Learning in a Net with Input and O u t p u t . . . . 1.4.4 O t h e r U p d a t e Rules . . . . . . . . . . . . . . . . 1.4.5 Energy Surface . . . . . . . . . . . . . . . . . . . 1.4.6 Higher Order Networks . . . . . . . . . . . . . . . 1.4.7 Layout . . . . . . . . . . . . . . . . . . . . . . . . List of Symbols . . . . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 Backpropagation 2.1 2.2 2.3

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . How the Network Functions . . . . . . . . . . . . . . . . Calculation of Weights by Backpropagation . . . . . . . . 2.3.1 Calculation of the Error Signals of the O u t p u t Neurons . . . . . . . . . . . . . . . . . . . . . . .

7 7 9 11 11 12 12 16 17 20 22 22 23 23 26 28 28 31 32 32

35 35 36 39 39

2

CONTENTS 2.3.2

2.4 2.5 2.6 2.7

4

5

T h e Weight U p d a t e s of t h e Weights b e t w e e n Hidden Neurons a n d O u t p u t Neurons . . . . . . . . . 39 2.3.3 T h e Error Signals for t h e H i d d e n Neurons . . . . 41 2.3.4 T h e Weight U p d a t e s of t h e Weights b e t w e e n t h e I n p u t Neurons a n d t h e H i d d e n Neurons . . . . . . 41 2.3.5 R e m a r k s on E p o c h s a n d Multiple P a t t e r n s . . . 42 A General T h r e e Layer Network . . . . . . . . . . . . . . 43 T h e General B a c k p r o p a g a t i o n A l g o r i t h m . . . . . . . . . 44 Applications of Neural Networks . . . . . . . . . . . . . 47 Problems ........................... 51

Neurons in the Brain

53

3.1 3.2 3.3 3.4 3.5

54 58 61 65 67

T h e Physical Model . . . . . . . . . . . . . . . . . . . . . O p e r a t i o n of t h e N e u r o n . . . . . . . . . . . . . . . . . . T h e M a t h e m a t i c a l Model . . . . . . . . . . . . . . . . . . T h e Transfer F u n c t i o n . . . . . . . . . . . . . . . . . . . Problems ...........................

The Fundamental System of Differential Equations

69

4.1 4.2 4.3 4.4 4.5 4.6 4.7

70 72 74 76 82 89 94

Existence of a Solution . . . . . . . . . . . . . . . . . . . Numerical Solution of t h e E q u a t i o n s . . . . . . . . . . . An Analog Circuit . . . . . . . . . . . . . . . . . . . . . The Energy Function .................... Stability of Equilibria . . . . . . . . . . . . . . . . . . . . A L i a p u n o v T h e o r e m for neural networks . . . . . . . . . Problems ...........................

Synchronous and Discrete Networks 5.1 5.2 5.3

F r o m Analog to Digital Neural Networks . . . . . . . . . S y n c h r o n o u s d y n a m i c s a n d cycles . . . . . . . . . . . . . Problems ...........................

6 Linear Capacity 6.1 6.2 6.3

T h r e s h o l d Functions . . . . . . . . . . . . . . . . . . . . Linear C a p a c i t y . . . . . . . . . . . . . . . . . . . . . . . Problems ...........................

97 98 100 103

105 106 113 115

CONTENTS 7

8

3

Capacity from a Signal to Noise Ratio 7.1 S u m - o f - O u t e r p r o d u c t Weights . . . . . . . . . . . . . . . 7.2 C a p a c i t y D e p e n d e n t on Representation . . . . . . . . . . 7.3 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .

117

N e u r a l N e t w o r k s and M a r k o v C h a i n s

131

8.1 8.2 8.3

131 136 140

Simulated annealing . . . . . . . . . . . . . . . . . . . . T h e Fokker-Planck equation . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .

117 124 128

Preface When I was asked to teach a course on neural networks at Imperial College of Science, Technology and Medicine, London, I surveyed the literature for any text suitable for Engineers at a Master's level. There seemed to be two categories of text. One treated neural networks as a collection of recipes. These books are easy to read, and full of examples. Unfortunately, the reader is given no insight why an example works. The result is that the book is only useful if the reader's problems are similar to the examples in the book. If one wants to apply neural networks to new and challenging problems, one needs a deeper insight. This is offered by a second class of texts. They are based on statistical physics. If they have to appeal to an engineer, the statistical physics is usually watered down. Statistical physics, though important, is not the only foundation on which to base a deeper insight in neural networks. In the last seven years, I have done research into neural networks, while working and teaching in an engineering environment. I have learned that the knowledge needed to develop successful engineering applications of neural networks, is based on dynamical systems, information theory, and statistical physics. This book uses these three paradigms to explain neural networks. The novelty of this approach is that dynamical systems and information theory concepts are used for the first time in an expository text about neural networks. Where other books try to give an encyclopedic overview, I have chosen a pedagogical approach. Attention is focused on the main issues, and they are developed from scratch. This will allow a wide audience to understand the main issues about neural networks in depth. The reader will learn the most important parameters for the design of neural

PREFACE

5

network applications. In this way, he or she will start an application with the building blocks ready at hand, without the need for a blind imitation of other existing applications. This book has been used as a text for the Analysis of Neural Networks course at Imperial College since 1990. The audience consisted of final year undergraduates, Master's degree students, and PhD students from Electrical Engineering, Computing, and Biomedical Systems. The book is also suitable for students and interested readers with a background in Applied Mathematics, Physics, and General Engineering. This book is exclusively about quantitative results in neural networks. The reader can verify most formulas by writing a small simulation program. This will help a lot in understanding the theory. There are no extensive results of simulations in this book. Because neural networks are high dimensional systems, often with hundreds of degrees of freedom, many simple simulations are statistically insignificant. In no engineering field has the author seen so many simulations that cannot be generalized. An exception to this is the statistical physics literature, but it is not very accessible to engineers. For this reason, we wilt concentrate on qualitative results. Chapter t gives a bird's eye view of neural networks. It introduces all concepts that will be studied in depth in further chapters. The most popular neural networks are layered in structure, and they are studied in Chapter 2. Neural networks are derived from models of the brain. This is the subject of Chapter 3. In this book, brain models are used as inspiration to build engineering systems, as the wings of a bird were the inspiration for the wings of an airplane. In Chapter 4, we will see that neural networks are essentially dynamical systems, and we derive results about the stability of their equilibria. Chapter 5 is about oscillations in neural networks. The number of equilibria is calculated in Chapters 6 and 7. Finally, Chapter 8 introduces statistical models of neural networks. It is assumed that the reader knows what a differential equation is. This is used in Chapters 3, 4, and 5. For Chapter 6, the reader needs to know what a binomial coefficient is, and what the equation of a line is. In Chapter 7, the expression for the normal distribution is used. Chapter 8 uses Markov chains. All chapters can be read independently of each other. The reader who does not like a chapter can just skip it

6

PREFACE

and go on with the next chapter. Somebody in a great hurry can just read the first and the last chapter/ Three types of neural networks are not studied in this book. Kohonen's self-organizing networks require a treatment that is based on statistical data processing. Networks using radial basis functions need to be considered in a function approximation framework. Both types of networks have been omitted, not because they are unimportant, but because the necessary mathematical preliminaries would disturb the unity of the present book. A third type of network, based on the ideas of Grossberg, is not studied explicitly, but is present implicitly in our treatment of networks as dynamical systems. The author can be reached via electronic mail to [email protected], and will be happy to answer any technicM questions you may have about subjects treated in this book. Many thanks to the people who have encouraged me to do basic neural network research, in particular I. Aleksander, E. C. Van Der Meulen, A. N. Chet~ev, R. Smith, and my wife, Janet. In addition, the following people have inspired me via lectures or talks: D. Rumelhart, H. Haken, V. I. Kryukov, E. Caianiello, and J. Eccles. They have strengthened my belief that applied science and engineering are not a collection of gadget applications but that they are part of the edifice of science.

Philippe De Wilde London July 1995

Chapter 1 Key concepts in neural networks This chapter gives a "quick and dirty" overview of neural networks. The key concepts are introduced, and the basic operation of the network is explained. Everything introduced in this chapter is taken up again later in the book. There is also a guide to the literature, referring to some of the most inspiring books on neural networks. But this book stands on its own. In fact, it introduces and derives many results that you will not find in any of the existing books about neural networks. After going through this chapter, you will be able to impress any audience at a cocktail party with your knowledge about neural networks!

1.1

K e y w o r d s and a K e y C o n c e p t

Neural networks are about associative m e m o r y or content-addressable memory. You give a content to the network, and you get an address or an identification back. You could store images of people in the network. When you show an image of a person to the network, it will return the name of the person, or some other identification, e.g. the social security number. Neural networks are also about parallel processing . They consist

8

CHAPTER 1. K E Y CONCEPTS IN NEURAL NETWORKS

of a network of processors that operate in parallel. This means they will operate very fast. To date, the most complex neural networks that operate in parallel consist of a few hundred neurons. But the technology is evolving fast. To recognize images, one needs about one processor per pixel. The processors, also called neurons, are very simple, so they can be kept small. Neural networks are also fail-soft devices. If some of the processors fail, the network will degrade slowly. This is highly desirable from the consumer's point of view, and also for the design engineer, who has to cram many processors on one chip, and who can now use some less reliable, but smaller components. Neural networks are often used to classify or categorize. If the images of a group of persons were stored in the network, it could infer who looks sad, and who looks happy. Of course, the only information the network has is the image, and someone could look sad, but still be very happy! The major key concept in neural networks is the interaction between microscopic and macroscopic phenomena. A macroscopic phenomenon of the image of a face of a person can be a frown, or a smile. A microscopic phenomenon is the correlation between the pixels in the image. Macroscopic phenomena are related to form. The spiral in Fig. 1.1 for example is generated by varying the curvature of a curve. When doing this, one only considers a few successive points on the curve. There is no global knowledge necessary to draw the spiral. Growth of a crystal is another example of how microscopic or molecular forces have a macroscopic effect. Clouds are yet another example. The cloud is large scale. The water molecules in it are small scale. Much of the research in neural networks is about explaining global properties from interaction between particles. A good model of this is the derivation of the gas laws from the movement of molecules. In the eighteenth century, Robert Boyle observed that. for a given mass of a gas held at a constant temperature, the pressure is inversely proportional to the volume. This is a macroscopic property of a gas. At the end of the nineteenth century, Ludwig Bottzmann derived this relation using the speed and mass of the molecules in the gas. The molecules are the microscopic constituents of the gas. For the network that recognizes faces of people, the macroscopic

1.2. A GUIDE TO THE LITERATURE

9

Figure 1.1: A spiral is a large scale form (right), generated by small scale interactions (magnified in the circle on the left) between the cells when the spiral is grown. properties that we will study include the number of faces that the network can recognize, the speed at which it does this, the number of classes into which it can split up the faces (smiling, frowning, ... ), etc. The microscopic interactions will be the calculations that the neurons, the little processors attached to each pixet, perform. We will have to specify these calculations so that the network can recognize a particular set of faces. Most applications of neural networks can be reduced to pattern recognition, as explained in Section 2.6. So our example turns out to be a very general one.

1.2

A G u i d e to t h e L i t e r a t u r e

This little guide is deliberately selective, and presents the author's view on the most important neural network books and review papers. This book is closest in spirit to J. Hertz et al., Introduction to the Theory of Neural Computation [41], and S. Haykin, Neural Networks [39]. They are the two main reference works for the theory of neural networks. They give a broad overview, and Hertz et al. also establish the roots of neural networks in statistical mechanics. Several books present, neural networks in a similar way, but are

10

CHAPTER I. K E Y CONCEPTS IN NEURAL N E T W O R K S

more specialized. Y. Kamp and M. Hasler, Recursive Neural Networks for Associative Memory [47] is good for Hopfield networks. E. Goles and S. Martinez, Neural and Automata Networks [33] stresses the link with cellular automata. S. Grossberg, The adaptive brain [37] studies biologically plausible neural networks as dynamical systems. The works [50, 55] study unsupervised classification. General iterative systems are studied in F. Robert, Discrete Iterations [68]. R. Hecht-Nielsen, Neuroeomputing [40] is good for backpropagation, if one is tolerant of some peculiar notation. Many theoretical results about neural networks stem from the statistical physics of spin glasses. A standard work is M. M~zard et al., Spin Glass Theory and Beyond. The neural network side is presented in D. Amit, Modeling Brain Function [7]. A different approach, based on the Fokker-Planck equation, is taken by H. Haken in Synergetics [38]. This pioneering book predates neural networks themselves, and is a jewel to read. Other stochastic models of neural networks are studied in two largely unknown Russian books: A. Chetaev, Neural Nets and Markov Chains [14], and V. Kryukov et al., The Metastable and Unstable States in the Brain [52]. Some of Chetaev's work has been published in English as [13]. Two seminal papers by J. Hopfield are still good reading, and they have the advantage of brevity [45, 46]. Neural networks is an interdisciplinary field. Apart from statistical physics, it is also deeply embedded in cognitive science. When the link with cognitive science is stressed, neural networks are often referred to as parallel distributed processing or connectionism. Fundamental work in this field was done by a group of people around D. Rumelhart and J. McClelland. It is written up in their books about parallel distributed processing [69, 59, 58]. Another original approach, combining Boolean networks and cognitive science, is taken by I. Aleksander and H. Morton, in [5, 6]. All these books are easy to read, even without any knowledge about psychology or artificial intelligence. A link between neural networks and quantum mechanics is described in [28, 29, 24, 25]. On the Internet, there is an active newsgroup comp. a i . n e u r a l - n e t s which has a regular posting answering frequently asked questions about neural nets. It is worthwhile reading this. It also contains information about simulation software.

1.3.

THE O P E R A T I O N OF A B A S I C N E T W O R K

1.3 1.3.1

11

The Operation of a Basic Network Static D e s c r i p t i o n of the N e t w o r k

In Fig. 1.2, the neurons are simply represented as dots. To every neuron is a t t a c h e d a n u m b e r t h a t can be +1 or -1. This is called the state of the neuron. For example, the state of neuron 4 is denoted by x4, and x4 = +1. We will encounter neurons with more general states in Section 1.4.4.

x2=-1

x I ~+1

• x3=+1

x4=+1 x5=-1 Figure 1.2: A network of five neurons, with their states indicated. T h e states of all neurons can be grouped in a vector, called the state vector. For the network in Fig. 1.2, the state vector is x=(+1,-1,+1,+1,-1)

r.

We will denote vectors by bold characters. T h e .T indicates the transpose. This means that x is actually a column vector with 5 rows and 1 column. In general, the state vector has as m a n y components, or rows, as there are neurons. If the neurons were used to store pixels in the image of a face, a state +1 could indicate a black pixel, a state -1 could indicate a white one. W h y +1 and -1 are used instead of 1 and 0 is a deep problem t h a t will be discussed in Section 7.2. For the m o m e n t , it is sufficient to r e m a r k t h a t the numbers +1 and -1 can be transformed to arbitrary

12

C H A P T E R 1. K E Y CONCEPTS IN N E U R A L N E T W O R K S

numbers a and b via the transformation y=

1.3.2

a(1 -- x) + b(1 + x) 2

E v o l u t i o n in T i m e

Neural networks are dynamical. This means that their state changes in time. In Fig. 1.3, the network is drawn at time t=0. The state vector -1

+1

•

+1

+1 -1 Figure 1.3: The network at time t=0. at this time is denoted by x(0). In Fig. 1.4, the network is drawn at time t = l . Remark that neuron 1 has not changed state, but neuron 2 has changed state from -1 to +1. The state vector at time t = l is denoted by x(1). Do not confuse this with Xl, the state of neuron 1. As time evolves further, t = 0, 1 , 2 , . . . , the state vector evolves too, x(0), x(1), x(2), ....

1.3.3

C o n s t r u c t i o n of the U p d a t e R u l e

The rule according to which the neurons change in time is called the update rule. Let's concentrate on neuron 4. In Fig. 1.5, you can see that there are ].inks between neurons 1, 2, 3, and 4. This means that neuron 4 will be influenced by neurons 1, 2, and 3. This happens according to the following rule.

x,(t + 1) = sgn(T~x~(t) + T4~x~(t) + T~x~(t)).

1.3.

THE OPERATION OF A BASIC NETWORK

13

+1

+1

•

+1

-1 -1 Figure 1.4: T h e network at time t = l .

x2

×

x3

•

x4

×5 Figure 1.5: How neurons 1, 2, and 3 influence neuron 4.

14

CHAPTER 1. K E Y CONCEPTS IN NEURAL NETWORKS

This formula shows how the state of neuron 4 at time t + l is derived from the states of neurons 1, 2, and 3 at time t. The T's can be any number, and the function sgn is drawn in Fig. 1.6.

Ira-

0

-1 Figure 1.6: The function sgn. The value at 0 equals 1. If we plug in some actual values in the update rule, x4(1) = sgn[2(+l) + 2 ( - 1 ) - 3(+1)] = s g n ( - 3 ) = - 1 , indicating that the state of neuron 4 at time t = l is -1. To understand the meaning of the numbers T, assume that neuron 4 is only connected to neuron 1. The update rule is then

x4(t + 1)=sgn(T41x~(t)). If T41 is positive, this further simplifies t o x4(t + 1) = x l ( t ) . This means that neuron 4 will switch to the same state as neuron 1. In this case, the connection between neurons 4 and 1 is called excitatory. If T41 is negative, the update rule becomes x4(t + 1) = - x l ( t ) , and neuron 4 will be in the state with opposite sign from the state of neuron 1. In this case, the connection is said to be inhibitory. The biological motivation for the terms excitatory and inhibitory will be explained in Chapter 3. The elements of the matrix T are called weights or synapses. In Fig. 1.7, all connections for a particular five neuron network are drawn. The update rule for a general neuron i in this network can be written

xi(t + l) = sgn (~'~Tijxj(t))

1.3. THE OPERATION OF A BASIC NETWORK

15

x2 x1

x3

Figure 1.7: Some connections that can occur in a network of 5 neurons. This can be written down for i = 1,2, 3,4, 5. The matrix T is called the interconnection matrix, or synaptic matrix. Synaptic is another term from biology that witl be explained in Chapter 3. When there is no connection, the corresponding element in the matrix T is zero. For the network in Fig. 1.7,

(00 ) 0 0 0

T-

O0

0

.

0 0

0

Remark that T is symmetric, and has zero diagonal. This means that neuron i influences neuron j in the same way that neuron j influences neuron i (T symmetric). Also, no neuron influences itself (zero diagonal). For a network with n neurons, the update rule can be written

xi(t + l) =sgn (~-~ T~jxj(t)) , i =

(1.1)

If every neuron calculates its new state at the same time, the update is called synchronous. If, at every time step t = 1, 2, . . . , only one neuron calculates its new state, the update is called asynchronous. Problems of synchronous and asynchronous updating are discussed in Chapter 5.

16

1.3.4

C H A P T E R i. K E Y CONCEPTS IN N E U R A L N E T W O R K S

Trajectories in State Space

Following the evolution of a state vector through time, you can obtain something like (remember that T stand for the transpose of the vector, it makes a row vector into a column vector) x(0)

= (+1-1+1+1-1)

1

T,

i

x(1)

=

(+1+1+1+1-1)

x(2)

=

(+1 + 1 + 1 - 1 - 1) T,

x(3)

=

(+1+1+1-1-1)

l

T,

T.

1

After t=2, the state vector does not change any more. The network has converged. This is illustrated in Fig. 1.8.

Figure 1.8: Convergence to a single state. For the network that stores faces, x(0) could be a digitized face from a security camera, with low resolution and noise. The network would then converge to one of the faces previously stored, for example from a database of criminals. The network does not always converge, for example x(0)

:

( 3 - 1 - 1 - 1 - t - 1 - 1 - 1 ) T,

x(1)

-

(-1-1-1-13-1)

T,

x(2)

-- ( + 1 + 1 + 1 - 1 - 1 )

T,

x(3)

=

T,

(-1-1-1-1+1)

This network is oscillating, see Fig. 1.9. The network hesitates between

1.3. THE OPERATION OF A BASIC N E T W O R K

17

Figure 1.9: Oscillation between two states. two states, just as a h u m a n can hesitate between two persons, when recalling a face. Convergence and oscillation will be studied in Chapter 4 and 5. T h e sequence of states, as time evolves, is called a trajectory. The t r a j e c t o r y is a p a t h in a higher dimensional space. Because the state vector has as m a n y components as there are neurons, the higher dimensional space has as m a n y dimensions as there are neurons: This space is called the state space. T h e components of the state vectors are either +1 or -1, and we will denote the state space by { - t , 1}% This is for n neurons. T h e state space consists of the corners of an n-dimensional hypercube. An example of a state space for a network with 5 neurons is sketched in Fig. 1.10. T h e r e are 25 = 32 states. T h e end points of the trajectories are called fundamental memories or attractors. All states t h a t are on trajectories going to the same f u n d a m e n t a l m e m o r y are said to be in the same attraction basin. T h e network can be set up to perform error correction in the following way: the f u n d a m e n t a l memories are the pure images. T h e other points on the trajectories are the noisy or distorted images. As time evolves, the state of the network will converge to one of t h e pure images, or oscillate between a n u m b e r of images. T h e starting state or initial state is sometimes called a probe.

1.3.5

Capacity

T h e capacity of a neural network is the n u m b e r of f u n d a m e n t a l memories t h a t it can have. Suppose a user wants to store m f u n d a m e n t a l memories X 1 , X 2 , • . . , X TM .

T h e i-th c o m p o n e n t of f u n d a m e n t a l m e m o r y x ~ will be denoted by x~. T h e following recipe for the synaptic matrix T was proposed by

18

C H A P T E R 1. K E Y C O N C E P T S IN N E U R A L N E T W O R K S

I

~"

_ ~ ' ~

\ \

t

I

.-.. -...-...,,-.

fundamental memory

/ it

Figure 1.10: An example of the trajectories in the state space of a 5 neuron network.

1.3. THE OPERATION OF A BASIC NETWORK

19

J. Hop field [45] m

Tij=~(xTx~-Sij),

i=l,...,n,

j=l,...,n.

(1.2)

o~1

We will call a network with this choice of weights, a Hopfield network. We will prove in Chapter 7 that, if you use this recipe, the capacity 6'1 is of the order of

C1=0

(n) 41

n

'

where O means the following: f(n) = O(g(n)) if there exists a number nl such that for all n > nl, f(n) O. (4.8)

With the convention (4.7) for the time constant, this becomes

dui

C, -dt- -

ui R~

+ ~ --~fj(ui) T~j j=l

+ - ~Ii.

(4.9)

The left hand side of this equation can be interpreted as an input current charging a capacitor Ci to a potential ui. In the right hand side, -uJR,: is a leakage current, and Ii/Ri an input from outside the network. If the sum term stands for input currents from other neurons, fj(uj) is a potential, the output of an amplifier, and T~j/R~

76

CHAPTER 4. FUNDAMENTAL S Y S T E M

are conductances. The numbers Ri are just scaling factors here, not resistances. Conductances are always positive, so a solution has to be found for negative synapses. This is done by inverting the output of the amplifier, giving a signal fj(uj) as well as - f j ( u j ) , see Fig. 4.6.

!

iR

- out'out Figure 4.6: Negative synapses implemented using an inverter. This analysis leads to the analog electric network sketched in Fig. 4.7. It is the basic layout of many chip implementations of neural networks, even digital implementations, [35, 31, 62, 15]. Remark that all the synapses take up much more space than all the neurons.

4.4

The Energy Function

The time evolution, or dynamic behaviour, of a mechanical system is governed by a differential equation, Newtons law, F TF~

relating the second time derivative of the place coordinates x to the forces F and the mass m. When the system evolves in time, it evolves towards an energy minimum. This is illustrated in Fig. 4.8 for two mechanical systems, a ball rolling down a bowl, and a pendulum. A mechanical system where the outside forces are independent of time is called autonomous. In this section we will assume that the neural network is autonomous, i.e.

dli dt

0,

4.4.

THE E N E R G Y FUNCTION

77

conduct~e T

~In R~

I II il m

II

;~u~ ,~0~ ~°~

,~u~ ~u~

Figure 4.7: A basic electronic circuit for a neural network.

78

CHAPTER

4. F U N D A M E N T A L

SYSTEM

J

Figure 4.8: Two physical systems that evolve towards an energy minimum. or the external inputs Ii are independent of time. There exists no recipe for finding the energy function of a neural network. Moreover, the dynamic equations (4.1) for a neural network contain first order derivatives in time, whereas Newtons law contains second order time derivatives. We will therefore propose an expression for the energy function, and verify whether it decreases as the system evolves in time. The proposal is[35]

1

E = - - ~ i=1 j=l

Tijfi(ui)fj(ttj) -- E Iifi(ui) + ~=1

/z=li(udfi-l(x) dx.

~=1 J,=o

(4.10) If the transfer functions were linear, the first term would be quadratic in the state variables ui. The second term, too, is usual for an external force Ii. It is the third term that is rather surprising. The symbol f ~ l is the inverse f u n c t i o n o f / ,

defined b y / 7 1 ( f ~ ( x ) )

- x, see also Fig. 4.9.

Consider the upper limit of the integral x = f~(ui) = ri ql + 1 tanh(vi(ui - 0,)),

4.4.

79

THE ENERGY FUNCTION

f

°

!

S Figure 4.9: The transfer function fi and its inverse. or

vi(ui

Oi)

~

arctanh vi + 1 X , qi

hence ui = Oi + larctanhri + 1 X - f/--l(x). ui qi

Using this expression, we have for the integral in the energy function, with formula 2.741.3 from [34], ~ox f/-l(x) dx = ~o~ Oi dx + --1 ~ ~ arctanh r!,+ 1 x dx u~

--

Oix +

qi

--uil

{xarctanhri qi + 1x

2 ( r , + 11 in t ' , ; ; T - f J qi

=

qi

x 2]

2

Oix + _1 ~xarctanhri + 1 x

vi [, 2 ( r i + l ) ln 1 Remark that

-

1 l+y arctanhy = ~ In ----~" 1

qi k q~

x

.

(4.11)

CHAPTER 4. FUNDAMENTAL S Y S T E M

80

The result of this calculation is that the energy function (4.10) is just a complicated expression in fi(u~), i = 1 , . . . , n, with nothing indeterminate. Let us now abbreviate vi = fi(ui), i = 1 , . . . , n, so that 4.10 becomes E = -2

1 Y!,J~ T i j v i v j - ~~ Iivi + ~

v'fi-l(x)dx.

(4.12)

T h e o r e m 4.3 If the matrix T is symmetric, has zero diagonal, and if

the functions fj are monotonically increasing, then the function E =

--i1 ~ T,jv~v~- ~ I~ + ~

foV~ fVl(x) dx

is an energy function. Proof. In expression 4.12, vi is a function of time, and dE/dt will be the evolution of energy with time. Using the chain rule, dE dt

k=l

k=l

OE dvk Ovk dt

o(

\

n ~v~ ~ dvk ~Zij?)ivj-~Ii~)iJI-i~=l]O fi-l(x)dx] d~.... i=1 j = t

S" n

i¢k

k=l

i=1

TikViVk --

1 ~n Tkjvk,j - ~Tkkv~ 1 2)

j#k

dvk + 0 [~oV~f f a(x) dx ] --~

-~ k=l~-~[( -~Tikvi--Ik + f[l(vk)]

dvk dt '

"2 l jCk (4.13)

where we have also used Leibnitz's rule for differentiation under the integral sign [70]. We will assume now that the matrix T is symmetric, Tij = Tj~, i = 1 , . . . , n , j = 1 , . . . , n , and has zero diagonal, Tii = 0, i = 1 , . . . , n .

4.4.

81

THE ENERGY FUNCTION

In biological terms, these assumptions mean that the synapse from neuron i to j is the same as the synapse from j to i, and that no axon branches connect to the neurons own dendrites. This is not very plausible biologically, but helps a great deal in the engineering design of neural networks. The mathematical analysis is much simpler when T is symmetric and has zero diagonal. Asymmetric T and non-zero diagonal T matrices are a topic of current research. We will relax the conditions on T in Chapter 6 and 8. Continuing the derivation of d E / d t , and using (4.1), we find dE dt

-

k= 1

=

Tkivi--Ik+f;

k=l

(vk)

dt

i=1

T k i f i ( u i ) - Ik + f ~ l ( f k ( u k ) )

dt

i=1

=

duk dvk - 2 k_=, l# k dt dt

=

dukdfk(uk) -- k---1 #k dt dt

=

- 2k--1 _ , u k attk - r - \ dt ]

(4.14)

The time constants #i > 0, k = 1 , . . . , n, and if the transfer functions are monotonically increasing, dfk/duk > 0. Moreover, (duk/dt) 2 :> O, and thus dE~dr < 0, showing that the energy decreases in time. During the operation of the network, energy will decrease until it does not change any more. Q.E.D. From equation (4.14), we can also deduce dE dt

= 0 =:~

dui dt

= O,

i = 1,...,n.

(4.15)

This means that stationary points for the energy are stationary points for the whole whole neural network. For some approximations of the transfer function, dfk/dt = 0 in some interval, as illustrated in Fig. 4.10. Even in this case, the energy decreases during the operation of the network.

CHAPTER 4. FUNDAMENTAL S Y S T E M

82

V Figure 4.I0: A function with zero derivative for large absolute values of its input.

4.5

Stability of Equilibria

When the network is observed to be in a particular initial state Uo at time to, the numbers ul(to), u2(t0),..., un(to) all have values. The vector containing those values is called the state vector of the neural network. If there are n neurons, this is a vector in an n-dimensional space, the state space. In Fig. 4.11, you can see the trajectory in the state space, between time to and t2. For many applications of neural

u(t f • u(t(J

U(tl)

statespace Figure 4.11: Time evolution in state space. networks, we want to know what happens to the trajectory as time proceeds, t --~ c(~. The most useful situation, termed convergence in engineering terms, is sketched in Fig. 4.12. Other possibilities, often a nightmare for the engineer, are in Fig. 4.13. Analogous to a ball rolling in a bowl, the equilibrium can be stable or unstabte, see Fig. 4.14. Different initial conditions, close to a stable

4.5. S T A B I L I T Y OF E Q U I L I B R I A

83

state space

Figure 4.12: Convergence. equilibrium, will tend to make the system evolve towards this equilibrium. An unstable equilibrium will only be reached starting from one particular initial condition, and then only if there is no noise. This is possible in a digital system when the variables are discrete. In the presence of noise, the system will never be in an unstable equilibrium. A word of caution is necessary here. The dynamical systems we are studying are high dimensional if there are many neurons. The mathematical properties of the state spaces of these systems are very complicated, and are often open research problems. For example, next to stable and unstable equilibria, saddle points play an important role. In this book we only discuss the very basics that seem necessary to us for applying and understanding neural networks. For more detail, the reader should consult [2, 2, 8, 9, 54, 64, 23, 42] . Time can be added as an extra dimension in the state space. This is illustrated in Fig. 4.15 We will assume for simplicity from now on that there is no external input, and that M1 transfer functions go through the origin. This means I d t ) = 0, t >__to _> 0 and 0~ = 0. The fundamental system of differential equations is now

du,(t) #i

dt

- -ui(t) +~Tijfj(uj(t)), j=l

The trajectory ui(t) = O,i = 1 , . . . , n , t

i-

1,...,n,

#~ > 0.

(4.16)

>_ to >_ 0 is a solution of this

CHAPTER 4. FUNDAMENTAL S Y S T E M

84

u(te~ state space

state space

state space

state space

• u(t~

state space

Figure 4.13: Some possible b e h a v i o u r in s t a t e space as t ~ c¢.

4.5. S T A B I L I T Y OF EQUILIBRIA

85

r%

stable

unstable

S

(t~

u(t(J u(t(J u(t(J

state space

state space

Figure 4.14: Stable and unstable different initial conditions.

equilibria.

The trajectories are for

t

Figure 4.15: Time evolution of the state of one neuron.

C H A P T E R 4. F U N D A M E N T A L

86

SYSTEM

system. This solution is independent of time, and is an equilibrium of (4.16). More formally, we define an equilibrium as a point in state space for which dui dt = O, i = 1 , . . . , n . (4.17) As we will investigate convergence, we need a notion of distance in state space. For two vectors u I and u 2, the Euclidean distance is

~/(u~ - u~) ~ + (u~ - u~) 2 + . . .

+ (u~ - u~) 2.

(4.18)

Another distance is (4.19) In general, we will denote by lu 1 - u21 any function obeying tU 1 - - U21

=

0 @ U 1 - - U 2,

lu 1 - u21 in I _ u31

=
0, from Condition 4.2. The main part of the proof consists in showing that the derivative of v along the solutions of the system is positive definite. From the definitions and Condition 4.1,

Ov(u)

-

dr(u)[ dt u

solution of (4.16)

1(

--

i=10~iUi-~

U solution of (4.16)

=

~,ui-i=1 #i

-ui + ~Tofj(uj) j=l

lu, I < r.i

= 1,...,n.

) (4.25)

4.6. A LIAPUNOV THEOREM FOR NEURAL N E T W O R K S

93

This expression is entirely in terms of the absolute values Jui], and we will abbreviate Iuit = wi. We can write this expression a s wTRw if we choose for the elements of the matrix R

Rij =

{ m(-l+a,), ~'" ~ai4, #i J

i=j, i ¢ j.

We now will simplify the condition lUll < ri, i = 1 , . . . , n . define r=min(ri), i=l,...,n,

(4.26) If we

then lul < r will imply l u d < ri, i = 1 , . . . , n. This can easily be verified for the norms based on the Euclidean distance (4.18). From now on, we will use the Euclidean norm lul =

ui. i=l

So far, we have found the following upper bound for Dv D r ( u ) < w TRw,

lul < r.

(4.27)

We now want to relate the matrix R to the matrix S from Condition 4.2. First consider the following equality. -

w.

(4.28)

This can be verified by calculating the coefficient of wiwj in left and right hand sides. One finds rijWiWj -~ r j i w j w i --

ri d + rji rij + rjl 2 w i w j ~2 wjwi.

If we denote by t ( S ) the largest eigenvalue of S, the matrix in Condition 4.2, and if ]u I < r, Dv(u)

0.

(5.1)

The digital form of neural networks is usually derived from another set of equations, very similar to (5.1), apart from the fact that the function fj is now in front of the summation sign [56],

The thresholds Oi, i = 1 , . . . , n, are independent of time. In order to relate the set of equations (5.2) to (5.1), consider the transformation rt

~, = ~ T , jxj,

i= 1,...,n,

(5.3)

j=l

and assume that the determinant of the synapse matrix is nonzero, so that the transformation is invertible,

det T # 0.

(5.4)

If the left and right hand sides of (5.2) are multiplied with Tij and summed over j, one obtains n

ET~j~xJ(t) j----1

T~jzj(t) +

at

j=l

T~sfj

Tjkxk(t)--0j

,

j=l

i = 1,...,~.

(5.5)

or,

d~(t) dt

n

- -w~(t) + ~ Tijfj(wj - 0j) j=l

(5.6)

5.1. FROM ANALOG TO DIGITAL NEUJ~AL N E T W O R K S

99

We can now perform the transformations ui = wi - Oi, Oi = -Ii, and t' = #t,# > 0. The last transformation is a change in clock speed. Equation (5.6) now becomes n

dui(t) -ui(t) + ~ Tijfj(uj(t)) + Ii(t), # dt j=l

i = 1,...,n,

# > O,

(5.7) which is entirely similar to (5.1), apart from the fact that the time constant or decay rate ~ is now the same for all neurons. The conclusion up to now is that the dynamics (5.2) is equivalent to the dynamics (5.1) if the time constants ~t~ are the same for all neurons, the determinant of the synapse matrix is nonzero, and the external inputs Ii are independent of time. It is now easy to discretize the time in (5.2). Use the approximation

dxi(t) _ xi(t + 1 ) - xi(t), dt

(5.8)

and (5.2) can be written as

xi(t+l)=fi(~Tijxj(t)-Oi),j=l

i=l,...,n.

(5.9)

The digital version of the network is now obtained by letting the slope of fi in the origin become infinite. The dynamics of the network are then described by

xi(t+l)=sgn(~Tijxj(t)-Oi),j=l

i=e,...,n,

(5.10)

with sgnx=

1, x k 0, --1, x < 0 .

(5.11)

The sgn function is a threshold function, and is studied in detail in Chapter 6. The sgn function can also be,. mapped into a Boolean function, this is studied in Section 7.2. The network dynamics (5.10) are particularly easy to understand and to implement on chip. This is one of the reasons for the success of

C H A P T E R 5. S Y N C H R O N O US A N D D I S C R E T E

100

neural networks. Despite the simplifications, they have retained much of the rich dynamical behaviour of the non-linear systems (5.1). This is the subject of the rest of this book.

5.2

Synchronous dynamics and cycles

A network operating in discrete time can function in two modes. They are called synchronous and asynchronous. In asynchronous mode, the equation

xi(t + 1 ) = s g n

(5.12)

Tijxj(t)-Oi

is applied for a neuron i chosen randomly for each time step. This random updating makes the analysis quite difficult. There is no simple answer, for example, to the problem when to stop updating the neurons. This opens up a whole unexplored field of research. In the sequel, we will assume synchronous updating, i.e. the set of equations

0)

.

(5.13)

is applied for all i, so that all neurons are updated every time step. To investigate the convergence behaviour of the network (5.13), we will need to construct an energy function, as in Chapter 4. There is no recipe for an energy function, and we make the following choice n

E(x(t))

=

4=1

=

n

n

- ~ xi(t) E T4jxj(t - 1) + E Oi[xi(t) + xi(t - 1)], j=l

4=1

- x T ( t ) T x ( t - 1) + 0T[x(t) - x(t - t)],

(5.14)

where x T is the transpose of the column vector x. In the definition (5.11) of the function sgn, the value 0 has been arbitrarily assigned a sgn equal to 1. This arbitrariness is a very deep problem in the theory of neural networks. For networks in discrete time, we wilt suffice with

5.2. S Y N C H R O N O U S D Y N A M I C S A N D CYCLES

101

T h e o r e m 5.1 In the network (5.13), the thresholds can always be adjusted so that the argument of the sgn function is never zero, and without disturbing the trajectories of the network.

Proof. Assume that for certain values of the state vector x and for certain neurons i the argument ~ j Tijxj - Oi of the sgn function is zero. As there are 2 ~ states and n neurons, it is possible to calculate the smallest value of the absolute value of the argument that is non-zero. This value can be mathematically expressed as

I min E T i j x j - Oil, x,, j [

E Tijxj - Oi 7~ O. j

Call this value e. If we now subtract e/2 from all thresholds Oi, i = 1 , . . . , n,

O~ = O i - e , then

0

i=l,...,n,

00

as all arguments ~ j T i j x j - 0 i of the sgn function will keep the same sign. The arguments ~ j T i j x j - 0 i that were zero and had a sgn 1, are now equal to e/2, and still have a sgn equal to 1. This proves that nothing has changed to the dynamics of the network. Q.E.D. We are now ready to prove that the function (5.14) is indeed an energy function, i.e. T h e o r e m 5.2 The energy E(x(t)) = - E xi(t) E T i j x j ( t - 1) + E Oi[xi(t) + x i ( t - 1)1 i j i

is decreasing during the synchronous operation of the network (5.13), if T is symmetric.

CHAPTER 5. SYNCHRONO US AND DISCRETE

102

Pro@ E(x(t))

For a symmetric synapse matrix T, -

E(x(t-

1)

= -Exi(t)Erijxj(ti

1)+

~-~Oi[xi(t)+ xi(t-

j

+ ~ xi(t-

1) ~_,Tijxj(t

i

-

2) + ~

j

1)+

•[xi(t)- xi(t-

j

+ E xj(tj

Oi[xi(t-

1) +

x,(t-

2)]

i

= -}-~xi(t)ET~jxi(ti

1)1

i

2)]0i

i

1)E i

Tijxi(t-

2)

We have to establish the sign of this expression. Assume i is fixed, equivalent to choosing one term from the sum over i in (5.15). Listing all possibilities for this term we can establish the following table xi(t-2) x i ( t - 1 ) xi(t) x i ( t - 2 ) - x i ( t ) s g n ( ~ T i j x j ( t - 1 ) - O i ) -1 -1 -1 0 -1 -1 -1 1 -2 1 -1 1 -1 -1 0 -2 -1 1 1 1 2 1 -1 -1 -1 1 -1 1 0 1 1 1 -1 2 -1 1 1 1 0 1. In this table, sgn(~jTijxj(t- 1 ) - O i ) was derived from the transition x~(t - 1) --~ x~(t). It was also assumed that the thresholds were adapted so that the argument of sgn was never zero. That this is possible was proven in Theorem 5.1. It follows from the table that

--[xi(t)-xi(t--2)](~j Tijxj(t-1)-Oi) 2, then x(2) -# x(0). In the table in the proof of Theorem 5.2, single out the rows for which x~(t) ¢ x~(t- 2). For the remaining rows, you can see that the energy is strictly decreasing, or

E(x(1))- E(x(0)) < 0. This contradicts (5.16), so that we conclude that the period p _< 2. Q.E.D. There exist similar results for non-symmetric matrices. They are analyzed with the same techniques used for symmetric synapse matrices

[33]. 5.3

Problems

1. If the matrix T is symmetric and has zero diagonal, is its determinant always different from zero?

104

C I t A P T E R 5. S Y N C H R O N O U S A N D D I S C R E T E

2. Is it possible to circumvent the restriction that the time constants have to be equal when establishing the equivalence of the neural networks (5.1) and (5.2)? (This is a difficult problem) 3. Is the discretization error in (5.8) somehow dependent on the time constant #? 4. What are the essential differences between an analog and a discrete neural network model? Discuss the advantages and disadvantages of both. 5. Are all equilibria stable in a network with discrete neuron states and discrete time?

Chapter 6 Linear Capacity In this chapter and the following, we will concentrate exclusively on the number of attractors or equilibria that can be stored in the network. This number is called the capacity of the network, and it is one of the most important characteristics of a network. If a neural network is used as an associative memory, the first property the user will want to know is how much can be stored in the memory. There are several ways to define capacity, dependent on what sort of stability is required for the equilibria. In this book, we follow an approach which we think is most useful for engineers [4]. The reader should know that there exists a whole parallel field of investigation into the capacity and many more aspects of neural networks, based on the statistical physics of spin glasses [7]. In this chapter, we will prove one result, namely that the capacity is linearly proportional to the number of neurons. In preparing for this result, we will encounter several concepts and techniques which are useful in their own right. Hyperplanes, for example, are also used in the analysis of multilayer feed-forward networks. We will also compare the number of threshold functions with the number of Boolean functions. This is a classical result in engineering mathematics, and is useful in the comparison of networks with threshold nodes and networks with Boolean nodes [5]. The linear capacity result in itself is a very general result, encompassing the more specific results that will be derived in Chapter 7, and some results from spin glasses.

CHAPTER 6. LINEAR CAPACITY

106

6.1

Threshold Functions

In order to calculate the capacity of a network, we need to know how many really different networks there are. If you multiply all weights in the network

xi(t+l)=sgn(~Tijxj(t)-Oi),j=l

i=l,...,n,

(6.1)

by the same positive number, for example, the new network will have the same attractors and the same trajectories as the old one. In order to clarify the equivalence of networks, we will study the functions

in some detail. A function f with values 1 and -1 is called a threshold function if the inverse image of 1, f - l ( 1 ) , and o f - l , f - a ( - 1 ) , are separated by a hyperplane. The geometric space we work in is the state space, and because only the sgn function is used, this space is reduced to the corners of the hypercube [10]. For n neurons, we denote this state space by { - 1 , +1} n. It consists of vectors with n coordinates, all equal to +1 or -1. The space { - 1 , +1} '~ is sometimes called n-dimensional binary space, or ndimensional bipolar space. A hyperplane ~r in this space has equation

a.x=0,

x E {--1,+1} ~,

a E R ".

This concept is illustrated for two dimensions in Fig. 6.1, and for three dimensions in Fig. 6.2. The threshold function f does not have to defined on all 2 '~ points of { - 1 , +1} n. We will understand this better if we can calculate B~,

the number of threshold functions of n binar9 variables defined on rn points. In order to calculate B,~, we will augment the dimension of the space, so that all hyperplanes go through the origin. Instead of the n

6.1.

TItRESHOLD FUNCTIONS

107

l-

0: ° i

f'I(_i)

, i

IP,

f-1 (1)

-l Figure 6.1: A threshold function in one dimension.

.

.

.

.

.

.

.

.

.

.

/i ~f"~ /. /"

/

-11. I

i

Figure 6.2: A threshold function in two dimensions.

C H A P T E R 6. L I N E A R C A P A C I T Y

108

variables x l , . . . , xn, we consider the n + 1 variables x a , . . . , x~, 0, and

a.x-O=O,

(x,O) c { - i , + l } n

×R

(6.3)

is the equation of a hyperplane in n + 1 dimensions, going through the origin, because xi = 0, i = 1 , . . . ,n, /~ = 0 is a solution of (6.3). See Fig. 6.3. x2 aeX =0

J.

Figure 6.3: A hyperplane in two dimensions is a straight line. Assume now that the threshold function f is" defined on m points x l , . . . , x m. These are points in n-dimensional space. Up to now, in the equation a . x - 0 = O, we have considered x and 0 to be the variables. We will now consider a l , . . . , an, 0 to be the variables. This is motivated by the fact that, further on in the calculations, ra will be the number of equilibria, and x l , . . . , x m equilibria or patterns will be given by the user, and fixed. T h e set of equations

a.x i-0=0,

i=l,...,m

(6.4)

defines m hyperplanes in n + 1-dimensional space. Refer to Fig. 6.4 for an illustration. If you take into account that the lines drawn in Fig. 6.4 are hyperplanes separating the inverse images of +1 and -1 under the threshold function, it becomes clear t h a t two regions as indicated in Fig. 6.4 correspond to different functions because, if x i as a point is m a p p e d to -t-1 in one region, it is m a p p e d to -1 in the other region.

6.1. THRESHOLD FUNCTIONS

109

ie I

a1

/

I '

Figure 6.4: Region 1 and region 2 correspond to different threshold functions. R e m e m b e r from elementary geometry that a line a • x - ~ = 0 separates a plane in two halves, with a . x - ~ > 0 for all points in one half plane, and a . x - ~ < 0 for the points in the other half plane. Again referring to Fig. 6.4, all values (ai, ~) in the same region correspond to the same function. Two points in different regions correspond to different functions as at least one x ~ of the m vectors x l , . . . , x m is m a p p e d to +1 by one function and to -1 by the other. W h a t we have established here is a link between the number of regions in a space and the number of threshold functions. More precisely, if C~+ 1 is the maximum number of regions in n + 1-dimensional space, made by m hyperplanes through the origin, then

BnTM ~_~C : + 1 .

(6.5)

We will now try to calculate cnm+l . First, we will establish a recursion equation, and then solve this equation. For clarity, this is formulated a two theorems. Theorem

6.1 If C~+1 is the maximum number of regions in n + 1-

dimensional space, made by m hyperplanes through the origin, then Cnm+l = Cnm-1 -~ Cnm+~1.

CHAPTER 6. LINEAR C A P A C I T Y

110

Proof. Consider cnm+~1, the number of regions made by m - 1 hyperplanes. Now add an m - t h hyperplane to make the most possible number of new regions. Refer to Fig. 6.5.

f

1

m-th

plane

Figure 6.5: How the m - t h plane intersects m - 1 planes in at most m - 1 hyperlines.

The m - t h plane intersects the m - 1 planes in at most m - 1 hyperlines. Hyperlines have one dimension less than hyperplanes. In three dimensions, for example, the hyperplanes are planes, and the hyperlines are lines. T h e hyperlines in n + 1-dimensional space are hyperplanes in ndimensional space. T h e m - t h plane itself is a space of n dimensions, containing the m - 1 hyperlines. We can now use the definition of the numbers C~+~1, to conclude that the m - 1 hyperlines divide the m - t h plane in at most C ~ -1 regions.

6.1.

THRESHOLD FUNCTIONS

111

In Fig. 6.5, you can see that region 1 has been split in two by the m-th plane. In general, in the n + 1 dimensional space, one region is added per region in the m-th plane. As there are at most C~ -1 such regions, we find the following recursion equation

Cnm.{_l~---Cnm-I -Jr-C:.t_~1.

(6.6)

Q.E.D. We now have to solve this recursion equation. T h e o r e m 6.2 If Cnm÷l i8 the maximum number of regions in n + 1dimensional space, made by ra hyperplanes through the origin, then m Cn+l = 2~ i=o

m

1 i

"

Proof. The recursion equation (6.6) is a two-dimensional difference equation, the two independent variables being n and m. Just as for a two-dimensional partial differential equation, we will need to find boundary conditions in both variables. For n = 0, Cnm+iis about m hyperplanes in dimension one. These hyperplanes are points. These points divide the one-dimensional space, a line, in two half lines, so that

C~-2. For n = 1, the hyperplanes are lines in a two dimensional space, so that C~ = 2m,

(6.7)

what can be easily verified by drawing m lines through the origin and observing that they divide the plane in 2m regions. These are boundary conditions in the varia,ble n. As the maximum number of regions in a one-dimensional space is always 2,

cnl+l = 2, providing a boundary condition in m. Having established boundary conditions, we need to find a general solution to the recursion equation (6.6). This is usually found with

112

C H A P T E R 6. L I N E A R C A P A C I T Y

the help of generating functions [36], but we will use a more intuitive approach. F i r s t ' ° b s e r v e t h a t t h e b i n ° m i a l c ° e f f i c i e n t ( i sm a s ° l )u t i ° n n + l of (6.6). But then, ( m 7~- 1 ) is also a solution. We will therefore look for a general solution of the form

Za, ( P,q

\ m

-

n+l

p

-q } "

(6.8)

Using the initial conditions and the recursion equation (6.6), we can make a table of C~+ 1. For simplicity, we will list C~+1/2 , and we will also list the differences between C~+1/2 and C ~ / 2 . 0 1 2 3 4 mJ~n~ 1 1 1 1 1 1 1 1 2 2 2 2 2 1 2 3 1 4 4 4 3 1 3 4 3 7 1 8 8 4 1 4 5 6 11 4 15 1 16 5 1 5 6 10 16 10 26 5 31 6 Observe now that the differences form the table of binomial coefficients! This means ,+x = C,~ + 2

.

n

(6.9)

Combined with the boundary condition C~+ 1 1 = 2, this gives

Q.E.D. For more background to these calculations, refer to [65]. We can now combine the inequality (6.5) and (6.10) into T h e o r e m 6.3 If B ~ is the number of threshold functions of n binary variables defined on m points, then

--

i=o

i

"

6.2. LINEAR C A P A C I T Y

6.2

Linear

113

Capacity

The equations governing the operation of the network we are studying are

x i ( t + l ) = s g n ( ~ Tji=jlx j ( t ) - O i ) ,

i = 1,...,n,

(6.12)

with

1, x > O , sgnx=

-1,

x 2 m. --

(6.16)

We always assume t h a t the binomial coefficients are zero w h e n the p a r a m e t e r s are out of range, e.g.

(°) b

=0,

a n, then (6.16) implies

i=o

m-1 i

)m< l( ----" i=0

rn-1 i

)

= 2m_1.

(6.17)

From (6.16) and (6.17) it would now follow t h a t 2 m > 2 "~, a contradiction, so t h a t our assumption must have been false, and m _< n. Q.E.D. T h e requirement t h a t T be zero-diagonal is a very severe one. Indeed, if T was the identity matrix, more or less the opposite of a zerodiagonal matrix, t h e n condition (6.13) becomes xi(t)=sgn(xi(t)-Oi),

i=l,...,n,

which is trivially fulfilled for Oi = O, i = 1 , . . . , n. This means t h a t , if T is the identity matrix, the capacity m = 2 n, exponential in the n u m b e r

6.3. PROBLEMS

115

of neurons. This is true for many variants of neural networks. If the capacity is superlinear, the diagonal of T is usually non-zero. To calculate the capacity, one could also look for the eigenvalues of T. But this is complicated by the non-tinearity of sgn, and by the fact that only eigenvectors with positive eigenvalues are fixed. Unlike Chapter 5, we have never assumed in this chapter that the matrix T was symmetric. This makes Theorem 6.4 very general indeed. The formulation of Theorem 6.4 is subtle, the condition for every set being very important. As we saw on page 19, one particular n × n matrix T can have more than n fixed vectors, and they are not spurious patterns, because we did not consider learning.

6.3

Problems

1. What boundary conditions does the equation C : + , = C : -1 "Jl-C:+~1

(6.18)

need? . Describe how a dynamic, fully interconnected, neural network recognizes patterns. Do this without giving practical examples. Then, in ~ second part of your answer, show how some practical problems can be reduced to pattern recognition (you will of course give examples in this part). 3. Are the weights in a network derived from a biological concept? . Suppose that a neural network is used to segment a medical image. Assume an idealized case where you always expect to find the same number of segments, and each segment has the same number of "typical" contents, e.g., tumor present or no tumor present. To which of the following will the capacity of the network be most relevant: the number of segments, the number of "typical" contents per segment, or the product of both?

Chapter 7 Capacity from a Signal to Noise Ratio In this chapter, we answer the same question as in the last one, namely how many equilibria or fixed vectors the network has. The difference is that we have made a particular choice for the weight matrix, using the sum-of-outerproducts [45]. Because an extra restriction is imposed on the weight matrix, one expects to find a lower capacity, and we will see that this is indeed the case. The sum-of-outerproduct rule is a popular choice for the weights, and therefore the calculations in this chapter are of great importance. Moreover, the techniques presented here are used in calculating many other properties of neural networks. We follow an approach loosely based on information theory. The success of this technique has been illustrated in [61, 11, 16, 57, 51, 66, 72, 49]. There exists a complementary theory based on spin glasses, for which we refer the reader to [32, 7, 74, 17, 48, 63].

7.1

Sum-of-Outerproduct Weights

We will be studying the network xi(t + 1) = sgn (j=,

Tijxj(t)) ,

i-

1,...,n,

(7.1)

118

CHAPTER

7. C A P A C I T Y

FROM SIGNAL TO NOISE RATIO

with 1, x > 0 , -1, x -1

j ¢ i fl¢o~ "~i ",j ",i "~j

lim n.---~oo,(n-1)(m--1).-}oo ~ 1

--

~i > 0 = 1 < ,"

)

= 1 ' ;""

f_,~1 e - ~~2 dx = 1.

(7.13) (7.14)

The integral is visualized in Fig. 7.1 The limit will be one if the variance of the probability distribution becomes very small, as indicated in Fig. 7.2. The stability condition (7.14) has to hold for n independent bits, so that the condition for stability of the complete pattern a is n~,(~-l)(,~-l)~

1 e-~--~" dx

= 1.

(7.15)

121

7.1. S U M - O F - O U T E R P R O D U C T W E I G H T S

x)

0.8

0.6

0.4

0.2

0 -3

-2

-I

0 X

1

2

Figure 7.1: The shaded region is the probability that x _> -1.

exp(-(20*x*x))

-

-

0.8

0.6

0.4

0.2

J

-3

-2

-I

i,

0 x

1

2

Figure 7.2: For small cr, Pr(x > - 1 ) nearly equals 1.

122

C H A P T E R 7. C A P A C I T Y F R O M S I G N A L T O N O I S E R A T I O We want to prove now that lim 0.2 = O. n---+oo,(n-1)(rn--1)--+oo

This does not follow trivially from (7.11), because we do not know how m varies with n. We will assume tim 0.2 = c > O, n--+oo,(n--1)(m-1)--+oo and try to derive a contradiction. If we know that the limit of the variance is c, we can use this in (7.15), which becomes n---+~

2V/'~

I e

= 1.

2c d x

(7.16)

This is impossible, because

~

1

/_)o

x~

le 2o&

(7.19)

-,'

(7 . 20)

1

lira

n--*oo,(n-1)(rn-1)-.+eo lim

n~re-~=O

n V / - ~n-

= o.

',,

(7.21)

T h i s last e q u a t i o n n o w defines m implicitly as a f u n c t i o n of n. T h e r e is no s t r a i g h t f o r w a r d w a y to solve for m , a n d we will m a k e s o m e guesses, a n d check w h e t h e r t h e limit is zero. F i r s t guess m - 1 = n. W e find 1

lira n e - ~ = e c ,

(7.22)

n--+oo

so we r e d u c e m for o u r s e c o n d guess, m - 1 = n~ I n n . W e o b t a i n

1

In u

lim n ~ e '~ x/ln n

~ - -

n

2 = lim ,~--+oo ~

= lira Vl-nnn = oo. ~--,~

(7.23)

For o u r t h i r d guess, we r e d u c e m even more, to m - 1 = n/(21nn). T h e limit is n o w 1

l i m n ¢gi7

- k~n

=

lira

1

v ln n

0,

(7.24)

a n d we h a v e finally f o u n d a valid value for m. R e m a r k t h a t for m - 1 = n/(21nn), n --* oo implies ( n - 1)(rn - 1) --* oc. It m a y b e p o s s i b l e t h a t a value for rn - 1 b e t w e e n n~ i n n a n d n/(2 In n) gives a zero limit, b u t we will b e c o n t e n t here w i t h t h e solution r n - 1 = n/(21nn). As a c o n c l u s i o n , we h a v e p r o v e n t h e following 7 . 1 A pattern will be stable with probability I for a Hopfield network with sum-of-outerproduct weights, if n -+ oo and the number of patterns m obeys the condition

Theorem

n - - ° m - 1 _< 2 I n n

(7.25)

124

C H A P T E R 7. C A P A C I T Y F R O M SIGNAL TO NOISE R A T I O

7.2

C a p a c i t y D e p e n d e n t on R e p r e s e n t a tion

Up to now, we have always been using +1 and -1 for the states of the neurons. This is not common, in digital electronics one usually works with the binary 0 and 1. One can of course always transform a pattern v" E {0, 1} n to a pattern x" E { - 1 , +1} ~ via the transformation 2v~ - 1 = xT, i = 1 , . . . , n , but there is much more to that. We use here a derivation from [12]. In the following, v will always be used for a pattern with bits with values 0 and 1, and x for a pattern with +1 and -1 bits. With the v-notation, the sum-of-outerproducts rule (7.3) becomes m

T i j = -I E ( ( 2 v ~ - l ) ( 2 v ; - 1 ) - S i J ) ,

i= l,...,n,

j= l,...,n.

a=l

(7.26)

The condition (7.5) for a stable bit i becomes T i j ( 2 v ~ - 1)

( 2 v ] - 1) _> 0.

(7.27)

\j=a We will change the state of neuron j to [(1 - a) + ( 1 + ~)(2vj - 1)1,

0 _
O. We choose to keep the weights (7.26) in the same form.

(7.28)

7.2. C A P A C I T Y D E P E N D E N T ON R E P R E S E N T A T I O N

125

Just as in the preceding section, we switch from v~ to uniformly distributed random bits "y~'. We can now substitute the weights, and split into a signal and noise term, taking into account that (23`/z 1)(27/z -- 1) - 5ij = O. Pr

{I lm ((2v/~ =1 n ~__

1)(2v~ - 1 ) - 5ij)

1

×5[(1-t)+(1+t)(2v Pr

2-1)]

] ( 2 v ~ - l ) > 0 } =1.-',~

~',- (7.29)

~n Y'~(21'~ - 1)(27~ - 1)(272 - 1)[(1 - 1) + (1 + t)(2')' 2 - 1)] 1

n

ra

+ N E E(23`

- 1)(2 g - 1)(2 9 - 1)

j----1 fl:/:a

x[(1 - t) + (1 + t)(2-y 7 - 1)]] ___0} = 1 . . ' ~

(7.30)

~ [ ( 1 + t) + (1 - t)(2-y 7 - 1)]

Pr

j¢i +~n

( 2 3 ' 7 - 1)(23`g- 1)(27jz - 1) j=l

x[(1

-

t ) + (1 + t)(23` 2

-

1)]] _> 0} = 1.

(7.31)

This is again a sum of a signal term and a noise term. Let's first analyze the signal term 1

n

2-~ ~ [ ( 1 + t ) + (1 - t)(23` 2 - 1)]. j¢i The random variable 2 7 ~ - 1 has mean 0, so that (1 + t ) + ( 1 - 1 ) ( 2 3 , 7 - 1 ) has mean 1 + 1. The variance +1

[(1 - 1)(277 - 1)]2 P r ( 7 7 ) = (1 - t ) 2

(7.32)

~7=0 From this result follows that ~[(1 1 + 1) + (1 - A)(27~ - 1)] has mean (1 + A)/2n and variance (1 - A)2/4n 2.

CHAPTER 7. C A P A C I T Y FROM SIGNAL TO NOISE RATIO

126

We can now use the central limit theorem to find that the signal term 1

n

2-~ ~--~.[(1 + 3`) + (1 - 3,)(272 - 1)] j¢i

(7.33)

has a Gaussian distribution with mean n--

#-

2~ 1 (1 + $),

and variance

1

n 4n 2

(7.34)

(1 - 3`)2.

(7.35)

The noise term n

2--~

m

Y'~(23~ - 1)(23/~- 1)(23jz - 1)[(1 - )~) + (1 + 3 ` ) ( 2 7 ; - 1)1

j ¢ i fl•a

is a sum of (n - 1)(m - 1) random variables

1--~-2' ( ~3i 2n . 1)(2"//z .

1)(23~ .

.1)[(1

. 3`)+(1+3`)(23'2

1)].

(7.36)

They are not independent, but as in the preceding section, we will suppose that they are. The random variables (7.36) have mean 0, and variance

111

4~2

[(1 - 3`) + (1 + 3`)]2 +

[(1 - 3`)

(1 + 3`)]2

]

1 + 3`2 2n 2

Using the central limit theorem, we find that the noise term has a Gaussian distribution with mean 0 and variance cr 2 = (n - 1 ) ( m - 1 ) ( 1 + A2).

(7.38)

2n 2

Condition (7.31) is very similar to (7.13). The signal term now has a variance (7.35), but this variance becomes zero in the limit n c¢. If we assume the limit has been taken, so that the variance is zero, condition (7.31) contains the area under a Gaussian probability distribution, as illustrated in Fig. 7.1, but now with - # instead o f - 1 for the lower boundary.

7.2.

CAPACITY

DEPENDENT

127

ON REPRESENTATION

If we now also take into account that n independent bits have to be stable, condition (7.35) has become equivalent to (remark the similarity with (7.15)) ,~,(,~-1)(m-1)--.~

. e - ~- ~ dx

~

n.-*oo,(n-1)(m-1)--,,oo

lim n-~oo.(.-1)(m-1)-~

1

1 2

oo e - ~

1 ~erf

-#

= 1 -: :- (7.39)

dx

= 1

1

< :.

= 1 -'

lim ~1 ~--,oo,(,~-1)(,~- 1)--~

=1.

.

(7.40)

The argument of the error function is !

#

V/'20"

-

n - 1 (1 + A) | 2n2 2n ~ (1 + A 2 ) ( n - 1 ) ( m 1 1+,~

=

1,/v

1 1)V~

/n-1

(7.41)

Vm- 1'

and in order to simplify calculations we will suppose that this argument is large as n ~ oc, i.e. m increases slower than n. Condition (7.40) is now

n-+oo,(n--1)(m--1)--~oo

vf~

=1

--e~t

O"

lim

n-..,,oo,(n-1)(m-1)--*oo

lim

n

x/-f + A ~ ~

I+A

V

2

n - e - 2~-v-~= 0 #

- 1 _~ 0+~ 2 .-1

n

,~ ',-(7.42)

e ~ 1+~ ,~-1 ____O.

(7.43)

We now have again to guess the solution of this equation for m as a function of n. We will immediately give the right guess, but the reader should do some experiments on his or her own. For the guess m-l=

n (1 +)~)2 l n n 4(1 + )~2)

CHAPTER 7. CAPACITY FROM SIGNAL TO NOISE RATIO

128

the limit (7.43) becomes n ,~--,oo lira 2 ~

1 e - han = lim - -0. n--,~ 2 ( I n n

(7.44)

We can now formulate this result as a little theorem. T h e o r e m 7.2 A pattern will be stable with probability I for the Hopfield network with sum-of-outerproduct weights, and with the output of the neurons ~[(1 - ),) + (1 + A)xj], 0 o,

(8.9.)

It can be verified that Pr(~dt + r) = +1) + Pr({i(t + T) = --1) =

2 + e '~F~ + e -AF~ (1 + e-e~F, )(1 + eZ,~ )

=1. (8.3)

8.1. SIMULATED ANNEALING

133

In order to understand the meaning of the number/~i, consider the formula (8.2) in the limit ¢~i --+ co.

Pr(~i(t + T ) = +1)

Pr(~i(t + T ) = --1)

1 Fi(t)>0, 0 F~(t) < 0, 7 F~(t) = 0,

=

0 Fi(t) > O, 1 Fi(t) < O,

=

(8.4)

7' F~(t) = 0,

For F~(t) ~ O, this

is the deterministic network (5.10). It is possible to avoid the situation Fi(t) = 0 by an adjustment of the thresholds 0i, as was explained in Theorem 5.1. Consider now (8.2) in the limit fli --* 0. Pr({i(t + r ) = +1)

-

Pr(~i(t + T ) - - - 1 )

=

1 1

2

(8.5)

This shows that for/3i -4 0, the neurons are throwing unbiased dice, they are not influenced by their input F / a n y more. We could call this chaos, but it is different from the chaos that can occur in the networks studied in Chapter 4. We will now try to relate /3i to the evolution of the energy. A deterministic equiva.lent of (8.2) would be

xi(t + v) =sgn (~--~

xj(t) +1

Oil.

]

(8.6)

÷ ~ xj(t)oj.

(8.7)

2

A candidate for the energy function is

j-_lk=l

2

2

j=l

As usual with an energy function, we will assume that the synapse matrix T is symmetric.

134

CHAPTER 8. NEURAL NETS AND MARKOV CHAINS Calculate now the partial derivative

OE Oxi

l ~ TikXk + l -

:

2

k=l

j=a

I @'T. Xj +I - -

2

2~=1

2

00 "

s,

2

+0i,

(8.8)

Switching to finite differences, this formula becomes

AE=-Axi(

~-~Tijxj+lj=l 2

0i)

(8.9)

One can verify now that, if ~j TiJ xJ+12- Oi >_ 0, then Axi >_ 0, because of (8.6), and A E _< 0. Similarly, if ~2j T~j ~j+l _ 0~ < 0, then Ax~ _< 0, and again A E _< 0, proving that the energy E decreases during the operation of the network. In the probabilistic situation, however, with the rule (8.2), the condition ~j TiJ x~+12-Oi _> 0 can result in xi(t + T) = --1 SO that, if xi(t) = 1, Axi < O, and A E >_ 0. We see here that the energy can increase if the network operates probabilistically! This is illustrated in Fig. 8.1.

Figure 8.1: The energy can increase during simulated annealing.

8.1. S I M U L A T E D A N N E A L I N G

135

How likely is the increase in energy? Without doing complicated calculations, one sees that the energy increases if (8.2) does not obey the sgn rule. This is most likely for small/~i. So far, we have seen that the probabilisticalty operating neuron (8.2) becomes deterministic for t 3 / ~ co. As/3i becomes smaller and smaller, it becomes more and more likely that the energy will increase. For /34 = 0, the neuron i behaves like an unbiased dice. This is true for every neuron. This leads to the interpretation of 1//~i as a temperature. For low temperature, the system is deterministic, for high temperature, the system is noisy, and chaotic. We have proven T h e o r e m 8.1 In a probabilistic network, the energy can increase if the temperature is not zero. This interpretation is extremely important for applications where a neural network is used such that its energy function corresponds to a cost function. By letting the network evolve in time, the cost function is minimized. Usually, one finds a local minimum. To find the global minimum, the temperature is increased, so that the energy can increase. After a while, the network is in another region of the state space, and the temperature can be decreased again. By timing this process carefully [t], one will find the global optimum. The state of the network is a vector !~ in the state space { - 1 , +1} n. From (8.2), we can calculate the probability that neuron i has a certain value, given the state of the network one time step earlier, Pr

+ T)=

=

{1 + e x p - / 3 i x ~ ( t ) ( ~ T i j j=l

x~(t) 2

+t

o,)]}

, (8.1o)

where a and/~ are subscripts running over all the states, a , f l = 1 , . . . , 2% The probabilities (8.10) are uncorrelated for different i, they depend only on the state one time step earlier. From them, we can calculate the transition rate per unit time, for a transition from state/~ to a state

CHAPTER 8. NEURAL N E T S AND M A R K O V CHAINS

136

c~ in the state space, R~O = =

1

t

T

T

1.{

-O~e=-Pr(~( r ~"=

l+exp

.a, . 13. .1,

[

t+r)=x"[~0(t)=x

-/3ix~(t)

j__ =l

)]}1

0)

ij

2

- - Oi

. , 2".

'

(8.11)

This shows that the states of a neural network follow a Markov chain [19]. The transition rates R~ e _> 0, and even Roe > 0, if/~ < oo. The numbers Q~z are the transition probabilities. From now on, we drop the restriction that the synapse matrix T has to be symmetric.

8.2

The

Fokker-Planck

equation

In a network that operates probabilistically, it is not possible to sketch a particular trajectory in state space, as the next state is only given in a probabilistic sense, via the transition probabilities Q~z in (8.11). What we can talk about in a deterministic sense is the occupation probability of state a at time nr, denoted by p~(nr). This quantity evolves in a deterministic way according to the equation 2n

p~(nr) = ~ Q~Zpz(nT -- r).

(8.12)

/3=1

This can be formulated in words as the occupation probability of state

(~ is the occupation probability of state t3 multiplied by the probability of a state transition from ~ to ~, and summed over all states/3. The matrix Q is called stochastic if its column sums equal 1. This means that the sum of the probabilities to go from state/3 to any other state is 1. T h e o r e m 8 . 2 The matrix Q with elements Qoe/3

{

II

i=1 a,/3

1 +exp _=

1,...,

-flix i 2n

j=l

Tij x~

+ 1

o,)]} -1

2 (8.13)

8.2. THE FOKKER-PLANCK EQUATION

137

is stochastic. Proof. 2 n¸

E Q-, =

o~----1

1

1

1

E

E

"'" E

x~=--I x ~ = - - I

E

1

...

1

l'I1+e-Z,~3F,,

xna=--I i=1

1 x~=-t

n

E

n--1

1

1

II 1 + ~ ~,~,

E

x ~ _ 1 = - I /=1

1

1

n-1

x~=-I

1

1+ ~ - ~ o ~ '

1

E... xn°E1=-1 i=1 II1+~,~,,

x~=-I 1

=

E x~=--I 1 + e - & ~ y~ '

=

1.

(8.14)

Q.E.D. We will now try to write down an equation describing how the occupation probabilities evolve in time. Using the fact that Q is a stochastic matrix, 2n

p~(t + ~ ) - p . ( t )

= ~ Q ~ p ~ ( t ) - p,(t), fl=l 2n

2n

= ~_, Q~zp~(t)- ~_~ Qz~p,(t). fl=l

(8.15)

B=I

Dividing the left and right hand side by T, we find an expression for the finite difference of the occupation probabilities,

p~(t+v)-p~(t)

2" = y~(R~zp~(t)- Rz~p,~(t)). fl=l

(8.16)

This can be formulated in words as follows: the change in occupation probability of state (~ is the probability to be in state fl multiplied by the transition rate of going from fl to ~ minus the probability to be in

138

C H A P T E R 8. N E U R A L N E T S A N D M A R K O V C H A I N S

P9 c~a

Figure 8.2: A picture of the state space, with states a and ft. The probability p~ is represented by a region of size proportional to p~. Either the probabilities Q or the rates R can be used in the drawing. c~ multiplied by the rate of going from a to fl, summed over all states. This is illustrated in Fig. 8.2. In the limit for small ~', equation (8.16) becomes a differential equation, and is called the Fokker-Planck equation, or Master equation, or Chapman-Kolmogorov equation. In this limit, and also for the distinction between the different sorts of equations, there arise many mathematical subtleties, for which we refer the reader to [19]. Finally, we will investigate the steady-state solution of (8.16). When the occupation probabilities are not changing anymore in time, 2n

~_,(R,~zp~ - R~,~p,~) = O,

a = 1 , . . . , 2 n.

(8.17)

~=1

In this system of linear equations, R,Z is known, and the p~ are the unknowns. The equations could be solved numerically, if it was not that there are 2n of them! Instead we will prove T h e o r e m 8.3 The system of linear equations 2n

~ - ~ ( n ~ p z - R ~ p ~ ) = O, /~=1

a-- 1,...,2 ~

8.2. THE FOKKER-PLANCK EQUATION

139

for the steady state occupation probabilities has a non-zero solution. Proof. Consider the matrix W with elements ~n

W~ = R~ - 6~ ~ R~.

(8.18)

This means W~ =

R~,

2"

E.=I

i.e. from each diagonal element of R, the sum of all its column elements has been subtracted. The system (8.17) is now equivalent to 2n

W ~ p z = 0,

a = 1,...,2L

(8.19)

fl=l

Indeed, (8.19) is equivalent to 2n

2n

~--~.(R~,p,-5~P~p~)=O, 3=1

a= 1,...,2".

(8.20)

~'=1

The system (8.19) will have a non-zero solution if and only if det W = 0. That det W = 0 can be verified in the following way.

c~

a

a

"t

a

"~

This proves that the sum of all columns of W is 0, so that its determinant has to be 0. This result is the stepping stone for a further analysis of the steady state occupation probabilities of neural networks, for which we refer to [18].

140

8.3

C H A P T E R 8. N E U R A L N E T S A N D M A R K O V CIIAINS

Problems

1. How can uncertainty be modelled with neural networks? How do you handle uncertainty in data and their classification? Describe what properties of the network play a role, and also give practical examples. 2. What happens to a neural network at zero temperature? Is Theorem 8.2 still valid?

Bibliography [1] E. Aarts and J. Korst. Simulated Annealing and Boltzmann Machines. Wiley, New York, 1989. [2] Ralph Abraham and Jerrold E. Marsden. Foundations of Mechanics. Benjamin, New York, 1967. [3] Milton Abramowitz and Irene A. Stegun. Handbook of Mathematical Functions. Dover, New York, 1970.

[4] Yaser S. Abu-Mostafa and Jeannine-Marie St.Jacques. Information capacity of the Hopfield model. IEEE Transactions on Information Theory, IT-31(4):461-464, July 1985. [5] Igor Aleksander and Helen Morton. An Introduction to Neural Computing. Chapman and Hall, London, 1990. [6] Igor Aleksander and Helen Morton. Neurons and Symbols. Chapman and Hall, London, 1993. [7] Daniel J. Amit. Modeling Bruin Function, The World of Attractor Neural Networks. Cambridge University Press, Cambridge, 1989.

[8] Vladimir Igorevitch Arnol'd.

Geometrical Methods in the Theory of Ordinary Differential Equations. Springer-Verlag, New York, 1983.

[9] D. K. Arrowsmith and C.M. Place. An Introduction to Dynamical Systems. Cambridge University Press, Cambridge, 1990.

142

BIBLIOGRAPHY

[10] Pierre Baldi. Neural networks, acyclic orientations of the hypercube, and sets of orthogonal vectors. Siam Journal on Discrete Mathematics, 1(1):1-13, February 1988. [11] Pierre Baldi and Santosh S. Venkatesh. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters, 58(9):913-916, March 1987. [12] A. D. Bruce, E. J. Gardner, and D. J. Wallace. Dynamics and statistical mechanics of the Hopfield model. Journal of Physics A: Mathematical and General, 20:2909-2934, July 1987. [13] A. N. Chetaev. Some problems on markov chains arising in connection with the modelling of neural networks. Russian Mathematical Surveys, 31(4):77-87, July/August 1976. [14] A. N. Chetaev. Nejronnye Seti i Tsepi Markova (Neural nets and Markov chains). Nauka, Moskva, 1985. [15] P.Y.K. Cheung, A. Ferrari, Ph. De Wilde, and G. Benyon-Tinker. A neural network processor - a vehicle for teaching system design. IEE Proceedings-G, 139(2):244-248, April 1992. [16] Philip A. Chou. The capacity of the Kanerva associative memory. IEEE Transactions on Information Theory, 35(2):281-298, March 1989. [17] Barry A. Cipra. An introduction to the Ising model. American Mathematical Monthly, 94(10):937-959, December 1987. [18] John W. Clark. Statistical mechanics of neural networks. Physics Reports, 158(2):91-157, February 1988. [19] D. R. Cox and H. D. Miller. The Theory of Stochastic Processes. Chapman and Hall, London, 1984. [20] Philippe De Wilde. A Marquardt learning algorithm for neural networks. In A. M. Barb@, editor, Proceedings of the Tenth Symposium on Information Theory in the Benelux, pages 51-57, Enschede, The Netherlands, 1989. Werkgemeenschap voor Informatie- en Communicatietheorie.

BIBLIOGRAPHY

143

[21]

Philippe De Wilde. Time and area requirements for a hybrid learning algorithm. In M. Novak and E. Pelik£n, editors, Proceedings of the International Symposium on, Neural Networks and Neural Computing NEURONET'90, pages 3.54-356, Prague, 1990. Czechoslovak Academy of Sciences.

[22]

Philippe De Wilde. Conditions for active states in neural networks. In A. Holden and V. Kryukov, editors, Neurocomputers and Attention, pages 695-700, Manchester, 1991. Man&ester University Press.

[23] Philippe De Wilde. Class of Harniltonian neural networks. Physical Review E, 47(2):1392-1396, February 1993.

[24]

Philippe De Wilde. Reduction of representations and the modelling of consciousness. In Harald Hfining et al., editor, Aachener Beitriige zur Informatik, Band 3, pages 40-43, Aachen, 1993. Verlag der Augustinus Buchhandlung.

[25]

Philippe De Wilde. Physical and linguistic problems in the modelling of consciousness by neural networks. In J. Mira and F. Sandoval, editors, Lecture Notes in Computer Science 930, pages 584588. Springer Verlag, Berlin, 1995.

[26]

Philippe De Wilde and A.M.C.-L. Ho. Symmetries of general feedforward neural networks and equivalent classification tasks. In I. Aleksander and J. Taylor, editors, Artificial Neural Networks II, Proceedings of the International Conference on Artificial Neural Networks, Amsterdam, 1992. North-Holland.

[27] R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. Wiley, New York, 1973.

[28]

John C. Eccles. A unitary hypothesis of mind brain interaction in the cerebral cortex. Proceedings of the Royal Society of London Series B - Biological Sciences, 240(1299):433-451, 1990.

[29] John C. Eccles. Evolution of consciousness. Proceedings of the National Academy of Sciences of the USA, 89(16):7320-7324, 1992.

144

BIBLIOGRAPHY

[3o] G. B. Ermentrout and J. D. Cowan. Large scale spacially organized activity in neural nets. ,SIAM Journal of Applied Mathematics, 38(1):1-21, 1980.

[31] Wai-Chi Fang, Bing J. Shen, Oscal T.-C. Chen, and Joongho Choi. A VLSI neural processor for image data compression using selforganization networks. IEEE Transactions on Neural Networks, 3(3):506-518, May 1992.

[32] Elizabeth Gardner. The space of interactions in neural networks. Journal of Physics A: Mathematical and General, 21(1):257-270, January t988. [33] Eric Goles and Servet Martlnez. Neural and Automata Networks. Ktuwer, Dordrecht, 1990. [34] I. S. Gradshteyn and I. M. Ryzhik. Table of Integrals, Series, and Products. Academic Press, New York, 1980.

[35] Hans P. Graf, Lawrence D. Jackel, and Wayne E. Hubbard. VLSI implementation of a neural network model. Computer, 21(3):4149, March 1988.

[36] Ronald L. Graham, Donald E. Knuth, and Oren Patashnik. Concrete Mathematics. 1994.

Addison-Wesley, Reading, Massachusetts,

[37] Stephen Grossberg. The adaptive brain, volume I & II. NorthHolland, Amsterdam, 1986. [38] Hermann Haken. Syne~yetics. Springer-Verlag, Berlin, 1983. [39] Simon Haykin. Neural Networks, a Comprehensive Foundation. Macmillan, New York, 1994. [40] Robert Hecht-Nielsen. Neurocomputing: picking the human brain. IEEE Spectrum, 25(3):36-41, March 1988.

BIBLIOGRAPHY

145

[41]

John Hertz, Anders Krogh, and Richard G. Palmer. Introduction to the Theory of Neural Computation. Santa Fe Institute Studies in the Sciences of Complexity. Addison-Wesley, Redwood City, California, 1991.

[42]

A.M.C.-L. Ho and Philippe De Wilde. General transient length upper bound for recurrent neural networks. In J. Mira and F. Sandoval, editors, Lecture Notes in Computer Science 930, pages 202208. Springer Verlag, Berlin, 1995.

[43]

A. L. Hodgkin and A. F. Huxley. A quantitative description of current and its application to conduction and excitation in nerve. Journal of Physiology, 117:500-544, 1952.

[44]

John E. Hopcroft and Jeffey D. Ullman. Introduction to Automata Theory, Languages, and Computation. Addison-Wesley, Reading, Massachusetts, 1979.

[45]

John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the USA, 79:2554-2558, April 1982.

[46] John J. Hopfield and David W. Tank. Computing with neural circuits: a model. Science, 233(4764):625-633, August 1986. [47] Yves Kamp and Martin Hasler. Recursive Neural Networks for Associative Memory. Wiley, Chichester, 1990. [48] I. Kanter and H. Sompolinsky. Associative recall of memory without errors. Physical Review A, 35(1):380-392, 1987. [49] James D. Keeler. Information capacity of outer-product neural networks. Physics Letters A, 124(1/2):53-58, September 1987. [50] Teuvo Kohonen. An introduction to neural computing. Neural Networks, 1(1):3-16, 1988. [51] J£nos Koml6s and Ramamohan Paturi. Convergence results in an associative memory model. Neural Networks, 1(3):239-250, 1988.

146

BIBLIOGRAPHY

[52] V. I. Kryukov, G. N. Borisyuk, R. M. Borisyuk, A. B. Kirillov, and E. I. Kovalenko. The Metastable and Unstable States in the Brain (in Russian). Academy of Sciences of the USSR, Pushchino, 1986.

[53]

Solomon Lefschetz. Differential equations: geometric theory. Wiley, New York, 1963.

[54]

Jian Hua Li, Anthony N. Michel, and Wolfgang Porod. Qualitative analysis and synthesis of a class of neural networks. IEEE Transactions on Circuits and Systems, CAS-35(8):976-985, 1988.

[55]

Ralph Linsker. Self-organization in a perceptual network. Computer, pages 105-117, March 1988.

[56]

C. M. Markus, F. R. Waugh, and R. M. Westervelt. Associative memory in an analog iterated-map neural network. Physical Review A, 41(6):3355-3364, March 1990.

[57]

E. Marom. Associative memory neural networks with concatenated vectors and nonzero diagonal terms. Neural Networks, 3(3):311318; 1990.

[58]

James L. McClelland and David E. Rumelhart. Ezplorations in Parallel Distributed Processing: A Handbook of Models, Programs, and Exercises. MIT Press, Cambridge, Massachusetts, 1988.

[59]

James L. McClelland, David E. Rumelhart, and the PDP Research Group. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 2, Psychological and Biological models. MIT Press, Cambridge, Massachusetts, 1986.

[60]

Warren S. McCulloch. Embodiments of Mind. MIT Press, Cambridge, Massachusetts, 1989.

[61]

Robert J. McEliece, Edward C. Posner, Eugene R. Rodemich, and Santosh S. Venkatesh. The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory, IT33(4):461-482, July 1987.

BIBLIOGRAPHY

147

[62] Carver Mead. Analog VLSI and Neural Systems. Addison-Wesley, Reading, Massachusetts, 1989. [63] Marc M@zard, Giorgio Parisi, and Miguel Angel Virasoro. Spin Glass Theory and Beyond. World Scientific, Singapore, 1987.

[64] Anthony N. Michel, Jay A. Farrell, and Wolfgang Porod. Qualitative analysis of neural networks. IEEE Transactions on Circuits and Systems, CAS-36(2):229-243, 1989. [65] Saburo Muroga. Threshold Logic and its Applications. Interscience, New York, 1971.

Wiley-

[66] Charles M. Newman. Memory capacity in neural network models: Rigorous lower bounds. Neural Networks, 1(3):223-238, 1988. [67] Athanasios Papoulis. Probability, Random Variables, and Stochastic Processes. McGraw-Hill, New York, 1981. [68] Francois Robert. Discrete Iterations: a Metric Study. SpringerVerlag, Berlin, 1986.

[69] David E. Rumelhart, James L. McClelland, and the PDP Research Group. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1, Foundations. MIT Press, Cambridge, Massachusetts, 1986.

[70] Murray R. Spiegel. Advanced Calculus. McGraw-Hill, New York, 1974.

[71] Dawid W. Tank and John J. Hopfield. Simple "neural" optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit. IEEE Transactions on Circuits and Systems, CAS-33(5):533-541, May 1986.

[72] Santosh S. Venkatesh. Robustness in neural computation: Random graphs and sparsity. IEEE Transactions on Information Theory, 38(3):1114-1119, May 1992. [73] Ferdinand Verhulst. Nonlinear Differential Equations and Dynamical Systems. Springer-Verlag, Berlin, 1990.

148

BIBLIOGRAPHY

[74] K. Y. M. Wong and D. Sherrington. Theory of associative memory in randomly connected Boolean neural networks. Journal of Physics A: Mathematical and General, 22(12):2233-2263, 1989.

Index absolute refractory period, 60 action potential, 58 adaptation, 35 address, 22 analog circuit, 74 applications, 47, 53 associative memory, 22, 105 asymmetric, 81 asymptotic approximation, 122 asynchronous, 15, 100 attraction basin, 17 attractor, 17 auto-associativity, 20 autonomous, 76, 132 axon, 54, 81 backgammon, 48 backpropagation, t0, 35 convergence, 47 battlefield robot, 48 binary, 124 binary space, 106 binomial coefficient, 112 bipolar, 124 bipolar space, 106 black box, 23 Boltzmann machine, 131 Boltzmann, L., 8 boundary condition, 111 Boyle, R., 8

broomstick balancer, 48 capacitor, 75 capacity, 17, 43, 105, 113, 123, 128 exponential, 114 superlinear, 115 car navigation, 50 category, 8, 20 cell wall, 54 cellular automaton, 10 central limit theorem, 120, 126 channel, 31 chaos, 28, 133, 135 Chapman- Kolmogorov equation, 138 clamped, 36 classification, 8, 20 cloud, 8 cocktail party, 7 codeword, 20 cognitive science, 10 conductance, 76 connection, 12, 31 connectionism, 10 content, 22 content-addressable, 7 convergence, 16, 20, 23, 82, 103 correlation, 8 cortex, 60

150 cost function, 135 crystal, 8 current input, 75 leakage, 75 cycle, 90, 103 database, 16 decoder, 20 decrescent, 89 degradation, 8 delay line, 49 dendrite, 54, 81 derivative along a solution, 89 deterministic, 28, 131 diagonal, 15, 81 difference equation, 111 digital, 76, 83, 97, 124 digitize, 16 dimension, 17 discrete time, 97 distance, 86 dynamic behaviour, 69 dynamical system, 10, 69, 70 efficacy, 60 eigenvalues, 115 encoder, 48 energy, 133 function, 28, 69, 89, 94, 100 minimum, 76 potential, 28 surface, 28 epoch, 42 equilibrium, 86, 113, 131 asymptotically stable, 86 exponentially stable, 86

INDEX

stability, 82, 86, 89 unstable, 82 error, 26 correction, 17 signal, 39 error function, 122 evolution in time, 69 example, 23, 39 exchange rate, 49 excitatory, 14, 58 exclusive-OR, 48 existence of solution, 70 exponentially stable, 89 external input, 62, 78, 83 face recognition, 48 fail-soft device, 8 feedforward, 23 firing rule, 132 firing threshold, 60 fixed vector, 113 Fokker-Planck equation, 10, 138 forecasting, 49 form, 8 fundamental memory, 17, 20 fuzzy inference, 50 game, 48 gas laws, 8 Gaussian, 119, 121,126 generation function, 112 global behaviour, 69 global property, 8 gradient descent, 46 halting problem, 90 handwriting, 49 hetero-associativity, 22

151

INDEX

hidden layer, 41 hidden neurons, 23 Hopfield, J., 19 hypercube, 17, 26, 106 hyperline, 110 hyperptane, 106, 108 hyphenation, 48 identification, 7 image, 7, 11, 36 compression, 50 distortion, 17 multiple, 42 recognition, 48 independency, 126 information, 56 information theory, 117 inhibitory, 14, 58 initial condition, 70 initial state, 82 input, 23 input-output pair, 39 interconnection matrix, 15 intermediate neurons, 29 Internet, 10 ion, 57 layered network, 23 layout, 31 leakage current, 75 learning, 45 by example, 23 supervised, 23, 35 unsupervised, 10 learning rate, 40 Leibnitz's rule, 80 Liapunov hmction, 89

Liapunov theorem, 89 Liapunov, A.M., 89 limit cycle, 90, 103 linearization, 69, 78 loan application scoring, 49 lobe, 54 local, 69 local optimum, 131 macroscopic, 8 management information system 48 Markov chain, 131,136 Master equation, 138 matrix stochastic, 136 synaptic, 14, 15, 17, 19, 26 mechanical system, 76 medical diagnosis, 50 membrane, 57, 61 memory associative, 7 content-addressable, 7 microscopic, 8 minimization, 26, 69 minimum, 135 molecule, 8 multilayer, 53 multiple patterns, 42 negative definite, 92 nerve cell, 54 network Boolean, 10 feedforward, 23 higher order, 28 Hopfield, 10, 19

152 layered, 23 multilayer, 105 size of, 8 network geometry, 39 neuron, 8, 11, 54 hidden, 23, 41, 42 intermediate, 29 neurotransmitter, 58 Newtons law, 76 noise, 17, 23, 43, 49, 50, 60, 83, 120, 125, 135 non-linearity, 115 non-symmetric, 103 nonlinearity, 26, 46, 66 norm, 86 numerical solution, 72 occupation probability, 136 optimization, 49, 69 optimum, 135 oscillation, 16, 23, 103 output, 23 parallel distributed processing, 10 parallel processing, 7 parameter, 37 pendulum, 76 period, 103 phase transitions, 10 pixel, 8, 11 placement, 31 positive definite, 92 post-synaptic potential, 58 prediction, 49 probabilistic operation, 27 probability, 119 probe, 17

INDEX

proceqsor, 8 protein structure, 49 prototyp~ 20 quadratic, 78 RC circuit, 74 recursion, 111 refractory period, 60, 66 reliability, 8 representation, 124 Runge-Kutta method, 72 saddle point, 83 security, 16, 48 sgn, 106 signal, 49, 120, 125 signal processing, 120 simulated annealing, 131 smoothness, 70 software, 10 soma, 54 sonar, 50 speech recognition, 50 synthesis, 48 spin glass, 10, 105, 117 spiral, 8~ spurious patterns, 28, 129 squashing, 113 stability, 82, 86, 118, 120 starting state, 20 state, 11, 36, 62 clamped, 36 random, 132 spurious, 28 state space, 17, 135 state vector, 11, 82

153

INDEX

statistical physics, 9, 105 steady state, 138 steepest descent, 46 stochastic matrix, 136 stochastic models, 10 stored pattern, 113 sum-of-outerproducts, 117 supervised learning, 23 symmetric, 15, 81,101,115, 118, 133 synapse, 14, 37, 56, 76, 81,113 synaptic bouton, 56 synaptic matrix, 15, 17, 26 synchronous, 15, 100, 103, 132 synergetics, 10 tanh, 27 approximation of, 70, 72 target value, 39 tax, 50 technology, 8 temperature, 28, 135 tensor, 29 threshold, 27, 37, 38, 66, 98, 101, 113, 118, 132 firing, 60 threshold function, 109 time, 12 constant, 75, 81 evolution, 69 time series, 49 time-dependent system, 53 trajectory, 16, 82, 101,136 transfer function, 65, 81, 83, 97 transition probability, 136 transition rate, 135 travelling salesman, 49

Travelling Salesman Problem, 131 uncertainty, 36, 132 update rule, 12, 20, 26, 29, 132 VLSI implementation, 66 voltage spike, 58 weight, 14, 37 adaptation, 39 initialization, 39 update accumulation, 42 wiring, 20 zero diagonal, 15, 114, 118

Lecture Notes in Control and Information Sciences Edited by M. Thoma 1 9 9 2 - 1 9 9 5 Published Titles: Vol. 167: Rao, Ming Integrated System for Intelligent Control 133 pp. 1992 [3-540-54913-7]

Vol. 166: Dorato, Peter; Fortuna, Luigi; Muscato, Giovanni Robust Control for Unstructured Perturbations: An Introduction 118 pp. 1992 [3-540-54920-X[ Vol. 169: Kuntzevich, Vsevolod M.; Lychak, Michael Guaranteed Estimates, Adaptation and Robustness in Control Systems 209 pp. 1992 [3-540-54925-0] Vol. 170: Skowronski, Janislaw M.; Flashner, Henryk; Guttalu, Ramash S. (Eds) Mechanics and Control. Proceedings of the 4th Workshop on Control Mechanics, January 21-23, 1991, University of Southern California, USA 302 pp. 1992 [3-540-54954-4] Vol. 171: Stefanidis, P.; Paplinski, AoP.; Gibbard, M.J. Numerical Operations with Polynomial Matrices: Application to Multi-Variable Dynamic Compensator Design 206 pp. 1992 [3-540-54992-7]

Vol. 172: Tolle, H.; ErsO, E. Neurocontrol: Learning Control Systems Inspired by Neuronal Architectures and Human Problem Solving Strategies 220 pp. 1992 [3-540-55057-7] Vol. 173: Krabs, W. On Moment Theory and Controllability of Non-Dimensional Vibrating Systems and Heating Processes 174 pp. 1992 [3-540-55102-6]

Vol. 174: Beulens, A.J. (Ed.) Optimization-Based Computer-Aided Modelling and Design. Proceedings of the First Working Conference of the New IFIP TC 7.6 Working Group, The Hague, The Netherlands, 1991 268 pp. 1992 [3-540-55135-2]

Vol. 175: Rogers, E.T.A.; Owens, D.H. Stability Analysis for Linear Repetitive Processes 197 pp. 1992 [3-540-55264-2] Vol. 176: Rozovskii, B.L.; Sowers, R.B. (Eds) Stochastic Partial Differential Equations and their Applications. Proceedings of IFIP WG 7.1 International Conference, June 6-8, 1991, University of North Carolina at Charlotte, USA 251 pp. 1992 [3-540-55292-8]

Vol. 177: Karatzas, I.; Ocone, D. (Eds) Applied Stochastic Analysis. Proceedings of a US-French Workshop, Rutgers University, New Brunswick, N.J., April 29-May 2, 1991 317 pp. 1992 [3-540-55296-0] Vol. 178: Zol~sio, J.P. (Ed.) Boundary Control and Boundary Variation. Proceedings of IFIP WG 7.2 Conference, Sophia-Antipolis, France, October 15-17, 1990 392 pp. 1992 [3-540-55351-7|

Vol. 179: Jiang, Z.H.; Schaufelberger, W. Block Pulse Functions and Their Applications in Control Systems 237 pp. 1992 [3-540-55369-X]

Vol. 180: Kall, P. (Ed.) System Modelling and Optimization. Proceedings of the 15th IFIP Conference, Zurich, Switzerland, September 2-6, 1991 969 pp. 1992 [3-540-55577-3] Vol. 181: Drane, C.R. Positioning Systems - A Unified Approach 168 pp. 1992 [3-540-55850-0]

Vol. 182: Hagenauer, J. (Ed.) Advanced Methods for Satellite and Deep Space Communications. Proceedings of an International Seminar Organized by Deutsche Forschungsanstalt fSr Luft-und Raumfahrt (DLR), Bonn, Germany, September 1992 196 pp. 1992 [3-540-55851-9] Vol. 183: Hosoe, S: (Ed.) Robust Control. Proceesings of a Workshop held in Tokyo, Japan, June 23-24, 1991 225 pp. 1992 [3-540-55961-2]

Vol. 184: Duncan, T.E.; Pasik-Duncan, B. (Eds) Stochastic Theory and Adaptive Control. Proceedings of a Workshop held in Lawrence, Kansas, September 26-28, 1991 500 pp. 1992 [3-540-55962-0] Vol. 185: Curtain, R.F. (Ed.); Bensoussan, A.; Lions, J.L.(Honorary Eds] Analysis and Optimization of Systems: State and Frequency Domain Approaches for Infinite-Dimensional Systems. Proceedings of the lOth International Conference, Sophia-Antipolis, France, June 9-12, 1992. 648 pp. 1993 [3-540-56155-2] Vol. 186: Sreenath, N. Systems Representation of Global Climate Change Models. Foundation for a Systems Science Approach. 288 pp. 1993 [3-540-19824-5]

Vol. 187: Morecki, A.; Bianchi, G.; Jaworeck, K. (Eds) RoManSy 9: Proceedings of the Ninth CISM-IFToMM Symposium on Theory and Practice of Robots and Manipulators. 476 pp. 1993 [3-540-19834-2]

Vol. 188: Naidu, D. Subbaram Aeroassisted Orbital Transfer: Guidance and Control Strategies 192 pp. 1993 [3-540-19819-9] Vol. 189: Ilchmann, A. Non-Identifier-Based High-Gain Adaptive Control 220 pp. 1993 [3-540-19845-8] Vol. 190: Chatila, R.; Hirzinger, G. (Eds) Experimental Robotics I1: The 2nd International Symposium, Toulouse, France, June 25-27 1991 580 pp. 1993 [3-540-19851-2] Vol. 191: Blondel, V. Simultaneous Stabilization of Linear Systems 212 pp. 1993 [3-540-19862-8]

Vol. 192: Smith, R.S.; Dahleh, M. (Eds) The Modeling of Uncertainty in Control Systems 412 pp. 1993 [3-540-19870-9] Vol. 193: Zinober, A.S.I. (Ed.] Variable Structure and Lyapunov Control 428 pp. 1993 [3-540-19869-5] Vol. 194: Cao, Xi-Ren Realization Probabilities: The Dynamics of Queuing Systems 336 pp. 1993 [3-540-19872-5]

Vol. 195: Liu, D.; Michel, A.N. Dynamical Systems with Saturation Nonlinearities: Analysis and Design 212 pp. 1994 [3-540-19888-1] Volo 196: Battilotti, S. Noninteracting Control with Stability for Nonlinear Systems 196pp. 1994 [3-540-19891-1]

Vol. 197: Henry, J.; Yvon, J.P. (Eds) System Modelling and Optimization 975 pp approx. 1994 ]3-540-19893-8]

Vol. 204: Takahashi, S.; Takahara, Y. Logical Approach to Systems Theory 192 pp. 1995 [3-540-19956-X]

Vol. 198: Winter, H.; N(il~er, H.-G. (Eds) Advanced Technologies for Air Traffic Flow Management 225 pp approx. 1994 [3-540-19895-4]

Vol. 205: Kotta, U.

Vol. 199: Cohen, G.; Quadrat, J.-P. (Eds) 1 l t h International Conference on Analysis and Optimization of Systems Discrete Event Systems: Sophia-Antipolis, June 1 5 - 1 6 - 1 7 , 1994 648 pp. 1994 [3-540-19896-2]

Inversion Method in the Discrete-time Nonlinear Control Systems Synthesis Problems 168 pp. 1995 [3-540-19966-7] Vol. 206: Aganovic, Z.;.Gajic, Z.

Linear Optimal Control of Bilinear Systems with Applications to Singular Perturbations and Weak Coupling 133 pp. 1995 [3-540-19976-4]

Vol. 200: Yoshikawa, T.; Miyazaki, F. (Eds)

Vol. 207: Gabasov, R.; Kirillova, F.M.;

Experimental Robotics II1: The 3rd International Symposium, Kyoto, Japan, October 28-30, 1993 624 pp. 1994 [3-540-19905-5]

Prischepova, S.V. (Eds) Optimal Feedback Control 224 pp. 1995 [3-540-19991-8] Vol. 208: Khalil, H.K.; Chow, J.H.;

Vol. 201: Kogan, Jo Robust Stability and Convexity 192 pp. 1994 [3-540-19919-5] Vol. 202: Francis, B.A.; Tannenbaum, A.R. (Eds) Feedback Control, Nonlinear Systems, and Complexity 288 pp. 1995 [3-540-19943-8] Vol. 203: Popkov, Y.S. Macrosystems Theory and its Applications: Equilibrium Models 344 pp. 1995 [3-540-19955-1]

Ioannou, P.A. (Eds) Proceedings of Workshop on Advances in Control and its Applications 300 pp. 1995 [3-540-19993-4] Vol. 209: Foias, C.; C)zbay, H.;

Tannenbaum, A. Robust Control of Infinite Dimensional Systems: Frequency Domain Methods 230 pp. 1995 [3-540-19994-2]