No title

ARTICLE Communicated by David Zipser Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Betw...

Author: MIT Press

8 downloads 878 Views 22MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

ARTICLE

Communicated by David Zipser

Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors M. Griniasty M.V. TsodyW Racah lnstitute of Physics and Center for Neural Computation, Hebrew University, Jerusalem

Daniel J. Amit' INFN, Sezione di Roma, lstituto di Fisica, Universita di Roma, La Sapienza, Ple Aldo Moro, Roma, Italy

It is shown that a simple modification of synaptic structures (of the Hopfield type) constructed to produce autoassociative attractors, produces neural networks whose attractors are correlated with several (learned) patterns used in the construction of the matrix. The modification stores in the matrix a fixed sequence of uncorrelated patterns. The network then has correlated attractors, provoked by the uncorrelated stimuli. Thus, the network converts the temporal order (or temporal correlation) expressed by the sequence of patterns, into spatial correlations expressed in the distributions of neural activities in attractors. The model captures phenomena observed in single electrode recordings in performing monkeys by Miyashita et al. The correspondence is as close as to reproduce the fact that given uncorrelated patterns as sequentially leamed stimuli, the attractors produced are significantly correlated up to a separation of 5 (five) in the sequence. This number 5 is universal in a range of parameters, and requires essentially no tuning. We then discuss leaming scenarios that could lead to this synaptic structure as well as experimentalpredictions following from it. Finally, we speculate on the cognitive utility of such an arrangement. 1 Introduction

1.1 Temporal to Spatial Correlations in Monkey Cortex. The remarkable sequence of neurocognitive experiments by Miyashita (1988), Miyashita and Chang (19881, and Sakai and Miyashita (1991) is the most direct evidence of the relevance of attractor dynamics in cortical cognitive processing. It is at the same time detailed and structured enough 'On leave of absence from The Institute of Higher Nervous Activity, Moscow. +Onleave of absence from Racah Institute of Physics, Hebrew University, Jerusalem.

Neural Computation 5,1-17 (1993)

@ 1993 Massachusetts Institute of Technology

2

M. Griniasty, M. V,Tsodyks, and D. J. Amit

to guide and confront attractor neural network (ANN) modeling. In the first experiment (Miyashita and Chang 1988), the monkey is trained to recognize and match a set of visual patterns. As a result, one observes selective enhancement of neural spike activity, which persists for 16 sec after the removal of the stimulus. The fact that selective, stimulus-related, enhancement of neural activity persists for 16 sec in the absence of the provoking stimulus is evidence of nonergodic attractor dynamics (see, e.g., Amit 1992). The same encouraging evidence has forced a confrontation on the question of activity rates on retrieval by attractors. The rates in the Miyashita attractors were many times lower than what models of the Hopfield type (Hopfield 1982; Amit 1989) predicted. This fruitful confrontation led to a study (Amit and Tsodyks 1991) that showed that when neural description is taken in greater detail, as well as the conditions prevailing in cortex, attractors can appear having stochastic behavior and low rates. The second study (Miyashita 1988) went further to provide information about coding in the particular module of the anterior ventral temporal cortex of the monkey: It was discovered that despite extreme precaution in producing visual stimuli uncorrelated in their spatial form, spatial correlations appeared in the patterns of sustained activities, evoked by the stimuli, during the delay period. These persistent activities we interpret as the structure of the attractors. There was one kind of correlation that was preserved in the stimuli, the temporal order of their presentation was maintained fixed during training. What the monkey’s brain appears to be doing, is to convert the temporal correlation into a spatial one. Namely, spatial correlations were observed among the attractors, corresponding to the stimuli that were close temporally in the training session. These attractors are the result of retrieval dynamics. The spatial correlations between the activities of the neurons investigated persisted to a fifth neighbor in the temporal sequence. The correlation figure of Miyashita (1988) is reproduced in Figure 1. 1.2 Modeling Correlation Conversion. The main result of the Hopfield program has been to connect the intuitive call for selective (stimulus dependent) attractor dynamics (associative memory) with specific constructions of synaptic matrices, and therefore a bridge to unsupervised learning. The program was limited by the requirement that the attractors be as close as possible to the patterns of which the matrices were constructed, that is, the presumed items in the learning process. This went under the name of autoassociation. Here we shall show that a simple modification of the synaptic matrices used for autoassociation in ANNs leads to a relaxation dynamics that associates with stimuli near one of the random, uncorrelated underlying patterns, an attractor that is correlated with several patterns. The patterns that have the largest correlations with a given attractor are the neighbors of the stimulus leading to the attractor, in the sequence of stored patterns.

Conversion of Temporal to Spatial Correlations

c

3

0.4t

L.

c

.g -

-

0.3 -

F

6 0 0

-6

0.2

-

0.1

-

0

CI

g

0

0

-1

5

10th neighbour

Figure 1: Spatial correlations between attractors, in monkey's anterior ventral temporal cortex, corresponding to structurallyuncomlated patterns, as a function of the difference in the position of the learned stimuli in the fixed training sequence. From Miyashita (1988).

It then follows that attractors are correlated among themselves. Again, the attractors that are correlated are the neighbors in the sequence of the underlying patterns. These are just the type of correlations observed by Miyashita. In fact, the number of attractors that are found to be correlated significantly in the model is the same as in the experiment. The extended model is discussed in two different variants: one is the original formulation of k1 neurons, with the artificial symmetry between active and passive states of neurons; the second is a 0-1 formulation (Tsodyks and Feigel'man 1988; Buhmann et al. 19891, in which this symmetry is removed and that can naturally be interpreted in terms of high and low activity rates of neurons in attractors. The results differ in detail, but the main qualitative features, of converting sequential order among uncorrelated patterns to a set of correlated attractors, is present in both. Both models have symmetric synaptic matrices, which are unrealistic but convenient. The study of autoassociative ANNs, over the last several years, has made it clear that most of the attractor properties of these extensively connected networks are rather robust to the introduction of synaptic asymmetry (see, e.g., h i t 1989). We then proceed to interpret the proposed synaptic matrices in terms of learning dynamics. It is argued that rather plausible synaptic dynamics, accompanying the relaxation in the ANN, may produce a synaptic

M. Griniasty, M. V. Tsodyks, and D. J. Amit

4

matrix with correlated attractors for uncorrelated external stimuli. Within such learning scenarios, one is led to predict that the presentation of uncorrelated patterns in a random sequence would produce attractors that are uncorrelated, and are each close to the representation of the original patterns, as would be the case in the Hopfield model. Finally, we discuss the potential utility of such conversions of temporal correlations to spatial correlations in modeling several aspects of cognitive behavior. 2 The Model with fl Neurons

The original way of pursuing the Hopfield ANN program was to choose the variables describing the instantaneous states of each neuron S;(t)=fl, where i labels the neuron (i = I , . . . ,N). The patterns, to be stored in an N neuron network, are N-bit words of fls, the value of each bit chosen independently, with probability 0.5. Denoting the components of the activity of neuron number i in pattern number p, tr, the proposed synaptic matrix is written as:

where p is the total number of patterns stored in the connections. The patterns, p, are considered to form an ordered sequence, which corresponds to the order of presentation in the training phase. For simplicity, the sequence is taken to be cyclic. Each pattern in the construction of the matrix is connected to one preceding pattern. Note, in particular, that this extended matrix still preserves the symmetry of the Hopfield matrix, which implies that all attractors will be fixed points, and makes analysis so much simpler. How this relates to a learning scenario is discussed in Section 4. The matrix of equation 2.1, for a = 0, reduces to the original Hopfield matrix. This matrix is accompanied, as usual, by a schematic spike emission dynamics that, in the noiseless case, determines the new state, Si(t), of the neuron according to Si(t

+ 6t) = sign[hi(t + bt)]

(2.2)

where

h; mimics the value of the postsynaptic potential, relative to the threshold, on neuron i. The linear superposition of bilinear terms in the neural activities of the stored patterns is sometimes referred to as a Hebbian learning from a “tabula rasa.” We shall return to the question of learning later.


5

The natural variables for the description of the nonergodic asymptotic behavior of the network are the “overlaps” m”(t) of the current state of the network, Si(t), with the stored pattern p. They are defined as

See, for example, Amit et al. (1985) and Amit (1989). The value of overlap

measures how close is the state of the network to the stored pattern mp = 1, the state is identical, as a binary word to the pattern p, that is, Si = #‘ for all i. If mp = 1 for the asymptotic state, the attractor, then the corresponding pattern is retrieved perfectly. With the matrix, equation 2.1, the “field hi can be expressed in terms of the overlaps (Amit et al. 19851, which implies that so can the dynamics of the network as well as its attractors. Namely, we can write equation 2.3 as mp

p. If

from which one derives the mean-field equations determining the attractors. For a symmetric matrix, those are simple fixed points. They read, in the limit of a network of a large number of neurons with a relatively low number of stored patterns:

The double angular brackets imply an averaging over the distribution of the bits in the patterns (see, e.g., Amit et al. 1985; Amit 1989). Autoassociation was the interpretation of the fact that, in the absence of noise, for low loading, the equations 2.6 had solutions with one single m p # 0, which attracted a wide set of initial states in the neighborhood of each pattern. Away from these large basins, ‘,spurious states” were found to exist (Amit et al. 1985). Moreover, the artificial symmetry of the +1 and -1 states produced attractors of the sign reversed states of each pattern. These, retrieval properties of the Hopfield ANN have been found very robust to extensive noise and synaptic disruption, including asymmetric disruption. If one tries a pure pattern solution for equations 2.6, with p = 2 for example, one has1:

m2 = ( ( t 2 w(m2[t2 + a(tl

+ t3)i)))

(2.7)

For a < 0.5, it is the first term in the square brackets that dominates the sign of the argument of the sign-function and m2 = 1 is a fixed point solution, as in the case a = 0. For a > 0.5, this is no longer the case. For ‘The 2s are superscripts not squares.


6

25% of the sites t3= t1= -t2 and the argument has the sign opposite to that of Starting from a state with m2 = 1, and all other overlaps 0,

c2,

one arrives, after one step to a state with m2 = m1 = m3 = 0.5. This is no fixed point either. The solutions of equations 2.6 have several overlaps different from zero. The previous discussion suggests a numerical procedure for arriving at the solution: start from a pure pattern state and iterate until convergence. This is what the network would do, if given one of the pure patterns it learned, Si = t’, as an initial state, until it relaxes to a fixed point. The symmetry of the dynamics under pattern permutations implies that this has to be done for one pattern only The equations were solved in this way. One finds that starting from a pure pattern, one arrives at a stable solution after several iterations. The solution reached is a state with nonzero overlaps with several stored patterns, symmetrically distributed around the pattern, which served as the stimulus. Only a small number, actually 5, of these overlaps are significantly large, provided u < 1.2 This distribution of overlaps in an attractor, corresponding to one of the underlying patterns, is shown in Figure 2. In this case p = 13 patterns are stored, and u = 0.7. It is remarkable that the structure of the attractor does not depend on the number of patterns p, nor on the value of a, in the entire range 0.5 < u < 1. For a > 1, the network develops attractors that have overlaps with all stored patterns. The values of the overlaps decrease as the number of patterns increases. This means that after learning sufficiently many patterns, the network loses its ability to associate attractors with the stimuli. One can read from Figure 2 that the retrieval attractor has substantial overlaps with several patterns, symmetrically disposed, before and after, in the sequence relative to the pattern corresponding to the stimulus. Clearly, if each attractor is correlated with several patterns, then the attractors corresponding to different patterns must themselves be correlated. These correlations would correspond to the correlations measured by Miyashita and Chang (1988) (Fig. 1). The correlation of activities in two attractors, u p and u’, is defined as i

N

C(ui”- a)(aiv- a ) 1‘1 j=l

C(p,v) = i

(2.8)

where 8 is the average activity in a given attractor and the normalization constant ICI is chosen so that C ( p ,p ) = 1. In the present case, T = 0, and (C( = N. Hence, the correlation of attractors p and v can be written as

1

= -

C sign(ha)sign(hr)= ((sign(hj’)sign(h”)))

N i

(2.9)

2Noteadded in proof L. Cugliandolo has recently proved that beyond 5 all overlaps are exactly zero.


7

pattern

Figure 2 Overlaps between an attractor and stored patterns, as a function of the separation of the pattern in the sequence from the pattern underlying the attractor. where hy is the local field on neuron i when the network is in attractor number p. The last equality is an expression of self-averaging, giving an average over the distribution of patterns. Finally, substituting the fields from equation 2.5 in equation 2.9, we arrive at the correlation coefficient:

(2.10)

where m; is the overlap of the attractor corresponding to stimulus number p with pattern number p. These attractor overlaps are illustrated in Figure 3, where we plot the correlations among different pairs of attractors vs. the distance between their corresponding patterns. Figure 3 clearly demonstrates that while stored patterns are completely random, and hence uncorrelated, the states reached by the network on presentation of these same patterns, have a substantial degree of correlations, which decreases with the separation of the patterns in the sequence. Note that while an attractor "sees" two-three patterns on each side, it sees five attractors on each

8


Figure 3: Correlations between attractors as a function of the separation in the sequence of the patterns to which the attractors belong. side. The qualitative form of the correlations captures the experimental trend (Fig. 1). The absolute values of the correlations differ. This can be due to a different normalization used in Miyashita and Chang (1988), where the normalization is not given explicitly. It may also be that the absolute values will be different in more realistic networks. Finally, the analysis given above is based on the exact symmetry of the matrix 2.1, in which case the system has only fixed point attractors. Asymmetry can enter in two ways: either as a local, random disruption of the synaptic elements, or as a coherent asymmetry of the two transition terms in the symmetric matrix. The first type of asymmetry is in the realm of the robustness of the attractor dynamics of the network. Concerning the second type: The mean-field equations 2.6 hold even if the coefficients of the two transition terms in Jq are not equal. We have found that the behavior of the network is robust against some amount of asymmetry between the two coefficients. However, if the asymmetry becomes too large, equations 2.6 first acquire another solution, with the maximum overlap shifted to another pattern. At still higher asymmetry, the fixed point solution is lost, in favor of a time-dependent attractor. In this attractor the network moves from one pattern to another, in a direction determined by the major nondiagonal term in the matrix 2.1.


9

3 ANN with Discrete 0-1 Neurons

The above description was extended to deal with a 0-1 representation of information, allowing for the removal of the symmetry between active and refractory states of the neurons (Tsodyks and Feigel’man 1988; Buhmann ef aI. 1989). This description has several further advantages: its terms are very close to a representation in terms of spike rates, which are positive analog variables; moreover it allows for very efficient storage of patterns as the coding rate, that is, the fraction of Is in the patterns, becomes very low-sparse coding. Since it is important to show that the correlation effects to be discussed take place also for this case, we shall recall that formulation as well. The dynamics is described in terms of instantaneous neural variables, V;(t), which take the values (0,U as V;(t

+ St) = 0

,]jjVj(t) [ j

I

-0

(3.1)

where Q ( x ) = 1 for x > 0, and 0 otherwise, and 0 is a neural threshold. These variables can be directly interpreted as high and low activity (spike rates) of each neuron. Such would be a description in terms of analog rates, in a high gain limit. In this case the patterns to be stored by a learning dynamics, q”, are chosen as N-bit words of independently chosen O,ls, that is, q;=o;1

p=ll...lp

(3.2)

where the probability for a 1,O-bit is f l (1- f ) , respectively. An extension of the symmetric synaptic matrix, appropriate for autoassociation (Tsodyks and Feigel’man 1988; Buhmann et al. 1989), to our requirements would be

(3.3)

and the corresponding overlaps are generalized to (3.4)

With the couplings 3.3, the dynamics can be expressed in terms of the above overlaps and so can the fixed points of the retrieval attractors. The latter have the form


10

When a = 0, the system of equations reduces to that of Tsodyks and Feigel'man (1988) and Buhmann et al. (1989). In this case, at low loading, the exact stored patterns are the retrieval attractors of the network, that is, the equations admit solutions with a single nonvanishing overlap, which in turn is equal to 1. These attractors persist until a reaches the critical value of

Above this value of a, the pure patterns are unstable and the network, having a symmetric synaptic matrix, finds new fixed points. The equations 3.5 have to be solved numerically for the values of the overlaps in the retrieval attractors. This we do, again following the network, as was explained in the previous section. A typical solution is shown in Figure 4, for parameter values: f = 0.01, 6 = 0.2, p = 11, and a = 0.25. Figure 4a represents the overlaps vs. the pattern number in the sequence, relative to the pattern of the stimulus. Figure 4b is the correlationbetween the attractors. Note that in distinction to the fl case, the significant overlaps here, of which there are five in total, are all equal. They are all unity, up to terms of OU).This implies that the attractor is approximately the union of the 1-bits in the five patterns centered around the stimulus. In particular, the mean spatial activity level in the attractors is higher than in the pure patterns. A fact that can be tested experimentally. The correlation Figure 4b may seem somewhat simple compared with the experimental one of Figure 1. Clearly, the experimental correlations are not a straight line going to zero at a separation of five patterns. We find the appearance of the correlations as well as their clear trend to decrease with the separation in the training sequence, down to very small values at a separation of five, very significant. All that was put in was the synaptic structure connecting successive patterns in the sequence. The remaining differences may be attributed to several factors, all of which are under study. These factors are a

The neurons in the experiment are analog neurons, represented by spike rates, and not discrete 0-1 neurons.

a

In the experiment the neurons operate in the presence of noise, while here for simplicity we dealt with a noiseless situation.

a

The matrix we chose is surely not the matrix in the monkey's cortex. One consequence is that all our attractors are identical.

a

In the experiment the sample groups of neurons are small and are chosen in special ways. This leads inter alia to large fluctuations. O u r correlations are ideal in that they take into account an infinite number of neurons.


11

All these effects can be studied either by an extension of the above reasoning to analog neurons with noise, or by simulations. These studies are under way. The attractor to attractor correlations are computed according to equation 2.8. What remains is to determine r and ICI for this case. If the mean proportion of 1-bits in the attractors is g, then 8 = g and ICI = Ng(1 -g). It is rewarding to see that the correlations between neighboring attractors are monotonically decreasing with the separation in the sequence, and are disappearing after the fifth neighbor, as in the experimental data. In the present case, the number of condensed patterns, those having large values of the overlap with the attractor corresponding to a stimulus, depends on the value of u. This variation leaves finite intervals of u in which the attractors are invariant. Increasing u, we observe a sequence of bifurcations, where the number of condensed patterns increases by two. Correspondingly, the number of significantly correlated attractors increases by four on crossing a bifurcation value of a. Between any two bifurcation points the solution does not change, that is, the number of significantly correlated attractors as well as the magnitude of the correlations remain invariant.

4 Learning

In this section we will try to discuss possible learning scenarios, which could lead to a synaptic structure of the type considered in the previous sections. At the present time there is not enough information about the learning mechanism and memory preservation in the cortex, and our discussion can at best be tentative. We feel though that such a discussion may not be completely premature, just because of the level of specific detail provided by the experiments of Miyashita et al., and the ability of theory to approach a similar level of detail. Moreover, it is our feeling that a discussion of the implications to learning, of such findings, may lead to experiments which may shed additional light on constraints on learning through neurophysiological correlates of behavior. It is plausible to describe the synaptic dynamics as

(4.1)

where y is the rate of decay of the synaptic value and Ki,Lj are, respectively, the post- and pre-synaptic contributions to the synaptic efficacy. Both K and L depend on the activity of the corresponding neuron. A simple mechanism that would lead to the matrix 2.1 could be to apply a usual Hebbian modification rule, with both pre- and post-synaptic

M. Griniasty, M. V. Tsodyks, and D. J- Amit

12

a

0

2

4

6

pattern

8

10

12

b

Figure 4: Correlations between attractors (a) and overlaps of attractor with patterns (b) vs. the separation in the sequence of the patterns to which the attractors belong.


13

terms as linear combinations of the current and preceding patterns, for example,

(4.2)

This form may result from two different scenarios. In both we assume that strong presentations of the individual, uncorrelated patterns create attractors for those patterns themselves. Then, during training, which consists of many repeated presentations, the network, which remains in an attractor between presentations, is made to move to the next attractor by a new presentation. It should be emphasized that in this description the role of the attractors is quite crucial. Before the patterns themselves are stored in the synapses as attractors, at the presentation of a consecutive pattern (to be learned) in the sequence during training, there is no memory of the previous pattern. This is especially true if the time between presentations of consecutive patterns is as long as in the experiments of Miyashita et al. The difference between the two scenarios is in the way we view the origin of the source terms, K and L, for the synaptic change. In the first we assume that the values of the neuronal spike frequencies represent, in an analog way, the transition between the two attractors. In this picture, K l ( t ) = L l ( t ) = 0, before the transition starts, which is about when the next pattern is presented. When the network is well established in the new attractor, K 2 ( t ) and L*(t) tend to zero. In the second scenario, one assumes that it is the local synaptic variable that remembers some short history of the pre- or postsynaptic activity of the corresponding neuron. For example, it may be the case that the synaptic mechanism modifies its effectiveness depending on the mean of the neuron's activity in some prior time window 7. The pre- or postsynaptic change may be enhanced, or suppressed, by a history of high average mean activity. Since the moving mean of the activity history is a linear combination of the activity in two consecutive patterns, while the network is moving from one attractor to the next, the end result is the same, provided, of course, that the averaging window is short in comparison with the time spent in each attractor. But this does not seem to be a strong requirement, given that the network stays in these attractors for many seconds. As a simple assumption we can take K = L, as functions of the p r e and post-synaptic activities, which implies KI = LI, K2 = L2. Relaxing this constraint would lead us to asymmetric synaptic transition terms, of the type discussed in Section 2. If the resulting asymmetry is not large we expect the performance of the network to be robust. In the symmetric case, the contribution to the synaptic dynamics is

14


As one pattern follows the other, these contributions sum up, when equation 4.1 is integrated. If we neglect the exponential decay, y, the summation is direct and after a long time, when all patterns have been presented many times in a fixed order, the resulting matrix would be proportional to

where the time integration is over an interval 7, in which synaptic modification is taking place. This matrix has the same form as the one we introduced in the previous sections. It corresponds though to a case in which u 5 0.5. This fact should not be considered too adverse. Synaptic decay, for example, is sufficient to raise u above 0.5. In the final analysis one should consider analog neurons, toward which the 0-1 neurons are an intermediate stage. Even for the discrete 0-1 neurons, the critical value of u is much lower than 0.5, while the heuristic learning mechanism can remain essentially the same. Finally, if the patterns are presented in a random order during training, one can expect every pattern to be followed by any other one, given that a large number of presentations is required for satisfactory learning. This implies that the transition terms in equation 4.4, containing any particular pattern, will be multiplied by a sum over all other patterns. That sum vanishes on the average and the transition terms become negligibly small. No correlations are then generated by the netwprk, from uncorrelated patterns. 5 Experimental Predictions and Some Speculations

Given that a synaptic matrix, which can be learned without supervision, is able to convert temporal correlations into spatial ones, one is tempted to make some preliminary speculations about the computational and behavioral utility of such synaptic development. One directly measurable application had been pointed out in Sakai and Miyashita (1991). In this experiment the monkeys are trained to recognize 24 visual patterns, organized in 12 fixed pairs. The pairs are presented in a random order. Correlations are generated among the two members of each pair only. Those correlations are then shown to be correlated with the ability of the monkey to retrieve the second member of a pair, after being presented with the first. The basic nature of this type of association for the construction of cognitive behavioral patterns is quite immediate. What is special about this particular experiment is that the associative retrieval of the paired member is directly connected to the presence of the correlations in the representation of the pairs of attractors in the part of cortex under electrophysiological observation.


15

The interpretation of this experiment does not require speculation. To

go one step beyond, one can expect the generation of such correlations to underlie the effect of priming (Besner and Humphreys 1990). In other words, if the network is in one of its attractors, and a new stimulus is presented, the transition between two attractors that are highly correlated (i.e., have a particularly large number of active neurons common to their representations) is much faster than the transition between less correlated attractors. This effect was observed in a simulation with realistic neurons (Amit et al. 1991), when the pure patterns, involved in the construction of the synaptic matrix, included as explicit correlations (Rubin 1991). This effect can be directly measured in a Miyashita (1988) type experiment. One would expect that the transition time between different attractors would increase with the distance of the two patterns in the sequence of presentation. In cognitive psychology the effect is familiar in experiments in which the reaction time is measured for the recognition of distorted words or other images. This reaction time is significantly shortened if the pattern to be recognized is preceded by a cognitively correlated pattern (Besner and Humphreys 1990). In the language of the model we would say that the "priming" image leads the network into its corresponding attractor. That attractor is correlated with the attractor corresponding to the test stimulus. Hence, the transition between the two is faster than the transition from some other state in which the network may find itself in otherwise. Complementing this scenario with the suggestion that at least part of our basic cognitive correlations is related to temporal contiguity of afferent stimuli completes this speculation. This interpretation can be extended one small step further. As attractors get increasingly correlated, there is an increase in the probability that noise would cause transitions between them, transitions of the BuhmannSchulten type (Buhmann and Schulten 1987). This opens the way for the scenario in which such transitions can be provoked in a cortical network by random afferent activation of the module. The transitions will tend to take place between correlated attractors, which in the present model are related to temporal proximity during learning. Note that this process can also be observed in the experiments of the Miyashita type, though their cognitive content is more difficult to investigate. One could hope to be able to investigate the process of learning the matrix that generates the correlations. We have argued in Section 4 that the process will go through the intermediate stage of learning the pure pattern attractors first. This was based on the assumption that there is autonomous learning in the particular module under observation. This is not self-evident, and it may be that the pure patterns are quickly learned as attractors in a different area, as hippocampus, for example, and those attractors then assist in learning the correlated attractors. Since the question is open, one could attempt to clarify it by presenting different parts of the training sequence, in an experiment such as Miyashita (19881, with

16


different frequencies. Then, if learning actually first goes through the creation of individual attractors for the pure patterns, one should observe lower correlations in the parts shown less frequently, as well as lower coding rates. In other words, pure patterns are expected to use fewer neurons than the composite patterns correlated by the dynamics (see, e.g., Section 3). On the other hand, if the module learns the correlated attractors directly, no group of patterns should show the appearance of uncorrelated attractors. Acknowledgments

The authors acknowledge useful discussions with N. Rubin, S. Seung, and H. Sompolinsky DJA has benefited from many useful discussions of the learning mechanisms with Stefan0 Fusi. We are indebted to N. Rubin for information concerning priming effects. MVT's research is supported in part by the Ministry of Science and Technology. References Amit, D. J. 1989. Modeling Brain Function. Cambridge University Press, New York. Amit, D. J. 1992. In defence of single electrode recordings. NETWORK 4(4). Amit, D. J., Gutfreund, H., and Sompolinsky, H. 1985. Spin-glass models of neural networks. Phys. Rev. A32,1007. Amit, D. J.,Evans, M. R., and Abeles, M. 1991. Attrador neural networks with biological probe neurons. NETWORK 1,381. Amit, D. J., and Tsodyks, M. V. 1991. Quantitative study of attractor neural network retrieving at low spike rates I Substrate-spikes, rates and neuronal gain. NETWORK 2, 259; and Low-rate retrieval in symmetric networks. NETWORK 2,275. Besner, D., Humphreys, G. eds. 1990. Basic Processes in Reading: Visual word recognition. Hillside, NJ: Erlbaum; Tweney, R. D., Heiman, G. H., and Hoemann, H. W. 1977. Effects of visual disruption on sign intelligibility. J. Exp. Psycol. Gen.106,255. Buhmann, J., Divko, R., and Schulten, K. 1989. Associative memory with high information content. Phys. Rev. A39,2689. Buhmann, J.,and Schulten, K. 1987. Noise driven associations in neural networks. Europhys. Lett. 4, 1205. Hopfield, J. J. 1982. Neural networks and physical systems with emergent selective computational abilities. Proc. Natl. A d . Sci. USA. 79,2554. Miyashita, Y. 1988. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature (London) 335,817. Miyashita, Y.,and Chang, H. S. 1988. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature (London) 331,68. Rubin, N. 1991. Private communication.


17

Sakai, K., and Miyashita, Y. 1991. Neural organization for the long-term memory of paired associates. Nature (London) 354, 152. Tsodyks, M. V., and Feigel'man, M. V. 1988. The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 46, 101.

Received 21 January 1992; accepted 5 May 1992.

This article has been cited by: 1. Valentina Daelli, Alessandro Treves. 2010. Neural attractor dynamics in object recognition. Experimental Brain Research 203:2, 241-248. [CrossRef] 2. Nicolas Brunel, Frédéric Lavigne. 2009. Semantic Priming in a Cortical Network ModelSemantic Priming in a Cortical Network Model. Journal of Cognitive Neuroscience 21:12, 2300-2319. [Abstract] [Full Text] [PDF] [PDF Plus] 3. F L Metz, W K Theumann. 2009. Symmetric sequence processing in a recurrent neural network model with a synchronous dynamics. Journal of Physics A: Mathematical and Theoretical 42:38, 385001. [CrossRef] 4. Tomoyuki Kimoto, Tatsuya Uezu, Masato Okada. 2008. Multiple Stability of a Sparsely Encoded Attractor Neural Network Model for the Inferior Temporal Cortex. Journal of the Physical Society of Japan 77:12, 124002. [CrossRef] 5. F. Metz, W. Theumann. 2007. Period-two cycles in a feedforward layered neural network model with symmetric sequence processing. Physical Review E 75:4. . [CrossRef] 6. Sawako Tanimoto, Masato Okada, Tomoyuki Kimoto, Tatsuya Uezu. 2006. Distinction of Coexistent Attractors in an Attractor Neural Network Model Using a Relaxation Process of Fluctuations in Firing Rates -- Analysis with Statistical Mechanics --. Journal of the Physics Society Japan 75:10, 104004. [CrossRef] 7. Masato Okada. 2006. Part 3: Brain science, information science and associative memory model. New Generation Computing 24:2, 185-201. [CrossRef] 8. F. Metz, W. Theumann. 2005. Pattern reconstruction and sequence processing in feed-forward layered neural networks near saturation. Physical Review E 72:2. . [CrossRef] 9. Tatsuya Uezu, Aya Hirano, Masato Okada. 2004. Retrieval Properties of Hopfield and Correlated Attractors in an Associative Memory Model. Journal of the Physics Society Japan 73:4, 867-874. [CrossRef] 10. Gianluigi Mongillo, Daniel J. Amit, Nicolas Brunel. 2003. Retrospective and prospective persistent activity induced by Hebbian learning in a recurrent cortical network. European Journal of Neuroscience 18:7, 2011-2024. [CrossRef] 11. Kazushi Mimura, Tomoyuki Kimoto, Masato Okada. 2003. Synapse efficiency diverges due to synaptic pruning following overgrowth. Physical Review E 68:3. . [CrossRef] 12. Masahiko Yoshioka, Masatoshi Shiino. 2000. Associative memory storing an extensive number of patterns based on a network of oscillators with distributed natural frequencies in the presence of external white noise. Physical Review E 61:5, 4732-4744. [CrossRef] 13. Kaname Toya, Kunihiko Fukushima, Yoshiyuki Kabashima, Masato Okada. 2000. Journal of Physics A: Mathematical and General 33:14, 2725-2737. [CrossRef]

14. Tomoki Fukai, Tomoyuki Kimoto, Makoto Doi, Masato Okada. 1999. Journal of Physics A: Mathematical and General 32:30, 5551-5562. [CrossRef] 15. Masahiko Yoshioka, Masatoshi Shiino. 1998. Associative memory based on synchronized firing of spiking neurons with time-delayed interactions. Physical Review E 58:3, 3628-3639. [CrossRef] 16. Néstor Parga , Edmund Rolls . 1998. Transform-Invariant Recognition by Association in a Recurrent NetworkTransform-Invariant Recognition by Association in a Recurrent Network. Neural Computation 10:6, 1507-1525. [Abstract] [PDF] [PDF Plus] 17. Asohan Amarasingham , William B. Levy . 1998. Predicting the Distribution of Synaptic Strengths and Cell Firing Correlations in a Self-Organizing, Sequence Prediction ModelPredicting the Distribution of Synaptic Strengths and Cell Firing Correlations in a Self-Organizing, Sequence Prediction Model. Neural Computation 10:1, 25-57. [Abstract] [PDF] [PDF Plus] 18. Daniel J. Amit, Stefano Fusi, Volodya Yakovlev. 1997. Paradigmatic Working Memory (Attractor) Cell in IT CortexParadigmatic Working Memory (Attractor) Cell in IT Cortex. Neural Computation 9:5, 1071-1092. [Abstract] [PDF] [PDF Plus] 19. Nicolas Brunel. 1996. Hebbian Learning of Context in Recurrent Neural NetworksHebbian Learning of Context in Recurrent Neural Networks. Neural Computation 8:8, 1677-1710. [Abstract] [PDF] [PDF Plus] 20. Hans Liljenström. 1996. Neuromodulation can significantly change the dynamical state of cortical networks. Behavioral and Brain Sciences 19:02, 303. [CrossRef] 21. Paul L. Nunez. 1996. Multiscale neocortical dynamics, experimental EEG measures, and global facilitation of local cell assemblies. Behavioral and Brain Sciences 19:02, 305. [CrossRef] 22. J. J. Wright, D. T. J. Liley. 1996. Dynamics of the brain at global and microscopic scales: Neural networks and the EEG. Behavioral and Brain Sciences 19:02, 285. [CrossRef] 23. Ichiro Tsuda. 1996. The form of chaos in the noisy brain can manifest function. Behavioral and Brain Sciences 19:02, 309. [CrossRef] 24. Valerie Gray Hardcastle. 1996. Modeling for modeling's sake?. Behavioral and Brain Sciences 19:02, 299. [CrossRef] 25. Andrew Oliver. 1996. Dynamics of the brain — from the statistical properties of neural signals to the development of representations. Behavioral and Brain Sciences 19:02, 306. [CrossRef] 26. Hubert Preissl, Werner Lutzenberger, Friedemann Pulvermüller. 1996. Is there chaos in the brain?. Behavioral and Brain Sciences 19:02, 307. [CrossRef] 27. Márk Molnár. 1996. Chaos in induced rhythms of the brain – the value of ERP studies. Behavioral and Brain Sciences 19:02, 305. [CrossRef]

28. Harry R. Erwin. 1996. Multiscale modeling of the brain should be validated in more detail against the biological data. Behavioral and Brain Sciences 19:02, 297. [CrossRef] 29. M. N. Zhadin. 1996. Rhythmicity in the EEG and global stabilization of the average level of excitation in the cerebral cortex. Behavioral and Brain Sciences 19:02, 309. [CrossRef] 30. Zbigniew J. Kowalik, Andrzej Wrobel, Andrzej Rydz. 1996. Why does the human brain need to be a nonlinear system?. Behavioral and Brain Sciences 19:02, 302. [CrossRef] 31. Robert Miller. 1996. Empirical data base for simulation: Firing rates and axonal conduction velocity for cortical neurones. Behavioral and Brain Sciences 19:02, 304. [CrossRef] 32. J. J. Wright, D. T. J. Liley. 1996. Multiscale modeling of brain dynamics depends upon approximations at each scale. Behavioral and Brain Sciences 19:02, 310. [CrossRef] 33. Walter S. Pritchard. 1996. The EEG data indicate stochastic nonlinearity. Behavioral and Brain Sciences 19:02, 308. [CrossRef] 34. Edgar Koerner. 1996. Comparative reduction of theories — or over-simplification?. Behavioral and Brain Sciences 19:02, 301. [CrossRef] 35. Lester Ingber. 1996. Nonlinear nonequilibrium nonquantum nonchaotic statistical mechanics of neocortical interactions. Behavioral and Brain Sciences 19:02, 300. [CrossRef] 36. Theodore H. Bullock. 1996. Is the distribution of coherence a test of the model?. Behavioral and Brain Sciences 19:02, 296. [CrossRef] 37. Péter Érdi. 1996. Levels, models, and brain activities: Neurodynamics is pluralistic. Behavioral and Brain Sciences 19:02, 296. [CrossRef] 38. Daniel J. Amit. 1996. Is the time ripe for integration of scales?. Behavioral and Brain Sciences 19:02, 295. [CrossRef] 39. Walter J. Freeman. 1996. Neural system stability. Behavioral and Brain Sciences 19:02, 298. [CrossRef] 40. Xiangbao Wu, Robert A. Baxter, William B. Levy. 1996. Context codes and the effect of noisy learning on a simplified hippocampal CA3 model. Biological Cybernetics 74:2, 159-165. [CrossRef] 41. Ehud Ahissar. 1995. Are single-cell data sufficient for testing neural network models?. Behavioral and Brain Sciences 18:04, 626. [CrossRef] 42. Masahiko Morita. 1995. Another ANN model for the Miyashita experiments. Behavioral and Brain Sciences 18:04, 639. [CrossRef] 43. Ralph E. Hoffman. 1995. Additional tests of Amit's attractor neural networks. Behavioral and Brain Sciences 18:04, 634. [CrossRef]

44. Shimon Edelman. 1995. How representation works is more important than what representations are. Behavioral and Brain Sciences 18:04, 630. [CrossRef] 45. Maartje E. J. Raijmakers, Peter C. M. Molenaar. 1995. How to decide whether a neural representation is a cognitive concept?. Behavioral and Brain Sciences 18:04, 641. [CrossRef] 46. Daniel J. Amit. 1995. The Hebbian paradigm reintegrated: Local reverberations as internal representations. Behavioral and Brain Sciences 18:04, 617. [CrossRef] 47. Walter J. Freeman. 1995. The Hebbian paradigm reintegrated: Local reverberations as internal representations. Behavioral and Brain Sciences 18:04, 631. [CrossRef] 48. David C. Krakauer, Alasdair I. Houston. 1995. An evolutionary perspective on Hebb's reverberatory representations. Behavioral and Brain Sciences 18:04, 636. [CrossRef] 49. Joaquin M. Fuster. 1995. Not the module does memory make – but the network. Behavioral and Brain Sciences 18:04, 631. [CrossRef] 50. Peter M. Milner. 1995. Attractors – don't get sucked in. Behavioral and Brain Sciences 18:04, 638. [CrossRef] 51. Friedemann Pulvermüller, Hubert Preissl. 1995. Local or transcortical assemblies? Some evidence from cognitive neuroscience. Behavioral and Brain Sciences 18:04, 640. [CrossRef] 52. J. J. Wright. 1995. How do local reverberations achieve global integration?. Behavioral and Brain Sciences 18:04, 644. [CrossRef] 53. Elie Bienenstock, Stuart Geman. 1995. Where the adventure is. Behavioral and Brain Sciences 18:04, 627. [CrossRef] 54. Frank der van Velde. 1995. Association and computation with cell assemblies. Behavioral and Brain Sciences 18:04, 643. [CrossRef] 55. Wolfgang Klimesch. 1995. The functional meaning of reverberations for sensoric and contextual encoding. Behavioral and Brain Sciences 18:04, 636. [CrossRef] 56. Eric Chown. 1995. Reverberation reconsidered: On the path to cognitive theory. Behavioral and Brain Sciences 18:04, 628. [CrossRef] 57. Jean Petitot. 1995. The problems of cognitive dynamical models. Behavioral and Brain Sciences 18:04, 640. [CrossRef] 58. G. J. Dalenoort, P. H. de Vries. 1995. What's in a cell assembly?. Behavioral and Brain Sciences 18:04, 629. [CrossRef] 59. Josef P. Rauschecker. 1995. Reverberations of Hebbian thinking. Behavioral and Brain Sciences 18:04, 642. [CrossRef] 60. Michael Hucka, Mark Weaver, Stephen Kaplan. 1995. Hebb's accomplishments misunderstood. Behavioral and Brain Sciences 18:04, 635. [CrossRef] 61. Anders Lansner, Erik Fransén. 1995. Distributed cell assemblies and detailed cell models. Behavioral and Brain Sciences 18:04, 637. [CrossRef]

62. Morris W. Hirsch. 1995. Mathematics of Hebbian attractors. Behavioral and Brain Sciences 18:04, 633. [CrossRef] 63. Daniel J. Amit. 1995. Empirical and theoretical active memory: The proper context. Behavioral and Brain Sciences 18:04, 645. [CrossRef] 64. W Whyte, D Sherrington, A C C Coolen. 1995. Journal of Physics A: Mathematical and General 28:12, 3421-3437. [CrossRef] 65. Michel Kerszberg, Claudine Masson. 1995. Signal-induced selection among spontaneous oscillatory patterns in a model of honeybee olfactory glomeruli. Biological Cybernetics 72:6, 487-495. [CrossRef] 66. Tomoki Fukai. 1995. A model cortical circuit for the storage of temporal sequences. Biological Cybernetics 72:4, 321-328. [CrossRef] 67. Iris Ginzburg, Haim Sompolinsky. 1994. Theory of correlations in stochastic neural networks. Physical Review E 50:4, 3171-3191. [CrossRef] 68. L. F. Cugliandolo . 1994. Correlated Attractors from Uncorrelated StimuliCorrelated Attractors from Uncorrelated Stimuli. Neural Computation 6:2, 220-224. [Abstract] [PDF] [PDF Plus] 69. L F Cugliandolo, M V Tsodyks. 1994. Journal of Physics A: Mathematical and General 27:3, 741-756. [CrossRef]

NOTE

Communicated by Hal White

On the Realization of a Kolmogorov Network Ji-Nan Lin Rolf Unbehauen LehrstuhlfUr Allgemeine und TheoretischeElektrotechnik, Universitdt Erlangen-Niirnberg,Cauerstrasse 7, D-8520 Erlangen, Germany

It has been suggested that the theorem by Kolmogorov (1957) about the multivariate function representation in the form (0.1)

with Q 2 2N + 1 provides theoretical support for neural networks that implement multivariate mappings (Hecht-Nielson 1987; Lippmann 1987). Girosi and Poggio (1989) criticized Kolmogorov's theorem as irrelevant. They based their criticism mainly on the fact that the inner functions p,,. are highly nonsmooth and the output functions gq are not in a parameterized form. However, this criticism was not convincing: Kurkova (1991) argued that highly nonsmooth functions can be regarded as limits or sums of infinite series of smooth functions, and the problems in realizing a Kolmogorov network can be eliminated through approximately implementing (~4. and g, with known networks. In this note we present our view on the discussion from a more essential point of view. Since P(., in equation 0.1 should be universal, Kolmogorov's theorem can be regarded as a proof of a transformation of representation of multivariate functions in terms of the Q univariate output functions g,. [In some improved versions of Kolmogorov's theorem it is proved that only one g in equation 0.1 is necessary (Lorentz 19661.1 Such a strategy is embedded in the network structure as shown in Figure 1. (Note that the block T is independent off.) If Figure 1 is thought of as a general network structure for approximation of multivariate functions, a question is whether an arbitrarily given multivariate function f can be (approximately) implemented through an (approximate) implementation of the corresponding Q univariate functions g,. To this question we have an answer as stated below: Proposition. In Figure 1, an approximate implementation of gq does not in general deliver an approximate implementation of the original function f, unless g, can be exactly implemented. Neural Computation 5,18-20 (1993)


Realization of a Kolmogorov Network

19

: IT1 Figure 1: The basic strategy in Kolmogorov’s theorem is embedded in a network structure where a universal transformation T maps the multidimensional Euclidean space (the domain of multivariate functions) into one or several unidimensional ones. Here we mean by function approximation a mechanism that provides an estimation of the corresponding output (function value) at an arbitrary point in the input space (domain), which is meaningful in some sense (e.g., an interpolation of some given samples). Such a procedure is closely related to the “spatial relation’’ (e.g., the Euclidean distance) defined in the input space. For instance, from the viewpoint of interpolation, an estimation of the function value at a point depends on its position relative to that of the sample points in the domain, as well as on the sample values. As we know, there does not exist a homeomorphism between a multidimensional Euclidean space and an interval in the real line, i.e., there does not exist a way to map points from the former into the latter while preserving the spatial relations between them. (It is due to this nonhomeomorphism that the inner functions vqnmust be highly nonsmooth;) That means, approximating gq in the real line may not have an equivalent meaning to that of approximating the original function f in a multidimensional space. For instance, a ”reasonable” interpolation in an interval in the real line between two samples of a gq may lose its meaning in the corresponding regions in the multidimensional domain off. In fact, there exist various methods of one-to-one mapping from a multidimensional space to a unidimensional one. (The construction of the inner functions vqnin Kolmogorov‘s theorem is only one of them.) Therefore, it is not difficult to understand that the structure in Figure 1

20

Ji-Nan Lin and Rolf Unbehauen

is, theoretically, for an exact representation of multivariate functions. If the uni-input subnetwork in Figure 1 are assumed to be adjustable for each input point, that is, they are able to provide for each input point an arbitrarily desired output independent of the other points, then g, and thus f can be exactly implemented by the network. However, such an assumption has little significance in practice. On the other hand, from the viewpoint of information theory, the implementation of the univariate functions g, does not mean a simplification of that of the original multivariate one f , since the universality of the inner structure T implies that all the information describing f should be carried by 8,. Based on the above discussion, we believe that Kolmogorov’s theorem is irrelevant to neural networks for mapping approximation. The discussion is illustrated by a network model with an example of function approximation (Lin and Unbehauen 1992). The consequence of our discussion is not encouraging for efforts toward constructing mapping approximation networks along the lines of Kolmogorov’s theorem and its proofs. However, if we take equation 0.1 as a general approximation representation of multivariate functions in terms of summation and superposition of univariate functions, it is closely relevant to mapping networks. Some useful neural networks (e.g., the perceptron type network) can be represented by equation 0.1.

References Girosi, F., and Poggio, T. 1989. Representation properties of networks: Kolmogorov’s theorem is irrelevant. Neural Comp. 1,465469. Hecht-Nielson,R. 1987. Kolmogorov‘s mapping neural network existence theorem. In Proceedings of the International Conference on Neural Networks, pp. 111, 11-14. IEEE. Kolmogorov, A. N. 1957. On the representation of continuous functions of several variables in the form of a superposition of continuous functions of one variable and addition. Dokl. Akad. Nauk USSR 114(5),953-956. Kurkova, V. 1991. Kolmogorov’s theorem is relevant. Neural Comp. 3, 617-622. Lin, J.-N., and Unbehauen, R. 1992. A simplified model for understanding the Kolmogorov network. In Proceedings URSl Int. Symp. on Signals, Systems and Electronics, ISSSE 92, Paris, 11-14. Lippmann, R. P. 1987. An introduction to computing with neural nets. I E E E ASSP Mag. 4-22. Lorentz, G. G. 1966. Approximation offunctions, Holt, Rinehart & Winston, New York. Received 6 February 1992; accepted 5 June 1992.

This article has been cited by: 1. Allan Pinkus. 1999. Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143. [CrossRef] 2. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef] 3. Mohammad Bahrami. 1995. Issues on representational capabilities of artificial neural networks and their implementation. International Journal of Intelligent Systems 10:6, 571-579. [CrossRef]

Communicated by Lawrence Abbott

Statistical Mechanics for a Network of Spiking Neurons Leonid Kruglyak' William Bialek Department of Physics, and Department of Molecular and Cell Biology, University of California at Berkeley, Berkeley, CA 94720 USA and NEC Research Institute, 4 Independence Way, Princeton, N]08540 USA

We show that a simple statistical mechanics model can capture the collective behavior of large networks of spiking neurons. Qualitative arguments suggest that regularly firing neurons should be described by a planar "spin" of unit length. We extract these spins from spike trains and then measure the interaction Hamiltonian using simulations of small clusters of cells. Correlations among spike trains obtained from simulations of large arrays of cells are in quantitative agreement with the predictions from these Hamiltonians. We comment on the novel computational abilities of these "XY networks."

1 Introduction Understanding the computations performed by biological nervous systems requires methods for describing the collective behavior of large networks of neurons. For a physicist, the natural tool is statistical mechanics. The development of the Hopfield model (Hopfield 1982) and its relation to king spin glasses have resulted in the many recent efforts along these lines (Amit 1989). While these neural network studies have produced a number of interesting results, they have been based on extremely simplified models of neurons-often just two-state devices. The hope has been that most microscopic biological details are not important for the collective computational behavior of the system as a whole. Renormalization group ideas have taught us that microscopic details are often irrelevant, but in the neural case it is unclear exactly where to draw the line between important features and incidental details. Most real neurons produce trains of identical action potentials, or spikes, and it is the timing of these spikes that carries information; in many cases significant computations are carried out on time scales comparable to the interspike intervals (de Ruyter van Steveninck and Bialek 1988; Bialek et al. 1991). 'Present address: Theoretical Physics, Oxford University, 1 Keble Rd., Oxford OX1 3NP.

Neural Computation 5,21-31 (1993)


22

Leonid Kruglyak and William Bialek

How do we relate the observable sequences of spikes to the local spins or fields in a statistical mechanics model? In this paper we begin with a semirealistic model for spiking neurons in a network and systematically construct a corresponding statistical mechanics model for interacting spins. This model is successful in that it accurately describes the correlations among spike trains observed in simulations of large arrays of interconnected neurons. The construction is largely numerical, and while we can offer analytical explanations only for some of its successes, we believe that the numerical results themselves are of considerable interest. We emphasize that the goal of this work is not to model a particular part of the nervous system but rather to show that an explicit reduction of a network of spiking cells to a statistical mechanics spin system is possible.

2 A Model for Spiking Cells

We use the Fitzhugh-Nagumo (FN) model (Fitzhugh 1961; Nagumo et al. 1962) to describe the electrical dynamics of an individual neuron. This model demonstrates a threshold for firing action potentials, a refractory period, and single-shot as well as repetitive firing-in short, all the qualitative properties of neural firing. It is also known to provide a reasonable quantitative description of several cell types (Rinzel and Ermentrout 1989; Fitzhugh 1969). To be realistic it is essential to inject into each cell a noise current bin(t ) , which we take to be gaussian, spectrally white, and independent in each cell n. We model a synapse between two neurons by exponentiating the voltage from one and injecting it as current into the other. Our choice is motivated by the fact that the number of transmitter vesicles released at a synapse is exponential in the presynaptic voltage (Aidley 1971); other synaptic transfer characteristics, including small delays, give results qualitatively similar to those described here. We emphasize that all of our methods for analyzing the FN model with exponential synapses have been applied to other models of spike generation with essentially identical results (Kruglyak 1990). In particular, both a simpler "integrate and fire" model (Gerstner 1991) and a more complicated channel model (Rinzel and Ermentrout 1989; Morris and Lecar 1981) in which regular bursts of spikes play the same role as the spikes do in the FN model were examined in detail. The equations of motion for a network of interacting noisy FN neurons are

(2.1)

Network of Spiking Neurons

23

where V , is the transmembrane voltage in cell n, l o is the dc bias current, and the W,,are auxiliary variables; VOsets the scale of voltage sensitivity in the synapse. Voltages and currents are dimensionless, and the parameters of the system are expressed in terms of the time constants T~ and T~ and a dimensionless ratio a. The FN model with noise has the standard sigmoidal input/output relation if one plots the firing rate vs. dc injected current in a single cell.' Most neural network models take this simple relation to be the central feature of neural firing. However, the sigmoidal i/o relation hides the experimentally salient distinction between regular and irregular patterns of firing. Regular firing is characterized by a tightly bunched interspike interval distribution; in the irregular case the distribution is approximately Poisson. These different regimes correspond to different parameter values in the model, and experiments on sensory neurons suggest that many cells are confined to one regime or the other under natural conditions (Teich and Khanna 1985; Goldberg and Fernandez 1971). Networks of neurons in the two regimes should exhibit very different collective properties. We study in detail the case of regular firing. If the intervals between spikes cluster tightly about a mean value, the firing is nearly periodic, and we expect that it should be describable by an underlying oscillation. Note that we are not talking about a perturbed oscillation of the noiseless FN equations; in fact, the simulations described below are carried out in a parameter region where those equations do not oscillate-the oscillations we see are noise-driven. Also, when cells are coupled together the interactions cannot be treated as a small effect since a single spike from one cell can trigger a spike in another cell. Hence the standard analytic methods for reducing complex nonlinear oscillations to phase equations (Kuramoto 1984) do not apply, and we have to follow our intuition and attempt to extract the underlying oscillation numerically.

3 From Spike Trains to Spins

We assume that spikes occur at times ti when the total phase of an oscillation crosses zero (mod27r). Then this oscillation has a mean frequency wo that is simply related to the mean interspike interval and a slowly varying phase 4(t) that describes the deviations from perfect periodicity. If this description is correct, the spike train s ( t ) = xi 6 ( t - t i ) should have a power spectrum with well resolved peaks at kwo, f2w0,.. .. This is what we observe in simulations of the FN model for one isolated cell. 'The noiseless behavior is quite different-there is a threshold current below which the rate is zem and above which it jumps to a finite value. There is also a second threshold current above which the cell ceases to fire.


24

We then low-pass filter s ( t ) to keep only the f w o peaks, obtaining a phase and amplitude modulated cosine,

[Fs](t)x A ( t )C O S [ U ~+~ 4(t)]

(3.1)

where [Fs](t)denotes the filtered spike train, and the amplitude A ( t ) and the phase 4(t) vary slowly with time. Plotting the filtered spike train against its time derivative results in a phase portrait characteristic of a noisy sinusoidal oscillation, confirming equation 3.1. As indicated earlier, very similar phase portraits are obtained for the Gerstner and bursting Morris-Lccar models (Kruglyak 1990). Hence the spike train can be described by a two-vector in the phase plane. We do not expect the magnitude of this vector to matter since A(t) is related to the unimportant details of the filtering process and to the biologically irrelevant differences among spikes. The orientation of the vector, now assumed to be of unit length, gives us the phase. Using the phase portrait it is thus possible to process the spike train from a neuron and recover a time-dependent, planar unit spin S ( t ) . We now want to see how these spins interact when we connect two cells via synapses. 4 From Synapses to Spin-Spin Interactions

We characterize the two-neuron interaction by accumulating a histogram of the phase differences between two neurons connected via a model synapse? A variety of synaptic interactions have been examined; the results below though not the exact form of the interaction hold in every case. The probability distribution of the phase difference defines an effective Hamiltonian, P(&,&) o( exp[-H($l - 42)]. Note that this Hamiltonian is simply another way to characterize the equilibrium distributionit is not meant to describe the time evolution of the system. Hence it remains a useful concept even when the standard notion of an energy function breaks down. This is in contrast with the usual statistical mechanics approach to neural networks, which assumes a Liapunov dynamics for the noiseless case and then treats all noise by promoting the Liapunov function to the role of a true Hamiltonian and placing it at finite temperature. The assumption of Liapunov dynamics cannot be justified for biological networks, and the noise can be more complicated (Crair and Bialek 1990). Figure 1 shows the effective Hamiltonian for a pair of symmetrically connected cells. We see that with excitatory synapses (1 > 0) the interaction is ferromagnetic, as expected. Once again, the Hamiltonians for the Gerstner and bursting Morris-Lecar models show only minor variations from the FN Hamiltonian (Kruglyak 1990). *An interaction that depends only on the phase difference is only the simplest case; there could in principle be a dependence on the absolute phase of one of the cells as well. As mentioned below and described in detail elsewhere, no such dependence is seen when we look at probability distributions of phases in small clusters of cells.


25

Effective Hamiltontans 6.0

4.0

I

*r^2, $0

8.0

'

V

-0.0

0.0

P

2.0

4.0

h o(llmne.

Figure 1: Effective Hamiltonians for three values of the coupling strength I. From simulations of equation 3.1 for two cells with 71 = 0.1, Q = 10,Io = -0.25, Vo = 0.3,a = 1.1, and 61, of spectral density S, = 1.25 x With arrays of more than two neurons it is possible that the effective Hamiltonian includes more than just a simple nearest-neighbor interaction. We have searched for these effects in simulations on small clusters of cells, and always find that the observed phase histograms can be accounted for by appropriate convolutions of the histograms found in the two-neuron simulations. This leads us to predict that the statistical mechanics of an entire network will be described by the effective Hamiltonian H = CijHij(4i - 4j), where Hq(+i - 4j) is the effective Hamiltonian measured for a pair of connected cells i , j as in Figure 1. 5 Correlation Functions

One crucial consequence of equation 3.1 is that correlations of the filtered spike trains are exactly proportional to the spin-spin correlations that are the natural objects in statistical mechanics. Specifically, if we have two cells n and rn,

(Sn . S m ) = (cos(4n - 4m)) = ~,-~([FsnI(t)[FsmI(t))

(5.1)

This relation shows us how the statistical description of the network can be tested in experiments that monitor actual neural spike trains.

Lmnid Kruglyak and William Bialek

26

5.1 One Dimension. It is well known that when planar spins are connected in a one-dimensional chain with nearest-neighbor interactions, correlations between spins drop off exponentially with distance. The mapping from spike trains to spins predicts that this exponential decay will also be observed in the spike train correlation functions. To test this prediction we have run simulations on chains of 32 Fitzhugh-Nagumo neurons connected to their nearest neighbors. Correlations computed directly from the filtered spike trains as indicated above indeed decay exponentially, as seen in Figure 2. More complicated correlation functions are predicted and observed with, for example, delayed or asymmetric synaptic interactions (Kruglyak 1990). In Figure 3 we compare the correlation lengths predicted from the effective Hamiltonians determined as described above with the correlation lengths extracted from the realistic simulations; the agreement is excellent. We emphasize that while the “theoretical” correlation lengths are based on the statistical mechanics of a simple spin model, the “experimental” correlation lengths are based on measured correlations among spike trains generated by realistic neurons. There are no free parameters.

Correlation us. Distance i n 1 Damsns-ion

0.0

10.0

6.0 M - e

(n &tUw

16.0

zo.0

Spdnoa

Figure 2: Correlation function in 1 dimension for three values of the coupling strength (lowest curve, J = 0.0005, middle curve, J = 0.001, highest curve, J = 0.0015). The points are obtained from simulations of chains of neurons; the lines are best-fit exponentials.


27

6.0

4.0

!

j

3.0

E

i

2.0 1.0

0.0

0.0

1.0

2.0

3.0

4.0

6.0

Ihdlcted C m l o l l a Length

Figure 3: Correlation length obtained from fits to the simulation data vs. correlation length predicted from the Hamiltonians. 5.2 ’Itvo Dimensions. In the twodimensional case we connect each neuron to its four nearest neighbors on a square lattice. The corresponding spin model is essentially the XY model, but from Figure 1 we see that the interaction potential between neighbor spins has a somewhat different form, with the energy rapidly reaching a plateau as the spins tilt apart. The XY model itself is of course well understood (Kosterlitz and Thouless 1973; Nelson 1983), and models with potentials as in Figure 1 should be in the XY universality class. To get a feeling for the relation between the Hamiltonian in Figure 1 and the XY model we follow Jose et al. (1977) and carry out a MigdalKadanoff renormalization. We recall that for the XY model itself this approximation leads to an “almost” fixed line at low temperatures, while above the transition the flow is to infinite temperature. We find the same results in our model; the “almost” fixed line is the same and flow to infinite temperature is asymptotically along the same curve as in the XY case. The similarity of flows in the XY model and in our model is so great that we feel confident in predicting the conditions under which the network should exhibit algebraic or exponential decay of correlations. We then check these predictions directly by simulating two-dimensional arrays of neurons with toroidal boundary conditions. Figure 4 shows


28

a

0.0

-2.0

I

-4.0

0.0

6.0

10.0

16.0

DLttonce tn LcJt(ca Spoclns.

b

..

-0.4

I

I 0.

_."

"tttt

-n

0.0

1.0

2.0

8.0

4.0

Figure 4: Conelation function in 2 dimensions. (a) Above the transition, loglinear plot; linear behavior indicates exponential fall-off. (b) Below the transition, log-log plot; linear behavior indicates algebraic fall-off. the spike train correlations as a function of distance for two coupling strengths, one below and one above the transition point suggested by the renonnalization calculation. The low coupling (high T ) data are taken on a 32 x 32 lattice and are well described by exponential fall-off, at least until the correlations are so small as to be lost in the statistical noise. In the strong coupling (low T ) case we study a 128 x 128 lattice and find that


29

the correlations first decay algebraically and then plateau and become irregular due to finite size effects. In the algebraic phase the long-distance behavior of the system should be describable in terms of spin waves (Kosterlitz and Thouless 1973). We can fit the data of Figure 4b, except at the shortest distances, with an exact lattice spin wave calculation in which the critical exponent r] x 0.2. This is reasonable since we are just below the apparent phase transition, 0.25. We have also looked at the effects of several kinds of where r] disorder (deleting a fraction of connections, randomly choosing coupling strengths, randomly assigning the inherent firing frequencies of the cells) and find at most a shift in the transition coupling with no apparent qualitative changes when the disorder is small (Kruglyak 1990; Kruglyak and Bialek 1991b). In particular, the phase with algebraic decay of correlations is preserved, albeit at stronger couplings. This result also holds for the other neural models described in Section 2. This robustness to changes in connectivity, connection strengths, and internal dynamics of neurons gives us confidence that the model discussed in this paper is applicable to real biological systems. 6 Discussion

To summarize, we have found a systematic procedure for extracting spin variables from spike trains for a particular class of realistic model neurons. We then measure an effective interaction Hamiltonian by simulating small clusters of cells. This allows us to formulate a statistical mechanics model for a network of spiking cells by directly referring to a more microscopic model rather than by simply postulating a coarsegrained description. We use correlations between spike trains to test the collective properties of networks predicted from the model and find that the predictions are both qualitatively and quantitatively correct. The fact that our particular network is described by an XY-like model is especially interesting. There are many computational problems, especially in vision, where one would like to make comparisons among signals carried by neurons separated by large physical distances in a given layer of the brain. It has been traditionally assumed that the range of possible comparisons is limited by the range of physical interconnections, which is typically quite short. Physically we know that this is wrong, since we can have long-range correlations near a critical point even when the microscopic interactions are short ranged. The difficulty is that the system must be carefully poised at the critical point, but this is not a problem in XY systems, which have a critical line and hence an entire regime of long-range correlations. Reasonably regular two-dimensional architectures are common in regions of the nervous system devoted to sensory information processing. In many of these systems one can observe neural responses to stim-

30


uli that provide direct input only to very distant neurons (Allman et al. 1985);such responses are described as coming from ”beyond the classical receptive field.” The power-law decay of correlations in XY-like models may provide an apt description of the gradual decrease in responsiveness to more distant stimuli found in these experiments. If the XY scenario is applicable, we expect that these neurons should be regularly firing, and there is recent evidence for such oscillatory behavior in cortical cells that exhibit long-range responses (Gray and Singer 1989; Gray et al. 1989; Eckhorn et al. 1988). Simulations show that local temporal correlations in such networks can indeed carry information about large-scale spatial properties of the stimulus (Kruglyak 1990; Kruglyak and Bialek 1991a).

Acknowledgments We thank 0. Alvarez, D. Arovas, A. B. Bonds, K. Brueckner, M. Crair, E. Knobloch, and H. Lecar for helpful discussions. The work at Berkeley was supported in part by the National Science Foundation through a Presidential Young Investigator Award (to W. B.), supplemented by funds from Cray Research, Sun Microsystems, and the NEC Research Institute, by the Fannie and John Hertz Foundation through a Graduate Fellowship (to L. K.), and by the USPHS through a Biomedical Research Support Grant.

References Aidley, D. J. 1971. The Physiology of Excitable Cells. Cambridge University Press, Cambridge. Amit, D. J. 1989. Modeling Brain Function. Cambridge Universisty Press, Cambridge. Allman, J., Meizin, F., and McGuiness, E. 19851 Stimulus specific responses from beyond the classical receptive field: Neurophysiological mechanisms for local-global comparisons in visual neurons. Annu. Rev.Neurosci. 8,407. Bialek, W., Rieke, F., de Ruyter van Steveninck, R. R., and Warland, D. 1991. Reading a neural code. Science 252,1854. Crair, M. C., and Bialek, W. 1990. Non-Boltzmann dynamics in networks of spiking neurons. In Advances in Neural lnformation Processing Systems, 2, D. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. de Ruyter van Steveninck, R. R., and Bialek, W. 1988. Real-time performance of a movement-sensitive neuron in the blowfly visual system: Coding and information transfer in short spike sequences. Proc. R. SOC.Lond. B 234,379. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern. 60,121. Fitzhugh, R. 1961. Impulses and physiological states in theoretical models of nerve membrane. Biophys. J. 1,445-466. Fitzhugh, R. 1969. Mathematical models of excitation and propagation in nerve. In Biologicul Engineering, H. P. Schwan, ed., Chap. 1. McGraw Hill, New York.


31

Gerstner, W. 1991. Associative memory in a network of “biological” neurons. In Advances in Neurallnformation Processing Systems, 3, D. Touretzky,ed. Morgan Kaufmann, San Mateo, CA. Goldberg, J. M., and Fernandez, C. 1971. Physiology of peripheral neurons innervating semicircularcanals of the squirrel monkey. 111: Variations among units in their discharge properties. 1.Neurophys. 34, 676. Gray, C. M., KONg, P., Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature (London) 338,334. Gray, C. M., and Singer, W. 1989. Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl. Acad. Sci. U.S.A. 86, 1698. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79,2554. Josh,J. V., Kadanoff, L. P., Kirkpatrick, S., and Nelson, D. R. 1977. Renormalization, vortices, and symmetry-breakingperturbations in the two-dimensional planar model. Phys. Rev.B 16,1217. Kruglyak; L. 1990. From biological reality to simple physical models: Networks of oscillating neurons and the XY model. Ph.D. thesis, University of California at Berkeley, Berkeley, CA. Kruglyak, L., and Bialek, W. 1991a. Analog computation at a critical point: A novel function for neuronal oscillations? In Advances in Neural Information Processing Systems, 3, D. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. Kruglyak, L., and Bialek, W. 1991b. From biological reality to simple physical models: Networks of oscillating neurons and the XY model. In Neural Networks: From Biology to High Energy Physics, 0. Benhan, C. Bosio, P. Del Giudice, and E. Tabet, eds., ETS Editrice, Pisa, 1992. Kosterlitz, J. M., and Thouless, D. J. 1973. Ordering, metastability, and phase transitions in two-dimensional systems. J. Phys. C: Solid State Phys. 6,1181. Kuramoto, Y. 1984. Chemical Oscillations, Waves, and Turbulence. Springer, Berlin. Morris, C., and Lecar, H. 1981. Voltage oscillations in the barnacle giant muscle fiber. Biophys. J. 35, 193-213. Nagumo, J. S.,Arimoto, S., and Yoshizawa, S. 1962. An active pulse transmission line simulating a nerve axon. Proc. I. R. E . 50, 2061. Nelson, D. R. 1983. Defect-mediated phase transitions. In Phase Transitions and Critical Phenomena, C. Domb and J. L. Lebowitz, eds., Vol. 7, Chap. 1. Academic Press, London. Rinzel, J., and Ermentrout, G. B. 1989. Analysis of neural excitability and oscillations. In Methods in Neuronul Modeling, C. Koch and I. Segev, eds., Chap. 5. The MIT Press, Cambridge, MA. Teich, M. C., and Khanna, S. M. 1985. Pulse-number distribution for the neural spike train in the cat‘s auditory nerve. J.Acoust. SOC.Amer. 77, 1110. Received 3 September 1991; accepted 29 May 1992.

This article has been cited by: 2. Henry Tuckwell, Laurent Toubiana, Jean-Francois Vibert. 2000. Enhancement of epidemic spread by noise and stochastic resonance in spatial network models with viral dynamics. Physical Review E 61:5, 5611-5619. [CrossRef] 3. Henry Tuckwell, Laurent Toubiana, Jean-Francois Vibert. 1998. Spatial epidemic network models with viral dynamics. Physical Review E 57:2, 2163-2169. [CrossRef] 4. Roger Rodriguez, Henry Tuckwell. 1996. Statistical properties of stochastic nonlinear dynamical models of single spiking neurons and neural networks. Physical Review E 54:5, 5585-5590. [CrossRef]

Communicated by James Anderson

Acetylcholine and Learning in a Cortical Associative Memory Michael E. Hasselmo Department of Psychology, Harvard University, Cambridge, MA 02138 USA

Implementing associativememory function in biologically realistic networks raises difficulties not dealt with in previous associative memory models. In particular, during learning of overlapping input patterns, recall of previously stored patterns can interfere with the learning of new patterns. Most associative memory models avoid this difficulty by ignoring the effect of previously modified connections during learning, thereby clamping activity to the patterns to be learned. Here I propose that the effects of acetylcholine in cortical structures may provide a neurophysiological mechanism for this clamping. Recent brain slice experiments have shown that acetylcholine selectively suppresses excitatory intrinsic fiber synaptic transmission within the olfactory cortex, while leaving excitatory afferent input unaffected. In a computational model of olfactory cortex, this selective suppression, applied during learning, prevents interference from previously stored patterns during the learning of new patterns. Analysis of the model shows that the amount of suppression necessary to prevent interference depends on cortical parameters such as inhibition and the threshold of synaptic modification, as well as input parameters such as the amount of ovelc lap between the patterns being stored. 1 Introduction

A wide range of neural network models have suggested that associative memory function may depend on excitatory intrinsic connections within the cortex. These include both linear associative matrix memories (Anderson 1983; Kohonen 1984) and models related to spin glass systems (Hopfield 1982). However, the majority of these models have focused on network dynamics during the recall of previously stored memories. During learning of new memories, most associative memory models ignore the effect of intrinsic connections within the network by clamping the activity of units to the desired pattern, by computing synaptic modification independently of network dynamics, or by applying learning before the spread of activation. This allows use of the Hebb rule for computation Neural Computation 5 , 3 2 4 (1993)


Acetylcholine and Learning

33

of a clean outer product for each input pattern being stored. The sum of these outer products is computed for m different patterns and stored as an intrinsic excitatory connectivity matrix By as follows, where A!p) represents element i of pattern p:

Despite the common use of clamping in associative memory models, no neurophysiological mechanism has previously been presented describing how the brain might suppress normal network recall dynamics during learning. Without clamping, and with learning rates that are similar to or slower than the update of activation, recall of previously stored patterns will interfere with the learning of new patterns. This paper presents a neurophysiological mechanism that may prevent interference during learning in a cortical associative memory. 2 The Problem: Interference during

Learning

The learning rule in associative memory models is taken to be analogous to the phenomenon of long-term potentiation within cortical structures. However, biological evidence suggests that long-term potentiation cannot occur until the presynaptic activity (aj) has reached the terminal bouton and influenced postsynaptic activity (ai). Thus, associative memory models should not apply a learning rule until activity has propagated across the synapse being modified. Rather than applying the learning rule immediately, before the spread of activity, as in ABij(t) = ai(t)aj(t), learning should be applied only after synaptic transmission has had time to influence the postsynaptic activity, as in ABij(t 1) = ai(t l)aj(t). This presents a difficulty for associative memory models. Unless activity is clamped to the desired pattern during learning, activity will be influenced by intrinsic connections modified by previously stored patterns. Thus, when a new pattern is presented to the network, the activity will depend partly on previously learned patterns that overlap with the new pattern. For instance, after learning of one pattern A ( ] ) ,the connectivity matrix Bfi = A&')&(*).In this case, presentation of a second pattern A@)will result in postsynaptic activity ai combining the input pattern and the spread of activation along excitatory intrinsic connections:

+

+

+ where n = the number of neurons in the network. If this activity is taken as the postsynaptic activity in the Hebbian learning rule, and presynaptic

Michael E. Hasselmo

34

activity is taken as U j = A:), the modification of synaptic strength takes the form: ABij(t + 1) =

Ui(t

+ l ) ~ j ( =t )

Thus, in addition to the outer product of the pattern with itself, Ai@)Aj@), the connectivity matrix also contains the outer product between the two patterns stored, Ai(1)Aj(2), scaled to the dot product between these two patterns. For orthogonal patterns, the dot product equals zero, and no interference during learning occurs. However, for nonorthogonal patterns, interference during learning will occur between each new pattern p and all previously stored patterns q according to their direct overlap, and also according to the overlap between these patterns and intervening patterns stored within the network. This adds an interference term to the learning rule as follows:

[

AB!@ '1 = A!P)A!P) ' I + ql=l

91 0 for any matrix M # 0. (For the mathematically minded reader this shows that E can be thought of as a Lyapunov functional.) The next step is to show that the update 2.5 can be implemented as an algorithm requiring no nonlocal calculations. For this we rewrite 2.5 as an update for K-I instead of KT, using the fact 6K-' = -K-'.bK.K-'. For later convenience, we also change notation now and define W = K-I; the variables Wv will designate the actual synaptic strengths in the feedback network introduced in the next section. Then in terms of W the algorithm 2.5 becomes (2.8)

where T is a time constant setting the update rate. Although W-I does appear in 2.8, it appears in the combination (W-l)T, which as will be discussed in the next section is equal to (SiOj),and thus in the network Re

Algorithm for Sensory Receptive Field Development

49

implementation no nonlocal computations will be needed. The algorithm 2.8 can be recognized as the ensemble averaged form of the Goodall (1960) algorithm. Since direct minimization of E{K} in 2.2 gives the global minimum 2.3 with no local minima, our demonstrating that 2.8 or 2.5 always reduces E{K} is sufficient to prove convergence. However, for completeness we shall also give the old proof of convergence (Goodall 1960) that does not require a minimization measure. One reason we give this proof here is that it leads us to rewrite the update algorithm 2.8 in yet another form that will be useful in our later discussion of symmetry fixing and locality. To start the proof we need, in addition to 2.8, the transpose equation (2.9) using R = RT. By multiplying 2.8 from the right by WT and 2.9 from the left by W and then adding the resulting equations, we arrive at the integrable differential equation r dt d(W.WT)=2(R-W.WT)

(2.10)

which has the solution

where C is a constant matrix determined by initial conditions. In the limit where t / r becomes large We WT = R, which is just the minimum E{K} solution 2.3 since W = K-'. 3 Network Implementation

In this section we discuss a network implementation of the above algorithm. As shown in Figure 1 the network has one nontrivial layer, with feedforward connections directly from the input Si and lateral feedback connections among the neurons. It is assumed that the input Si is directly connected only to the ith neuron, with link strength unity. The plastic links in this network are the lateral feedback links that connect the output of the jth neuron back to the input of the ith neuron with link strength Wq. The neurons are assumed to be linear, thus the dynamics of this network can be written as

where Oi is the output of the ith neuron, Sj is its feedforward input, and

50

Joseph J. Atick and A. Norman Redlich

01

Figure 1: The architecture of the network used for decorrelation. It is assumed that the ith neuron receives direct input with synaptic weight unity from Si and feedback input from the outputs of all neurons with weights W,. Only the feedback links Wv are updated during learning.

T is a time constant. Below we shall assume that T 1 (see Fig. 2) defines a linear chain with nearest-neighbor transitions. Note, that the structure of the clustering cost functions 2.1 and 2.2 is the same. We have introduced only a generalized distortion measure ((Dial in the topological case. Topology preserving clustering reduces to nontopological clustering if we identify ,T = ,S with ,S being the Kronecker symbol (S, = 1 if (I: = 7; ,S = 0 if a # y). The clustering schemes based on 2.1 or 2.2 do not take any a priori knowledge about data classes into account. Many applications, however, provide information how to partition the training data into a set of classes y. Given such a priori class information for the training data, it is desirable that clusters do not overlap with class boundaries, for example, data belonging to different classes should not be assigned to the same cluster. This knowledge about the class membership Fir E [0,1], C, Fir = 1Vi of data point i in class y allows us to supervise a clustering process. The costs associated with supervised clustering depend on the joint probability pra = CiFi,Mi,/N of a data point out of class y being assigned to cluster a. The uniqueness of class membership implies the relation pa = &p,,. We introduce a new cost term S,, called supemision costs, that penalizes assignments of data from different classes to the same cluster. S, should be positive and it should vanish if the conditional probability p,la, that a data point assigned to cluster a is member of class y, approached certainty, that is, p,la = p,,/p, = 1. A supervision cost term that satisfies this condition is defined by the ”conditional class entropy” S, = - C, p,l, log(p,l,). Adding the supervision costs S, to the logarithmic complexity costs C, = -log@,) both cost terms being weighted equally (AS = AC = A) we arrive at the “supervised complexity costs” C; = - C, p , ~ , log(p,,) which penalizes too complex clustering solutions and includes a priori class information. As discussed in Bichsel and Seitz (1989) the ”conditional class entropy” as a penalty term favors a partitioning of the input space with clusters of unique class adherence, a very valuable property for any subsequent classification stage. The simplest functional form of S, that is polynomial in p,, with a root at P,l, = 1 is S, = C,P,l,(l - p,la). 3 Maximum Entropy Estimation of y, and p ,

Different combinations of complexity terms, topology constraints, and class information for supervised clustering define a variety of clustering algorithms that are relevant in very different information processing contexts. To derive robust, preferably parallel algorithms for these data clustering cases, we study the optimization problem 2.2 in a probabilistic framework (see Rose et al. 1990). This design philosophy for

Joachim Buhmann and Hans Kiihnel

80

optimization algorithms is motivated by the success of simulating annealing (Kirkpatrick et al. 1983) and of neural optimization algorithms (Durbin and Willshaw 1987; Yuille 1990; Simic 1990). The most likely distribution of assignment variables Mi, can be determined by the maximum entro y principle (Jaynes 1957), and equals the Gibbs distribution P ( { M } ) = exp(-PEK). The "computational temperature" T = 1/P, which plays the role of a Lagrange parameter for the average clustering costs, controls the randomness in the assignment process. The partition function Z normalizes the Gibbs distribution. Statistical physics (see, e.g., Amit 1989; Rose et al. 1990) states that maximizing the entropy at a fixed temperature is equivalent to minimizing the free energy

P

K

i

N

with respect to the variables paryurparyu (see Buhmann and Kuhnel 1992c for details). The auxiliary variables p a , ya are Lagrange parameters that enforce the constraints pa = Ezl Mi,/N and Ci ,&Mi7Tya(a/?a)Dia = 0, respectively. It can be shown that all minima of FK have ya I 0 and pa = Xpa(aCa/apa). The fact that ye vanishes3 is implied by the definition of ya as the centroid of cluster a. The resulting reestimation equations for the expected cluster probabilities and the expected centroid positions are (3.2) (3.3)

Equations 3.2 and 3.3 are necessary conditions of FKbeing minimal. We have identified pa, y. in 3.1-3.4 with their expectation values, which is 31n self-organizing feature maps yo is usually defined as the center of mass of the Voronoi cell a instead of the generalized centroid C i C 7 M i Y T f u ~ Z ) i o ( X i , y=u )0, which produces a nonvanishingconjugate field ya.

Complexity Optimized Data Clustering

81

justified in the large N limit. The expectation value (Mi,)of the assignment variable Mi, can be interpreted as a fuzzy membership of data point xi in cluster a. The fuzziness that is controlled by the temperature T sets a resolution limit below which different clusters cannot be discriminated. In the case of supervised clustering with the cost function 2.2 supplemented by supervision costs, the free energy generalizes to

&

The term Firpya = A c p a ( X a / d p a )+ Aspa C,"=Ir i y ( a & / @ y a ) enforces the constraint pya = & I'iyMi,/N. Equation 3.2 has to be generalized to a reestimation of the average joint probabilities pay, that is,

The reestimation equation for the cluster centroids has the same form as in 3.3 with (Mi,) given by 3.7. See Buhmann and Kuhnel (1992a) for a more detailed derivation of the equations 3.5-3.7. The global minimum of the free energy 3.1 with respect to p,,y, (y,, jj, already inserted) determines the maximum entropy solution of the cost function 2.1. Note that the optimization problem 2.1 or 2.2 of a KN state space has been reduced to a K(d+l)-dimensional minimization of the free energy FK(3.1 or 3.5). To find the optimal parameters p a , y, and the number of clusters K that minimize the free energy we start with one cluster located at the centroid of the data distribution, split that cluster, and reestimate p,, ya using equations 3.2 and 3.3. The new configuration is accepted as an improved solution if the free energy 3.1 has been decreased. This splitting and reestimation loop is continued until we fail to find a new configuration with lower free energy. An iterative splitting strategy is required because the cost function exhibits numerous local minima, especially since we compare solutions with different numbers of clusters. Cluster splitting in the topological case should be performed such that a new cluster does not violate the topology constraint severely,

Joachim Buhmann and Hans Kuhnel

82

~~

I

* *

+

*

* ' * * I * ** 1 . * . * * * *+* * c ' * 1 . * . * . * I * * +** * * * ' t

Figure 1: A data distribution (4000 data points) (a), generated by four normally distributed sources, is clustered with the complexity measure C, = -logpa. Two zero temperature solutions for X = 2.5,0.4 are shown in (b,c) where plus signs (+) denote the positions of the gaussians and stars (*) denote cluster centers.

that is, it should not give rise to a topological defect. The resulting configuration is stable under single cluster splitting, but we are not assured to find the global minimum. Note the difference between our approach to determine the number of clusters and the suggestion by Rose et al. (1990) to use the temperature as a control parameter. The temperature determines the fuzziness of a clustering solution, whereas the complexity term penalizes excessively many clusters. In the case of hard clustering (T + 0) we still like to limit the number of clusters by an application dependent complexity measure. Nontopological (T,? = 6,J clustering results at zero temperature for the logarithmic complexity measure (C, = - log p,) are shown in Figure 1. At high complexity costs (b) the algorithm finds four clusters located at the centers of the gaussians. In the limit of very small complexity costs (c) the best clustering solution densely covers the data distribution. The specific choice of logarithmic complexity costs causes an almost homogeneous density of cluster centers, a phenomenon that is explained by the vanishing average complexity costs (C,) = -pa logp, of very sparsely occupied clusters (see Gish and Pierce 1968; Buhmann and Kuhnel 1992~). Analytical results of the asymptotic (K large) cluster density for different complexity measures are discussed in Buhmann and Kiihnel (1992~). Figure 2 shows a clustering configuration assuming a onedimensional topology in index space with nearest-neighbor transitions. The short links between neighboring nodes of the neural chain indicate that the distortions due to external noise have also been optimized. Note that complexity optimized clustering determines the length of the chain or, for a more general noise distribution, an optimal size of the cluster


83

Figure 2: Topology presetving clustering with C, = l/p,: a chain of 50 clusters covers the data distribution of Figure 1. The short average distances between consecutive clusters demonstrate that the total distortion error has been minimized, including the distortions induced by external noise (17 = 0.05).

set. This stopping criterion for adding new cluster nodes generalizes selforganizing feature maps (Kohonen 1984)and removes arbitrariness in the design of topological mappings. Furthermore, our algorithm is derived from an energy minimization principle in contrast to self-organizing feature maps, which “cannot be derived as a stochastic gradient on any energy function” (Erwin et al. 1992). The complexity optimized clustering scheme has been tested on the real-world task of image compression (Buhmann and Kuhnel 1992b). Entropy optimized vector quantization of wavelet decomposed images has reduced the reconstruction error of the compressed images up to 30%. Furthermore, the tendency of entropy optimized clustering to represent outlier regions with low data density yields psychophysicallymore pleasing image reconstructions since rare image features like edges are more faithfully encoded than by K-means clustering (Buhmann and Kuhnel 1992~). Figure 3 summarizes our experiments with supervised clustering controlled by the supervision costs Ci. To test the performance of supervised clustering compared to an unsupervised scheme as K-means clustering

84


Number of Clusters h’

Figure 3: Comparison of K-means clustering results (4 with supervised clustering ( 0 ) of a data set generated by a homogeneous two-dimensional data distribution with a disk shape ( ( ( x i / (_< 1). The data set (800data points) is divided into two classes of eight equally sized segments per class which alternate. Supervised clustering outperforms K-means clustering significantly for K 216 as indicated by the decrease of the average “conditional class entropy” (So).

we generated a disk-shaped data set with homogeneous density inside the disk llxill 5 1 and zero density outside. The data set was divided into two classes of eight segments per class with segments of different classes adjacent to each other. This artificial data set of two strongly intermingled classes allows us to benchmark the supervised clustering scheme with K-means clustering. Figure 3 shows that supervised clustering outperforms K-means clustering significantly for K 216 clusters as indicated by the decrease of the average “conditional class entropy” (So).

4 Online Clustering -

The described clustering procedures are butch algorithms since they require all data points {xi} to be available at the same time, a very restrictive constraint for applications dealing with a large data volume. Online clustering algorithms process a data stream sequentially by updating y, and pa iteratively. Such an algorithm is implemented by the first two layers of a competitive neural network with an architecture depicted in Figure 4. The third layer composed of classification units assigns data points to classes provided that class labels are available for the training data. The third layer is absent in unsupervised clustering schemes. The synaptic vectors y, define localized receptive fields of the clustering units


85

a, which can be interpreted as radial basis function units. Learning after N - 1 data points have been processed results in the incremental changes

(4.1) (4.2)

The upper indices (N) and (N - 1) denote the estimates of pa, yo after XN and XN-~ have been processed, respectively, To derive equations 4.14.3we have expanded the equations 3.2, 3.6, and 3.3 u p to linear terms in AporApyo,Aya,keeping ( M N , ~fixed. ) All terms proportional to derivatives of ( M N , ~vanish ) in the hard clustering limit exponentially fast IO[exp(-c/T)/q, c > 0 as T + 01.

Figure 4 A three layer competitive neural network for data clustering with d units in the input layer, K units in the clustering layer, and G units in the classification layer implements the complexity optimized clustering algorithm. Unit a in the clustering layer receives activity from the input units weighted by the synaptic vector yo. The units in the clustering layer are connected by a winner-take-all network. Due to mutual competition each data point XN is assigned to exactly one cluster a in the case of hard clustering and to a small number of clusters for fuzzy clustering. The third layer calculates the conditional class probability of a data point i being in class 7. This layer is absent for unsupervised clustering.

Joachim Buhmann and Hans Kiihnel

86

In the case of squared Euclidean distortion costs (Dia = equation 4.3 reduces to the form

I1Xi

-~

~ 1 1 ~ ) (4.4)

which corresponds to MacQueen's update rule for K-means clustering (MacQueen 1967) and which is similar to learning in competitive neural networks. yiN-') is moved toward the most recent data point XN proportionally to the mismatch (XN - yLN-'))and proportional to XN'S effective membership C, T,,(MN,,) in cluster a. In addition to that, the update formula 4.4 weights any change in yiN)by the effective number of data points NC, T,,piN), which are already assigned to cluster a. The learning rate 1/(NC, T,,piN)) treats different clusters according to their history. That generalizes conventional topological feature maps which suggest the same learning rate for all clusters (Kohonen 1984). Is the learning rate c/(NC,T,,piN)) with c = 1 the fastest rate that guarantees the correct maximum entropy estimation of the cluster centers? This question of optimally efficient stochastic approximation of the clustering solution has received considerable attention recently, especially since there exists evidence from numerical simulation that fast adaptive K-means clustering (Darken and Moody 1990) and "search then converge" clustering schedules (Darken and Moody 1992) outperform the original MacQueen's update rule in speed. According to numerical and theoretical results there exists a critical c* below which convergence in the asymptotic limit N + 00 is dramatically slowed down (Darken and Moody 1992). Unfortunately, c* is unknown for clustering and the numerically supported conjecture that the optimal learning rate c"p' = c* > 1 has to be proved. Online optimization of the number of clusters K relies on a heuristics for cluster merging and cluster creation. We have explored the following heuristics for cluster creation: A data point xi initializes a new cluster K 1 with YK+I = Xi if the clustering costs by assigning xi to an already existing cluster exceeds the complexity costs of the new cluster K + 1, that is, CK+I < min,( llxi - y,ll' + XC,). We found in a series of clustering experiments that this strategy causes a slight overestimation of the number of clusters but the resulting cluster configurations have comparable cluster costs to configurations found in batch runs.

+

5 Conclusion

Complexity optimized clustering is a maximum entropy approach to unsupervised and supervised data clustering that determines the optimal number of clusters as a compromise between distortion errors and the complexity of a cluster set. The complexity term turns out to be as important for the design of a cluster set as the distortion measure. We have de-


87

rived a batch and an online version of the proposed clustering algorithms. Complexity optimized clustering maps onto a winner-take-all network, which suggests hardware implementations in analog VLSI (Andreou et al. 1991; Lazzaro et al. 1989). Topology preserving clustering suggests a cost function based approach to limit the size of self-organizing maps. The proposed framework for data clustering unifies traditional clustering techniques like K-means clustering, entropy constraint clustering or fuzzy clustering with neural network approaches as topological vector quantizers. The network size and the cluster parameters are determined by a problem adapted complexity function, which removes considerable arbitrariness present in other nonparametric clustering methods. A related approach to clustering (Wong 1992) was recently brought to our attention. The clustering algorithm proposed by Wong is complementary to ours since it relies on cluster melting rather than cluster splitting.

Acknowledgments It is a pleasure to thank H. Sompolinsky, T. Tishby, and P.Tavan for helpful discussions. JB has been supported by the German Federal Ministry of Science and Technology and by the Air Force Office of Scientific Research (C. von der Malsburg, PI) while working at the Center for Neural Engineering of the University of Southern California. HK is a recipient of a graduate fellowship, Technical University Munich.

References Amit, D. 1989. Modelling Brain Function. Cambridge University Press, Cambridge. Andreou, A. G., Boahen, K. A., Pouliquen, P. O., Pavasovii., A,, Jenkins, R. E., and Strohbehn, K. 1991. Current mode subthreshold MOS circuits for analog VLSI neural systems. I E E E Transact. Neural Networks 2, 205-213. Bezdek, J. C. 1980. A convergence theorem for the fuzzy isodata clustering algorithms. I E E E Transact. Pattern Anal. Machine Intelligence 2(1), 1-8. Bichsel, M., and Seitz, P. 1989. Minimum class entropy: A maximum information approach to layered networks. Neural Networks 2, 133-141. Buhmann, J., and Kiihnel, H. 1992a. Unsupervised and supervised data clustering with competitive neural networks. IJCNN International Conference on Neural Networks, Baltimore, pp. N796-IV801. IEEE. Buhmann, J., and Kiihnel, H. 1992b. Complexity optimized vector quantization: A neural network approach. In Data Compression Conference, J. Storer, ed., pp. 12-21. IEEE Computer Society Press, Los Alamitos, CA. Buhmann, J., and Kiihnel, H. 1992c. Vector quantization with complexity costs. I E E E Transact. I n f o r m . Theory, in press. Chou, P. A., Lookabaugh, T., and Gray, R. M. 1989. Entropy-constrained vector quantization. I E E E Transact. Acoust. Speech Signal Process. 37, 31-42.

88


Cover, T. M.1991. Elements of lnformation Theory. Wiley, New York. Darken, C., and Moody, J. 1990. Fast adaptive k-means clustering: Some empirical results. lnternational Joint Conference on Neural Networks, Sun Diego, pp. I1 233-11 238. IEEE. Darken, C., and Moody, J. 1992. Towards faster stochastic gradient search. In Neural Information Processing Systems 4 . Morgan Kaufmann, San Mateo, CA. Duda, R. O., and Hart, P E. 1973. Pattern Classification and Scene Analysis. Wiley, New York. Durbin, R., and Willshaw, D. 1987. An analogue approach to the travelling salesman problem using an elastic net method. Nature (London) 326, 689691. Erwin, W., Obermayer, K., and Schulten, K. 1992. Self-organizingmaps: Ordering, convergence properties, and energy functions. Biol. Cybern. 67,47-55. Farvardin, N. 1990. A study of vector quantization for noisy channels. IEEE Transact.Inform. Theory 36(4), 799-809. Gish, H., and Pierce, J. N. 1968. Asymptotically efficient quantizing. IEEE Transact.Inform. Theory IT 14,676-683. Hanson, R., Stutz, J., and Cheeseman, I? 1991. Bayesian classification theory. Tech. Rep. FIA-90-12-7-01, NASA Ames Research Center. Japes, E. T. 1957. Information theory and statistical mechanics. Phys. Rev. 106, 620-630. Kirkpatrick, S., Gelatt, C., and Vecchi, M. 1983. Optimization by simulated annealing. Science 220, 671-680. Kohonen, T.1984. Self-Organization and Associative Memory. Springer, Berlin. Lazzaro, J., Ryckebusch, R., Mahowald, M. A., and Mead, C. A. 1989. Winnertake-all networks of o(n) complexity. In Neural Information Processing Systems I, pp. 703-711. Morgan Kaufmann, San Mateo, CA. Linde, Y., Buzo, A., and Gray, R. M. 1980. An algorithm for vector quantizer design. l E E E Transact. Commun. COM 28, 84-95. Luttrell, S. P.1989. Hierarchical vector quantisation. IEE Proc. 136,405-413. MacQueen, J. 1967. Some methods for classification and analysis of multivariate observations. Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281-297. Ritter, H., Martinetz, T., and Schulten, K. 1992. Neural Computation and Selforganizing Maps. Addison-Wesley, New York. Rose, K., Gurewitz, E., and Fox, G. 1990. Statistical mechanics and phase transitions in clustering. Pkys. Rev. Lett. 65(8), 945-948. Rumelhart, D. E., and Zipser, D. 1985. Feature discovery by competitive learning. Cog. Sci. 9,75-112. Simic, P. 1990. Statistical mechanics as the underlying theory of "elastic" and "neural" optimizations. Network 1,89-103. Wong, Y. 1992. Clustering data by melting. Neural Cornp. 5(1), 89-104. Yuille, A. L. 1990. Generalized deformable models, statistical physics, and matching problems. Neural Comp. 2(1), 1-24. Received 4 December 1991; accepted 18 August 1992.

This article has been cited by: 2. Ya-Jun Zhang, Zhi-Qiang Liu. 2002. Self-splitting competitive learning: a new on-line clustering paradigm. IEEE Transactions on Neural Networks 13:2, 369-380. [CrossRef] 3. J. Vesanto, E. Alhoniemi. 2000. Clustering of the self-organizing map. IEEE Transactions on Neural Networks 11:3, 586-600. [CrossRef] 4. Isaac David Guedalia , Mickey London , Michael Werman . 1999. An On-Line Agglomerative Clustering Method for Nonstationary DataAn On-Line Agglomerative Clustering Method for Nonstationary Data. Neural Computation 11:2, 521-540. [Abstract] [PDF] [PDF Plus] 5. T. Hofmann, J.M. Buhmann. 1998. Competitive learning algorithms for robust vector quantization. IEEE Transactions on Signal Processing 46:6, 1665-1675. [CrossRef] 6. Chi-Sing Leung, Lai-Wan Chan. 1998. An error control scheme for transmission of vector quantization data over noisy channels. IEEE Transactions on Signal Processing 46:10, 2767-2780. [CrossRef] 7. T. Hofmann, J.M. Buhmann. 1997. Pairwise data clustering by deterministic annealing. IEEE Transactions on Pattern Analysis and Machine Intelligence 19:1, 1-14. [CrossRef] 8. Ibrahim H. Osman, Gilbert Laporte. 1996. Metaheuristics: A bibliography. Annals of Operations Research 63:5, 511-623. [CrossRef] 9. Steven Gold, Anand Rangarajan, Eric Mjolsness. 1996. Learning with Preknowledge: Clustering with Point and Graph Matching Distance MeasuresLearning with Preknowledge: Clustering with Point and Graph Matching Distance Measures. Neural Computation 8:4, 787-804. [Abstract] [PDF] [PDF Plus] 10. David Miller , Kenneth Rose . 1996. Hierarchical, Unsupervised Learning with Growing via Phase TransitionsHierarchical, Unsupervised Learning with Growing via Phase Transitions. Neural Computation 8:2, 425-450. [Abstract] [PDF] [PDF Plus] 11. Jianchang Mao, A.K. Jain. 1996. A self-organizing network for hyperellipsoidal clustering (HEC). IEEE Transactions on Neural Networks 7:1, 16-29. [CrossRef] 12. N. Barkai, H. Sompolinsky. 1994. Statistical mechanics of the maximum-likelihood density estimation. Physical Review E 50:3, 1766-1769. [CrossRef] 13. T L H Watkin, J -P Nadal. 1994. Journal of Physics A: Mathematical and General 27:6, 1899-1915. [CrossRef]

Communicated by John Moody and Steve Zucker

Clustering Data by Melting Yiu-fai Wong Department of Electrical Engineering, 116-81, California Institute of Technology,Pasadena, CA 91125 USA We derive a new clustering algorithm based on information theory and statistical mechanics, which is the only algorithm that incorporates scale. It also introduces a new concept into clustering: cluster independence. The cluster centers correspond to the local minima of a thermodynamic free energy, which are identified as the fixed points of a one-parameter nonlinear map. The algorithm works by melting the system to produce a tree of clusters in the scale space. Melting is also insensitive to variability in cluster densities, cluster sizes, and ellipsoidal shapes and orientations. We tested the algorithm successfully on both simulated data and a Synthetic Aperture Radar image of an agricultural site with 12 attributes for crop identification. 1 Introduction

Clustering is an important problem that can be found in many applications where a priori knowledge about the distribution of the observed data is not available (Duda and Hart 1973; Jain and Dubes 1988). Simply stated, the goal is to partition a given data set into several compact groups. Each group indicates the presence of a distinct category in the measurements. It is widely used for exploratory data analysis in diverse disciplines. The literature is therefore spread among many different fields over many years. It is almost impossible to cite each contribution individually. One of the early algorithms was invented by Lloyd (1982), which was later extended by Linde et al. (1980) for vector quantization. In pattern recognition, the ISODATA algorithm (Ball and Hall 1967) and its sequential version, the k-means clustering algorithm, have been extensively used. Other algorithms include the fuzzy techniques (Ruspini 1969; Bezdek 1981; Gath and Geva 1989; Rose et al. 1990) and the hierarchical techniques such as the agglomerative and divisive methods (see Wishart 1969). These algorithms, however, suffer from several difficulties: (a) they are highly sensitive to the initialization; (b) they perform poorly if the data contain overlapping clusters; and (c) they also suffer from the inability to handle variabilities in cluster shapes, cluster densities, and cluster Neural Computation 5,89404 (1993)


90

Yiu-fai Wong

sizes. The most urgent problem is the lack of cluster validity criteria (Ekzdek 1981). All the algorithms tend to create clusters even when no natural clusters exist in the data. In this paper, we examine a fundamental way of looking at the problem of clustering, and derive a new algorithm based on information theory and statistical mechanics. We identify clustering with heating up a thermodynamic system, giving rise to hierarchical clustering in the scale space. Melting can also account for variability in cluster densities, cluster sizes, and cluster shapes (ellipsoids). The algorithm was tested successfully on both simulated data and a SyntheticAperture Radar (SAR) image of an agricultural land with 12 attributes for crop identification (Wong et al. 1992;Wong and Posner 1992). A main contribution of this paper is that this interdisciplinary approach from information theory, thermodynamics, and nonlinear dynamics can provide a proper formulation for effective clustering and related optimization problems.

2 Scale and Cluster Independence

Intuition tells us that the number of clusters depends on the scale we look at the data. At a very coarse scale, the whole data set is a cluster, whereas at a very fine scale, every datum is itself a cluster. Scale has not been exploited by the other clustering techniques, though the idea of scale space has been around for a long time (Gabor 1946;Koenderink 1984). Wong (1992)introduced a concept called “cluster independence.” To explain it, consider the situation where several people are given the same data and the same rule about clusters. Each is told to stop once a cluster is found. If they do not communicate, it is clear that the assignments of the clusters are independent. If clusters indeed exist, the information should be present in the data itself. The notion of scale implies that the data points near the cluster centers should give more information while the data points far away should give less. This can be implemented by assigning a cost of having a data point reveal the cluster locations. To make a cluster robust, the information should be spread among the data. If we treat the contributions to the determination of a cluster from all the data points as a probability distribution, this means that this probability distribution should be chosen such that its entropy is maximized subject to a linear cost constraint Uaynes 1957). Cluster independence allows us to consider one cluster at a time. Suppose the cost function is e ( x ) = ( x - Y ) ~where x is a datum and y is a cluster center. This means that we use the squared distance as a measure of the compactness of a cluster. Let P ( x ) denote the contribution

Clustering Data by Melting

91

of datum x to y . Maximizing the entropy

-

c

P(X)logP(x)

(2.1)

X

subject to the constraint

C P ( x ) e ( x )= C

(2.2)

X

one obtains

p( x) = e-P(X-Y)’ /z

(2.3)

where Z = Cxe-fl(X-y)2.To make the connection with thermodynamics, we define the “free energy”

F=--

;

l0gZ

(2.4)

At equilibrium, it is known that a thermodynamic system settles into configurations that minimize its free energy. That is, we want a F / a y = 0, or equivalently, (2.5) the weighted mean of the data. We point out that equation 2.5 is very different from that obtained by the maximum-likehood estimate of a Gaussian mixture (Wolfe 1970; Cheeseman et al. 1988). Unlike these Bayesian approaches, our method does not assume any particular data distribution. Without loss of generality, we restrict the notation and the exposition to the case of one-dimensional data. The case of higher dimensional data was treated in Wong (1992). The good news is that the dynamics are essentially the same. Definition 1. A nominal cluster is centered at y if and only if y is u local minimum of the free energy of the thermodynamic system described above. Equation 2.5 is only a necessary condition for y to be a cluster center. The sufficient conditions will become clearer as the “melting” process is explained. The details can be found in Wong (1992). Because of that, we will use “cluster” instead of “nominal cluster.” Without worrying whether nominal clusters are real clusters, one can ask the following questions: 1. Do clusters exist? This depends on whether the equation has any solutions. 2. How many clusters are there? This depends on the number of

solutions the equation has.

Yiu-fai Wong

92

3. How do the clusters evolve? The answer is given by the trajectories of the solutions of equation 2.5 as p varies.

The above list could have been longer but it suffices to illustrate the importance of equation 2.5. Since we are concerned only with local minima, this is a great advantage over other applications of physical optimization where global optima are sought. 3 Melting and Its Dynamics

Solutionsof equation 2.5 cannot be computed analytically. However, they are identical to the fixed points of the following one-parameter map':

This connects our problem with nonlinear dynamics. Figure 1 is a plot of the map for 20 data points along a unit interval with a large p. It is clear that the free energy F acts as the Lyapunov function (Wiggins 1990) for the mapping. The difference between successive ys is -$3F/&j. Thus, the ys march down the surface of the free energy and settle down in some local minimum. The mapping 3.1 exhibits no chaotic behavior. Hence, solutions can always be computed iteratively and the convergence is exponentially fast. One can see that /3 truly captures the notion of scale. At a very large p, every datum is itself a cluster, while at a very small p, the whole data set is a cluster. The essence of the algorithm is thus as follows. Start with a huge p ( h e scale); initialize every datum as a cluster. As /3 is gradually decreased, the number of clusters decreases due to the merging of the clusters. When two clusters merge, the associated data points are merged as well. Eventually the whole data set is a cluster. Specifically, the Melting Procedure is as follows:

pmXis a number related to the dynamic range and an assumed noise in the observations;

1. Choose p-; 2. set i = 1,

=

3. let every data point be a cluster; 4. iterate according to the mapping 3.1 N times or until the clusters converge. In our simulations, N = 200;

5. record the new cluster centers; 'We could have instead established a similar equivalence with the differential equa- y)e-o(x-Y)', but the analysis is very similar and the results are the tion dyldt = same.

cx(x


93

c

I 0

0.2

0.6

0.4

0.8

1

Y

Figure 1: The map for 20 data points along the unit interval. 6. if more than two clusters that previously are distinct share the same center, the set of data associated with the new cluster is the union of those with the original clusters; 7. i = i

+ 1, pi = &1/1.05;

8. if there is more than one cluster, go to 4; else Melting is complete.

It is clear that the Melting Procedure generates a strict tree structure in the scale space, analogous to a dendrogrum. Figure 2 shows an example of the Melting Procedure for a set of onedimensional data that has two clusters. The graphs are obtained by computing the fixed points of equation 3.1 as scale increases. The horizontal axis indexes i in the Melting Procedure. We merely identify scale with i, which is plotted logarithmically because of the exponential terms in equation 2.5. The original data are plotted as *s at i = 0. The dynamics involved in the merging process can be studied using local bifurcation theory (Wiggins 1990). The necessary condition for bifurcation to occur is af/tIy = 1. That is,

Yiu-fai Wong

94

-1.5' 0

20

40

60

80

100

I

I20

Figure 2: The fixed points versus scale. The leftmost points are the data.

In Wong (1992), two types of bifurcations were identified: pitchfork and saddle-node. Their bifurcation diagrams are shown in Figure 3, which show the trajectories of the fixed points as the parameter ,f3 is varied around its critical value. In a pitchfork bifurcation, two clusters continuously merge into a cluster while in a saddle-node bifurcation, a cluster becomes unstable and is siphoned into another cluster. Such bifurcations can be seen in Figure 2. The interpretation of these two bifurcations for cluster analysis is as follows (Wong 1992): A pitchfork bifurcation indicates (1) uniformly spaced or nonclustered data, or (2) clustered data but with a high degree of symmetry at certain scales. A saddle-node bifurcation indicates the inhomogeneous spatial distribution present in the data. As one expects, in clustering data, saddle-node bifurcations will be most frequently observed. It is now clear why we choose to "melt" the system starting from a low temperature, as contrasted with annealing (Kirkpatrick et al. 1983). Annealing would fail since saddle-node bifurcation implies that we do not know how much hill-climbing is needed to reach the other local minimum. We can also find an information-theoreticbasis for condition 3.2. The rate distortion function deals with the question of the minimum number of bits needed to encode a source symbol subject to an expected distortion


a

Y

b

Y

95

stable branch unstable branch

5

1n=p

Figure 3 (a) Pitchfork bifurcation in our clustering scheme. bifurcationin OUT clustering scheme.

(b)

Saddle-node

constraint (Pierce and Posner 1980). For a Gaussian source with variance u2 and the average distortion 5 6,

(3.3) Thus, when equation 3.2 becomes an equality/ R(6) = 0 signifies that there is no need to waste bits to encode the source. The cluster should either disappear or be merged. 4 What Is a Good Cluster and

How Many Are There?

One needs a criterion to decide the good clusters among all the clusters in the scale space. We will briefly outline the ideas in Wong (1992). Recall that p ( x ) is the contribution of a data point to a cluster. Thus the quantity fractional free energy (FFE) of a nominal cluster (4.1)

Yiu-fai Wong

96

I

0

0.2

0.4

0.6

0.8

1

Figure 4 (a) Data and the computed clusters illustratingability to handle many

clusters.

is a measure how good a cluster Q is. A large FFE indicates that most of the contributions come from the data belonging to the cluster itself and vice versa. What is large or small is set by a threshold MT,which expresses a degree of confidence. Hence, by keeping track of the fixed points and their FFE values, a criterion for deciding "good clusters" was defined in Wong (1992). We need to select the real clusters among the good clusters. It is very difficult to define a universally accepted criterion because clusters really need to be interpreted in the context of the specific applications. Nonetheless, an attempt to define a scale-based criterion was carried out in Wong (19921, which has found to be applicable in the radar application (Wong and Posner 1992). , If distinct good clusters exist in the data, their FFEs should remain good over a large range of logarithmic scale in p even though the fixed points may vary their positions slightly. In addition, the FFEs of these clusters should start out with very high value, only to drop quickly when they are about to bifurcate, hence, for a good cluster, the longer its FFE remains high, the more robust it is.


'

97

O t

-0.2 0

10

20

30

40

50

60

70

80

90

0

Figure 4 (b) x-components of the trajectories of the cluster centers versus scale.

1-

0.9 0.8 -

0.7 0.6 -, 0.5

0 Using the vector notation yik

=

[t]

(2.8)

the cell equations can be transformed into the Fourier domain, resulting in CoS(k~0Tij) - sin(kw0Tij) ’ yjk (2.9) ujk = x i k -k wjj sin(kwoTij) cos(kw0Tjj) i

[

vik

=

I

1

(2.10)

(2.11)

For linear cells with yi(t) = mivi(t)

(2.12)

where mi is a gain constant, the spectral cell output is simply Yjk

= miVjk

(2.13)

Note that the transformation into the Fourier domain causes the time delays Tij to become a simple quadrature phase matrix. Similarly, the cell time constant ru becomes an amplitude scaling factor that depends on the spectral hequency component kwo. The spectral error criterion, which is the basis for the SBP algorithm, is obtained by comparing the actual output sequence yi(t) to the desired

James A. Kottas

118

output sequence yjd'(t). In the time domain, the error as a function of time is (2.14) ei(t) = yjd'(t) - yi(t) where i refers only to a network output cell here, The total error for the current output sequence over all output cells (No) is given by (2.15) In the same way, the total error over all output sequences is simply the sum of E for each sequence. Since this is a linear operation, the derivation below will be done as if there was only one desired output sequence. The results are then summed over all output sequences to obtain the complete error criterion. Using Parseval's theorem, the error in equation 2.15 can be approximated by (2.16) where Ef, and E:k are the Fourier series coefficients of the temporal error sequence defined in equation 2.14. As in the conventional backpropagation algorithm, the weights and time delays are adapted according to the gradient descent driving term, (2.17) where zi, can be either a weight w, or a time delay T, and ~a(r) is the correspondingadaptation time constant with units of numbers of training epochs. Using an approach analogous to that given for the conventional backpropagation algorithm in Rumelhart et al. (19861,expressions for Awij and AT, can be derived. For output cells, the spectral cell errors for the ith cell at the kth frequency index (k = 0,1,2, . .. ,K) can be defined as r aE 1

;1

av,F,

(2.18)

6ik = -

allowing the driving term to be written as (2.19) using the chain rule. Note that the spectral cell errors 6ik are independent of z,, the weight or time delay being adapted.

Training Periodic Sequences

119

The term that is dependent on Zij, dVik/azij VZijVik, can be derived from the spectral cell equations in 2.9 and 2.10. When zij is the weight wij, d V i k / h i j = VwijVik is given by (2.20) where

and

These coefficients incorporate the effects of both the cell filter and the interconnect time delais. When zi, is the time delay Tq, aVik/aTV = VTijVik is given by2 Alk (Tij)

8Tij

(2.23)

- -wij

AdTij)

where (2.24)

-av,c,8Tij

avik

-

- aTij -

= -Wijkwo

[

A2k(Tij)

1 [J

Alk(Tij)

-A~k(Tij) Aa(Tij)

y k

James A. Kottas

120

Given the expressions for aVik/dzij,only the spectral cell errors need to be determined in order to compute the adaptation driving term Azij. For an output cell, (2.27)

in general. If the output cell is linear so yi(t) = mivi(t), the spectral cell error simplifies to

6ik = miEjk

(2.28)

where E i k is the set of Fourier series components of the error sequence defined in equation 2.14. However, if the output cell has a nonlinear output function Si, the spectral components of vi(t) are spread across the frequency spectrum. This effect is captured by the term,

(2.29)

The components of this matrix can be calculated using (2.30) (2.31)

(2.32)

(2.33)

where (2.34)

and Si(v) = dSi(v)/dv. Clearly, the nonlinear case involves much more computation than when the cells are linear.


121

For a hidden cell (one whose output is not an output of the network), the spectral cell errors can be computed using the recursive backpropagation relationship, T

(2.35)

When the hidden ce is linear, the backpropagation expression can be simplified to (2.36)

Using these expressions, the adaptation driving terms Awij and ATq can be computed for each adaptable interconnect. 3 The Training Process

A training epoch consists of the set of desired training sequences, each repeated n, times, in succession. An example epoch for two training cycles is shown in Figure 1. At the beginning of an epoch, the error gradients Azij(t) are initialized to 0. The first training sequence, having period TI (so TO= TI),is presented for ne - 1 cycles to allow transients to decay away. During the neth cycle, the output error El for the first sequence is computed along with the spectral Coefficients Yik, dYik, and Yik. At the end of this cycle, the gradient information Azij(t) is computed and saved. Next, the second training sequence with period 'T (so TOnow is set to Tz) is applied to the network. After n, - 1 transient cycles, the output error E2 and a new set of spectral coefficients are measured during the n,th cycle. The gradient information for this sequence is calculated and added to the previous Azij(t)from the first training sequence. If there was a third training sequence, the above procedure used for the second sequence would be applied again using this third sequence. After all training sequences have been cycled f i e times and the resulting gradient terms Azij(t ) accumulated, the weights and/or time delays are updated and the total training error is found by summing the individual output errors, Ci Ei. Alternatively, the weights and/or time delays can be adapted immediately after each training sequence. In this case, AzV(t)is not integrated over all sequences but used after each sequence. While this method is not a true gradient descent, it usually converges faster in practice than when the interconnect updates are done only at the end of the epoch.

James A. Kottas

122

One Training Epoch

Figure 1: An example epoch with two training cycles. Note that each cycle is repeated n, = 6 times to allow transients to decay. 4 Training Considerations 4.1 Time Delay Wrap-Around. If there is only one training sequence with period TI,or if all the training sequences have the same period (TI= T2 = ... = TN),Ti, can wrap around the period if the gradient information in ATi,(t) tries to make Tij negative. Then, if Tu does become negative, it can be changed to Tij TO(where TO= TI).

+

4.2 Zero Weights. During the training process, it is quite possible that a weight may have to change sign. However, such a transformation can cause the weight to become zero and stay there for the remainder of the adaptation. This condition can introduce an undesirable local minimum into the error surface by effectively reducing the number of degrees of freedom in the network. The interconnects at risk are those associated with the outputs of the hidden cells. From equations 2.35 and 2.36, the hidden cell errors 6, are proportional to wi,. As wi, + 0, 6, also approaches 0, causing the driving term Awii(t) to become very small. The solution wi, = 0 is the


123

trivial solution to the adaptation, since 6, =. Aw,(t) = 0 in this case. A further consequence is that the backpropagation process effectively stops because the cell error is zero. The solution to the problem is to monitor w, and Aw,. When wij is within toof 0 in magnitude and the sign of Awij is such that w, will be adapted toward 0, it is necessary to force wq to "jump" over 0. Assuming to> 0, this can be done using the following algorithm: If 0 5 W, 5 €0 and Awij < 0, set W , = - E O . If

-60

5 Wij 5 0 and AWV> 0, set W ,

= €0.

4.3 Finite Cell Bandwidth. The overall effect of having cells with finite bandwidth is to limit the spectral content of the input/output training sequences. Nonlinear networks have a mechanism for counteracting the attenuation that is unavailable in linear networks. Nonlinear sigmoidal output functions Si can mix the spectral energy in different frequencies via the driving term '7VitYik2 given in equations 2.29-2.33. For frequencies much greater than the cutoff frequency fc = 1/(27rrv), the attenuation is too great for this coupling mechanism to overcome. However, for frequencies around fc, a nonlinear network can learn to compensate for the attenuation via the SBP training algorithm. Since there is no corresponding mechanism in a linear network, it cannot be expected to learn a set of input/output sequences that contain frequencies around or above fc. For reliable learning to occur, the bandwidth of the output sequences must be less than fc, preferably by at least half a decade M 3.16). Another effect of a positive rv is that a phase delay is introduced between the input and output signals of a cell. This delay can be thought of as a form of time delay for propagating signals through the cell. Its influence.in the training process is maintained by the coefficients Alk(t) and AZ(t) defined in equations 2.21 and 2.22, respectively. The amount of the effective delay cannot be controlled because it is determined by rv and the frequency content of the input sequence. However, the interconnect time delays can be used to compensate accordingly. Multiple interconnect time delays must be adapted, though, because the amount of compensation available from any single interconnect is limited. The limit is imposed not by the interconnects but by the spectral content of the input signal-the propagation delay varies between frequencies of the input signal. This situation reveals another benefit of having adaptable time delays in the interconnects.

5 Discrete-Time Considerations

In the continuous-time formulation, the time delays are continuous. However, in a discrete-time simulation of a network, the delays must become

James A. Kottas

124

discrete in some form because of storage limitations. To approximate continuous delays, the function value at a nonintegral delay can be interpolated from neighboring function values at integral delays. For example, suppose Tv = 5.3 time steps and y,[t - Tij] is needed (where t is the current time step). Using linear interpolation,

yj[t - 5.31 x (0.7)yj[t - 51 + (0.3)yj[t - 61

(5.1)

For a general linear interpolation function, Tq can be broken down into an integral part and a fractional part,

+

Tij = LTvJ 6Tij

(5.2)

where LTij) is the largest integer less than or equal to Tij (i.e., the truncation funchon) and 6Tjj is the fractional offset (0 5 6Tij < 1). The form for the general linear interpolator is

The discrete-time equations for the continuous-time cells governed by equations 2.1-2.3 are (5.4) (5.5) (5.6) where a, = ed1Iruand y,[t - Tij] is computed using the above approximation. Note that if all the time delays are 0, each interconnect effectively has a minimum delay of 1 time step. In the remaining discussions, the time delays Tij will refer to the amount of additional time delay in the interconnects. The change in yi(t) over time can be approximated using the backward difference formula,

@ !!

M Ayi[t] = yi[t] -yi[t - 11 (5.7) dt The corresponding Fourier series coefficients for yi[t] are approximated by

(5.9) in which there is an implied At = 1 in the summation. Note that the approximations in these formulas go beyond the integral-to-sum conversion. Depending on TOand K, the sine/cosine basis set of functions


125

may not be approximately orthogonal in discrete time because of too few samples in the waveforms. The highest sampling frequency for the sinusoids, KIT0 where TOis the duration of the shortest training sequence, should be larger than the Nyquist rate, 2/To. A factor of 5 is sufficient in practice. The changes in the weights and time delays, governed by equation 2.17, are implemented in discrete time using ~ij[t] = Z q [ t - 11

where

+ (1 - ~$')A~ij[t]

(5.10)

= e-1/7?).

6 Choosing Parameters

There are two parameters that must be chosen a priori: K, the maximum spectral frequency, and n,, the number of cycles per sequence in an epoch. The criteria for choosing K are varied and depend upon the available computation power as well as the duration of the shortest training sequence. The argument for making K large is to capture the details of the shapes for all the training sequences. However, larger values for K require more computation, especially if nonlinear cells are used so the VV*Yik, terms in equations 2.30-2.33 must be calculated. Furthermore, when training with short sequences, K cannot be too large or else the sine/cosine set of basis functions in the Fourier series decomposition ceases to be orthogonal. The net effect is akin to aliasing whereby the spectral coefficients become inaccurate, usually too big. A reasonable guideline is to have at least 10 samples per period of the sine and cosine functions with the highest harmonic. Therefore, if TOis the duration of the shortest training sequence, K should be at most T0/10. The criteria for choosing n, is based on making sure the network is at a steady state (limit cycle or fixed point) in order for the Fourier analysis to be valid. Therefore, the number of transient cycles per sequence in an epoch, n, - 1, must be large enough for all transients to have decayed away. However, as n, increases, the training run time will lengthen because of the increased computational load. The best way to choose n, is with a dynamic algorithm that monitors the network outputs and signals when a steady state has been reached. However, this algorithm could become rather complicated when trying to detect limit cycles. For a fixed value of n, and an arbitrary network topology, the best way to determine n, is empirically. Using reasonable guesses for the weights and time delays, the network can be simulated in the normal runtime mode and its transient response measured. Then, n, can be set to the smallest number of periods that will provide this amount of transient time. In general, it is a good idea to increase this value for n, by 1 or 2 for a safety margin.

James A. Kottas

126

However, the training process still can fail due to insufficient transient time because the exact path of the evolution in weight and time delay space is not known. For example, when training recurrent linear networks, the poles could temporarily migrate close to the imaginary axis, resulting in longer transient times. If n, is not large enough, the adaptation could fail or converge to a false "solution" that incorporates the long transient effects. 7 Sample Simulation Results

The simple recurrent network shown in Figure 2 was used to test the SBP algorithm on an infinite impulse response (IIR) network. Both the weights wI2and wzl and the time delays T12 and 7'21 were allowed to be adaptable. The initial weights were set to 0.5 (with no random perturbations) and the time delays to 0. The input interconnect was fixed with wl0 = 1 and TIO = 0. A single input/output pair of sequences was used for training. The input sequence was the one-dimensional limit cycle defined by x1it] = sin

(T)

(7.1)

where TO = 200 time steps. The desired output sequence was generated by the network by setting rf,= 49.498 time steps (so a, = 0.98), w10 = WZI = 1, w12 = -1, TIO = 0, and T21 = 7'12 = 30 time steps. Both cell output functions were linear with a gain mi = 1. Although using the network to generate the desired output sequence is not indicative of a practical problem, it does ensure that the network has a chance of learning the input/output sequence relationship.

~

Figure 2: A simple linear filter/oscillator with an infinite impulse response.


127

A total of five spectral components were computed, so K = 4. Strictly speaking, K could have been 1 since the input sequence contains only one frequency component. The number of cycles per epoch was set to n, = 2, allowing for one transient cycle. To speed convergence, the SuperSAB adaptive gain algorithm (Tollenaere 1990) was employed whereby the common learning gain q in equation 2.17 becomes the set of time-varying gains $)[t]. In this algorithm, the gain for each weight or time delay is increased as long as the error gradient for the particular weight or time delay does not change sign. If it does, the gain is penalized and the gradient driving term is temporarily reset to 0. The resulting evolution of the training error E during adaptation is shown in Figure 3a. The corresponding evolutions of the weights and time delays are shown in Figure 3b and c. The initial and final limit cycles are illustrated in Figure 4. These plots show the successful a d a p tation performed by the SBP algorithm. The oscillations apparent in the evolutions of the total training error and the weights are induced by the gradient resets made by the SuperSAB algorithm when the gradient AT,, [f] changes sign at about t = 270 epochs. Since only one training sequence was used here, the time delays were allowed to wrap around 0 to TO.The delay T12 takes advantage of this ability as illustrated in Figure 3c. If the wrap-around is disabled, the adaptation settles into an unsatisfactory local minimum, thus preventing the training from completing successfully. Other simulations showed that various combinations of the weights and time delays can be adapted. However, if the input interconnect (wl0,Tlo) is allowed to vary along with both sets of weights and time delays, the adaptation path is such that the network develops an eigenvalue that is very close to 1 in magnitude. The resulting long time constant prevents the network from reaching a steady state within the allotted transient time. Coincidentally, the adaptation converges to a solution, but this solution is not the desired one because it cancels out the long transient response. For the correct solution to be obtained, ne would have to be increased significantly (two orders of magnitude in this particular case). 8 Discussion

The same network in Figure 2 was trained on two input/output limit cycles with different fundamental periods. As before, the network was used to generate the output cycles given two input cycles. The SBP successfully trained the network to learn the new set of weights and time delays. In this case, the time delays could not wrap around TOsince TOwas not the same for both cycles. Several other cases have been tested with the SBP algorithm (Kottas 1991). It has been used to train a nonperiodic impulse response h[t]

128

JamesA. Kottas

Figure 3 The evolution of the weights and time delays for the network shown in Figure 2. (a) The total training error E . (b)The weights. (c) The time delays. The negative delays indicate wraparound has occurred. The actual delay is given by T12 + TOwhen T12 < 0. TO= 200 time steps here.


129

Figure 4 (a) The initial (before training) and desired (final)output limit cycles. (b) Snapshots of 8 output cycles taken during the training process with 100 epochs between cycles. into a conventional finite-impulse-response (FIR) filter network. Sharp transitions in h [ f ]can only be approximated because of the finite number of Fourier components used in the training. The SBP algorithm also can train the time delays in a FIR filter network. If all weights are held constant at 1, the resulting network realizes a phase-only FIR filter. If the time delays are permitted to wrap around To in this case, the SBP algorithm will not necessarily find the optimal solution, that is, one with a minimum of time delay. However, the time delays can be modified easily by subtracting off unnecessary delays such that the relative delays between the FIR taps are preserved. Finally, the SBP algorithm can train vector patterns and not just sequences into a network. For example, it successfully trained a feedforward network with one hidden layer to learn the exclusive-OR operation between two binary inputs. In this case, only the weights were adaptable and the time delays were set to 0. Furthermore, only the bias component (K = 0) was used for training. The number of cycles per sequence (a binary vector here) per epoch was set to n, = 3 to allow the discrete time simulation to completely propagate an input pattern through the network and produce the corresponding output pattern. In all these cases, the SBP algorithm requires that a steady state solution exists for each input. If the network is inherently unstable such as with a recurrent linear network with eigenvalues greater than one in magnitude, the SBP will not work and thus will not be able to restore stability to the network.

130

James A. Kottas

Adaptable gains (@[ t]) can significantly decrease the convergence time, especially when the weights and time delays select a point near a quadratic well in the error surface. However, the gradient-reset step in the SuperSAB algorithm often induces oscillations in the adaptation. A simple solution is to ignore this part of the SuperSAB algorithm and let the SBP algorithm continue from the current point (as opposed to the previous point) on the error surface. Compared with the conventional recurrent backpropagation algorithm, the SBP algorithm requires more computational resources. However, the added computational complexity is small compared with the ease in generating networks for solving certain problems. Consider the task of recognizing limit cycles. Using the SBP algorithm, it is possible to train very quickly two linear networks, both with no hidden cells, to recognize arbitrarily shaped limit cycles in less than three periods with a high degree of phase sensitivity. This application is discussed more fully in the context of a finite state machine based on limit cycles in Kottas (1991). 9 Conclusions

The spectral backpropagation (SBP) algorithm can train the weights and time delays in a network given a set of periodic input/output sequences. Its adaptation criterion is based on minimizing the spectral energy in a Fourier series decomposition of the output error sequences. A useful property of the SBP algorithm is that it allows the cells in the network to have finite bandwidth. The SBP algorithm has been demonstrated successfully with continuous-time limit cycles and recurrent networks (varying combinations of the weights and time delays), nonperiodic discrete-time sequences and FIR networks (varying the weights and time delays independently), and static vector patterns in feedforward networks (weights only). Since it requires steady-state behavior to exist in the network, the SBP algorithm cannot train inherently unstable networks. The main advantage of having trainable time delays is that it allows the phase characteristics of the network to be controlled more easily than with the conventional recurrent backpropagation algorithm. This feature is desirable when designing networks to process periodic signals. Acknowledgments This work was sponsored in part by DARF'A through Grant AFOSR-860301. I would like to thank Vernon Shrauger, Thomas McNamara, and the reviewer for their useful comments on the manuscript. I also would like to acknowledge several helpful discussions with Cardinal Warde and Michael Jordan.


131

References Jordan, M. I. 1989. Supervised learning and systems with excess degrees of freedom. In Proceedings of the 1988 Connectionist Models Summer School, G . Hinton, D. Touretzky, and T. Sejnowski, eds., pp. 62-75. Morgan Kaufmann, San Mateo, CA. Kottas, J. A. 1991. Limit cycles in neural networks for information processing. Ph.D. thesis, MIT. Pearlmutter, B. A. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1,263-269. Pineda, F. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1, 161-172. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing. Vol. 1: Foundations, D. E.Rumelhart, J. L. McClelland, and the PDP Research Group, eds., pp. 318-362. The MIT Press, Cambridge, MA. Tollenaere, T. 1990. SuperSAB Fast adaptive back propagation with good scaling properties. Neural Nehvorks 3,561-573. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. 1989.Phoneme recognition using time-delay neural networks. IEEE Trans.ASSP 37,328-339. Yao, Y.,and Freeman, W. J. 1990. Model of biological pattern recognition with spatially chaotic dynamics. Neural Networks 3, 153-170. Received 3 September 1991; accepted 5 May 1992.

This article has been cited by:

Communicated by Halbert White

Generalization and Approximation Capabilities of Multilayer Networks Yoshikane Takahashi NTT Network Information Systems Laboratories, 1-2356 Take Yokosuka-shi,Kanagawa 283-03, Japan

This paper develops a theory for constructing 3-layered networks. The theory allows one to specify a finite discrete set of training data and a network structure (minimum intermediate units, synaptic weights and biases) that generalizes and approximates any given continuous mapping between sets of contours on a plane within any given permissible error.

1 Introduction A real object world usually consists of a continuum each point of which also consists of a continuum, for example, objects such as pictures and hand-written characters. Humans perceive the whole object world continuously both inter-object and intra-object from a certain finite discrete set of perceived data. These human perception capabilities (generalization and approximation) are represented by a continuous mapping between the object worlds. Multilayer networks are expected to simulate these generalization and approximation capabilities in an effective manner. Ji et al. (1990) modified the backpropagation learning algorithm for 3layered networks to generalize continuous curves from the set of integer points. However, they do not further investigate specific construction methods of optimal training data and network structures (minimum intermediate units, synaptic weights and biases) to generate neural mappings that generalize continuous curves, which is left as an open problem (the generalization problem). It has been proven that any continuous mapping from an m-dimensional real space to an n-dimensional real space can be approximated within any given permissible distortion by the 3-layered network with enough intermediate units (Hornik et al. 1989; Funahashi 1989). However, this result assures just the network existence and provides no construction methods for optimal network structures to generate neural mappings Neural Computation 5,132-139 (1993) @ 1993 Massachusetts Institute of Technology

Capabilities of Multilayer Networks

133

that approximate the continuous mapping; this is also left as another open problem (the approximation problem). This paper gives a theoretical solution to both the generalization and the approximation problems. That is, it develops a theory to construct a set of training data and 3-layered network structure that generates a neural mapping which generalizes and approximates any given continuous mapping between sets of contours on a plane within any given permissible distortion. 2 Problem Formulation

2.1 Definitions. 2.1.1 Metric Spaces of Contours. Let x denote a contour on the plane.

where I = [O, 11. Set C ( I )= { x I x : contour}. C(I) is a metric space with the distance d: d ( f , x b ) = max I f ( t ) - xb(t)l for V f , V xb E C(I) tE1

t EI

(2.2)

Ie(t)

for V t E I. where IP(f)- Xb(t)l = Ci - $(t)I Assume that X and Y denote any compact subspaces of C(I). 2.1.2 Continuous Mappings. Define a continuous mapping f from X

onto Y f : X Y such that V ( x ) ] ( t ) = y ( t ) and f is a restriction of a mapping F : C(I) + C(I) that satisfies these double continuity conditions: (2.3) a. For each x in C(I),F ( x ) and F-'(x) are continuous on I. b. F and F-' are continuous on C(I). Examples off include smooth deformations on contours such as rotation and similarity extension. Any one continuous mapping f is fixed and discussed. 2.2.3 Three-layered Networks. Each constituent of the 3-layered network 52 is defined as follows. 1. Inputs and outputs are values of contours x in X and y in Y respec-

tively.

Yoshikane Takahashi

134

2. The input layer has two units. Outputs from unit i are represented by the set { x , ( t ) I t E I} (i = 1,2). 3. The intermediate layer has r units with r in 1 5 r 5 YM where YM is assumed to be a fixed natural number. The mapping function of unit j is defined as: r

1

(2.4) where cp is any bounded, monotone increasing and continuously differentiable function. Aij is a synaptic weight from unit i to unit j with the condition [Aij[ 5 A (A: positive constant). 0, is a bias at unit j with the condition 0 5 j 5 0 (0: positive constant) (1 5 i 5 2, 1 5 j 5 r). All these mapping functions altogether are expressed as z ( t ) = cp[Ax(t) - 191. 4. The output layer has two units. The mapping function of unit k is defined as

(2.5) where c,k is a synaptic weight from unit j to unit k with the condition lcpl 5 C (C: positive constant) (1 5 j 5 r, 1 I k 5 2).

5. Assume that YM and cp are known and fixed while T, A, 19, and c are unknown variables. r = (r, A, 6, c) is called a structure of 52. 2.2 Generalization and Approximation. Let any structure r be given. Then r generates a neural mapping [ defined as [: C(I) + C(I) such that

y(t) = [ [ ( x ) ] ( t )= cp[A x ( t ) - 291 for V x

E C(I),y E

2.2.1 Generalization. Assume any nonnegative real number given. Define XN x YN as a finite discrete subspace of X x Y

XN x YN n

=

(2.6)

C(I) E

1 0 is

{(x",y") I x" E X,y" E Y,y" =f(x"),

= 1,2,..., A!}

(2.7)

Also define A as a partitioning of I: A = {P E O = t O < t' t9"

r 1 4 = 0, I , . . . , K }

< ... < P < ... < p-' < t K = I - P = 1 / K for 9 = O , l , . . . , K - 1

(2.8)

Yoshikane Takahashi

136

3 Solving GAP

GAP is solved in three steps, each of which is described in each section. 3.1 Tkaining Data Construction. As it is assumed that X and Y are compact in C(Z) and f is continuous, the subspace XN x YN of X x Y in 2.7 is constructed as V x E X , 3n : d ( x ,x") < &/40r~Ch M,

and d [ f ( x ) , f ( x " )< ] &/5 (3.1)

As f is continuous, a positive real number 6 > 0 is found such that

d V ( P ) , F ( x b ) ]< &/loif d(x@,xb)< 6 where V P E XN,Vxb E C(Z)

(3.2)

Applying the Weierstrass plynomial approximation theorem to each continuous function x" E X ,a polynomial x p E C(Z) is constructed with M reselected if necessary so that

d(x",XP) < &/40r~CA M 0, it is more likely that the output of the machine is y = 1, and when f(x, w) < 0, it is more likely that the output is y = -1. The parameter l / p is the so-called "temperature" parameter. When p = 00, the machine is deterministic, emitting y = 1 when f(x, w) > 0 and y = -1 when f(x, w) < 0. Let us consider the case where the true machine that generates examples is specified by WO. More specifically, let p(x) be a nonsingular probability distribution of input signals x, and let x l , . .. ,xt be t randomly and independently chosen input signals subject to p(x). The true machine generates answers yl, . . . ,yt using the probability distribution p(yi I xi,wo), i = I , . . . , t. Let be t pairs of examples thus generated,

. . .,(xt,yt)}

p(-l

I Xt+l,Wt)

and is -1 otherwise. The prediction error is given by ut = 0.51yt+l-y;+]I.

This is a random variable depending on the t training examples tt and Xt+l*

Its expectation (uJgen with respect to tt and xt+l is called the generalization error, because it denotes the average error when the machine trained with t examples predicts the output of a new example. On the other hand, the training error is evaluated by the average of ui(i = 1,. . .,t), which are the errors when the machine wtpredicts the past outputs yi for the training inputs xi retrospectively, using the distribution p(yi I xi, wt), that is

This error never converges to 0 when a machine is stochastic, because even when wtconverges to the true parameter wothe machine cannot be free from stochastic errors. The prediction error can also be measured by the logarithm of the predictive probability for the new input-output pair (yt+l,xt+l),

e(t) = - logp(yt+l I

xt+l, f i t )

(3.1)

This is called the entropic loss, log loss or stochastic complexity (Rissanen 1986; Yamanishi 1991). The generalizationentropic error is its expectation over the randomly generated training examples ttand new input-output pair (xt+~,YI+I 1, (e(t))gen

= -(logp(yt+1 I xt+l,wt))

(3.2)

Since the expectation of - logp(y I x) is the conditional entropy,

the generalization entropic loss is the expectation of the conditional entropy H(Y 1 X;wt) over the estimator wt. The entropic error of the true

Shun-ichi Amari and Noboru Murata

146

machine, specified by WO,is given by the conditional entropy,

Ho = H(Y I XWO)= El- logp(y I X,WO)l

(3.3)

Similarly, the training entropic error is the average of the entropic loss over the past examples (yjl Xi) that are used to obtain wt, (3.4) Obviously, the training error is smaller than the generalization error. It is interesting to know the difference between the two errors. The following theorem gives the universal behaviors of the training and generalization entropic errors in a faithful model, that is, when there is a wo specifying the true machine. Theorem 1. Universal Convergence Theorem for Training and Generalization Errors. Theasymptotic learning curve for the entropic training error is given by

and for the entropic generalization error by

where m is the number of parameters in w. The result of l / t convergence is in good agreement with the results obtained for another model by the statistical-mechanical approach (e.g., Seung et al. 1991). It is possible to compare our result with Yamanishi (19911, where the cumulative log loss,

is used. Here wjis the maximum likelihood estimator based on the i observations ti. From (3.61, we easily have

in agreement with Yamanishi (19911, because of

cI 71

= log t

+ o(l0g t)

i=l

The proof of Theorem 1 uses the standard techniques of asymptotic statistics and is given in the Appendix.

Statistical Theory of Learning Curves

147

4 Learning Curves for Unfaithful Model

It has so far been assumed that there exists wosuch that the true distribution p(y I x) is written as

P(Y I 4

I

= P(Y %WO)

(4.1)

This implies that the model M = {p(y I x,w)} of the distribution parameterized by w is faithful. When the true distribution is not in M, that is, there exisits no wosatisfying (4.11, the model M is said to be unfaithful. We can obtain learning curves in the case of unfaithful models, in a quite similar manner as in the faithful case. Let p(y I x,w:) be the best approximation of the true distribution p(y I x) in the sense that w; minimizes the Kullback-Leibler divergence

where the expectation E is taken with respect to the true distribution

p(x)p(y 1 x). We define the following quantities: H,' = E[- logp(y I x,will G* = E[{Vl(Y I x,w,')l{vr(y I X,Wi)}T1 K* = -E[VVl(y I X, w:)]

(4.2)

(4.3) (4.4)

where V is the gradient operator, V implying the column vector

vl=

(&)

the suffix T denotes the transposition of a vector, and VVl is the Hessian matrix. In the faithful case, w: = wo,H,'= Ho,and G' = K' = G is the Fisher information matrix. However, in general, G' # K* in the unfaithful case.

Theorem 2. Convergence Theorem for Learning Curves : Lhfiithful Case. The asymptotic learning cume for the entropic training error is given by m* (e(t))@&= H,'- (4.5) 2t and for the entropic generalization error by m* (e(t))@&= H,'+ (4.6) 2t where m' = tr(K*-'G') is the trace ofK*-'G'. See the Appendix for the proof. It is easy to see that m* = m in the faithful case, because of K' = G*.The above relations can be used for selecting an adequate model (see Murata et al. 1991; Moody 1992).

148


5 Bayesian Approach

The Bayesian approach uses a prior distribution 9(w),and then calculates the posterior probability distribution Q(w I 5') based on t observations (training examples). The predictive distribution based on Ct is defined by (5.1) P(Y I x;ttt) = JP(Y I x,w)Q(w I W w One idea is to use this predictive distribution for predicting the output. Another idea is to choose one candidate parameter w; from the posterior distribution Q(w I &) and to use p(y I x; w;) for predicting the output. The former one is called the Bayes algorithm and the latter is called the Gibbs algorithm (Opper and Haussler 1991). The entropic generalization loss is evaluated by the expectation of - log p(y I x; &) for a new example (y, x) in the Bayes algorithm case and the expectation of - log p(y I x; w;) in the Gibbs algorithm case. The entropic training loss is given, correspondingly, by

We first study the case of using the predictive distribution p(y I x; 4). By putting (5.2)

the predictive distribution is written as (5.3) p(yt+1 I X'+1, 5') = Zt+l/Z' [Amari et al. (1992); see also the statistical-mechanical approach, for example, Levin et al. (1990); S u n g et al. (1991); Opper and Haussler 1991)l. Therefore, (5.4)

We can evaluate these quantities by statistical techniques (see the Appendix). Theorem 3. The learning curves for the Bayesian predictive distribution are the same as those for the maximum likelihood estimation. We can perform similar calculations in the case of the Gibbs algorithm. Theorem 4. The learning curves for the Gibbs algorithm is for the training error (e(f1)tIain = Ho and for the generalization error

(5.5)

(5.6)


149

Conclusions We have presented a statistical theory of learning curves. The characteristics of learning curves for stochastic machines can easily be analyzed by the ordinary asymptotic method of statistics. We have shown a universal l / t convergence rule for the faithful and unfaithful statistical models. The difference between the training error and the generalization error is also given in detail. These results are in terms of the entropic loss, which fits very well with the maximum likelihood estimator. The present theory is closely related with the AIC approach (Akaike 1974; Murata et al. 1991; Moody 1992) and the MDL approach (Rissanen 1986). Our statistical method cannot be applied to deterministic machines, because the statistical model is nonregular in this case, where the Fisher information diverges to infinity. However, we can prove

for the entropic loss without using the annealed approximation (Amari 1992). But this does not hold for the expected error ut. Appendix: Mathematical Proofs In order to prove Theorem 1, we use the following fundamental lemma in statistics. Lemma. The maximum likelihood estimator wt based on t observations ttis asymptotically normally distributed with mean woand covariance matrix (tG)-',

C )

W , N N wo,-G-'

(A11

where wois the true parameter and G = (gij) is the Fisher information matrix defined by

where E denotes the expectation with respect to the distribution p(x)p(y I x, wo).

When the probability distribution is of the form (Z.l), the Fisher information matrix can be calculated to be

(see Amari 1991). This shows that G diverges to 00 as the temperature tends to 0, the estimator wtbecoming more and more accurate.


150

Proof of Theorem 1. In order to calculate ( W g e n = -E[logp(y

I x,Wt)l

we expand

I(y I x, W:)= logp(y I x, Wt) at WO,giving

I(y I x,Wt) = qy I x,wo) + Vl(y I x,wo)(Wt - wo) 1 +i(Wt - wo)TVVl(y I x, wo)(W, - wo) + * * *

(A3)

where VI is the gradient with respect to w, VVI = ($l/aW&j) is the Hessian matrix, and the superscript T denotes the transposition of a column vector. By taking the expectation with respect to the new input-output pair (y,x), we have

because of the identity

-W"I(y

I x,wo)l = E[(Vl)(VOT1

Taking the expectation with respect to Wt, we have

E[Wt - WO] = O(l/t) 1 E[(Wt - WO)(W: - W O ) ~ ]= -G-' -I-O(l/$) t and hence

m E[(Wt - W O ) ~ G ( W - WO)] ~ = - + O(l/P) t


151

and substituting this in (A9),and then summing over i, we have t

C l(yi I X, i=l

fit)

=

C l(yi I xi, wo)

because the maximum likelihood estimator wtsatisfies t

Since the xis are independently generated, by the law of large numbers, we have N

-H0

N

E[VVl(y I X, WO)]= -G

I

- C VVl(yi 1 Xi, wo) t i=l

Since (wt- WO)/& is normally distributed with mean 0 and covariance matrix G-I, (Wf - WO)~G(W' - WO)

can be expressed as a s u m of squares of m independent normal random variables with mean 0 and variance 1, implying that it is subject to the X2-distribution of degree m. Therefore, we have

where xi is a random variable subject to the Xz-distribution of degree m. Since its expectation is m,

This proves Theorem 1. In order to prove Theorem 2, we use the following lemma. Lemma. The maximum likelihood estimator wtunder an unfaithful model is asymptotically normally distributed with mean w; and covariance matrix t-'K*-'GK'-',


152

We do not give the proof of the lemma, because it is too technical. Refer to Murata et al. (1991).The proof of the theorem is almost parallel to the faithful case, if we replace woby wC; and taking account that K' # G*. The Bayesian case can be proved by using the relations p(w I 6 )

t

N

q(w)tm/21G1'/2exp{--(w 2 - WJTG(w - W t ) }

However, the proof is much more complicated and we omit it. One can complete it by using the asymptotic statistical techniques. Acknowledgments The authors would like to thank Dr. K. Judd for comments on the manuscript. The present research is supported by the Japanese Ministry of Education, Science and Culture under Grant-in-Aid on Special Priority Area of Higher Order Brain Functioning. References Akaike, H. 1974. A new look at the statistical model identification. IEEE Trans. AC-19, 716-723.

Amari, S. 1967. Theory of adaptive pattern classifiers. IEEE Vans. EC-16(3), 299-307.

Amari, S. 1985. Differential-Geometrical Methods in Statistics. Springer Lecture Notes in Statistics, 28, Springer, New York. Amari, S. 1991. Dualistic geometry of the manifold of higher-order neurons. Neural Networks 4, 443445. Amari, S. 1992. Universal property of learning curves. METR92-03, Univ. of Tokyo. Amari, S., Fujita, N., and Shinomoto, S. 1992. Four types of learning curves. Neural Comp. 4(4), 605-618. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neural Comp. 1,151-160. Gyorgyi, G., and Tishby, N. 1990. Statistical theory of learning a rule. In Neural Networks and Spin Glasses, K. Thuemann and R. Koeberle, eds., pp. 3-36. World Scientific, Singapore. Haussler, D., Kearns, M., and Shapire, R. 1991. Bounds on the sample complexity and the VC dimension. Proc. 4th Ann. Workshopon Computational Learning Theory, pp. 61-73. Morgan Kaufmann, San Mateo, CA. Haussler, D., Littlestone, N., and Warmuth, K. 1988. Predicting (0,l) functions on randomly drawn points. Proc. COLT'BB, pp. 280-295. Morgan Kaufmann, San Mateo, CA.

StatisticalTheory of Learning Curves

153

Hansel, D., and Sompolinsky,H. 1990. Learning from examples in a single-layer neural network. Europhys. Lett. 11, 687-692. Heskes, T. M., and Kappen, B. 1991. Learning processes in neural networks. Phys. Rev.A 440,2718-2726. Levin, E., Tishby, N., and Solla, S. A. 1990. A statistical approach to learning and generalization in layered neural networks. Proc. IEEE 78(10), 1568-1574. Moody, J. E. 1992. The effective number of parameters: An analysis of generalization and regularization in nonlinear systems. In Advances in Neural Information Processing Systems, J . E. Moody, S. J. Hanson, and R. P. Lippmann, eds. Morgan Kaufmann, San Mateo, CA. Murata, N., Yoshizawa, S., and Amari, S. 1991. A criterion for determining the number of parameters in an artificial neural networks model. In Artificial Neural Networks, T. Kohonen, K. Makisara, 0. Simula, and J. Kangas, eds. Elsevier Science Publishers 8. V., North-Holland. Opper, M., and Haussler, D. 1991. Calculation of the learning curve of Bayes optimal classfication algorithm for learning a perceptron with noise. Proc. 4th Ann. Workshopon Computational Learning Theory. Morgan Kaufmann, San Mateo, CA. Rissanen, J. 1986. Stochastic complexity and modeling. Ann. Statist. 14, 10801100.

Rosenblatt, F. 1961. Principles of Neurodynamics. Spartan, New York. Rumelhart, D., Hinton, G. E., and Williams, R. J. 1986. Learning internal representationsby error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. Foundations. MIT Press, Cambridge, MA. Seung, S., Sompolinsky, H., and Tishby, N. 1991. Learning from examples in large neural networks. To be published. Valiant, L. G. 1984. A theory of the learnable. Comm. ACM. 27(11), 11341142. White, H. 1989. Learning in artificial neural networks: A statistical perspective. Neural Comp. 1,42!j-464. Widrow, B. 1966. A Statistical Theory of Adaptation. Pergamon Press, Oxford. Yamanishi, K. 1990. A learning criterion for stochastic rules. Proc. 3rd Ann. Workshop on Computational Learning Theory, pp. 67-81. Morgan-Kaufmann, San Mateo, CA. Yamanishi, K. 1991. A loss bound model for on-line stochastic prediction strategies. Proc. 4th Ann. Workshop on Computational Learning Theory. MorganKaufmann, San Mateo, CA. Received 14 November 1991; accepted 18 August 1992.

This article has been cited by: 2. Yu Nishiyama, Sumio Watanabe. 2007. Stochastic complexity of complete bipartite graph-type Boltzmann machines in mean field approximation. Electronics and Communications in Japan (Part III: Fundamental Electronic Science) 90:9, 1-9. [CrossRef] 3. Shun-ichi Amari , Hyeyoung Park , Tomoko Ozeki . 2006. Singularities Affect Dynamics of Learning in NeuromanifoldsSingularities Affect Dynamics of Learning in Neuromanifolds. Neural Computation 18:5, 1007-1065. [Abstract] [PDF] [PDF Plus] 4. Koichiro Nishiue, Sumio Watanabe. 2005. Effects of priors in model selection problem of learning machines with singularities. Electronics and Communications in Japan (Part II: Electronics) 88:2, 47-58. [CrossRef] 5. Kazushi Ikeda. 2004. An Asymptotic Statistical Theory of Polynomial Kernel MethodsAn Asymptotic Statistical Theory of Polynomial Kernel Methods. Neural Computation 16:8, 1705-1719. [Abstract] [PDF] [PDF Plus] 6. Koji Tsuda, Shotaro Akaho, Motoaki Kawanabe, Klaus-Robert Müller. 2004. Asymptotic Properties of the Fisher KernelAsymptotic Properties of the Fisher Kernel. Neural Computation 16:1, 115-137. [Abstract] [PDF] [PDF Plus] 7. Toshiaki Aida. 2001. Reparametrization-covariant theory for on-line learning of probability distributions. Physical Review E 64:5. . [CrossRef] 8. Sumio Watanabe . 2001. Algebraic Analysis for Nonidentifiable Learning MachinesAlgebraic Analysis for Nonidentifiable Learning Machines. Neural Computation 13:4, 899-933. [Abstract] [PDF] [PDF Plus] 9. Didier Herschkowitz, Manfred Opper. 2001. Retarded Learning: Rigorous Results from Statistical Mechanics. Physical Review Letters 86:10, 2174-2177. [CrossRef] 10. Wenxin Jiang, M.A. Tanner. 2000. On the asymptotic normality of hierarchical mixtures-of-experts for generalized linear models. IEEE Transactions on Information Theory 46:3, 1005-1013. [CrossRef] 11. Toshiaki Aida. 1999. Field Theoretical Analysis of On-Line Learning of Probability Distributions. Physical Review Letters 83:17, 3554-3557. [CrossRef] 12. Silvia Scarpetta, Magnus Rattray, David Saad. 1999. Journal of Physics A: Mathematical and General 32:22, 4047-4059. [CrossRef] 13. S. Guarnieri, F. Piazza, A. Uncini. 1999. Multilayer feedforward networks with adaptive spline activation function. IEEE Transactions on Neural Networks 10:3, 672-683. [CrossRef] 14. Terrence L. Fine , Sayandev Mukherjee . 1999. Parameter Convergence and Learning Curves for Neural NetworksParameter Convergence and Learning Curves for Neural Networks. Neural Computation 11:3, 747-769. [Abstract] [PDF] [PDF Plus]

15. Magnus Rattray, David Saad. 1999. Analysis of natural gradient descent for multilayer neural networks. Physical Review E 59:4, 4523-4532. [CrossRef] 16. Didier Herschkowitz, Jean-Pierre Nadal. 1999. Unsupervised and supervised learning: Mutual information between parameters and observations. Physical Review E 59:3, 3344-3360. [CrossRef] 17. A. Uncini, L. Vecci, P. Campolucci, F. Piazza. 1999. Complex-valued neural networks with adaptive spline activation function for digital-radio-links nonlinear equalization. IEEE Transactions on Signal Processing 47:2, 505-514. [CrossRef] 18. Magnus Rattray, David Saad, Shun-ichi Amari. 1998. Natural Gradient Descent for On-Line Learning. Physical Review Letters 81:24, 5461-5464. [CrossRef] 19. Jianfeng Feng. 1998. Journal of Physics A: Mathematical and General 31:17, 4037-4048. [CrossRef] 20. A.J. Zeevi, R. Meir, V. Maiorov. 1998. Error bounds for functional approximation and estimation using mixtures of experts. IEEE Transactions on Information Theory 44:3, 1010-1025. [CrossRef] 21. Shun-ichi Amari . 1998. Natural Gradient Works Efficiently in LearningNatural Gradient Works Efficiently in Learning. Neural Computation 10:2, 251-276. [Abstract] [PDF] [PDF Plus] 22. S. Raudys. 1997. On dimensionality, sample size, and classification error of nonparametric linear classification algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 19:6, 667-671. [CrossRef] 23. A. Atiya, Chuanyi Ji. 1997. How initial conditions affect generalization performance in large networks. IEEE Transactions on Neural Networks 8:2, 448-451. [CrossRef] 24. Sepp Hochreiter, Jürgen Schmidhuber. 1997. Flat MinimaFlat Minima. Neural Computation 9:1, 1-42. [Abstract] [PDF] [PDF Plus] 25. S. Amari, N. Murata, K.-R. Muller, M. Finke, H.H. Yang. 1997. Asymptotic statistical theory of overtraining and cross-validation. IEEE Transactions on Neural Networks 8:5, 985-996. [CrossRef] 26. Manfred Opper. 1996. On-line versus Off-line Learning from Random Examples: General Results. Physical Review Letters 77:22, 4671-4674. [CrossRef] 27. K.-R. Müller, M. Finke, N. Murata, K. Schulten, S. Amari. 1996. A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward NetworksA Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks. Neural Computation 8:5, 1085-1106. [Abstract] [PDF] [PDF Plus] 28. Manfred Opper, David Haussler. 1995. Bounds for Predictive Errors in the Statistical Mechanics of Supervised Learning. Physical Review Letters 75:20, 3772-3775. [CrossRef] 29. Florence d'Alché-Buc, Jean-Pierre Nadal. 1995. Asymptotic performances of a constructive algorithm. Neural Processing Letters 2:2, 1-4. [CrossRef]

30. M. B Gordon, D. R Grempel. 1995. Learning with a Temperature-Dependent Algorithm. Europhysics Letters (EPL) 29:3, 257-262. [CrossRef] 31. Peter Sollich. 1994. Query construction, entropy, and generalization in neural-network models. Physical Review E 49:5, 4637-4651. [CrossRef]

Communicated by Haim Sompolinsky

Learning in the Recurrent Random Neural Network Erol Gelenbe Ecole des Hautes Etudes en Informatique, Universitt Rent Descartes (Paris V),45 rue des Saints-P$res, 75006 Paris, France The capacity to learn from examples is one of the most desirable features of neural network models. We present a learning algorithm for the recurrent random network model (Gelenbe 1989, 1990) using gradient descent of a quadratic error function. The analytical properties of the model lead to a "backpropagation" type algorithm that requires the solution of a system of n linear and n nonlinear equations each time the n-neuron network "learns" a new input-output pair. 1 Introduction

The capability to learn from examples is one of the most desirable features of neural network models. Therefore this issue has been at the center of much research in neural network theory and applications (Ackley et al. 1985; Le Cun 1985; Rumelhart et al. 1986). Learning theory in general is of major interest because of its numerous implications in machine intelligence, as well as its ability to provide a better understanding of the relationship between natural and artificial intelligence. In the area of artificial neural networks, learning has been extensively studied in the context of feedfonvard networks, primarily on the basis of the backpropagation algorithm (Rumelhart et al. 1986). Designing effective learning algorithms for general (i.e., recurrent) networks is a current and legitimate scientific concern in neural network theory. There are numerous examples where recurrent networks constitute a natural approach to problems. Such examples include, in particular, image processing and pattern analysis and recognition (see, for instance, Atalay et al. 19911, where local interactions between picture elements lead to mutual interactions between neighboring neurons, which are naturally represented by recurrent networks. In such cases, it is clear that effective learning algorithms for recurrent networks can enhance the value of neural network methodology. Another area where recurrent networks are indispensable is in combinatorial optimization, and it would be interesting to explore further the relationship between the application Neural Computation 5,154-164 (1993) @ 1993 Massachusetts Institute of Technology

Learning in the Recurrent Random Neural Network

155

of neural networks to control and optimization (Gelenbe and Batty 1992) and network learning. Several authors have considered learning algorithms for recurrent connectionist networks (Almeida 1987; Pineda 1987, 1989; Pearlmutter 1989; Behrens et al. 1991). These are based on neural network dynamics, which exhibit a fixed-point behavior. The work presented in this paper extends this approach to the random network model (Gelenbe 1989, 1990), which has the advantage of possessing well-defined fixed-point equations representing the stationary solution of the stochastic network equations. Applications of the random network model to image texture generation, associative memory, pattern recognition, and combinatorial optimization have been described elsewhere (Atalay et al. 1991; Gelenbe et al. 1991; Mokhtari 1991; Gelenbe and Batty 1992). In this paper we present a “backpropagation”type learning algorithm for the recurrent random network model (Gelenbe 1989, 1990), using gradient descent of a quadratic error function when a set of input-output pairs is presented to the network. Both the excitation and inhibition weights of the random network model must be learned by the algorithm. Thus, it requires the solution of a system of 2n linear and n nonlinear equations each time the n-neuron network ”learns” a new input-output pair. The system of nonlinear equations describes the networks fixedpoint, while the linear equations are obtained from the partial derivatives of these equations with respect to the network weights. To justify the use of the algorithm, we prove (in the Appendix) a general theorem concerning necessary and sufficient conditions for the existence of the stationary or fixed-point solution to the network. This general result completes the work presented in Gelenbe (1990) where only more restrictive sufficient conditions were given. Note that for our network existence implies uniqueness of the solution, due to the fact that the random network model is characterized by Chapman-Kolmogorov equations. Furthermore existence implies stability, since all moments of the state distribution can be explicitly computed from the model’s product-form property. 2

The Random Network Model

In the random network model (RN), n neurons exchange positive and negative impulse signals. Each neuron accumulates signals as they arrive, and fires if its total signal count at a given instant of time is positive. Firing then occurs at random according to an exponential distribution of constant rate, and signals are sent out to other neurons or to the outside of the network. Each neuron i of the network is represented at time t by its input signal potential ki(t), constituted only by positive signals that have accumulated, which have not yet been cancelled by negative signals,

Erol Gelenbe

156

and which have not yet been sent out by the neuron as it fires. Positive signals represent excitation, while negative signals represent inhibition. A negative signal reduces by 2 the potential of the neuron to which it arrives (i.e., it “cancels” an existing signal) or has no effect on the signal potential if it is already zero, while an arriving positive signal adds 2 to the neuron potential. This is a simplified representation of biophysical neural behavior (Kandel and Schwartz 1985). In the RN, signals arrive at a neuron from the outside of the network (exogenous signals) or from other neurons. Each time a neuron fires, a signal leaves it depleting its total input potential. A signal leaving neuron i heads for neuron j with probability p+(i,j) as a positive (or normal) signal, or as a negative signal with probability p-(i, j) or it departs from the network with probability d(i). p ( i , j ) = p+(i,j ) + p-(i, j) is the transition probability of a Markov chain representing the movement of signals between neurons. We have j) + d(i) = 1 for 1 5 i 5 n. External (or exogenous) inputs to each neuron i of the network are provided by stationary Poisson processes of rate A(i), and X(i). A neuron is capable of firing and emitting signals if its potential is strictly positive, and firing times are modeled by iid exponential neuron firing times with rate r(i), at neuron i. In Gelenbe (1989) it was shown that this network has a product form solution. That is, the network‘s stationary probability distribution can be written as the product of the marginal probabilities of the state of each neuron. This does not imply that the neurons have a behavior that is independent of each other. Indeed the probabilities that each neuron is excited are obtained from the coupled nonlinear signal flow equations (2) below, which yield the rate of signal arrival and hence the rate of firing of each neuron in steady state. The RN has a number of interesting features:

cip(i,

1. It represents more closely the manner in which signals are trans-

mitted in a biophysical neural network where they travel as spikes rather than as fixed analog signals. 2. It is computationally efficient.

3. It is easy to simulate, since each neuron is simply represented by a

counter; this may lead to a simple hardware implementation. 4. It represents neuron potential and therefore the level of excitation

as an integer, rather than as a binary variable, which leads to more detailed information on system state; a neuron is interpreted as being in the “firing state” if its potential is positive. Let k(t) = [k,(t), . . . ,kn(t)] be the vector of signal potentials at time t, and k = (kl,.. . ,kn) be a particular value of the vector. p(k) denotes the stationary probability distribution p(k) = limt,, Prob[k(t)= k] if it exists. Since { k ( t ) : t 2 0) is a continuous time Markov chain it satisfies the


157

usual Chapman-Kolmogorov equations; thus in steady state p(k) must satisfy the global balance equations:

+ ~ ( i ) ] l [ >k i 0 ] ]= C [ p ( k ; ) r ( i ) d ( i )+ p(kr)A(i)l[ki> 01

p(k) x [ A ( i ) + [A(i) i

i

+ p(kr)X(i)+ E { p ( k ; - ) r ( i ) p + ( i , j ) l [ k j > 01 i

+ ~ ( k $ + ) r ( i ) ~ - ( i ,+j )p(ki+)r(i)p-(i,j)l[kj ) = O]}] where the vectors used are

k; k17 klk$+

( k i , . . . , k i + l ,...,kfl) ( k l , .. . ,ki - 1,. . . , k f l ) = (kl, . . . ,ki + 1,. . . ,kj - 1,.. . ,k f l ) = (k1,. . . ,ki + 1,. . . ,kj 1,.. . ,kn) = =

+

and 1[X]is the usual characteristic function which takes the value 1 if X is true and 0 otherwise.

Theorem (Gelenbe 1989). Let

qi = X+(i)/[r(i)

+ X-(i)]

(2.1)

where the X+(i), A-(i) for i = 1,.. . ,n satisfy the following system of nonlinear simulataneous equations: X+(i) = x q j r ( j ) p + ( j ,i) i

+ A(i),

X-(i)

= x q j r ( j ) p - ( j , i)

i

+ X(i)

(2.2)

I f a unique nonnegative solution {A+(i), X-(i)} exists to equations 2.1 and 2.2 such that each qi < 1, then n

~ ( k=)

n[l-qiI&

i=l

As a consequence of this result, whenever the qi < 1 can be found, the network is stable in the sense that all moments (marginal or joint) of the neural network state can be found from the above formula, and all moments are finite. For instance, the average potential at a neuron i is simply qi/[l- q i ] . The rate (frequency) of the emission of spikes from neuron i in steady state is then 9ir(i). Furthermore because the underlying model is described by Chapman-Kolmogorov equations, whenever there is a solution, it is necessarily unique and given by the above product form formula. If for some neuron, we have X+(i) > [r(i) X-(i)], we say that the neuron is unstable or saturated. This implies that it is constantly excited in steady state: lim,,,Prob[ki(t) > 01 = 1. Its rate of spike emission

+

Em1 Gelenbe

158

is then r ( i ) : to another neuron j of the network its output appears as a constant source of positive or negative signals of rates r ( i ) p + ( i , j ) and Y(Op-(i,j). For notational convenience let us write w+(j, i ) = r ( i ) p + ( i , j ) 2 0, w-v, i) = r ( i ) p - ( i , j ) 2 0 N ( i ) = Cqjw+o,i)+ A(i), and i

D(i)

r(i)

=

+ C qjw-u, i) + x ( i ) i

Then 1 becomes qi

= N(i)/D(i)

(2.3)

and r ( i ) = Ci(w+(j,i) + w-v, i)]. 2.1 The Role of the Parameters w+U, i ) and w-v, i). The weight parameters w+(j,i) and w-u, i) have a somewhat different effect in the RN model than the weights w(/,i)in the connectionist model. In the RN model, all the w+(j,i) and w-(j,i) are nonnegative since they represent rates at which positive and negative signals are sent out from any neuron i to neuron j . Furthermore, in the RN model, for a given pair ( i , j ) it is possible that both w+(i,j) > 0 and w-(i,j) > 0; in general, it is not possible to transform an RN into an equivalent network in which certain connections are only excitatory, while others are only inhibitory, as would be the case in the usual connectionist model. Therefore, in the RN, for each pair 0, i) it will be necessary to learn both w+(i,j ) and w-(i, j ) . 3 Learning with the Recurrent Random Network Model

We now present an algorithm for choosing the set of network parameters W in order to learn a given set of K input-output pairs ( L , Y) where the set of successive inputs is denoted L = ( ~ 1 ,... ,L K } / and Lk = (Ak, A,) are pairs of positive and negative signal flow rates entering each neuron: Ak = [Ak(l), . . . ,Adn)],

xk

= [Xk(l), - .. ,Ak(n)1

The successive desired outputs are the vectors Y = {yl, . . . ,y ~ } ,where each vector yk = (ylk, . . . ,yn& whose elements yik E [O,I] correspond to the desired values of each neuron. The network approximates the set of desired output vectors in a manner that minimizes a cost function E k : n

Ek = (1/2)Cai(qi - yik)’, i=l

Ui

20


159

Without loss of generality, we treat each of the n neurons of the network as an output neuron; if we wish to remove some neuron j from network output it suffices to set Uj = 0 in the cost function, and to disconsider qj when constructing the output of the network. O u r algorithm lets the network learn both n by n weight matrices Wz = {wk+(i,j)}and W; = {w;(i, j ) } by computing for each input Lk = (Ak, Xk), a new value Wt and W; of the weight matrices, using gradient descent. Clearly, we seek only solutions for which all these weights are positive. Let us denote by the generic term w(u,v) either w(u,v) = w-(u,v), or w(u, v) = w+(u,v). The rule for weight update may be written as n

~ ( u , v=)w - I ( u , ~ ) - vCai(qik - ~ i k ) [ a q r / W ~ , v ) ] k

(3.1)

i=l

where 77 > 0 is some constant, and 1. q& is calculated using the input Lk and w(u,v) = ~ k - ~ ( u , vin) ,equation 3. 2. [aqi/bw(u,v)]kis evaluated at the values 9i wk-1

=

qik and w(u,v) =

(u,v).

To compute [aqi/aW(u,u)]kwe turn to the expression 3, from which we derive the following equation:

a q i / h ( u ,v) =

C a q j / h ( u ,v)[w+(j,i ) - w-(i, i)qil/D(i) i

l[u = i]qi/D(i) l[w(u,v) = w+(u,i)]qu/D(i)- l[w(u,u) = w-(u,i)lquqi/D(i) Let q = (ql,.. . ,qn),and define the n x n matrix .. W={[w+(i,j)- w - ( i , j ) q j ] / D ( i ) } i , 1 = 1 , ...,n We can now write the vector equations: -

+

= aq/&+(u,vw+ 7+(u,v)qu 8q/h-(u, = aq/aw- (u,v) W 7 - (u,v)qu where the elements of the n-vectors -y+(u, v ) = [-$(u, v), . . . ,$ ( u , v)], Y ( u , ~= ) [ n ( ~ , v ). ., ., - ~ ( ~are ,41 ^li+(u,v)= -l/D(i) if u = i,v # i,

aq/&+(u,v)

= =

~,:(u,v) = = =

=

+

+l/D(i) ifu#i,v=i, 0 for all other values of (u,v), -(1 + q i ) / D ( i ) if u = i,v = i, -l/D(i) i f u = i , v # i , -9i/D(i) if u # i,v = i, 0 for all other values of (u,v)

Em1 Gelenbe

160

Notice that

(3.2) where I denotes the n by n identity matrix. Hence the main computational effort in solving 3.2 is simply to obtain [I which can be done in time complexity O(n3), or O(mnz)if an m-step relaxation method is used. Since the solution of 3 is necessary for the learning algorithm, in the Appendix we derive necessary and sufficient conditions for the existence of the q j . We now have the information to specify the complete learning algorithm for the network Initiate the matrices W$ and W; in some appropriate manner. This initiation will be made at random (among nonnegative matrices) if no better information is available; in some cases it may be possible to choose these initial values by using a Hebbian learning rule. Choose a value of in 4.

v-’,

1. For each successive value of k, starting with k = 1 proceed as follows. Set the input values to Lk = (Ak, Xk).

2. Solve the system of nonlinear equations 3 with these values. 3. Solve the system of linear equations 3.2 with the results of (2). 4. Using equation 3.1 and the results of (2) and (31, update the matrices Wt and Wi.Since we seek the “best” matrices (in terms of gradient descent of the quadratic cost function) that satisfy the nonnegutivity constraint, in any step k of the algorithm, if the iteration yields a negative value of a term, we have two alternatives:

a. set the term to zero, and stop the iteration for this term in this step k; in the next step k + 1 we will iterate on this term with the same rule starting from its current null value; b. go back to the previous value of the term and iterate with a smaller value of 77. In our implementation we have used (a). Note that we may either proceed with a complete gradient descent [iterating on Steps (21, (3), and (4) until the change in the cost function or in the new values of the weights is smaller than some predetermined value], or only one iteration can be carried out for all the weights for each successive value of k (new input).


161

Clearly, one may either update the weight matrices separately for each successive value of k (i.e., successive input) as suggested, or sum the updates for all inputs at each iteration of the algorithm. 3.1 Complexity. Several authors have examined the complexity of neural network learning algorithms (Pineda 1989; Baum 1991). One viewpoint (Pineda 1989) is to consider the complexity of each network weight update, while another is to consider the complexity of learning a given family of input-output functions (Baum 1991). In the latter approach, it is known that learning even elementary boolean functions using the backpropagation algorithm is NP complete. In fact, the complexity of our learning algorithm is of the same order as that of the algorithms described in Pineda (1989). Here we merely discuss the complexity of weight update for the algorithm we have presented. Notice that the algorithm requires that for each (u,v) and for each input (successive k) we solve the nonlinear system of equations 3, and the linear system 3.2. Equations 3.2 have to be solved for each (u,v ) . [I, is obtained in time complexity O(n3),or in time complexity O(rnn2)if a relaxation method with rn iterations is adopted as suggested, for instance in Pineda (1989). The remaining computations for 3.2 are trivial. Similarly for 3, which is a nonlinear system of e uations (to be solved once for each step), the complexity will be O(mn ).

w-'

9

Appendix: Existence and Uniqueness of Network Solutions As with most neural network models, the signal flow equations 1 and 2,

which describe the manner in which each neuron receives inhibitory or excitatory signals from other neurons or from the outside world, are nonlinear. These equations are essential to the construction of the learning algorithm described above. Yet only sufficient conditions for the existence (and uniqueness) of their solution had previously been established for feedforward networks, or for so-called hyperstable networks (Gelenbe 1989, 1990). Thus in order to implement the learning algorithm it is useful to have necessary and sufficient conditions for their existence. This is precisely what we do in this appendix. Rewrite 1 and 2 as follows:

+

+ h(i), = C X+(j)p-(j, i)r(i)/[r(i)+ ~ ( i )+]X(i)

X+(i) = cX+(j)p+(j, i)r(i)/[r(i) X-(i)] i

(A.1) i where the qi have disappeared from the equations. The X+(i) and X-(i) represent the total arrival rates of positive and negative signals to neuron i.

X-(i)

Em1 Gelenbe

162

Define the following vectors: A' with elements X+(i) X- with elements X-(i)

A with elements A(i) X with elements X(i)

Let F be the diagonal matrix with elements fi = r(i)/[r(i) + X-(i)] I 1. Then A.l may be written as

+ A,

A' = X'FP'

X'(1-

X- = X'FP-

+X

FP') = A

(A.2)

+X

(A.3)

A- = X'FP-

Proposition 1. Equations A.2 and A.3 have a solution (A', A-). Proof. Since the series CEo(FP+)n is geometrically convergent (Kemeny and Snell 1960, p. 43 ff), we can write A.2 as

n=O

so that A.3 becomes (A.4)

Now define y = X- - A, and call the vector function m

where the dependence of G on y comes from F, which depends on A-. Notice that G is continuous. Therefore by Brouwer's fixed-point theorem,


163

has a fixed-point y*. This fixed point will in turn yield the solution of A.2 and A.3: m

n=O

completing the proof. 0 If this computation yields a fixed-point y* such that for any neuron i such that X+(i) 2 [r(i) X-(i)], the stationary solution for neuron i does not exist; this simply means that in steady state neuron i is constantly excited, and we set qi(y*)= 1. If on the other hand we obtain X+(i) < [r(i) X-(i)], then we set qi(y*) = A+(i)/[r(i)+ X-(i)]. Since p ( k ) is a probability distribution it must s u m to 1, which is the case if qi(y*)< 1, for all i and hence p ( k ) exists. Let us insist on the fact that p ( k ) is indeed unique, and that 9 4 ~ ’ ) < 1 for all i implies stability (in the sense of finiteness of all moments of the state).

+

+

Remark. This reduces the problem of determining existence and uniqueness of the steady-state distributions of a random network to that of computing y’ from A.5, which always exists by Proposition 1, and then of verifying the condition qi(y*)< 1, for each i = 1,.. . ,YZ. Acknowledgments The author acknowledges the support of Pale Algorithmique Rbpartie, C3 CNRS, the French National Program in Distributed Computing, and of a Grant from the Minist6re de la Recherche et de la Technologie(Paris, France).

References Ackley, D. H., Hinton, G. E., and Sejnowski, T. J. 1985. A learning algorithm for Boltzmann machines. Cog. Sci. 9, 147-169. Almeida, L. B. 1987. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. Proc. IEEE First International Conf. Neural Networks, San Diego, CA, Vol. 11, pp. 609-618. Atalay, V., Gelenbe, E., and Yalabik, N. 1991. Texture generation with the random neural network model. In Artificial Neural Networks, Vol. I, T. Kohonen, ed., pp. 111-117. North-Holland, Amsterdam. Baum, E. B. 1991. Neural net algorithms that learn in polynomial time from examples and queries. Draft Paper May 11 (private communication). Behrens, H., Gawronska, D., Hollatz, J., and Schurmann, B. 1991. Recurrent and feedforward backpropagation: Performance studies. In Artqcial Neural Networks, Vol. 11, T. Kohonen et al., eds., pp. 1511-1514. North-Holland, Amsterdam. Gelenbe, E. 1990. Stability of the random neural network model. Neural Cornp. 2(2), 239-247.

164

Erol Gelenbe

Gelenbe, E. 1989. Random neural networks with negative and positive signals and product form solution. Neural Comp. 1(4),502-510. Gelenbe, E.,and Batty, F. 1992. Minimum cost graph covering with the random network model. ORSA TC on Computer Science Conference, Williamsburg, VA, January. Pergamon Press, Oxford. Gelenbe, E., Stafilopatis, A., and Likas, A. 1991. In Artificial Neural Networks, Vol. I, T. Kohonen, ed., pp. 307-315. North-Holland, Amsterdam. Kandel, E. C., and Schwartz, J. H. 1985. Principles of Neural Science, Elsevier, Amsterdam. Kemeny, J. G., and Snell, J. L. 1965.FiniteMarkov Chins. Van Nostrand, Princeton, NJ. Le Cun, Y. 1985.A learning procedure for asymmetric threshold networks. Proc. Cognitiva 85, 599-604. Mokhtari, M. 1992. Recognition of typed images with the random network model. lnt. 1.Artificial Intelligence Pattern Recognition, in press. Pearlmutter, 8. A. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1(2), 263-269. Pineda, F. J. 1987. Generalization of backpropagation to recurrent and higher order neural networks. In Neural Information Processing Systems, D. Z. Anderson, ed., p. 602. American Institute of Physics. Pineda, F. J. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1(2), 161-172. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, eds., Vols. I and II, Bradford Books and MIT Press, Cambridge, MA. Received 3 September 1991;accepted 27 May 1992.

This article has been cited by: 2. Zhihao Guo, Shaya Sheikh, Camelia Al-Najjar, Hyun Kim, Behnam Malakooti. 2010. Mobile ad hoc network proactive routing with delay prediction using neural network. Wireless Networks 16:6, 1601-1620. [CrossRef] 3. Samar Samir Mohamed, J. M. Li, M. M. A. Salama, G. H. Freeman, H. R. Tizhoosh, A. Fenster, K. Rizkalla. 2010. An Automated Neural-Fuzzy Approach to Malignant Tumor Localization in 2D Ultrasonic Images of the Prostate. Journal of Digital Imaging . [CrossRef] 4. Jin-ting Wang, Peng Zhang. 2009. A single-server discrete-time retrial G-queue with server breakdowns and repairs. Acta Mathematicae Applicatae Sinica, English Series 25:4, 675-684. [CrossRef] 5. Erol Gelenbe, Peixiang Liu, Boleslaw K. Szymanski, Christopher Morrell. 2009. Cognitive and self-selective routing for sensor networks. Computational Management Science . [CrossRef] 6. Erol Gelenbe, Stelios Timotheou. 2008. Random Neural Networks with Synchronized InteractionsRandom Neural Networks with Synchronized Interactions. Neural Computation 20:9, 2308-2324. [Abstract] [PDF] [PDF Plus] 7. P. G. Harrison, T. Kocak, E. Gelenbe. 2008. Discussant Contributions for the Computer Journal Lecture by Erol Gelenbe. The Computer Journal 51:6, 731-734. [CrossRef] 8. E. Gelenbe, S. Timotheou. 2008. Synchronized Interactions in Spiked Neuronal Networks. The Computer Journal 51:6, 723-730. [CrossRef] 9. Erol Gelenbe. 2007. Steady-state solution of probabilistic gene regulatory networks. Physical Review E 76:3. . [CrossRef] 10. Erol Gelenbe. 2006. Introduction. Computational Management Science 3:3, 175-176. [CrossRef] 11. Pavel Bocharov, Ciro D’Apice, Alexandre Pechinkin. 2006. Product form solution for exponential G-networks with dependent service and completion of service of killed customers. Computational Management Science 3:3, 177-192. [CrossRef] 12. Alper Teke, Volkan Atalay. 2006. Texture Classification and Retrieval Using the Random Neural Network Model. Computational Management Science 3:3, 193-205. [CrossRef] 13. M. A. Karkoub. 2006. Prediction of Hydroforming Characteristics using Random Neural Networks. Journal of Intelligent Manufacturing 17:3, 321-330. [CrossRef] 14. Erol Gelenbe, Peixiang Liu, Jeremy LainLaine. 2006. Genetic Algorithms for Route Discovery. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 36:6, 1247-1254. [CrossRef] 15. Ricardo Lent. 2006. Linear QoS Goals of Additive and Concave Metrics in <emphasis emphasistype="italic">Ad Hoc Cognitive Packet Routing.

IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 36:6, 1255-1260. [CrossRef] 16. T. Kocak, J. Seeber, H. Terzioglu. 2003. Design and implementation of a random neural network routing engine. IEEE Transactions on Neural Networks 14:5, 1128-1143. [CrossRef] 17. S. Mohamed, G. Rubino. 2002. A study of real-time packet video quality using random neural networks. IEEE Transactions on Circuits and Systems for Video Technology 12:12, 1071-1083. [CrossRef] 18. E. Gelenbe, K.F. Hussain. 2002. Learning in the multiple class random neural network. IEEE Transactions on Neural Networks 13:6, 1257-1267. [CrossRef] 19. E. Gelenbe, E. Seref, Z. Xu. 2001. Simulation with learning agents. Proceedings of the IEEE 89:2, 148-157. [CrossRef] 20. C.E. Cramer, E. Gelenbe. 2000. Video quality and traffic QoS in learning-based subsampled and receiver-interpolated video sequences. IEEE Journal on Selected Areas in Communications 18:2, 150-167. [CrossRef] 21. E. Gelenbe, Zhi-Hong Mao, Yan-Da Li. 1999. Function approximation with spiked random networks. IEEE Transactions on Neural Networks 10:1, 3-9. [CrossRef] 22. E. Gelenbe, A. Ghanwani, V. Srinivasan. 1997. Improved neural heuristics for multicast routing. IEEE Journal on Selected Areas in Communications 15:2, 147-155. [CrossRef] 23. E. Gelenbe, Yutao Feng, K.R.R. Krishnan. 1996. Neural network methods for volumetric magnetic resonance imaging of the human brain. Proceedings of the IEEE 84:10, 1488-1496. [CrossRef] 24. C. Cramer, E. Gelenbe, H. Bakircloglu. 1996. Low bit-rate video compression with neural networks and temporal subsampling. Proceedings of the IEEE 84:10, 1529-1543. [CrossRef]

REVIEW

Communicated by Steven Nowlan

Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms 0. Nerrand

P. Roussel-Ragot L. Personnaz G. Dreyfus Ecole Supirieure de Physique et de Chimie Industrielles de la Ville de Paris, 10, rue Vauquelin, 75005 Paris, France

S . Marcos Laboratoire des Signaux et S y s t h e s , Ecole Supirieure d’Electricit6, Plateau de Moulon, 91192 Gif sur Yvette, France

The paper proposes a general framework that encompasses the training of neural networks and the adaptation of filters. We show that neural networks can be considered as general nonlinear filters that can be trained adaptively, that is, that can undergo continual training with a possibly infinite number of time-ordered examples. We introduce the canonical form of a neural network. This canonical form permits a unified presentation of network architectures and of gradient-based training algorithms for both feedforward networks (transversal filters) and feedback networks (recursive filters). We show that several algorithms used classically in linear adaptive filtering, and some algorithms suggested by other authors for training neural networks, are special cases in a general classification of training algorithms for feedback networks. 1 Introduction

The recent development of neural networks has made comparisons between “neural” approaches and classical ones an absolute necessity, to assess unambiguously the potential benefits of using neural nets to perform specific tasks. These comparisons can be performed either on the basis of simulations-which are necessarily limited in scope to the systems that are simulated--or on a conceptual basis-ndeavoring to put into perspective the methods and algorithms related to various approaches. The present paper belongs to the second category. It proposes a general framework that encompasses algorithms used for the training of neural networks and algorithms used for the estimation of the parameters of filters. Specifically, we show that neural networks can be used Neural Computation 5,165-199 (1993) @ 1993 Massachusetts Institute of Technology

166

0.Nerrand et al.

adaptively, that is, can undergo continual training with a possibly infinite number of time-ordered examples-in contradistinction to the traditional training of neural networks with a finite number of examples presented in an arbitrary order; therefore, neural networks can be regarded as a class of nonlinear adaptive filters, either transversal or recursive, which are quite general because of the ability of feedforward nets to approximate nonlinear functions. We further show that algorithms that can be used for the adaptive training of feedback neural networks fall into four broad classes; these classes include, as special instances, the methods that have been proposed in the recent past for training neural networks adaptively, as well as algorithms that have been in current use in linear adaptive filtering. Furthermore, this framework allows us to propose a number of new algorithms that may be used for nonlinear adaptive filtering and for nonlinear adaptive control. The first part of the paper is a short presentation of adaptive,filters and neural networks. In the second part, we define the architectures of neural networks for nonlinear filtering, either transversal or recursive; we introduce the concept of canonicalform of a network. The third part is devoted to the adaptive training of neural networks; we first consider transversal filters, whose training is relatively straightforward; we subsequently consider the training of feedback networks for nonlinear recursive adaptive filtering, which is a much richer problem; we introduce undirected, semidirected, and directed algorithms, and put them into the perspective of standard approaches in adaptive filtering (output error and equation error approaches) and adaptive control (parallel and series-parallel approaches), as well as of algorithms suggested earlier for the training of neural networks.

2 Scopes of Adaptive Filters and of Neural Networks 2.1 Adaptive Filters. Adaptive filtering is of central importance in many applications of signal processing, such as the modeling, estimation, and detection of signals. Adaptive filters also play a crucial role in system modeling and control. These applications are related to communications, radar, sonar, biomedical electronics, geophysics, etc. A general discretetime filter defines a relationship between an input time sequence {u(n),u(n - l),. . .} and an output time sequence { y ( n ) , y(n - l),. . .}, u(n) and y ( n ) being either uni- or multidimensional signals. In the following, we consider filters having one input and one output. The generalization to multidimensional signals is straightforward. There are two types of filters: (1) trunsversuljilters (termed finite impulse response or FIR filters in linear filtering) whose outputs are functions of the input signals only; and (2) recursive filters (termed infinite impulse response or IIR filters in linear filtering) whose outputs are func-

Neural Networks and Adaptive Filtering

167

tions both of the input signals and of a delayed version of the output signals. Hence, a transversal filter is defined by y(n) = @[u(n),~ ( -nl ) ,. . . , u(n - M

+ l)]

(1)

where M is the length of the finite memory of the filter, and a recursive filter is defined by y(n) = @ [ u ( n ) , u ( n - l,..., ) u(n-M+l),y(n-l),y(n-2) ,...,y(n-N)](2)

where N is the order of the filter. The ability of a filter to perform the desired task is expressed by a criterion; this criterion may be either quantitative, for example, maximizing the signal-to-noise ratio for spatial filtering (see for instance Applebaum and Chapman 1976), minimizing the bit error rate in data transmission (see for instance Proakis 19831, or qualitative, for example, listening for speech prediction (see for instance Jayant and No11 1984). In practice, the criterion is usually expressed as a weighted sum of squared differences between the output of the filter and the desired output (e.g., LS criterion). An adaptive filter is a system whose parameters are continually updated, without explicit control by the user. The interest in adaptive filters stems from two facts: (1) tailoring a filter of given architecture to perform a specific task requires a priori knowledge of the characteristics of the input signal; since this knowledge may be absent or partial, systems that can learn the characteristics of the signal are desirable; and (2) filtering nonstationary signals necessitates systems that are capable of tracking the variations of the characteristics of the signal. The bulk of adaptive filtering theory is devoted to linear adaptive filters, defined by relations (1)and (21, where @ is a linear function. Linear filters have been extensively studied, and are appropriate for many purposes in signal processing. A family of particularly efficient adaptation algorithms has been specially designed in the case of transversal linear filtering; they are referred to as the recursive least square (RLS) algorithms and their fast (FRLS) versions (Bellanger 1987; Haykin 1991). Linear adaptive filters are widely used for system and signal modeling, due to their simplicity, and due to the fact that in many cases (such as the estimation of gaussian signals) they are optimal. Despite their popularity, they remain inappropriate in many cases, especially for modeling nonlinear systems; investigations along these lines have been performed for adaptive detection (see for instance Picinbono 19881, prediction, and estimation (see for instance McCannon et al. 1982). Unfortunately, when dealing with nonlinear filters, no general adaptation algorithm is available, so that heuristic approaches are used. By contrast, general methods for training neural networks are available; furthermore, neural networks are known to be universal approximants (see for instance Hornik et al. 19891, so that they can be used to approximate any smooth nonlinear function. Since both the adaptation of filters (Haykin 1991; Widrow and Stearns 1985) and the training of

0. Nerrand et al.

168

neural networks involve gradient techniques, we propose to build on this algorithmic similarity a general framework that encompasses neural networks and filters. We do this in such a way as to clanfy how neural networks can be applied to adaptive filtering problems. 2.2 Neural Networks. The reader is assumed to be familiar with the scope and principles of the operation of neural networks; to help clarlfy the relations between neural nets and filters, the present section presents a broad classification of neural network architectures and functions, restricted to networks with supervised training.

2.2.1 Functions of Neural Networks. The functions of neural networks depend on the network architectures and on the nature of the input data: 0

0

Networkarchitectures: neural networks can have either a feedforward structure or a feedback structure; Input data: the succession of input data can be either time-ordered or arbitrarily ordered.

Feedback networks (also termed recurrent networks) have been used as associative memories, which store and retrieve either fixed points or trajectories in state space. The present paper stands in a completely different context we investigate feedback neural networks that are never left to evolve under their own dynamics, but that are continually fed with new input data. In this context, the purpose of using neural networks is not that of storing and retrieving data, but that of capturing the (possibly nonstationary) characteristics of a signal or of a system. Feedforzuard neural networks have been used basically as classifiers for patterns whose sequence of presentation is not significant and carries no information, although the ordering of components within an input vector may be significant. In contrast, the time ordering of the sequence of input data is of fundamental importance for filters: the input vectors can be, for instance, the sequence of values of a sampled signal. At time n, the network is presented with a window of the last M vaIues of the sampled signal {u(n),u(n- l),. . . , u(n - M + l)}, and, at time n + 1, the input is shifted by one time period {u(n + l),u(n),. . . ,u(n - M + 2 ) ) . In this context, feedfonoard networks are used as transversal filters, and feedback networks are used as recursive filters. A very large number of examples of feedforward networks for classification can be found in the literature. Neural network associative memories have also been very widely investigated (Hopfield 1982; Personnaz et al. 1986; Pineda 1987). Feedforward networks have been used for prediction (Lapedes and Farber 1988; Pearlmutter 1989; Weigend et al. 1990). Examples of feedback networks for filtering can be found in Robinson and Fallside (1989), Elman (1990), and Poddar and Unnikrishnan (1991).


169

Note that the above classification is not meant to be rigid. For instance, Chen et al. (1990) encode a typical filtering problem (channel equalization)into a classificationproblem. Conversely, Waibel et al. (1989) use a typical transversal filter structure as a classifier.

2.2.2 Nonadaptive and Adaptive Training. At present, in the vast majority of cases, neural networks are not used adaptively: they are first trained with a finite number of training samples, and subsequently used, for example, for classification purposes. Similarly, nonadaptive filters are first trained with a finite number of time-ordered samples, and subsequently used with fixed coefficients. In contrast, adaptive systems are trained continually while being used with an infinite number of samples. The instances of neural networks being trained adaptively are quite few (Williams and Zipser 1989a,b; Williams and Peng 1990; Narendra and Parthasarathy 1990,1991). 3 Structure of Neural Networks for Nonlinear Filtering 3.1 Model of Discrete-TimeNeuron. The behavior of a discrete-time neuron is defined by relation 3:

where

f i is the activation function of neuron i vi is the potential of neuron i zj can be either the output of neuron j or the value of a network

input j

Pi is the set of indices of the afferent neurons and network inputs to neuron i ~ i j is , ~ the weight of the synapse that transfers information from neuron or network input j to neuron i with (discrete) delay T

qij is the maximal delay between neuron j and neuron i. It should be clear that several synapses can transfer information from neuron (or network input) j to neuron if each synapse having its own delay T and its own weight C Y , ~ .Obviously, one must have Cii,o = 0 V i for causality to be preserved. If neuron i is such that i 4 Pi and qi, = 0 V j E Pi, neuron i is said to be static.

0. Nerrand et al.

170

Output at time n

State variables at

k ....I

Feedforward network 1

.....

Unit [,I.*.[iJ I 3 delays

.......

Figure 1: General canonical form of a feedback neural network.

3.2 Structure of Neural Networks for Filtering. The architecture of a network, that is, the topology of the connections and the distribution of delays, may be fully or partially imposed by the problem that must be solved: the problem defines the sequence of input signal values and of desired outputs; in addition, a priori knowledge of the problem may give hints which help designing an efficient architecture [see for instance the design of the feedforward network described in Waibel et al. (1989)l. To clarify the presentation and to make the implementation of the training algorithms easier, the canonical form of the network is especially conve nient. We first introduce the canonical form of feedback networks; the canonical form of feedforward networks will appear as a special case.

3.2.1 The Canonical Form of Feedback Networks. The dynamics of a discretetime feedback network can be described by a finite-difference equation of order N , which can be expressed by a set of N first-order difference equations involving N variables (termed state variables) in addition to the M input variables. Thus,any feedback network can be cast into a canonical

form that consists of a feedforward (static) network whose outputs are the outputs of the neurons that have desired values, and the values of the state variables, whose inputs are the inputs of the network and the values of the state variables, the latter being delayed by one time unit (Fig. 1).


171

Note that the choice of the set of state variables is not necessarily unique: therefore, a feedback network may have several canonical forms. The state of the network is the set of values of the state variables. In the following, all vectors will be denoted by uppercase letters. The behavior of a single-input-single-outputnetwork is described by the state equation 4 and output equation 4a:

s ( n + 1) = cp[S(n),W)]

(4)

where U ( n ) is the vector of the M last successive values of the external input u and S ( n ) is the vector of the N state variables (state vector). The output of the network may be a state variable. The transformation of a noncanonical feedback neural network filter to its canonical form requires the determination of M and of N . In the single-input-single-output case, the computation of the maximum number of external inputs E (M I E) is done as follows: construct the network graph whose nodes are the neurons and the input, and whose edges are the connections between neurons, weighted by the values of the delays; find the direct path of maximum weight D from input to output; one has E = D + 1. The determination of the order N of the network from the network graph is less straightforward; it is described in Appendix 1. If the task to be performed does not suggest or impose any structure for the filter, one may use either a multilayer perceptron, or the most general form of feedforward network in the canonical form, that is, a fully connected network; the number of neurons, of state variables and of delayed inputs must be found by trial and error. If we assume that the state variables are delayed values of the output, or if we assume that the state of the system can be reconstructed from values of the input and output, then all state variables have desired values. Such is the case for the NARMAX model (Chen and Billings 1989) and for the systems investigated in Narendra and Parthasarathy (1990).Figure 2 illustrates the most general form of the canonical form of a network having a single output y(n) and N state variables {y(n - l),. . .,y(n -N)}. It features M external inputs, N feedback inputs, and one output; it can implement a fairly large class of functions a; the nonrecursive part of the network (which implements function CP) is a fully connected feedforward net. More specific architectures are described in the literature, implementing various classes of functions cp and Q. Some examples of such architectures are presented in Appendix 2. 3.2.2 Special Case: The Canonical Form of Feedforward Networks. Similarly, any feedforward network with delays, with input signal u, can be cast into the form of a feedforward network of static neurons, whose inputs are the successive values u(n),u(n - l ) ,. . . ,u(n - M + 1);this puts

0. Nerrand et al.

172

. . . .....

......

Figure 2 Canonical form of a network with a fully connected feedforward net, whose state variables are delayed values of the output.

the network under the form of a transversal filter obeying relation 1:

y(n) = @[u(n),u(n - l),. . . ,u(n - M

+ l)]= @[U(n)]

The transformation of a noncanonical feedforward neural network filter to its canonical form requires the determination of the maximum value M, which is done as explained above in the case of feedback networks. An example described in Appendix 1 shows that this transformation may introduce the replication of some weights, known as “shared weights.”

4 Training Neural Networks for Adaptive Filtering 4.1 Criterion. The task to be performed by a neural network used as a filter is defined by a (possibly infinite) sequence of inputs u and of corresponding desired outputs d. At each sampling time n, an error e(n) is defined as the difference between the desired output d(n) and the actual output of the network y(n): e(n) = d(n) - y(n). For instance, in process identification, d(n) is the output of the process to be modeled; in a predictor, d(n) is the input signal at time n + 1. The training algorithms aim at finding the network coefficientsso as to satisfy a given quality criterion. For example, in the case of nonadaptive


173

training (as defined in Section 2.2.2), the most popular criterion is the least squares (LS)criterion; the cost function to be minimized is

Thus, the coefficients minimizing J ( C ) are first computed with a finite number K of samples; the network is subsequently used with these fixed coefficients. In the context of adaptive training, taking into account all the errors since the beginning of the optimization does not make sense; thus, one can implement a forgetting mechanism. In the present paper, we use a rectangular "sliding window'' of length N,;hence the following cost function: 1 I(n,C) = 2

" C p=n-Nc+l

e(pI2

The choice of the length N, of the window is task-dependent, and is related to the typical time scale of the nonstationarity of the signal to be processed. In the following, the notation ](n) will be used instead of J ( n ,C). The computation of e(p) will be discussed in Sections 4.3 and 4.4.2. 4.2 Adaptive Training Algorithms. Adaptive algorithms compute, in real time, coefficient modifications based on past information. In the present paper, we consider only gradient-based algorithms, which require the estimation of the gradient of the cost function, V J ( n )and , possibly the estimation of I(n);these computations make use of data available at time n. In the simplest and most popular formulation, a single modification of the vector of coefficients A C ( n ) = C ( n ) - C(n - 1) is computed between time n and time n + 1; such a method, usual in adaptive filtering, is termed a purely recursive algorithm. The modification of the coefficients is often performed by the steepestdescent method, whereby A C ( n ) = - p V J ( n ) . To improve upon the steepest-descent method, quasi-Newton methods can be used (Press et al. 1986), whereby A C ( n ) = + p D ( n ) , where D ( n ) is a vector obtained by a linear transformation of the gradient. Purely recursive algorithms were introduced in order to avoid timeconsuming computations between the reception of two successive samples of the input signal. If the application under investigation does not have stringent time requirements, then other possibilities can be considered. For instance, if it is desired to get closer to the minimum of the cost function, several iterations of the gradient algorithm can be performed between time n and time n + 1. In that case, the coefficient-modification

0. Nerrand et al.

174

vector AC(n) is computed iteratively as AC(n) = CK,(n) - Co(n), where K, is the number of iterations at time n, with

Ck(n) = Ck-l(n) -k pk&-l(n)

(k = 1 to Kn)

where Dk-l(n) is obtained from the coefficients computed at iteration k - 1, and Co(n) = CKnvI(n- 1). If K, > 1, the tracking capabilities of the system in the nonstationary case, or the speed of convergence to a minimum in the stationary case, may be improved with respect to the purely recursive algorithm. The applicability of this method depends specifically on the ratio of the typical time scale of the nonstationarity to the sampling period. As a final variant, it may be possible to update the coefficients with a period T > 1 if the time scale of the nonstationarity is large with respect to the sampling period:

cO(n) = CK,,-,(n - T ) Whichever algorithm is chosen, the central problem is the estimation of the gradient, VJ(n):

At present, two techniques are available for this computation: the forward computation of the gradient and the popular backpropagation of the gradient. 1. The forward computation of the gradient is based on the following

relation:

The partial derivatives of the output at time n with respect to the coefficients appearing on the right-hand side are computed recursively in the forward direction, from the partial derivatives of the inputs to the partial derivatives of the outputs of the network. 2. In contrast, backpropagation uses a chain derivation rule to com-

pute the gradient of J(n). The required partial derivatives of the cost function J(n) with respect to the potentials are computed in the backward direction, from the output to the inputs. The advantages and disadvantages of these two techniques will be discussed in Sections 4.3 and 4.4.2.


175

In the following, we show how to compute the coefficient modifications for feedforward and feedback neural networks, and we put into perspective the training algorithms developed recently for neural networks and the algorithms used classically in adaptive filtering. 4.3 Training Feedforward Neural Networks for Nonlinear 'Ransversal Adaptive Filtering. We consider purely recursive algorithms (i.e., T = 1 and K,,= 1). The extension to non-purely recursive algorithms is straightforward. As shown in Section 3.2.2, any discrete-time feedforward neural network can be cast into a canonical form in which all neurons are static. The output of such a network is computed from the M past values of the input, and the output at time n does not depend on the values of the output at previous times. Therefore, the cost function

is a sum of N, independent terms. Its gradient can be computed, from the N, + M + 1 past input data and the N, corresponding desired outputs, as a sum of N, independent terms: therefore, the modification of the coefficients, at time n, is the sum of N, elementary modifications computed from N, independent, identical elementary blocks [each of them with coefficients C ( n - l)],between time n and time n 1. We introduce the following notation, which will be used both for feedforward and for feedback networks: the blocks are numbered by m; all values computed from block m of the training network will be denoted with superscript m. For instance, y"(n) is the output value of the network computed by the mth block at time n: it is the value that the output of the filter would have taken on, at time n - N, m, if the vector of coefficients of the network at that time had been equal to C(n - 1). With this notation, the cost function taken into account for the modification of the coefficients at time n becomes

+

+

+

where em(n)= d(n - N, m ) - y"(n) is the error for block m computed at time n. As mentioned in Section 4.2, two techniques are available for computing the gradient of the cost function: the forward computation technique (used classically in adaptive filtering) and the backpropagation technique (used classically for neural networks) (Rumelhart et al. 1986). Thus, each

0. Nerrand et al.

176

block, from block m = 1 to block rn = N,, computes a partial modification AcF of the coefficients and the total modification, at time n, is NC

Ac,( n ) =

Acy (n) m=l

as illustrated in Figure 3. It was mentioned above that either the forward computation method or the backpropagation method can be used for the estimation of the gradient of the cost function. Both techniques lead to exactly the same numerical results; it has been shown (Pineda 1989) that backpropagation is less computationally expensive than forward computation. Therefore, for the training of feedforward networks operating as nonlinear transversal filters, backpropagation is the preferred technique for gradient estimation. However, as we shall see in the following, this is not always the case for the training of feedback networks. 4.4 Training Feedback Neural Networks for Nonlinear Recursive Adaptive Filtering. This section is devoted to the adaptive training of feedback networks operating as recursive filters. This problem is definitely richer, and more difficult, than the training of feedforward networks for adaptive transversal filtering. We present a wide variety of algorithms, and elucidate their relationships to adaptation algorithms used in linear adaptive filtering and to neural network training algorithms.

4.4.1 General Presentation ofthe Algorithmsfor Training Feedback Networks. Since the state variables and the output of the network at time n depend on the values of the state variables of the network at time n - 1, the computation of the gradient of the cost function requires the computation of partial derivatives from time n = 0 up to the present time n. This is clearly not practical, since (1) the amount of computation would grow without bound, and (2) in the case of nonstationary signals, taking into account the whole past history does not make sense. Therefore, the estimation of the gradient of the cost function is performed by truncating the computations to a fixed number of sampling periods N f into the past. Thus, one has to use N f computational blocks (defined below), numbered from rn = 1 to rn = N f : the outputs y"(n) are computed through Nt identical versions of the feedforward part of the canonical form of the network [each of them with coefficients C(n - 1)l. Clearly, N f must be larger than or equal to N, to compute the N, last errors P(n).Here again, we first consider the case where T = 1 and K, = 1. Figure 4 shows the mth computational block for the forward computation technique: the state input vector is denoted by Sg(n); the state output vector is denoted by Stut(n).The canonical feedforward (FF)net computes the output from the external inputs P ( n )and the state inputs

Figure 3 Computation of two successive coefficient modifications for a nonlinear transversal filter (N,= 3).

At time n

178

0. Nerrand et al.

Figure 4 Training block m at time n with a desired output value: computation of a partial coefficient modification using the forward computation of the gradient for a feedback neural network. If the output of block m has no desired value, it has no "products" part and does not contribute directlv to coefficient modifications: it just transmits the state variables and their derivatives to the next block.

Sg(n). The forward computation (FC) net computes the partial derivatives required for the coefficient modification, and the partial derivatives of the state vector which may be used by the next block. The N f blocks compute sequentially the Nt outputs {y"} and the partial derivatives {@"'/acij}, in the forward direction (rn = 1 to Nt). The N, errors { P } (computed from the outputs of the last Ncblocks) and the corresponding partial derivatives are used for the computation of the coefficient modifications, which is the sum of N, terms:

Details of the computations are to be found in Appendix 3. For the blocks to be able to perform the above computations, the values of the state inputs SE(n) and of their partial derivatives with respect to the weights must be determined. The choice of these values is of central importance; it gives rise to four families of algorithms.


179

4.4.2 Choice of the State Inputs and of Their Partial Derivatives. Choice of thestateinputs: The most "natural" choice of the state inputs of block m is to take the values of the state variables computed by block m-1: S z ( n ) = St;'(n) with Sk(n) = S,!,,,(n - 1). Thus, the trajectory of the network in state space, computed at time n, is independent of the trajectory of the process: the input of block m is not directly related to the actual values of the state variables of the process to be modeled by the network, hence the name undirected algorithm. If the coefficients are mismatched, this choice may lead to large e m r s and to instabilities. Figure 5a shows pictorially the desired trajectory of the state of the network and the trajectory which is computed at time n when an undirected algorithm is used (Nt = 3, N, = 2). We show in the next section that in that case, one must use the forward computation technique to compute the coefficient modifications (Fig. 5b). This choice of the state inputs has been known as the output error approach in adaptive filtering and as the parallel approach in automatic control. It does not require that all state variables have desired values. In order to reduce the risks of instabilities, an alternative approach may be used, called a semidirected algorithm. In this approach, the state of the network is constrained to be identical to the desired state for m = 1:

Sk(n) = [d(n- N t ) l d ( n- Nt - l)l.. . , d ( n - Nt - N + l)] and Sz(n) = S?;'(n). This is possible only when the chosen model is such that desired values are available for all state variables; this is the case for the NARMAX model. Figure 6a shows pictorially the desired trajectory of the state of the network and the trajectory that is computed at time n when a semidirected algorithm is used (Nt = 4, N, = 2). We show in the next section that in that case, one can use the backpropagation technique to compute the coefficient modifications (Fig. 6b). The trajectory of the state of the network can be further constrained by choosing the state inputs ofall blocks to be equal to their desired values:

SE(n) = [d(n - Nt + m - l ) , d ( n - Nt + m - 2), . . . , d ( n - Nt + m

-N)]

for m = 1 to Nt. With this choice, the training is under control of the desired values, hence of the process to be modeled, at each step of the computations necessary for the adaptation (hence the name directed algorithm); therefore, it can be expected that the influence of the mismatch of the model to the process is less severe than in the previous cases. Figure 7a shows pictorially the desired trajectory of the state of the network and the trajectory that is computed at time n when a directed algorithm is used (Nt = N, = 3). We show in the next section that in that case, one can use the backpropagation technique to compute the coefficient modifications (Fig. %). In directed algorithms, all blocks are independent, just as in the case of the training of feedforward networks (Section 4.3); therefore, one has Nt = N,.

180

0. Nerrand et al.

Figure 5: Undirected algorithm (with Nt = 3 and Nc = 2). (a) Pictorial representation of the desired trajectory, and of the trajectory computed at time n, in state space; the trajectory at time n is computed by the blocks shown in b. (b)Computational system at time n. The detail of each block is shown in Figure 4. Note that the output of block 1 has no desired value.

This choice of the values of the state inputs has been known as the equation error approach in adaptive filtering and as the series-parallel approach in automatic control. It is an extension of the teacher forcing technique (Jordan 1985) used for neural network training. If some state inputs do not have desired values, hybrid versions of


181

Figure 6: Semidirected algorithm (with Nr = 4 and Nc = 2). (a) Pictorial representation of the desired trajectory, and of the trajectory computed at time n, in state space; the trajectory at time n is computed by the blocks shown in b. (b) Computational system at time n. The detail of each block is shown on Figure 8. Note that the outputs of blocks 1 and 2 have no desired values, but do contribute an additive term to the coefficient modifications. the above algorithms can be used: those state inputs for which no desired values are available are taken equal to the corresponding computed state variables (as in a n undirected algorithm), whereas the other state inputs may be taken equal to their desired values (as in a directed or in a semidirected algorithm). Consistent choices of the partial derivatives of the state inputs: The choices of the state inputs lead to corresponding choices for the initialization

0. Nerrand et al.

182

b-l A d 1 (n)

Ad2(n)

Ad3(n)

(b)

Figure 7 Directed algorithm (with Nf= Nc = 3). (a) Pictorial representation of the desired trajectory, and of the trajectory computed at time n, in state space; the trajectory at time n is computed by the blocks shown in b. (b) Computational system at time n. The detail of each block is shown on Figure 8. Note that in a directed algorithm, each block is independent from the others and must have a desired output value.

of the partial derivatives, as illustrated in Figures 5a, 6a, and 7a. In the case of the undirected algorithm, one has S$(n) = SrG1(n);therefore, a consistent choice of the values of the partial derivatives of the state inputs consists in taking the values of the partial derivatives of the state outputs computed by the previous block:


183

except for the first block where one has as!m ( n ) - W"An - 1) ac, dCij In the case of the semidirected algorithm, the state input values of the first block are taken equal to the corresponding desired values; the latter do not depend on the coefficients; therefore, their partial derivatives can consistently be taken equal to zero. The values of the partial derivatives of the state inputs of the other blocks are taken equal to the values of the partial derivatives of the state outputs computed by the previous block. In the case of the directed algorithm, one can consistently take the partial derivatives of the state inputs of all blocks equal to zero. The parameters T, K,, N f , Nc being fixed, the first three algorithms described above are summarized on the first line of each section of Table 1. The first part of the acronyms refers to the choice of the state inputs and the second part refers to the choice of the partial derivatives of the state inputs. They include algorithms which have been used previously by other authors: the real-time recurrent learning algorithm (Williams and Zipser 1989a) is an undirected algorithm (using the forward computation technique) with N f = N, = 1, This algorithm is known as the recursive prediction error algorithm, or IIR-LMS algorithm, in linear adaptive filtering (Widrow and Steams 1985). The teacher-forced real-time recurrent learning algorithm (Williams and Zipser 1989a) is a hybrid algorithm with Nt = Nc = 1. The above algorithms have been introduced in the framework of the fomard computation of the gradient of the cost function. However, the estimation of the gradient of the cost function by backpropagation is attractive with respect to computation time, as mentioned in Section 4.3.4. If this technique is used, the computation is performed with N f blocks, where each coefficient cij is replicated in each block m as cr. Therefore, one has

The training block m is shown in Figure 8: after computing the N, errors using the N f blocks in the forward direction, the N f blocks compute the derivatives of ](n)with respect to the potentials { v y } , in the backward direction. The modification of the coefficients is computed from the N f blocks as

It is important to notice that backpropagation assumes implicitly that the partial derivatives of the state inputs of the first copy are taken equal to zero. Therefore, the backpropagation technique will lead to the same coefficient modifications as the forward propagation technique if and only if it is used

Desired values

Desired values

Desired values

Desired values

Desired values

Desired values

Semidhcted (SD) algorithm

SDD algorithm

SDUD algorithm

Directed algorithm (D) (equationerror) (teacher forcing) (series parallel)

DSD algorithm

DUD algorithm

D e s i i values

Desired values

Desired values

S:;'(n)

S%%)

s:;'(n)

as:,,

acij

-(n as:,,

- 1)

Zero

zero

- 1)

zero

zero

zero

-(n acij

=

- 1)

zero

acij

-as:,, (n

OC,

-(n)

Initialization: partial derivatives for the first block

'In each section, the first line describes the algorithms with consistent choices of the state inputs.

SLAn - 1) s:ilw

SZ1(n)

1)

S:u,(n -

s:;W

S:dn - 1)

SE(n) =

sk(n) =

UDSD algorithm

Undirected (UD) algorithm (output error) (parallel) U D D algorithm

State input of a current block

Initialization: state input of the first blodc

Table 1: Three Families of Algorithms for the Training of Feedback Neural Networks'

as%;' -@acij I

zero

- (acij n)

=

as:;:;' acij ( n ) zero as:;:;'

- (acij n)

as:;'

zero

as:;' -@Iacij

-as2 (n) acij

Partial derivatives for a current blodc


185

Figure 8: Training block m at time n with a desired output value: computation of a partial coefficient modification using the backpropagation technique for the estimation of the gradient for a feedback neural network. If block m has no desired value, then P = 0, but it does contribute an additive term to the coefficient modification. It should be noticed that forward propagation through all blocks must be performed before backpropagation.

within algorithms complying with this condition, that is, within directed or semidirected algorithms (Figs. 6b and 7b);backpropagationcannot be used consistently within undirected and hybrid algorithms. When both backpropagation and forward computation techniques can be used, backpropagation is the best choice because of its lower computational complexity. An example of the use of a directed algorithm for identification and control of nonlinear processes can be found in Narendra and Parthasarathy (1990). Other choices of the partial derivatives of the state inputs: Because adaptive neural networks require real-time operation, tradeoffs between consistency and computation time may be necessary: setting partial derivatives i3S;/i3cy equal to zero may save time by making the computation by backpropagation possible even for undirected algorithms (UD-Dor UD-SDalgorithms). The full variety of algorithms is shown on Table 1: in each group, the first line shows the characteristics of the fully con-

186

0. Nerrand et al.

sistent algorithm, whereas the other two lines show other possibilities which are not fully consistent, but which can nevertheless be used with advantage. The SD-UD, D-SD,and D-UD algorithms have been included for completeness: computation time permitting, the accuracy of the computation may be improved by setting the partial derivatives of the state inputs to nonzero values in the directed or semidirected case. Undirected algorithms have been in use in linear adaptive filtering: the extended L M S algorithm is a UD-D algorithm (see Table 1) with Nf = N, = 1 (Shynk 1989); the u posteriori error algorithm is also a UD-D algorithm with Nt = 2, N, = l (Shynk 1989). The truncated buckpropagution through time algorithm (Williams and Peng 1990) is a UD-D algorithm with N, = 1 and Nt > 1, with a special feature: to save computation time, the coefficients of the blocks 1 to Nf - 1 are the coefficients that were computed at the corresponding times. 5 Conclusion

The present paper provides a comprehensive framework for the adaptive training of neural networks, viewed as nonlinear filters, either transversal or recursive. We have introduced the concept of canonical form of a neural network, which provides a undying view of network architectures and allows a general description of training methods based on gradient estimation. We have shown that backpropagation is always advantageous for training feedforward networks adaptively, but that it is not necessarily the best method for training feedback networks. In the latter case, four families of training algorithms have been proposed; some of these algorithms have been in use in classical linear adaptive filtering or adaptive control, whereas others are original. The unifying concepts thus introduced are helpful in bridging the gap between neural networks and adaptive filters. Furthermore, they raise a number of challenging problems, both for basic and for applied research. From a fundamental point of view, general approaches to the convergence and stability of these algorithms are still lacking; a preliminary study along these lines has been presented (Dreyfus et ul. 1992); from the point of view of applications, the real-time operation of nonlinear adaptive systems requires specific silicon implementations, thereby raising the questions of the speed and accuracy required for the computations.

Appendix 1 We consider a discrete-time neural network with any arbitrary structure, and its associated network graph as defined in Section 3.2. The set of state variables is the minimal set of variables that must be initialized to allow the computation of the state of all neurons at any


187

time n > 0, given the values of the external inputs at all times from 0 to n. The order of the network is the number of state variables. Clearly, the only neurons whose state must be initialized are the neurons which are within loops (i.e., within cycles in the network graph). Therefore, to determine the order of the network, the network graph should be pruned by suppressing all external inputs and all edges which are not within cycles (this may result in a disconnected graph). To determine the order, it is convenient to further simplify the network graph as follows: (1) merge parallel edges into a single edge whose delay is the maximum delay of the parallel edges; (2) if two edges of a loop are separated by a neuron that belongs to this loop only, suppress the neuron and merge the edges into a single edge whose delay is the sum of the delays of the edges. We now consider the neurons which are still represented by nodes in the simplified network graph. We denote by N the order of the network. If, for each node i of the simplified graph, we denote by Ai the delay of the synapse, afferent to neuron i, which has the largest delay (i.e., the weight of the edge directed toward i, which has the largest weight), then a simple upper bound for N is given by i

The state Xi of a neuron i, which has an afferent synapse of delay Aj, cannot be computed at times n < Ai; the computation of the states of the other neurons may require the values of xi at times 0,1,. . . ,Ai - 1; thus, the contribution of neuron i to the order of the network is smaller than or equal to Aj. Let the quantity wi be defined as wi = Ai - min(Aj - 7 j . j ) jER,

if Ai - min(A, - 7j.i) > 0 jERi

wi = 0 otherwise where Ri stands for the set of indices of the nodes that are linked to i by an edge directed from i to j (i.e., the set of neurons to which neuron i projects efferent synapses). Then the order of the network is given by

N=Cwj i

The necessity of imposing the state of neuron i at time k (0 < k < Ai - 1) depends on whether this value is necessary for the computation of the state of a neuron j to which neuron i sends its state: if k 7j.i is smaller than the maximum delay A, of the synapses afferent to j, it is not necessary to transmit the state of neuron i at time k to neuron j , since the latter does not have the information required to compute its state at time k + qi; the information on the state of neuron i at time k is necessary only

+

0. Nerrand et al.

188

if one has k 2 Aj - 7j.j. Therefore, the minimum number of successive values required for neuron i is equal to

Ai - min(Aj - 7j.J FRi

if Ai - @(A, 1ERi

- 7j.i)

> 0,

zero otherwise

Clearly, this result is in accord with the upper bound given above. The above results determine the number of state variables related to each neuron. The choice of the set of state variables is not unique. The presence of parallel edges within a loop, or the presence of feedforward connections between loops, may require the replication of some neurons and of some coefficients. Figure Al.la shows a feedback network and Figure Al.lb shows its canonical form; the order of the network is 6. The example shows that some weights are replicated.

Appendix 2 This appendix describes several architectures of feedback neural networks that have been proposed in the literature. We present their canonical form, so that they can be easily compared. The discrete-time mathematical model of a time-invariant dynamical process is of the form

where vector U is the input of the dynamical system, vector S denotes the state of the system, and vector Y is the output of the system. Since neural networks with hidden neurons are able to approximate a large class of nonlinear functions, they can be used for implementing functions cp and Q. The network proposed by Jordan (1986) is trained to produce a given sequence y(n) for a given constant input P ("plan"). Thus it is used as an associative memory. The network and its canonical form are shown in Figure A2.1. The representation of the network under its canonical form shows that the network is of order 2, although the representation used by Jordan exhibits four connections with unit delays. Note that the state variables are not delayed values of the output. The presence of hidden neurons allows this network to learn any function y(n) = Q[S(n),U(n)]. The network suggested by Elman (1988) is used as a nonlinear filter. Its canonical form is shown on Figure A2.2.

Figure Al.1: (a) Example of a feedback neural network. Numbers in rectangles are synapse delay values, u is the external input, and y is the output of the network. (b) Canonical form of the network ( E = 8, M = 2, N = 6). The cij,. notation of relation 3 is wed.

0. Nerrand et al.

190

Figure A2.1: (a) Network architecture proposed by Jordan. (b)Canonical form.

4

4

Figure A2.2 Canonical form of the network architectureproposed by Elman. Each state variable is computed as a fixed nonlinear function f of a weighted sum of the external inputs and state inputs. Therefore, the class of functions cp that can be implemented is restricted to the form:

v")W)l , = f[AS(fl) + W n ) l where A and B are the synaptic matrices. SimiIarly, the output is computed as a fixed nonlinear function f of a weighted sum of the state


191

variables, so that the class of functions @ that can be implemented is restricted to

@[S(n),W)l = f[CS(n)l where C is the synaptic matrix. The network proposed in Williams and Zipser (1989a) and Williams and Peng (1990) is used as a nonlinear filter. The state of the network at time n 1 is computed as a weighted sum of the inputs and of the state values at time n, followed by a fixed nonlinearityf i . As a result, the network can only implement nonlinear functions of the form fi[AS(n)+ BU(n)l. The network used by Poddar and Unnikrishnan (1991) consists of a “feedforward network of pairs of neurons; each neuron, except the output neuron, and each external input is associated to a “memory neuron.” If xi(n) is the value of the output of neuron i and x,(n) the value of the output of the associated memory neuron j at time n, the output of the memory neuron at time n + 1 is

+

xj(n + 1) = aixi(n) + ( 1 - ai)xj(n),

0 < ai I 1

If ai = 0, the memory neurons introduce only delays, so that the network is a nonlinear transversal filter. If ai # 0, the memory neurons are linear low-pass first order filters, and the network is actually a feedback network. A state output is associated to each memory neuron. Figure A2.3a shows an example of such an architecture where neurons 3,4,7, and 8 are the memory neurons associated to the two inputs 1 and 2 and to the two neurons 5 and 6, respectively. The canonical form is shown in Figure A2.3b where x3, x4, x7, x8 are chosen as state variables. For process identification and control problems, the most general structure used by Narendra and Parthasarathy (1991) is a model of the specific form y(n) = @ I [ u ( n - l ) , u ( n - 2 ) ,

...I+

@p[y(n-l),y(n-2),

...I

where 91 and \Tr2 are implemented by MLP networks with 20 neurons in the first hidden layer and 10 neurons in the second hidden layer. Appendix 3

For simplicity, we present the training of the fully connected neural net of Figure 2 we denote the external inputs by z1 to ZM! the feedback inputs by Z M + ~ to Z M + N , and the outputs of the neurons by ZM+N+~ to ZM+N+“ (where v is the number of neurons). The neurons are ordered in the following way: the pth neuron receives the outputs of neurons indexed 4 < p (fully connected).

IE

F 4-

I


193

At time n, we have to consider the following cost function:

J(n)= I

5

(ey

m=N,-N,+I

where Nt is the number of blocks used to compute the N, values e"' (Nt 2 Nc). In this appendix, we present the contribution of block m (1 5 m 5 Nf) to the gradient estimation. This contribution is computed from the external input vector, the desired value and the state input vector. We denote the available values of the coefficients at time n by { q } . The canonical FF net of the mth block, with coefficients { c y } = {cij}, computes the outputs zy = fi(v7) of all neurons and the state output vector Stut(n) from the external input vector

If"(#)

+

n Nt m ) , u ( n- Nf = [ ~ (= [z?, zz", . . . ,Z G ]

+ m - 'I), . . . ,u(n - Nt + m - M + l ) ]

and the state input vector S g ( n ) = [~;t;+~, Z Z + ~. ., . ,z;t;+N]

as follows:

1. For i = 1 to M (external inputs): zy

= u(n

2. For i = M

- Nt + m - i + 1)

+ 1 to M + N (state inputs):

zy is given by the chosen algorithm (Table 1)

3. For i = M

+ N + 1to M + N + v - 1 (hidden neurons):

4. For i = M

+ N + v (linear output neuron):

Thus, the state output vector is szut(n)

[ZG+N+vr Z G + N + v + l ,

*.

m ZM+N+v+N-ll

.?

= [Y"? z k + l , .. . I z ~ + N - l l

and, ifN,-N,+l 5 m 5 Nt, weobtain from the desired valued(n-Nt+m) and the output ym: em = d ( n - Nf + m ) - ym.

0.Nerrand et al.

194

In the following, we present two methods for the computation of the gradient of J(n): the forward computation and the backpropagation techniques. 1. Forward Computation (Fig.4): We consider the whole set of Nfblocks as a static network on which we perform the forward computation technique. It is based on the following relation:

The linear FC net of the mth block computes, with coefficients {q}and cf:(q)}l the set of partial derivatives of the state output (including y") with respect to all coefficients Cjj: aSk:,/acjj(n). For the v(M + N) + (v - l)v/2 coefficients cq (i > 17: 1. For p = 1 to M (external inputs):

2. For p = M

+ 1 to M + N (feedback inputs):

az; - is given by the chosen algorithm (Table 1 ) acij

3. F o r p = M + N + l t o M + N + v - l (hiddenneurons):

4. For p = M

+ N + v (linear output neuFon):

Thus the partial derivatives of the state output are given by:


195

Once all partial derivatives of the output values y" are computed for the Nt blocks, the gradient of J ( n ) is obtained from

If the steepest-descent method is used, the coefficient modifications are given by Aci,(n) = -p- a'(n)

e"-w

=p m=Nt-N,+l

- C

-

Nt

Ac!(n)

m=Nt-N,+I

2. Backpropgufwn (Figure 8). Considering the effect of the coefficient ci, only, one has

thus

Then the gradient of J ( n )can be written as

where

This means that standard backpropagation can be applied to the whole set of NIblocks considered as a static network with replicated coefficients. The linear BP net of the mth block computes, with coefficients {cT} and { fi(q)},the set of partial derivatives of J(n) with respect to the potentials of all neurons. We define the following set of variables 97: 1. f o r i = M + N + v + N - 1 d o w n t o M + N + v + l : if m = Nt then qT = 0, otherwise L$" = 9L&+l

0. Nerrand et al.

196

2. for i = M

+ N + v (linear output neuron): +

if m = Nt then qy = e"', otherwise qy = em ; : ;4 note that qy = -3. for i = M

+ N + v - 1 down to M + N + 1 (hidden neurons):

where Ri is the set of indices of the neurons to which the ith neuron transmits its output

4. for i = M

+ N (last feedback input):

qr =

c c;qr

hERi

5. for i = M

q?

+ N - 1 down to M + 1 (other feedback inputs):

=

c chmi@

f qZN+v

hERi

Note that computation by backpropagation assumes implicitly that the derivatives of the feedback inputs of the first block (m = 1)with respect to the coefficients are equal to zero; this is in contrast to the forward computation of the gradient, where these values can be initialized arbitrarily. Note also that with the forward computation technique, the number of partial derivatives to compute for each block is v[vM (v - l)v/2] whereas with the backpropagation method this number is v. Once all partial derivatives of ](n) with respect to the potentials vy of all neurons are computed for the Nt blocks, the gradient of I(n) is obtained from

+

If the steepest-descent method is used, the coefficient modifications are given by


197

Acknowledgments

The authors are very grateful to 0. Macchi for numerous discussions that have been very helpful in putting neural networks into the perspective of adaptive filtering. C. Vignat has been instrumental in formalizing some computational aspects of this work. We thank H. Gutowitz for his critical reading of the manuscript. This work was supported in part by EEC Contract ST2JO312C.

References Applebaum, S. P.,and Chapman, D. J. 1976. Adaptive arrays with main beam constraints. IEEE Trans. Antennas and Propagation, AP-24, 650-662. Bellanger, M. G. 1987. Adaptive Digital Filters and Signal Analysis: Marcel Dekker, New York. Chen, S., and Billings S. A. 1989. Representations of non-linear systems: the NARMAX model. Int. 1.Control 49,1013-1032. Chen, S., Gibson, G. J., Cowan, C. E N., and Grant, P. M. 1990. Adaptive equalization of finite nonlinear channels using multilayer perceptrons. Signal Process. 20, 107-119. D r e y h , G., Macchi, 0, Marcos, S., Personnaz, L., Roussel-Ragot, P., Urbani, D., and Vignat, C. 1992. Adaptive training of feedback neural networks for non-linear filtering and control. In: Neural Networks for Signal Processing 11, S. Y. Kung, F. Fallside, J. Aa. Sorenson, C. A. Kamm, eds., pp. 550-559. IEEE. Elman, J. L. 1990. Finding structure in time. Cog. Sci. 14, 179-211. Fallside, F. 1990. Analysis of linear predictive data as speech and of ARMA processes by a class of single-layer connectionistmodels. In Neurocomputing: Algorithms, Architectures and Applications, F. Fogelman-Soulie and J. Herault, eds., pp. 265-283, Springer. Haykin, S. 1991. Adaptive Filter Theory: Prentice-Hall International Editions, Englewood Cliffs,NJ. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Hornik, K., Stinchcombe, M., and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2,359-366. Jayant, N. S., and Noll, P. 1984. Digital Coding of Waveforms. Principles and Applications to Speech and Video. Signal Processing Series, A. Oppenheim, Ed. Prentice-Hall, Englewood Cliffs, NJ. Jordan, M. I. 1985. The Learning of Representations for Sequential Performace. Doctoral Dissertation, University of California, San Diego. Jordan, M. I. 1989. Serial order: A parallel, distributed processing approach. Proc. Eighth Annu. Conf. Cog. Sci. SOC., 531-546. Lapedes, A., and Farber, R. 1988. How neural nets work. In Neural Information

198

0.Nerrand et al.

Processing Systems, D. Z. Anderson, ed., pp. 442-456, American Institute of Physics. McCannon, T. E., Gallagher, N. C., Minoo-Hamedani, D., and Wise, G. L. 1982. On the design of nonlinear discrete-time predictors. IEEE Trans. Inform. Theory 28,366-371. Narendra, K. S., and Parthasarathy, K. 1990. Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Networks 1,4-27. Narendra, K.S.,and Parthasarathy,K. 1991. Gradient methods for the optimization of dynamical systems containing neural networks. IEEE Trans. Neural Networks 2, 252-262. Nicolau, E.,and Zaharia, D. 1989. Adaptive arrays. In Studies in Electrical and Electronic Engineering 35. Elsevier, Amsterdam. Pearlmutter, B. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1, 263-269. Personnaz, L., Guyon, I., and Dreyfus, G. 1986. Collective computational prop erties of neural networks: New learning mechanisms. Phys. Rev. A 34,42174228. Picinbono, B. 1988. Adaptive methods in temporal processing. In U n d e m t e r Acoustic Data Processing, Y. T. Chan, ed., pp. 313-327. Kluwer Academic Publishers, Dordrecht. Pineda, F. 1987. Generalization of backpropagation to recurrent neural networks. Phys. Rev. Lett. 59,2229-2232. Pineda, F. J. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1,161-172. Poddar, P., and Unnikrishnan, K. P. 1991. Non-linear prediction of speech signals using memory neuron networks. In Neural Networks for Signal Processing, Proceedings of the 2991 IEEE Workshop, B. H. Juang, s. Y. Kung, and C. A. Kamm, eds., pp. 395-404. Press, W. H., Flannery, B. P., Teukolsky, S. A., and Vetterling, W. T. 1986. Numerical Recipes. Cambridge University Press, Cambridge. Proakis, J. G. 1983. Digital Communications. McGraw-Hill, New York. Robinson, A. J., and Fallside, F. 1989. A dynamic connectionist model for phoneme recognition. In Neural Networks from Models to Applications, L. Personnaz and G. Dreyfus, eds., pp. 541-550. IDSET, Paris. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, D. Rumelhart and J. McClelland, eds. MIT Press, Cambridge. Shynk, J. J. 1989. Adaptive IIR filtering. IEEE ASSP Mag. April, 4-21. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K.1989. Phoneme recognition using time-delay neural networks. IEEE Trans. Acoustics, Speech, Signal Process. 37, 328-339. Weigand, A. S., Huberman, B. A., and Rumelhart, D. E. 1990. Predicting the future: A connectionist approach. Int. 1.Neural Syst. 1,193-209. Widrow, B., and Steams, S. D. 1985. Adaptiw Signal Processing. Prentice-Hall, Englewood Cliffs, NJ. Williams, R. J., and Zipser, D. 1989a. A learning algorithm for continually


199

running fully recurrent neural networks. Neural Comp. 1,270-280. Williams, R. J., and Zipser, D. 1989b. Experimental analysis of the real-time recurrent learning algorithm. Connect. Sci. 1, 87-111. Williams, R. J., and Peng, J. 1990. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Comp. 2,490-501. Received 30 January 1992; accepted 21 August 1992.

This article has been cited by: 1. L. Saad Saoud, A. Khellaf. 2010. Nonlinear dynamic systems identification based on dynamic wavelet neural units. Neural Computing and Applications 19:7, 997-1002. [CrossRef] 2. A. Yilmaz, K. E. Akdogan, M. Gurun. 2009. Regional TEC mapping using neural networks. Radio Science 44:3. . [CrossRef] 3. Chi Sing Leung, Ah Chung Tsoi. 2006. Combined learning and pruning for recurrent radial basis function networks based on recursive least square algorithms. Neural Computing and Applications 15:1, 62-78. [CrossRef] 4. Su Lee Goh , Danilo. P. Mandic . 2004. A Complex-Valued RTRL Algorithm for Recurrent Neural NetworksA Complex-Valued RTRL Algorithm for Recurrent Neural Networks. Neural Computation 16:12, 2699-2713. [Abstract] [PDF] [PDF Plus] 5. Shih-Lin Hung, C. S. Huang, C. M. Wen, Y. C. Hsu. 2003. Nonparametric Identification of a Building Structure from Experimental Data Using Wavelet Neural Network. Computer-Aided Civil and Infrastructure Engineering 18:5, 356-368. [CrossRef] 6. Danilo P. Mandic . 2002. Data-Reusing Recurrent Neural Adaptive FiltersData-Reusing Recurrent Neural Adaptive Filters. Neural Computation 14:11, 2693-2707. [Abstract] [PDF] [PDF Plus] 7. P.A. Mastorocostas, J.B. Theocharis. 2002. A recurrent fuzzy-neural model for dynamic system identification. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 32:2, 176-190. [CrossRef] 8. Stefan C. Kremer . 2001. Spatiotemporal Connectionist Networks: A Taxonomy and ReviewSpatiotemporal Connectionist Networks: A Taxonomy and Review. Neural Computation 13:2, 249-306. [Abstract] [PDF] [PDF Plus] 9. Paolo Campolucci , Aurelio Uncini , Francesco Piazza . 2000. A Signal-Flow-Graph Approach to On-line Gradient CalculationA Signal-Flow-Graph Approach to On-line Gradient Calculation. Neural Computation 12:8, 1901-1927. [Abstract] [PDF] [PDF Plus] 10. L.S.H. Ngia, J. Sjoberg. 2000. Efficient training of neural nets for nonlinear adaptive filtering using a recursive Levenberg-Marquardt algorithm. IEEE Transactions on Signal Processing 48:7, 1915-1927. [CrossRef] 11. I. Rivals, L. Personnaz. 2000. Nonlinear internal model control using neural networks: application to processes with delay and design issues. IEEE Transactions on Neural Networks 11:1, 80-90. [CrossRef] 12. P. Campolucci, A. Uncini, F. Piazza, B.D. Rao. 1999. On-line learning algorithms for locally recurrent neural networks. IEEE Transactions on Neural Networks 10:2, 253-271. [CrossRef]

13. D. P. Mandic, J. A. Chambers. 1999. A posteriori error learning in nonlinear adaptive filters. IEE Proceedings - Vision, Image, and Signal Processing 146:6, 293. [CrossRef] 14. Gérard Dreyfus , Yizhak Idan . 1998. The Canonical Form of Nonlinear Discrete-Time ModelsThe Canonical Form of Nonlinear Discrete-Time Models. Neural Computation 10:1, 133-164. [Abstract] [PDF] [PDF Plus] 15. G.C. Mouzouris, J.M. Mendel. 1997. Dynamic non-Singleton fuzzy logic systems for nonlinear modeling. IEEE Transactions on Fuzzy Systems 5:2, 199-208. [CrossRef] 16. I. Kamwa, R. Grondin, V.K. Sood, C. Gagnon, Van Thich Nguyen, J. Mereb. 1996. Recurrent neural networks for phasor detection and adaptive identification in power system control and protection. IEEE Transactions on Instrumentation and Measurement 45:2, 657-664. [CrossRef] 17. Eric A. Wan , Françoise Beaufays . 1996. Diagrammatic Derivation of Gradient Algorithms for Neural NetworksDiagrammatic Derivation of Gradient Algorithms for Neural Networks. Neural Computation 8:1, 182-201. [Abstract] [PDF] [PDF Plus] 18. Tsungnan Lin, B.G. Horne, P. Tino, C.L. Giles. 1996. Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks 7:6, 1329-1338. [CrossRef] 19. C.W. Omlin, C.L. Giles. 1996. Rule revision with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering 8:1, 183-188. [CrossRef]

NOTE

Communicated by William W. Lytton

Fast Calculation of Synaptic Conductances Rajagopal Srinivasan Department of Electrical Engineering, Case Western Reserve University, Cleveland, OH 44206 USA Hillel J. Chiel, Departments of Biology and Neuroscience, Case Western Reserve University, Cleveland, OH 44106 USA Synaptic conductances are often modeled as sums of cr functions

where t is the current time, ti is the time of the ith spike in the presynaptic neuron, and T is the time constant of the synapse. If the time of decay of the synapse, TD, is not equal to its time of onset, TO, the conductance at time t after k spikes have occurred is

The drawback of these solutions is that one must keep track of the times of occurrences of each spike that initiated the synaptic potentials, and recalculate each exponential in the summation at each time step. This creates a large storage and computational overhead. Since both these equations represent the impulse response of a second-order differential equation, another approach is to numerically integrate additional differential equations for each synapse in the network (Wilson and Bower 1989). We have developed an improved method for computing synaptic conductances that separates equations 1 and 2 into two components: one that is a function of the current time of the simulation and one that accumulates the contributions of previous spike events to the synaptic conductance. We demonstrate that this method requires only the storage of two running sums and the time constants for each synapse, and that it is mathematically equivalent to equations 1 and 2. We will then demonstrate that it is also faster for a given level of precision than numerically integrating differential equations for each synapse. We will first describe our algorithm for equation 1, and then for equation 2. Neurul Computation 5,200-204 (1993) @ 1993 MassachusettsInstitute of Technology

Fast Calculation of Synaptic Conductances

201

Equation 1 can be rewritten as follows:

+

When the k 1st spike occurs at time tk+l, single terms can be added to each of the two summations in brackets to update them, eliminating the need to store spike times. To keep the exponentials inside the summation and outside the brackets from growing too large or small (respectively) as t increases over time, the exponentials can be rescaled. The left-hand term can be rewritten as

(4)

We will refer to the terms within the summation as Suml(tk). It can be updated once the k + 1st spike occurs as follows:

sum1( f k + l ) = e - ( f k + l - t k ) / T because after the k

Sum1(fk)

+I

(5)

+ 1st spike occurs, k

1

The right-hand term in equation 3 can also be rewritten as

I

i=l

+

Once the k 1st spike occurs, the new form of the terms within the summation [which we refer to as sud(tk+l)]would be

c

c k

k+l

tie(ti++l)/T

= e-(b+l-k)/T

tie-(tk-'g)/T

+ tkfl

(8)

i= 1

i=l

so that Sud(tk) is updated using the following rule: sud(tk+l) = e-(fk+l-tk)/T Sud(tk) + t k + l

(9)

Thus, at time t > f k + l , from equations 3-9, the synaptic conductance is equal to e-(t-fk+l)/r

r

[t(Suml(tk+l))- (sud(tk+l))]

(10)

Rajagopal Srinivasan and Hillel J. Chiel

202

The conductance needs to be evaluated at each time step of the simulation, that is, from time t to time t + At. This can be accomplished by which yields the term multiplying the term at time t, e-(r-tk+l)/+, by cAt/+, Thus, the conductances can be updated as at time t At, e-[('+A')-tk+ll/T. follows:

+

Suml(t + At) = e-*'/' Suml(t) + S Sum2(t + At) = e-At/r Sum2(t)+ St where S = 1 if a spike occurred at time t and S = 0 otherwise. The synaptic conductance can then be calculated at time t At from

+

1 g = -[(t 7

+ At) Suml(t + At) - Sum2(t + At)]

(13)

Equations 11, 12, and 13 summarize our algorithm for updating equation l, which requires storage of only Suml, Surd, and T . Since these equations are mathematically identical to equation 1, the accuracy of this method does not depend on the step size At, unless the step size becomes so large that spikes are missed. Of course, this constraint on step size is true for equation 1 as well. By the same logic, equation 2 can be updated as follows: Sum3(t + At) = e-A'"%un3(t) + S Sum4(t + At) = e-A'/roSum4(t)+ S where S = 1 if a spike occurred at time t and S = 0 otherwise, Sum3(t) = cb, e-(f-ti)/w,and Sum4(t) = e-(t-fl)/ro. The synaptic conductance can then be calculated at time t At from mTo [Suml(t + At) - Sum2(t At)] (16) g=

$, +

I, ,[

+

Determining the value of the conductance requires only that Suml,Sum2, TO, and TD be saved for each synapse. In addition, TDTO/(TD - TO) is a constant for a given synapse, and therefore can be precalculated for each synapse. This method requires far fewer exponentiations and additions than does the original closed-form solution (compare equations 1 and 2 to equations 11-16). Furthermore, the accuracy of our method is limited only by the machine precision. It also requires far less memory storage to maintain this accuracy. The number of spikes that must be stored to maintain the precision of equations 1 or 2 depends on (1) the spike frequency, (2) the synaptic time constant, and (3) the required precision level, E . To ensure that a spike that has occurred at some time to in the past will add less than E to equation 1, it must be true that ( ( t- to/t)e < E. It can be shown that, for E < setting (t - tO)/r equal to P ( E )= ln(l/E)

+ (1+ l / l n ( l / ~ ) ] [ ~ ( l n ( l / ~ ) ) ]

(17)

Fast Calculation of Synaptic Conductances

203

will always satisfy this constraint. For example, if E < the value of P ( E )would be 16.63, which implies that t-to must be equal to 16.637, that is, spike to must be stored until a time of 16.637 has elapsed, after which its contribution to equation 1 will be less than If the time constant T of this synapse is 50 msec, this requires that a spike be stored for 831.5 msec. The worst case size of the storage queue for a synapse would then be determined by this storage time, divided by the minimum period between spikes (which determines the maximum number of spikes that may occur during this storage time). If the input cell spikes with a minimum period of 100 msec between spikes, the queue would have to have room to store 831.5/100 M 9 spike times. In general, Max queue size = P(E)T/Tmin where Th,, is the minimum firing period of the input cell. Thus, storage requirements for equation 1 or 2 increase logarithmically with increasing precision [since, as E decreases, the fastest growing term in equation 17 is ln(l/~)I,increase linearly with the synaptic time constant 7, and increase inversely with the minimum firing period Tmh. If one chooses to implement a queue dynamically, one has the computational overhead of keeping track of which spikes have aged sufficiently to be dropped. Whether one uses a fixed size array or a dynamic array, one must use more storage as the precision, input firing frequency, or time constant of a synapse increases. How does our method compare to numerically integrating a secondorder differential equation, injecting new impulses each time an action potential occurs in the presynaptic neuron? A variety of techniques exist for numerically integrating differential equations (Press et al. 1988). One efficient, stable, and fairly accurate technique that is frequently employed is referred to as the exponential technique. For the second-order differential equation yielding equation 1 or 2, applying this technique yields the following finite difference equations (Wilson and Bower 1989, p. 328):

where x ( t ) is nonzero at the time a spike occurs, and zero otherwise. A drawback of this approach is that it is not inherently as precise as our method. One must trade off speed versus precision for equations 19-20. For example, if we choose to use a 1 msec time step for our method, in order to guarantee that deviations from it are smaller than we found that this numerical integration technique must be run with a step size 4 times smaller; to obtain deviations smaller than requires a step size 8 times smaller, and to obtain deviations smaller than requires a step size 16 times smaller. Of course, we could choose a larger step size for our method without loss of accuracy (see above), and the step sizes

204

Rajagopal Srinivasan and Hillel J. Chiel

for the numerical integration technique would then be proportionally smaller. We directly compared the time taken by the three methods by writing three benchmark programs in C (code listings available from the authors on request), and timing them on a Decstation 5000/200. As a reasonable precision limit for the methods, we chose For equation 1 and equations 11-13, a l msec step size was used. For equations 19-20, a l msec step size gave relatively poor precision (deviations were on the order of a step size of 0.125 msec was necessary to limit deviations to less than from the other two methods. The time constant for the synapse was 50 msec, the input spike frequency was 10 Hz, and we chose a queue size for the method of equation 1 that would maintain spike times until they had decayed to values smaller than (from equation 18, we determined that the queue should hold 9 spike times). Simulation time was 200 sec. Using these parameters, our method (equations 11-13) required only 2.1 sec of real time, whereas the method of equation 1 required 15.8 sec of real time, and the method of equations 19-20 required 18.1 sec of real time. These results suggest that our method is superior both in terms of speed and accuracy to previous methods. Acknowledgments

R. S. was supported by a Research Experience for Undergraduates Supplement to H. J. C.’s NSF Grant, BNS 88-10757.H. J. C.thanks the NSF for its support of this research. We are grateful for the comments of Dr.Randall Beer and two anonymous reviewers on an earlier draft of this manuscript. References Press,W. H., Flannery, B. P., Teukolsky, S. A,, and Vetterling, W. T. (eds.) 1988. Numerical Recipes in C . Cambridge University Press, Cambridge. Wilson, M. A., and Bower, J. M. 1989. The simulation of large-scale neural networks. In Methods in Neuronal Modeling, C. Koch and I. Segev, eds., pp. 291333. MIT Press, Cambridge. Received 11 February 1992; accepted 3 August 1992.

This article has been cited by: 1. Mark E. Nelson. 2010. Electrophysiological models of neural processing. Wiley Interdisciplinary Reviews: Systems Biology and Medicine n/a-n/a. [CrossRef] 2. Hans E. Plesser, Markus Diesmann. 2009. Simplicity and Efficiency of Integrate-and-Fire Neuron ModelsSimplicity and Efficiency of Integrate-and-Fire Neuron Models. Neural Computation 21:2, 353-359. [Abstract] [Full Text] [PDF] [PDF Plus] 3. Jan Reutimann , Michele Giugliano , Stefano Fusi . 2003. Event-Driven Simulation of Spiking Neurons with Stochastic DynamicsEvent-Driven Simulation of Spiking Neurons with Stochastic Dynamics. Neural Computation 15:4, 811-830. [Abstract] [PDF] [PDF Plus] 4. Maurizio Mattia , Paolo Del Giudice . 2000. Efficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical SynapsesEfficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical Synapses. Neural Computation 12:10, 2305-2329. [Abstract] [PDF] [PDF Plus] 5. Michele Giugliano . 2000. Synthesis of Generalized Algorithms for the Fast Computation of Synaptic Conductances with Markov Kinetic Models in Large Network SimulationsSynthesis of Generalized Algorithms for the Fast Computation of Synaptic Conductances with Markov Kinetic Models in Large Network Simulations. Neural Computation 12:4, 903-931. [Abstract] [PDF] [PDF Plus] 6. Michele Giugliano , Marco Bove , Massimo Grattarola . 1999. Fast Calculation of Short-Term Depressing Synaptic ConductancesFast Calculation of Short-Term Depressing Synaptic Conductances. Neural Computation 11:6, 1413-1426. [Abstract] [PDF] [PDF Plus] 7. J. Köhn , F. Wörgötter . 1998. Employing the Z-Transform to Optimize the Calculation of the Synaptic Conductance of NMDA and Other Synaptic Channels in Network SimulationsEmploying the Z-Transform to Optimize the Calculation of the Synaptic Conductance of NMDA and Other Synaptic Channels in Network Simulations. Neural Computation 10:7, 1639-1651. [Abstract] [PDF] [PDF Plus] 8. William W. Lytton . 1996. Optimizing Synaptic Conductance Calculation for Network SimulationsOptimizing Synaptic Conductance Calculation for Network Simulations. Neural Computation 8:3, 501-509. [Abstract] [PDF] [PDF Plus] 9. Alain Destexhe, Zachary F. Mainen, Terrence J. Sejnowski. 1994. Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. Journal of Computational Neuroscience 1:3, 195-230. [CrossRef] 10. A. Destexhe , Z. F. Mainen , T. J. Sejnowski . 1994. An Efficient Method for Computing Synaptic Conductances Based on a Kinetic Model of Receptor BindingAn Efficient Method for Computing Synaptic Conductances Based on a Kinetic Model of Receptor Binding. Neural Computation 6:1, 14-18. [Citation] [PDF] [PDF Plus]

Communicated by David Willshaw

NOTE

The Variance of Covariance Rules for Associative Matrix Memories and Reinforcement Learning Peter Dayan Terrence J. Sejnowski Computational Neurobiology Laboratory, The Salk Institute, P. 0.Box 85800, San Diego, CA 92186-5800 USA Hebbian synapses lie at the heart of most associative matrix memories (Kohonen 1987; Hinton and Anderson 1981) and are also biologically plausible (Brown et al. 1990; Baudry and Davis 1991). Their analytical and computational tractability make these memories the best understood form of distributed information storage. A variety of Hebbian algorithms for estimating the covariance between input and output patterns has been proposed. This note points out that one class of these involves stochastic estimation of the covariance, shows that the signal-to-noise ratios of the rules are governed by the variances of their estimates, and considers some parallels in reinforcement learning. Associations are to be stored between R pairs [a(w),b(w)] of patterns, where a(w) E (0, and b(w) E (0,l}”,using the real-valued elements of an m x n matrix W . Elements of a(w) and b(w) are set independently with probabilities p and r, respectively, of being 1. A learning rule specifies how element Wi, changes in response to the input and output values of a particular pair-the model adopted here (from Palm 1988a,b) considers local rules with additive weight changes for which: n

Wij =

C Aij(w),

where Aij(w) = f [ai(w), bj(w))

W=l

and f can be represented as [a,p, y,61 based on

One way to measure the quality of a rule is the signal-to-noise ratio (S/N) of the output of a single “line” or element of the matrix, which is a measure of how well outputs that should be 0 can be discriminated from outputs that should be 1. The larger the S/N, the better the memory will perform (see Willshaw and Dayan, 1990 for a discussion). A wide variety of Hebbian learning rules has been proposed for heteroNeural Computation

5,205-209 (1993)


Peter Dayan and Terrence Sejnowski

206

and autoassociative networks (Kohonen 1987; Sejnowski 1977a; Hopfield 1982; Perez-Vincente and Amit 1989; Tsodyks and Feigel'man 1988). The covariance learning rule fcov = [PT, -p(l - T), -(1 - p ) ~(1, - p)(l - T)], has the highest S/N (Willshaw and Dayan 1990; Dayan and Willshaw 1991); however, both it and a related rule, fprd = [-pr, -pr, -pr, 1 - pr] (Sejnowski1977a),have the drawback that (Y # 0, that is, a weight should change even if both input and output are silent. Note the motivations behind these rules: fcov

-

fprd

(input - p ) x (output - T) input x output - pr

Alternative rules have been suggested that better model the physiological phenomena of long-term potentiation (LTP) and depression (LTD) in the visual cortex and hippocampus, including the heterosynaptic rule fhet = [0, -p, 0 , l - p] (Stent 1973; Rauschecker and Singer 1979), and the homosynaptic rule fhm = [O,O, -T, 1 - T] (Sejnowski 197%; Stanton and Sejnowski 1989), motivated as fhet fhom

N

(input - p) x output input x (output - T)

These have been shown to have lower S/Ns than the covariance rule (Willshaw and Dayan 1990); however, for the sparse patterns, that is, low values of p and T, this difference becomes small. The sparse limit is interesting theoretically, because many more patterns can be stored, and empirically, because the cortex has been thought to employ it (see, for example, Abeles et al. 1990). All of these rules are effectively stochastic ap roximations of the covariance between input and output ((ai(w)- Lii) F j ( U ) - bj)) where the averages Ow are taken over the distributions generating the pltterns; they all share this as their common mean.' If inputs and outputs are independent, as is typically the case for heteroassociative memories, or autoassoaative ones without the identity terms, then their common expected value is zero. However, the rules differ in their variances as estimates of the covariance. Since it is departures of this quantity from its expected value that mark the particular patterns the matrix has learned, one would expect that the lower the variance of the estimate the better the rule. This turns out to be true, and for independent inputs and outputs the S/N of the rules:

S/N

[fcovl

w!p* r(1-p

1-r)

Variance of Covariance Rules for Matrix Memories

207

are inversely proportional to their variances. fcov is the best, fprd the worst, but the ordering of the other two depends on p and r. Circumstances arise under which the optimal rule differs, as for instance if patterns are presented multiple times but input lines can fail to fire on particular occasions-this would favor the homosynaptic rule. Exactly the same effect underlies the differences in efficacy between various comparison rules for reinforcement learning. Sutton (1984) studied a variety of two-armed bandit problems, which are conventional tasks for stochastic learning automata. On trial w, a system emits action y(w) E (0,l) (i.e., pulls either the left or the right arm) and receives a probabilistic reward r(w) E (0,l) from its environment, where

In the supervised learning case above, the goal was to calculate the covariance between the input and output. Here, however, the agent has to measure the covariance between its output and the reward in order to work out which action it is best to emit (i.e., which arm it is best to pull). Sutton evaluated r(w)b(w)- (y(w))] and an approximation to [r(w)- (r(w))][y(w) - (~(w))], where (y(w)) averages over the stochastic process generating the outputs and

is the expected reinforcement given the stochastic choice of y(w). These are direct analogues of fhst orfhom [depending on whether y(w)is mapped to a(w) or b(w)l and fcov, respectively, and Sutton showed the latter significantly outperformed the former. There is, however, an even better estimator. In the previous case, a and b were independent; here, by contrast, r(w) is a stochastic function of y(w). The learning rule that minimizes the variance of the estimate of the covariance is actually

+

where f = P[y(w) = O]p1 Pb(w) = l ] p ~pairs the probability of emitting action 0 with the reinforcement for emitting action 1. Williams (personal communication)suggested ? on just these grounds and simulations (Dayan 1991)confirm that it does indeed afford an improvement. Four previously suggested Hebbian learning rules have been shown to be variants of stochastic covariance estimators. The differences between their performances in terms of the signal-to-noise ratio they produce in an assodative matrix memory may be attributed to the differences in the variance of their estimates of the covariance. The same effect underlies the performance of reinforcement comparison learning rules, albeit suggesting a different optimum.

208

Peter Dayan and Terrence Sejnowski

Acknowledgments

We are very grateful to Steve Nowlan and David Willshaw for helpful comments. Support was from the SERC and the Howard Hughes Medical Institute.

References Abeles, M., Vaadia, E., and Bergman, H. 1990. Firing patterns of single units in the prefrontal cortex and neural network models. Network 1, 13-25. Anderson, J. A., and Rosenfeld, E., eds. 1988. Neurocomputing: Foundations of Research. MIT Press, Cambridge, MA. Baudry, M., and Davis, J. L. 1991. Long-Term Potentiation: A Debate of Current Issues. MIT Press, Cambridge, MA. Brown, T. H., Kairiss, E. W., and Keenan, C. L. 1990. Hebbian synapses: Biophysical mechanisms and algorithms. Annu. Rev. Neurosci. 13, 475-512. Dayan, P. 1991+ Reinforcement comparison. In Connectionist Models: Proceedings of the 2990 Summer School, D. S . Touretzky, J. L. Elman, T. J. Sejnowski and G. E. Hinton, eds. Morgan Kaufmann, San Mateo, CA. Dayan, P., and Willshaw, D. J. 1991. Optimal synaptic learning rules in linear associative memories. Biol. Cybernet. 65, 253-265. Hinton, G. E., and Anderson, J. A., eds. 1981. Parallel Models of Associative Memory. Lawrence Erlbaum, Hillsdale, NJ. Hopfield, J. J. 1982. Neural networks and physical systems with emergent computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kohonen, T. 1987. Content-addressableMemories, 2nd ed. Springer-Verlag,Berlin. Palm, G. 1988a. On the asymptotic information storage capacity of neural networks. In Neural Computers. NATO AS1 Series, R. Eckmiller and C. von der Malsburg, eds., Vol. F41, 271-280. Springer-Verlag,Berlin. Palm, G. 1988b. Local synaptic rules with maximal information storage capacity. In Neural b Synergetic Computers, Springer Series in Synergetics, H. Haken, ed., Vol. 42, 100-110. Springer-Verlag, Berlin. Perez-Vincente, C. j., and Amit, D. J. 1989. Optimised network for sparsely coded patterns. J. Phys. A: Math. General 22, 559-569. Rauschecker, J. P., and Singer, W. 1979. Changes in the circuitry of the kitten’s visual cortex are gated by postsynaptic activity. Nature (London) 280,58-60. Sejnowski, T. J. 1977a. Storing covariance with nonlinearly interacting neurons. J. Math. Biol. 4, 303-321. Sejnowski, T. J. 1977b. Statistical constraints on synaptic plasticity. J. Theoret. Biol. 69, 385-389. Stanton, P., and Sejnowski, T. J. 1989. Associative long-term depression in the hippocampus: Induction of synaptic plasticity by Hebbian covariance. Nature (London) 339, 215-218. Stent, G. S. 1973. A physiological mechanism for Hebb’s postulate of learning. Proc. Natl. Acad. Sci. 70, 997-1001.

Variance of Covariance Rules for Matrix Memories

209

Sutton, R. S. 1984. Temporal Credit Assignment in Reinforcement Learning. Ph.D. Thesis, University of Massachusetts, Amherst, MA. Tsodyks, M. V., and Feigel'man, M. V. 1988. The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 6, 101-105. Willshaw, D. J., and Dayan, P. 1990. Optimal plasticity in matrix memories: What goes up MUST come down. Neural Comp. 2,8593. Received 20 January 1992; accepted 14 September 1992.

This article has been cited by: 1. Max Garagnani, Thomas Wennekers, Friedemann Pulvermüller. 2009. Recruitment and Consolidation of Cell Assemblies for Words by Way of Hebbian Learning and Competition in a Multi-Layer Neural Network. Cognitive Computation 1:2, 160-176. [CrossRef]

Communicated by Geoffrey Hinton

NOTE

Optimal Network Construction by Minimum Description Length Gary D. Kendall Trevor J. Hall Departmentof Physics, King’s College London, Strand, London WC2R 2LS, UK

1 Introduction

It has been established that the generalization ability of an artificial neural network is strongly dependent on the number of hidden processing elements and weights (Baum and Haussler 1989). There have been several attempts to determine the optimal size of a neural network as part of the learning process. These typically alter the number of hidden nodes and/or connection weightings in a multilayer perceptron by either heuristic methods (Le Cun et al. 1990; Fahlman and Lebiere 1990) or inherently via some network size penalty (Chauvin 1989; Weigend et al. 1991; Nowlan and Hinton 1992). In this note an objective method for network optimization is proposed that eliminates the need for a network size penalty parameter. 2 Network Complexity

Rissanen has proposed a formalism of Ockham’s razor with the minimum description length (MDL) principle (Rissanen 1989). It is asserted that the smallest representation of an observed training set is the statistical process by which it was generated. Hence the most probable model 8 of any system S given a set of observed data is the model that minimizes the total description length L.

L = L(S I 8 ) + q e ) In a perfect coding system the description length of a set of symbols approaches the sum of the self information of the symbols. It is therefore straightforward to closely approximate the total description length of a neural network system by summing the self information of the parameters of the network itself and the input/output values of each training pair unaccounted for by the network.

L = z(s I e) + z(e) Neural Computation 5,210-212 (1993) @ 1993 Massachusetts Institute of Technology

Optimal Network Construction

211

To calculate the self information I associated with each of these parameters the sum of the self information I(Ek) of the constituent .bytes is taken.

where

Ek

are the individual events of each byte in each parameter and

P(Ek) is the a priori probability of Ek given the distribution of bytes in the corresponding set of parameters (k E (0, . . .,2551). Network optimization using this method can take place for any parameterized neural network model with supervised learning. In minimizing the total description length, both network learning and network construction take place simultaneously. An approach that can be used to optimize an adaptive network architecture by minimum description length is to include additional parameters indicating the status of each node/weight (i.e., used or unused) from a maximum network size. In then calculating the description length of the neural network model it is necessary to include only the network parameters used.

3 Discussion

It has been found experimentally that a genetic optimization algorithm (Goldberg 1988)is successful in minimizing the total description length over the discrete space involved in this method. Weights and training data parameters are coded as signed 16 bit words with each gene representing 8 bits. A population size of 500 with a mutation probability of has been found successful for a network with a maximum of fifteen hidden nodes. Results from network simulations have demonstrated that learning on a distribution about the XOR problem, with a general recurrent network, has resulted in the most simple feedforward design to prevail within 1000 generations: A multilayer perceptron with one hidden node and direct bottom to top connections. Simulations on line detection in 9 x 9 pixel images given 200 training examples, using a multilayer perceptron architecture, have taken 300,000 generations to converge to an optimal matched filter. The final network size and weight distribution produced are implicit in the quantity of and variation in the training data provided. Training by this method not only encourages weight elimination by the reduction of parameters as in Weigend et al. (1991)but also weight sharing through the minimization of the self information of the parameters as in Nowlan and Hinton (1992).

212

Gary D. Kendall and Trevor J. Hall

Acknowledgments The work of G. D. Kendall is supported by the UK Science and Engineering Research Council and Smith System Engineering Limited. References Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization. Neural Comp. 1(1),151-160. Chauvin, Y. 1989. A back-propagation algorithm with optimal use of hidden units. In Advances in Neural Information Processing Systems I , D. S. Touretzky, ed., pp. 519-526. Morgan Kaufmann, San Mateo, CA. Fahlman, S. E., and Lebiere, C. 1990. The cascade-correlation learning architecture. In Advances in Neural Information Processing Systems 2, D. S. Touretzky, ed.,pp. 524-532. Morgan Kaufmann, San Mateo, CA. Goldberg, D. E. 1988. Genetic Algorithms in Search, Optimisation and Machine Learning. Addison-Wesley, Reading, MA. Le Cun, Y., Denker, J. S., and Solla, S. A. 1990. Optimal brain damage. In Advances in Neural Information Processing Systems 2, D. s. Touretzky, ed., pp. 598605. Morgan Kaufmann, San Mateo, CA. Nowlan, S. J., and Hinton, G. E. 1992. Simplifying neural networks by soft weight-sharing. Neural C O M ~4(4), . 473-493. Rissanen, J. 1989. Stochastic Complexity in Statistical Inquiry. World Scientific. Weigend, A. S., Rumelhart, D. E., and Huberman, 8. A. 1991. Generalization by weight-elimination with application to forecasting. In Advances in Neural Infomation Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 875-882. Morgan Kaufmann, San Mateo, CA. Received 10 April 1992; accepted 3 September 1992.

This article has been cited by: 1. X.M. Gao, X.Z. Gao, J.M.A. Tanskanen, S.J. Ovasaka. 1997. Power prediction in mobile communication systems using an optimal neural-network structure. IEEE Transactions on Neural Networks 8:6, 1446-1455. [CrossRef] 2. Guozhong An . 1996. The Effects of Adding Noise During Backpropagation Training on a Generalization PerformanceThe Effects of Adding Noise During Backpropagation Training on a Generalization Performance. Neural Computation 8:3, 643-674. [Abstract] [PDF] [PDF Plus] 3. Gary D. Kendall, Trevor J. Hall, Timothy J. Newton. 1993. An investigation of the generalisation performance of neural networks applied to lofargram classification. Neural Computing & Applications 1:2, 147-159. [CrossRef]

Communicated by Jack Byme

A Neural Network Model of Inhibitory Information Processing in Aplysia Diana E. J. Blazis Thomas M. Fischer Thomas J. Carew Departments of Psychology and Biology, Yale University,

New Haven, CT 06520 USA

Recent cellular studies have revealed a novel form of inhibitory information processing in the siphon withdrawal reflex of the marine mollusc ApZysia: Motorneuronal output is significantly reduced by activity-dependent potentiation of recurrent inhibition within the siphon withdrawal network (Fischer and Carew 1991, 1993). This inhibitory modulation is mediated by two types of identified interneurons, L29s and L30s. In an effort to describe and analyze this and other forms of inhibitory information processing in Aplysia, and to compare it with similar processing in other nervous systems, we have constructed a neural network model that incorporates many empirically observed features of these interneurons. The model generates important aspects of the interactions of cells L29 and L30, and with no further modification, exhibits many network level phenomena that were not explicitly incorporated into the model. 1 Introduction

Recurrent inhibitory circuits are a common architectural feature of nervous systems, providing a powerful mechanism for rapid and precise control of neural output (Pompeiano 1984). Moreover, intrinsic and extrinsic modulation of recurrent inhibitory circuitry endows those networks with a high degree of flexibility and an enhanced capacity for adaptive modification. This type of modulation has been described in several systems including spinal motor neurons (Pompeiano 1984; Fung et al. 1988) and hippocampal circuits (Miles 1991). Recent cellular studies in the marine mollusc Aplysia have identified a recurrent inhibitory circuit in the neural network underlying the siphon withdrawal reflex (SWR)(Frost et al. 1988; Hawkins and Schacher 1989; Fischer and Carew 1991, 1993). The circuit is formed between identified interneurons L29 and L30: L29 provides excitatory input to L30, which projects back on L29 with an inhibitory synapse. The output element of this circuit, cell Neural Computation 5, 213-227 (1993) @ 1993 Massachusetts Institute of Technology

214

D. E. J. Blazis, T. M. Fischer, and T. J. Carew

L29, provides substantial input to siphon motor neurons (MNs). In the L29/L30 circuit, recurrent inhibition of L29 can be modulated in at least two ways: (1)it is reduced by stimulation mimicking tail shock (Frost et al. 1988), and (2) it is increased by prolonged direct activation of either L29 or L30 (Fischer and Carew 1991, 1993). Thus, L29/L30 interactions provide an example of a recurrent inhibitory circuit that can be modified by both extrinsic and intrinsic inputs. Moreover, the L29/L30 circuit is of additional interest because it exhibits an intrinsic form of plasticity, activity-dependent potentiation of recurrent inhibition (described more fully below). This form of use-dependent regulation of recurrent inhibition endows the L29/L30 circuitry with the capacity for dynamic gain control in the SWR (Fischer and Carew 1993). The SWR of Aplysia has been widely used as a model system for studies of learning and memory. The siphon is located in the mantle cavity and is used as an exhalent funnel for respiration (Fig. 1A). In response to tactile stimuli delivered directly to the siphon or to other sites on the body the animal withdraws its siphon into the mantle cavity. The SWR and its circuitry show a variety of adaptive modifications (for review, see Carew and Sahley 1986). Large cells and restricted neural networks have facilitated the identification and analysis of neural circuitry underlying this response and its modification. Interneurons L29 and L30 are located in the abdominal ganglion (a central ganglion that contains most of the circuitry underlying the SWR) (Fig. IB). L29 interneurons (about 5 in number) were previously shown to play an important role in excitatory information processing in the SWR circuit (Hawkins et al. 1981; Frost et al. 1988; Hawkins and Schacher 1989). L29s are activated by siphon stimulation, provide substantial excitatory input to identified siphon motor neurons (MNs, e.g., the LFS MNs, Frost et al. 1988, see also Fischer and Carew 1993), and receive recurrent inhibition from L30 interneurons (about 2 in number). Recently, Fischer and Carew (1991,1993) have shown that direct activation of a single L29 produces transient inhibition of the total reflex input to the MNs. They further showed that this inhibition occurs because L29s recruit recurrent inhibition onto themselves from L30 interneurons. The net effect of this recruitment of inhibition is that the L29 response to siphon input is significantly reduced for up to 40 sec following intracellular activation, resulting in a smaller net EPSP in the MN. The mechanism subserving this inhibition appears to be use-dependent facilitation of the inhibitory post-synaptic potential (IPSP) from L30 to L29 (Fischer and Carew 1991, 1993). We have recently developed a neural network model to describe this form of inhibitory information processing in Aplysia both to quantitatively analyze this type of adaptive modification in the SWR and to examine the possible behavioral significance of the L29/L30 circuit in the SWR neural network. Our overall strategy is to first represent in computational form the principal features of each identified cell, and then to

Inhibitory Information Processing in Aplysia

A.

215

B.

C.

Figure 1: Schematicillustrationof the siphon withdrawal reflex of Aplysia examined at several levels of analysis. (a) Diagram of intact Aplysia, illustrating the position of the siphon (adapted from Kandel 1979). (b) Diagram of a reduced preparation used to analyze neuronal correlates of behavior, showing approximate locations of interneurons and motorneuronsdescribed in the present study. Reflex input can be elicited with a tactile stimulus to the siphon while simultaneously recording from identified elements in the circuit for siphon withdrawal. (c) Connectivity of neural network model. The asterisk denotes a synapse that exhibits use-dependent potentiation.

216


progressively refine the model by continually adding empirically derived cellular parameters. Thus, a number of important cellular parameters and interactive mechanisms are not yet fully incorporated in the model. [Some of these features have been modeled by other investigators: sensorimotor facilitation and inhibition have been analyzed by Gingrich and Byrne (1985)and Byrne et al. (19891,and interneuronal contributions to MN response duration have been studied by Frost et al. (1991).1Nonetheless, our model describes key features of the relationship between L29 and L30 and generates network-level phenomena of the SWR, including activity-dependent potentiation of recurrent inhibition.

2 Methods

Biophysical and synaptic parameters in the model are derived or estimated from cellular studies in Aplysia and other invertebrates. Parameters of the model are listed in Table 1, and equations for selected features of the model are listed below. We have implemented our model with the network simulator GENESIS (Wilson et al. 19891, using an exponential Euler integration method and a time step of 0.1 msec.

2.1 Network Architecture. The network consists of a sensory module and four single compartment cells: three interneurons and a motor neuron (Fig. 1C).The sensory module consists of four receptors that mimic siphon sensory input. The number of sensory receptors selected was arbitrary. In Aplysia, sensory input from the siphon accelerates rapidly and then declines over the course of continued stimulation. The mechanisms underlying this accommodation are not yet incorporated into the model; instead, receptor firing rates are set so as to decline over stimulus presentation. Another feature of sensory neurons is synaptic depression, which occurs with repeated sensory neuron stimulation at relatively short interstimulus intervals (ISIs; Castelluci et al. 1970; Byrne et al. 1989). In our empirical studies, siphon stimulation was presented at an IS1 of 5 min, an interval that precludes habituation at both behavioral and synaptic levels. Therefore, the current model does not incorporate a representation of synaptic depression. The MN has only three synaptic inputs, one from the sensory module, one from a generic excitatory interneuron (denoted as E in Fig. 10, and one from a single L29. Three features of the biological circuit are incorporated in the network: (1) L29 makes an excitatory synapse onto L30; (2)L30, in turn, projects back upon L29 with an inhibitory synapse that shows marked use-dependent potentiation (indicated by an asterisk); and (3) there is a rectifying electrical synapse between L29 and L30.


217

Table 1: Parameters of the Model. Parameter

L29

L30

E

MN

c,

.63 -61 12

.63 -37 12

.63 -40 12

.63 -4s 12

1.7 +55 .0025 -65 2.85 -65

1.98 $55 0.61 -60

3.14 +55 0.28 -60

3.14 +55 0.61 -60

(nF) Em (mV) Rill

Active conductances G N (lo-' ~ S) (mV) GK, s) EK, (mV) G K ~ s) E m 9 (mV) Leak conductance G b , a (lO-'OS) E ~ e a k(mV) sensory inputs G (10-5 s) 71 (msec), 72 (msec) Em, (mV) output Target G (10-5 s) 71 (msec), (msec) E , (mV) Electrical s y n a p d Target

r Saturation (nA)

80 -75

1.7 13 55 L30/MN 0.11 1,3/ 1,7 55

L30 02-7

0.55

-

-

-

0.1

0.004

-75

-75

0.004 -75

5.5 13 55

7.5 L3 55

1.7 13 55

-

L29

MN

0.054-0.4ob

0.11

9,s -65

13

L29 0.05e-' 0.08

55

-

-

-

@Hyperpolarizedto -70 mV in all simulations. bConductancerange, computed as a function of WO activation frequency and time; see text equation 3. CCurrentacross electrical synapse: See equations 4a and 4b and accompanying text for explanation of terms.

2.2 Synaptic Conductances. The model incorporates time-dependent synaptic conductances. Ionic current resulting from activation of a synaptic conductance is calculated as follows:

where I = current, E,, reversal potential of the ion, and Emem= resting potential of the cell. G(t) is computed with the dual exponential form:

D.E.J. Blazis, T. M.Fischer, and T.J. Carew

218

where w is a weighting factor that is free to vary. For the EPSP and IPSP between L29 and L30 (respectively), the reversal potential and time constants of the simulated conductances reflect those empirically observed. The magnitudes of synaptic contributions from each of the three pathways converging onto the MN were specified in the following way. First, the strength of the sensory input was set to result in peak complex EPSP amplitude of 5-10 mV (in all cases, the MN was hyperpolarized to -70 mV, as in empirical studies). The strength of the L29 EPSP onto the MN was set to produce a depolarization of about 10 mV. The strength of the sensory input to L29 was set to yield an appropriate firing pattern in response to a short stimulus (that is, 4-6 action potentials to a 100 msec activation of the sensory receptors). Since the complex EPSP seen in the MN in response to sensory stimulation typically has a peak amplitude of 25-30 mV (Fischer and Carew 19921, the strength of the connection from interneuron E was then set to make up the difference between that amount and the contribution of the sensory array and the L29 input. A look-up function describing activity-dependent potentiation of the L30 IPSP has been incorporated in the model. This preliminary representation of potentiation of the L30 IPSP is based on a linear regression of data obtained by Fischer and Carew (1991,1992)and specifies that the inhibitory synaptic conductance from L30 onto L29 changes as a function both of L30 activation and of time:

+

+

~ ( t =) k ( O . 1 ~ 0.591~ I)

(3)

where x = spike count of L30 during activation, y = time after stimulation offset, and k = 3400 (a value which yields approximately a 1 mV IPSP in L29 at baseline). In the simulations presented here, activation of L30 by sensory stimulation is not taken into account. Thus the model slightly underestimates the weight of the inhibitory connection from L30 to L29. 2.3 Active Conductances. Active conductances are based on the Hodgkin and Huxley (1952)formulations. At present, each cell has a sodium conductance (GNJ, a delayed rectifier potassium conductance the implethat exhibits inactivation (GK"), and a leak conductance (GL); mentations of these conductances were drawn from existing GENESIS version 1.3 libraries. L29 has an additional, noninactivating potassium current (GKU~,derived from Byrne 1980). Parameters for the various conductances were set to approximate the firing patterns of each cell. In more detailed versions of the model, other conductances will be integrated as well (e.g., the Ca-dependent potassium conductance described for cell L30,Frost et al. 1988).

2.4 Electrical Synapse. The model includes a representation of the electrical synapse between L29 and L30. The current through the electri-


219

cal synapse at a given time is proportional to the differencebetween the potentials of cells L29 and L30:

where rl and r2 (values in Table 1) are coupling coefficients for the simulated electrical synapse between cells L29 and L30 that reflect the results of empirical studies (Fischer and Carew 1993). Like the biological synapse, the simulated electrical synapse is recbfying, such that current passes more readily from L29 to L30 than in the reverse direction. The current from one cell to another is also bounded (“saturation” term in Table 1)to reflect the fact that large voltage trajectories, such as those due to action potentials, are severely attenuated by the electrical synapse. Taken collectively, the above assumptions allow the model to generate salient features of the L29-L30 interaction. In the sections that follow, we compare our empirical results with our simulations of a variety of physiological manipulations of this simple network. 3 Results 3.1 Characteristic L29/L30 Interactions. Figure 2 shows that the model produces key features of the firing pattern of interneuron L29. The left column shows empirically obtained results (Fischer and Carew 1991,1993),and the right column shows the simulations. Traces A, through D1 of Figure 2 show characteristic features of cell L29: (1)L29 responds to siphon input with a brief burst of action potentials (Fig. 2A1). (2) L29 exhibits a characteristic response to maintained intracellular injection of depolarizing current, responding initially with a brisk burst of action potentials that becomes arrhythmic (Fig. 2B1). This %uttering” occurs because L29 activates L30, thereby recruiting inhibition onto itself. (3) Subthreshold depolarization of L29 recruits IPSPs back onto itself (Fig. 2C1) by virtue of current flow to L30 through the electrical synapse (L30has a lower voltage threshold for action potential initiation than L29). (4) L29 receives an IPSP from L30 that is potentiated by repetitive firing of L30 (Fig. 2D1).In the experiment shown in Figure 2D, cell L29 has been hyperpolarized to a membrane potential more negative than the reversal potential of the L30 IPSP, resulting in a depolarizing synaptic response. A single action potential from cell L30 (indicated by the arrow) results in a small IPSP in L29 (PRE).When cell L29 is then repeatedly activated, thereby activating L30 (not shown), dramatic usedependent facilitation results in a 2- to 3-fold increase in the L30 IPSP amplitude (POST). Our simulations of the L29-L30 interactions are qualitatively quite similar to the empirical data (Fig. 2, A2-41, but several aspects of the


220

EMPIRICAL RESULTS

SIMULATIONS

B,

m

WO

Splke

Figure 2 Comparison of empirical results and simulations showing characteristic features of the firing pattern of cell L29 and interactions between cells L29 and L30. (Al, A2) Response of cell L29 to brief tactile stimulation. (B1, b)Response of cell L29 to intracellular current injection. (C1, Cz) Recruitment of IPSPs onto L29 from L30 resulting from subthreshold current injection into L29. (D1,D2)Use-dependent potentiation of the IPSP from L30 to L29 (the IPSP is depolarizing due to hyperpolarization [HYP]of L29, see text). In both the empirical and simulated experiments, L29 was activated for 5 sec by current sufficient to produce a firing rate of 30 Hz (approximately 3 4 nA),which generates sufficient activation of L30 (not shown) to potentiate the L30-L29 IPSP

simulations warrant comment. First, the frequency of firing to siphon input is somewhat lower for the simulated L29 (Fig. 2A2)than that empirically observed. Second, the simulated firing frequency of L29 to current injection does not completely match the empirical data (Fig. 2B2). Third, there is a reasonable match of the empirical data to simulations of subthreshold recruitment of inhibition onto L29 (Fig. 2C2)and facilitation of the IPSP from L30 to L29 (Fig. 2D2).


221

In summary, the model produces essential features of the interaction between L29 and L30. In addition, as will be discussed below, with no further changes, the model also generates important aspects of the synaptic input to the MNs as well as inhibitory modulation of that input. 3.2 Reduction of M N Output by Voltage Clamp of L29. Empirical results shown in Figure 3 (upper half) illustrate the effect on the complex EPSP in a siphon MN of voltage clamping a single L29 during siphon stimulation. On the left, synaptic input to the MN and L29 was elicited by tactile stimulation of the siphon (see Fig. 1B). On the right are shown the responses of these same cells to siphon tap when a single L29 is functionally removed from the circuit by voltage clamping it at its resting potential (approximately -60 mV). In the example shown in Figure 3, the evoked complex EPSP in the MN is diminished to 50% of control. The simulated data (Fig. 3, lower half) capture this result qualitatively, although on average the simulated complex EPSP is shorter in duration than the empirically measured EPSP (note differences in gain and time base in Fig. 3). 3.3 Inhibition of M N Output by Intracellular Activation of L29. As described above, Fischer and Carew (1991, 1993) found that intracellular activation of L29 results in sigruficant inhibition of the tap-evoked complex EPSP in the MN for about 40 sec following L29 activation. In the example shown in Figure 4A1, activation of the S W R network by a siphon tap resulted in a complex E S P of about 25 mV in the MN and a brisk burst of action potentials in L29. A single L29 was then activated with intracellular current for 5 sec (not shown). Twenty seconds after L29 activation, the complex EPSP elicited by an identical siphon tap was significantly reduced (to about 5 mV), as was the response of L29 itself. Both the EPSP and the response of L29 to siphon tap recovered 5 min later. As shown in Figure 4A2, our simulations of activity-dependent potentiation of recurrent inhibition induced by activation of L29 are qualitatively similar to the empirical results. Specifically, 20 sec after L29 activation (and the consequent activity-dependent potentiation of the L30 IPSP onto L29) the tap-evoked complex EPSP is reduced (although the magnitude of the inhibitory effect is smaller than that empirically observed, see Discussion), and the response of L29 itself is also markedly reduced. Both the complex EPSP and the response of L29 to siphon tap recovered 5 min later. The mechanism subserving the inhibitory effects of L29 appears to be that it recruits inhibition onto itself from L30. Figure 4B1 is an empirical record showing that injection of depolarizing current into L29 activates L30 through both chemical and electrical synapses (Fig. 1C). This empirical result is also produced by the model (Fig. 4B2). That L29 activation potentiates the L30 IPSP implies that direct activation of L30 should also produce inhibition of the complex EPSP at the MN; this is indeed the

D.E. J. Blazis, T. M. Fischer, and T. J. Carew

222

case (Fischer and Carew 1991,1993). Moreover, the magnitude and time course of IPSP potentiation by activation of L29 (Fig. 2D1)map onto the inhibition of the tap-evoked response of the MN (Fischer and Carew 1991, 1993).

EMPIRICAL RESULTS

SIMULATIONS Baseline -------

Clamp

LFS Motor Neuron

L29

Vm

1

000 Mln SlphonTap

I

500 Mln

J15 mV 5 o m


223

4 Discussion

We have presented a preliminary computational analysis of a neural network that exhibits activity-dependentpotentiation of recurrent inhibition. We are encouraged that the model captures some of the key features of the L29-L30 interaction and, without further modification, exhibits some of the empirically observed circuit interactions, including inhibitory modulation, that were not explicitly incorporated into the model. 4.1 Discrepancies between Empirical Results and Simulations. The simulations of activation (through current injection) and inactivation (through voltage clamp) of L29 (Figs. 3 and 4) show that the model describes the empirical results qualitatively, but, at least in some cases, not quantitatively. First, the EPSPs in the simulated MN are of shorter duration than those observed empirically; and in the case of activation of L29, the magnitude of inhibition observed is somewhat smaller than that shown in Figure 4A (although the magnitude of the simulated inhibition is quite close to the average L29-induced inhibition of 20% observed by Fischer and Carew 1993). It is likely that the discrepancies between simulated and empirical results due to the fact that the current model incorporates only a single L29 into the circuit, whereas there are at least 5 L29 cells in the actual reflex circuit. Most of the L29s appear to contribute to the EPSP recorded in the MN (Frost et aZ. 1988; Hawkins and Schacher 1989; Fischer and Carew 1993), and thus our model underestimates the summed contribution of the entire L29 class. Ongoing simulations incorporating multiple L29s can diredly test this hypothesis (Blazis ef aZ. 1992) and, in addition, may shed light on the unique contribution of a single L29 to various features of network behavior, a determination that would be difficult (if not impossible) to achieve in cellular experiments alone. Related questions of interest in this computational analysis will focus on the functional significanceof particular architectural and synaptic features of the network, such as the role of multiple L29s and W s , the contribution of the redifyhg electrical synapse between L29 and L30, and the dynamics of facilitation of L30 synapses under different activation conditions.

Figure 3 Facing p g e . Comparison of empirical results and simulations showing the response of L29 and a siphon MN to tactile stimulation before and during voltage clamp of L29. For both the empirical and simulated results, left-hand traces show the mponse of the MN and a single L29 to a siphon tap (arrow) before voltage clamp. Right-hand traces show the response of the h4N and L29 when L29 is voltage clamped at its resting potential. In this and subsequent figures, the MN is hyperpolarized to reveal underlying EPSPs.

224

D.E. J. Blazis, T.M. Fischer, and T.J. Carew

4.2 Role of the L29L30 Circuit in Information Processing in the SWR. Our current cellular and computational analyses are aimed at determining the functional relationships between activity-dependentpotentiation of recurrent inhibition and other forms of plasticity observed in


225

the SWR. For example, since the L30 synapse potentiates at relatively low rates of activation (Fischer and Carew 1993)one form of plasticity that the L29-L30 circuit could theoretically mediate is habituation (habituation is know to involve reduction of afferent input to MNs with repeated sensory stimulation, see Castelluci et al. 1970). Build-up of use-dependent potentiation of the L30 IPSP, achieved via repeated siphon stimulation at short ISIs known to produce habituation, could progressively remove the contribution of L29 interneurons to siphon MN output. Indeed, our ongoing simulations suggest that the L29/L30 circuit alone can mediate at least partial habituation of reflex output, depending on stimulus duration and IS1 (Blazis et al. 1992). Thus, inhibition created by the L29/L30 recurrent inhibitory circuit could augment other mechanisms, such as homosynaptic depression, thought to subserve habituation in Aplysia (Castelluci et al. 1970). In addition to a role in nonassociative learning, L29 and L30 might also contribute to associative processes. For example, as described by Fischer and Carew (19931, this interneuronal network could play a role in changes in response topography associated with classical conditioning of the SWR (Hawkins et al. 1989; Walters 1989). In conclusion, cellular and computational analyses of simple nervous systems like that of Aplysia can yield insights that are useful for understanding both natural and artificial intelligent systems (see Hawkins and Kandel 1984; Card and Moore 1990). To date, most studies of the SWR of Aplysia have focused on a single synapse, that between the siphon sensory neurons and motor neurons (for review, see Carew and Sahley 1986). However, several studies (Frost et al. 1988; Hawkins and Schacher 1989; Fischer and Carew 1991, 1992) as well as the present work show that the SWR network also contains functionally important recurrent inhibitory circuits that can be modulated by both extrinsic

Figure 4: Facing page. Comparison of empirical results and simulationsshowing inhibition of the complex EPSP in the MN and recruitment of L30 firing following intracellular activation of L29. (Al, A2) Empirical result (All and simulation (A2) of L29-induced inhibition of the complex ESP in the h4N. For each pair of traces, the top trace shows the response of the MN md the lower trace shows the response of L29. 0.00 min: response of MN and L29 to siphon tap (arrow). 5.00 min: response of MN and L29 to siphon tap 20 sec following intracellular activation of a single L29 (3-4 nA, 5 sec, as in Fig. 2D). The diminished EPSP amplitude of the MN and reduced responding of L29 occurs because activation of L29 recruits inhibition from L30 onto L29, effectively removing L29 from the circuit (see text). 10.00 min: response of the MN and L29 to siphon tap 5 minutes after L29 activation. (B1, B2) Empirical result (B1) and simulation (B2) of activation of L30 by intracellular current injection into L29. Note that, in response to maintained depolarization of L29, L30 continues to fire action potentials (even after L29 has been silenced by recurrent inhibition)due to the electrical synapse between L29 and L30 (see Fig. 2B,C).

226


inputs and intrinsic activity. The ability of cellular studies in Aplysia to pmvide critical physiological constraints for a biologically realistic model greatly facilitates a computational analysis. In tun,a computational analysis can pmvide general insights into the principles of operation underlying different forms of information processing in an identified neural network. Acknowledgments

We thank Kent Fitzgerald, Edward Kairiss, and Emilie Marcus, and two anonymous reviewers for valuable comments on an earlier version of the manuscript. We are also grateful to David Berkowitz and Edward Kairiss for many helpful discussions. This research was supported by National Research Institute Service Award IF32-MH10134-02to D. E. J. B., Grant PHS T32-MH1839705to T.M. F., and AFOSR Award AF 89-0362to T.J. C. References Blazis, D. E. J., Fischer, T. M., and Carew, T. J. 1992. A neural network model of use-dependent gain control in the siphon withdrawal reflex of Aplysia. SOC. Neurosci. Abstr. 18,713. Byrne, J.H.1980. Quantitative aspects of ionic conductance mechanisms contributing to firing pattern of motor cells mediating inking behavior in Aplysia californica. J. Neurophysiol. 43, 651-668. Byrne, J. H., Gingrich, K. J., and Baxter, D. A. 1989. Computational capabilities of single neurons: Relationship to simple forms of associative and nonasd a t i v e learning in Aplysia. In Computational Models of Learning in Simple Neural Systems, R. D. Hawkins and G. H. Bower, eds., pp. 31-63.Academic Press, San Diego. Card, H.C., and Moore, W. R. 1990. Silicon models of associative learning in Aplysia. Neural Networks 3,333-346. Carew, T. J., and Sahley, C. L. 1986. Invertebrate learning and memory: From behavior to molecules. Annu. Rev. Neumci. 9,435487. Castelluci, V., Pinsker, H., Kupfermann, I., and Kandel, E. R. 1970. Neuronal mechanisms of habituation and dishabituation of the gill-withdrawal reflex of Aplysia. Science 167,1745-1748. Fischer, T. M.,and Carew, T. J. 1991. Activation of the facilitatory interneuron L29 produces inhibition of reflex input to siphon motor neurons in Aplysia. Soc. Neurosci. Abstr. 17,1302. Fischer, T. M.,and Carew, T. J. 1993. Activity dependent potentiation of recurrent inhibition: A mechanism for dynamic gain control in the siphon withdrawal reflex of Aplysia. I. Neurosc., in press. Frost, W. N., Clark, G. A., and Kandel, E. R. 1988. Parallel processing of shortterm memory for sensitization in Aplysia. J. Neurobiol. 19,297-334.


227

Frost, W. N., Wu., L. G., and Lieb, J. 1991. Simulation of the Aplysia siphon withdrawal circuik Slow components of internmnal synapses contribute to the mediation of reflex duration. SOC.Neurosci. Abstr. 17,1390. Fun& S. J., Pompeiano, O., and Barnes, C. D. 1988. Coerulospinal influence on recurrent inhibition of spinal motonuclei innervating antagonistic hindleg muscles of the cat. PflugmArch. 412,346-353. Gingrich, K. J., and Byme, J. H. 1985. Simulation of synaptic depression, posttetanic potentiation, and presynaptic facilitation of synaptic potentials from sensory neurons mediating gill-withdrawal reflex in Aplysia. J.Neurophysiol. 53,652-669. Hawkins, R. D., and Schacher, S. 1989. Identified facilitator neurons L29 and L28 are excited by cutaneous stimuli used in dishabituation, sensitization, and classical conditioning of Aplysia. J. Neurosci. 9,42364245. Hawkins, R. D., and Kandel, E. R. 1984. Is there a cell biological alphabet for simple forms of learning? Psychol. Rev. 91,375-391. Hawkins, R. D., Castelluci, j7. F., and Kandel, E. R 1981. Interneurons involved in mediation and modulation of gill-withdrawal reflex in Aplysia. I. Identification and characterization. J. Neumphysiol. 45,304-314. Hawkins, R. D., Lalevic, N., Clark, G. A., and Kandel, E. R. 1989. Classical conditioning of the Aplysia siphon-withdrawal reflex exhibits response specificity. Proc. Natl. Acad. Sci. U.S.A. 86,7620-7624. Hodgkin, A. L., and Huxley, A. F. 1952. A quantitativedescription of membrane current and its application to conduction and excitation in nerve. J. Physiol. (London) 138,253-281. Kandel, E. R. 1979. Behavioral Biology of Aplysia: A Contribution to the Comparative Study of OpisthobranchMolluscs. W. H. Freeman, San Francisco. Miles, R. 1991. Tetanic stimuli induce a short-term enhancement of recurrent inhibition in the CA3 region of guinea-pig hippocampus in vitro. J. Physiol. 443, 669-682. Pompeiano, 0. 1984. Recurrent inhibition. In Handbook of the Spinal Cord, N. Davidoff, ed., pp. 461-557. Marcel Dekker, New York. Walters, E. T. 1989. Transformation of siphon responses during conditioning of Aplysia suggests a model of primitive stimulus-response association. Proc. Natl. Acad. Sci. U.S.A. 86, 7616-7619. Wilson, M. A., Bhalla, U. S., Uhley, J. D., and Bower, J. M. 1989. GENESIS: A system for simulating neural networks. In Advances in Neural Information Processing Systems I, D. S. Touretzky, ed.,pp. 485492. Morgan Kaufmann, San Mateo, CA. Received 27 February 1992; accepted 21 August 1992.

This article has been cited by: 2. S. Yamada, M. Nakashima, S. Shiono. 1998. Reinforcement learning to train a cooperative network with both discrete and continuous output neurons. IEEE Transactions on Neural Networks 9:6, 1502-1508. [CrossRef] 3. Joe L. Martinez, Brian E. Derrick. 1996. LONG-TERM POTENTIATION AND LEARNING. Annual Review of Psychology 47:1, 173-203. [CrossRef]

Communicated by Gordon Shepherd

Computational Diversity in a Formal Model of the Insect Olfactory Macroglomerulus C . Linster Ecole Supkrieure de Physique et de Chimie Industrielles de la Ville de Paris, Luboratoire d'Electronique, 10, rue Vauquelin, 75005 Paris, France

C. Masson Luboratoire de Neurobiologie Comparie des Invertkhris, INRAICNRS (URA1190), 92140 Bures Sur Yvette, France

M. Kerszberg Biologie Molkulaire, CNRS (USA 1284), Institut Pasteur, 25, rue du Docteur Roux, 75015 Paris, France

L. Personnaz G . Dreyfus Ecole Suphieure de Physique et de Chimie Industrielles de la Ville de Paris, Luboratoire d'Electronique, 10, rue Vauquelin, 75005 Paris, France

We present a model of the specialist olfactory system of selected moth species and the cockroach. The model is built in a semirandom fashion, constrained by biological (physiological and anatomical) data. We propose a classification of the response patterns of individual neurons, based on the temporal aspects of the observed responses. Among the observations made in our simulations a number relate to data about olfactory information processing reported in the literature; others may serve as predictions and as guidelines for further investigations. We discuss the effect of the stochastic parameters of the model on the observed model behavior and on the ability of the model to extract features of the input stimulation. We conclude that a formal network, built with random connectivity, can suffice to reproduce and to explain many aspects of olfactory information processing at the first level of the specialist olfactory system of insects. 1 Introduction

We study the detection of sexual pheromones by insects, with a view to the more general modeling of the olfactory pathway. We use the known anatomical data of the olfactory system, and retain the level of detail we deem necessary to produce biologically relevant behavior. Thus, we do Neural Computation 5, 228-241 (1993) @ 1993 Massachusetts Institute of Technology

Model of the Insect Olfactory Macroglomerulus

229

not attempt to represent the particulars of dendritic passive propagation, or the precise local input-output functionalityof individual synapses. We model them by simple ingredients such as propagation delays and activation thresholds. Precise biological data on the wiring is not available, therefore we introduce randomness in the connectivity. A variety of approaches to the modeling of olfactory systems have been presented thus far: the pioneering work of Rall and Shepherd (1968) exploits a wealth of detail concerning the precise shape of mitral cell dendrites in order to compute electrical potentials; Wilson and Bower (1988, 1989) and Haberly and Bower (1989) replicate certain basic features of responses by extensive simulations of larger cell sets exploiting the same data; Lynch and Granger (Lynch et al. 1989; Lynch and Granger 1989) study associative memory and synaptic adaptation in piriform cortex, including considerable detail about synaptic processes, and a Hebb-type learning rule; Li and Hopfield (1989) attempt to abstract a set of relevant parameters from the biological details of the olfactory modular organization, with a highly simplified model: interneurons are lumped into single variables. In contrast, we study the individual and collective behavior of neurons whose dendrites make contacts within the so-called macroglomerulus (or macroglomerular complex, MGC), which is responsible for sexual pheromone recognition. The aim of our work is to analyze the emergence of the responses necessary for odor recognition and localization. 2 Biological Background

In the olfactory system of insects, two subsystems process behaviorally important odor classes: the specialist subsystem detects sexual pheromones, while the generalist subsystem recognizes food odors (for a review see Masson and Mustaparta 1990). In the following, we focus on the specialist subsystem. It receives information from sensory neumns, which are sensitive to non-overlapping molecule spectra ("labeled lines"). The axons of sensory neurons project onto the antennal lobe local interneurons, which possess no axons, and onto the antennal lobe projection or output neurons. The latter transfer signals to other centers for further integration with other sensory modalities. The huge convergence between pheromone-sensitive and projection neurons, which Ernst and Boeckh (1983) estimate as 5000:l in the cockroach, leads to a characteristic spatial organization of all synaptic connections in subassemblies termed gzomeruli, which are identifiable and species-specific. In the case of interest to us, (e.g., in certain moth species and in the cockroach),this reduces to a single MGC (Fig. 1). We use data pertaining to the moth species Manducu sexta and to the cockroach Penplaneta americana (for reviews see Christensen and Hildebrand 1987a; Boeckh and Ernst 1987). The complex responses to stim-

230

C. Linster et al.

Figure 1: schematic representation of the specialist oIfactory system. In the macmglomdus, receptor cell axons connect with local interneurons(restricted to the antennal lobe), and with pmjection n m n s , which convey information to higher centers.

ulation by pheromone blends, as observed intracellularly in projection newom, indicate that integrative pmesses take place in the MGC. In the moth, the depolarization of a local intemeuron can cause inhibition of background activity in a projection neuron. There is also evidence that local interneurons are responsible for much or all of the inhibitory synaptic activity (Christensen and Hildebrand 198%). Furthermore, the long-latency excitation exhibited by some projection neurons suggests that polysynaptic pathways are present between pheromone-responsive primary afferent axons and the projection neurons. In fact, it has been demonstrated, in the cockroach, that the receptor axons synapse mainly with local interneurons (Boeckh et al. 1989; Distler 1990). 3 The Formal Model

Neurons may be at rest ( x = 0) or above firing threshold ( x = 1). They are probabilistic neurons with memory: the probability P[Xi(t) = 11 that the state x i ( t ) of neuron i at time t is 1 is given by a sigmoid function of


231

the neuron membrane potential q ( t ) at time t:

which is biased by a positive threshold 8j, and where T is a parameter, called temperature, which determines the amount of noise in the network (random fluctuations of the membrane potential). In discrete time, the fluctuation of the membrane potential around the resting potential, due to input e;(t) at its postsynaptic sites, is expressed as At * V j ( t - At) - .ej(t - At)

+ 7i

where 7; is the membrane time constant and At is the sampling interval, with e;(t) =

C[W - xj(t~- Q)]

w h m Wij is the weight of the synapse between neuron j and neuron i, and q is its delay. The weights are binary. The value of the transmission delay associated with each synapse is fixed but chosen randomly; it is meant to model all sources of delay, transduction, and deformation of the transmitted signal from the cell body or dendrodendritic terminal of neuron j to the receptor site of neuron i. The mean value of the delay distribution is longer for inhibition than for exatation: we thereby take into account approximately the fact that IPSCsusually have slower decay than EPSCs, and may accumulate to act later than actually applied. We consider three types of neurons: receptor, inhibitory and exatatory. Two types of receptor neurons (A and B ) are sensitive only to input A or B, where A and B represent two odor components. For all A (respectively B ) type receptor neurons, we have ei(t) = A(t), [respectively B(t)l, where A(t) is the concentration of component A. Receptor neurons may make axodendritic (YY = 0), exatatory synapses with both types of interneurons. Interneurons may make dendrodendritic synapses (Tij # 0 ) with any other intemeuron, but the connectivity c will be sparse.

4 Results

To analyze the behavior of such a network, we first introduce a classification of the possible response patterns of the neurons, which has been found useful for the analysis of olfactory response patterns (Meredith

232

C. Linster et al.

1986; Kauer 1974; Fonta et al. 1991). In the network under investigation,' which exhibits a typical distribution of response patterns, we observe three classes of patterns: purely excitatory, purely inhibitory, and mixed (both inhibitory and excitatory) responses. Excitation and inhibition are defined in relation to the neuron spontaneous activity. The mixed response patterns subdivide into three groups, according to the relative durations of the inhibition and excitation phases (Fig. 2A and 28). We analyze the behavior of the network in response to four characteristics of the input patterns (pure odors, A or B, and mixed odors, A and B), which are behaviorally important (see Kaissling and Kramer 1990): (1) amplitude, (2) stimulus shape, (3)frequency of stimulus presentation, and (4) ratio of the components in mixed odors. The behavior of the model network exhibits several characteristicsthat agree with biological data: selective neurons respond to only one of the two odor components and nonselective neurons respond to both components. The neurons exhibit a limited number of response patterns, most of them a combination of excitation and inhibition (Fig. 3A and 3B). The recognition of the concentration ratio of odor components is of behavioral importance, but it is not known whether the detection of a precise ratio is achieved at the level of the glomerulus or at higher olfactory centers. Here, we observe amplitude and temporal variations of the response patterns of individual interneurons as a function of the concentration ratio. Interneurons with oscillatory responses code, by temporal changes in their response patterns (Fig. 4A), for ratio variations of the input stimulation. In addition, pairs of neurons respond simultaneously to mixed input of a specific input ratio: in contrast, the first spikes of the responses to other ratios are separated by 25-50 msec; thus, the response latency could be one of the response parameters that indicate ratio detection (Fig. 5). The odor plume formed downwind from the calling female possesses a highly variable structure. Pulsed stimulation improves a male moth ability to orient toward an odor source (Baker et al. 1985; Kennedy 1983). We have therefore observed the behavior of the interneurons in response to pulsed stimulation. We find that some interneurons cannot follow pulsed stimulation beyond a specific cut-off frequency (Fig. 4B). The ability of these neurons to detect a certain frequency range depends on their response pattern; the cut-off frequency of each neuron depends on

'Fifty neurons; connectivity c = 10%(190 synapses); synaptic strength w,j = +l/-l; 30% receptor neurons, 30% excitatory intemeurons, 40%inhibitory intemeurons; sampling step At = 5 msec, which is enough to study the maximal physiological spiking frequencies (Christensen el al. 1989a); membrane time constant 7' - 25 msec; synaptic delays are chosen from a uniform distribution between 10 and for excitatory synapses, and between 10 and 100 msec for inhibitory synapses; the parameters of the sigmoids are T = 1 and 8 = 1 for receptor neurons, T = 0.375 and 8 = 1.5 for the others.

6-m


233

20-40 At

!! 15-30 At

j Lnunu

2040 &

Rs:

1

R6:

15-30 At

I

Figure 2 (A) Response patterns: the amount of activation and inactivation is shown as a function of the stimulation (there At = 5 msec). R1, Activation for the duration of the stimulation. The spiking frequency varies as a function of the amplitude of the input. R2, The activation is followed by an inactive phase after the end of the stimulation. R3, Phasic burst, followed by an inactive phase of the same duration as the stimulus. R4, Phasic burst, followed by a tonic phase of diminished activation or by a phase of nonresponse, and by a short inactive phase after the end of the stimulation. R5, Phasic burst, followed by several phases of inactivation and activation (oscillatory response). R6, Inactivation during the application of the stimulation. The amplitude of the negative potential is a function of the amplitude of the stimulation.

234

C. Linster et al.

Figure 2 (B) Neurons responding with Rl-R6 (duration of each stimulation: 200 msec).

the duration of the stimulation and on the interstimulus interval. Neurons that respond with mixed exatation and inhibition show irregular responses and cannot follow high-frequency stimulation. Neurons that respond with exatation mostly respond continously to high-frequency stimulation. These behaviors depend mainly on the relations between the stimulation frequency, the interstimulus interval, and the temporal parameters of the model. Synaptic delays determine the behavior of mixed responses, while membrane time constants determine the behavior of excitatory responses. The stimulus profiles (rise and fall times of the odor signal) i n d i c a t e irrespective of the stimulus concentration-the distance between the lo-


235

Figure 3 Responses of selective (A) and nonselective(B) neurons to stimulation with one and both odors (duration of each stimulation: 200 msec).

cation of odor perception and the odor source. We observe a number of interneurons that reflect the profile of the stimulation irrespective of its concentration. This depends again on the response patterns; neurons that exhibit purely excitatory responses reflect the input profile by response latency and response duration, whereas neurons that exhibit an oscillatory response have completely different temporal response patterns as a function of the input profile (Fig. 4C). In this section, we have shown that the response patterns of individual neurons reflect various characteristics of the input pattern. Selective

236

C. Linster et al.

neurons indicate the presence, amplitude, and stimulus profile of one component (depending on their response pattern); nonselective neurons indicate the presence, amplitude, and stimulus profile of the mixture of the two components. Some nonselective neurons also reflect the quality of the mixture, that is, the ratio of the components. 5 Influence of the Distribution of Neurons and Synapses

The number and the diversity of the response patterns depend on the total number of neurons, on the distribution of excitation and inhibition in the network, on the number of connections and feedback loops, and on the temporal parameters (i.e., synaptic delays, membrane time constants). The diversity of response patterns grows with the percentage of synapses in the network (all other parameters remaining unchanged). At connectivity c < 2%, afferent synapses cause purely excitatory responses (Rl); around c = 2%, simple mixed responses (R2) and inhibitory responses (R6) appear; at about c = 8%, the majority of the interneurons respond mainly with excitation (R1 and R2). The full diversity and distribution of response patterns described above are observed for most networks around c = 10%. With an increasing number of synapses, the number of response patterns decreases. Due to an increasing network activity, the response patterns tend to oscillate, and the network saturates. Similarly, increasing the inhibitory synapse number beyond 50% introduces oscillations; the total activity in the network decreases. Beyond 60% inhibition, only R3 responses (phasic burst followed by a long inhibitory period) survive. If there is too much excitation in the network (more than forty percent excitatory neurons or more than forty percent receptor neurons), the network becomes unstable and saturates.

Figure 4 Facing page. Responses of selective and nonselective neurons with varying response patterns to stimulation with varying input characteristics. (A) Stimulation with varying ratios of the input components,the sum of the amplitudes of the two components being constant. Several neurons respond with varying temporal response patterns to changing ratios (duration of each stimulation: 50 msec). (B)Neuron 7 responds with phasic bursts to stimulation at low frequencies,and responds continually to stimulation at the same frequency but with shorter interstimulus intervals, because the interstimulus interval approches the membrane constant of the neuron (upper diagram: stimulation duration 30 msec, interstimulus interval 20 msec; middle diagram: stimulation duration 40 msec, interstimulus interval 10 msec; bottom diagram: stimulation duration 20 msec, interstimulus interval 10 msec). (C) Stimulation by input with varying profiles, the rise and fall times vary from 10 msec to 50 msec (stimulationduration 100 msec).


237

6 Discussion

In this section, we discuss the relevance of the results to the specialist system of insects. The model exhibits several behaviors that agree with biological data, and it allows us to state several predictive hypotheses about the processing of the pheromone blend. In the model, we observe two broad classes of interneurons: selective (to one odor component) and nonselective neurons. The fact that a distinct representation of pheromone components in parallel pathways

c.Linster et al.

238

Figure 5 Importance of response latenaes for ratio detection (stimulationduration 50 msec).

coming from the antenna is preserved by some antennal lobe neurons (local interneurons and projection neurons), but not all of them, has been reported in several species: in moths, Mmrducvr sexta (Christensen and 19831, and in the Hildebrand 1987a,b, 1989b) and Bombyx mMj (0cockmach, P e u a e (Boeckh 1976; Burrows etal. 1982; Boeckh and Selsam 1984; Hiisl1990). Selective and nonselective neurons exhibit a variety of mpnse patterns, which fall into three classes. inhibitory, excitatory, and mixed. Such a classification has indeed been pmlpposed for olfactory antennal lobe neurons (localinterneumns and pmjection neurons)in the specialist olfactory system in Mnduaa (Qlristensen et al. 1989a; Christensen and Hildebrand 1987a,b). Similar observations have been reported for Bombyx mon' (0% 1983)and for the cocboach (8urrows et d.1982; Boeckh and Ernst 1987). In our model we observe a number of local interneurons that cannot follow pulsed stimulationbeyond a neuroMIpeclficmt-off fmquency. This frequency depends on the neuron respanse pattern and on the duration of the interstimulusinterval. These d t s agree with data pertainins to antennal lobe neurons (intern-ns and prapction neurons) in Manduca serta (Christensen and Hildebrand 1988) and in Heliothis Oirescens (Christensen et al. 1989b). In both species, some antennal lobe neurons follow pulsed input with phasic bursts up to a cut-off fnquency. Physiological evidencein several species (ChxWemmand Hildebrand 198%; Burrows et al. 1982)has led to the hypothesis that some projection neurons (or local interneurons), may code for pheromone concentration and quality by measuring differences in response latency and duration, instantaneous spike frequency, and total number of spikes. Furthermore,

-


239

the overall response to the correct blend of pheromones may be qualitatively different from the response to some other ratio of pheromones (Christensen and Hildebrand 198%). Our model exhibits characteristics (Figs. 4A and 5) that could substantiate these suggestions. They will be analyzed and discussed in more detail in a forthcoming publication. 7 Conclusion

We have presented an original model of olfactory information processing in the macroglomerulus of insects. This model incorporates very simple ingredients; its c o ~ e c t i v i t yis chosen randomly, from distributions which take into account complete, albeit approximate, biological knowledge. From these simple assumptions, a variety of neuronal responses emerge, some of them strongly resembling those observed in living systems. Our model performs feature extraction on the signal represented in separate input lines. A number of feahws concerning the single odor components as well as their blend are represented in parallel lines by the interneuron network. These results agree with the hypothesis that there are ‘,separate but parallel lines of olfactory information flow between the antenna1 lobe and the protocerebrum, each line carrying information about different aspects of a pheromonal stimulus” (Christensen et al. 1989a). The use of random c o ~ e d i ~ iand t y synaptic delays gives us a means to study the conditions under which such feature extraction can arise, and the diversity of output patterns that are thereby exhibited. Thus, a model built with random c o ~ e d i v i t ysuffices to explain, reproduce, and predict a number of signal processing properties on the olfactory specialist subsystem. The variation of random distribution parameters and delays gives insights into the means whereby natural neural nets may be modulated by higher control mechanisms, be they genetic, adaptive, or instructive.

Acknowledgments This work has been supported in part by EEC BRAIN contract ST2J-0416C and by Ministere de la Recherche et de la Technologie (Sciences de la Cognition).

References Baker, T. C., W i l l i s ,M. A., Haynes, K. F., and Phelan, P.L. 1985. A pulsed cloud of sex pheromones elicits upwind flight in male moths. Physiol. Entomol. 10, 257-265. Boeckh, J. 1976. Aspects of nervous coding of sensory quality in the olfactory pathway of insects. Proceedings of the XV International Congress of Entomology, Washington,19-27 August 1976.

240

C. Linster et al.

Boeckh, J., and Selsam, P.1984.Quantitative investigation of the odor specificity of central olfactory neurons in the American cockroach. Chem. Senses 9(4), 369-380. Boeckh, J., and Emst, K. D. 1987. Contribution of single unit analysis in insects to an understanding of olfactory function. I. Comp. Physiol. A161, 549-565. Boeckh, J.,Ernst, K. D., and Selsam, P. 1989. Double labelling reveals monosynaptic connections between antennal receptor cells and identified intemeurons of the deutocerebrum in the American cockroach. Zool. I&.Anut. 119, 303-312. Burrows, M., Boeckh, J., and Esslen, J. 1982. Physiological and morphological properties of interneurons in the deutocerebrum of male cockroaches which respond to female pheromone. I. Comp. Physiol. 145,447-457. Christensen, T. A., and Hildebrand, J. G. 1987a. Functions, organization, and physiology of the olfactory pathways in the lepidopteran brain. In Arthropod Bruin: Its Evolution, Development, StructureundFunctions, A. P. Gupta, ed. John Wiley, New York. Christensen, T. A., and Hildebrand, J. G. 1987b. Male-specific, sex pheromoneselective projection neurons in the antennal lobes of the moth Munducu sextu. I. Comp. Physiol. A 160, 553-569. Christensen, T. A., and Hildebrand, J. G. 1988. Frequency coding by central olfactory neurons in the sphinx moth Manducu sextu. Chem. Senses 13(1), 123-130. Christensen, T. A., Hildebrand, J. G., and Tomlinson, J. H. 1989a. Sex pheromone blend of Munducu sextu: Responses of central olfactory interneurons to antennal stimulation in male moths. Arch. Insect Biochem. Physiol. 10, 281-291. Christensen, T.A., Mustaparta, H., and Hildebrand, J. G. 1989b. Discrimination of sex pheromone blends in the olfactory system of the moth. Chem. Senses 14(3),463-477. Distler, P. 1990. GABA-immunohistochemistryas a label for identifying types of local intemeurons and their synaptic contacts in the antennal lobe of the American cockroach. Histochemistry 93,617-626. Emst, K. D., and Boeckh, J. 1983. A neuroanatomical study on the organization of the central antennal pathways in insects. Cell Tissue Res. 229, 1-22. Fonta, C.,Sun, X. J.,and Masson, C. 1991. Cellular analysis of odour integration in the honeybee antennal lobe. In The Behuviour and Physiology of Bees, L. J. Goodman and R. C. Fischer, eds., pp. 227-241. C.A.B. International, London. Haberly, L. B., and Bower, J. M. 1989.Olfactory cortex: Model circuit for study of associative memory? TINS 12(7), 133. Hod, M. 1990. Pheromone-sensitive neurons in the deutocerebrum of Periplunetu umericunu: Receptive fields on the antenna. I. Comp. Physiol. A 167,321-327. Kaissling, K-E., and Kramer, E. 1990. Sensory basis of pheromone-mediated orientation in moths. Verh. Dtsch. Zoolo. Ges. 83, 109-131. Kauer, J, S. 1974. Response patterns of amphibian olfactory bulb neurons to odor stimulation. I. Physiol. 243, 695-715. Kennedy, J. S. 1983. Zigzagging and casting as programmed response to windborne odor: A review. Physiol. Entomol. 8, 109-112.


241

Li, Z., and Hopfield, J. J. 1989. Modeling the olfactory bulb and its neural oscillatory processings. Biol. Cybernet. 61,379-392. Lynch, G., and Granger, R. 1989. Simulation and analysis of a simple cortical network. In Computational Models of Learning in Simple Neural Systems, R. D. Hawkins and G. H. Bower, eds., pp. 205-238. Academic Press, New York. Lynch, G., Granger, R., and Larson, J. 1989. Some possible functions of simple cortical networks suggested by computer modeling. In Neural Models of Plasticity, J. H. Byrne and W. 0. Berry, eds., pp. 329-361. Academic Press, New York. Masson, C., and Mustaparta, H. 1990. Chemical information processing in the olfactory system of insects. Physiol. Rev. 70(1), 199-245. Meredith, M. 0.1986.Patterned response to odor in mammalian olfactory bulb: The influence of intensity. J. Neurophysiol. 56(3), 572-597. Olberg, R. M. 1983. Interneurons sensitive to female pheromone in the deutocerebrum of the male silkworm moth, Bombyx mori. Physiol. Entomol. 8, 419-428. Rall, W., and Shepherd, G. M. 1968. Theoreticalreconstructionof field potentials and dendrodendritic synapse interactions in olfactory bulb. J. Neurophysiol. 31,884-915. Wilson, M.A., and Bower, J. M. 1988. A computer simulation of olfactory cortex with functional implications for storage and retrieval of olfactory information. In Neural Information Processing Systems, D. Z . Anderson, ed., pp. 114-126. American Institute of Physics, New York. Wilson, M. A., and Bower, J. M. 1989. The simulation of large scale neural networks. In Methods in Neuronal Modelling: From Synapses to Networks, C . Koch and I. Segev, eds., pp. 291-334. MIT Press, Cambridge, MA. Received 6 December 1991; accepted 3 August 1992.


Communicated by Fernando Pineda

Learning Competition and Cooperation Sungzoon Cho James A. Reggia Department of Computer Science, University of Maryland, College Park, MD 20742 USA

Competitive activation mechanisms introduce competitive or inhibitory interactions between units through functional mechanisms instead of inhibitory connections. A unit receives input from another unit proportional to its own activation as well as to that of the sending unit and the connection strength between the two. This, plus the finite output from any unit, induces competition among units that receive activation from the same unit. Here we present a backpropagation learning rule for use with competitive activation mechanisms and show empirically how this learning rule successfully trains networks to perform an exclusive-OR task and a diagnosis task. In particular, networks trained by this learning rule are found to outperform standard backpropagation networks with novel patterns in the diagnosis problem. The ability of competitive networks to bring about context-sensitive competition and cooperation among a set of units proved to be crucial in diagnosing multiple disorders. 1 Introduction

Competitiveactivation mechanisms have recently been proposed as a method for producing competitive or inhibitory interactions between units (nodes) through functional mechanisms instead of inhibitory connections (Reggia 1985). A unit sends output to another unit proportional to the receiving unit's activation as well as the connectionweight between them. Since the total output from a unit is finite, competition arises among the receiving units; a stronger receiving unit with a higher activation level gets more input, indirectly reducing the input to weaker (lower activation) units, often until a clear winnerb) emerges. This process has the same effect as inhibiting other competing units without having direct inhibitory connections. This approach brings about more flexible and context-sensitive information processing than more traditional methods: a set of units can ' A similar idea was independently described in Rumelhart and McClelland (1986).

Neural Computation 5,242-259 (1993) @ 1993 Massachusetts Institute of Technology

Learning Competition and Cooperation

243

compete or cooperate depending on the context (this will be elaborated on later). Applications including print-to-sound transformation, diagnosis, satellite communication scheduling, and models of cerebral cortex have demonstrated that competitive activation mechanisms can work successfully, in many cases with substantially fewer connections than standard approaches (Reggia et al. 1988; Bourret et al. 1989; Peng and Reggia 1990; Reggia et al. 1992; Cho and Reggia 1992). Existing learning methods, however, cannot be used with competitive activation mechanisms because even in feedforward networks there is an implicit recurrent flow of information (to guide the competitive process). This lack of learning methods has greatly limited the usability of competitive activation mechanisms in the past. Thus, we have derived an error backpropagation (EBP) learning rule for networks employing these mechanisms (Cho and Reggia 1991). This new learning rule, referred to as competitive EBP, can be applied to networks with an arbitrary connection structure, but we restrict our attention to simple architectures where one is interested in the final state of a competition, not its process? In other words, our learning rule could be described as training networks to learn a set of fixed points of a dynamic system, unlike some recurrent learning rules that are concerned with a set of trajectories. This paper describes the competitive EBP learning rule and two applications of it where context-dependent competition and cooperation among units play key roles. The first application involves training networks with three units to learn to perform an exclusive-OR (XOR) operation on their inputs. This application illustrates clearly how competition and cooperation among units can function effectively for this simple, linearly nonseparable problem. A second, diagnostic application involves localization of damage in the human central nervous system given the hdings on a patient’s neurological examination. Networks are initially trained to identify the location of single disorders (sites of damage) given a set of manifestations associated with each disorder. Then previously unseen sets of manifestations are presented to the trained networks. It is found that the networks trained with competitive EBP produce better diagnostic hypotheses than networks trained with a standard backpropagation learning rule when multiple disorders are present simultaneously. Solving such multimembership diagnosticproblems is widely recognized to be a difficult task (Peng and Reggia 1990). The following section briefly describes competitive activation mechanisms and the competitive EBP rule in an informal manner [refer to Cho and Reggia (1991) for more details]. Simulation results involving the two applications described above are then presented to demonstrate that this approach can work effectively.

20ur learning rule has also proven to be effective in learning continuous-valued functions (Cho and Reggia 1992).

Sungzoon Cho and James A. Reggia

244

2 Activation Mechanisms and Learning Rule

Given an arbitrarily connected network, let the activation level of unit k at time t, ak(t), be given as

where output O U t k j ( t ) from unit j to unit k is distributed competitively by r

1

(2.2) LIEN

J

The function f k denotes any differentiable activation function. The weight on the connection from unit k to unit j is denoted by wkj, which is assumed to be zero when there is no connection between the two units. The term E k denotes a constant external input to unit k. The network-wide constant parameters a, P, and y represent decay, input gain, and output gain, respectively. Values of Q and p control how fast the activation decays while that of y determines how much output a unit sends in terms of its activation level. The parameter p determines how much competition exists among the units. The larger the value of p, the more competitive the model's behavior. The output O U t k j ( f ) is proportional not only to the sender's activation level aj(t), but also to the receiver's activation level, &(t). Therefore, a stronger unit receives more activation. Another unit I, which also gets input from unit j, can be seen as competing against unit k for the output from unit j because the normalizing factor ClEN w p ; ( f )in the denominator of equation 2.2 constrains the sum of the outputs from unit j to be equal to its activation level, u,(t), when y = 1. The activation sent to unit k, therefore, depends not only on those of the units sending it activation such as unit j , but also on those of its competitors to which unit k has no explicit connections. This has the effect of introducing implicit "feedback (recurrent effects) into networks in locations where explicit connections do not exist. This is why conventional backpropagation and other learning methods are not directly applicable in models where equation 2.2 controls the spread of activation. Applying gradient descent to the usual sum-of-square error measure (Rumelhart et al. 19861, we derived the following weight updating rule (2.3)

where B, = ElEN wlSu;.Since is zero except for those units I that have connections from unit s (i.e., wjs # 0), the summation is actually over only


245

those units to which unit s sends its activation. Therefore, learning can be carried out by a local computation. The bk values in equation 2.3 are the fixed point solutions of the dynamic system

(2.4) where Ak denotes the difference between the desired and the actual values at output unit k (zero at other units). Equation 2.4 appears to be complex but, in fact, is quite simple: all factors other than the &s in equation 2.4 are constants since this backward dynamic system is run after the forward system (equations 2.1-2.2) reaches equilibrium and the p becomes error signals at the output units have been c o m p ~ t e d When .~ zero (no competition), equation 2.4 reduces to the same backward dynamic system as derived in recurrent backpropagation (Almeida 1987; Pineda 1987) except for normalized weights. Equation 2.4 can be interpreted intuitively as follows. The first term, CjfN d j k b j ( t), represents ak's partial responsibility for the error signal at to unit j . This term also exists in a unit j since it influenced signal standard EBP learning rule (Rumelhart ef al. 1986) and in a recurrent EBP learning rule for noncompetitive activation mechanisms (Pineda 1987). The second term akAk is the error signal arising directly from external teaching signals and is nonzero only for output units. The third term, PCmeNoUth{bk(t) - (I/?) C I € N O U t f r n [ ~ f ( t ) / a r n ] } ,occurs only in Our cornpetitive EBP learning rule for competitive activation mechanisms. The quantity p CmEN o U t h b k ( f ) can be interpreted as accounting for the fact that unit k is indirectly responsible for its own error signal through its influence on the incoming activation signal outh from unit rn (the term outh has factor a k in it). However, unit k is not solely responsible for that error since the activation level of unit k itself is indirectly influenced by those of its competitors, that is, the other units to which unit m sends its output signals. The sum (I/?) C l E N o u t ~ , [ 6 / ( ( t ) / a m ] can be viewed as the amount that needs to be subtracted from unit k to compensate for this indirect influence by competitors. This term was derived from the denominator term B, of o u t h . Learning proceeds as follows. A training pattern is presented as input to the network, the forward dynamic system (equations 2.1-2.2) is run until it reaches equilibrium, the error signal A for each output unit is cal~~~

~~~

3Benaimand Samuelides proved convergence of feedforward networks employing a slightly different competitive activation mechanism (Benaim and Samuelides 1990). Empirically, we encountered neither oscillations nor divergence.

246


culated, the backward dynamic system (equation 2.4) is run to compute the error signals for all the units in the network, and then the weight on each connection is changed according to the learning rule (equation 2.3). These steps are taken for all training patterns and this is said to comprise one epoch. Learning continues until either all training patterns are correctly learned or a preset time limit of 400 epochs expires. Simulations were conducted using Maryland/MIRRORS 11, a simulation software environment that constructs and runs a neural network as specified by the user.

3 Exclusive-OR (XOR)

Exclusive-OR (XOR) has been a popular problem to test learning algorithms due to the historic fact that one-layer linear models and elementary perceptrons are unable to learn the task. Because of its linear nonseparability property, a hidden layer of units is required to learn the task with backpropagation algorithm (Rumelhart et al. 1986). Here we use a competitive activation mechanism and our competitive EBP learning rule to train a network with two input units and an output unit to learn XOR.4 Competition and cooperation among a set of units were found to play a crucial role in performing the task. Figure 1 depicts the three-unit network used in the simulation. Units 1 and 2 represent input units and unit 3 represents the output unit. A sigmoid function, f ( x ) = 1/(1+ e-s(x-0.5)),was chosen as f k ( x ) in equation 2.1 for all k. With larger values of s, the sigmoid function becomes steeper, producing clearer winners and losers during a simulation. However, large values of s also cause the derivatives of the activation functions to vanish, making error signals much less significant. Such "flat spots" have been recognized previously and various cures suggested (Fahlman 1988; Hinton 1989; Cho and Reggia 1991). Here we start with a small s value of 1 and gradually increase it to 10 (this process is not simulated annealing). All of the parameters a,P, and y and competition parameter p were set to 1 and fixed throughout learning. Four input patterns (l,O), (O,l), (l,l),and (0,O)were used with the corresponding target outputs l,l,O, and 0, respectively. A total of 45 different nonsolution initial weight states were used, 32 cases of which were randomly generated and the remaining 13 cases of which were handcrafted so that the initial weight state was located far away from a solution weight state. In all 45 cases, the competitive EBP learning rule successfully changed the initial nonsolution state into a solution state. The mean, median, minimum, and maximum number 40Ur input units are doing more than what ordinary input units do, but since they receive direct external inputs, they are not hidden units.


247

I

I

El

E2

Figure 1: The three-unit network trained to learn XOR. Weight wi, represents the weight from unit j to unit i. External inputs E l and E2 are fed to input unit 1 and 2, respectively. The activation level at unit 3 at equilibrium is considered to be the output of the network.

of learning epochs taken were 145, 44, 33, and 840, respectively. The networks with handcrafted initial weights were found to take a much longer time than the networks with randomly generated initial weights as expected. All of the final solution states obtained share the property that the connections from input to output units are stronger than those between input units (i.e., wZ1< w31 and wI2< w32). How such sets of weights perform XOR with a competitive activation mechanism is as follows. When the pattern (1,O) is applied to the input units, input unit 2 and output unit 3 compete for the output from input unit 1. Since ~ 3 >1 W Z I ,unit 1 sends a larger amount of activation to unit 3 than unit 2. With a larger input from unit 1, the activation level of unit 3 surpasses that of unit 2. Then, with the help of a slightly larger activation level, unit 3 gets an even larger amount of input from unit 1than does unit 2 since the input amount to unit 3 is proportional not only to the connection weight but also to the activation level of unit 3. This process of increasing the activation level of unit 3 accelerates while unit 2 remains close to where it started, that is, near zero. The network finally reaches equilibrium with unit 3 as the winner and unit 2 as the loser, which means that the activation levels of unit 1 and unit 3 are close to one and that of unit 2 is close to zero. This is exactly what is desired. By symmetry, unit 3 wins with the input pattern (0,l) since ~ 3 >2 w12. With the pattern (l,l),initially two competitions occur at the same time: between unit 1 and unit 3 for

248


unit 2’s activation, and between unit 2 and unit 3 for unit 1’s activation. With both unit 1 and unit 2 receiving external inputs and feeding each other (cooperation),they both win over unit 3, overcoming the fact that w 3 1 > w 2 1 and ~ 3 > 2 ~ 1 2 . 5 Finally with pattern (O,O), unit 3 does not get any significant input from either of the input units so it stays at the initial activation level which is zero. Through competition and cooperation among units, a competitive activation mechanism implements XOR task. 4 Diagnostic Associations

Recently, backpropagation models have been applied to diagnostic problem solving with some success. However, these backpropagation models for diagnosis apply to small, circumscribed decision problems where it is usually assumed that at most a single disorder is present. For example, the system for diagnosing myocardial infarction was limited to determining the presence or absence of that single disorder (Baxt 1990). The previous backpropagation models thus typically perform a pattern classification task (selection of one diagnostic category out of several when given a problem description), and do not in any sense solve general diagnostic problems where one must construct a multiple disorder solution from individual disorders. This is a major limitation because in many diagnostic problems multiple disorders may be present simultaneously (referred to here as multiple disorder patterns) (Peng and Reggia 1990). Further, a diagnostician is often presented with manifestations that are a proper subset of the manifestations associated with a disorder (referred to here as partial manifestation patterns). What would be desirable is to train a network with “textbook examples” consisting of all manifestations associated with each individual disorder (referred to here as prototypical patterns), yet for the trained network to produce reasonably good diagnosis with nonprototypical multiple disorder patterns and partial manifestation patterns. Connectionist models based on backpropagation have not been demonstrated previously to perform well for multiple disorder problems. Our hypothesis was that networks trained by competitive backprop agation learning rule would produce better diagnoses given multiple disorder patterns and partial manifestation patterns since they do not employ inhibitory connections. To assess this, we trained networks using both standard EBP (referred to as standard networks) and competitive EBP (referred to as competitive networks), and then compared their respective performance with previously unseen multiple disorder patterns and partial manifestation patterns.

’This process does not work when w31 >> wZ1,and w32 B w12.


249

4.1 The Localization Task and Training Networks. The medical information used for training is based on standard reference sources and personal knowledge of one of the authors OR) (Adams and Victor 1985; Brazis et al. 1985). We selected 29 different manifestations (Table 1) and 16 localizations of brain damage, so 16 associations between a disorder and a set of prototypical manifestations had to be learned (Table 2). Each disorder represents damage to a specific part of the brainstem, cerebellar hemispheres, or cerebral hemispheres. With each ,manifestation and disorder assigned a numbered index, these associations can be viewed as 16 binary 1 / 0 patterns (Table 3). Presence of a 1 at the ith row and jth column indicates a causal relation between the ith manifestation and the jth disorder. For all 16 patterns, during training only one output unit was to be turned on, representing the presence of the disorder whose prototypical manifestations were present. These prototypical patterns, created as described above, were found to be linearly separable since the 16 input patterns are linearly independent when viewed as a binary vector (Hertz et al. 1991). Because these prototypical patterns were linearly separable, we used networks with two layers of units, namely, input and output units only. However, this is not an easy problem to solve: we are not concerned here with just classdying input patterns into one of n categories. We are interested in the far more difficult task of identdymg a set of categories when a network has only been trained with singledisorder exemplars. Thus, for example, with diagnosis involving n disorders the cardinality of the output space is 2", not n. This kind of multimembership problem is widely recognized to

Table 1: Manifestations (Input Units) Nos. 1, 2

3,4 5, 6 7, 8 9, 10 11, 12 13, 14 15, 16

17 18, 19 20,21

22,23 24'25 26,27 28'29

Manifestations Left, right hemiparesis Left, right facial paresis Left, right tongue paresis Left, right gaze palsy (conjugate) Left, right intemuclear opthalmoplegia Left, right 3rd nerve palsy Left, right 6th nerve palsy Left, right Horner's syndrome Nystagmus Left, right hemiataxia Left, right touch-proprioceptionimpairment Left, right pain-temperature impairment Left, right facial sensory impairment Left, right hemianopsia Sensory, motor aphasia


250

Table 2 Disorders (outputunits) and Their Corresponding Manifestations No. Disorder

Left medial medulla Right medial medulla Left lateral medulla Right lateral medulla Left medial pons Right medial pons Left lateral pons Right lateral pons Left midbrain Right midbrain Left cerebellum Right cerebellum Left frontal lobe 14 Right frontal lobe 15 Left pareitotemporal lobe 16 Right pareitotemporal lobe

1 2 3 4 5 6 7 8 9 10 11 12 13

Manifestations (see Table 1) 2, 5, 21 1, 6, 20 15, 17, 18, 20, 23, 24 16, 17, 19, 21, 22, 25 2, 4, 6, 7, 9, 13, 17, 18, 21 1, 3, 5, 8, 10, 14, 17, 19, 20 3, 7, 15, 17, 18, 21, 23, 24 4, 8, 16, 17, 19, 20, 22, 25 2, 4, 6, 11, 17, 18, 21, 23 1, 3, 5, 12, 17, 19, 20, 22 17, I8 17,19 2, 4, 6, 8, 29 1, 3,5, 7 21, 23, 25,27, 28 20,22, 24, 26

be a very challenging problem in statistical pattern recognition and diagnostic problem solving (Peng and Reggia 1990). There is no generally recognized ideal solution to such problems at present, and any approach (neural network or otherwise) that can approximate solutions is worth investigation. The standard network we used is shown in Figure 2a and the competitive network is shown in Figure 2b. The standard network used a single bias unit that was always on. The multiple "bias units" in the competitive network were an experiment: they were introduced originally to develop as feature detectors. In other words, they were expected to selectively turn on for certain features in the input manifestation patterns. 4.2 Learning Prototypical Patterns (Single-Disorder Patterns). Both standard and competitive networks learned the prototypical patterns rather easily! First, we successfully trained standard networks with zero, four, and eight hidden units in around 90, 300, and 210 epochs, respectively. For each input pattern of a set of manifestations, the correct 6Parameters a, p, 7, and p in equations 2.1 and 2.2 were set to 1.0. Parameter s increased linearly from 3 to 8 over 200 epochs and then remained fixed. Constant 0.1 was added to the derivative of the activation function to accelerate learning (Fahlman 1988). For both forward and backward dynamic systems equilibrium was assumed if every dynamic variable changed less than 0.01.


251

Table 3: Manifestation-Disorder Associations (I/O patterns) Manif.

No.

1

1

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

1 1

1

Disorder No. 2 3 4 5 6 7 8 9 10 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

12 13 14 15 16 1 1 1 1 1 1 1

1

1 1 1

1 1 1 1 1 1 1 1 1

disorder unit became strongly activated while others did not. Since the standard networks with no hidden units gave best generalization performance, we describe results only from the network with no hidden units (shown in Fig. 2a). We also trained 35 different competitive networks. Five different numbers of bias units were used, ranging from zero to four. For each case, seven different competitive networks were generated by randomly assigning initial weights. Any competitive network with at least one bias unit learned the training patterns successfully in between 21 and 45 epochs, with an average of 36 epochs? The bias units did not learn to become feature detectors, but functioned in a fashion equivalent to a single bias unit that always came on. Furthermore, the competitive networks with exactly one bias unit produced the best diagnosis for multiple disorder patterns and partial manifestation patterns. Thus, we 'Although competitive networks took fewer epochs in training than standard networks, each epoch with a Competitive network takes more real time due to the recurrent nature of competitive activation mechanisms.


252

b.

a.

OUTPUT

biaa

OUTPUT

I . . . ]

'1 000

... INPUT

0

000

...

0

INPUT

Figure 2: (a) Standard network and (b) competitive network. The standard network shown in (a) consists of 29 input units representing the manifestations listed in Table 1, 16 output units representing the disorders listed in Table 2, and a bias unit. Each input unit and the bias unit is connected to every output unit. Standard networks with a hidden layer of four and eight units were also trained, but did not give a better performance in either training or testing and so are not considered furthermore. Initially, weights were randomly chosen from a uniform distribution in the range of [-0.3,0.3]. The competitive network shown in (b)has the same set of input and output units and connections as the standard network, except that it has a set of zero to four bias units, receiving connections from input units (see text for explanation). A total of 35 competitive networks (seven for each number of bias units) were randomly generated with initial weights picked from a uniform distribution in the range of [0,1]. Input units are fully connected to bias units, which are in turn fully connected to output units. Both incoming and outgoing connections of the bias units were updated according to the learning rule.

present only results from one of the competitive networks with one bias unit. The weight values of the trained standard network with no hidden unit (but one bias unit) and those of the trained competitive network with one bias unit are shown in Figure 3a and b, respectively. Note that both networks developed a similar but not identical pattern of excitatory connections, but the competitive network does not have any inhibitory connections. If the trained networks were pruned, that is, if connections with very small valued weights were removed from the networks, the competitive network would become much smaller than the standard network. As we explain below, examination of these weight values provides insight as to why competitive networks produce a better diagnosis than standard networks with multiple disorder patterns and partial manifestation patterns.


253

a.

b.

Figure 3: Connection weights of trained (a) standard and (b) competitive networks. Each box represents a connection from an input to an output unit. The size of a circle in the ith row and jth column is proportional to the size of the connection weight from input unit j - 1 to output unit i with the bias unit denoted as unit 0 (leftmost column). The filled circles represent positive weight values and the open circles represent negative weight values. Note that the trained standard network has a large number of inhibitory weights, and that the bias unit in the standard network inhibits output nodes while that in the competitive network excites them.

4.3 Testing Multiple Disorder Patterns. For multiple disorder patterns, the combined manifestations of two disorders were presented simultaneously to the trained networks. A total of 48 multiple disorder

254

Sungzoon

Cho and James A. Reggia

input pattern pairs were selected for testing (with Pi denoting the pattern of input manifestations associated with disorder i, PI, plus each of the preceding 15 patterns, PI, plus each of the preceding 13 patterns, PI, plus each of the preceding 11 patterns, and PI, plus each of the preceding 9 patterns were presented). It turns out that 4 of these 48 multiple disorder patterns are the same as one of the two combined original input patterns, so there are only 44 truly multiple disorder patterns. The trained networks would be performing ideally if they produced two clear winning disorder units corresponding to those disorders whose manifestations were presented together.8 The standard network activated the two correspondingdisorder units in only 16 cases, and only one disorder unit in the remaining 28 cases. In addition, most of the winners in a total of 44 cases were not fully activated (see Table 4). This weak activation in the output layer of the standard network, representing the failure to generate two clear correct winners, is attributed to the large number of inhibitory connections that are present (see negative weights in Fig. 3a). When manifestations are turned on that are associated with more than one disorder, the corresponding disorder units receive strong inhibitory input as well as excitatory input. Consider, for instance, the case of disorders 1 and 16. Disorder 1 is associated with manifestations 2, 5, and 21 while disorder 16 is associated with manifestations 20, 22, 24, and 26. When all seven of these manifestations were presented, however, neither disorder 1 nor disorder 16 turned on strongly (first row of Table 4). This is because disorder 1 has inhibitory connections from manifestations 20,22,24, and 26, all of which are associated with disorder 16 (see Fig. 3a), thus sending inhibition to disorder 1. Similarly, manifestations 2, 5, and 21 which are associated with disorder 1 send inhibitory activations to disorder 16, thus resulting in a weak activation for disorder 16 (see Fig. 3a). Manifestation units not only send excitatory signal to the associated disorder units, but also send inhibitory signal to the disorder units that are not associated with them. This prevents the standard network from producing clear-cut multiple winners when multiple disorders are present. The competitive network, on the other hand, turned on both of the two associated disorder units very strongly in 38 out of the 44 cases, clearly outperforming the standard network (Table 5). Use of only excitatory connections coupled with the competitive activation mechanism enabled the network to produce multiple winners in the disorder layer

*The actual situation is more complex than this as it is possible that the union of manifestations for two disorders could correspond to the manifestations of another single disorder.


255

Table 4 Multiple-Disorder Testing with Standard NetworkQ Disorder

No.

1

116 2 16 3 16 4 16 5 16 6 16 7 16 8 16 9 16 10 16 11 16 12 16 13 16 14 16 15 16 114 2 14 3 14 4 14 5 14 6 14 7 14 8 14 9 14

.5

2

3

4

Activation Level at Each Output Unit 5 6 7 8 9 10 11 12 13

2 12 3 12 5 12 7 12 9 12 11 12 1 10 2 10 3 10 4 10 5 10

6 10 7 10 8 10 9 10

15

.4

.5

16 .6 .7 .3

.4

.6

.8 .4

.8

.5 .8 .6 .4

.7

.5 .5 .5 .5 .5

.6 .6

.4

.3 .2

.5 .6

.7 .5 .5 .4

.8

10 14 11 14 12 14 13 14

112

14

.3

.6 .6

.2

.6

.4

.2 .2

.6

.5 .9

.9 .9

.9

.5

.5

.7 .9 .6 .6

.5

.2

.8 .6

.5

'Entries less than 0.2 not shown.

when the two winners are necessary to account for the input manifestations (see Fig. 3b). Although there are some additional disorder units

Sungzoon Cho and JamesA. Reggia

256

Table 5 Multiple-Disorder Testing with Competitive Networka Disorder

No. 116 2 16 3 16 4 16 5 16 6 16 7 16 8 16 9 16 10 16 11 16 12 16 13 16 14 16 15 16 114 2 14 3 14 4 14 5 14 6 14 7 14 8 14 9 14 10 14 11 14 12 14 13 14 112 2 12 3 12 5 12 7 12 9 12 11 12 110 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 10

1 1

1

1

Activation Level at Each Output Unit 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

#Entriesless than 0.2not shown.

turned on, those "nonperfect" diagnoses are not necessarily undesirable in a clinical sense. For instance, when P,and PI,are presented, not only


257

disorders 3 and 14, but also disorder 7 is activated. This is a reasonable diagnosis given the fact that seven out of eight manifestations associated with disorder 7 are present (Peng and Reggia 1990). 4.4 Testing Partial Manifestation Patterns. A total of 29 partial manifestation patterns, each involving only one input unit (manifestation) being activated, were presented to the standard network as well as to the competitive network. The majority (19) of manifestations are associated with more than one disorder while the rest (10) of them are associated with only one disorder (manifestation 9 through 14 and 26 through 29). In the majority of cases, both the standard and the competitive network activated multiple disorder units partially, producing no clear winners but a set of alternatives. These results are similar to a "differential diagnosis'' that a human diagnostician might give. It is with the remaining 10 cases where the input manifestation can be caused by only a single disorder that the two types of networks performed differently. The standard network did not produce a clear winner in any of these 10 cases; most of the disorder units were activated rather weakly. Input from a single manifestation unit was not enough to fully activate the disorder units? The competitive network, however, turned on the respective corresponding disorder units as clear winners since it is the relative amount of input to the disorder units, not the absolute amount, that determines the result of competition. Note that even with a small difference in the input, competitive networks can produce clear winners and losers.

5 Conclusion

Competitive activation mechanisms are an alternative way of controlling the spread of activation in neural networks. They have been shown to be effective in some applications, but until now lacked an effective supervised learning method. We therefore have derived a backpropagation learning rule for use with competitive activation mechanisms and have described it in this paper. This learning rule can be viewed as a generalization of a previous form of recurrent backpropagation with normalized weights (Pineda 1987). To demonstrate that our new learning rule can work effectively, we first applied it to the task of exclusive-OR. Competitive networks with three units were successfully trained to perform this operation on their 9This was studied further by independently training and testing standard networks with normalized input patterns. Such networks produced strong winners in the corresponding single disorder units. However, when the single manifestation presented was associated with more than one disorder, the standard networks trained with normalized patterns produced strong winners in all the associated disorder units, which is an unacceptable result. Using normalized input patterns also produced poor results with multiple-disorder patterns.


258

inputs. Having hidden units was unnecessary because the two input units and one output unit could compete to produce correct behavior by the network. The second application involved locating areas of brain damage given manifestations (neurological signs and symptoms) as input. It should be noted that a diagnosis problem like this is not just a pattern classification problem where the given set of manifestations is classified into one of several categories. Rather, it also involves constructing a hypothesis that includes more than one disorder. Standard backpropagation networks have not been demonstrated to handle multiple-disorder diagnosis problems effectively. The standard backpropagation network we studied developed many inhibitory connections when trained with just prototypical cases. When manifestations of multiple disorders were subsequently presented simultaneously, these inhibitory connections prevented the network from producing appropriate diagnostic hypotheses. Competitive networks, on the other hand, performed qualitatively better in such cases. In competitive networks, the disorder units were able to "cooperate" in producing multiple winners when appropriate even though they were only trained on prototypical single-disorder cases. Competitive networks also did better with partial manifestation patterns than standard networks. In summary, we have derived and demonstrated the effectiveness of a supervised learning rule for competitive activation mechanisms. Together with an unsupervised learning rule previously developed for competitive activation mechanisms (Sutton et al. 19901, this greatly increases the range of tasks to which this approach can be applied. Acknowledgments

This work was supported by N M awards NS29414 and NS16332. Dr.Reggia is also with the Institute for Advanced Computer Studies at the University of Maryland. References Adam, R., and Victor, M. 1985. Principles of Neurology. McGraw-Hill, New

York. Almeida, L. 1987. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. Proceedings of the IEEE First Annual International Conference on Neural Nehoorks, Vol. 11, pp. 609418,San Diego, CA. Baxt, W. 1990. Use of an artificial neural network for data analysis in clinical decision making. Neural Camp. 2,480-489. Benaim, M., and Samuelides, M. 1990. Dynamical properties of neural nets using competitive activation mechanisms. Proceedings of lnternational Joint Conference on Neural Networks, Vol. III, pp. 541546, San Diego, CA.


259

Bourret, P., Goodall, S., and Samuelides, M. 1989. Optimal scheduling by competitive activation: Application to the satellite antennae scheduling problem. Proceedings of International Joint Conferenceon Neural Networks, Vol. I , pp. 565572, Washington, DC. Brazis, I?, Masdeu, J., and Biller, J. 1985. Localization in Clinical Neurology. Little, Brown, Boston. Cho, S., and Reggia, J. 1991. A recurrent error back-propagation rule for competitive activation mechanisms. Tech. Rep. CSTR-2661, Department of Computer Science, University of Maryland. Cho, S., and Reggia, J. 1992. Learning visual coordinate transformations with competition. Proceedings of International Joint Conference on Neural Networks, Baltimore, MD, Vol. IV,pp. 49-54. Fahlman, S. 1988. Faster-learningvariations on back-propagation: An empirical study. Proceedings of the 1988 Connectionist Models Summer School, pp. ,3845, Pittsburgh, PA. Hertz, J., Krogh, A., and Palmer, R. 1991. Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, CA. Hinton, G. 1989. Connectionist learning procedures. Artificial Intelligence 40, 185-234. Peng, Y.,and Reggia, J. 1990. Abductive Inference Models for Diagnostic ProblemSolving. Springer-Verlag,Berlin. Pineda, F. 1987. Generalization of back-propagation to recurrent neural networks. Phys. Rev.Lett. 59(19), 2229-2232. Reggia, J. 1985. Virtual lateral inhibition in parallel activation models of associative memory. Proceedings of the 9th International Joint Conferenceon Artificial Intelligence, Vol. 1, pp. 244-248, Los Angeles, CA. Reggia, J., D’Autrechy, C. L., Sutton, G., and Weinrich, M. 1992. A competitive distribution theory of neocortical dynamics. Neural Comp. 4, 287-317. Reggia, J., Marsland, P., and Berndt, R. 1988. Competitive dynamics in a dualroute connectionist model of print-to-sound transformation. Complex Syst. 2, 509-547. Rumelhart, D., Hinton, J., and Williams, R. 1986. Learning representations by back-propagating errors. Nature (London) 323,533-536. Rumelhart, D., and McClelland, J. 1986. On learning the past tenses of English verbs. In Parallel Distributed Processing, Vol 2: Foundations, D. Rumelhart, J. McClelland, and the PDP Research Group, eds., pp. 216-271. MIT Press, Cambridge, MA. Sutton, G., Reggia, J., and Maisog, J. 1990. Competitive learning using competitive activation rules. Proceedings of International Joint Conference on Neural Networks, Vol. II, pp. 285-291, San Diego, CA. Received 16 May 1991; accepted 26 June 1992.

This article has been cited by: 2. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef]

Communicated by Christof Koch

Constraints on Synchronizing Oscillator Networks David E. Cairns Roland J. Baddeley Leslie S. Smith Centrefor Cognitiveand Computational Neuroscience, University of Stirling, Stirling, Scotland, FK9 4LA

This paper investigates the constraints placed on some synchronized oscillator models by their underlying dynamics. Phase response graphs are used to determine the phase locking behaviors of three oscillator models. These results are compared with idealized phase response graphs for single phase and multiple phase systems. We find that all three oscillators studied are best suited to operate in a single phase system and that the requirements placed on oscillatory models for operation in a multiple phase system are not compatible with the underlying dynamics of oscillatory behavior for these types of oscillator mode1s

.

1 Introduction Following observations of oscillations and synchronization behavior in cat visual cortex (Eckhorn et al. 1989; Gray et al. 1989a) a number of interpretations have been put forward to explain these results (Gray et al. 1989b; Eckhorn et al. 1988; Grossberg and Somers 1991; Shastri 1989; Sompolinsky et al. 1990). It has been suggested that a possible interpretation of the observed synchronization behavior is that the brain could be using synchronized oscillations as a method of solving the binding problem (Von der Malsburg and Schneider 1986). If a cluster of nodes that share a common property are synchronized, they are thus labeled as belonging to one group. Other synchronized nodes that are in a different phase of an oscillatory cycle are effectively labeled as a separate group. By using this method, it can be seen that a number of different entities may be stored simultaneously, each represented by a different phase in an oscillatory cycle. A fundamental requirement behind these theories is that groups of nodes should be able to move into and remain in separate synchronized phases. A simple but effective architecture that enables synchronization to take place is lateral coupling. Lateral connections between node pairs Neural Compufufion 5,260-266 (1993) @ 1993 Massachusetts Institute of Technology

Synchronizing Oscillator Networks

261

transfer a measure of the activation state of one node to the other. This causes a change in the period of the receiving node and thereforea change in its phase. We investigate the response of three generic oscillator models to this type of effect and determine whether or not they are capable of supporting multiple phases as required by the above theories in order to perform binding. 2 Method

Three studies were performed, one for each of the oscillator models. To provide a small but general cross section, we chose one simple oscillator model and two biological models. For the simple oscillator, a leaky integrator model was chosen to illustrate the most basic phase response one can obtain from a nonlinear system (Appendix A.1). As an example of models of cellular oscillations or potential pacemaker cells, a reduced version of the Hodgkin-Huxley cell membrane model (the Morris-Lecar oscillator) was chosen [Appendix A.2 (Rinzel and Ermentrout 1989)l. At the multicellular level, an oscillator based on an original model of excitatory/inhibitory cell cluster interactions by Wilson and Cowan (1972) was used [Appendix A.3 (Wang et al. 199011. The technique for obtaining the phase response graphs was obtained from Rinzel and Ermentrout’s original study of the dynamics of the Morris-Lecar model (Rinzel and Ermentrout 1989). Each oscillator was driven by a constant input until the period of the oscillation had stabilized. This gave a base period Ab. Driving the oscillator by the constant input and starting from a point just after peak activation, trials were made for points across the phase of the oscillation. For each successive trial, the instant of delivery of an entraining signal to the oscillator was increased. Each entraining input was of a constant size and was delivered for a constant proportion (0.025) of the period of the oscillator. The entraining input caused a change in the period of the oscillator. A measure of the relative phase shift caused by the entraining input was calculated according to equation 2.1. The cumolative results of the trials allowed the production of phase response graphs for each oscillator. (2.1) where A0 is the phase shift, Ab is the normal period, and A, is the new period. 3 Discussion

The following discussion relates how the results of the study (shown in Fig. 1) compare with an idealized phase response behavior that one

262

D. E. Cairns, R. J. Baddeley, and L. S. Smith

would like in a single phase and a multiple phase system. For a single phase system where all nodes move toward a globally synchronized activation, the ideal phase response behavior can be represented by the graph in Figure 2a. An entraining signal causes the phase of a receiving node to move in the direction of the phase of the node producing the signal. The degree of phase shift is proportional to the difference between the two nodes and thus causes a steady convergence with minimal possibility of overshoot. The direction of phase shift is determined by the difference in phase, the phase shift being in the direction of the shortest "route" to synchrony. A node with this form of behavior will always attempt to synchronize with the originator of any signal and will remain unperturbed only when in synchrony. This represents an idealized behavior for a single phase system, however any system that has zero phase shift at 0 and 1 with a monotonic decrease in phase in the region 0-0.5 and a monotonic increase in phase in the region 0.5-1.0 (with a discontinuity at 0.5) will cause synchronization to occur (Niebur et al. 1991; Sompolinsky et al. 1990). An example of the phase response for an oscillator in a multiple phase system is shown in Figure 2b. The oscillator maintains the requirements for phase locking with entraining signals arriving close to the phase of a receiving node. However, if the entraining signals arrive further out of phase then no phase shift occurs. This "dead zone" allows for the coexistence of multiple phase groups where inputs arriving from each out-of-phase group do not perturb the receiving group. The above phase response behavior is atypical of most oscillatory dynamics. The frequency of a node is usually increased or decreased as a result of extra input. Only in cases where the "activation" of the node is saturated (for example, when it has reached its peak or is in a refractory period), will little phase shift occur. The extended region of low response required for multiple phases is unlikely to be present in the basic dynamics of most oscillator models. Comparing these requirements with the phase graphs of the oscillators under study, it can be seen that they are best suited to single phase/synchronized activation systems. All three models exhibit an almost linear positive phase convergence in the latter half of their phase. In the case of the two neurophysiologicallybased systems some negative movement is also observed in the first half of the phase plane. Although none of the phase responses is ideal, they are sufficient to allow all of the models to exhibit effective synchronization behavior. Conversely, the oscillators studied do not show the type of behavior necessary for a multiple phase system. They do not possess significant regions of low response to entraining input in their mid-phase region. Consequently these oscillators do not allow for separate phases to coexist stably in a system. They will always be attempting to cause global synchronization. This would favor a network where one population is synchronized against a background of incoherent activity (Sompolinsky et al. 1990; Niebur et al.

SynchronizingOscillator Networks

263

Phase Shift

' u a 4

0.2

0.6

0.4

0.8

1

Input 0.50 ._._Input 0.25

-0.2 0.2 (C)

0.4

0.6

0.E

1

Phase

Figure 1: Phase response graphs. (a) Leaky integrator model, (b) Morris-Lecar cell membrane model, and (c) Wang et al. cell cluster model. Each graph shows the change in phase that occurs when an oscillator is perturbed at a given point in its phase. The x axis gives the phase of the oscillator at the point it is perturbed and the y axis the degree of perturbation in terms of phase shift. The amount of entraining input I by which the oscillator is stimulated is given as a fraction of the driving input of the oscillator. 1991; Koch and Schuster 19921, thus allowing figure ground separation but not labeling of multiple objects by phase (Shastri 1989). 4 Conclusion

This paper indicates there is a limit to the number of stable phases one can expect a system of interacting oscillators to maintain and that this limit is low. The results give support to models that use similar oscillators to achieve low level synchronization for the purposes of coherent activation. For models that use synchronized oscillations and multiple phases


264

Figure 2: Idealized phase response graphs. (a)Single phase system; (b)multiple phase system. as a method to solve the binding problem, they show that the number of phases available is likely to be significantly less than the minimum required to perform useful computation.

Appendix A Oscillator Models A.l Leaky Integrator

where T is the tonic input (1.0) k = 0.95 E is the entraining input (0.5,0.25) O = 19.93

A.2 Morris-Lecar

dv dt = -iion(V,

+T+ E

W)

(A.2)

Synchronizing Oscillator Networks

265

where

gG, = 1.1

v is the voltage

Vl = -0.01

q5 = 0.2

w is the fraction of K+ channels open g~ = 2.0 T is the tonic input (0.28) g~ = 0.5 E is the entraining input (0.14,0.07)

~2 ~3 ~4

= 0.15 = 0.0 = 0.3

V K = -0.7 V L = -0.5

A.3 Wang et al.

dyi-- --Yi + G, dt

T,

F ( x ) = (1 - V ) X T~ = 0.9

6, = 0.4

+ Tyx;)

(-T

yyY

+ ~2

(0 I Q I 1)

T,, = 1.0

1.0 Oy = 0.6 TW = 1.9 f = 0.2 A, = 0.05 Tyx= 1.3 j = 0.2 A, = 0.05 Tw = 1.2 Q = 0.4 a! = 0.2 /3 = 0.14

T~ =

(A.9)

(A.12)

I; is the tonic input (0.3) Siis the entraining input (0.15,0.075)

Acknowledgments The authors would like to thank the members of CCCN for useful discussions in the preparation of this paper, in particular Peter Hancock and Mike Roberts for their helpful comments on the draft versions. Roland Baddeley and David Cairns are both funded by SERC and Leslie Smith is a member of staff in the Department of Computing Science and Mathematics at the University of Stirling. References Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? B i d . Cybernet. 60, 121-130. Eckhorn, R., Reitboeck, H. J., Arndt, M., and Dicke, I? 1989. Feature linking via stimulus-evoked oscillations: Experimental results from cat visual cortex

266


and functional implications from a network model. Proc. Intl. Joint Conf. Neural Networks (Washington), pp. 723-730. Gray, C. M.,Konig, P., Engel, A. K., and Singer, W. 1989a.Oscillatory responses in cat visual cortex exhibit inter-columnar synchronisation which reflects global stimulus properties. Nature (London) 338, 1698-1702. Gray, C. M.,Konig, P., Engel, A. K., and Singer, W. 1989b. Synchronisation of oscillatory ~psponsesin visual cortex: A plausible mechanism for scene segmentation. Proc. lntl. Symp. Synergetics Cognition, Vol. 45,C24,pp. 82-98. Grossberg, S., and Somers, D. 1991. Synchronised oscillations during cooperative feature linking in a cortical model of visual perception. Neural Networks 4,453-466. Koch, C., and Schuster, H. 1992. A simple network showing burst synchronization without frequency locking. Neural Comp. 4(2), 211-223. Niebur, E., Schuster, H. G., Kammen, D. M.,and Koch, C. 1991. Oscillator-phase coupling for different two-dimensional network connectivities. Phys. Rev. A 44,6895-6904. Rinzel, J.,and Ermentrout, G. B. 1989. Analysis of neural excitability and oscillations. In Methods in Neuronal Modeling-From Synapses to Networks. C. Koch and I. Segev, eds. MIT Press, Cambridge. Shastri, L. 1989. From simple associntions to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings. Tech.Rep. University of Pennsylvania. Sompolinsky, H., Golomb, D., and Kleinfeld, D. 1990. Global processing of visual stimuli in a neural network of coupled oscillators. Proc. Natl. Acad. Sci. U.S.A. 87, 7200-7204. von der Malsburg, C., and Schneider, W. 1986. A neural cocktail-partyprocessor. Biol. Cybernet. 54, 29-40. Wan& D.,Buhmann, J., and von der Malsburg, C. 1990. Pattern segmentation in associative memory. Neural Comp. 2,96106. Wilson, H.P.,and Cowan, J. D. 1972. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. 1 - 1 2 . Received 19 May 1992; accepted 3 September 1992.

This article has been cited by: 2. G. Frank, G. Hartmann, A. Jahnke, M. Schafer. 1999. An accelerator for neural networks with pulse-coded model neurons. IEEE Transactions on Neural Networks 10:3, 527-538. [CrossRef] 3. S. Campbell, DeLiang Wang. 1996. Synchronization and desynchronization in a network of locally coupled Wilson-Cowan oscillators. IEEE Transactions on Neural Networks 7:3, 541-554. [CrossRef] 4. Alfred Nischwitz, Helmut Glünder. 1995. Local lateral inhibition: a key to spike synchronization?. Biological Cybernetics 73:5, 389-400. [CrossRef]

Communicated by Ralph Liiker

Learning Mixture Models of Spatial Coherence Suzanna Becker Geoffrey E. Hinton Department of Computer Science, University of Toronto, Toronto, Ontario, Canada M5S 1A4

We have previously described an unsupervised learning procedure that discovers spatially coherent properties of the world by maximizing the information that parameters extracted from different parts of the sensory input convey about some common underlying cause. When given random dot stereogramsof curved surfaces, this procedure learns to extract surface depth because that is the property that is coherent across space. It also learns how to interpolate the depth at one location from the depths at nearby locations (Beckerand Hinton 1992b). In this paper, we propose two new models that handle surfaces with discontinuities. The first model attempts to detect cases of discontinuities and reject them. The second model develops a mixture of expert interpolators. It learns to detect the locations of discontinuities and to invoke specialized, asymmetric interpolators that do not cross the discontinuities. 1 Introduction

Standard backpropagation is implausible as a model of perceptual learning because it requires an external teacher to speclfy the desired output of the network. We have shown (Becker and Hinton 1992b) how the external teacher can be replaced by internally derived teaching signals. These signals are generated by using the assumption that different parts of the perceptual input have common causes in the external world. Small modules that look at separate but related parts of the perceptual input discover these common causes by striving to produce outputs that agree with each other (see Fig. la). The modules may look at different modalities (e.g., vision and touch), or the same modality at different times (e.g., the consecutive 2-D views of a rotating 3-D object), or even spatially adjacent parts of the same image. In previous work, we showed that when our learning procedure is applied to adjacent patches of images, it allows a neural network that has no prior knowledge of depth to discover stereo disparity in random dot stereograms of curved surfaces. A more general version of the method allows the network to discover the best way of interpolating the depth at one location from the depths at nearby locations. We first summaNeural Computation 5,267-277 (1993) @ 1993 Massachusetts Institute of Technology

268

Suzanna Becker and Geoffrey E. Hinton

rize this earlier work, and then introduce two new models that allow coherent predictions to be made in the presence of discontinuities. The first assumes a model of the world in which patterns are drawn from two possible classes: one which can be captured by a simple model of coherence, and one which is unpredictable. This allows the network to reject cases containing discontinuities. The second method allows the network to develop multiple models of coherence, by learning a mixture of depth interpolators for curved surfaces with discontinuities. Rather than rejecting cases containing discontinuities, the network develops a set of location-specific discontinuity detectors, and appropriate interpolators for each class of discontinuities. An alternative way of learning the same representation for this problem, using an unsupervised version of the competing experts algorithm described by Jacobs et al. (1991), is described in Becker and Hinton (1992a). 2 Learning Spatially Coherent Features in Images

Using a modular architecture as shown in Figure la, a network can learn to model a spatially coherent surface, by extracting mutually predictable features from neighboring image patches. The goal of the learning is to produce good agreement between the outputs of modules that receive input from neighboring patches. The simplest way to get the outputs of two modules to agree is to use the squared difference between the outputs as a cost function, and to adjust the weights in each module so as to minimize this cost. Unfortunately, this usually causes each module to produce the same constant output that is unaffected by the input to the module and therefore conveys no information about it. We would like the outputs of two modules to agree closely (i.e., to have a small expected squared difference) relative to how much they both vary as the input is varied. When this happens, the two modules must be responding to something that is common to their two inputs. In the special case when the outputs, da, db, of the two modules are scalars, a good measure of agreement is

where V is the variance over the training cases. Under the assumption that da and db are both versions of the same underlying gaussian signal that have been corrupted by independent gaussian noise, it can be shown that I is the mutual information (Shannon and Weaver 1964)between the underlying signal and the average of d, and db. By maximizing I we force the two modules to extract as pure a version as possible of the underlying common signal.

2.1 The Basic Stereo Net. We have shown how this principle can be applied to a multilayer network that learns to extract depth from ran-

Learning Mixture Models

269

max I

vu

Q Q hidden units

Figure 1: (a)Two modules that receive input from corresponding parts of stereo images. The first module receives input from stereo patch A, consisting of a horizontal strip from the left image (striped) and a corresponding strip from the right image (hatched). The second module receives input from an adjacent stereo patch B. The modules try to make their outputs, da and db, convey as much information as possible about some underlying signal (i.e., the depth) which is common to both patches. (b) The architecture of the interpolating network, consisting of multiple copies of modules like those in (a) plus a layer of interpolating units. The network tries to maximize the information that the locally extracted parameter dc and the contextually predicted parameter dc convey about some common underlying signal. We actually used 10 modules and the central 6 modules tried to maximize agreement between their outputs and contextually predicted values. We used weight averaging to constrain the interpolating function to be identical for all modules. dom dot stereograms (Becker and Hinton 1992b). Each network module received input from a patch of a left image and a corresponding patch of a right image, as shown in Figure la. Adjacent modules received input from adjacent stereo image patches, and learned to extract depth by trying to maximize agreement between their outputs. The real-valued depth (relative to the plane of fixation) of each patch of the surface gives rise to a disparity between features in the left and right images; because that disparity is the only property that is coherent across each stereo image, the output units of modules were able to learn to accurately detect relative depth. 2.2 The Interpolating Net. The basic stereo net uses a very simple model of coherence in which an underlying parameter at one location is assumed to be approximately equal to the parameter at a neighboring location. This model is fine for the depth of frontoparallel surfaces but it

270


is far from the best model of slanted or curved surfaces. Fortunately, we can use a far more general model of coherence in which the parameter at one location is assumed to be an unknown linear function of the parameters at nearby locations. The particular linear function that is appropriate can be learned by the network. We used a network of the type shown in Figure lb. The depth computed locally by a module, d,, was compared with the depth predicted by a linear combination d, of the outputs of nearby modules, and the network tried to maximize the agreement between d, and d,. The contextual prediction, d,, was produced by computing a weighted sum of the outputs of two adjacent modules on either side. The interpolating weights used in this sum, and all other weights in the network, were adjusted so as to maximize agreement between locally computed and contextually predicted depths. To speed the learning, we first trained the lower layers of the network as before, so that agreement was maximized between neighboring locally computed outputs. This made it easier to learn good interpolating weights. When the network was trained on stereograms of cubic surfaces, it learned interpolating weights of -0.147, 0.675, 0.656, -0.131 (Becker and Hinton 1992b). Given noise free estimates of local depth, the optimal linear interpolator for a cubic surface is -0.167, 0.667, 0.667, -0.167. 3 Mixture Models of Coherence

The models described above were based on the assumption of a single type of coherence in images. We assumed there was some parameter of the image that was either constant for nearby patches, or varied smoothly across space. In natural scenes, these simple models of coherence may not always hold. There may be widely varying amounts of curvature, from smooth surfaces, to highly curved spherical or cylindrical objects. There may be coherent structure at several spatial scales; for example, a rough surface like a brick wall is highly convoluted at a fine spatial scale, while at a coarser scale it is planar. And at boundaries between objects, or between different parts of the same object, there will be discontinuities in coherence. It would be better to have multiple models of coherence, which could account for a wider range of surfaces. One way to handle multiple models is to have a mixture of distributions (McLachlan and Basford 1988). In this section, we introduce a new way of employing mixture models to account for a greater variety of situations. We extend the learning procedure described in the previous section based on these models.

3.1 Throwing out Discontinuities. If the surface is continuous, the depth at one patch can be accurately predicted from the depths of two patches on either side. If, however, the training data contains cases in


271

which there are depth discontinuities (see Fig. 2) the interpolator will also try to model these cases and this will contribute considerable noise to the interpolating weights and to the depth estimates. One way of reducing this noise is to treat the discontinuity cases as outliers and to throw them out. Rather than making a hard decision about whether a case is an outlier, we make a soft decision by using a mixture model. For each training case, the network compares the locally extracted depth, d,, with the depth predicted from the nearby context, d,. It assumes that d, - d, is drawn from a zero-mean gaussian if it is a continuity case and from a uniform distribution if it is a discontinuity case, as shown in Figure 3. It can then estimate the probability of a continuity case: (3.1) where N is a gaussian, and kdiscont is a constant representing a uniform density.’ We can now optimize the average information d, and d, transmit about their common cause. We assume that no information is transmitted in discontinuity cases, so the average information depends on the probability of continuity and on the variance of d, + dc and d, - d, measured only in the continuity cases: (3.2) where Pcont = (pcont(dc - 4)). We tried several variations of this mixture approach. The network is quite good at rejecting the discontinuity cases, but this leads to only a modest improvement in the performance of the interpolator. In cases where there is a depth discontinuity between d, and db or between dd and d, the interpolator works moderately well because the weights on d, or d, are small. Because of the term Pcontin equation 3.2 there is pressure to include these cases as continuity cases, so they probably contribute noise to the interpolating weights. In the next section we show how to avoid making a forced choice between rejecting these cases or treating them just like all the other continuity cases. ‘We empiccally select a good (fixed)value of kdixont, and w,e choose a starting value of ccont(dc - d c ) (some proportion of the initial variance of d, -dJ, and gradually shrink it during learning. The learning algorithm’s performance is fairly robust with respect to variations in the choice of kdk& the main effect of changing this parameter is to sharpen or flatten the network‘s probabilistic decision function for labeling cases as continuous or discontinuous (equation 3.1). The choice of Vmnt(dc- &), on the other hand, turns out to affect the learning algorithm more critically; if this variance is too small, many cases will be treated as discontinuous, and the network may converge to very large weights which overfit only a small subset of the training cases. There is no pmblem, however, if this variance is too large initially; in this case, all patterns are h a t e d as continuous, and as the variance is shrunk during learning, some discontinuous cases are eventually detected.

272


Figure 2: (Top) A curved surface strip with a discontinuity created by fitting 2 cubic splines through randomly chosen control points, 25 pixels apart, separated by a depth discontinuity. Feature points are randomly scattered on each spline with an average of 0.22 features per pixel. (Bottom)A stereo pair of “intensity” images of the surface strip formed by taking two different projections of the feature points, filtering them through a gaussian, and sampling the filtered projections at evenly spaced sample points. The sample values in corresponding patches of the two images are used as the inputs to a module. The depth of the surface for a particular image region is directly related to the disparity between corresponding features in the left and right patch. Disparity ranges continuously from -1 to +1 image pixels. Each stereo image was 120 pixels wide and divided into 10 receptive fields 10 pixels wide and separated by 2 pixel gaps, as input for the networks shown in Figure 1. The receptive field of an interpolating unit spanned 58 image pixels, and discontinuities were randomly located a minimum of 40 pixels apart, so only rarely would more than one discontinuity lie within an interpolator’s receptive field.

3.2 Learning a Mixture of Interpolators. The presence of a depth discontinuity somewhere within a strip of five adjacent patches does not necessarily destroy the predictability of depth across these patches. It may just restrict the range over which a prediction can be made. So instead of throwing out cases that contain a discontinuity, the network could try to develop a number of different, specialized models of spatial coherence across several image patches. If, for example, there is a depth discontinuity between d, and d, in Figure lb, an extrapolator with weights of -1.0, $2.0, 0, 0 would be an appropriate predictor of d,. The network could also try to detect the locations of discontinuities, and use this information as the basis for deciding which model to apply on a given case. This information is useful not only in making clean decisions about which coherence model to apply, but it also provides valuable cues for interpreting the scene by indicating the locations of object boundaries in the image. Thus, we can use both the interpolated depth map, as well


273

A

continuity

d:

d:

Figure 3: The probability distribution of d,, P l ( d c ) , is modeled as a mixture of two distributions: a gaussian with mean = dr and small variance, and P 2 ( d c ) , a uniform distribution. Sample points for d, and d,, @ and dr are shown. In this case, $ and d t are far apart so $ is more likely to have been drawn from Pz. as the locations of depth discontinuities, in subsequent stages of scene interpretation. A network can learn to discover multiple coherence models using a set of competing interpolators. Each interpolator tries, as before, to achieve high agreement between its output and the depth extracted locally by a module. Additionally, each interpolator tries to account for as many cases as possible by maximizing the probability that its model holds. The objective function maximized by the network is the sum over models, i, of the agreement between the output of the ith model, &,, and the predicted depth, d,, weighted by the probability of the ith model:

(3.3) where the V’s represent variances given that the ith model holds. The probability that the ith model is applicable on each case a, pp, can be computed independently of how well the interpolators are doing;’ this can be done by adding extra “controller” units to the network, as shown in Figure 4, whose sole purpose is to compute the probability, pi, that each interpolator’s model holds. The weights of both the controllers and the interpolating experts can be learned simultaneously, so as to maximize *More precisely, this computed probability is conditionally independent of the interpolators’ performance on a particular case, with independence being conditioned on a fixed set of weights. As the reviewer has pointed out, when the weights change over the course of learning, there is an interdependence between the probabilities and interpolated quantities via the shared objective function.


274

controller 1 exoert 2

controller 2

controller 4 expert 5

controller 5

Figure 4 An architecturefor learning a mixture model of curved surfaces with discontinuities, consisting of a set of interpolators and discontinuity detectors. We actually used a larger modular network and equaIity constraints between the weights of correspondingunits in different modules, with 6 copies of the architectureshown here. Each copy received input from different but overlapping parts of the input. I”. By assigning a controller to each expert interpolator, each controller should learn to detect a discontinuity at a particular location (or the absence of a discontinuity in the case of the interpolator for pure continuity cases). And each interpolating unit should learn to capture the particular type of coherence that remains in the presence of a discontinuity at a particular location. The outputs of the controllers are normalized, so that they represent a probability distribution over the interpolating experts’ models. We can think of these normalized outputs as the probability with which the system selects a particular expert. Each controller’s output is a normalized exponential function of its squared total input, xi: (3.4)

Squaring the total input makes it possible for each unit to detect a depth edge at a particular location, independently of the direction of contrast change. We normalize the s uared total input in the exponential by an estimate of its variance, 6(x$ = k w,:. (This estimate of the variance of the total weighted input is exact if the unweighted individual inputs are independent, gaussian, and have equal variances of size k.) This discourages any one unit from trying to model all of the cases simply by having huge weights. The controllers get to see all five local depth estimates, da . . .d,. As before, each interpolating expert computes a h e a r function of four contextually extracted depths, d i c = Wbda +Wi& wddd wiede, in order to try to predict the centrally extracted depth d,.

cji

+

+


275

We first trained the network using the original continuous model, as described in Section 2, on a training set of 1000 images with discontinuities, until the lower layers of the network became well tuned to depth. So the interpolators were initially pretrained using the continuity model, and all the interpolators learned similar weights. We then froze the weights in the lower layers, added a small amount of noise to the interpolators’ weights (uniform in [-0.1,0.1]), and applied the mixture model to improve the interpolators and train the controller units. We ran the learning procedure for 10 runs, each run starting from different random initial weights and proceeding for 10 conjugate gradient learning iterations. The network learned similar solutions in each case. A typical set of weights on one run is shown in Figure 5. The graph on the right in this figure shows that four of the controller units are tuned to discontinuities at different locations. The weights for the first interpolator (shown in the top left) are nearly symmetrical, and the corresponding controller’s weights (shown immediately to the right) are very small; the graph on the right shows that this controller (shown as a solid line plot) mainly responds in cases when there is no discontinuity. The second interpolator (shown in the left column, second from the top) predominantly uses the leftmost three depths; the corresponding controller for this interpolator (immediatelyright of the top left interpolator’s weights) detects discontinuitiesbetween the rightmost two depths, d, and d d . Similarly, the remaining controllers detect discontinuities to the right or left of d,; each controller’s corresponding interpolator uses the depths on the opposite side of the discontinuity to predict d,. 4 Discussion

We have described two ways of modeling spatially coherent features in images of scenes with discontinuities. The first approach was to simply try to discriminate between patterns with and without discontinuities, and throw away the former. In theory, this approach is promising, as it provides a way of making the algorithm more robust against outlying data points. We then applied the idea of multiple models of coherence to a set of interpolating units, again using images of curved surfaces with discontinuities. The competing controllers in Figure 4 learned to explicitly represent which regularity applies in a particular region. The output of the controllers was used to compute a probability distribution over the various competing models of coherence. The representation learned by this network has a number of advantages. We now have a measure of the probability that there is a discontinuity that is independent of the prediction error of the interpolator. So we can tell how much to trust each interpolator’s estimate on each case. It should be possible to distinguish clear cases of discontinuities from cases that are simply noisy, by the entropy of the controllers’ outputs.

276


Figure 5 (a) Typical weights learned by the five competing interpolators and correspondingfive discontinuitydetectors. Positive weights are shown in white, and negative weights in black. (b) The mean probabilities computed by each discontinuity detector are plotted against the distance from the center of the units’ receptive field to the nearest discontinuity. The probabilistic outputs are averaged over an ensemble of lo00 test cases. If the nearest discontinuity is beyond f30 pixels, it is outside the units’ receptive field and the case is therefore a continuity example.

Furthermore, the controller outputs tell us not only that a discontinuity is present, but exactly where it lies. This information is important for segmenting scenes, and should be a useful representation for later stages of unsupervised learning. Like the raw depth estimates, the location of depth edges should exhibit coherence across space, at larger spatial scales. It should therefore be possible to apply the same algorithm recursively to the outputs of the controllers, to find object boundaries in two-dimensional stereo images. The approach presented here should be applicable to other domains that contain a mixture of alternative local regularities across space or time. For example, a rigid shape causes a linear constraint between the locations of its parts in an image, so if there are many possible shapes, there are many alternative local regularities (Zemel and Hinton 1991).


277

Acknowledgments

This research was funded by grants from NSERC and the Ontario Information Technology Research Centre. Hinton is Noranda fellow of the Canadian Institute for Advanced Research. Thanks to John Bridle and Steve Nowlan for helpful discussions.

References Becker, S., and Hinton, G. E. 1992a. Learning to make coherent predictions in domains with discontinuities. In Advances in Neural Information Processing Systems 4. Morgan Kaufmann, San Mateo, CA. Becker, S., and Hinton, G. E. 1992b. A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature (London) 355, 161163. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. 1991. Adaptive mixtures of local experts. Neural Cornp. 3(1), 79-87. McLachlan, G. J., and Basford, K. E. 1988. Mixture Models: Inference and Applications to Clustering. Marcel Dekker, New York. Shannon, C. E., and Weaver, W. 1964. The Mathematical Theory of Communication. The University of Illinois Press, Urbana, IL. Zemel, R. S., and Hinton, G. E. 1991. Discovering viewpoint-invariant relationships that characterize objects. In Advances in Neural Information Processing Systems 3, pp. 299-305. Morgan Kaufmann, San Mateo, CA.

Received 5 June 1992; accepted 26 June 1992.

This article has been cited by: 2. K. Lee, H. Buxton, J. Feng. 2005. Cue-Guided Search: A Computational Model of Selective Attention. IEEE Transactions on Neural Networks 16:4, 910-924. [CrossRef] 3. Ryotaro Kamimura, Taeko Kamimura, Thomas R. Shultz. 2001. Information Theoretic Competitive Learning and Linguistic Rule Acquisition. Transactions of the Japanese Society for Artificial Intelligence 16, 287-298. [CrossRef] 4. Suzanna Becker, Mark Plumbley. 1996. Unsupervised neural network learning procedures for feature extraction and classification. Applied Intelligence 6:3, 185-203. [CrossRef] 5. Suzanna Becker. 1996. Network: Computation in Neural Systems 7:1, 7-31. [CrossRef]

Communicated by Steve Suddarth

Hints and the VC Dimension Yaser S. Abu-Mostafa California lnstitute of Technology, Pasadena, CA 91225 USA

Learning from hints is a generalization of learning from examples that allows for a variety of information about the unknown function to be used in the learning process. In this paper, we use the VC dimension, an established tool for analyzing learning from examples, to analyze learning from hints. In particular, we show how the VC dimension is affected by the introduction of a hint. We also derive a new quantity that defines a VC dimension for the hint itself. This quantity is used to estimate the number of examples needed to "absorb" the hint. We carry out the analysis for two types of hints, invariances and catalysts. We also describe how the same method can be applied to other types of hints. 1 Introduction Learning from examples deals with an unknown function f that is represented by examples to the learning process. The process uses the examples to infer an approximate implementation off. Learning from hints (Abu-Mostafa 1990) generalizes the situation by allowing other information that we may know about f to be used in the learning process. Such information may include invariance properties, symmetries, correlated functions (Suddarth and Holden 19911, explicit rules (Winand Giles 1992),minimum-distance properties (Al-Mashouq and Reed 1991), or any other fact about f that narrows down the search. In many practical situations, we do have some prior information about f, and the proper use of this information (instead of just using "blind examples off) can make the difference between feasible and prohibitive learning. In this paper, we develop a theoretical analysis of learning from hints. The analysis is based on the VC dimension (Blumer et al. 19891, which is an established tool for analyzing learning from examples. Simply stated, the VC dimension VC(G) furnishes an upper bound for the number of examples needed by a learning process that starts with a set of hypotheses G about what f may be. The examples guide the search for a hypothesis g E G that is a good replica off. Since f is unknown to begin with, we start with a relatively big set of hypotheses G to maximize our chances of finding a good approximation off among them. However, the larger G is, the more examples off we Neural Computation 5,278-288 (1993) @ 1993 Massachusetts Institute of Technology

Hints and the VC Dimension

279

need to pinpoint the good hypothesis. This is reflected in a bigger value of V C ( G ) . How do we make G smaller without the risk of losing good approximations off? This is where the hints come in. Since a hint is a known property of f, we can use it as a litmus test to weed out bad gs thus shrinking G without losing good hypotheses. The main result of this paper is the application of the VC dimension to hints in two forms. 1. The VC dimension provides an estimate for the number of examples needed to learn f. When a hint H is given about f , the number of examples off can be reduced. This is reflected in a smaller “VC dimension given the hint’’ V C ( G I H). 2. If H itself is represented to the learning process by a set of examples,

we would like to estimate how many examples are needed to absorb the hint. This calls for a generalization of the VC dimension to cover examples of the hint as well as examples of the function, which is reflected in a “VC dimension for the hint” V C ( G ; H ) . We will study two types of hints in particular, invariances and catalysts. We will discuss how the same framework can be used to study other types of hints. A detailed account of the VC dimension can be found in (Blumer et al. (1989) and Vapnik and Chervonenkis (1971). We will provide a brief background here to make the paper self-contained. The setup for learning from examples consists of an environment X and an unknown function f : X ---t ( 0 , l ) that we wish to learn. The goal is to produce a hypothesis g : X -, (0,l) that approximates f . To do this, the learning process starts with a set of hypotheses G and tries to select a good g E G based on a number of examples [ x l , f ( x l ) ]; . . . ; [xN,f(xN)]off. To generate the examples, we assume that there is a probability distribution P ( x ) on the environment X. Each example is picked independently according to P ( x ) . The hypothesis g that results from the learning process is considered a good approximation off if the probability [wxt. P(x)l that g ( x ) # f ( x ) is small. The learning process should have a high probability of producing a good approximation off when a sufficient number of examples is provided. The VC dimension helps determine what is “sufficient.” Here is how it works. Let rg= Pr[g(x) = f ( x ) ] ,where Pr[.] denotes the probability of an event. We wish to pick a hypothesis g that has rgx 1. However, f is unknown and thus we do not know the values of these probabilities. Since f is represented by examples, we can compute the frequency of agreement between each g and f on the examples and base our choice of g on the frequencies instead of the actual probabilities. Let hypothesis g agree with f on a fraction vg of the examples. We pick a hypothesis that has vg M 1. The VC inequality asserts that the values of vgs will be close to rgs. Specifically, r

1

Yaser S. Abu-Mostafa

280

where “sup” denotes the supremum, and rn is the growth function of G. m(N)is the maximum number of different binary vectors g(x1) . - . g ( x ~ ) that can be generated by varying g over G while keeping XI, . . . ,X N E X fixed. Clearly, m(N)5 2N for all N. The VC dimension V C ( G )is defined as the smallest N for which m(N) < 2N. We assume that G has a finite VC dimension. If VC(G)= d, the growth function m(N)can be bounded bY

When this estimate is substituted in the VC inequality, the right-hand side of the inequality becomes arbitrarily small for sufficiently large N. This means that it is almost certain that each vgis approximately the same as the corresponding 7rg. This is the rationale for considering N examples sufficient to learn f . We can afford to base our choice of hypothesis on vg as calculated from the examples, because it is approximately the same as 7rg. How large N needs to be to achieve a certain degree of approximation is affected by the value of the VC dimension. In this paper, we assume that f E G. This means that G is powerful enough to implement f . We also assume that f strictly satisfies the hint H. This means that f will not be excluded as a result of taking H into consideration. Finally, we assume that everything that needs to be measurable will be measurable. 2 Invariance Hints

It is often the case that we know an invariance property of an otherwise unknown function. For example, speaker identification based on a speech waveform is invariant under time shift of the waveform. Properties such as shift invariance and scale invariance are commonplace in pattern recognition, and dozens of methods have been developed to take advantage of them (e.g., Hu 1969). Invariances have also been used in neural networks, for example, group invariance of functions (Minsky and Papert 1988) and the use of invariances in backpropagation (Abu-Mostafa 1990). An invariance hint H can be formalized by the partition

x=ux, (2

of the environment X into the invariance classes X,, where a is an index. Within each class X,, the value off is constant. In other words, x , x‘ E X, implies that f ( x ) = f (x‘) . Some invariance hints are “strong” and others are “weak,“ and this is reflected in the partition X = U, X,. The finer the partition, the weaker the hint. For instance, if each X, contains a single point, the hint is extremely weak (actually useless) since the information that x,x‘ E X,


281

implies that f ( x ) = f ( x ' ) tells us nothing new as x and x' are the same point in this case. On the other extreme, if there is a single X, that contains all the points (X, = X), the hint is extremely strong as it forces f to be constant over X (either f = 1 or f = 0). Practical hints, such as scale invariance and shift invariance, lie between these two extremes. In what follows, we will apply the VC dimension to an invariance hint H. We will start by assessing the impact of H on the original VC dimension. We will then focus on representing H by examples and address what an example of H is, how to define a VC dimension for H, and what it means to approximate H. Finally, we will discuss relations between different VC dimensions.

2.1 How the Hint Affects VC(G). The VC dimension is used to estimate the number of examples needed to learn an unknown function f . It is intuitive that, with the benefit of a hint about f, we should need fewer examples. To formalize this intuition, let the invariance hint H be given by the partition X = U, X,. Each hypothesis g E G either satisfies H or else does not satisfy it. Satisfying H means that whenever x, x' E X,, then g ( x ) = g(x'). The set of hypotheses that satisfy H is G G = {g E G I x,X' EX, + - g ( x ) = g ( x ' ) }

G is a set of hypotheses and, as such, has a VC dimension of its own.

This is the basis for defining the VC dimension of G given H VC(G 1 H ) = VC(G) Since G C_ G, it follows that VC(G I H) 5 VC(G).Nontrivial hints lead to a significant reduction from G to G, resulting in VC(G [ H) < VC(G). On the other hand, some hints may have VC(G I H) = VC(G). For instance, in the case of the weak hint we talked about, every g trivially satisfies the hint, hence G = G. VC(G 1 H ) replaces VC(G) following the "absorption" of the hint. Without the hint, VC(G)provides an estimate for the number of examples needed to learn f . With the hint, VC(G 1 H) provides a new estimate for the number of examples. This estimate is valid regardless of the mechanism for absorbing the hint, as long as it is completely absorbed. If, however, the hint is only partially absorbed (which means that some gs that do not strictly satisfy the invariance are still allowed), the effective VC dimension lies between VC(G)and VC(G I H). 2.2 Representing the Hint by Examples. What is an example of an invariance hint? If we take the hint specified by X = U, X,, an example would be " f ( x ) = f(x')," where x and x' belong to the same invariance class. In other words, an example is a pair ( x , x ' ) that belong to the same X,.


282

The motivation for representing a hint by examples is twofold. The hint needs to be incorporated in what is already a learning-from-examples process. The example f ( x ) = f(x') can be directly included in descent methods such as backpropagation along with examples of the function itself. To do this, the quantity [g(x) - g(x')I2 is minimized the same way [g(x) - f ( x ) l Zis minimized when we use an example o f f . In addition, we may represent a hint by examples if it cannot be easily expressed as a global mathematical constraint. For instance, invariance under elastic deformation of images does not readily yield an obvious constraint on the weights of a feedforward network. In contrast to the function f that is represented by a controlled number of examples and is otherwise unknown,a hint can be represented by as many examples as we wish, since it is a known property and hence can be used indefinitely to generate examples. Examples of the hint, like examples of the function, are generated according to a probability distribution. One way to generate ( x , x ' ) is to pick x from X according to the probability distribution P ( x ) , then pick x' from X, (the invariance class that contains x ) according to the conditional probability distribution P(x' 1 Xa).A sequence of N (pairs of) examples (XI, x:); (XZ, 2');.. .;( X N , YN)would be generated in the same way, independently from pair to pair. 2.3 A VC Dimension for the Hint. As we discussed in the introduction, the VC inequality is used to estimate how well f is learned. We wish to use the same inequality to estimate how well H is absorbed. To do this, we transform the situation from hints to functions. This calls for definitions of new X, P, G, and f. Let H be the invariance hint X = U,X,. The new environment is defined by

x=ux; a

(pairs of points coming from the same invariance class) with the probability distribution described above P ( x , 2)= P(x)P(x' 1 X,) where X, is the class that contains x (hence contains x'). The new set of hypotheses G, defined on the environment X, contains a hypothesis g for every hypothesis g E G such that

and the function to be "learned" is f(x,x') = 1


283

The VC dimension of the set of hypotheses G is the basis for defining a VC dimension for the hint. VC(G;H ) = VC(G) VC(G;H)depends on both G and H since G is based on G and the new environment X (which in turn depends on HI. 2.4 Approximation of the Hint. If the above learning process resulted in the hypothesis g = f (the constant 11, the corresponding g E G would obviously satisfy the hint. Learning from examples, however, results only in g that approximates f well (with high probability). The approximation is in terms of the distribution P(x,x') used to generate the examples. Thus, w.r.t. to P, Pr[g(x,x') # 11 0 as the number of examples N becomes large. Can we translate this statement into a similar one based only on the original distribution P(x)? To do this, we need to rid the statement of x'. Let --f

P r W # g(x')l = 7 By definition of g, Pr[g(x,x') # 11 is the same as Pr[g(x) # g(x')]. This implies that y 0 as N + 00. In words, if we pick x and x' at random according to P(x,x'), the probability that our hypothesis will have different values on these two points is small. To get rid of x' from this statement, we introduce hint-satisfying versions of the gs. For each g E G, let g be the best approximation of g that strictly satisfies the hint. This means that, within each invariance class X,, g(x) is constant and its value is the more probable of the two values of g(x) within X, (ties are broken either way). We will argue that P r W # g(4l I Y Since y + 0, this statement [which is solely based on P ( x ) ] implies that "g approximately satisfies the hint" in a more natural way. Here is the argument. Let q be the probability that g(x) # g(x). Given X,, let be the conditional probability that g(x) # g ( x ) , and let ya be the conditional probability that g(x) # g(x'). From the definition of g, q, must be I (otherwise, the value of g in X, should be flipped). Within each X,, since g is constant, g(x) # g(x') if, and only if, g agrees with g on either x or x' and disagrees on the other. This means that

+,

1

Ya = 2%(1 -%)

2 ?a (since 1 - q, 2 f). This is true for every class X,. Averaging over a, we get y 2 9, hence Prk(x) #Wl = 3 5 7 4

0

This establishes the more natural notion of approximating the hint.


284

2.5 A Bound on VC(G;H). As in the case of the set G and its growth function m ( N ) , the VC dimension V C ( G ; H )= V C ( G )is defined based on the growth function m(N)of the set G. m(N) is the maximum number of patterns of 1s and 0s that can be obtained by applying the g’s to (fixed but arbitrary) N examples (x,, x i ) ; (XZ, x i ) ; . . .;( X N , xh). VC(G;H ) is the smallest N for which m(N)< 2N. The value of V C ( G ; H )will differ from hint to hint. Consider our two extreme examples of weak and strong hints. The weak hint has VC(G;H ) as small as 1 since each g always agrees with each example of the hint [hence every g is the constant 1, and m(N)= 1 for all NI. The strong hint has V C ( G ; H )as large as it can be. How large is that? In Fyfe (19921, it is shown that for any invariance hint H ,

V C ( G ; H )< X V C ( G ) where X = 4.54. The argument goes as follows. For each pattern generated by the g’s on x1, x{i x2i x;i . . . X N , there is at most one distinct pattern generated by the g’s on 9

(XI 1

4

1; (x2,d);.. ‘ ;(XN,xk)

because g ( X n 1 xb) is uniquely determined by g ( x n ) and g(x‘,). Therefore,

m(N)5 m(2N) If V C ( G )= d, we can use Chernoff bounds (Feller 1968)to estimate m(2N) for N 2 d as follows

< - 2U(d/Wx2N where X ( 8 ) = -0 log, B - (1- 0 ) log,( 1- 6 ) is the binary entropy function. Therefore, once X(dI2N) 5 m(N) will be less than 2N and N must have reached, or exceeded, the VC dimension of G. This happens at N l d x 4.54. In many cases, the relationship between V C ( G I H ) and V C ( G ; H ) can be roughly stated as follows: the smaller one is, the bigger the other is. Strong hints generally result in a small value of VC(G I H) and a large value of V C ( G ; H ) ,while weak hints result in the opposite situation [the loose similarity with the average mutual information I(X;Y) and the conditional entropy H(X 1 Y) in information theory is the reason for choosing this notation for the various VC dimensions]. This relationship between V C ( G I H ) and V C ( G ; H )may suggest that we do not save when we use examples of a hint and, as a result, use fewer examples of the function. However, it should be noted that examples of the hint can be generated at will, while examples of the function may be limited in number or expensive to generate.

it


285

3 Catalyst Hints

Catalyst hints (Suddarth and Holden 1991) were introduced as a means of improving the learning behavior of feedfonvard networks. The idea is illustrated in Figure 1. A network attempting to learn the function g = f is augmented by a catalyst neuron out of the last hidden layer. This neuron is trained to learn a related function g' = f'. In doing so, the hidden layers of the network are influenced in a way that helps the main learning task g = f . After the learning phase is completed, the catalyst neuron is removed. The catalyst function f' is typically a "well-behaved version" of f that can be learned more easily and more quickly. When f' is learned, the internal representations in the hidden layers of the network will be suited for the implementation of the main function f. As a hint, namely a piece of information about f l the catalyst is the assertion that there is a way to set the weights of the network that simultaneously implements g = f and g' = f'. Unlike invariances, catalysts are very particular to the network we use. To formalize the catalyst hint, let 0 be the set of pairs of hypotheses (g,g') that can be simultaneously implemented by the network (when the catalyst neuron is present). The values of the weights in the different

Figure 1: A network that uses a catalyst hint.


286

layers of the network determine (g,g‘). A particular g may appear in different pairs (8, g‘) and, similarly, a particular g‘ may appear in different pairs (g,g‘). Since the catalyst hint puts a condition on g‘, its impact on g is indirect through these pairings of (g,g‘). This suggests the following notation: (g,8’) denotes the hypothesis g when the catalyst hypothesis is g‘ and (g,g’) denotes the hypothesis g‘ when the main hypothesis is g. Applied to a point x E X, we use the convention (g,gO(x) = g(x) (g,g’)(x) = g’(4

Thus (g,g‘) and (g,8’) provide an inflated notation for the hypotheses g and g‘, respectively. In these terms, the set of hypotheses G is defined by G = {(g,g’) I (g,g’) E 8)

To apply the VC dimension to catalyst hints, we will follow the same steps we used for invariance hints. The catalyst hint H is given by the constraint g‘ = f’. When H is absorbed, G is reduced to G

Obviously, G E G. The VC dimension of G given H is VC(G I H ) = VC(G)

Again, VC(G I H) 5 VC(G). How small VC(G I H) will be depends on the catalyst function f’. For instance, the degenerate case of a constant f’ results in VC(G I H) = VC(G) since the constant can be implemented by the catalyst neuron alone and would not impose any constraint on the weights of the original network. On the other hand, a complex f’ will take specific combinations of weights to implement, thus significantly restricting the network and resulting in VC(G I H) VC(G). If the hint is only partially absorbed, the effective VC dimension lies between VC(G) and VC(G I H). One situation that leads to partial absorption is when the hint is represented by examples. An example of the hint H: g‘ = f’ takes the form g’(x) = f’(x). In this case, examples of H are of the same nature as examples off; x is picked from X according to P ( x ) and f ’ ( x ) is evaluated. The definition of examples of H leads to the definition of G, the set of agreement/disagreement patterns between the hypotheses and the hint. For each hypothesis (g,g’) E G, there is a hypothesis g E G such that


287

The VC dimension of G is the basis for defining VC(G;H), the VC dimension that will indicate how many examples [x,f’(x)] are needed to absorb H. It is given by

VC(G;H)= V C ( G ) Unlike an invariance hint, the particular choice of a catalyst hint (the function f’) does not affect the value of VC(G;H). The VC inequality asserts that a sufficient number of examples will lead to a hypothesis (g,g‘) that satisfies

W(g, g W ) # fW1

+

0

where the probability is taken wxt. P ( x ) . Therefore, we will get a hypothesis g that pairs up with a good approximation off’. This establishes a natural notion of approximating the hint. 4 Conclusion

We have analyzed two different types of hints, invariances and catalysts. The highlight of the analysis is the definition of VC(G I H ) and VC(G;H). These two quantities extend the VC inequality to cover learning f given the hint, and learning the hint itself. Other types of hints can be quite different from invariances and catalysts, and will require new analysis. However, the common method for dealing with any type of hint in this framework is as follows. 1. The definition of the hint should determine for each hypothesis in G whether or not it satisfies the hint. The set G contains those hypotheses which do satisfy the hint. V C ( G 1 H) is defined as VC(G). 2. A scheme for representing the hint by examples should be selected.

Each example is generated according to a probability distribution P that depends on the original distribution P. Different examples are generated independently according to the same distribution. 3. For every hypothesis and every example of the hint, we should be able to determine whether or not the hypothesis agrees with the example. The agreement/disagreement patterns define the set of hypotheses G,and V C ( G ) defines VC(G;H). A hypothesis will agree with every possible example if, and only if, it satisfies the hint. 4.

How well a hypothesis approximates the hint is measured by the probability (w.r.t. P) that it will agree with a new example. An approximation in this sense should imply a partial absorption of the hint.

288


5. How the hint is represented by examples may not be unique. The choice of representation affects the definition of VC(G;H) and also affects what partial absorption means. A minimum consistency requirement is that no hypothesis that strictly satisfies the hint should be excluded as a result of the partial absorption process. A good process will exclude as many hypotheses as possible without violating this requirement.

Our analysis here dealt with the situation where the unknown function f strictly satisfies the hint, and strictly belongs to G. Relaxing these conditions is worth further investigation. It is also worthwhile to extend this work to cover real-valued functions, as well as average-case measures instead of the worst-case VC dimension. Finally, schedules for mixing examples off with examples of the hint in learning protocols are worth exploring.

Acknowledgment This work was supported by AFOSR Grant 92-J-0398 and the FeynmanHughes fellowship. The author wishes to thank Dr.Demetri Psaltis for a number of useful comments.

References Abu-Mostafa, Y. 1990. Learning from hints in neural networks. 1. Complex. 6, 192-1 98. Al-Mashouq, K.,and Reed, I. 1991. Including hints in training neural networks. Neural Comp. 3, 418427. Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. 1989. Learnability and the Vapnik-Chervonenkisdimension. 1.ACM 36,929-965. Feller, W. 1968. An Introduction to Probability Theory and Its Applications, Vol. 2. Wiley, New York. Fyfe, A. 1992. Invariance hints and the VC dimension. Ph.D. Thesis, Caltech. Hu, M. 1962. Visual pattern recognition by moment invariants. IRE Trans. Inform. Theory IT-8, 179-187. Minsky, M.,and Papert, S. 1988. Perceptrons, expanded edition. MIT Press, Cambridge, MA. Omlin, C., and Giles, C. 1992. Training second-order recurrent neural networks using hints. In Machine Learning: Proceedings of the Ninth lnternational Conference (ML-92), D. Sleeman and P. Edwards, eds. Morgan Kaufmann, San Mateo, CA. Suddarth, S., and Holden, A. 1991. Symbolic neural systems and the use of hints for developing complex systems. Intl. 1.Machine Stud. 35,291. Vapnik, V., and Chervonenkis, A. 1971.On the uniform convergence of relative frequencies of events to their probabilities. Theory Probabil.Awl. 16,264-280. Received 21 April 1992; accepted 15 July 1992.

This article has been cited by: 2. Eshel Faraggi, Bin Xue, Yaoqi Zhou. 2009. Improving the prediction accuracy of residue solvent accessibility and real-value backbone torsion angles of proteins by guided-learning through a two-layer neural network. Proteins: Structure, Function, and Bioinformatics 74:4, 847-856. [CrossRef] 3. J. Ghosn, Yoshua Bengio. 2003. Bias learning, knowledge sharing. IEEE Transactions on Neural Networks 14:4, 748-765. [CrossRef] 4. E.A. Rietman, S.A. Whitlock, M. Beachy, A. Roy, T.L. Willingham. 2001. A system model for feedback control and analysis of yield: A multistep process model of effective gate length, poly line width, and IV parameters. IEEE Transactions on Semiconductor Manufacturing 14:1, 32-47. [CrossRef] 5. P. Niyogi, F. Girosi, T. Poggio. 1998. Incorporating prior information in machine learning by creating virtual examples. Proceedings of the IEEE 86:11, 2196-2209. [CrossRef] 6. E.A. Rietman, D.J. Friedman, E.R. Lory. 1997. Pre-production results demonstrating multiple-system models for yield analysis. IEEE Transactions on Semiconductor Manufacturing 10:4, 469-481. [CrossRef] 7. Yaser S. Abu-Mostafa . 1995. HintsHints. Neural Computation 7:4, 639-671. [Abstract] [PDF] [PDF Plus] 8. Vladimir Vapnik , Esther Levin , Yann Le Cun . 1994. Measuring the VC-Dimension of a Learning MachineMeasuring the VC-Dimension of a Learning Machine. Neural Computation 6:5, 851-876. [Abstract] [PDF] [PDF Plus]

Communicated by Ralph Linsker

Redundancy Reduction as a Strategy for Unsupervised Learning A. Norman Redlich The Rockefeller University,2230 York Ave., New York, NY 10021 USA A redundancy reduction strategy, which can be applied in stages, is proposed as a way to learn as efficiently as possible the statistical properties of an ensemble of sensory messages. The method works best for inputs consisting of strongly correlated groups, that is features, with weaker statistical dependence between different features. This is the case for localized objects in an image or for words in a text. A local feature measure determining how much a single feature reduces the total redundancy is derived which turns out to depend only on the probability of the feature and of its components, but not on the statistical properties of any other features. The locality of this measure makes it ideal as the basis for a ”neural” implementation of redundancy reduction, and an example of a very simple non-Hebbian algorithm is given. The effect of noise on learning redundancy is also discussed. 1 Introduction Given sensory messages, for example, the visual images available at the photoreceptors, animals must idenhfy those objects or scenes that have some value to them. This problem, however, can be very tricky since the image data (e.g., photoreceptor signals) may underdetermine the scene data (e.g., surface reflectances) needed to find and idenhfy objects (Kersten 1990). In the case of very primitive organisms crude special purpose filters may suffice, such as the “fly detector” in frogs. But for more general object detection and for the reconstruction of physical scenes from noisy image data, some additional clues or constraints are needed. One type of clue is knowledge of the statistical properties of scenes and images (Attneave 1954, Barlow 1961, 1989). Such information can be used to recover physical scene data from noisy image data, as shown for example by Geman and Geman (1984). Barlow (1989) has also argued that such information is necessary for object recognition, since it allows objects to be discriminated from irrelevant background data. Also, since objects are encoded redundantly in sensory messages, knowing this redundancy can aid in their recognition. But how can an organism go about learning the statistical properties of sensory messages? And second, what is the most efficient way of sforNeural Computation 5,289304 (1993) @ 1993 Massachusetts Institute of Technology

290

A. Norman Redlich

ing this statistical knowledge? The enormity of these problems becomes obvious when one considers just how many numbers in principle must be learned and stored. In vision this amounts to storing the probability of every possible set of pixel values in both space and time. For a conservative estimate of this number for humans, assume one million cones sampled in space only-temporal sampling would add considerably to this. Then assume a grey scale of roughly 100, which is less than the number of contrast units that can be discriminated in bright light-and ignores luminance data. This gives 100’@“)~ooo v b l e images whose probabilities could not possibly be stored as 100 wo*wo numbers in the brain, which has no more than 1OI6 synapses. However, there are two very important properties of images which allow this number to be decreased enormously. The first and most obvious is noise: most images differ from each other only by noise or by symmetries, so there is no need to learn and store their individual probabilities. The second s i m p w n g property is that sensory message probabilities can often be derived from a far smaller set of numbers. This is the case when the set of probabilities P(Z) for the images I = ( 1 1 , h,l 3 , . . .l,}, with pixel values I,, can be factorized into a far smaller set of statistically independent probabilities for the subimages ( 1 1 , Izr4,. . .I,} as P ( I ) = P(Il)P(12)P(13). . .P(Zm). Thus, as Barlow (1989) has emphasized, the most efficient way to store the probabilities P(Z) would be to find a transformation, a factorial code, from the pixel representation I, to the statistically independent representation I, with smallest m. It can be demonstrated (Atick and Redlich 1990a) that this explains one purpose of the retinal transfer function, which approximately removes (second-order) statistical dependence from the optic nerve outputs { I , , Zzr1 3 1 . . .Im}. Finding a transformation to a factorial representation is an unsupervised learning problem that typically requires many learning stages. At each stage I assume that only the local probabilities P(I,) are measured, but as statistical independence is “increased, products of these give better approximations to the joint probabilities P ( I ) . To quantdy just how statistically independent a representation is at each stage, it is necessary to define a global learning measure L,which should be a function only of the local probabilities P(l,) (global denotes a property of the entire representation at each stage). Such a measure is defined here based on the redundancy,’ a quantity that is minimal only when the code is factorial. Learning a redundancy reducing transformation at each stage can be very difficult and may depend on the nature of the redundancy at that stage. In the retina, the greatest source of redundancy is due to ‘I use “redundancy reduction” to refer to statistical dependence between pixels.

This is not strictly speaking the only source of redundancy, which also can come from the uneven probability distribution of grayscale values. Nevertheless, I use the term “redundancy reduction” because it im lies an information preserving transformation (unlike, e.g., “entropy reduction”) ancfalso because the word “redundancy” has an intuitive appeal. The precise meaning of redundancy reduction here is a transformation which increases the learning measure L,to be defined in Section 2.

Redundancy Reduction Strategy

291

multiscale second-order correlations between pixels-corresponding to scenes being very smooth over large spatial and temporal regions-and this redundancy can be removed easily through a linear filter (Atick and Redlich 1991). But this is the exception, since in general quite complicated nonlinear coding is required (for some progress see, e.g., Barlow and Foldiak 1989; Redlich 1992). However, there is one common type of nonlinear redundancy reduction that is relatively straightforward to learn. This is the redundancy in images coming from strong correlations within sharply delineated features which are in turn weakly correlated with each other (the features can be spatially extended as long as they decouple from other parts of the image). The procedure for factorizing in this case is to look first for the subfeatures that are most tightly bound, and therefore are responsible for the most redundancy. These may then be pieced together in stages, until eventually a statistically independent set is found. What makes this much simpler than I expected is the existence of a completely local measure of how much an individual feature (or subfeature) contibutes to the global redundancy. By local I mean that this measure is only a function of the probabilities of the feature and its components, but not of the probabilities of any other feature or components. The locality of this feature measure also allows simple implementation of redundancy reduction through unsupervised ”neural” learning algorithms. One such non-Hebbian algorithm will be discussed here, and compared to some other unsupervised algorithms (von der Malsburg 1973; Bienenstock et al. 1982; Hinton and Pearlmutter 1986). The closest connection is with Hinton and Pearlmutter’s algorithm because their single-unit feature measure is mathematically related to the one here, though this is manifest only in a particular approximation. This connection is not surprising since (see,e.g., Hinton and Sejnowski 1983) their aim was also to learn statistical regularities. Some of the major distinctions between this work and theirs are the focus here on efficiency of storage and learning (on statistical independence) and also the insistence here on transformations which preserve information (see Section 6). To demonstrate the power of the present approach, I apply it to strip the redundancy from English text-to learn the text’s statistical properties. This example is used because we all know a fairly good solution to the problem: transform from the letter representation to the word representation. Of course to make the problem sufficiently difficult all clues to the solution, such as spaces between words, punctuation, and capitalization, are first eliminated from the text. The algorithm eventually segments the text just as desired into words and tightly bound groups of words. Although it is not the main purpose of this paper, I shall also indicate how useful the algorithm can be for recovering messages from noisy signals. This works best when the useful information is coded redundantly while the noise is random. It then turns out that the algorithm used here

A. Norman Redlich

292

finds only the useful portion of the input, and this will be demonstrated using noisy text. Finally, I should emphasize that my aim is not to find redundancy in language or to claim that words are learned or stored in the brain as found here. Instead, my ultimate motivation is to find an environmentally driven, self-organizing principle for the processing of visual images (and other sensory signals) to facilitate object or pattern identification (see Redlich 1992). So by "words" here I always wish to imply visual features, with letter positions in a text corresponding to pixel locations in an image and particular letters corresponding to image grayscale (or color) values. The next step of applying the algorithms derived here to visual images will appear in future papers.

2 Global Learning Measure C

Taking the English text example, the input consists of an undifferentiated stream of "pixel" values L = {a, b, c, . . .} (as in Fig. la) and the goal is to learn the probability functions P(Z) = P(ll ,l p , /J, . . .l,), with the subscript n denoting the position of letter 1, in the text. In practice, the aim is to learn P ( I ) for string length n roughly equal to the correlation length of the system. But even for n as small as 12 this in principle requires storing and updating as many as 2612 numbers. To find P(Z) more efficiently, at each stage letters will be grouped together into "words," which at first will be only pieces of real English words. Then at successive stages the new set of words W = {w~, w2,w3,. . .wm} will be built by combining the previous "words" into larger ones, among which will be real English words and also tightly correlated groups of real words (from now on quotes around words are dropped). At the very first stage P ( I ) = P(ll,l 2 , l 3 , . . . In) is very poorly approximated by the product of letter probabilities P ( l 1 ) P ( l ~ ) P ( l 3. .) .P(ln), but as the redundancy is reduced, products of word probabilities P ( w I ) P ( w $ ( w ~ ) . . . P ( W m ) give better and better approximations to P(1). To quantitatively measure how well P(I) is known at each stage, we can use information theory (see also Barlow 1961) to define a global learning measure (2.1) where HLis the entropy in the original letter code,

HL = -

cPI log(P1) IEL

(2.2)


293

and Hw/S is the word entropy per letter at a particular stage in learning:

Hw

= -

c

Pwlog(Pw)

WEW

s

=

CP,SW

(2.3)

with 3 the average of the word lengths sw.

a~iccwasheginningtogetverytiredofsittinghyhersisteron thehankandofhavingnothingtodoonceortw iceshehadpeep edintothehookhersisterwasreadingbutithadnopi~tures~r conversationsinitandwhatistheuseofahookthoughtalicew ithoutpicturesorconversationssoshewasconsider inginhe r o w n m i n d a s w e l l a s s h e c o u l d f o r t h a h o t d a y m ad e h e r f e e l v e r y sleepyandstupidwhetherthepleasureofmakingada isychai nwouldheworthlhetrouhleofgettingupandpickinglhedaini eswhensuddenlyawhiterahhitwithpinkeyesranclosehyher t h e r e w a s n o t h i n g s o v e r y r e m a r k a b l e i n t h a t n o r d i d a l i c e thi n k i Is o v e r y m u c h o u t o f I h e w a y t o h e a r t h e ra h h i t s a y to i t s e If o hdearohdear (a)

alice was b e g in n ing t o g e t very I i r e d o f s i t t ing b y her s i s t e r 0 n the h a n k and o f h a v ing n o th ing t o d o o n c e o r t w i c e she h a d p e e p e d in t o the b o o k her s is I e r was r e a d ing hut i th a d n o p i c t u r e s o r c o n ver s a t i o n s in i t and w h a t i s the u s e o f a h o o k th ough t alice with ou t p i c t u r e s fl r c o n vet s a t i o n s s o she was c o n s i d e ring in hero w n m in d a s w e I I a s she c ould for the h o I d a y m a d e her f e e I very s I e e p y and s t u p i d w he the r the p 1 e a s u r e o f m a k ing a d a i s y c h a in w ould h e w o r th the t r ou h I e o f g e t t ing u p and p i c k ins the d a i s i e s w h e n s u d d e n 1 y a w h i t e r a h h i t with p in k e y e s r a n c I o s e h y her the r e was n o th ing s o very r e m a r k a h I e in that n o r d i dalic e th ink i t s o very m u c h ou t o fthe w a y t o he a r the r a h h i t s a y t o i t s e lfohdearohdear

(h) alice was beg in n ing to g e t very t i r ed of s it 1 ing h y her s i s t e r on the b a n k and of ha v ing n o thins to d o on c e o r I w i c e shehad p e e p ed in to the h ook her s i s t e r was r e a d ing hut i th a d n o p i c t u r e s o r c o n ver s a t i on s in it and what i s the u s e of a h ook thought alice with ou t p i c I u r e s o r c on ver s a t i on s s o she was c on s i d e r ing in her o w n m in d a s w e 1 I a s shecould for the h o t d a y m a d e her f e e I very s I e e p y and s t u p i d w he thu the p I e a s u r e of m a k ing a d a i s y ch a in would he w o r th the t r ou b 1 e of g e t t ing u p and p i c k ing the d a i s i e I when s u d d e n ly a whi t e r a h h it with p i n k e y e s r a n c I0 s e h y her her e was n o thing s o very r e m a r k a h I e in that n o r d'i dalic e think it s o very m uch ou t ofthe way to he a r ther a h h its a y to it s elf o h d e a r o h d e a r (C)

Figure 1: A small sample of the text with all clues to the redundancy removed. In (a) single letters are treated as words, indicated by the spaces between them, and the entropy is HL = 4.16 bits. As the redundancy is reduced letters are combined into words, indicated by removing spaces between them. Only some of the redundancy reduction stages are shown in (b)-(fl, with the entropy per letter reduced to Hw/S = 3.75, 3.46, 2.84, 2.51, and 2.35 bits, respectively (the real word entropy per letter is Hw/S = 2.17 bits). Continued.

A. Norman Redlich

294

alice was heg in n ing toge I very t ir ed of s it t ing hy her s is ter on the h an k and of having nothing to do on ce on w ice shehad p e e p ed intothe h ook her s is ter was read ing hut i thad no p i c tur e s or conversation s in it and what is the us e of a h ook thoughtalice without p i c tur e s or conversation s s o shewas con side r ing in her own m ind as well asshecould for the h o t day made her fee I very s I e e p y and s t up id w he ther the pleas ure of mak ing ad a is y ch a in would he wor th the t r ou hle of getting up and p i ck ing the d a is i e s when suddenly a whiterahhit with p in key e s r an close by her therewas nothing s o very re mark a hle in that no r d i dalic e think it Y o very much outofthe way to hear therahhit say to it self ohdear ohdear

(d) alice was heginn ing toget verytiredof s it I ing hy her s is tcr on the hank and of having nothing to do on ce ortwice shehad peeped intothe hook her s is ter was read ing hut ithad no picture s or conversation s in it and what is the us e of a hook thoughtalice without picture s or conversation s so shewas consider ing in her own mind aswell asshecould for the hot day made her fee I very sleep y and stupid whether the pleas ure of making ad a is y ch a in wouldhe wor th the t rouhle of getting up and p ick ing the d a is i e s when suddenly a whiterahhit with p in key e s r an close hy her therewas nothing so very remark a hle in that no r d i dalic e think it so very much outofthe way to hear therahhit say to it self ohdear ohdear (a)

alice was heginning toget verytiredof sitting hy her s is ter onthehank and of having nothingtodo onceortwice shehad peeped intothe hook her s is ter was read ing hut ithad no picturesor conversation s in it and what is the us e of a hook thoughtalice without picturesor conversation s so shewas consider ing in her own mind aswcll asshecould for the hot Jay made her feelvery sleepyand stupid whether the pleas ure of making ad a is y ch a in wouldhe wor th the t rouhle of getting up and p ick ing the d a is i e s when suddenly a whiterahhit with p in key e s r an close hy her therewas nothing so very remark able in that no r d i dalic e think it so very much outofthe way to hear therahhit say toitself ohdear ohdear

(f )

Figure 1: Continued.

Initially, the set of words W is also the set of letters L so Hw/S = HLand C = 0, indicating that no learning has occurred. At the other extreme the word code is factorial for which2Hw/S = H, where H is the total entropy per letter of the text: (2.4)

[It is well known (Shannon and Weaver 1949) that H w / s 5 H with equality only when the words w in W become completely independent.] So 2This is true because the word code is reversible, .so H is invariant; for another type of reversible code see Redlich (1992).


295

the learning measure C starts out equal to zero and grows as redundancy is reduced until it approaches its maximum value

where Rc is the total redundancy in the text due to correlations between letters. If there are no correlations between letters then H = H L and Rc = 0. It is important to note that although C is bounded from above by Rc (Hw/S is bounded from below by H ) , it can go negative, so the system can in effect unlearn or increase redundancy. This happens when words (or letters) at one stage which are already independent of each other are mistakenly combined into new words. 3 Local Feature Measure F

Now that we have a global learning measure C, how do we go about finding the word/letter combinations W -+ W’, which increase C? For this purpose it is useful to have a local measure of how much an individual new word or feature increases L. Such a local feature measure F can be derived directly from C by calculating the change in C caused by including in W + W’a single new feature. Actually, since increasing C corresponds to decreasing Hw/S, we need to calculate the change in Hw/S. For extra clarity, let us first calculate the change in Hw/S when only two words in Ware combined to form a new word. Assume for simplicity that the current word set W still contains many single letters, including the letters 9’’ and ”n.” Let us see, as an example, how combining these letters into the particular word w = “in” changes Hw/S in 2.3. Following

w+

W’

where Pi,,, Pi, and P , denote the probabilities of the example word “in,” and of the letters ’?“ and “n.“ Also the “in“ terms have been separated out, so the sum C still runs over the old set W (assuming “i” and/or “n” still exist as independent elements in the set W’). To calculate the change HL/S’ - Hw/S, the new probabilities P‘ must be expressed in terms of the old probabilities P. This is easily accomplished using Pw = N w / N , where Nw = number of times word w appears, and N = total word count for the text (later N can be taken to infinity). After combining ”i” and “n” into “in,” the number N -+ N’= N - Nin, since every time the word “in” occurs it is counted as one word in W’, but was

A. Norman Redlich

296

counted as two words in W. Likewise, Ni Therefore,

-+

Ni - Nin,and N ,

-+

N , - Ni,.

Substituting these P’ into 3.1 gives

=

s/(1 -Pin)

(3.3)

so that

which defines the feature measure 3

The original average word length 3 has not been included in the definition of 3 because it is the same for all new features built out of W. As promised, this feature measure depends only on the local data Pi,, Pi, and P,. The local measure 3 can also be derived in general for new words of any length. We need to take into account the number m of old words making up the new word, as well as the number of times, m,, each old word w appears. For example if the new word “trees” is built out of the old words ”tr,” “e,” and “s” in W,then m = 4, while mt,= 1, m, = 2, m, = 1 giving Em, = m. The new probabilities P can then be derived from Pw and m, using counting arguments only slightly more complicated than before. Thus, denoting the new word by f, for feature,


297

N -t N‘ = N - (m - l)Nf, while N, + N , - m,Nf for w in the set W, of old words in f , and N , + N , otherwise. With these adjustments, the general feature measure defined by 3.4 is

F

=

F(P,,P, w E W,)

I

-[1 - (m- l)P,] log[l - (m - l)Pf]

(3.6)

This reduces to 3.5 in the special case f = “in,” W,= {”i ” ”n”) , m = 2 , mi = 1, m, = 1. To gain some intuition into just what statistical properties are measured by F it is useful to approximate 3 for the case Pf > n P w . This term plus one is the mutual information (Shannon and Weaver 19491, which measures how strongly the components that make up the feature? are correlated. A good example in English is “qu” which has high mutual information because “q” always predicts “u” so the “u’’ following “q” is redundant. The second requirement is that the feature be relatively frequent since Pf multiplies the mutual information. Otherwise, the feature could be highly self-correlated, but not common enough to significantly reduce the global redundancy. This is very important, since the mutual infor31n physics language, the feature is analogous to a bound state like an atom built out of protons and electrons. The mutual information is then proportional to the difference between the bound state (feature)energy and the sum of the energies of its components. This is the amount of energy that is gained by building the bound state (atom).

298

A. Norman Redlich

mation alone tends to favor very rare features composed of very rare elements. On the other hand, large P, alone is a dangerous criterion since there are many common features with small or even negative mutual information. Including these in the new set W actually increases the redundancy, since it effectively creates a correlated structure out of already statistically independent elements. In English text an example of a redundancy increasing feature is ”tte,” built out of “t” and “e.” 4 Experimental Results

F can be applied to devise a redundancy reduction algorithm for English text. One simple strategy would be to find at each stage the single new word which has the largest F,and thus find W from W. However, in practice it turns out that a far more time efficient approach is to find the set of new words with largest F,say 10 to 100 new words at each step. Another computational efficiency is gained by limiting the number m of component words to some small number, such as three or four. It turns out that for English text-likely also for many other ensembles-using only up to third-order correlations at each stage (m = 2,3)is sufficient, since larger words are most often composed of redundant subwords. To experimentally test how well this works I applied it to learning about 25 pages of a well known children’s book (Carroll 1865), chosen for its moderately sized vocabulary of roughly 1700 (real English) words. After eliminating all punctuation, capitals, and word spaces, the excerpt contained approximately48,000 characters. The letter entropy was found to be HL= 4.16 bits, while the entropy per letter for real English words is 2.17 bits. Figure l a shows a small piece of the text after it was stripped of any redundancy clues. Spaces are used between letters to indicate that they are being treated here as separate “words.” Figure lb-f then show the text sample in various stages of redundancy reduction. At each stage, when new words are built the spaces between their component words are eliminated. Figure 1 shows the results of using only second- and third-order joint probabilities at each step to find roughly 10 to 100 new words per stage. About 20 such stages were required to get the redundancy down to the Hw = 2.35 bits of Figure l e (only 5 of the 20 stages are actually shown in the figure) which is close to the real word entropy. Even computing all second- and third-order joint probabilities, these results represent only a few hours computation on a Macintosh computer. But the computation time and array storage needed can be reduced even further by calculating the joint probabilities only for a sample of possible new words, as will be discussed in the next section. Figure 2 shows the improvement possible using up to fourth-order probabilities; only the last stage is shown in the figure. Since there is only a small improvement over the third-order result, this demonstrates that fourth-order is not absolutely necessary.


299

alice was heginning toget verytiredof sitting hy hersister onthehan k and of having nothingtodo onceortwice shehad peeped intothe hook hersister was read ing hut ithad no pictures or conversation s in it and what is theuseof a hook thoughtalice without pictures or conversation s so shewas consider ing in her ow n mind aswell asshecould for the hot day made her feel very sleepy and stupid whether the plea sure of making ad a is y ch a in wouldhe wor th the trouhle of gettinypand p i c king the d a is ies when suddenly a whiterahhit with p in k eyes r an close hy her therewas nothing so very remark ahle in that n or did alice think it so very much outoftheway to hear therahhit saytoitself ohdear ohdear

Figure 2: The same sample of text, but using up to fourth-order correlation per stage instead of the third-order limit in Figure 1. Only the last stage is shown. It has Hw/S = 2.28 bits. Reviewing the results in Figure 1, one may note that some real English words, such as “daisies,” are not found, but this is due to the relatively small sample of English text used. In fact, the word “daisies” appears in the text only once so it would have been an error for it to quahfy as a redundant feature. However, the algorithm is superbly sensitive to redundant words which appear in the text as few as two or three times. Another thing to observe is that many groups of real words are combined into single features. Some of this reflects actual redundancy in English, for example “ofthe” is likely a truly redundant combination in English, but many of these, such as “whiterabbit” are only redundant for this sample text. Such real word groupings would have far lower redundancy (lower 37 in a much larger sample text which includes many different subjects and writing styles. The most significant success of the redundancy reduction algorithm is the segmentation of the text, which is almost always broken at the boundary between real words. This efficient segmentation corresponds to finding a cover W (Fig. le) of the entire sample with a small number of words-less than the number of real words. This is close to the smallest number of (approximately) statistically independent words. Such efficient segmentation would not have been found using an algorithm that chooses only high probability words.

5 Neural Implementation

In a ‘,neural” implementation of the algorithm used in Section 4, neurons, or dendrites, calculate the local data Pf and P,. Actually, only Pf needs to be calculated since the P, are computed by the previous stage and may be encoded in the neural output strengths. Finding F(Pf,Pw)still requires some computation, but (especiallyin 3.7) this reduces essentially

300

A. Norman Redlich

to computing logarithms! The real problem is not how to calculate Pf or F(Py), but how to search the space of possible features for those with largest 3. One option is to convert this search to one over a set of (continuous) synaptic weights and then apply gradient descent to maximize 3. This is the technique used by Hinton and Pearlmutter (1986) to maximize the Kullback information. Though its application to 3is somewhat different, I believe it might work, although I have not attempted it. Instead, I wish to explore here a more direct approach which avoids the convergence problems often associated with gradient descent. The simplest and most direct approach would be to exhaustively calculate Pr for all features of size 5 m. Of course m small enough to make this computationally feasible might be too small to discover the redundancy. But, there really is no need for an exhaustive search, since a prerequisite for large 3is large P f , and a more limited sampling will usually find these common features. Then only those common features with sufficiently large 3 need be kept. I now use this to develop a temporal search algorithm. Suppose first that there are a fixed number (smaller than needed for an exhaustive search) of feature neurons at each learning stage, which can be in one of two states, occupied or pee. Occupied neurons respond to one feature, and their job is to quickly calculate a good approximation for F. As soon as the occupied neuron discovers that 3 is below some constant threshold 3*, it becomes free and is available to test another feature. The neurons are mutually inhibiting so no two neurons can be occupied by the same feature. Also there is some ordering to decide which free neuron takes the next possible feature. To approximate 3,a neuron only needs an approximation for Pfsince the P, were calculated by the previous stage. How big Pf needs to be for F(Pf,P,) > F* depends on the probabilities of the input elements P, that make up the feature. In effect, the feature neuron uses a featuredependent threshold A(P,) for P f . (If the criterion were simple frequency of the feature, on the other hand, one would use a fixed threshold A for Pf.) Features that are built out of infrequent inputs w have lower threshold for P f , as can be seen most easily in 3.7. The final ingredient is an approximation for P f ( t ) at time t, where t = 0 is the time when the neuron first picks up its feature. For this, I make a very simple choice: If the feature has occurred only once at time t = 0, then for t > 0 approximate P f ( t )= l/t; if the feature occurs a second time at t = TI use for t > TI, P f ( t ) = 2 / t ; and if the feature has 41t should be noted that - log(P) has a very nice interpretation as the infomation in or improbability of the signal. If neurons have output strengths proportional to the information they carry, then the mutual information, one of the ingredients needed for 3,can be calculated through simple addition of neuronal outputs. This was suggested by Uttley (1979) as one of the attractions of using the mutual information to build a conditional probability computer (Singh 1966). Also, the idea that neurons signal improbability has been proposed by Barlow (19891, and there is evidence for this in the retina.


301

+

occurred n 1 times use P f ( t ) = n/t, which eventually approaches the true Pf for large n. If at any time P f ( t ) drops below the threshold A(P,), that is, F ( t ) drops below 7, then the occupied neuron is freed to search for other features. Of course, since for small t P f ( t ) may be a poor approximation, good features will be dropped occasionally, but these are likely to be picked back up again since they must be relatively frequent. On the other hand, the longer a neuron is occupied by a feature, the better the approximate Pf(f)becomes and the less susceptible to such errors. In fact, I have simulated this algorithm for the beginning stages of learning for the sample text used in Section 4,and it finds exactly the same set of features as does an exhaustive search, but it requires far less memory. One may also ask how this learning algorithm compares with other unsupervised “feature” detection algorithms. First, as has been discussed, this approach is related to Hinton and Pearlmutter’s: both favor features with large Pf and with Pf >> n P W ,although theirs is not guaranteed to find a factorial code. The greater distinction is between algorithms that use these criteria and algorithms of the type proposed by von der Malsberg (1973) and by Bienenstock et al. (1982). Those also favor features with large Pf,but they prefer features composed of elements with large P,. This may lead to features with small mutual information, and thus may include false background elements. For words in text this leads to poor segmentation, since many very tightly bound words are composed of relatively rare subwords. 6 Noise and Generalization

As mentioned in the introduction, desirable input information is often encoded redundantly (e.g., words in text) so redundancy can be used to distinguish true signal from noise. This is the case for example when the noise is not correlated with the true signal or with itself. Then the feature detection algorithm still finds the true signal redundancy-the true signal statistics-even though the total signal is noisy. To show this, consider an English text with random noise, that is, a certain fraction of letters, chosen randomly, are incorrect. Taking the same sample text used in Section 4, but with 1/13 letters randomly incorrect, I applied the same algorithm as before. The result, shown in Figure 3, is that only real words and word combinations are chosen by the algorithm, while noisy letters are ignored. So noise does not confuse the feature detection. Once the features have been found, the text can be restored by using the probabilities of the redundant words to predict the noiseincorrect letters, that is, to build Bayesian filters. It should be noted that in order to reconstruct the true text, one needs to know more than just the statistical properties of the noisy input messages. In the above example, one additionally needs to know that the

A. Norman Redlich

302

a licewashegiuningtogetverytiredofsittinghyhersisteron thehankandofharingnothitgtodoonceortwzkeshehadpeep e d i k t o z h e h o o k h e r s i s t e r w a s r e a d in g h u t i Ih a d n o p i I t u r e s hr chnversationsinitandwhvtistheuseofahookthoughtalicew ithoutpiqturesosconv~rsationssrshewasconsideringfnhe rownmiodasweclasshecouldforthehotdaymadeherfeelvery sleepyondstupidwhethfrthL?pleasureofmaiingadahsuchai gwouldheworththetrouhljofgsttingupanzpickingthedaqs~ eswhensuddenlyawhinerahhitwithpinkeyesranclosehahe rtherewasnothingsoderyremarkahleinthatnordidalicethi n e i t s o v e r j m u a h o u t o f t h e w a y t o h e a r t h e r c h h i t s a y toitselfo hdearohdeas (a) alice was beg i u ning toget verytiredof sitting by hersister onthe hank and of ha r ing no th it g todo onceortw z k e shehad peeped i k to z he hook hersister was read ing hut ithad no pi1 tures hr c h n ver sation s in it and w h v t i s thcuseof a hook thoughtalice without p i q tures o s conversation s s r shcwas considering f n her own m i o d a swec las shecould for the hot day madeher feelvery sleepyand stupi d whe th f r the plea sure of m a i ing a d a h sucha i g wouldhe worth theu ou h I j of gening up an z pick ing the d a q s i e s whensudden ly a w h in e rahhit with p in key e s r anc 10s e b a her therewas nothing s o der y remark able in that nor d idalice thine its over j m u a h outoftheway tohear ther c hhit say toitself ohdear oh d e a s (b)

Figure 3: Again, the same sample of text as in Figure 1, but with one out of 13 letters randomly incorrect. The noisy text before any redundancy reduction is shown in (a); it has HL = 4.26 bits, which is slightly higher than the original text because it is less correlated. One of the later stages in redundancy reduction is shown in (b);it has entropy per letter Hw/3 = 2.99 bits. Note that the noise does not confuse the algorithm into finding false words or word combinations. noise is random. In other words, one needs at least some outside knowledge or supervision. For example, mean squared filtering that uses the autocorrelator of an ensemble to filter out noise, can be implemented through a supervised perceptron-type algorithm (see Atick and Redlich 1990b). This leads to an important point: purely unsupervised learning based strictly on statistics, does not lead to conceptualization. This is due to the implicit assumption that every distinguishable input state potentially carries a different message. In conceptualizing, on the other hand, different input states which carry the same useful message are grouped together. This grouping requires some further knowledge that distinguishes signal from noise, or provides a measure of closeness on the signal space (Kohonen 19841, or provides active supervision as in perceptron learning. Also, the information that distinguishes between different members of a concept can be thrown away, as in noise filtering. Since this information reduction effectively lowers the number of input states, it also simplifies the problem of learning and storing statistics. So one challenge is to in-


303

corporate in the present redundancy reduction strategy a controlled or supervised information reduction. Some first steps in this direction have been taken by Linsker (1989) and by Atick and Redlich (1990a), both using the mutual information between the desired scene data and the noisy image signal (for a different application of redundancy reduction to supervised learning, see Redlich 1992).

Acknowledgments I thank J. Atick for his very perceptive comments on the manuscript. Also, this work was supported in part by a grant from the Seaver Institution and in part by the DOE DE-FG02-90ER40.542.

References Atick, J. J., and Redlich, A. N. 1990a. Towards a theory of early visual processing. Neural Comp. 2,308-320. Atick, J.J., and Redlich, A. N. 199Ob. Predicting ganglion and simple cell receptive field organizations. Int. J. Neural Syst. 1,305. Atick, J.J., and Redlich, A. N. 1991. Convergent algorithm for sensory receptive field development. Neural Comp. In press. Atick, J. J., and Redlich, A. N. 1992. What does the retina know about natural scenes? Neural Comp. 4,196-210. Attneave, F. 1954. Some informational aspects of visual perception. Psychol. Rev. 61,183-193. Barlow, H. B. 1961. Possible principles underlying the transformation of sensory messages. In Sensory Communication, W. A. Rosenblith, ed. MIT Press, Cambridge, MA. Barlow, H. B. 1989. Unsupervised learning. Neural Comp. 1,295-311. Barlow, H.B., and Foldiak, F! 1989. In The Computing Neuron. Addison-Wesley, New York. Bienenstock, E. L., Cooper, L. N., and M u m , F! W. 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. I. Neurosci. 2,3243. Carroll, L. 1865. Alice in Wonderland. Castle, Secaucus. Eriksson, K., Lindgren, K., and Mansson, B. A. 1987. Structure, Context, Complexity, Organization, Chap. 4. World Scientific, Singapore. Geman, S., and Geman, D. 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. Trans. Patt. Anal. Machine Intell. PAMI-6, 72141. Hinton, G. E., and Pearlmutter, B. A. 1986. G-maximization: an unsupervised learning procedure for discovering regularities. In Neural Nehvorks for Computing, A l p Conference Proceedings, Snowbird, UT,J. S. Denker, ed. AIP Press, New York. Hinton, G. E.,and Sejnowski, T. J. 1983. Optimal perceptual inference. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 448453.

304

A. Norman Redlich

Kersten, D., 1990. Statistical limits to image understanding. In Vision: Coding and Eficiency, C. Blakemore, ed. Cambridge University Press, Cambridge. Kohonen, T., 1984. Self Organization and Associative Memory, Springer-Verlag, Berlin. Kullback, S., 1959. Information Theory and Statistics. Wiley, New York. Linsker, R. 1989. An application of the principle of maximum information preservation to linear systems. In Advances in Neural Information Processing Systems, D. S. Touretzky, ed., Vol. 1, pp. 186-194. Morgan Kaufmann, San Mateo, CA. Redlich, A. N. 1992. Supervised factorial learning. Preprint. Shannon, C. E., and Weaver, W. 1949. The Mathematical Theory of Communication. The University of Illinois Press, Urbana. Singh, J., 1966. Great Ideas in lnformation Theory, Languageand Cybernetics, Chap. 16, Dover, New York. Uttley, A. M., 1979. Information Transmission in the Nervous System. Academic Press, London. von der Malsburg, C. 1973. Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85-100. Received 12 December 1991; accepted 29 September 1992.

This article has been cited by: 2. Jim W. Kay, W. A. Phillips. 2010. Coherent Infomax as a Computational Goal for Neural Systems. Bulletin of Mathematical Biology . [CrossRef] 3. S. M. Boker, J. F. Cohn, B.-J. Theobald, I. Matthews, T. R. Brick, J. R. Spies. 2009. Effects of damping head movement and facial expression in dyadic conversation using real-time facial expression tracking and synthesized avatars. Philosophical Transactions of the Royal Society B: Biological Sciences 364:1535, 3485-3495. [CrossRef] 4. M.A. Sanchez-Montanes, F.J. Corbacho. 2004. A New Information Processing Measure for Adaptive Complex Systems. IEEE Transactions on Neural Networks 15:4, 917-927. [CrossRef] 5. J. Michael Herrmann. 2001. Dynamical systems for predictive control of autonomous robots. Theory in Biosciences 120:3-4, 241-252. [CrossRef] 6. Sepp Hochreiter , Jürgen Schmidhuber . 1999. Feature Extraction Through LOCOCODEFeature Extraction Through LOCOCODE. Neural Computation 11:3, 679-714. [Abstract] [PDF] [PDF Plus] 7. D. Obradovic , G. Deco . 1998. Information Maximization and Independent Component Analysis: Is There a Difference?Information Maximization and Independent Component Analysis: Is There a Difference?. Neural Computation 10:8, 2085-2101. [Abstract] [PDF] [PDF Plus] 8. Jean-Pierre Nadal , Nestor Parga . 1997. Redundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive ApproachesRedundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive Approaches. Neural Computation 9:7, 1421-1456. [Abstract] [PDF] [PDF Plus] 9. J. Gerard Wolff. 1995. Computing as compression: An overview of the SP theory and system. New Generation Computing 13:2, 187-214. [CrossRef] 10. Gustavo Deco, Bernd Schürmann. 1995. Learning time series evolution by unsupervised extraction of correlations. Physical Review E 51:3, 1780-1790. [CrossRef] 11. G. Deco , W. Finnoff , H. G. Zimmermann . 1995. Unsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer NetworksUnsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer Networks. Neural Computation 7:1, 86-107. [Abstract] [PDF] [PDF Plus] 12. A. Norman Redlich . 1993. Supervised Factorial LearningSupervised Factorial Learning. Neural Computation 5:5, 750-766. [Abstract] [PDF] [PDF Plus]


Approximation and Radial-Basis-Function Networks Jooyoung Park Irwin W.Sandberg Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712 USA

This paper concerns conditions for the approximation of functions in certain general spaces using radial-basis-function networks. It has been shown in recent papers that certain classes of radial-basis-function networks are broad enough for universal approximation. In this paper these results are considerably extended and sharpened.

1 Introduction

This paper concerns the approximation capabilities of radial-basis-function (RBF) networks. It has been shown in recent papers that certain classes of RBF networks are broad enough for universal approximation (Park and Sandberg 1991; Cybenko 1989). In this paper these results are considerably extended and sharpened. Throughout this paper, we use the following definitions and notation, in which N and R denote the natural numbers and the set of real numbers, respectively, and, for any positive integer Y, R' denotes the normed linear space of real r-vectors with norm 11 11. (., .) denotes the standard inner product in R'. LP(%'), Lm(%'),and C,(Rr), respectively, denote the usual spaces of R-valued maps f defined on 92' such that f is pth power integrable, essentially bounded, and continuous with compact support. With W c R', C(W) denotes the space of continuous %-valued maps d e fined on W.The usual LP and uniform norms are denoted by 11 . (Ip and 11. l m, respectively. The characteristic function of a Lebesgue measurable subset A of R' is denoted by 1 ~ The . convolution operation is denoted by "*," and the Fourier transform (Stein and Weiss 1971) of a Fouriertransformable function f is written as By a cone in X' we mean a set C C R' such that x E C implies that a x E C for all a 1 0. By a proper cone we mean a cone that is neither empty nor the singleton (0). The block diagram of a typical RBF network with one hidden layer is shown in Figure 1. Each unit in the hidden layer of this RBF network has its own centroid, and for each input x = (xl, x2, . . . ,xr), it computes the distance between x and its centroid. Its output (the output signal at one

i.


306

JooyoungPark and Irwin W.Sandberg

Figure 1: A radial-basis-functionnetwork.

of the kernel nodes) is some nonlinear function of that distance. Thus, each kernel node in the RBF network computes an output that depends on a radially symmetric function, and usually the strongest output is obtained when the input is at the centroid of the node. Each output node gives a weighted summation of the outputs of kernel nodes. We first consider RBF networks represented by functions q : '8' 4 B of the form

where M E N is the number of kernel nodes in the hidden layer, Wi E W is the weight from the ith kernel node to the output node, x is an '1,and K is the common radially symmetric input vector (an element of 8 kernel function of the units in the hidden layer. Here zi E 92' and CY > 0 are the centroid and smoothing factor (or width) of the ith kernel node, respectively. We call this family S o ( K ) . Note that the networks in this family have the same positive smoothing factor in each kernel node. Families with a translation-invariant vector space structure are also often important. For example, networks are widely used in which the smoothing factors are positive real numbers as in So(K), but can have different values across kernel nodes. This family is the smallest vector space among those containing So(K) as a subset. We call this vector space

Approximation and Radial-Basis-Function Networks

S,(K). Its general element q : R'

-+

307

?J? is represented by

where M E N, oi > 0, wi E R, and zi E R' for i = 1,2,. . . ,M. For the sake of clarity and convenience, we consider only a onedimensional output space instead of outputs represented by multiple nodes as in Figure 1. The extension of our results to multidimensional output spaces is trivial. Notice that the kernel function K characterizes the families So(K) and S l ( K ) , and that each kernel node has its output derived from K indexed by two parameters (the centroid and smoothing factor), one for position and the other for scale. Ordinarily K is radially symmetric with respect to the norm 11 11 in the sense that llxll = llyll implies K ( x ) = K(y). However, as we shall see in the next section, radial symmetry of the kernel function K : R' R is needed in the development of only one of the approximation results in this study, Except where indicated to the contrary, radial symmetry of the kernel function K is not assumed. In Park and Sandberg (1991)it is shown that S o ( K ) is dense in LP(R'), p E [l,00) if K is an integrable bounded function such that K is continuous almost everywhere and Jsl,K ( x ) dx # 0. In Cybenko (1989)it is pointed out that a consequence of a generalization of a theorem due to Wiener is that the elements of a vector space related to & ( K ) are capable of approximating functions in L1 (8').The purpose of this paper is to report on a substantial sharpening of the results in Park and Sandberg (1991) and Cybenko (1989). -+

2 Approximation Results

As mentioned above, in Park and Sandberg (1991)it is shown that So(K) is dense in Lp(%'), p E [l,00) if K is an integrable bounded function such that K is continuous almost everywhere and Jslr K ( x ) dx # 0. Our first theorem concerns the p = 1 case; a necessary and sufficient condition is

given for approximation with So(K).

Theorem 1. Assuming that K : 3' -+ R is integrable, S,-,(K) is dense in L'(R') ifand only ifJR,K ( x )dx # 0. Proof. Suppose first that JR, K ( x ) dx # 0, and define ] = I Jslr K ( x )dxl. Let f E L' (3') and E > 0 be given. Since Cc(3') is dense in L' (R') (Rudin 19871, we can choose a nonzero fc E Cc(Rr)such that

Ilf - f c l l 1

0 and a > 0 conditions are replaced by the conditions that a; E S and a E S, where S is any subset of (0,oo) such that zero is a cluster point of S. Also, note that the denseness of SI(K) in L1(Rr) is a corollary of Theorem 1. 2. When K : R' -+ R is integrable, S I ( K ) is dense in L'(R') only if JslrK ( x ) dx # 0. The "only if" part of the proof of Theorem 1 shows this. The above observations give the following theorem:

Theorem 2. Assuming that K : R' if and only if JR, K ( x ) dx # 0.

+R

is integrable, SI(K) is dense in L1(Rr)

Up to this point our results concern the approximation of functions in L'(8') under the condition that JRr K ( x ) dx # 0. As shown above, this condition is necessary for approximation with So(K) or SI(K). A natural question that arises is whether the nonzero-integral condition is necessary for approximation in Lp(Rr), p E (1,m). We will see below that it is not necessary for p = 2.


311

In the following theorem, attention is focused on kernel functions K : 8' .+ R with the property that for all M c R' with positive measure there is a ~7 > 0 such that K(a.)# 0 almost everywhere on some positive measure subset of M . We call such K pointable. We shall use the fact that the negation of this condition on K is that for some M of positive measure, K(P) = 0 almost everywhere on M for all (T > 0.

Theorem 3. Assuming that K : R' + R is a square integrablefunction, SI(K) is dense in L2(Rr) ifand only if K is pointable.

Proof. We make use of the following characterization of closed translationinvariant subspaces of L2(Rr), which is an easy modification of (Rudin, 1987, Theorem 9.17). 0 Lemma 2. Associate to each measurable set E c R' the linear space MEof all f E L2(Rr) such that = 0 almost everywhere on E . Then each M E is a closed translation-invariant subspace of L2(Rr),and every closed translation-invariant subspace of L2(Rr) is M Efor some E . Consider any K satisfying the indicated conditions, and suppose that the closure of S I ( K ) is not L2(Rr). Then, since this closure is translationinvariant, by Lemma 2 there is a measurable subset E of R' having positive measure such that

f =0

almost everywhere on E

for any f in the closure of

S1 (K).

a' exp(-2742, .))K(a.)= 0

In particular, almost everywhere on E

for any z E $2' and ~7 > 0. Thus, K(a.)= 0 almost everywhere on E for all a > 0, which contradicts our supposition. To show the "only if" part, we prove the contrapositive: Assume that there is a measurable set M c R' with positive measure such that

K(P) =0

almost everywhere on M

for all a > 0. Then for any f E L2(Rr) with ] g E S, ( K ) satisfying [If - g((2< ]/2, because

2 This completes the proof.

IlflMll2 =

l l f l ~ 1 1 2>

0,' there is no

I 0

'Here we use (1 . 112 to denote also the usual norm on the space of complex-valued square-integrable functionals.

JooyoungPark and Irwin W. Sandberg

312

A large class of kernel functions satisfies the conditions of pointability. For example, kernel functions K such that K # 0 almost everywhere on some ball centered at the origin are pointable. Note that this class includes functions K with JW, K ( x ) dx = 0. A result for the general LP(Rr) case along the lines of the "if part" of Theorem 2 is:

Proposition 1. With p that

E

(1,oo),let K : Rr + R be an integrublefunction such

and

Then SI(K) is dense in LP(gr). Proof. Suppose that S I ( K ) is not dense in LP(Rr). Then by the HahnBanach theorem (Rudin 19871, there exists a bounded linear functional A on Lp(Rr) such that A[the closure of S , ( K ) ] = (0)

(6)

but

w p ( n r )#)(01 By the Riesz representation theorem (Rudin 1987), A : L p ( a r )+ R can be represented by

for some function 8,f in Lq(Rr)? where q is the conjugate exponent of p defined by 1/p + l / q = 1. In particular, from equation 6

for any z E Rr and > 0. Define K : Rr -+ R and KO : Rr + R for u > 0 by

and

(-)

1- x K b ( x ) = 7K D

O

*The strategy of using the Hahn-Banach theorem together with representations of linear functionals was first used in the neural-networks literature in Cybenko (1989).


313

Note that for any u > 0 and z in IR',

Since K E L1(IRr) and

JR,

K ( x )dx = 1, by Lemma 1 (in the appendix),

II&* g n -gAllq -, 0

as

(r

-,0

(8)

By 7 and 8, we conclude that gA is zero almost everywhere. This implies that A is the zero functional, which contradicts our supposition. Our focus has been on LP approximation. We next give a theorem concerning the uniform approximation of continuous functions on compact subsets of W. Theorem 4. Let K : IR' -,IR bean integrablefunction such that K is continuous Then &(K) is dense in C(W )with and such that K - I ( O ) includes no proper respect to t h e n o m 11 . Ilmforany compact subset WofIR".

Proof. Consider any compact subset W of IR'. Suppose that & ( K ) is not dense in C(W). Then proceeding as in the proof of Proposition 1, we see that there is a nonzero finite signed measure p that is concentrated on W and that satisfies

for any z E IR' and u > 0. With z E R', u > 0, and any function h E L1(Sr)n Lw(Rr) whose Fourier transform has no zeros4 (e.g., the gaussian function exp(-all. I[$) with a > 0), consider the integral

Note that

where lpl is the total variation of p. By equation 9 and Fubini's theorem (see, e.g., Rudin 19621,we have

- (' -

O

= Jr[k1K[

u

] d p ( y )] h(x)dx

3Since k(-w) equals the conjugate of K(w) for any w in R', this condition can be stated in terms of subspaces instead of cones. "ere we use a strategy along the lines of Hornick (1991, proof of Theorem 5).

JooyoungPark and Irwin W. Sandberg

314

By the change of variable x+y equation 10 is equivalent to

--+

x and Theorem 3:4.5 of Petersen (1983),

Note that h * p is integrable (by Theorem 1:4.5 of Petersen 1983). It is also essentially bounded, because Kh * P)(X)l

J,, lh(x - Y)I dlPl(Y)

I

5 Ilhllml/4(Rr) for almost all x E V. Consider the closed translation-invariant subspace I of L'( !I? defined ) as the L'-closure of S l ( K ) . By equation 11 and the essential boundedness of h * p, it easily follows that

for any f in I. Following the notation in Rudin (19621, define the zero set Z ( I ) of I to be the set of w where the Fourier transforms of all functions in I vanish. We claim that a nonzero element in Rr cannot be a member of Z ( I ) when K-'(0) includes no proper cone. Assume to get a contradiction that w # 0 and w E Z ( I ) . Then, using the definition of Z ( I ) ,

K

(5)(w)

= urexp(-2ai(z,w))K(aw) = o

for any z E R' and c > 0. This implies that &ow) = o

for all u > o

Since K is continuous, this means that k-'(O) includes the cone {uw E Rr : u 2 0}, which contradicts the cone condition. Thus, Z(1) is either the empty set or (0). In either case, by Theorems 7.1.2 and 7.2.4 of Rudin (19621, any integrable function from Rr to R with zero integral is a member of I. Thus, equation 12 gives

for any f in L'(Rr) wth JR,f(x)dx = 0.


315

Note that the property 13 can hold only for h * p in the class of almost everywhere constant functions. But since h * p E L'(R) and zero is the only constant function in L1(Rr),we have

h*p =0

almost everywhere.

(14)

Since h has no zeros, by Theorem 2:2.2 of Petersen (1983) and Theorem 1.3.6 of Rudin (19621, equation 14 implies p = 0. This contradicts our supposition, and thus proves the theorem. A corollary of this theorem is that S1(K)is dense in C( W) for any compact subset W of $2' when the kernel K : 8' + R is integrable, continuous and satisfies Jsl,K ( x ) dx # 0. Finally, when K : R' -+ R is integrable and radially symmetric with respect to the Euclidean norm, K is also radially symmetric with respect to the Euclidean norm (Bochner and Chandrasekharan 1949, p. 69). In this setting, every K not equivalent to the zero element of L'(%') satisfies the cone condition of Theorem 4. This observation gives the following: Theorem 5. Let K : 92' + R be a nonzero integrable function such that K is continuous and radially symmetric with respect to the Euclidean norm. Then S1( K ) is dense in C( W)with respect to the norm (1 . lloofor any compact subset W of Rr.

3 Concluding Remarks

The results in this paper significantly improve previous results. In particular, we have given sharp conditions on the kernel function under which radial-basis-function networks having one hidden layer are capable of universal approximation on R' or on compact subsets of Rr. A related result concerning uniform approximation using the elements of So(K) with integrable K is given in Park and Sandberg (1991, p. 254). The results in Section 2 concern the approximation of real-valued functions. Approximations for complex-valued functions are also of interest. In this connection, it is a straightforward exercise to verify that Theorems 1-5 and Proposition 1 remain true if " K : Rr R" is replaced with the condition that K maps R' into the set C of complex numbers, LP(Rr) denotes instead the corresponding space of C-valued functions, the elements of C( W)are taken to be C-valued, and So(K) and S1(K)refer instead to the corresponding sets in which the weights wi are drawn from C. An important problem we have not addressed is that of determining the network parameters so that a prescribed degree of approximation is achieved. -+

Jooyoung Park and Irwin W. Sandberg

316

Appendix

L e m m a 1. Letf function such that

E

Lp(Rr), p

E

[I,oo),and let q ? ~ : Rr + R be an integrable

s,, $44

dx = 1

Define q?Jc : Rr + 2 b y

q?JN = (l/Er)q?J(x/E) for 6 > 0. Then

*f -flip -, Oas E

.+0.

Theorem 9.4 of Rudin (1973). If Y is a closed translation-invariant subspace of L' (W), and if

Z(Y)= nfEv{sE R' :?(s)

= 01

is empty, then Y = I-.'(%'). References Bochner, S., and Chandrasekharan, K. 1949. Fourier Transforms. Princeton University Press, Princeton. Cybenko, G. 1989. Approximation by superpositions of a sigmoidal function. Math. Control, Signals, Syst. 2, 303-314. Hornik, K. 1991. Approximation capabilities of multilayer feedforward networks. Neural Networks 4, 251-257. Park, J., and Sandberg, I. W. 1991. Universal approximation using radial-basisfunction networks. Neural Comp. 3, 246-257. Petersen, B. E. 1983. Introduction to the Fourier Transform and Pseudo-Differential Operators. Pitman, Marshfield, M A . Rudin, W. 1962. Fourier Analysis on Groups. Interscience Publishers, New York. Rudin, W. 1973. Functional Analysis. McGraw-Hill, New York. Rudin, W. 1987. Real and Complex Analysis, 3rd ed. McGraw-Hill, New York. Stein, E. M., and Weiss, G. 197l. Introducfion to Fourier Analysis on Euclidean Spaces. Princeton University Press, Princeton. Received 13 April 1992;accepted 3 September 1992.

'This lemma is used in Park and Sandberg (1991)where it is observed to be a slight modification of a theorem in Bochner and Chandrasekharan (1949). We have since found earlier proofs of the lemma (e.g., Petersen 1983, p. 72).

This article has been cited by: 2. Hong-Jiang Wang, Chi-Sing Leung, Pui-Fai Sum, Gang Wei. 2010. Kernel Width Optimization for Faulty RBF Neural Networks with Multi-node Open Fault. Neural Processing Letters 32:1, 97-107. [CrossRef] 3. María D. Perez-Godoy, Antonio J. Rivera, Francisco J. Berlanga, María José Del Jesus. 2010. CO2RBFN: an evolutionary cooperative–competitive RBFN design algorithm for classification problems. Soft Computing 14:9, 953-971. [CrossRef] 4. Arta A. Jamshidi, Michael J. Kirby. 2010. Skew-Radial Basis Function Expansions for Empirical Modeling. SIAM Journal on Scientific Computing 31:6, 4715. [CrossRef] 5. He Huang, Ji-cheng Bai, Ze-sheng Lu, Yong-feng Guo. 2009. Electrode wear prediction in milling electrical discharge machining based on radial basis function neural network. Journal of Shanghai Jiaotong University (Science) 14:6, 736-741. [CrossRef] 6. Masoud Mirmomeni, Caro Lucas, Masoud Shafiee, Babak N. Araabi, Elaheh Kamaliha. 2009. Fuzzy descriptor systems and spectral analysis for chaotic time series prediction. Neural Computing and Applications 18:8, 991-1004. [CrossRef] 7. Hui Li, Yuping Zhang, Haiqi Zheng. 2009. Gear fault detection and diagnosis under speed-up condition based on order cepstrum and radial basis function neural network. Journal of Mechanical Science and Technology 23:10, 2780-2789. [CrossRef] 8. Dongchuan Yu, Stefano Boccaletti. 2009. Real-time estimation of interaction delays. Physical Review E 80:3. . [CrossRef] 9. Yan Hua, Wei Ping, Xiao Xian-Ci. 2009. A method to improve the precision of chaotic time series prediction by using a non-trajectory. Chinese Physics B 18:8, 3287-3294. [CrossRef] 10. Antonio Sánchez-García, Patricio Muñoz-Esparza, José Luis Sancho-Gomez. 2009. A novel image-processing based method for the automatic detection, extraction and characterization of marine mammal tonal calls. Journal of the Marine Biological Association of the United Kingdom 1. [CrossRef] 11. M. Islam, A. Sattar, F. Amin, Xin Yao, K. Murase. 2009. A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 39:3, 705-722. [CrossRef] 12. D. Achela K. Fernando, Asaad Y. Shamseldin. 2009. Investigation of Internal Functioning of the Radial-Basis-Function Neural Network River Flow Forecasting Models. Journal of Hydrologic Engineering 14:3, 286. [CrossRef] 13. Bor-Shyh Lin, Bor-Shing Lin, Fok-Ching Chong, Feipei Lai. 2009. Higher Order Statistics-Based Radial Basis Function Network for Evoked Potentials. IEEE Transactions on Biomedical Engineering 56:1, 93-100. [CrossRef]

14. Dong Nan, Wei Wu, Jin Ling Long, Yu Mei Ma, Lin Jun Sun. 2008. L p approximation capability of RBF neural networks. Acta Mathematica Sinica, English Series 24:9, 1533-1540. [CrossRef] 15. Alberto Guillén, Ignacio Rojas, Jesús González, Héctor Pomares, L. J. Herrera, O. Valenzuela, F. Rojas. 2007. Output value-based initialization for radial basis function neural networks. Neural Processing Letters 25:3, 209-225. [CrossRef] 16. Ali Gholipour, Caro Lucas, Babak N. Araabi, Masoud Mirmomeni, Masoud Shafiee. 2007. Extracting the main patterns of natural time series for long-term neurofuzzy prediction. Neural Computing and Applications 16:4-5, 383-393. [CrossRef] 17. Enrique Romero, Daniel Toppo. 2007. Comparing Support Vector Machines and Feedforward Neural Networks With Similar Hidden-Layer Weights. IEEE Transactions on Neural Networks 18:3, 959-963. [CrossRef] 18. Bor-Shyh Lin, Bor-Shing Lin, Fok-Ching Chong, Feipei Lai. 2007. Higher-Order-Statistics-Based Radial Basis Function Networks for Signal Enhancement. IEEE Transactions on Neural Networks 18:3, 823-832. [CrossRef] 19. Koldo Basterretxea, Jos Manuel Tarela, Ins del Campo, Guillermo Bosque. 2007. An Experimental Study on Nonlinear Function Computation for Neural/Fuzzy Hardware Design. IEEE Transactions on Neural Networks 18:1, 266-283. [CrossRef] 20. Alexey Kononov, Dries Gisolf, Eric Verschuur. 2007. Application of neural networks to traveltime computation. SEG Technical Program Expanded Abstracts 26:1, 1785. [CrossRef] 21. Masoud Mirmomeni, Caro Lucas, Babak Nadjar Araabi, Masoud Shafiee. 2007. Forecasting sunspot numbers with the aid of fuzzy descriptor models. Space Weather 5:8. . [CrossRef] 22. S. Padma, R. Bhuvaneswari, S. Subramanian. 2007. Application of soft computing techniques to induction motor design. COMPEL: The International Journal for Computation and Mathematics in Electrical and Electronic Engineering 26:5, 1324-1345. [CrossRef] 23. K. Schwab, M. Eiselt, P. Putsche, M. Helbig, H. Witte. 2006. Time-variant parametric estimation of transient quadratic phase couplings between heart rate components in healthy neonates. Medical & Biological Engineering & Computing 44:12, 1077-1083. [CrossRef] 24. Ali Gholipour, Babak N. Araabi, Caro Lucas. 2006. Predicting Chaotic Time Series Using Neural and Neurofuzzy Models: A Comparative Study. Neural Processing Letters 24:3, 217-239. [CrossRef] 25. D. Wedge, D. Ingram, D. Mclean, C. Mingham, Z. Bandar. 2006. On Global–Local Artificial Neural Networks for Function Approximation. IEEE Transactions on Neural Networks 17:4, 942-952. [CrossRef] 26. Nam Mai-Duy, Thanh Tran-Cong. 2005. An efficient indirect RBFN-based method for numerical solution of PDEs. Numerical Methods for Partial Differential Equations 21:4, 770-790. [CrossRef]

27. A. Krzyzak, D. Schafer. 2005. Nonparametric Regression Estimation by Normalized Radial Basis Function Networks. IEEE Transactions on Information Theory 51:3, 1003-1010. [CrossRef] 28. G. H. Schmitz, H. Puhlmann, W. Droge, F. Lennartz. 2005. Artificial neural networks for estimating soil hydraulic parameters from dynamic flow experiments. European Journal of Soil Science 56:1, 19-30. [CrossRef] 29. M. Arif, T. Ishihara, H. Inooka. 2004. Intelligent Learning Controllers for Nonlinear Systems using Radial Basis Neural Networks. Control and Intelligent Systems 32:2. . [CrossRef] 30. S.-H. Huh, J.-H. Park, I. Choy, G.-T. Park. 2004. Nonlinear uncertainty observer for AC motor control using the radial basis function networks. IEE Proceedings - Control Theory and Applications 151:3, 369. [CrossRef] 31. J. Gonzalez, I. Rojas, J. Ortega, H. Pomares, J. Fernandez, A. Diaz. 2003. Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation. IEEE Transactions on Neural Networks 14:6, 1478-1495. [CrossRef] 32. Ivan Tyukin , Cees van Leeuwen , Danil Prokhorov . 2003. Parameter Estimation of Sigmoid Superpositions: Dynamical System ApproachParameter Estimation of Sigmoid Superpositions: Dynamical System Approach. Neural Computation 15:10, 2419-2455. [Abstract] [PDF] [PDF Plus] 33. Yoshifusa Ito . 2003. Activation Functions Defined on Higher-Dimensional Spaces for Approximation on Compact Sets with and without ScalingActivation Functions Defined on Higher-Dimensional Spaces for Approximation on Compact Sets with and without Scaling. Neural Computation 15:9, 2199-2226. [Abstract] [PDF] [PDF Plus] 34. Irwin W. Sandberg. 2003. Gaussian radial basis functions and the approximation of input-output maps. International Journal of Circuit Theory and Applications 31:5, 443-452. [CrossRef] 35. E. Lavretsky, N. Hovakimyan, A.J. Calise. 2003. Upper bounds for approximation of continuous-time dynamics using delayed outputs and feedforward neural networks. IEEE Transactions on Automatic Control 48:9, 1606-1610. [CrossRef] 36. Haojian Xu, P.A. Ioannou. 2003. Robust adaptive control for a class of mimo nonlinear systems with guaranteed error bounds. IEEE Transactions on Automatic Control 48:5, 728-742. [CrossRef] 37. Miroslav K�rn�, Josef B�hm, Tatiana V. Guy, Petr Nedoma. 2003. Mixture-based adaptive probabilistic control. International Journal of Adaptive Control and Signal Processing 17:2, 119-132. [CrossRef] 38. Irwin W. Sandberg . 2003. Indexed Families of Functionals and Gaussian Radial Basis FunctionsIndexed Families of Functionals and Gaussian Radial Basis Functions. Neural Computation 15:2, 455-468. [Abstract] [PDF] [PDF Plus]

39. Michael Schmitt . 2002. Descartes' Rule of Signs for Radial Basis Function Neural NetworksDescartes' Rule of Signs for Radial Basis Function Neural Networks. Neural Computation 14:12, 2997-3011. [Abstract] [PDF] [PDF Plus] 40. Michael Schmitt . 2002. Neural Networks with Local Receptive Fields and Superlinear VC DimensionNeural Networks with Local Receptive Fields and Superlinear VC Dimension. Neural Computation 14:4, 919-956. [Abstract] [PDF] [PDF Plus] 41. Ming Zhang, Shuxiang Xu, J. Fulcher. 2002. Neuron-adaptive higher order neural-network models for automated financial data modeling. IEEE Transactions on Neural Networks 13:1, 188-204. [CrossRef] 42. J. Gonzalez, H. Rojas, J. Ortega, A. Prieto. 2002. A new clustering technique for function approximation. IEEE Transactions on Neural Networks 13:1, 132-142. [CrossRef] 43. Abelardo Errejon, E. David Crawford, Judith Dayhoff, Colin O'Donnell, Ashutosh Tewari, James Finkelstein, Eduard J. Gamito. 2001. Use Of Artificial Neural Networks In Prostate Cancer. Molecular Urology 5:4, 153-158. [CrossRef] 44. Irwin W. Sandberg. 2001. Gaussian radial basis functions and inner product spaces. Circuits Systems and Signal Processing 20:6, 635-642. [CrossRef] 45. Judith E. Dayhoff, James M. DeLeo. 2001. Artificial neural networks. Cancer 91:S8, 1615-1635. [CrossRef] 46. Irwin W. Sandberg. 2000. Constructive approximation of non-linear discrete-time systems. International Journal of Circuit Theory and Applications 28:2, 109-120. [CrossRef] 47. I.W. Sandberg. 2000. Time-delay polynomial networks and quality of approximation. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 47:1, 40-45. [CrossRef] 48. I.W. Sandberg. 1999. Corrections to "separation conditions and approximation of continuous-time approximately finite memo. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 46:10, 1305. [CrossRef] 49. I.W. Sandberg. 1999. Separation conditions and approximation of continuous-time approximately finite memory systems. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 46:7, 820-826. [CrossRef] 50. N.B. Karayiannis. 1999. Reformulated radial basis neural networks trained by gradient descent. IEEE Transactions on Neural Networks 10:3, 657-671. [CrossRef] 51. T. Chau, A.K.C. Wong. 1999. Pattern discovery by residual analysis and recursive partitioning. IEEE Transactions on Knowledge and Data Engineering 11:6, 833-852. [CrossRef] 52. M.M. Polycarpou, J.Y. Conway. 1998. Indirect adaptive nonlinear control of drug delivery systems. IEEE Transactions on Automatic Control 43:6, 849-856. [CrossRef]

53. Irwin W. Sandberg. 1998. Separation conditions and approximation of discrete-time and discrete-space systems. Circuits Systems and Signal Processing 17:3, 305-320. [CrossRef] 54. Zi-Jiang Yang, Koichi Hirata, Teruo Tsuji. 1997. Servo motor adaptive speed control under angle-dependent disturbances using an RBF network. Electrical Engineering in Japan 119:4, 77-86. [CrossRef] 55. I.W. Sandberg, L. Xu. 1997. Uniform approximation of multidimensional myopic maps. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 44:6, 477-500. [CrossRef] 56. Tin-Yau Kwok, Dit-Yan Yeung. 1997. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Transactions on Neural Networks 8:3, 630-645. [CrossRef] 57. Irwin W. Sandberg, Lilian Xu. 1997. Uniform Approximation of Discrete-Space Multidimensional Myopic Maps. Circuits Systems and Signal Processing 16:3, 387-403. [CrossRef] 58. N.B. Karayiannis, G.W. Mi. 1997. Growing radial basis neural networks: merging supervised and unsupervised learning with network growth techniques. IEEE Transactions on Neural Networks 8:6, 1492-1506. [CrossRef] 59. A. Roy, S. Govil, R. Miranda. 1997. A neural-network learning theory and a polynomial time RBF algorithm. IEEE Transactions on Neural Networks 8:6, 1301-1313. [CrossRef] 60. Tin-Yan Kwok, Dit-Yan Yeung. 1997. Objective functions for training new hidden units in constructive neural networks. IEEE Transactions on Neural Networks 8:5, 1131-1148. [CrossRef] 61. Irwin W. Sandberg, Lilian Xu. 1996. Network approximation of input-output maps and functionals. Circuits Systems and Signal Processing 15:6, 711-725. [CrossRef] 62. A. Krzyzak, T. Linder, C. Lugosi. 1996. Nonparametric estimation and classification using radial basis function nets and empirical risk minimization. IEEE Transactions on Neural Networks 7:2, 475-487. [CrossRef] 63. Maxwell B. Stinchcombe . 1995. Precision and Approximate Flatness in Artificial Neural NetworksPrecision and Approximate Flatness in Artificial Neural Networks. Neural Computation 7:5, 1021-1039. [Abstract] [PDF] [PDF Plus] 64. Bjørn Lillekjendlie, Dimitris Kugiumtzis, Nils Christophersen. 1994. Chaotic time series. Part II. System Identification and Prediction. Modeling, Identification and Control: A Norwegian Research Bulletin 15:4, 225-245. [CrossRef] 65. Lipo Wang, Kiuju FuArtificial Neural Networks . [CrossRef]

Communicated by Richard Lippmann

A Polynomial Time Algorithm for Generating Neural Networks for Pattern Classification: Its Stability Properties and Some Test Results Somnath Mukhopadhyay Asim Roy Lark Sang Kim Sandeep Govil Department of Decision and Information Systems, Arizona State University, Tempe, AZ 85287 USA Polynomial time training and network design are two major issues for the neural network community. A new algorithm has been developed that can learn in polynomial time and also design an appropriate network. The algorithm is for classification problems and uses linear programing models to design and train the network. This paper summarizes the new algorithm, proves its stability properties, and provides some computational results to demonstrate its potential.

1 Introduction One of the critical issues in the field of neural networks is the develop ment of polynomial time algorithms for neural network training. With the advent of polynomial time methods (Karmarkar 1984; Khachian 1979 and others), linear programming has drawn increased attention for its potential for training neural networks in polynomial time (Glover 1990; Mangasarian et al. 1990; Bennett et al. 1992; Roy and Mukhopadhyay 1991; Roy et al. 1992). This paper presents the method of Roy and Mukhopadhyay (1991)and Roy et al. (1992) in summary form and proves its stability properties under translation and rotation of data points. Application of the method to some well-known learning problems is also shown. 2 A Linear Programming Method for Neural Network Generation -

The following notation is used henceforth. An input pattern is represented by the N-dimensional vector x , x = (XI,X2,.. . ,XN).The pattern space, which is the set of all possible values that x may assume, is represented by 0,. K denotes the total number of classes. The method is for supervised learning where the training set xl, x2, . . . ,x, is a set of sample patterns with known classification. Neural Computation 5,317-330 (1993) @ 1993 Massachusetts Institute of Technology

S. Mukhopadhyay et al.

318

The basic idea of this method is similar to the hypersphere method of Reilly et al. (1982), where a class region is "covered" by a set of hyperspheres of varying size. This method, however, generates the "covers" in a completely different way so as to obtain a polynomial time algorithm. Any complex nonconvex region can be covered by a set of elementary convex forms of varying size, such as hyperspheres and hyperellipsoids. Nonconvex covers can also be used when there is no problem in doing so. Let p elementary covers or masks (henceforth generally referred to as masks) be required to cover the region of a certain class P. To classify an input pattern as being in class P, it is necessary to determine if it falls within the area covered by one of the p masks. If the pattern space is two-dimensional and one of the p masks is a circle centered at ( a , b) with a radius r, a simple masking functionf (XI,X 2 ) = t.2 - [(XI-a)2 (X,can determine if an input pattern is a member of this mask. If (Xi, X;) is an input pattern, and

+

if f(Xi, Xi)2 0, then (Xi,Xi) is inside this mask and (Xi, Xi) belongs to class P; if f ( X { ,Xi) < 0, then (Xi Xi) is not inside this mask. In the learning phase, the procedure actually requires f ( x ) to be at least slightly positive [ f ( x ) 2 E] for membership in the mask and at least slightly negative v ( x ) 5 -61 for nonmembership. The membership criteria can be relaxed to f ( x ) 2 -6 in the testing phase, if warranted by numerical accuracy considerations. So, in general, let p k be the number of masks required to cover a class k, k = 1,. . . K. Let f f ( x ) , . . . , f L ( x ) denote these masking functions for class k. Then an input pattern x' will belong to class j if and only if one or more of its masks is at least slightly positive, and the masks for all other classes are at least slightly negative. Here, each mask will have its own threshold value E as determined during its construction. Expressed in mathematical notation, an input pattern x' is in class j , if and only if

' x ' ) >-. j

fi(

for at least one mask i,i = 1,.. . , p i

and

fik(x') < -6;

for all k # j and i = 1 . . . p k

(1)

If all masks are at least slightly negative (i.e., below their individual - E thresholds), the input pattern cannot be classified. If masks from two or more classes are at least slightly positive, then also the input pattern cannot be classified, though an indication can be given about the possible contenders. Unlike the hypersphere method of Reilly et al. (19821, the standard masking function (if there is to be one) of this method can possibly be any

Polynomial Time Algorithm

319

function that is linear in terms of the parameters to learn. For example, the following functions of the input vector x:

i=l

i=l

i=l j=i+l

and

i=l

i=l i=i+l

are acceptable masking functions since they are linear functions of the parameters to learn-the UP, bjs, cijs, and d. In experimental studies of this procedure, a quadratic function, as shown in 2, has been used as the standard mask. A quadratic function is able to generate many shapes, such as hyperspheres and hyperellipsoids. It can also generate nonconvex shapes, which is acceptable as long as they help to cover a class region properly.

2.1 Generating Masks Via Linear Programming. The procedure is outlined on the basis of using a standard mask. But that need not be the case. A priori, it is not known how many masks will suffice for any of the class regions. Thus, the first attempt is to define a single mask that will cover the whole class region. If that fails, the sample patterns in that class are generally split into two or more clusters by using a clustering procedure that produces a predetermined number of clusters and then attempts are made to define separate masks for each of these clusters. If that should fail, or if only some are masked, then the unmasked clusters are further split for separate masking until masks are provided for each ultimate cluster. The general idea is to define as large a mask as possible to include as many of the sample patterns within a class in a given mask as is feasibly possible, thereby minimizing the total number of masks required to cover a given region and obtaining the best generalization. When it is not feasible to cover with a certain number of masks, the region is subdivided into smaller pieces and masking is attempted for each piece. That is, the unmasked sample patterns are successively subdivided into smaller clusters for masking. At any stage of this iterative procedure, there will be a number of clusters to be masked. It might be feasible to mask some of them, thereby necessitating the breakup only of the remaining unmasked clusters. This “divide and conquer” procedure is heuristic. One can explore many variations of it.


320

X"

x,'

Figure 1: Masking functions generate a multilayer perceptron. The feasibility of covering a set of sample patterns S, of class i with a mask f(x) is determined by solving the following linear program (LP): Minimize E s.t. f(xp) 2

f(xp) 5 E

2

E

-6

(4)

for all pattern vectors xp in the given set Si to be masked (xP E Si) for all pattern vectors xp in classes other than class i a small positive constant

If the LP solution is feasible and optimal, masking of set Si is complete and the LP solution to the parameters of f(x) defines the mask. If the LP is infeasible, masking of pattern set Si with mask f(x) is not feasible and it must be broken up for feasible masking with maskf(x). The infeasibility of an LP can be determined in polynomial time. 2.2 Constructing a Multilayer Perceptron from the Masks. The masking procedure actually generates a multilayer perceptron. Figure 1 shows how a multilayer perceptron is constructed from the masking functions when quadratics are used as masks. Suppose class A has k masks and


321

class B has p. Each mask is evaluated in parallel at nodes Al through Ak and B1 through B,. For a given input pattern, the output of a node is 1 if the corresponding mask is at least slightly positive (2 and zero otherwise. A hard limiting nonlinearity (linear threshold unit) is used at these nodes. Class A hidden nodes A, through Ak are connected to the final output node A for the class and likewise for class B. The output of node A is 1 if at least one of the inputs is 1, it is zero otherwise, and likewise for node B. Again, hard limiting nonlinearities are used at these output nodes. An input pattern is in class A if the output of node A is 1 and that of node B is zero and vice versa. The masking function coefficients correspond to the connection weights and are placed on the connections between the input nodes and hidden layer nodes, as shown in the figure. The higher order product and power terms have been shown as direct inputs to the network. Actually, one more layer is needed at the input end to compute these higher order terms. In this constructive procedure, each hidden node is generated in an incremental fashion-there is no predesigned, fixed net within which learning takes place. Unlike classical connectionist learning, learning here is based on complete access to available information and on the ability to explicitly compare cases (by means of constraints). The net generated is shallow, is allowed to grow laterally only, and learning takes place in a single layer. As can be seen, the masking procedure constructs a restricted high-order polynomial network (Giles and Maxwell 1987) that is allowed to grow laterally in the hidden layer. This incremental growth of the net is similar in spirit to the GMDH method (Farlow 1984) that adds nodes when necessary.

4)

2.3 Outliers in Classification Problems. Many classification problems, by their very nature, generate patterns that fall outside their core class regions. All classification systems attempt to extract the core regions of each class. Because outliers are not identified a priori in the training set, their weeding out has to be performed either before or in conjunction with drawing of the boundaries. In this procedure, some weeding out is performed prior to and some during masking. Weeding out some outliers prior to masking is done in the following way. A clustering procedure that can produce exactly k clusters, such as K-means and hierarchical clustering, is used to divide the training set into k small groups (clusters) and all minority class members in each group are discarded as outliers. This procedure, in essence, assigns a small neighborhood, as represented by one of the k clusters, to the class of its majority members. The process of breaking up core class regions into smaller pieces, cleansing them of outliers, and then masking them is, in effect, an attempt to minimize classification error. The weeding out is actually performed in three steps with the average cluster size


322

gradually decreasing to settle the allocation of unresolved neighborhoods. Sensitivity analysis is also performed in the first step. For example, if the training set has 700 samples, a clustering procedure can be used to divide the set into 100 clusters of average size 7. Suppose one such cluster has 5 class A and 2 class B patterns. The 2 class B patterns are in a minority in the cluster and are put in a candidate list of outliers. Suppose another cluster has 4 class A and 4 class B patterns. Since neither class commands a majority, the cluster cannot be assigned to any class. Such clusters are collected and broken up into smaller clusters in the next step. In the first step, the number of breakup clusters k is varied to test the sensitivity of the outlier diagnosis. Thus the training set is broken up twice into a different number of clusters and the consistency of each outlier diagnosis verified. For example, the training set of 700 samples may first be broken up into 80 clusters and an outlier candidate list developed. The same set may then be broken up into 120 clusters and a second outlier candidate list developed. Only those patterns that appear on both lists are discarded as outliers. In the second step, the unassigned clusters of step 1 are split into smaller clusters and a relaxed majority rule (> 50% membership defines majority class) used for territory allocation. So, for example, after the first step, out of the original 700 patterns, there might be 50 patterns unassigned, 70 thrown out as outliers and the remaining 580 retained for masking. In step 2, the 50 unassigned patterns may be split into 10 clusters of a reduced average size of 5. Suppose one such cluster has 3 class A and 2 class B patterns. The 2 class B patterns define a minority class and are discarded as outliers. The 3 class A patterns are retained for masking. In step 3, clusters can be arbitrarily assigned to a class if there is a membership tie-nothing remains unassigned. The LP algorithm is summarized below. Further details are available in Roy and Mukhopadhyay (1991) and Roy et al. (1992), including a proof of its polynomial time convergence. 2.4 The Algorithm. Phase I: Weed Out Outliers

step 1

1. Break up the training set into two different cluster sets A and B, each consisting of a different number of clusters, using a clustering procedure that can produce a predetermined number of clusters, such as K-means and hierarchical clustering. (The number of breakup clusters is determined, in turn,by the average cluster size chosen. We generally vary it between 6 to 8 to check sensitivity.) 2. In each cluster set, classify a pattern as an "outlier" if its class has less than one-third of the members, as a "core pattern" if its class has at least a two-thirds majority, and as "unassigned" otherwise.


323

3. Compare these classifications across the cluster sets A and B. If a pattern is classified as a “core pattern” in one set and as an “outlier” in another, reclassify it as a “core pattern.” All other inconsistently classified patterns are reclassified as “unassigned.” 4. “Unassigned” patterns are carried over to step 2, the ”outlier” pat-

terns discarded, and the ”core patterns” retained for masking in phase 11. step 2

1. Break up the remaining “unassigned” patterns into smaller sized clusters using a clustering procedure that can produce k clusters and classify patterns as ”outlier,” “core pattern,” or “unassigned using a relaxed majority rule (over 50% only) for the ”core pattern” classification. A pattern is an “outlier” when its class possesses less than 50% of the members and is “unassigned” otherwise. 2. “Unassigned” patterns are carried over to step 3, the “outliers”

discarded and the “core patterns” retained for masking in phase 11.

Step 3 Repeat step 2 with the remaining “unassigned” patterns, splitting them into smaller sized clusters. In this step, the ”simple majority” rule is used to classify “core patterns,” classification ties are resolved arbitrarily, and no patterns remain “unassigned.” Phase ZI: Construct Masking Functions 0. For each class i, i = 1,...K, perform the following two steps: 1. For each unmasked cluster of class i, set up the LP in (4). Initially, a class has only one unmasked cluster consisting of all the “core patterns” from phase I, unless that is split up to start with. Solve the LPs. If all LP solutions are feasible and optimal, masking of class i is complete; go to step 0 for next class. Otherwise, when some or all LPs are infeasible, save all feasible LP solutions (masks), if any, and go to step 2. 2. Collect all unmasked clusters of class i and split them into smaller

clusters using a clustering procedure. Discard all resulting clusters that are very small in size (e.g., has less than 2% of the total sample patterns) as “outliers.” Return to step 1. If the problem has no noise, phase I can be skipped. Outliers remaining after phase I can cause some mask break up. Phase I1 can be rerun as a cleanup pass to obtain bigger masks producing better generalization. In this procedure, the basic purpose of clustering is to dissect the data and not to uncover “real” clusters.


324

2.5 Stability Properties. To show the translation and rotation invariance properties of this method, the following have to be demonstrated: (1)stability of the steps of the algorithm that use clustering, and (2) stability of the linear programming solutions. Consider the following related pair of problems: Problem I .

set.

Minimize c f ( x p ) 2 p E GI f (xp) I - 6 , p E GZ E 2 a small positive constant €7

(5)

Problem Il. s.t.

Minimize E f ( x i R t ) 2 E , p E G1 f($R t ) I - E , P E Gz E 2 a small positive constant

+ +

(6)

where R is a rotation matrix, t a translation vector, GI the set of pattern vectors to be masked, GZ the set of pattern vectors belonging to classes other than that of GI, and f ( x p ) a linear mask. It is assumed that R is nonsingular and that the transpose of a rotation matrix is also its inverse. The following stability results are proven for a quadratic mask, f ( x ) = xTAx bTx c, where A is an N x N matrix, b a vector of size N, and c a scalar. Similar stability results can be shown for other linear masks.

+ +

Stability Theorem 1. The optimum objective function values for problems I and 11, when they are feasible, are the same and the solutions to the quadratic masks are equivalent. Proof. Let

A2 = RA1R-I b: = bfR-' - 2tTA2 cz = PA2t - bfR-'t + CI €2

=

€1

(7)

(8) (9) (10)

It is shown that if the solution (Al,b l , c l , q ) is optimal for problem I, then (Az,bz, CZ, Q), as defined in 7-10, is optimal for problem 11, and if (A2,bzrc2,c2) is optimal for 11, then ( A l ,bl,cl, q ) is optimal for I. Given the assumed optimality of the solutions ( A I b, l , c1, €1) and (Az,b2, c2, €2) for the respective problems (I and 111, they must also be feasible for these two problems. By substituting the solution 7-10 in problem 11, one obtains €1 2 €2, and by similar substitution in problem I, one obtains q 5 FZ. 0 Consequently, = €2 and the stated conclusions follow at once.

Corollary: Stability of Infeasible LPs. When problem I is infeasible, so is problem I1 and vice versa.


325

Proof. Follows directly from Stability Theorem 1 by means of contradiction. If one problem is feasible, then so is the other. CI Rotation and/or translation of data points do not affect the relative distances between them. If the distance matrix remains unchanged, any hierarchical clustering method used to produce k clusters will produce the same set of clusters (Everitt 1980; Hartigan 1975), independent of rotation and/or translation of data points. Hence, if the algorithm generates the same set of pattern vectors for the clustering step at every stage of phases I and 11, and if they are split into the same number of clusters, the resulting clusters at each stage will be identical. Stability Theorem 2. The masking and clustering outcomes, when a hierarchical clustering method is used to produce k clusters, are unaffected by any rotation andlor translation of the training set, when all other conditions of the algorithm remain unchanged. Proof. Follows directly from theorem 1, its corollary, and from the obser0 vations on clustering outcomes noted above. 3 Computational Results

All results have been obtained from an implementation of this algorithm on the SAS system (SAS Manual 1988). The problems were solved on an IBM 3090 operating under the MVS operating system. For clustering, the average linkage method of hierarchical clustering (Sokal and Michener 1958) was used. 3.1 The Parity Problem. Rumelhart et al. (1986) tried the parity problem ranging in size from two to eight. They used a single hidden layer architecture which requires at least N hidden units to solve parity with N inputs. Muhlenbein (1990) reports that the bp algorithm never converged for N 2 6 with N hidden units, but converged when provided with 2N hidden units. Tesauro and Janssens (1988) also used 2N hidden units to overcome the local minimum problem. Table 1 shows the clustering and LP solution times for this procedure. Since it is a two-class problem, only one class was masked. Phase I was not used since there is no noise in this problem. The table shows that the algorithm is extremely fast. 3.2 The Symmetry Problem. Rumelhart et al. (1986) discovered that it can always be solved with only two hidden units in a single layer of hidden units. Table 2 shows the results of this procedure on the symmetry problem. Since it is a two-class problem, only one class was masked. It is solved in all cases with a single hidden unit (mask) and the LP solution times are close to zero. A masking function with linear and square terms only was used.


326

Table 1: Solution Times for the Parity Problem

N, No. of bits

Clustering time (sec)

2 3 4 5 6 7 8

0 0.05 0.05 0.16 0.65 1.95 5.85

No. of masking functions

LP time (sec) 0 0 0 0 0 13 40

1 2 2 3 7 14 28

Table 2: Solution Times for the Symmetry Problem N, Clustering LP No. of No. of bits time (sec) time (sec) masking functions

2 3 4 5 6 7 8

-

0 0

0 0 0 0

0

1 1 1 1 1 1 1

3.3 Overlapping Gaussian Distributions. An obvious question about this method is, how well would the outlier detection heuristic work on classes with dense overlap? To test the heuristic under those circumstances, the following two problems were set up.

Problem I: The I-I Problem. A simple two-class problem where both classes are described by gaussian distributions with different means, and with covariance matrices equal to identity I. A four-dimensional problem with mean vectors [OOOO]and [1111] was tried. The Bayes error is about 15.2% in this case. Problem ZI: The I 4 Problem. A two-class problem where both classes are described by gaussian distributions with zero mean vectors and covariance matrices equal to I and 41. The optimal classifier here is quadratic, and Bayes error for a four-dimensional problem is 17.64% and for an eight-dimensional problem is 9%. Both problems were tried with randomly generated training sets of different sizes. Tables 3 and 4 show the results. Since both are two-class

12

10 13

16 16 16

23 27 27

M = ml,m2,m3

M = 6,.5,3

M = 7,5,3 M = 8,S,3

M=6,S M = 7,s M=8,5

M =6,s M =7 3

M = 8,s

phase I

No. of outliers found in phase I

Cluster sizes

Clustering + LP solution time (sec)

10.1 10.4 10.1 11.98 11.98 11.98

20

20

0 0

76.76

23

Number of Training Patterns (n = 180)

0 0 0


0 0 0

1 1 1

1 1 1

1 1 1

hCti0nS

No. of Masking


found in phase 11

No. of outliers

Table 3 LP Solution Ties and Results for Problem I (Bayes Error-15.2%)

21 17.1 17.1

18.6 18.6 18.6

23.6 20 2.5

Error (%)

17.8

17

17

Error (%) of SAS linear disaiminant function

M=7,5 M = 8,5

M=6,5

M = 7.5 M = 8,5

M = 6,5

Cluster sizes M = ml,m2,m3 phase I No. of outliers found in phase I1

Clustering + LP solution time (sec) No. of Masking functions

0 0

0 18.44

18.4 18.52

38 49 63 15 20 3

837.8 379.5 484.6

1 1 1

Number of Training Patterns (n = 4001,eight dimensions (Bayes error, 9%)

29 29 29

10.9 12.9 12

19 19 19

Error (%)

Number of Training Patterns ( n = 180),four dimensions (Bayes error, 17.64%)

No. of outliers found in phase I

Table 4 LP Solution limes and Results for Problem II

12.5

22.08

Error (%) of SAS linear discriminant function


329

problems, only one of the classes need be masked. As shown, a single hidden node (mask) suffices for all cases. For both problems, the error rate generally decreases as the training set size is increased and tends to the theoretical minimum Bayes error rate. A randomly generated test set of 400 examples was used for each problem. The table entries M = 7, 5, 3, etc. show the average cluster size that was used in each step of the three-step phase I procedure. Tables 3 and 4 also show the error rates of the SAS linear discriminant function on these same problems. For problem 11, the SAS linear discriminant function was provided with squares of the input values as additional inputs. 4 Conclusions

This paper describes a new algorithm that uses linear programming to find the weights of a network. Its advantages are that it can both design and train a net in polynomial time. The network design issue has often been overlooked by the neural network community. But a true neural network algorithm should be able to both design and train a network in polynomial time. This classifier, however, does not provide estimates of Bayesian probabilities and is currently implemented using SAS.

Acknowledgment This research was supported by the National Science Foundation Grant IRI-9113370.

References Bennett, K. P., and Mangasarian, 0. L. 1992. Neural network training via linear programming. In Advances in Optimization and Parallel Computing, P. M. Pardalos, ed. North Holland, Amsterdam. Everitt, B. S. 1980. Cluster Analysis, 2nd ed. Heinemann Educational Books, London. Farlow, S. 1984. Self-organizing Methods in Modeling. Marcel Dekker, New York. Giles, C. L., and Maxwell, T. 1987. Learning, invariance, and generalization in high-order networks. Appl. Optics 26(4),972-974,978. Glover, F. 1990. Improved linear programming models for discriminant analysis. Decision Sci. 21(4), 771-785. Hartigan, J. A. 1975. Clustering Algorithms. John Wiley, New York. Karmarkar, N. 1984. A new polynomial time algorithm for linear programming. Combinatorica 4,373-395. Khachian, L. G. 1979. A polynomial algorithm in linear programming. Dokl. Akad. Nauk SSR 244(5), 1093-1096; Soviet Math. Dokl. 20, 191-194.

330


Mangasarian, 0. L., Setiono, R., and Wolberg, W. H. 1990. Pattern recognition via linear programming: Theory and application to medical diagnosis. In Large-Scale Numerical Optimization, T. F. Coleman and Y. Li, eds., pp. 22-30. SIAM, Philadelphia. Miihlenbein, H. 1990. Limitations of multilayer perceptrons-Steps towards genetic neural networks. Parallel Comput. 14(3), 249-260. Reilly, D. L., Cooper, L. N., and Elbaum, C. 1982. A neural model for category learning. Biol. Cybern. 45, 35-41. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in Microstructure of Cognition, Vol. 1: Foundations, D. E . Rumelhart and J. L. Mdlelland, eds., pp. 318-362. MIT Press, Cambridge, MA. Roy, A., and Mukhopadhyay, S. 1991. Pattern classification using linear programming. ORSA J. Comput. 3(1), 66-80. Roy, A,, Kim, L. S.,and Mukhopadhyay, S. 1992. A polynomial time algorithm for the construction and training of a class of multilayer pemeptrons. Neural Nefwrks, in press. SAS Institute Inc. 1988. SAS Manual, Cary, NC. Sokal, R. R., and Michener, C. D. 1958. A statistical method for evaluating systematic relationships. Univ. Kansas Sci. Bull. 38, 1409-1438. Tesauro, G., and Janssens, R. 1988. Scaling relationships in backpropagation learning. Complex Syst. 2, 39-44. Received 1 November 1991; accepted 1 September 1992.

This article has been cited by: 2. Y. Takahashi. 2000. A mathematical solution to a network construction problem. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 47:2, 166-184. [CrossRef] 3. Ling Zhang, Bo Zhang. 1999. A geometrical representation of McCulloch-Pitts neural model and its applications. IEEE Transactions on Neural Networks 10:4, 925-929. [CrossRef] 4. C. Citterio, A. Pelagotti, V. Piuri, L. Rocca. 1999. Function approximation-fast-convergence neural approach based on spectral analysis. IEEE Transactions on Neural Networks 10:4, 725-740. [CrossRef] 5. J. Manuel Torres Moreno, Mirta B. Gordon. 1998. Efficient Adaptive Learning for Classification Tasks with Binary UnitsEfficient Adaptive Learning for Classification Tasks with Binary Units. Neural Computation 10:4, 1007-1030. [Abstract] [PDF] [PDF Plus]

Communicated by Fernando Pineda

Neural Networks for Optimization Problems with Inequality Constraints: The Knapsack Problem Mattias Ohlsson Carsten Peterson Bo Soderberg Department of Theoretical Physics, University of Lund, Shegatan 14A, $22362 Lund, Sweden

A strategy for finding approximate solutions to discrete optimization

problems with inequality constraints using mean field neural networks is presented. The constraints x I 0 are encoded by xO(x) terms in the energy function. A careful treatment of the mean field approximation for the self-coupling parts of the energy is crucial, and results in an essentially parameterfree algorithm. This methodology is extensively tested on the knapsack problem of size up to lo3 items. The algorithm scales like NM for problems with N items and M constraints. Comparisons are made with an exact branch and bound algorithm when this is computationally possible (N I 30). The quality of the neural network solutions consistently lies above 95% of the optimal ones at a significantly lower CPU expense. For the larger problem sizes the algorithm is compared with simulated annealing and a modified linear programming approach. For “nonhomogeneous“ problems these produce good solutions, whereas for the more difficult “homogeneous” problems the neural approach is a winner with respect to solution quality and/or CPU time consumption. The approach is of course also applicable to other problems of similar structure, like set covering. 1 Background

Feedback artificial neural networks (ANN)have turned out to be powerful in finding good approximate solutions to difficult combinatorial optimization problems (Hopfield and Tank 1985; Peterson and Soderberg 1989; Peterson 1990; Gislkn et al. 1989, 1991). The basic procedure is to map the problems onto neural networks of binary (Ising spin) or K-state (Potts spin) neurons with appropriate choice of energy functions, and then to find approximate minima of the energy using mean field theory (MFT) techniques. In this way essentially “black box” procedures emerge. The application areas dealt with in Hopfield and Tank (1985) and Peterson and Siiderberg (1989), Gislkn et al. (1989,1991) (traveling salesman, Neural Computation 5,331-339 (1993) @ 1993 Massachusetts Institute of Technology

M. Ohlsson, C. Peterson, and B. Soderberg

332

graph partition and scheduling) are characterized by global equality constraints, which can be implemented as quadratic penalty terms. These contain self-interaction parts (diagonal terms), which can be balanced by counterterms to assure reliable MFT dynamics. However, in many real-world optimization problems, in particular those of resource allocation type, one has to deal with inequalities. The objective of this work is to develop a mapping and MFT method to deal with this kind of problem. As a typical resource allocation problem we choose the knapsack problem for our studies. Although artificial, we feel it is a realistic enough test bed. A crucial ingredient in our approach is to avoid self-couplings by a proper MFT implementation of the constraint terms. 2

T h e Knapsack Problem

In the knapsack problem one has a set of N items i with associated utilities ci and loads ski. The goal is to fill a "knapsack" with a subset of the items such that their total utility, N

u = ccisj

(1)

i=l

is maximized, subject to a set of M load constraints, N

defined by load capacities bk. In equations 1 and 2 si are binary ( 0 , l ) decision variables, representing whether or not item i goes into the knapsack. The variables (ci, ski, and bk) that define the problem are all real numbers. We will consider a class of problems, where akj and ci are independent uniform random numbers on the unit interval, while bk are fixed to a common value b. With b = N / 2 , the problem becomes trivial-the solution will have almost all si = l. Conversely, with b CN

(9)

For problems where the constraints are "narrow" (b not too large) this method can require substantially lower computation needs. However, it is still based on exploration and it is only feasible for problem sizes less than M = N FZ 30 - 40. Greedy Heuristics (GH): This is a simple and fast approximate method for a nonhomogeneous problem. Proceeding from larger to smaller ci (cf. equation 91, collect every item that does not violate any constraint. This method scales like NM. Simulated Annealing (SA): Simulated annealing (Kirkpatrick et al. 1983) is easily implemented in terms of attempted single-spin flips, subject to the constraints. Suitable annealing rates and other parameters are given below. This method also scales like NM times the number of iterations needed for thermaliation. Linear Programming with Greedy Heuristics (LP):Linear programming based on the simplex method (Press et al. 1986) is not designed to solve discrete problems like the knapsack one. It does apply, however, to a modified problem with si E [0,l].For the ordered (equation 9) nonhomogeneous knapsack problem this gives solutions with a set of leading Is and a set of trailing Os, with a window in between containing real numbers. Augmented by greedy heuristics for the elements in this window, fairly good solutions emerge. The simplex method scales like

N2M. 5 Numerical Comparisons

Neural Network ("1: Convenient measures for monitoring the decision process are the saturation C = (4/N) Ci(v;- 0.5)2and the evolution rate A = (1/N) Ci(Avi)2,where Avi = vi(t At) - vi(t). The saturation starts off around 0 at high temperature T, and increases to 1 in the T -, 0 limit. We have chosen an annealing schedule where TO= 10, T , = kT,-I, where k = 0.985 if 0.1 < C < (N - l ) / N and 0.95 otherwise. At each temperature every neuron is updated once. We employ a progressive constraint coefficient, a = O.l/T, to avoid small final constraint violations. The algorithm is terminated when C > 0.999 and A < 0.00001. Should the final solution violate any constraint (which is very rare), the annealing is redone with a higher a. In Figure 2 we show a typical evolution of {v;} for an N = M = 40 problem. Simulated Annealing (SA): The performance of this method depends on the annealing schedule. To compare the performance of this method with that of the neural network approach we have chosen the parameters

+

Optimization Problems with Inequality Constraints

337

Table 1: Comparison of Performance and CPU Time Consumption for the Different Algorithms on an N = M = 30 problem! c; = rand[O,11

Algorithm

Perf.

BB

1

NN SA

0.98 0.98 0.98 0.97

LP GH

CPU time 16 0.80 0.80 0.10

0.02

c; = rand[0.45,0.55] Perf. CPU time 1 1500 0.95 0.70 0.95 0.80 0.93 0.25 0.88 0.02

C; = 0.5

Perf. 1 0.97 0.96 0.93 0.85

CPU time 1500 0.75

0.80 0.30 0.02

'The CPU consumption refers to seconds on a DEC3100 workstation.

such that the time consumption of the two methods is the same. This is accomplished with TO = 15, Tfinal= 0.01 and annealing factor k = 0.995. First we compare the NN, SA, and LP approaches with the exact BB for an N = M = 30 problem. This is done both for nonhomogeneous and homogeneous problems. The results are shown in Table 1. As expected LP and in particular GH benefit from nonhomogeneity both quality- and CPU-wise, while for homogeneous problems the NN algorithm is the winner. For larger problem sizes it is not feasible to use the exact BB algorithm. The best we can do is to compare the different approximate approaches, NN, SA, and LP. The conclusions from problem sizes ranging from 50 to 500 are the same as above. The real strength in the NN approach is best exploited for more homogeneous problems. In Figures 3 and 4 we show the performance and CPU consumption for N E [50,500] with M = N. 6 Summary

We have developed a neural mapping and MFT solution method for finding good solutions to combinatorial optimization problems containing inequalities. The approach has been successfully applied to difficult knapsack problems, where it scales like NM. For the difficult homogeneous problems the MFT approach is very competitive as compared to other approximate methods, both with respect to solution quality and time consumption. It also compares very well with exact solutions for problem sizes where these are accessible. In addition, the MFT approach of course has the advantage of being highly parallelizable. This feature was not explored in this work. In Vinod et al. (1990) an ANN approach different from ours was applied to the knapsack problem. The idea in Vinod et al. (1990) is to make orthogonal projections onto convex sets. Since the difficult parameter region was not explored there, a numerical comparison would not be meaningful.

M. Ohlsson, C. Peterson, and B. Werberg

338

I

U

Figure 3: Performance of the neural network (NN)and linear programming approaches (LP)normalized to simulated annealing (SA) for problem sizes ranging from 50 to 500 with M = N. (a) Ci = rand[0.45,0.55]and (b) ci = 0.5.

Figure 4: CPU consumption of the neural network (NN) and linear programming approaches(LP) normalized to simulated annealing (SA) for problem sizes ranging from 50 to 500 with M = N. (a) ci = rand[O.45,0.55] and 6)Ci = 0.5. The numbers refer to DEC3100 workstations.

Optimization Problems with Inequality Constraints

339

Note: We recently became aware of a similar neural approach to the knapsack problem (Hellstrom and Kanal 1992), where the authors also use a discretized form of the derivative. Their treatment is confined to integer problems with a single constraint, whereas ours treat the more general case. Another difference is that the problems probed in Hellstrom and Kanal (1992) are in a nondifficult region (b T =+ f ( A ) = 1

(3.2)

while p, a, u, and P are parameters obeying such restrictions that make the probabilistic interpretation of equations 3.1 possible. The probabilities of staying or going “ O N are both increased when Ai 2 T. It is interesting to note that while we shall be interested exclusively in regimes leading to smooth, uniform behavior of Si(t), values of p, u, a, and P are not precluded, which would yield oscillatory behavior of the activities, a situation that may be of interest in other experimental systems (Bargiello et al. 1984). Morphogen diffusion and synthesis are postulated to be described by the following equations for Ai:

Aj(t + 1) =

cb

+ Aj(t){

~i T,E(t) - %[I - b S j ( f ) ] }

+ k {[I- 0 S i - 1 ( t ) ] A i -(~t ) } + k {[I- ~ S i + l ( t ) ] A i + ~ ( t ) } .

(3.3)

Equations 3.1,3.2, and 3.3 are the central expressions defining our model. In 3.3, c b denotes the basal level of synthesis; k is the diffusion coefficient, or internuclear “hopping” probability per unit time for A molecules. We neglect, in this simple model, spatial fluctuations in k: the nuclei are indifferently spaced and the diffusion rates between them (assumed to be controlled mostly by the nuclear membrane barrier rather than by the internuclear distance, which may vary in the course of contraction) are the same for any two neighbors. Note that the equations must be modified for those nuclei closest to the tendinous ends of the fiber. It is important to realize that when the nuclear switch is turned “ON,” only a fraction 1 - 0 of the morphogen is available for diffusion: the rest is trapped in the nucleus where it directs enhanced mRNA transcription (see below). This interpretation of 3.3 requires that the involved transcription

Model for Motor Endplate Morphogenesis

347

factors be rather scarce, which seems indeed to be the case for, say, the MyoD molecule. The A-enhanced transcription of the a gene yields fresh A product at a rate described by q. The latter embodies the net effect of turnover (degradation) and autocatalytic biosynthesis. It depends on whether a nerve ending is located in the immediate vicinity of nucleus i and on the state of activity Si of this nucleus: isolated nucleus ri = under a synaptic terminal Ti =

70 70

+ TnSi + 7s + TnSi

(3.4)

When the switch is "ON," synthesis proceeds at a faster pace, as described by 7,. Transcription is also boosted further by anterograde factors such as CGRP present under a terminal ( T ~ )and acting there as "synaptotrophic" effectors. The latter are independent of the autocatalytic synthesis state Si, and their efficacy need not be very strong (i.e., Ts

T

(4.1) (4.2)

where Pi'stands for the probability that Sj = 1 at time t . It is easy to ensure that either of these equations relaxes to equilibrium in a time much less than td (see note 3).


349

the previous formulaa5This defines the one-active-nucleus solution completely. However, for the solution to be consistent, it is imperative that A0 be higher than the threshold T, and A1 lower. Furthermore, T must be higher than A,. With parameter values as described in the legend of Figure 2, we find a decay wavelength X = 7.09, while A0 = 2.97 and A, = 0.40 (all these under electrical stimulation). The wavelength is such that around the isolated nucleus, on the order of 7 nuclei will be inactivated on each side. The precise value depends on the threshold T above which the morphogen becomes effective. Why are the nuclei inactivated? The morphogen is trapped by the promoter sites in the nucleus that it activates. Its mobility is thereby reduced, as well as its concentrationin the vicinity: hence, the low value of A,. The concentration far away from the single active nucleus is A, = 0.52. Thus, by chosing a threshold T comprised between A, and Ao, one ensures that only one active nucleus may stably exist in the fiber. Direct dynamic simulations indeed confirm this expectation (see below), yet the exact position of the active nucleus appears difficult to assess analytically at this stage. In general, the value of A0 when the active nucleus is at the tendon6 will be lower than otherwise: for the case at hand, A0 becomes 2.38. This means the buildup of morphogen concentration required to stabilize the active nucleus occurs more readily at the tendinous end-where it has only one escape direction open. It is thus clear that the average transcription level will be rather high there. It must be stressed that X has a significance that goes well beyond the strict framework of the single-active-site solution. It will more generally be related to the approximate wavelength or spatial periodicity of any stable structure evolving from the system dynamics. Indeed, nothing in what we have said until now excludes the appearance, farther than a few X away from the active nucleus, of some additional activity pattern, in the form, for example, of another isolated active site. In general, periodic solutions may be expected with M active, N inactive alternating nuclei. Relative stability of those putative solutions is an extremely arduous problem to tackle analytically, so we shall presently turn to computer simulations. 5Morphogenconcentrations at and in the immediate vicinity of the active site are given as the solutions of the following set of linear equations: (4.6) (4.7)

In the first of these equations, L equals 2 when the active nucleus is in the midst of the fiber itself, or 1 when it is located at a tendinous end. 6That is, L = 1 in equations 4.6 and 4.7.

350

Michel Kerszberg and Jean-Pierre Changeux

Figure 2: (a) Dynamics of transcription states in a focally innervated muscle fiber. States are sampled at intervals of lo00 computation steps (successive lines, from top to bottom). Initial innervation is random. Where and when transcription takes place a box is drawn, full for subsynaptic nuclei, empty for the others. Before the onset of electrical activity, nuclei are mostly active; in the presence of electrical inputs (arrow), transcription is generally repressed but persists in two nuclei, one subsynaptic and located near the center of the fiber, the other near a tendinous end. Note that this latter feature is not always present, although finer time-dependence analysis always shows repression to proceed slowest near the tendons and at the center. (Fiber is innervated by 10 randomly distributed terminals and contains 30 nuclei with genetic switches having a = 0.88, B = 0.80, p = 0.10 and v = 0.95, u = 0.9. Parameter g of equation 3.5 is 0.2, and morphogen diffusion is controlled by k = 0.1. Net synthesis rates (refer to Eqs.3.3and 3.4)are T~ = 0.002, TO = 0.9997,T,, = 0.00025 and q = 0.00004.)(b) The effects of denervation. This is simulated here by switching off electrical input as well as anterograde signaling from the afferent motor endings at all times below the lower arrow. One observes "waves" of renewed transcriptional activity, spreading from the "central" nucleus, as well as from the tendinous ends (whether the nuclei there were initially active or not). Such a spread is typical of diffusion and this figure thus summarizes a strong prediction of the model as a modified reaction-diffusion system.


351

5 Computer Simulations of Developmental Dynamics Computer simulations of the system are consistent with analytical figures wherever the latter are available, that is, in the case of stabilized focal innervation. Moreover they open new perspectives into the behavior of the full nonlinear set of equations, which is out of the reach of analytical methods. Numerical calculations do not depend on the approximation used above, namely that the genetic switch adjustment is fast with respect to morphogen dynamics; yet they are compatible with our previous findings. They cover three cases, those of focal or multiple innervation, and that of noninnervated systems. We assume that 30 nuclei are present along the fiber, and that only a fraction of them are located under newly formed presynaptic terminals. Initially, 10 of those are distributed at random, each above one nucleus. This is a plausible description of the initial state in early endplate morphogenesis.

5.1 Focal Innervation. The upper part (a) of Figure 2 displays successive "snapshot" pictures of the state of activity of the 30 nuclei, at progressively later times from top to bottom. It can be seen how, with sustained electrical activity (which starts below the horizontal line), transcription in most of the nuclei is repressed. Analysis on a finer time scale shows transcription to diminish differently depending on position, with transcription around the fiber center and in nuclei situated near the tendons resisting longest. The final configuration usually comprises one active, subsynaptic nucleus located rather near the middle of the fiber (Salpeter and Loring 1985; Laufer and Changeux 1989; Changeux 1991). Persistent activity near the tendinous ends, while surprising at first, actually confirms the analytical calculations of the previous paragraph. When we take into account late growth of the myotube through myoblast fusion at its extremities (see Fig. 31, we find that with such growth, transcription may remain high in the nuclei incorporated latest in the syncytium, that is, those at the tendinous ends. Experimental support for this finding has been reported recently (Fontaine and Changeux 1989; Klarsfeld et al. 1991). Not pictured here, but observed in some cases, are anomalous patterns such as persistence of two nuclei actively engaged in transcription, or extinction of sustained transcription altogether. 5.2 Denervation. A situation of much interest concerns denervation. Experimental denervation usually leads to a reactivation of AChR gene transcription by many extrajunctional nuclei (Salpeter and Loring 1985; Laufer and Changeux 1989; Changeux 1991; Goldman and Staple 1989). We simulate denervation here by assuming that it simply causes the E variable (see equation 3.5) and the T~ parameter to decrease to zero values.

352

Michel Kerszberg and Jean-PierreChangeux

Figure 3: Myotube growth through myoblast fusion at the extremities is simulated on this figure. Growth starts after an initial innervation and stabilization phase. The myoblasts fusing with the fiber are assumed to express the AChR a-subunit gene at a high rate. It is seen how, through diffusion of the activator, this state is lost more or less quickly (compare the two sides of the figure!). On one side, strong transcription-related labeling would be expected, as observed experimentally(see text).

Figure 2b displays the results of a computer run where "denervation" occurs after a period of electrical stimulation (i.e., at the second horizontal line from the top). It shows waves of transcription onset starting from "seed" nuclei at which concentration of A was initially high, and spreading progressively to the whole fiber. Such seed nuclei include the initially active (subsynaptic) ones, as well as the near-tendon units. Indeed, denervation experiments performed on adult muscle reveal a nonuniform reappearance of the receptor protein and mRNAs, which are first re-expressed in the neighborhood of the endplate (Goldman and Staple 1989; Salpeter and Loring 1985; Neville et al. 1991).


353

a

b

Figure 4: (a) Noninnervated muscle cultures. Spontaneous electrical activity is present in this system, and leads to multiple transcriptional foci. Note that strong lateral labeling by transcription markers is predicted. For T~ = 0, T~ = 0.0003, T = 2.50 (all other parameters unchanged) we find that clusters of transcribing nuclei appear in the form of doublets or transient triplets. The groups themselves are more or less regularly spaced. When one takes T~ = 0.0006, four isolated, "regularly" spaced nuclei are found (not shown). (b) Dynamics of AChR gene expression in the multiply innervated muscle fiber. Here again, we assumed a reduced efficacy ( T ~= 0.00035) in the repressing effect of electrical activity, due, e.g., to spike "bunching." Under these conditions, more or less regularly spaced transcription sites persist, as indeed observed, e.g., in the chick anterior l a h i m u s dorsi muscle. Notice that nuclei at the tendinous ends are invariably the site of intense transcription in this case. From parts (a) and (b) of the figure it is apparent that the final innervation pattern is determined to a large extent by the underlying sarcoplasmic morphogenesis (a) interacting with a set of available exploratory synaptic boutons.

5.3 Cultured Fibers without Innervation. Figure 4a pertains to the situation where electrical activity occurs spontaneously, as is the case in primary cultures of chick embryonic muscle fibers (Fontaine and Chang-

354

Michel Kerszberg and Jean-PierreChangeux

eux 1989; Harris et al. 1989; Bursztajn et al. 1989; Horovitz et al. 1989; Berman et al. 1990). This corresponds formally to T~ = 0. The resulting structure may be called “imperfectly periodic”: there is obviously a preferred spacing between active units, but this is not a strict constraint. Strings of “doublets” or even the transient existence of “triplets” of active nuclei are predicted. For different numerical settings (not shown), strings of isolated, or “singlet” active nuclei become apparent. More complex but less stable configurations than those described here might occur if a larger range of parameter values were to be explored. Interestingly, in situ hybridization experiments with cultured chick myotubes reveal, within the same fiber, silent nuclei alternating with others expressing AChR subunit mRNA, following a pattern of rather poor regularity (Fontaineand Changeux 1989; Harris et al. 1989; Bursztajn et al. 1989; Horovitz et al. 1989; Berman et al. 1990). 5.4 Multiple Innervation (ALD). On Figure 4b, we have selected, for the simulation, numerical parameters which yield a final pattern reminiscent of ALD multiple-focus innervation. One notices rather short-lived clusters of active, mostly sub- or near synaptic nuclei. They are obviously reminiscent to the doublets seen in Figure 4a, and display again a “near-periodicity.” It is quite apparent again how transcription is actively proceeding at the tendons, which seem to “anchor” naturally the nearperiodic pattern of active nuclei. These results suggest very strongly that, in our model, the underlying pattern of gene activation is the dominant factor controlling synapse stabilization. We see here an illustration of how a reduction in the average depressing effect of electrical activity may cause a change in the spatial pattern of genetic expression. On the time scales considered, which are slow compared to nerve action potentials (or even interspike intervals) the reduction in T~ leading to this change may be seen as originating, for instance, from a time patterning of electrical stimulation. Yet, a detailed implementation of such time patterns in electrical stimulation, and their effects on synapse morphogenesis, has not been attempted at this stage.

6 Outlook

Many models, reviewed recently (Van Essen 19821, have been introduced in order to explain the inhomogeneous distribution of AChR along the muscle fiber. None of these previous attempts, however, took account of the inhomogeneity already present at the level of transcription. Here we present a theoretical hypothesis whereby differential AChR gene expression, as controlled by scarce transcription factors for which nuclei enter into competition, may be one of the critical determinative events in motor endplate morphogenesis.


355

The model, which emphasizes the autonomous evolution of the muscle nuclei, leads to satisfactory agreement with a set of known experimental facts concerning the development of the motor endplate upon innervation. It may, however, be of broader import and apply, with suitable modifications, to a variety of situations involving morphogenesis (Hafen et al. 1984; Izpis6a-Belmonte et al. 1991). Crucial for the relevance to cellularized embryonic structures may well be the recent report (Joliot et ul. 1991) that homeotic regulation factors might be able to cross membrane boundaries. The mechanisms proposed here apply to the case of syncytia; when individual, mononucleated cells (such as neurons) are present, one has to include in the model intercellular "morphogens," the mechanisms of their secretion (Torre and Steward 19921, of their recognition by membrane receptors, and of signal transduction. The model could, with suitable modifications, be generalized to synaptogenesis elsewhere in the peripheral and central nervous systems. Synaptogenesis through selective stabilization is of course taking place during development ( k w e s and Lichtman 1980) and has been reported in the cerebellum (Crepe1 et al. 1976; Mariani and Changeux 1980) and several areas of neocortex (Shatz et al. 19901, but has been observed directly in the living adult parasympathetic system as well (Purves and Voyvodic 1987). Recently, transient overproduction of neurotransmitter receptors has also been reported in diverse regions of the primate cerebral cortex (Lidow et al. 1991), and might lead to subsequent morphogenetic phenomena as modeled here if one considers a cortical column as a cellularized analog of the developing myotube. One of our findings that may be of significance for morphogenesis in general is the transcription onset waves we have observed on denervation. The latter are characteristic of a diffusion process and may help substantiate rather simply the morphogen hypothesis in a variety of situations. In mathematical terms, the formulation introduced here represents both a simplification of the original morphogenesis formalization (Turing 1952; Meinhardt 1986) and its generalization. It is a mathematical simplification because the nonlinear aspects are limited to a set of points (the nuclei) rather than being spread over the whole space, thus rendering the model highly tractable analytically; conversely, it represents a generalization since it includes an interaction of the morphogen with the genetic machinery involved in morphogen synthesis itself. The basic ingredients are an autocatalytic loop through the enhancement of activator (morphogen) transcription by its own gene product, long-range inhibition through trapping of the activator by DNA elements in nuclei, thus reducing its availability; and even longer (infinite) range inhibition by electrical activity. The model clearly produces a number of predictions at the level of the molecular mechanisms of gene expression that can be experimentally tested.

356


References Bargiello, R. A., Jackson, F, R., and Young, B. W. 1984. Restoration of circadian behavioral rhythms by gene transfer in Drosophila. Nature (London) 312,752754. Berman, S. A., Bursztajn, S., Bowen, B., Gilbert, W. 1990. Localization of acetylcholine receptor intron to the nuclear membrane. Science 247,212-214. Blau, H., Chiu, C.-P., Webster, C. 1983. Cytoplasmic activation of human nuclear genes in stable heterocaryons. Cell 32, 1171-1180. Britten, R. J., and Davidson, E. H. 1969. Gene regulation for higher cells: A theory. Science 165,349-356. Busztajn, S., Berman, S. A., and Gilbert, W. 1989. Differential expression of acetylcholine receptor mRNA in nuclei of cultured muscle cells. Proc. Natl. Acad. Sci. U.S.A. 06, 2928-2932. Changeux, J. P. 1991. Compartmentalized transcription of acetylcholine receptor genes during motor endplate epigenesis. New Biologist 3,413-429. Changeux, J. P., Babinet, C., Bessereau, J. L., Bessis, A., Cartaud, A., Cartaud, J., Daubas, P., Devillers-Thibry,A., Duclert, A., Hill, J. A., Jasmin, B., Klarsfeld, A., Laufer, R., Nghiem, H. O., Piette, J., Roa, M., and Salmon, A. M. 1990. Compartmentalizationof acetylcholine receptor gene expression during development of the neuromuscular junction. Cold Spring Harbor Symp. Quant. Biol. LV,381-396. Crbpel, F., Mariani, J., and Delhaye-Bouchaud, N. 1976. Evidence for a multiple innervation of Purkinje cells by climbing fibers in immature rat cerebellum. J. Neurobiol. 7, 567-578. Davis, R. L., Weintraub, H., Lassar, A. B. 1987.Expression of a single transfected cDNA converts fibroblasts to myoblasts. Cell 51,987-1000. Driever, W., and Niisslein-Vollhard, C. 1988a. A gradient of bicoid protein in Drosophila embryos. Cell 54, 83-93. Driever, W., and Nusslein-Vollhard, C. 1988b. The bicoid protein determines position in the Drosophila embryo. Cell 54, 95-104. Duclert, A., Piette, J., and Changeux, J. P. 1990. Induction of acetylcholine receptor a-subunit gene expression in chicken myotubes by electrical activity blockade requires ongoing protein synthesis. Proc. Natl. Acad. Sci. U.S.A. 07, 1391-1395. Eichele, G., and Thaller, C. 1987. Characterization of concentration gradients of a morphogenetically active retinoic acid in the chick limb bud. J. Cell. Biol. 105,1917-1923. Fontaine, B., Klarsfeld, A., Hokfelt, T., and Changeux, J. P. 1986. Calcitonin-gene related peptide, a peptide present in spinal cord motoneurons, increases the number of acetylcholine receptors in primary cultures of chick embryo myotubes. Neurosci. Lett. 71,59-65. Fontaine, B., Klarsfeld, A., and Changeux, J. P. 1987. Calcitonin-gene related peptide and muscle activity regulate acetylcholine receptor a-subunit mRNA levels by distinct intracellular pathways. J. Cell Biol. 105, 1337-1342. Fontaine, B., Sassoon, D., Buckingham, M., and Changeux, J. P. 1988. Detection of the nicotinic acetylcholine receptor a-subunit mRNA by in situ hybridiza-


357

tion at neuromuscular junctions of 15-day-old chick striated muscle. EMBO

J. 10, 625-632. Fontaine, B., and Changeux, J. P. 1989. Localization of nicotinic acetylcholine receptor a-subunit transcripts during myogenesis and motor endplate development in the chick. J. Cell Biol. 108, 1025-1037. Goldman, D.,and Staple, J. 1989. Spatial and temporal expression of acetylcholine receptor RNAs in innervated and denervated rat soleus muscle. Neuron 3, 219-228. Hafen, E., Kuroiwa, A., and Gehring, W. J. 1984. Spatial distribution of transcripts from the segmentation gene fushi tarazu during Drosophila embryonic development. Cell 37, 833-841. Hams, D. A., Falls, D. L., and Fischbach, G. D. 1989. Differential activation of myotube nuclei following exposure to an acetylcholine receptor-inducing factor. Nature (London) 337, 173. Harris, D. A., Falls, D. L., Johnson, F. A., and Fischbach, G. D. 1991. A prionlike protein from chicken brain copurifies with an acetylcholine receptorinducing activity. Proc. Natl. Acad. Sci. U.S.A. 88, 7664-7668. Horovitz, O.,Spitsberg,V., and Salpeter, M. M. 1989. Regulation of acetylcholine receptor synthesis at the level of translation in rat primary muscle cells. J. Cell Biol. 108, 1817. Izpisba-Belmonte, J. C., Tickle, C., Dollb, P., Wolpert, L., and Duboule, D. 1991. Expression of the homeobox Hox-4 genes and the specification of position in chick wing development. Nature (London) 350,585-589. Joliot, A. H., Triller, A., Volovitch, M., Pernelle, C., and Pmchiantz, A. 1991. a-2,8-polysialic acid is the neuronal surface receptor of antennapedia homeobox peptide. Nao Biologist 3, 1121-1134. Kaufmann, S. A. 1986. Boolean systems, adaptive automata, evolution. In Disordered Systems and Biological Organization, E. Bienenstock, F. FogelmanSoulib, and G. Weisbuch, eds., pp. 339-360. Plenum Press. Klarsfeld, A., Laufer, R., Fontaine, B., Devillers-Thiery, A., Dubreuil, C., and Changeux, J. P. 1989. Regulation of muscle AChR a-subunit expression by electrical activity: Involvement of protein kinase C and Ca++. Neuron 2, 1229-1236. Klarsfeld, A., Bessereau, J. L., Salmon, A. M., Triller, A., Babinet, C., and Changeux, J. I? 1991. An acetylcholine receptor a-subunit promoter conferring preferential synaptic expression in muscle of transgenic mice. EMBO J. 10, 625-632. Laufer, R., and Changeux, J. l? 1989. Activity-dependent regulation of gene expression in muscle and neuronal cells. Mol. Neurobiol. 3, 1-35. Lidow, M. S., Goldman-Rakic, P. S., and Rakic, P. 1991. Synchronized overproduction of neurotransmitter receptors in diverse regions of the primate cerebral cortex. Proc. Natl. Acad. Sci. U.S.A. 88, 10218-10221. Mariani, J., and Changeux, J. P. 1980. Multiple innervation of Purkinje cells by climbing fibers in the cerebellum of the adult staggerer mutant mouse. J. Neurobiol. 11,41-50. Meinhanit, H. 1986. Hierarchical inductions of cell states: A model for segmentation in Drosophila. J. Cell Sci. Suppl. 4, 357-381.


358

Monod, J., and Jacob, F, 1962. General conclusions: Teleonomic mechanisms in cellular metabolism. Cold Spring Harbor Symp. Quant. B i d . XXVI,389. Neville, C., Schmidt, M., and Schmidt, J. 1991. Kinetics of expression of Ach receptor a-subunit mRNA in denervated and stimulated muscle. NeuroReport 2, 655-657. New, H. V., and Mudge, A. W. 1986. Calcitonin-gene related peptide regulates muscle acetylcholine receptor synthesis. Nature (London) 323, 809-811. Purves, D., and Lichtman, J. 1980. Elimination of synapses in the developing nervous system. Science 210,153-157. Purves, D., and Voyvodic, J. 1987. Imaging mammalian nerve cells and their connections over time in living animals. Trends Neurosci. 10, 398-404. Salpeter, M., and Loring, R. H. 1985. Nicotinic acetylcholine receptors in vertebrate muscle: Properties, distribution and neural control. Prog. Neurobiol. 25, 297-325. Shatz, C. J., Gosh, A., McConnell, S. K., Allendoerfer, K. L., Friauf, E., and Antonioni, A. 1990. Pioneer neurons and target selection in cerebral cortical development. Cold Spring Harbor Symp. Quant. B i d . LV,469480. Thayer, M. J., Tapscott, S. J., Davis, R. L., Wright, W. E., Lasser, A. B., and Weintraub, H. 1989. Positive autoregulation of the myogenic determination gene MyoDl. Cell 58,241-248. Thomas, R., and D’Ari, 1990. Biological Feedback. CRC Press, Boca Raton, FL, and references therein. Torre, E. R., and Steward, 0. 1992. Demonstration of local protein synthesis within dendrites using a new cell culture system that permits the isolation of living axons and dendrites from their cell bodies. 1.Neurosci. 12,762-772. Toutant, M., Bourgeois, J. P., Toutant, J. P., Renaud, D., Le Douarin, G. H., and Changeux, J. P. 1980. Chronic stimulation of the spinal cord in developing chick embryo causes the differentiation of multiple clusters of acetylcholine receptor in the posterior latissimus dorsi muscle. Dev. B i d . 76, 384-395. Turing, A. M. 1952. The chemical basis of morphogenesis. Phil. Trans. R. SOC. (London) B 237, 37-72. Van Essen, D. C. 1982. Neuromuscular synapse elimination: Review. In Neuronal Development, N. C . Spitzer, ed. Plenum Press, New York. Wolpert, L. 1969. Positional information and the spatial pattern of cellular differentiation. 1. Theor. Biol. 25, 147. --

~~

Received 13 May 1992; accepted 24 September 1992.

This article has been cited by: 1. Michel Kerszberg, Jean-Pierre Changeux. 1998. A simple molecular model of neurulation. BioEssays 20:9, 758-770. [CrossRef]

NOTE

Communicated by John Platt

Universal Approximation by Phase Series and Fixed-Weight Networks Neil E. Cotter Peter R. Conwell Electrical Engineering Department, University of Utah, Salt Lake City, UT84112 USA

In this note we show that weak (specified energy bound) universal approximation by neural networks is possible if variable synaptic weights are brought in as network inputs rather than being embedded in a network. We illustrate this idea with a Fourier series network that we transform into what we call a phase series network. The transformation only increases the number of neurons by a factor of two. 1 Technical Preliminaries

Let g(x) be the bounded measurable real-valued function that we wish to approximate. We take the domain of g to be the unit hyper cube D = [-1/2, 1/2INwhere N is the number of entries in x = (XI,. . . , x N ) . To generalize our results below to an interval [ - X / 2 , X / 2 ] we would divide frequencies by X in our final phase series. By Lusin's theorem (Royden 19681, for any 6 > 0 there exists a continuous function, f, such that the measure of the set where f is not equal to g is less 6. Thus, by successfully approximating f we can restrict errors to an arbitrarily small percentage of inputs. We can represent f in turn with a Fourier series. We will assume that If1 is bounded by M / 2 & where M is a value we must specify before constructing our network. This limits us to what we call "weak universal approximation." Since we may take M as large as desired, however, this is a mild limitation. For the derivation we assume M = 1. This amplitude bound translates into a total energy bound of 1/8 on domain D. 2 Phase Series Derivation

Since the energy of a sinusoid on D is one-half its squared amplitude, an energy bound of 1/8 translates into a bound of f 1 / 2 for the coefficients of a Fourier series for f(x): f(x) =

C C abncos(2m o x + 6n/2)

(2.1)

~ = o , In

Neural Computation 5, 359-362 (1993) @ 1993 Massachusetts Institute of Technology

Neil E. Cotter and Peter R. Conwell

360

where n = (nl,. . . ,nN) is a frequency vector with entries ranging over all possible combinations of positive and negative integer values and 6 is a binary variable used to obtain sine terms by phase-shifting cosine terms. We observe that the left-hand side of the following trigonometric identity has the same form as the summand in equation 2.1: 2 COSAcos B = COS(B- A) + COS(B+ A)

(2.2)

We make the identification a6n = 2 cosA

(2.3)

Solving for A and substituting into equation 2.1 yields the phase series representation

f(x) =

CC C

cos(2m o x

+ 6a/2 + r]cos-'[uan/2])

(2.4)

6=0,1 n q=-i,i

The phase series has several features we wish to highlight 1. The phase and Fourier series have exactly the same value everywhere on D. 2. The coefficients of the cosine terms are unity regardless of f ( x ) . 3. The coefficients, c0s-~[a6,,/2],specifying f(x) are processed in the same way as input data-both the coefficients and the data are part of a weighted sum. 4. The multipliers, n and

r],

of the weighted s u m are independent of

f (XI. 3 Phase Series Neural Networks

We can implement both the original Fourier series in equation 2.1 and the phase series in equation 2.4 as neural networks having a single hidden layer and a linear output neuron. In the hidden layer, we could approximate the cosine term by summing familiar sigmoids such as the logistic squasher (Rumelhart and McClelland 1986) or the hyperbolic tangent (Hopfield 1984). For an exact representation, we sum copies of a "cosig" squasher (see Gallant and White 1988): 00

cosx = -1

+C C

+

cosig(xx - x2nk n/2)

(3.1)

k=-00 x=-I,l

where x 5 -R/2 - n/2) -7r/2 < x < 7r/2 x 2 R/2

(3.2)

Universal Approximation

361

Figure 1: Phase series network based on “cosig” sigmoid embedding one halfcycle of a cosine. Stacked boxes indicate a structure that is repeated for all values taken on by indices in the lower right-hand comer. The output sum thus has many inputs.

Substituting 3.1 into 2.4 yields the formula for a phase series neural network

c c c [-I + c c cosig(X2m oo

f(x)

=

6=o,1 n q=-i,i

ox

+ xb.rr/~

k=-oo x=-l,l

+ xq cos-’[&n/2] - X2Tk + */2)]

(3.3)

Figure 1 illustrates the phase series network. Note that the Fourier coefficients defining f(x) are embedded in network inputs. Thus, we have derived a neural network, capable of universal approximation, in which internal weights are fixed. Because the identity in equation 2.2 substitutes two cosine terms for one cosine term and a coefficient, this new network has only twice as many neurons as a Fourier network. 4 Discussion and Conclusion

There are possible advantages to using phase series in VLSI or optical circuits. First, we eliminate circuitry for accessing internal parameters. Second, more technologies are suitable for implementing the fixed synap tic weights inside the phase series network than are suitable for implementing varying synaptic weights inside a conventional neural network.

362

Neil E. Cotter and Peter R. Conwell

Third, constant parameters are less expensive and occupy less room than variable parameters in a circuit. A fixed resistance in a VLSI circuit or a fixed opacity in an optical circuit is relatively easy to manufacture.

Acknowledgments The authors are greatly indebted to an anonymous reviewer who pointed out that the identity in equation 2.2 is simpler than the one we originally

used.

References Gallant, A. R., and White, H. 1988. There exists a neural network that does not make avoidable mistakes. Proc. Int. Joint Conf. Neural Networks (IJCNN),Sun Diego, C A I, 657-664. Hopfield, J. J. 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 81,3088-3092. Royden, H. L. 1968. Real Analysis, 2nd ed. Macmillan, New York. Rumelhart, D. E., and McClelland, J. L., (eds.) 1986. Parallel Distributed Processing: Exploratioiis in the Microstructures of Cognition. Vol. 1: Foundations. MIT Press, Cambridge, MA. Received 18 May 1992; accepted 22 September 1992.

This article has been cited by: 1. Judith E. Dayhoff. 2007. Computational Properties of Networks of Synchronous Groups of Spiking NeuronsComputational Properties of Networks of Synchronous Groups of Spiking Neurons. Neural Computation 19:9, 2433-2467. [Abstract] [PDF] [PDF Plus] 2. A.S. Younger, P.R. Conwell, N.E. Cotter. 1999. Fixed-weight on-line learning. IEEE Transactions on Neural Networks 10:2, 272-283. [CrossRef]


NOTE

Backpropagation with Homotopy Liping Yang Wanzhen Yu Department of Applied Mathematics, Tsinghua University, 100084, Beijing, China

When training a feedforward neural network with backpropagation (Rumelhart et al. 19861, local minima are always a problem because of the nonlinearity of the system. There have been several ways to attack this problem: for example, to restart the training by selecting a new initial point, to perform the preprocessing of the input data or the neural network. Here, we propose a method which is efficient in computation to avoid some local minima. For a neural network, the output of every node is characterized by a nonlinear function (for example, the sigmoid function) that is the origin of local minima. Consider the following homotopy function fx(x) = Ax

+ (1 - A)s(x)

(1)

where s(x) is a sigmoid function and A E [0,1]. fx(x) forms a homotopy between linear and sigmoid function. Denote Nx to be the neural network characterized by f ~ We . start the training with A. = 1, that is, every node , choose &+I < Ak and is linear. After achieving a minimum of N A ~we continue the backpropagation procedures until Xks = 0 with which f x ( x ) is just the original sigmoid function. The learning of a feedforward network is to solve a nonlinear leastsquares problem,

I "

minF(w) = - c[gi(w) - siI2 2 i=l

(2)

where n is the number of training samples and gi is the output of network whose weight is w. This problem can also be treated by solving

VF(w) = 0

and

V2F(w)2 0

(3)

The homotopy method for solving nonlinear equations has been studied since 1978 (Chow et al. 1978; Li 1987). This method begins the homotopy process with an easily solvable equation (at X = 1) and tracing the zero point of homotopy function until getting the solution of original nonlinear equations (at A = 0). Neural Computation 5, 363-366 (1993) @ 1993 Massachusetts Institute of Technology

Liping Yang and Wanzhen Yu

364

As for the training of neural network, it is known that F(w) has a lot of local minima. For ordinary minimization methods, the algorithm often stops at local minima with large objective value and it is not easy to get a satisfactory solution. The homotopy method can overcome this difficulty to some extent. We can observe that the objective function F(w, A) is a polynomial of w when X = 1 and there exist very few minimum points. As X decreases to 0, the nonlinearity of objective function increases and more new minimum points appear. Because we have achieved a minimum point wk of F(w, Xk), which can provide a relatively better initial point for minimizing F(w, &+I), many unwanted local minima of F(w, are avoided. When computing with conventional BP method, it may occur that some components of w become so large that numerical instability will arise. This is because a very large change of x can only cause a very small change of s ( x ) , especially when 1x1 itself is large, that is, s'(x) -, 0 as 1x1 + 00. For this case, the usual treatment is to perform the scaling of w. The homotopy approach can avoid the infinity growth of w, because f { ( x ) > A. Our computational experiments show that the behavior of the homotopy method is good, although it does not guarantee the global minimum. In what follows, we shall study the process for decreasing A. Assume that we have solved minF(w, A) = F(w0, A) for a fixed X E (0,I]. Taking AX < 0, we need to solve minF(w, X + AX), which implies

vwF(w,

+ AX)lwo+Aw = 0

(4)

When w = WO, it gives

0

=

VWF(wo+ Aw, X + AX) - VWF(wo,A)

Whenever ViF(w0, A) is nonsingular, we can compute Aw by

a

AW = -V;F(wo, X ) - ' ~ V w F ( ~ X)AX o,

(7)

In practical computation, w o+ Aw can serve as a prediction of minimum point of F(w, X + AX). Table 1 illustrates an example of a separation problem. This problem contains 21 input points with corresponding outputs equal to +1 or -1. We use a neural network that has one hidden layer with 12 nodes. The sigmoid function s ( x ) is taken as 2/(1 + e-*) - 1. For this problem, the conventional BP algorithm fails to arrive at a totally correct solution, although many initial points have been tested. The homotopy method achieves a minimum point after 5172 iterations, which separate the input points correctly. The result is shown in Figure 1, where the region in white indicates the positive output of the neural network and the region in black indicates the negative output.

Backpropagation with Homotopy

365

Table 1: Separation Problem Input (0.05,0.05) (0.95,0.95) (0.05,0.95) (0.95,0.05)

Output

$1 +1 +1 +1 (0.50,0.50) -1 (0.95,0.50) +l (0.05,0.50) +1

Input

Output

Input

Output

(0.50,0.05) (0.50,0.95) (0.95,0.30) (0.30,0.05) (0.70,0.05) (0.70,0.95) (0.30,0.95)

+1 +1 -1 -1 -1 -1 -1

(0.05,0.30) (0.05,0.70) (0.40,0.50) (0.60,0.50) (0.50,0.40) (0.50,O.a) (0.95,0.70)

-1 -1 +1 +1 +1 +1 -1

Figure 1: Result of the homotopy method.

366

Liping Yang and Wanzhen Yu

Our computational experiments are made on about 20 separation problems whose sizes are similar to the illustrated one. The initial weights of neural networks are fixed for all problems. The conventional BP algorithm fails to get correct separation for one-third of the tested problems, especially for the problems for which the input data are irregular. The homotopy method can arrive at correct separation for all problems except one. It should be noted that the homotopy method usually takes more iterations compared to the conventional BP algorithm, because the former solves many minimization problems step by step as Xk decreases. This is a problem that needs further study. References Chow, S. N., Mallet-Paret, J., and Yorke, J. 1978. Finding zeros of maps: Homotopy methods that are constructive with probability one. Math. Cornp. 32, 887-889. Li, T. Y. 1987. Solving polynomial systems. The Math. Intelligencer 9, 33-39. Orfanidis, S. J. 1990. Gram-Schmidt neural nets. Neural Cornp. 2, 116-126. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representationsby error propagation. In Parallel DistributedProcessing, Vol. 1, D. E. Rumelhart and J. L. McClelland, eds. MIT Press, Cambridge, MA. Weymaere, N., and Martens, J. P. 1991. A fast and robust learning algorithm for feedforward neural networks. Neural Networks 4, 361-369.

Received 26 June 1992; accepted 29 September 1992.

This article has been cited by: 1. Nandakishore Kambhatla , Todd K. Leen . 1997. Dimension Reduction by Local Principal Component AnalysisDimension Reduction by Local Principal Component Analysis. Neural Computation 9:7, 1493-1516. [Abstract] [PDF] [PDF Plus] 2. B. Verma. 1997. Fast training of multilayer perceptrons. IEEE Transactions on Neural Networks 8:6, 1314-1320. [CrossRef] 3. F.M. Coetzee, V.L. Stonick. 1996. On a natural homotopy between linear and nonlinear single-layer networks. IEEE Transactions on Neural Networks 7:2, 307-317. [CrossRef] 4. Frans M. Coetzee , Virginia L. Stonick . 1995. Topology and Geometry of Single Hidden Layer Network, Least Squares Weight SolutionsTopology and Geometry of Single Hidden Layer Network, Least Squares Weight Solutions. Neural Computation 7:4, 672-705. [Abstract] [PDF] [PDF Plus]

NOTE


Improving Rejection Performance on Handwritten Digits by Training with ”Rubbish” Jane Bromley John S. Denker A T b T Bell Laboratories, Holmdel, NJ 07733, U S A

Introduction Very good performance for the classification of handwritten digits has been achieved using feedforward backpropagation networks (LeCun et al. 1990; Martin and Pittman 1990). These initial networks were trained and tested on clean, well-segmented images. In the real world, however, images are rarely perfect, which causes problems. For example, at one time one of our best performing digit classifiers interpreted a horizontal bar as a 2; in this example the most useful response would be to reject the image as unclassifiable. The aim of the work reported here was to train a network to reject the type of unclassifiable images (“rubbish”)typically produced by an automatic segmenter for strings of digits (e.g., zip codes), while maintaining its performance level at classifying digits, by adding images of rubbish to the training set.

Solution to the Problem Our data consisted of 39,740 handwritten characters, obtained from automatically segmented zip codes. The segmentation process used a number of heuristic algorithms that selected the best vertical cuts through the zip codes.’ Since the cuts were vertical and the heuristics imperfect, many of the zip codes were poorly segmented resulting in 94% images of single digits and 6% images of rubbish. An example segmentation is shown in Figure 1. There are, of course, innumerable other patterns that are not good digits, but the rubbish created in this way was particularly relevant to our task. This data was used to train two networks-GOOD

‘An improved zip code reader has since been developed,which relies less on heuristics and for which the training and segmenting are even more strongly coupled (Burges et al. 1992).


368

Jane Bromley and John S. Denker

Figure 1: A typical zip code showing, with dotted lines, one possible, but incorrect, segmentation. We designated as "rubbish" any subimage that could not be identified as a single digit because (1) there was more than one digit present, (2) there was only a small part of a digit present, or (3) the image was not of a digit. All images were labeled by hand. During training, the desired output vector for rubbish images was chosen to have all neurons low, while for a digit the corresponding neuron was high, the rest low.

was trained on 27,359 images of good digits and GOOD+RUBBISH was trained on these plus an extra 1642 images of rubbish. The architecture and training using backpropagation of these networks are described by LeCun et al. (1990). In Table 1 we see that training on rubbish caused no degradation when the nets were tested on good digits only, while it distinctly improved the ability of the network to reject rubbish-it went from rejecting 29.4 to 20.9% of the test patterns for 1% error on the test set. The rejection criterion was based on the difference between the two highest network outputs with highest confidence being assigned to classifications with the largest difference between these two outputs. An experimental investigation of rejection criteria for this network architecture has been made and this criterion came out as about the best (private communication from Yann LeCun). (No improvement is to be expected in the raw error rate, since the rubbish digits were scored as errors for both networks in this case.) The MSE showed no significant variability.

Improving Performance on Handwritten Digits

369

Table 1: Comparison of the Performance of the Two Networks after 20 Passes Through Their Respective Training Sets# Tested on: Trained on:

GOOD

MSE Error rate Reject

.026 4.3 8.5

Good GOOD+RUBBISH

Good+Rubbish GOOD GOOD+RUBBISH

4.4

.031 10.3

8.3

29.4

.027

.029 10.4 20.9

‘Each network was tested on two different testing sets: one consisting of only good digits and the other containing 6% of rubbish. MSE is the analog mean square error between desired and actual outputs. Reject is the number of patterns that had to be rejected (by the network) to achieve a 1% error rate on the remaining test digits.

Conclusions 0

0

0

0

0

Neural networks only do what they are trained to do. Contrary to other findings (Lee 19911, neural networks can generate an effective confidence judgment for rejecting ambiguous inputs. Rubbish subimages are ambiguous digits, multiple digits, partial digits, and noise. These are common in real-world images of handwritten digits. Our results show that including rubbish images in the training set improves the performance of a neural network digit classifier at rejecting such patterns. Performance on well-segmented digits is unaffected by this extra training. Accurate rejection is crucial in a system that automatically segments multidigit images and relies on its classifier to accept or reject possible segmentations of the image (Matan et al. 1992). Using this classifier in such a system led to an improvement in the recognition rate per zip code from 69 to 78%. Even more importantly, there was a vast improvement in the rejection performance of the whole system. Zip codes were sorted according to the neural network confidence measure of their being correct. Prior to training on rubbish, when the first 60%correctly classified zip codes are accepted, 10%erroneously classified zip codes were also accepted, while after training only 3% erroneously classified zip codes were included.

Acknowledgments Support of this work by the Technology Resource Department of the U.S.Postal Service under contract number 104230-90-C-2456is gratefully acknowledged.

370

Jane Bromley and John S. Denker

References Burges, C. J. C., Matan, O., LeCun, Y., Denker, J. S., Jackel, L. D., Stenard, C. E., Nohl, C. R., and Ben, J. I. 1992. Shortest path segmentation: A method for training a neural network to recognize character strings. Proc. Intl. Joint Conf. Neural Networks, IEEE, 3, 165-172. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. 1990. Handwritten digit recognition with a backpropagation network. In Neural Information Processing Systems,D. s. Touretzky, ed., pp. 396-494. Morgan Kaufmann, San Mateo, CA. Lee, Y. 1991. Handwritten digit recognition using K nearest-neighbor, radialbasis function, and backpropagation neural networks. Neural Comp. 3,440449. Matan, O.,Bromley, J., Burges, C. J. C., Denker, J. S., Jackel, L. D., LeCun, Y., Pednault, E. P. E., Satterfield, W. D., Stenard, C. E., and Thompson, T. J. 1992. Reading handwritten digits: A zip code recognition system. IEEE Computer 25,5943. Martin, G. L., and Pittman, J. A. 1990. Recognizing hand-printed letters and digits. In Neural Information Processing Systems, D. S. Touretzky, ed., pp. 405414. Morgan Kaufmann, San Mateo, CA. Received 27 July 1992; accepted 20 October 1992.

This article has been cited by: 1. Cheng-Lin Liu, H. Sako, H. Fujisawa. 2004. Effects of classifier structures and training regimes on integrated segmentation and recognition of handwritten numeral strings. IEEE Transactions on Pattern Analysis and Machine Intelligence 26:11, 1395-1407. [CrossRef] 2. C.-L. Liu, H. Sako, H. Fujisawa. 2004. Discriminative Learning Quadratic Discriminant Function for Handwriting Recognition. IEEE Transactions on Neural Networks 15:2, 430-444. [CrossRef] 3. D.R. Lovell, T. Downs, Ah Chung Tsoi. 1997. An evaluation of the neocognitron. IEEE Transactions on Neural Networks 8:5, 1090-1105. [CrossRef] 4. M. Revow, C.K.I. Williams, G.E. Hinton. 1996. Using generative models for handwritten digit recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 18:6, 592-606. [CrossRef] 5. G. C. Vasconcelos, M. C. Fairhurst, D. L. Bisset. 1995. Efficient detection of spurious inputs for improving the robustness of MLP networks in practical applications. Neural Computing & Applications 3:4, 202-212. [CrossRef]

NOTE

Communicated by Eric Baum

Vapnik-Chervonenkis Dimension Bounds for Two- and Three-Layer Networks Peter L. Bartlett' Department of Electrical and Computer Engineering, University of Queensland, Qld, 4072, Australia We show that the Vapnik-Chervonenkis dimension of the class of functions that can be computed by arbitrary two-layer or some completely connected three-layer threshold networks with real inputs is at least linear in the number of weights in the network. In Valiant's "probably approximately correct" learning framework, this implies that the number of random training examples necessary for learning in these networks is at least linear in the number of weights. This note addresses the question, How many training examples are necessary for satisfactory learning performance in a multilayer feedforward neural network used for classification? To define "satisfactory learning performance," we assume that the examples are generated randomly, and say that the trained network is approximately correct if it correctly classifies a random example with high probability. We require that the trained network will almost always be approximately correct, for any desired target function and any probability distribution of examples. This is known as "probably approximately correct" (or pac) learning (Valiant 1984). Blumer et al. (1989) show that the number of labelled examples necessary and sufficient for pac learning depends linearly on the VapnikCheruonenkis dimension (VC-dimension) of the set of functions that the learner can choose from. Definition 1. A class F of (0,1)-valued functions defined on a set X is said to shatter a finite set S C_ X if, for each of the 21sl classifications of the points in S, there is a function in F that computes the classification. The VC-dimension of F [written VCdim(F)l is the size of the largest subset of X that F shatters. We consider networks of processing units in layered, feedforward architectures with real-valued inputs and a single binary output. A feedfonvard architecture is a set of units (input units and processing units) arranged in a number of layers, and a set of connections, each of which joins 'Current address: Department of Systems Engineering, RSPhysSE, Australia National University, 0200, Australia.


Peter L. Bartlett

372

one unit to another unit in a later layer. An L-layer network contains L layers of processing units. A feedfonoard threshold network consists of a feedforward architecture that has a particular real-valued weight and threshold associated with each connection and processing unit, respectively. Each processing unit in the network computes a linear threshold function, f ( x ) = 3-1 (Cixiwi - 0), where xi, wi, and O are the real-valued inputs to the unit, weights, and threshold, respectively, and %(a) is 1 if a 2 0 and 0 otherwise. Notice that a network consists of an architecture together with the weights and thresholds, so it computes a particular (0, 1)-valued function of its inputs. We refer to the VC-dimension of the class of functions that can be computed by threshold networks with a particular feedforward architecture A as the VC-dimension of that architecture, and write VCdim(A). The VC-dimension of an arbitrary feedforward architecture is not known precisely. Baum and Haussler (1989) show that the VC-dimension of a feedforward architecture with N processing units and W weights is no more than 2Wlog, eN (where e is the base of the natural logarithm), and that the VC-dimension of a completely connected two-layer architecture with ko input units and kl first-layer units is at least 2[k1/2]ko. (A completely connected multilayer network has connections between all pairs of units in adjacent layers.) In this note, we give lower bounds on the VC-dimension of arbitrary two-layer architectures and some completely connected three-layer architectures. By the results of Blumer et al. (1989) the bounds indicate in all cases that the sample size necessary for pac learning is at least proportional to W, the number of weights in the network. Proofs of the results are given in the full version of this note (Bartlett 1992). To show that the VC-dimension of an architecture is at least d, we can construct a shattered set of size d . The problem of constructing such a set can be decomposed by separately constructing defining sets for units in a network.

Definition 2. A set S = { x , , x2, . . . ,x,} c U& is a defining set for a unit u in a feedforward threshold network with ko real-valued inputs if 1. We can classify the points in S in each of the 21sI distinct ways by slightly perturbing the weights and threshold of unit u.

2. Slightly perturbing the weights and threshold of units other than u will not affect the classification of the points in S. A point x E I& is an oblivious point for this network if the classification of x is unaffected by sufficiently small perturbations of the network weights.

Theorem 3. Let A be a feedforward architecture. Consider a set of processing units U in this architecture and a threshold network N with architecture A that has an oblivious point. If there is a finite defining set S , for each unit u in U , then VCdim(A) 2 CuEU IS,[ 1.

+

Vapnik-Chervonenkis Dimension

373

By finding appropriate defining sets, we can use Theorem 3 to give the following lower bounds.

Theorem 4. Let A be an arbitrary two-layer feedforward architecture. If A has I connectionsfrom the input units to other units, then VCdim(A) 2 I + 1. Theorem 5. Let A be the three-layer, completely connected architecture with ko > 0 input units, kl > 0 first-layer units, k2 > 0 second-layer units, and a single output unit.

+

(a) If ko 2 k l , and kz 5 2&1/(g/2 k1/2 + l), then VCdim(A) 2 kokl ki(kz - 1) 1. (b) If 1 < ko < k1 2 k2, then VCdim(A) 2 kok1 kl(k2 - 1)/2 1.

+

+

+

+

These results imply that for learning two-layer networks or completely connected three-layer networks with kz not too large, the sample size must increase at least linearly with the number of weights. The results also give lower bounds for learning in networks of processing units with a sigmoid transfer function, since a sigmoid network can compute any function on a finite set that a threshold network can compute.

Acknowledgments This work was supported by the Australian Telecommunications and Electronics Research Board. Thanks to T. Downs, R. Lister, D. Lovell, R. Williamson, and S. Young for comments on a draft.

References Bartlett, P. L. 1992. Lower bounds on the Vupnik-Chervonenkisdimension of multi-layer threshold networks. Intelligent Machines Laboratory, Department of Electrical and Computer Engineering, University of Queensland, Brisbane, Australia, Tech. Rep. IML92/3. October 1992. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neural Cornp. 1,151-160. Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. K. 1989. Learnability and the Vapnik-Chervonenkisdimension. J. Assoc. Computing Machin. 36(4), 929-965. Valiant, L. G . 1984. A theory of the learnable. Commun. ACM 27(11), 1134-1143. Received 24 September 1992;accepted 29 September 1992.

This article has been cited by: 1. Hiroki Suyari, Ikuo Matsuba. 1999. Information theoretical approach to the storage capacity of neural networks with binary weights. Physical Review E 60:4, 4576-4579. [CrossRef] 2. Ansgar H L West, David Saad. 1998. Journal of Physics A: Mathematical and General 31:45, 8977-9021. [CrossRef]

Communicated by Richard Andersen

A Neural Network for the Processing of Optic Flow from Ego-Motion in Man and Higher Mammals Markus Lappe Josef P. Rauschecker Laboratory of Neurophysiology, N M H , Poolesville, M D 20837, USA and Max Planck Institute for Biological Cybernetics, Tubingen, Germany

Interest in the processing of optic flow has increased recently in both the neurophysiological and the psychophysical communities. We have designed a neural network model of the visual motion pathway in higher mammals that detects the direction of heading from optic flow. The model is a neural implementation of the subspace algorithm introduced by Heeger and Jepson (1990). We have tested the network in simulations that are closely related to psychophysical and neurophysiological experiments and show that our results are consistent with recent data from both fields. The network reproduces some key properties of human ego-motion perception. At the same time, it produces neurons that are selective for different components of ego-motion flow fields, such as expansions and rotations. These properties are reminiscent of a subclass of neurons in cortical area MSTd, the triple-component neurons. We propose that the output of such neurons could be used to generate a computational map of heading directions in or beyond MST.

1 Introduction

The concept that optic flow is important for visual navigation dates from the work of Gibson in the 1950s. Gibson (1950)showed that the optic flow pattern experienced by an observer moving along a straight line through a static environment contains a singularity that he termed the focus of expansion. He hypothesized that the visual system might use the global pattern of radial outflow originating from this singularity to determine the translational heading of the observer. A host of studies in human psychophysics have followed up Gibson’s ideas (Regan and Beverly 1982;Rieger and Toet 1985;Warren et al. 1988; Warren and Hannon 1988, 1990). Regan and Beverly (1982)rejected his hypothesis on the basis that the optic flow pattern that arrives on the Neural Computation 5,374-391 (1993) @ 1993 Massachusetts Institute of Technology

Neural Network for Processing Optic Flow

375

retina is radically altered by eye movements of the observer. Then the flow field becomes a superposition of the radial outflow pattern with a circular flow field that is obtained when the eyes move in the orbita. Generally the resulting vector field may also have a singular point similar to a focus of expansion, but this point does not necessarily coincide with the heading direction. If, for instance, the eye rotation results from the fixation of a point in the environment, the singularity will be at the fixation point instead of the destination point. Nevertheless, Warren and Hannon (1990) found humans capable of judging their heading with great accuracy from optic flow patterns that simulated translation plus eye rotation. Their subjects were able to perceive their heading with a mean error between one and two degrees solely from the optic flow. No nonvisual information such as oculomotor signals was necessary. This ability persisted over a natural range of speeds and over a variation of the number of visible moving points between 10 and several hundred. The performance of the subjects was at chance, however, when no depth information in the form of motion parallax was available. In the visual system there are at least two (maybe three) main streams of information flow (Mishkin etal. 1983; Livingstone and Hubel 1988; Zeki and Shipp 1988). In the simplest depiction, there is an inferotemporal system that is mainly responsible for the processing of form, and a parietal system that processes motion (Ungerleider and Mishkin 1982). Within the cortical motion system, one of the prominent and most investigated areas in primates is the middle temporal area or area MT (Allman and Kaas 1971). In cats the probable homologue for MT is the Clare-Bishop area (Clare and Bishop 1954), also called area PMLS (Palmer et al. 1978). Evidence from both areas suggests that they participate in the processing of flow field information. Both areas contain neurons that are highly direction selective and respond well to moving stimuli. It has first been found in cat area PMLS that a majority of neurons prefer movement away from the area centralis, that is, centrifugal motion (Rauschecker et al. 1987a,b; Brenner and Rauschecker 1990). The same has been found in monkey area MT (Albright 1989), thus strengthening the likelihood of a homology between these two areas. Other studies have revealed single neurons in PMLS that respond well to approaching or receding objects (Toyama et al. 1990). More recently, a number of studies have described neurons in the dorsal part of monkey area MST (MSTd) that respond best to large expanding/contracting, rotating, or shifting patterns (Tanaka and Saito 1989a,b; Andersen et al. 1990; Duffy and Wurtz 1991a,b). The response of these neurons often shows a substantial invariance to the position of the stimulus. Duffyand Wurtz (1991a,b) found that a majority of the neurons in MSTd responded not only to one component of motion of the stimulus pattern (e.g., expansion or contraction), but rather to two or all three of them separately. About one-third of MSTd cells displayed selectivity to

376

Markus Lappe and Josef I? Rauschecker

expansions or contractions and clockwise or counterclockwise rotations and showed broad directional tuning for shifting dot patterns when tested with these stimuli one after another. It is these "triple component cells" that our model is mainly concerned with. Furthermore, cells in MSTd are unselective for the overall speed of a stimulus and for the amount of depth information available in the stimulus. There have been a number of computational approaches to extract navigational information from optic flow focusing on different mathematical properties of the flow field. The difficulty of the task is that in the mapping of three-dimensional movements onto a two-dimensional retina some information is lost that cannot be fully recovered. Models that use differential invariants (Koenderink and van Doorn 1981; LonguetHiggins and Prazdny 1980; Waxman and Ullman 1985) require dense optic flow to compute derivatives. By contrast, humans are quite successful with sparse fields (Warren and Hannon 1990). Models based on algorithms that solve a set of equations for only a small number of vectors (Prazdny 1980; Tsai and Huang 1984), on the other hand, require precise measurements and are very sensitive to noise. Methods that rely on motion parallax or local differential motion (Longuet-Higgins and Prazdny 1980; Rieger and Lawton 1985) are in agreement with the psychophysical data in that they fail in the absence of depth in the environment. However, they require accurate measurements at points that are close to each other in the image but are separated in depth, which is an especially difficult task to accomplish. Furthermore, recent psychophysical studies (Stone and Perrone 1991) have shown that local depth variations are not necessary. Least-square minimization algorithms (Bruss and Horn 1983; Heeger and Jepson 1990) that use redundant information from as many flow vectors as are available are robust and comparatively insensitive to noise. None of the above-mentioned algorithms is clearly specified in terms of a neural model. Given the current advances in visual neurophysiology, it seems desirable to construct a neural network for ego-motion perception that is consistent with the neurophysiological and psychophysical data. Recently a network model of heading perception in the simpler case without eye movements has been described (Hatsopoulos and Warren 1991), which accounts for some psychophysical findings. A neural model we presented in brief form earlier together with first results from the model described in this paper (Lappe and Rauschecker 1991) is also concerned with pure translations. It uses a centrifugal bias similar to the one found in PMLS and MT to achieve precise heading judgments with neuronal elements that are as broadly directionally tuned as the cells found in these areas. In this article we present a new neural network that succeeds when the radial flow pattern is disturbed by eye movements. The network is capable of reproducing many of the psychophysical findings, and the


377

single units exhibit great similarity to the triple component cells of Duffy and Wurtz (1991a,b) in area MSTd. 2 The Model

Our network is built in two layers. The first layer is designed after monkey area MT and represents the input to the network. The second layer is constructed to yield a representation of the heading direction as the output of the net and thus could form a model of MSTd. In each network layer we employ a population encoding of the relevant variables, namely the speed and direction of local movements in layer one and the heading direction of the individual in layer two. The computation of the direction of translation is based on the subspace algorithm by Heeger and Jepson (1990). Its main course of action is to eliminate the dependencies on depth and rotation first and thereby gain an equation that depends only on the translational velocity. Therefore it bears some similarity to Gibson's original claim that the visual system can decompose the optic flow into its translational and rotational components. We will restrict the scope of our model to such eye movements as occur when the observer keeps his eyes fixed on a point in the environment while he is moving. This is a natural and frequently occurring behavior, and we believe that using assumptions that are a reflection of the behavior of an animal or a human being makes it more likely to gain results that can be compared with experimental data. Although it is mathematically possible to include any type of eye movements, it is not very likely that the eyes would rotate around their long axis to a significant amount during locomotion. Note that our assumption includes the case of no eye movements at all, since it can be described as gazing at a point infinitely far away. 2.1 Optic Flow and the Subspace Algorithm. Optic flow is the projection of the motion of objects in the three-dimensional world onto a two-dimensional image plane. In three dimensions, every moving point has six degrees of freedom: The translational velocity T = (Tx,T,,TJ and the rotation 52 = (ax, R,, R#. When an observer moves through a static environment all points in space share the same six motion parameters. The motion of a point R = (X, Y, Z)' in a viewer-centered coordinate system is V = -(a x R T). This motion is projected onto an image plane. Writing two-dimensional image vectors in small letters, the perspective projection of a point is r = ( x , y)' = f (X/Z, Y/Z)', where f denotes the focal length. Following Heeger and Jepson (1990) the image velocity can be written as the s u m of a translational and a rotational component:

+

Markus Lappe and Josef l? Rauschecker

378

where p ( x ,y) = 1/Z is the inverse depth, and

A(x,y)=

( -f

")

-f Y

0

The unknown depth and translational velocity are multiplied together and can thus only be determined up to a scale factor. Regarding therefore the translation T as a unit vector, one is left with six unknowns: p, the two remaining components of T, and the three components of 51, but only two known quantities, Ox, Oy. The subspace algorithm uses flow vectors at five distinct image points to yield an overdetermined system of equations that is solved with a minimization method in the following way: The five separate equations are combined into one matrix equation 0 = C(T)q where 0 = (el,.. . ,&)' is now a 10-dimensional vector consisting of the components of the five image velocities, q = [P(x~,yl), . . . ,p(x5,ys),R,, R,, R,]' an eight-dimensional vector, and C(T) a 10 x 8 matrix composed of the A(xi,yi)Tand B(xi,yi)matrices:

C(T) =

(

A h y1)T 1

d

B(x1,y1)

A(x5,y5)T B("5, y5)

Heeger and Jepson (1990) then show that the heading direction can be recovered by minimizing the residual function

R(T) = IIQ'C'(T)112, where C'-(T)is a matrix that spans the two-dimensional orthogonal complement of C(T). 2.2 Restrictionto Fixations during Locomotion. We now restrict ourselves to only those eye movements that arise through the fixation of a ' the center of the visual field while the observer point F = (O,O,l / p ~ ) in moves along a straight line. The rotation that is necessary to h a t e this point can be derived from the condition that the flow at this point has to be zero:

(:)=p.(

-f

0 0 O - f O f) 0 -f O)T+(f 0 0)

Choosing R, = 0 we find

f) = PF(T,,-Tx,O)*. The

optic flow then is:


379

The case of a straight translation without any eye movements can easily be described within this framework by considering a fixation point that is infinitely far away. Then PF and the rotational velocity s1 are zero, resulting in a purely translational flow. The optic flow equation above has only four unknowns: p ( x , y ) , PF, T,, and Ty.Combining the equations for two different flow vectors into one matrix equation in the same way as before yields 0 = C (T) [ P ( x I , ~ ~ ) , P ( x ~ , Y ~ ) ,where ~ F ] ' , C(T) is now only a 4 x 3 matrix, the orthogonal complement of which is a line given by the vector C'(T). The residual function becomes the scalar product between this vector and the observed flow:

R(T) = IQtC'(T)I2

(2.2)

Since the optic flow is a linear function of the translational direction, R(T) does not have a single minimum but is equal to zero along a line in the (T,, Ty ) .plane. Therefore one such minimization alone cannot give the translational velocity, rather several pairs of flow vectors with different R(T) functions have to be used in conjunction. 2.3 The Network. In the first layer of the network, which constitutes the flow field input, 300 random locations within 50" of eccentricity are represented. We assume a population encoding of the optical flow vectors at each location by small sets of neurons that share the same receptive field position but are tuned to different directions of motion. Each such group consists of n' neurons with preferred directions ek, k = 1, . . . ,n'. The flow vector 8 is represented by the sum over the neuronal activities sk in the following way: n'

8 = Cskek

(2.3)

k=l

We do not concern ourselves with how the optic flow is derived from the luminance changes in the retina or how the aperture problem is solved. Neural algorithms that deal with these questions have already been developed (Bulthoff etal. 1989; Hildreth 1984; Yuille and Grzywacz 1988). A physiologically plausible network model that yields as its output a population encoding like the one we use here has been proposed by Wang et al. (1989). It can be thought of as a preprocessing stage to our network, modeling the pathway from the retina to area MT or PMLS. Since we start out with a layer in which the optic flow is already present, we have to guarantee that the tuning curves of the neurons and the distributions of the preferred directions match the requirement of equation 2.3. As the simplest choice for our model, we use a rectified cosine function with n' = 4. It preserves the most prominent feature of the observed directional tuning curves in MT/PMLS,namely broad

Markus Lappe and Josef P. Rauschecker

380

unidirectional tuning with no response in the null direction. The preferred directions are equally spaced, e k = [cos(nk/2),sin(xk/2)], and for the unit's response to a movement with speed 60 and direction $, the tuning curve is

6, cos(4 - nk/2) if cos($ - nk/2) > 0 s k = {

0

otherwise

The second layer represents a population encoding of the translational direction of the movement of the observer, which is represented by the intersection point of the 3D-movement vector T with the image plane. There are populations of n neurons at possible intersection points whose combined activities U I give the perceived direction. But here the sum of the activities U = Cbl U Iat each position yields a measure of how likely this position is to be the correct direction of movement. The perceived direction is chosen to be the one that has the highest total activity. The output of a second layer neuron is a sigmoid function g(x) of the sum of the activities of its m input neurons weighted by synaptic strengths Ijkl and compared to a threshold p:

Here IikI denotes the strength of the connection between the Ith output neuron and the kth input neuron in the population that represents image location i. The sigmoid function is symmetric such that g ( - x ) = 1-g(x). The connections and their strengths are set once before the network is presented with any stimuli, and are fixed afterward. First a number of image locations are randomly assigned to a second layer neuron. Then, values for the synaptic strengths are calculated so that the population of neurons encoding a specific T is maximally excited when R(T) equals zero. Although the neuron may receive input from a large number of image locations we start the calculation of the connections with only two in order to keep it simple. We want the sum in equation 2.4 to equal the scalar product on the right side of equation 2.2:

For every single image location i we have

Substituting equation 2.3 we find


381

Therefore we set the synaptic strengths to

If the neuron is connected to more than two image locations the input connections are divided into pairs and the connections are calculated separately for each pair. Now the question of when R(T) is minimal comes down to the question of when all the neurons’ inputs balance each other to give a net input of zero. Consider two output neurons U Iand UI’receiving input from the same set of first layer neurons but with inverse connections such that lik,’ = -Jiu. Then, if the threshold ,LA equals zero, the sum of both neurons’ activities is equal to 1 regardless of their inputs, since the sigmoid input/output function is symmetric. If, however, ,LA has a slightly negative value, both sigmoid functions will overlap and the sum will have a single peak at an input value of zero. Such a matched pair of neurons generates its maximal activity when R(T)= 0. MSTd neurons have very large receptive fields and do certainly receive input from more than 2 image locations. Also MSTd neurons show the same response in the case of as little as 25 visible moving dots as they do in the case of 300 (Duffyand Wurtz 1991a). We chose each of our model neurons to receive input from 30 image locations. We restrict the space for the encoded heading directions to the innermost 20 x 20” of the visual field, since this approximates the range over which the psychophysical experiments have been carried out. Nevertheless, each layer-two neuron may receive input from a much larger part of the visual field. The layer-two neurons form a three-dimensional grid with 20 x 20 populations encoding one degree of translation-space each, and 20 pairs of neurons in each population. 3 Results 3.1 Comparison of the Network’s Performance with Human Psychophysical Data. The network was tested with simulated flow fields with different motion parameters. We used a cloud-like pattern that consisted of a number of dots, the depths of which were randomly distributed within a given range. To test the behavior without eye movements a translational direction was randomly chosen within the innermost 20 x 20” and the rotation was set to zero. To test cases with eye rotation the translational direction was again chosen randomly and the fixation point was set in the center of the image plane and assigned a specific depth. The rotational component was then calculated from the condition that the flow at the fixation point must be zero. Each simulation run consisted of 100 presentations of different flow fields, after which we calculated the mean error as the mean angular difference between the network’s computed direction and the correct direction.


382

12

1-

-

10 -

1 e

4 -

.

2-

". ........................... 04

0

2

4

6

8

10

12

14

16

18

20

Number of flow vectors

Figure 1: Performance with sparse flow fields. The heading error becomes small with as little as 10 vectors. The number of dots necessary is about the same with or without eye movements.

We found the network's performance to be well within the range of human performers (Warren et al. 1988). For pure translation as well as with eye movements the mean error settled between 0.5 and 1.5", showing that the network always has its activity maximum at a position close to the translational direction. Consistent with the experiments of Warren et al. (1988) we found very little influence of speed on the performance of the network. Humans are able to detect their heading with very sparse flow fields consisting of only ten dots (Warren and Hannon 1988,1990). In order to test how many flow vectors are needed in our model under otherwise optimal conditions we made an additional assumption: We assumed that a given pair of vectors in the flow field serves as input to at least one pair of neurons in each population of the output layer. If this were not the case, some populations would receive more information than others and the number of dots neccessary for correct heading estimation would depend on the heading direction. Our assumption ensures that all heading directions are represented equally. Considering the large number of cortical neurons this assumption is biologically reasonable since it would be approximately fulfilled if the number of neurons in the output layer were large. For the simulations, we distributed the connections between input and output neurons in such a way as to fulfill the assumption. The results of the simulations are shown in Figure 1. The cloud of dots extended in depth from 11 to 31 m with a fixation point at 21 m. The translational


383

speed was 2 m/sec. In both the pure translation and in the eye rotation case the network started to detect the heading with the desired accuracy at approximately 10 points, although with eye rotation the error did not quite reach the optimum and continued to decrease as more flow vectors were provided. Mathematically two vectors are sufficient to compute the heading of a purely translational movement (Prazdny 19801, but humans fail to detect their heading with only two visible dots (Warren et al. 1988). Our network does not know a priori if the flow field is generated by a translation alone. It therefore has to rely on the flow pattern and needs about the same number of vectors as with the eye movements. Humans also fail when eye rotations are paired with a perpendicular approach to a solid wall, where all points are at the same depth (Rieger and Toet 1985; Warren and Hannon 1990). In this case the subjects' performances are at chance and they often report themselves as heading toward the fixation point. Because of a well-known ambiguity in planar flow fields (Tsai and Huang 19841, we were not able to test the depth dependence of the network with approaches to a plane at different angles. We therefore varied the depth range of the cloud. Doing this revealed that with decreasing depth the peak in the second layer grows broader and covers the fixation point as well as the heading direction. This can be seen in Figure 2 where the summed population activities in the output layer are shown on a grayscale map, together with reduced pictures of the input flow fields. Input and output are compared for situations that differ in the amount of depth in the image. In Figure 2a a flow field is shown in which the depth range of the cloud of dots is large, extending from 7 to 30 m. The observer moves toward the cross while he is keeping his eyes fixed on an object ( x ) in the center. There is no apparent focus of expansion. The network output (Fig. 2b) shows an easily localizable brightness peak in the upper left that corresponds to the correct heading direction as indicated by the cross. Figure 2c shows the same movement as Figure 2a, but here the depth range of the cloud is much smaller, ranging from 19 to 21 m. In this case the flow field looks very much like an expansion centered at the fixation point. In the corresponding network output (Fig. 2d), the peak is very broad and includes the fixation point in the center. A maximum nevertheless still exists, although much less pronounced, and in the simulations the network was still able to compute the right heading. However, the solution is unstable and very sensitive to noise. To illustrate this, we randomly varied the amplitudes of the flow vectors by stretching them by a factor distributed uniformly between 0.9 and 1.1, thus adding 10% noise. The results for all conditions are shown in Figure 3 for different depth ranges. This small amount of noise increases the error for the rotational movement to around 7", whereas in the purely translational case the network performance is unaffected. With growing depth differences this separation becomes less pronounced and the error values for the rotational case decrease.

384


Figure 2 Influence of image depth on the heading judgment of the network. (a) Depth-rich flow field. Movement is toward the cross (+) while the x in the center is hated. (b) Output of the network. The response peak gives the correct heading. (c) Same movement with only little depth differences. (d) Brightness maximum in the output of the network is very broad and includes the fixation point. 3.2 Comparison with Single Cell Properties in MSTd. The output layer cells of our model network exhibit a remarkable resemblance to some triple component neurons in MSTd. Figure 4 shows the response of one output layer cell to presentations of each of the components (e.g., expansions, rotations) at different places in the visual field. The neuron


385

-

.......... ..........

2-

Eye8 lixed on wga

. no nOiM

No aye movaments . no noiw E u a lixed on w0.1 - 10% noiw

.....

I....

'i.,

............................................................

* ..............................

Figure 3 Heading error versus depth. In the noise-free condition, heading calculation is accurate despite the broad peak in the network output depicted in Figure 2d. Adding a small amount of noise, however, shows that the solution in the eye movement case is unstable and gives rise to a large error. receives input from 30 positions distributed inside a 60 x 60" receptive field centered in the lower right quadrant of the visual field and extending u p to 10" into each of the neighboring quadrants, thus including the vertical and horizontal meridians and the fovea or area centralis (Fig. 4a). This receptive field characteristic is common for MSTd neurons ( h f f y and Wurtz 1991b). The neuron in our example is a member of the population that represents a heading direction in the upper right quadrant at an eccentricity of 11". Figure 4b shows the cell's broad unidirectional tuning and little selectivity for stimulus speed. The plots c-f in Figure 4 illustrate the responses of the neuron to expansions, contractions, clockwise rotations, and counterclockwise rotations, respectively. The ( x , y)-plane represents a visual field of 100 x loo", the height is the response of the neuron to a stimulus centered at (x,y). The size of the stimulus was always large enough to cover the whole receptive field of the cell. For a stimulus in the center of the visual field the cell responds favorably to counterclockwise rotations and expansions, although there also is a smaller response to contractions. There are very large areas of position invariance covering almost half of the visual field for a given stimulus movement. The response to counterclockwise rotations, for instance, is constant in most of the upper two quadrants. The cell also shows the reversals in selectivity observed in 40% of triple-component neurons in MSTd (Duffyand Wurtz 1991b). In our example, moving the center of the stimuli to the right causes the response

386


Figure 4 Responses of one output layer cell. (a) Receptive field of the cell as defined by its input connections. (b) Broad unidirectional response to global shifts of a dot pattern. No tuning to a particular stimulus speed. (c-f) Responses to expanding, contracting, and rotating patterns centered at different positions within the visual field reveal large areas of position invariance and sudden reversals of selectivity.


387

to contractions to disappear. Moving the center of the stimuli to the lower left causes the cell's selectivity to change to favor contractions and clockwise rotations. There are intermediate positions where the cell responds to both modes of one component. For example, in plots b and c, there is a vertical strip in the center where the cell responds to expansions as well as to contractions. The response reversals take place along edges running across the visual field, which is similar to the findings of h f f y and Wurtz (1991b). The reason for this is that the residual function, which is computed by the neuron, equals zero along a line in the (Tx,Ty)space, as mentioned before. The edge of the surface that marks the neuron's response to expansions follows this line. The neuron signals only that the heading direction lies somewhere along the edge. The edges of all neurons in one population overlap at the point that corresponds to the heading represented by that population, When the network is presented with a flow field, the population encoding the correct heading is maximally excited since all of their neurons will respond. In populations representing other directions, only part of the neurons will be active, so that the total activity will be smaller. It is worth noting that the edges of reversal do not necessarily cross the receptive field of the cell. In the example of Figure 4, the reversal from selectivity for expansion to selectivity for contraction takes place in the left half of the visual field outside the cell's receptive field, which occupies the lower right quadrant. Likewise, it sometimesoccurred in the simulations that the reversal for rotation was not even contained within the 100 x 100" visual field. Another interesting observation is that the edges for rotation and expansion/contraction often cross each other approximately orthogonally. The position of the intersectionpoint, on the other hand, can vary widely between cells.

4 Discussion

We have designed a neural network that detects the direction of egomotion from optic flow and is consistent with recent neurophysiological and psychophysical data. It solves the traditional problem of eye movements distorting the radial flow field by means of a biologically reasonable mechanism. The model reproduces some key properties of human ego-motion perception, namely, the ability to function consistently over a range of speeds, the ability to work with sparse flow fields, and the difficulties in judging the heading when approaching a wall while moving the eyes. The network also generates interesting neuronal properties in its output layer. Simple intuitive models for heading perception might expect a

388

Markus Lappe and Josef P.Rauschecker

single neuron to show a peak of activity for an expansion at a certain preferred heading direction. Instead, our model uses a population encoding in which single cells do not carry all the information about the perceived heading, but rather the combined activity of a number of cells gives that information. At the level of a single neuron, the position information is contained in the edges of reversal of the cell's preferred direction of stimulus motion. The resulting characteristics of the output neurons in our network show great similarity to the response properties of a particular cell class recently described in MSTd, the triplecomponent neurons (Duffy and Wurtz 1991a,b). These cells, which comprise about one-third of all neurons in MSTd, display selectivity not only for expansion or contraction, but also for one type of rotation and one direction of shifting patterns. Most of the neuronal outputs produced by our network have similar properties. It appears tempting to postulate, therefore, that the output of triple-component cells could be used to compute directional heading, either within MST or in another area. A potential problem for using the output of MSTd neurons to compute heading direction concerns their apparent position invariance. In a neural network that is supposed to signal the directional heading, the response of the output layer cells has to depend on the position of the stimulus in some way. Most neurons in MSTd seem to be insensitive against changes of stimulus position, although the proportions of position invariant cells reported in different studies vary and obviously depend on the exact stimulus paradigm (Andersen etal. 1990; Duffy and Wurtz 1991b; Orban et al. 1992). In our network model many output neurons would appear position invariant when tested over a limited, wide range of stimulus positions. Interestingly, the proportion of position dependent responses seems to be highest among triple-component neurons (Duffy and Wurtz, 1991b): In about 40% of these cells component selectivity for a flow field stimulus is reversed along oriented edges, which conforms exactly with the behavior of our model neurons. It is conceivable, therefore, that it is this subtype of triple-component neurons that is involved in the computation of heading direction. More neurons of this type might be encountered in MSTd if one specifically looks for them. Their frequency of occurrence may depend on laminar position, or they might be found even more frequently at another processing stage. A closer look at the experimental data reveals that the number of triple component cells in MSTd may indeed have been underestimated. The different cell types in MSTd do not fall in strictly separate classes but rather form a continuum changing smoothly from triple to single component cells (Duffy and Wurtz 1991a). Therefore, double and single component cells might be regarded as possessing some, albeit weak, responses to the other components. It is equally possible, however, that single and double component cells simply do not participate in the detection of heading direction, but serve some other purpose. Single com-


389

ponent cells, for example, could be involved in the analysis of object motion. The network can also generate cells that are selective to fewer components when the restriction is removed that rotations are due to the fixation of an object. Allowing arbitrary rotations, including ones around a sagittal axis through the eye, results in neurons that are unselective for rotations and respond only to translations and expansions/contractions. Under the different assumption that only frontoparallel rotations, including for instance pursuit eye movements, will occur, the neurons show strong, fully position invariant responses to rotational stimuli, which dominate over the selectivity for translation and expansion/contraction (Lappe and Rauschecker 1993). We would like to emphasize that the neurons in our model do not decompose the flow field directly. At no point is the translational part of the optic flow actually computed. The neurons rather test the consistency of a measured optic flow with a certain heading direction. In this way, a response selectivity for rotations, for example, does not mean that the neuron is actually tuned to the detection of a rotation in the visual field, but this property rather has to be regarded as the result of a more complex selectivity. The cells in the output layer of our model form a computational map of all possible heading directions. However, it would not be easy to find this map in an area of visual cortex, since the topography reveals itself only in the properties of cell populations. Simultaneous recording from an array of electrodes would perhaps be the only way to demonstrate this computational map experimentally. Our model suggests that one has to focus on the mapping of selectivity reversals and explore these more thoroughly, especially in triple component cells: Neurons in neighboring columns should show smooth shifts of their preferences. The concurrent activity of such cells in a hypercolumn would signal one particular heading direction in space, which is given by the intersection point of their reversal edges for expansion and contraction.

References Albright, T. D. 1989. Centrifugal directionality bias in the middle temporal visual area (MT) of the macaque. Visual Neurosci. 2, 177-188. Allman, J. M., and Kaas, J. H. 1971. A representation of the visual field in the caudal third of the middle temporal gyrus of the owl monkey (Aotus trivirgatus). Brain Res. 31, 85-105. Andersen, R., Graziano, M., and Snowden, R. 1990. Translational invariance and attentional modulation of MST cells. SOC.Neurosci. Abstr. 16, 7 . Brenner, E., and Rauschecker, J. I? 1990. Centrifugal motion bias in the cat’s lateral suprasylvianvisual cortex is independent of early flow field exposure. J. Physwl. 423,641460.

390


Bruss, A. R., and Horn, B. K. I? 1983. Passive navigation. Computer Vision, Grahics, Image Process. 21, 3-20, Bulthoff, H., Little, J., and Poggio, T. 1989. A parallel algorithm for real-time computation of optical flow. Nature (London) 337,549-553. Clare, M. H., and Bishop, G. H. 1954. Responses from an association area secondarily activated from optic cortex. 1.Neurophysiol. 17,271-277. Duffy, C. J., and Wurtz, R. H. 1991a. Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli. J. Neurophysiol. 65(6), 1329-1345. Duffy, C. J., and Wurtz, R. H. 1991b. Sensitivity of MST neurons to optic flow stimuli. 11. Mechanisms of response selectivity revealed by small-field stimuli. 1.Neurophysiol. 65(6), 1346-1359. Gibson, J. J. 1950. The Perception of the Visual World. Houghton Mifflin, Boston. Hatsopoulos, N. G., and Warren, W. H., Jr. 1991. Visual navigation with a neural network. Neural Networks 4(3),303-318. Heeger, D.J., and Jepson, A. 1990. Visual perception of three-dimensional motion. Neural Comp. 2, 129-137. Hildreth, E. C. 1984. The Measurement of Visual Motion. MIT, Cambridge, MA. Koenderink, J. J., and van Doom, A. J. 1981. Exterospecific component of the motion parallax field. 1.Opt. SOC.Am. 71(8), 953-957. Lappe, M., and Rauschecker, J. P. 1991. A neural network for flow-field processing in the visual motion pathway of higher mammals. SOC.Neurosci. Abstr. 17,441. Lappe, M., and Rauschecker, J. P. 1993. Computation of heading direction from optic flow in visual cortex. In Advances in Neural Information Processing Systems, Vol. 5, C. L. Giles, S. J. Hanson, and J. D.Cowan, eds. (in press). Morgan Kaufmann, San Mateo, CA. Livingstone, M., and Hubel, D. 1988. Segregation of form, color, movement, and depth: Anatomy, physiology, and perception. Science 240, 740-749. Longuet-Higgins, H. C., and Prazdny, K. 1980. The interpretation of a moving retinal image. Proc. R. SOC. London B 208,385-397. Mishkin, M.,Ungerleider, L. G., and Macko, K. A. 1983. Object vision and spatial vision: Two cortical pathways. Trends Neurosci. 6,414-417. Orban, G. A., Lagae, L., Verri, A., Raiguel, S., Xiao, D.,Maes, H., and Torre, V. 1992. First-order analysis of optical flow in monkey brain. Proc. Nutl. Acad. Sci. U.S.A. 89, 2595-2599. Palmer, L. A., Rosenquist, A. C., and Tusa, R. J. 1978. The retinotopic organization of lateral suprasylvian visual areas in the cat. 1. Comp. Neurol. 177, 237-256. Prazdny, K. 1980. Egomotion and relative depth map from optical flow. Biol. Cybern. 36,87-102. Rauschecker, J. P., von Griinau, M. W., and Poulin, C. 1987a. Centrifugal organization of direction preferences in the cat’s lateral suprasylvian visual cortex and its relation to flow field processing. J . Neurosci. 7(4),943-958. Rauschecker, J. P., von Griinau, M. W., and Poulin, C. 1987b. Thalamocortical connections and their correlation with receptive field properties in the cat’s lateral suprasylvian visual cortex. Exp. Brain Res. 67, 100-112.


391

Regan, D., and Beverly, K. I. 1982. How do we avoid confounding the direction we are looking and the direction we are moving? Science 215, 194-196. Rieger, J. H., and Lawton, D. T. 1985. Processing differential image motion. J. Opt. SOC.Am. A 2, 354-360. Rieger, J. H., and Toet, L. 1985. Human visual navigation in the presence of 3-D rotations. Biol. Cybern. 52, 377-381. Stone, L. S., and Perrone, J. A. 1991. Human heading perception during combined translational and rotational self-motion. In SOC. Neurosci. Abstr. 17, 857. Tanaka, K., and Saito, H.-A. 1989a. Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clustered in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neuropkysiol. 62(3), 626-641. Tanaka, K., and Saito, H.-A. 1989b. Underlying mechanisms of the response specificity of expansion/contraction and rotation cells in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neuropkysiol. 62(3), 642-656. Toyama, K., Fujii, K., and Umetani, K. 1990. Functional differentiation between the anterior and posterior Clare-Bishop cortex of the cat. Exp. Brain Res. 81, 221-233. Tsai, R. Y., and Huang, T. S. 1984. Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces. IEEE Trans. Pattern Anal. Machine lntelligence 6, 13-27. Ungerleider, L. G., and Mishkin, M. 1982. Two cortical visual systems. In Analysis of Visual Behavior, D. J. Ingle, M. A. Goodale, and R. J. W. Mansfield, eds., pp. 549-586. MIT Press, Cambridge, MA. Wang, H. T., Mathur, 8. P., and Koch, C. 1989. Computing optical flow in the primate visual system. Neural Comp. 1(1), 92-103. Warren, W. H., Jr., and Hannon, D. J. 1988. Direction of self-motion is perceived from optical flow. Nature (London) 336, 162-163. Warren, W. H., Jr., and Hannon, D. J. 1990. Eye movements and optical flow. J. Opt. SOC.Am. A 7(1), 160-169. Warren, W. H., Jr., Morris, M. W., and Kalish, M. 1988. Perception of translational heading from optical flow. J. Exp. Psyckol.: Human Percept. Perform. 14(4), 646460. Waxman, A. M., and Ulllgan, S. 1985. Surface structure and three-dimensional motion from image flow: A kinematic analysis. lnt. J . Robotics Res. 4, 72-94. Yuille, A. L., and Grzywacz, N. M. 1988. A computational theory for the perception of coherent visual motion. Nature (London) 335, 71-74. Zeki, S., and Shipp, S. 1988. The functional logic of cortical connections. Nature (London) 335,311-317. Received 7 January 1992; accepted 2 October 1992.

This article has been cited by: 2. Frank Bremmer, Michael Kubischik, Martin Pekel, Klaus-Peter Hoffmann, Markus Lappe. 2010. Visual selectivity for heading in monkey area MST. Experimental Brain Research 200:1, 51-60. [CrossRef] 3. Douglas A. Hanes, Julia Keller, Gin McCollum. 2008. Motion parallax contribution to perception of self-motion and depth. Biological Cybernetics 98:4, 273-293. [CrossRef] 4. Jason T. Richards, Ajitkumar P. Mulavara, Jacob J. Bloomberg. 2004. Postural Stability During Treadmill Locomotion as a Function of the Visual Polarity and Rotation of a Three-Dimensional Virtual EnvironmentPostural Stability During Treadmill Locomotion as a Function of the Visual Polarity and Rotation of a Three-Dimensional Virtual Environment. Presence: Teleoperators and Virtual Environments 13:3, 371-384. [Abstract] [PDF] [PDF Plus] 5. Markus Lappe, Charles J. Duffy. 1999. Optic flow illusion and single neuron behaviour reconciled by a population model. European Journal of Neuroscience 11:7, 2323-2331. [CrossRef] 6. Seth Cameron , Stephen Grossberg , Frank H. Guenther . 1998. A Self-Organizing Neural Network Architecture for Navigation Using Optic FlowA Self-Organizing Neural Network Architecture for Navigation Using Optic Flow. Neural Computation 10:2, 313-352. [Abstract] [PDF] [PDF Plus] 7. Richard A. Andersen, Lawrence H. Snyder, David C. Bradley, Jing Xing. 1997. MULTIMODAL REPRESENTATION OF SPACE IN THE POSTERIOR PARIETAL CORTEX AND ITS USE IN PLANNING MOVEMENTS. Annual Review of Neuroscience 20:1, 303-330. [CrossRef] 8. A V. van den Berg, J A. Beintema. 1997. Motion templates with eye velocity gain fields for transformation of retinal to head centric flow. NeuroReport 8:4, 835-840. [CrossRef] 9. J-C Hsu, R-H Lin, E C Yeh. 1997. Vision-based motion measurement by directly extracting image features for vehicular steering control. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 211:4, 277-289. [CrossRef] 10. Markus Lappe. 1996. Functional Consequences of an Integration of Motion and Stereopsis in Area MT of Monkey Extrastriate Visual CortexFunctional Consequences of an Integration of Motion and Stereopsis in Area MT of Monkey Extrastriate Visual Cortex. Neural Computation 8:7, 1449-1461. [Abstract] [PDF] [PDF Plus] 11. Ruye Wang . 1995. A Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTdA Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTd. Neural Computation 7:2, 290-306. [Abstract] [PDF] [PDF Plus]

12. Markus Lappe, Josef P. Rauschecker. 1995. Motion anisotropies and heading detection. Biological Cybernetics 72:3, 261-277. [CrossRef]

Communicated by Richard Durbin and Graeme Mitchison

Arbitrary Elastic Topologies and Ocular Dominance Peter Dayan Computational Neurobiology Laboratory, The Salk Institute,

PO.Box 85800, San Diego, CA 92186-5800 USA The elastic net, which has been used to produce accounts of the formation of topology-preserving maps and ocular dominance stripes (OD), embodies a nearest neighbor topology. A Hebbian account of OD is not so restricted-and indeed makes the prediction that the width of the stripes depends on the nature of the (more general) neighborhood relations. Elastic and Hebbian accounts have recently been unified, raising a question mark about their different determiners of stripe widths. This paper considers this issue, and demonstrates theoretically that it is possible to use more general topologies in the elastic net, including those effectively adopted in the Hebbian model. 1 Introduction

Durbin and Willshaw’s (1987) elastic net algorithm for solving the traveling salesperson problem (TSP) is based on a method for developing topology-preserving maps between the eye and brain (or lateral geniculate nucleus and cortex) due to von der Malsburg and Willshaw (1977) and Willshaw and von der Malsburg (1979). The elastic algorithm inspired a host of similar ones aimed at different optimization tasks, one of which is this topology problem, augmented by two associates-forming ocular dominance stripes and orientation selective cells (Goodhill and Willshaw GW 1990; Durbin and Mitchison 1990). SimiL. (1990, 1991) and Yuille (1990) looked at the relationship between elastic algorithms and Hebbian inspired ones (Hopfield and Tank 19851, showing that both mechanisms could be viewed as optimizing the same functions, albeit implementing differently the constraints (for the TSP, that each city should be visited exactly once). More recently, Yuille, Kolodny, and Lee (YKL 1991) repeated the feat and aligned elastic and Hebbian (Miller, Keller, and Stryker, MKS 1989) accounts of ocular dominance. The elastic net for the TSP consists of a set of points on a computational rubber band, pulled by forces toward the cities that have to be visited and by tension. The energy in a stretched rubber band is proportional to the square of its extension, which is incorrect for modeling the length of a tour (proportional just to the extension, in this model), Neural Computation 5, 392-401 (1993) @ 1993 Massachusetts Institute of Technology

Arbitrary Elastic Topologies

393

but Durbin (cited as a personal communication in Yuille 1990) suggests that changing the elastic net to use the absolute distance rather than its square is infelicitous. Hopfield and Tank’s (1985) model does in fact use the actual distances, and so, as they lucidly discuss, SimiC‘s and Yuille’s match between the elastic and Hebbian algorithms is not perfect. The nature of the topologies is even more mysterious in the match between Hebbian and elastic algorithms for ocular dominance. Topology enters MKS’ model through a cortical interaction function, which involves more than just the nearest neighbors. Conversely, these are the natural topology for the elastic version. This is one factor leading to an apparent difference between the predictions of MKS and GW. MKS suggested that the width of ocular dominance stripes is governed by the width of the cortical interaction function, whereas GW predicted that it is dependent on the relative correlations within and between the two eyes. This paper considers the issue by examining the two models of ocular dominance. The next section reviews YKLs analysis, and Section 3 looks at generalizing the nearest neighbor topology, testing the generalization in a onedimensional version of ocular dominance. 2 Yuille, Kolodny, and Lee’s Analysis

YKL un* the two models through the medium of a single cost function, which defines a generalized deformable model:’ E [VL,V R ,Y]

where V$ and V: are the variables matching the ith unit in the left and right eyes (more correctly lateral geniculate nucleus layers), respectively, to the ath unit in the cortex, xf and x; are the retinal “positions” of the ith unit in the left and right eyes, ya is the “position” of the ath unit in the cortex, and Y I {yo}. As GW and YKL say, these “positions” are defined somewhat abstractly; however, they are intended to capture something like the correlations between the firings of the retinal and cortical units. u is a constant that governs the relative weighting of the first term, which measures how close, correlationally, matching cells are, and the second term, which measures how close neighboring cortical cells are. This cost function owes its power of unification to having both matching V and continuous Y variables. The constraint on both retinal and cortical fields on a solution-that each cell should have a unique partner-is effectively duplicated in these ‘For convenience, this paper will look at the onedimensional versions of the various tasks. Extensions to the second dimension are straightforward, but messy, Also, YKL separate out the retinotopy dimension-whereas it is incorporated here into the continuous variables x and Y. MKS arbor functions are also neglected.

Peter Dayan

394

two sets of variables.* Minimizing E subject to these constraints leads to the optimal map. Hebbian and elastic methods are effectively different ways of minimizing this function, imposing different constraints in different manners on the way to deriving a solution. Both use Hopfield and Tank's key insight for the TSP that the constraints need not all hold throughout the optimization process, so long as they are guaranteed to be satisfied by the time the algorithm terminates. The reduction to an elastic net comes from eliminating the V Land V R variables using a Gibbs trick. The probability of a particular assignment of V and Y is declared to be proportional to e-flEIVL~VR~~, and these terms are summed over the set of V Rand V Lthat satisfy the partial constraint that each cell in the cortex maps to a unit in either the left or the right eye, but not both. The resulting elastic energy function is3

v

+5

c IYrl - Ya+*12

(2.2)

a

Note that the topology term survives this reduction intact, since it does not depend on the V . The alternative to eliminating the V variables is to eliminate the Y. YKL do this by regarding E[VL,VR,Y] as a quadratic form in Y, which has a minimum at Y,[VL,VR].Imposing the normalization constraint (see MKS) that each cortical cell receives a constant weight from the retina:

gives

where X L = {#} and X R = {xf}, and

21n terms of these variables: for each a, one of the collection over i of { V i ,V j } should be 1 and all the vt 0, and for each i the same should be true of the collection,overa. Also, for each a, ya should be the same as one # or .;",and for each i, there should be different 111 and a2 such that yal = # and y, = .;". 3Here and throughout, boundary conditions are avoided by assuming toroids and using modulo arithmetic.

Arbitrary Elastic Topologies

395

embodies the toroidal nearest neighbor topology. Therefore, at the minimum

1

+

YT = (2 q - 1 (ULXLT + URXR’ where the inverse exists for v > 0. Substituting back into equation 2.1, imposing the constraints in equation 2.3,and ignoring terms that do not depend on the U,gives

MKS’ Hebbian system regards the output oa of cortical cell a as coming partly from the input from the two eyes iL and iR through the connection matrix, [ULiL URiR],,and partly from the other cortical cells [Dola(MKS call C = (Z- D)-’the cortical interaction function):

+

o = VLiL+uRiR +DO

(W+ uRiR)

= (1 - D)-’

(2.5)

Hebbian learning changes U$ proportional to (oaik), where the angle brackets represent an averaging process. Defining Djk LL - ( i4j.i*L k), = (iFi;), and DF = (iii,”)

qt

YKL show that the U are moving down the gradient of

+

1

R RDRR

- D), vai’bj

ij

L RDLR + 2(Z-D)ab 1vaivbj ij }

(2.6) Compare equations 2.4 and 2.6. YKL argue that for the intent of comparing minima, one can identify K - lxf - x ~ I ’ with D f and similarly for and Df“, for some constant K. Therefore, if -23 = v 7 , these two expressions will have the same interesting minima-so, provided that the constraints are properly satisfied during learning, they should lead to the same ultimate solution: Note that this can make the effective correlations negative at some distance, which, as MKS discuss, allows correlation width to determine stripe width in their model. The cortical interaction function calculated from 7 (using v = 3/41 is shown in Figure 1. Although YKL show that this is enough to pro4YKL actually derive a different condition for matching-that (‘T = (Z- D)-I for some constant 1, the time needed to go away a distance E from the equilibrium x* is given by -Tc. Using equation 3.3, the convergence time step for Tii' is given by log€ log%? T,=--loga

where a =f'(O)

(3.4)

loga So L,(m';a) is estimated by

= 1/,-.

-

log[l + 2 / m ] 2 log u

log m1 loga

log E loga

(3.5)

4-

where E is replaced by E / , , / m since A' converges to when Ti7' becomes zero. For example, when a = 0 . 1 , ~= 0.001, and m1 = 0.1, the approximated convergence time Le(m';a)= 77. This approximation shows a good agreement with numerical simulation results of Figure 2. When Tii' is near Tii,, 5' converges to Tii, and the convergence time

T,

log E log b

= --

log I Tii' - Tii, log b

I

(3.6)

is obtained in the same manner where b = f'(Tii,). Putting ifi, N I/@', Tii' = m l / f i and replacing E by c/,/E, we get L,(m';a) =

log€ log b

--

log(1 - m') log b

(3.7)

and 1 =

-/,

1 +

+

a [ ( T a / ~ / a11

Hereafter, the recalling process will be called successful recall when the network converges to a fixed point m, and failed recall when the network converges to zero. The convergence time difference between successful recall (equation 3.7) and failed recall (equation 3.5) mainly depends on coefficients u and b. Numerical calculations show logallogb x 10 if a < 0.2. So the convergence times of successful recall are much smaller than those of failed recall. For example, when m1 = 0.9, a = 0.1, and E = 0.001, we obtain b = 0.02 and L,(m';a) = 1. When Tii' is near Mc, FI' converges to Tii, or zero. The convergence time Lc(ml;a ) is estimated by the sum of the time needed to move away from Tiic and the time needed to converge to the equilibrium from near

Convergence Time of Associative Neural Networks

469

it. The former time takes large values like equation 3.5 but the region where Lc(m';a ) is large at around 5' > E, is very small. So the "spike phenomena" is observed at around m' = m, shown in Figure 2. The convergence time L,(m'; a ) depends on the accuracy E . The problem is how to choose E to estimate the convergence time of finite size neural networks. The direction cosine m rtakes N + 1 discrete values 1, ( N - 2 ) / N , (N- 4)/N,.. . , -1 when N is finite. So it is sufficient to put E = 1/N. Then, it is easy to see that L,(m';a) is of order log(N) from equations 3.5 and 3.7. Numerical simulations of L,( m'; a ) based on the approximated macroscopic state equations obtained by Amari and Maginu are very similar to those of the layered associative net except for the critical value of a,. In this case, a, is about 0.16 (Amari and Maginu 1988). 4 Simulation Results of Autocorrelation Type Net

In the previous section, it is proved that the convergence time of a failed recall is much larger than that of a successful recall and that the convergence time is of order lo#) where N is the number of neurons of a layered associative net. In this section, it is examined by neural network simulation that these properties still hold even for an autocorrelation type of net, which is the discrete synchronous Hopfield net with Jii = 0 (i = 1,. . . ,N). Figure 3a and b shows the results of convergence times for a = O.OB,O.lO, respectively. The network output is synchronously updated until it converges to a fixed pattern. This criterion of convergence is different from that of the theoretical convergence time discussed. But there is little difference between them. In the figures, 0 indicates a successful recall (m*> 0.9) and + indicates a failed recall. We limit the iteration time to 100 and do not include cases in which the convergence time exceeds this upper limit. We often observe such cases for failed recall of m 1 < 0.4. In other words, the network always converges within an iteration time 100 for successful recall. From these simulation results, the convergence time distinction between successful recall and failed recall holds for this type of nets. And this tendency becomes clearer as a increases. It is concluded that in correlation-based associative networks the convergence time of failed recall is larger than that of successful recall. Figure 4 shows the average convergence times vs. the network sizes (the number of neurons) for a = 0.08. In the figure, 0, +, and 0 are the average convergence times over 100 successful recalls for 0.2 < m 1 < 0.3, 0.4 < m' < 0.5, and 0.8 < m1 < 0.9, respectively. The convergence time is shown to be of order log(N) for large values of m'. But for small values of m', it is not clear that the order estimate still holds. A more precise analysis and simulation for larger size networks are needed.

470

Toshiaki Tanaka and Miki Yamada

Figure 3: (a,b) Computer simulation of convergence time using neural network (N= 1OOO). 0 indicates a successful recall and + a failed recall. The horizontal bar is the initial direction cosine M I , and the vertical bar is the convergence time (iteration time). It is clear that the convergence time of successful recall is smaller than that of failed recall. 5 Discussion

An autocorrelation type associative net has two problems from the information processing point of view. One is that the convergence time depends deeply on the initial values. In some cases they need more than 10 times as much as those of successful recall. The other is that an automatic distinction between successful recall and failed recall needs to store each memorized pattern separately in order to calculate the direction co-

Convergence T i e of Associative Neural Networks

20

471

I

0.2-0.3 0 15

-

0.4-0.5

+

0

0.8-0.9 0

0

T

10

-

-

0

0

0

OO

-

0

0

OO

00

0 0

+++++

5 0 11

0

+0

+

0

+0 +oooo@

+

+ + + + o

0

0

-

on@

I

Figure 4: Average convergence time vs. network size. 0, +, and 0 are the average convergence times over 100 successful recalls for 0.2< m' < 0.3,0.4 < m' > 0.5, and 0.8 < m1 < 0.9,respectively. The convergence time is shown to be of order log(N)for large values of m'. sine between each memorized pattern and an obtained output pattern. Roughly speaking, the first problem is on the execution time, and the second one is on memory. These time and memory problems can be solved using the convergence time property that we have shown. If a network does not converge within a time limit, you can stop the execution of the network and ignore its output. This is because the network is shown to converge to a spurious state when the convergence time is larger than the time limit. On the other hand, if a network converges within the time limit, you can regard its process as successful recall and its output as a meaningful pattern. Acknowledgments We thank Professor S. Yoshizawa of Tokyo University for useful discussions and comments. References Amari, S. 1972. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transact. Computers C-21,1197-1206.

472

Toshiaki Tanaka and Miki Yamada

Amari, S., and Maginu, K. 1988. Statistical neurodynamics of associative memory. Neural Networks 1,63-73. Amit, D. J. 1989. Modeling Brain Function-The World of Attractor Neural Networks. Cambridge University Press, Cambridge. Cottrell, M. 1988. Stability and attractivity in associative memory networks. Biolog. Cybernet. 58, 129-139. Domany, E., Kinzel, W.,and Meir, R. 1989. Layered neural networks. J. Phys. A: Math. Gen. 22, 2081-2102. Florkn, P 1991. Worst-case convergence time for Hopfield memories. IEEE Transact. Neural Networks 2(5), 533-535. Gutfreund, H. 1988. Neural networks with hierarchically correlated patterns. Phys. Rev. A37,570-577. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kohonen, T. 1972. Correlation matrix memories. IEEE Transact. Computers C-21, 353-359. Koml6s, J., and Paturi, R. 1988. Convergence results in an associative memory model. Neural Networks 1,239-250. Meir, R., and Domany, E. 1987. Exact solution of a layered neural network model. Phys. Rev.Lett. 59,359-362. Nakano, K. 1972. Associatron-A model of associative memory. IEEE Transact. Syst., Man. Cybern. SMC-2, 381-388. Parisi, G. 1986. Asymmetric neural networks and the process of learning. J. Phys. A: Math. Gen. 19, L675L680. Received 28 May 1992; accepted 11 November 1992.

This article has been cited by: 2. Zhang Chengxiang , Chandan Dasgupta , Manoranjan P. Singh . 2000. Retrieval Properties of a Hopfield Model with Random Asymmetric InteractionsRetrieval Properties of a Hopfield Model with Random Asymmetric Interactions. Neural Computation 12:4, 865-880. [Abstract] [PDF] [PDF Plus]

Communicated by Gerald Tesauro

Robustness in Multilayer Perceptrons P. Kerlirzin F. Vallet Laborutoire Central de Recherches, Thornson-CSF 92404 Orsuy fcedex),France

In this paper, we study the robustness of multilayer networks versus the destruction of neurons. We show that the classical backpropagation algorithm does not lead to optimal robustness and we propose a modified algorithm that improves this capability.

1 Introduction

The distributed nature of information in neural networks suggests that they might be capable of "robust" computation, that is, a "graceful degradation" of performance with respect to damaged units and connections (Hinton et al. 1986;Le Cun 1987;DARPA 1988).In fact, such robustness is often observed empirically, but is usually uncontrolled. In this paper! we propose a method for optimizing during learning the ability of a network to compute after damage. Robustness to the destruction of neurons can be an interesting property for hardware implementation. One can note also that the problem of missing data corresponds to the destruction of input cells of the network. The degradation of a network due to a loss of precision of the synaptic weights ("clipping") has been studied (Hopfield 1982; Amit et al. 1985; Cottrell et al. 1987;Wong and Sherrington 1989)for associative memories (moderated drop of the capacity) and (Vallet and Cailton 1990)for linear classifiers. Here we study the case of the destruction of hidden and input neurons in a one-hidden-layer perceptron. The destruction of synapses can be tackled with similar solutions. The generalization to the case of a perceptron with more than one hidden layer is easy. Previously we have partially studied the case of linear networks (Kerlirzin 1990).We showed that the robustness optimizing algorithm behaves fairly, confirming theoretical results, and that it optimizes the robustness by distributing information on all the connections of the network. We now address the case of nonlinear networks. Neural Computation 5,473482 (1993) @ 1993 Massachusetts Institute of Technology

P. Kerlirzin and F. Vallet

474

2 Background

The network considered here performs a transformation F from R”to R”, through a hidden layer with p neurons (the output is linear for sake of clarity): P

X + F(X) = z~(’wl,X)W’, =Y i=l

where X is the input vector, Y the output one, o the neural transfer function [tanh(x) for our example), W, the input weight vector linking the input vector to the ith hidden cell, and Wi the output weight vector linking the ith hidden cell to the output vector (Fig. 1). Such a network tries to learn a desired function D (known on pattern examples: the learning set) with as few errors as possible. To achieve this, a cost function E, measuring the distance between the desired function D and the function F performed by the network (on the learning set) is minimized: N

E = C 11 D ( X ” ) - F(X”) [I2 ’”=I

the learning set being composed of N examples Xp. A method widely used to minimize this energy is a stochastic gradient descent procedure (Le Cun 1987; Rumelhart etal. 1986).Each convergence step (elementary learning) consists in modifying the weights W, and Wi in a direction opposite to the gradient, for the elementary contribution E(Xp) of the pattern Xfi to the total cost function E:

E ( X p ) =)I D(Xp) - F(X’”)[I2 We are interested here in partially damaged versions of the network, which implement the functions Fn: for which only a subset K of hidden cells is active:

F K ( X ) = z~(’wl,X)W’, iEE

K being a subset of {1,2,. . . p } , which represents the hidden neurons that are not destroyed. The measure of distance between the desired function D and the damaged one FK: is then N

[I D(Xp)- FK(X’”)It2

EK = p=l

The final goal is to minimize the average cost function, taking into account the probability P ( K ) of each configuration K occurring:

Robustness in Multilayer Perceptrons

input layer

475

hidden layer

output layer

Figure 1: Representation of the contribution of the ith hidden cells. We study here the simple and general example in which each cell has the same probability 7r to be damaged. The probability P ( K ) is thus given by

P ( K ) = (1 - 7r)lK17r(P-lKl)

(2.1)

1x1being the cardinal of K. It is now convenient to minimize the cost function N Ewer =

11 D ( x / ” -) F K ( X / ” )1)’

P(K)

(2.2)

K p=l

In order to minimize it, a stochastic gradient algorithm is used. It consists here in choosing one configuration K [randomly chosen according to the distribution given by P ( K ) l and one example X” (randomly chosen or chosen in a predefined order). A partial cost is defined for this configuration and this example:

Then the weights are modified in the opposite direction to the gradient direction of the elementary contribution E ( K , Xfi). One can note that the Probability P ( K ) is now implicitly taken into account by the distribution of the choice of the configuration K. The proposed algorithm is thus a doubly stochastic algorithm, that is, on examples and on damaged configurations.

476


3 Simulations and Results

The efficiency of the proposed learning algorithm is studied on a realworld problem: the identification of radio-transmitters characterized by relevant features. This a (supervised) classification problem with 9 classes and vectors (patterns) of dimension 23. The learning set contains 810 examples and the test set (for evaluating the generalization rates) contains 400 examples. The network used is totally connected with 3 layers (23 x 16 x 9). This is the one-hidden-layer network that provides the best generalization rate (86.9%), whereas a network without hidden layer provides a generalization rate of 79%. We present here the generalization results (the only interesting ones in practice), the learning results have a similar behavior with higher values. We compare the results between the classical network that has learned without destruction of cells (called "classical") and the "optimized" network for which learning was made by a stochastic gradient with cost function (2.2). The probability of destruction of neurons during the learning phase is 10% (so P ( K ) = 0.91n10.1'6-lKI = 91K1/10'6). The robustness of each network when d hidden cells (d = 0,1,. . .) are destroyed is then measured. For each case we present two generalization rates: the average rate, which is calculated for a given d in taking into account all the possible combinations of d destroyed hidden cells, and the worst rate, which is the smallest rate found in testing all the configurations with d destroyed hidden cells. The proposed algorithm has been studied first with randomly initialized weights. In that case, the proposed algorithm is not efficient and the classical one seems to be robust. Another strategy was studied to obtain an optimized network that is efficient even without destroyed cells. This strategy consists in learning in two steps. The network is first trained with the classical backpropagation algorithm and, second, the robustness optimizing algorithm is used to continue its training. It has been observed that the supplementary learning time is of the same order as the fore-learning time. The obtained results are summarized in Table 1. The conclusions are now clear. First, the classical backpropagation algorithm is not optimal with respect to the destruction of neurons. The proposed algorithm hardly degrades the results from the classical one without destruction. When one considers the destruction of neurons, the worst and average cases are better for the optimized network. Further, given a number of killed cells, the optimized network exhibits a smaller performance standard (the standard deviation of the performance of the network for each possible configuration). This general solution can also be used in the case of a network with several hidden layers: learning is still made with random destruction


477

Table 1: Comparison of Classical and Optimized Networks.

Worst case (%)

0 cell killed (0%) 1 cell killed (6%) 2 cells killed (12%) 3 cells killed (19%) 4 cells killed (25%) 8 cells killed (50%)

Average (%)

Classical

Optimized

Classical

Optimized

86.9 74.1 60.8 45.5

86.4 82.9 74.6 74.3 68.0 36.9

86.9 83.8 79.3 72.8 66.0 43.6

86.4 85.6 84.6 83.2 81.2 64.5

30.1 3.2

of neurons according to destruction laws, given for each layer. This approach should also be valid for the destruction of synaptic weights. It is interesting to study the effect of the mortality rate (probability of hidden neuron destruction during learning, A in equation 2.1) on the robustness of the network. The results are shown in Figure 2: average, worst cases and standard deviation of the generalization rates have been plotted versus the mortality rate A introduced during learning. These curves are shown for 6 different cases of the number d of destroyed cells in test phase: d = 0, 1, 2, 3, 4, and 8. It is interesting to notice that the recognition rates (worst and average cases) are roughly increasing functions of A and that their asymptotic values are reached for a value corresponding to d . Thus, if one wants to optimize a network for a destruction rate of A’, it is necessary to use a learning destruction rate A slightly greater than A’. The value of A need not be very precise, since we have shown that the generalization rate varies slowly with A. Another remarkable point is the decrease of the deviation of the results while A increases: the result with a given number d of destroyed hidden cells hardly depends on the choice of these cells. Finally, the proposed algorithm is able to solve the robustness problem under discussion. For d = 8 destroyed neurons during test, for example, the usual algorithm has an average test rate of 45% versus 75% for the proposed optimized version ( A = 50%). The previously proposed method can easily be applied to the case of the destruction of input cells. From an operational point of view, this is the problem of missing input data. Some data may arrive in an intermittent way or even completely disappear because of the failure of the upper processing step or of an out-of-order sensor. In this case, the aim is to have the most robust network with respect to this type of destruction. We have applied the learning algorithm described, in the case of input cells, to the problem previously described, and we show the results as previously. Figures 3 and 4 represent the generalization performances for the average and worst cases and the standard deviation for five values of the

I? Kerlirzin and F. Vallet

478

Averue nte on tat Ict

3 cells killed 2 cells killed

.....

I cell killed

- 0 cell killed

10

0

20

M

40

so

Word rate on teat r t

I,

. . I

10

.

..-. 8 cells kllled .--.4 calls kllled

-- .'*. 'b.

,

'. \.

.-.I.

.....

3 cells klUd 2 cells kllled I cell kUled

---..-.._.._..-.._. LO

m

M

40

Figure 2: Average and worst case and standard deviation of the generalization rate versus T probability of destruction of a hidden cell during learning. Six cases are represented, corresponding to a different number d of destroyed neurons during test: d = 0, 1, 2, 3,4, and 8.

479


Average rate on test wt (%)

3 cells kllled 2 cells killed

- 0 cell killed 0

10

20

30

40

Worst rate on test set (96)

~

..

-

4 cells kllled (17%) 3 cells killed (t3%) 2 cells killed (9% ) t cell kllled (4%) 0 cell killed ( 0 % )

Learning mortality rate (%) 3 0 , 10

0

20

30

40

Standard deviation (%)

10.0

-

8.0

--'*

2.0

--

4 cells killed 3 cells kllled 2 cells kllled

-. '.

, Learning mortglity rate (%) ...

0.0 0

10

20

30

Figure 3 Missing input data. Average case, worst case, and standard deviation for the generalization rate versus s probability of destruction of an input cell during learning. Five cases are shown, corresponding to a different number d of destroyed neurons during test: d = 0, 1, 2, 3, and 4. Curves correspond to the randomly initialized network.


480

Average rate on tesl set

90

..............................................................................................

86

82

...

--.

........... ............................................................................................ ...........*...... ................................ . .,...' ~

~

78 74

.,..-. -6

.,,,.

./

*.,'

,.*'

-*..

..*.

-.--

.__.4 cells kllled (17%)

....

--,/'

..........

3 cells kllled (13%) 2 cells killed ( 9 % )

-

I cell kllled (4%) 0 cell kllled ( 0 % )

.....

Learnlnp mortallty rate

70

-- n

p

Worst rate on test set .

I ............. ....... ...........................

1

........... .............................. ....... ................... ..-.'.". .................................................... , _...... ................. /,' _._._........ .. //.. ,,......... ..... ._-. 4 cells kllled ..... 3 cells kllled .... *........... 2 cells killed

8 O - t

70 60

so 40

~

~

~

~

--,.., '

.....

.a..

Learnlnp mortallty rate

3 0 , 0

2

--

10

20

........................................... .........

-

1 cell kllled 0 cell kllled

30

.............................................

.................... Learnlop mortallty rate

0

3

10

20

30

40

Figure 4 Same as Figure 3 with the fore-learning initialized network. number d of destroyed input cells during test. The figures on the left side correspond to the case of a randomly initialized network and those of the right side correspond to the case of an initialization by fore-learning. In opposition to the previous case, the fore-learning case does not seem to give notably better results than the randomly initialized case. For each d in {0,1,2,3,4,8},the value of the generalization rate for 7r = 10% of the


481

randomly initialized case is higher than the corresponding value of the fore-learning case. 4 Conclusions

In this article, we have examined the problem of improving the robustness of a multilayer perceptron versus the destructions of hidden or input cells. We have shown that the classical backpropagation is not optimal when one considers robustness. We have proposed a learning algorithm that takes into account the potential destructions of cells (destructions during learning) and that was evaluated on a real example. The case of destroyed input cells seems different from the one of hidden cells. In the first case, a random initialization and a weak destruction of neurons during learning seem to give better results. In the second case a fore-learning initialization and a strongly damaged learning seem to be better. Our first results are very encouraging, but other experiments on real data have to be done, especially when the network size is large.

Acknowledgment The authors want to thank P. Gallinari for his support of this work.

References Amit, D. J., Gutfreund, H., and Sompolinsky,H. 1985. Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Lett. 55(14), 1530-1 533.

Cottrell, D. M., Munro, P., and Zipser, D. 1987. Image compression by back-propagation: An example of extensional programming. In Advances in Cognitive Science, Vol. 3, N. E. Sharkey, ed. Ablex, Norwood, NJ. DARPA. 1988. Neural Network Study. AFCEA International Press, Fairfax. Hinton, G. E., McClelland, J. L., and Rumelhart, D. E. 1986. Distributed representations. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. I. Bradford Books, Cambridge, MA. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kerlirzin, P. 1990. Robustesse et capacitk des rkseaux multicouches. Rapport de stage (DEA Paris XI Orsay), LCR ASRF/90-8. Responsable: F. Vallet. Le Cun, Y. 1987. Modiles Connexionnistes de I'Apprentissage. Ph.D. thesis, Universite Pierre et Marie Curie, Paris, France. Rumelhart, D. E., McClelland, J. L., and the PDP research group. 1986. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. II. Bradford Books, Cambridge, MA.

482

I? Kerlirzin and F. Vallet

1990. Recognition rates of the Hebb rule for learning Vallet, F., and Cailton, J-G. boolean functions. Phys. Rev. A 41(6), 30593065, Wong, K. Y. M., and Sherrington, D. 1989. Theory of associative memory in randomly connected boolean neural networks. I. Phys. A 22, 2233-2263.

Received 28 October 1991; accepted 6 August 1992.

This article has been cited by: 2. Gaetano Perchiazzi, Rocco Giuliani, Loreta Ruggiero, Tommaso Fiore, G??ran Hedenstierna. 2003. Estimating Respiratory System Compliance During Mechanical Ventilation Using Artificial Neural Networks. Anesthesia & Analgesia 1143-1148. [CrossRef] 3. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef] 4. Sepp Hochreiter, Jürgen Schmidhuber. 1997. Flat MinimaFlat Minima. Neural Computation 9:1, 1-42. [Abstract] [PDF] [PDF Plus] 5. Terrence L. FineFeedforward Neural Nets . [CrossRef]

Communicated by Richard Lippmann

Pattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior Thorsteinn Rognvaldsson Department of Theoretical Physics, University of Lund, Solvegatan 14 A, $223 62 Lund, Sweden

The discrimination powers of multilayer perceptron (MLP) and learning vector quantization (LVQ) networks are compared for overlapping gaussian distributions. It is shown, both analytically and with Monte Carlo studies, that the MLP network handles high-dimensional problems in a more efficient way than LVQ. This is mainly due to the sigmoidal form of the MLP transfer function, but also to the fact that the MLP uses hyperplanes more efficiently. Both algorithms are equally robust to limited training sets and the learning curves fall off like UM, where M is the training set size, which is compared to theoretical predictions from statistical estimates and Vapnik-Chervonenkis bounds. 1 Introduction

The task of discriminating between different classes of input patterns has proven to be well suited for artificial neural networks (A"). Standard methods, like making cuts or discriminant analysis, are repeatedly being outperformed by nonlinear ANN algorithms, where the most extensively used algorithms are the feedforward multilayer perceptron (MLP) (Rumelhart and McClelland 1986) and the learning vector quantization (LVQ) (Kohonen 1990). Both algorithms have shown good discrimination and generalization ability, although some confusion prevails on their performance on realistic large sized problems, especially concerning their parsimony in parameters to fit data-an important issue when algorithms are transferred to hardware. This paper compares, analytically and with Monte Carlo simulations, the discrimination power of the MLP and LVQ algorithms on separating two gaussian distributions. Two classes are sufficient since the results carry over to problems with more classes. The problem is designed to resemble "real-life" situations with many input nodes and overlapping distributions, making the classification fuzzy. Discrimination is thus only possible down to a minimum error; the Bayes limit (Duda and Hart 1973). It is found, in contrast to previous results (Kohonen et al. 1988; Barna and Kaski 1990), that the MLP is more efficient than the LVQ algorithm on heavily overlapping distributions. The sensitivities of the two algorithms Neural Computation 5,483-491 (1993) @ 1993 Massachusetts Institute of Technology

Thorsteinn Rognvaldsson

484

Table 1: Bayes Limit for the “Hard” and “Easy” Cases for Dimensions 2 5 d 5 8 Dimension

Bayes limit (%)

d

“Hard case ”Easy” case

2 3 4

5 6

7 8

26.4 21.4 17.6 14.8 12.4 10.6 9.0

16.4 13.8 11.6 9.8 8.4 7.2 6.2

to limited training data are also examined and compared to theoretical predictions. 2

T h e Problem

The problem (Kohonen et al. 1988) consists of two overlapping gaussian distributions, PI and P2r of dimensionality d and normalized to unity with standard deviations (TI = 1.0 and 0 2 = 2.0. Two versions are generated; one where the distributions have the same mean, referred to as the “hard” case, and one where their means are separated by a distance El referred to as the “easy” case (notation follows Kohonen et al. 1988):

I &I

Pl(r) = (qd%)-dexp --

(2.1)

(2.2)

> d the end-cap of the cone is a (d - 1)dimensional hexagon, approximately a (d - 1)-dimensional hypersphere, and a will be given by

2 ) , r is the gamma function.' It is assumed where A d = 2 ~ ~ / ~ / I ? ( d /and in the last step that N is so large that a is small. 'Ad comes from the surface area A = Ad#-' radius r.

of a hypersphere of dimension d and

Thorsteinn Rognvaldsson

486

The volumes of the cone and the sector are given by (3.3) (3.4)

J2d

where R = ln[q/u1]u:u;/(u; - u:) is the radius of the optimal classification hypersphere. The deficit volume is approximated by expanding in power series

where equation 3.2 has been used in the last step. If AV is not too large, the density of patterns inside can be assumed constant. For u1 < a2 and AR &, where

~

Figure 2 Facing page. Aspects of the dynamics of a single cluster. (a) A state diagram of the output of the network. The fixed parameters for the network were JE = 15, J E ~= 12, xo = 1.1, /3 = 3, T = 0, and E = 0 and the stimulus profile was 1(0 - 00) = Il + (Ih - II)l0 - e01/(~/2)with - It = 3.5. The value of Jr and the average input I = 1/2(4 + 11) were varied. The boundaries of existence of the different states were determined from numerical simulations of equations 3.3 and 3.4. However, since 0 is large, the boundaries for the " O N or "OFF"states are approximately straight lines that can be determined from equation 3.1 by a consistency analysis. In an "OFT" state the maximum potential of the excitatory neurons must be less than XO. Since v ( 0 ) = I ( 0 ) in this state, we require maximum[I(0)]< XO, which yields the vertical line. In the " O N state the minimal potential has to be larger than XO. In this state v(0) = JE Jr I(0) i, = 0 and V = U = 1, which leads to JE JI minimum[I(0)] 21 xo and yields the oblique line. The asterisk corresponds to the values J, = -7 and I = -0.25 used in the simulations for b-c. (b) The average firing rate of two excitatory neurons (upper panel) and the inhibitory neuron (lower panel) found from a simulation of the equations (equation 2.1) for a network with 60 neurons using the above parameters, except that we include noise amplitude T = 0.0006 (equation 2.4). We chose the initial conditions such that the neurons did not get stuck in an " O N state. The heavy line for the excitatory neurons refers to one with an orientation preference 0 - 00 = 3' while the thin line refers to one with 0 - 00 = 27". The period is 2x/w = 3.4. (c) The tuning curve, or time-averaged output of an excitatory neuron as a function of its orientation preference relative to the orientation of the stimulus. The average was calculated from simulations of the network with an averaging time of approximately 20 periods. The dots indicate the orientation preferences of the excitatory neurons featured in b.

+ +

+ +

E. R. Grannan, D. Kleinfeld, and H. Sompolinsky

558

0, is the full extent of the tuning curve (0, = 80" in Figure 2c). This can be understood by recalling that only neurons with the same orientation preference are connected by the intercluster couplings. Thus for AOo > 0, there are no pairs of active neurons that have the same orientation preference and the effective interaction between the clusters must vanish. Second, r(A$; Ado) is not monotonic in A&. Third, r(A$; A&,) is not an odd function of A$. This implies that the phase equations cannot be described in terms of a potential, that is, the interaction terms in equation 4.2 cannot be written in the form SW/B$R. This has important consequences on the dynamics of a network with many stimulated clusters (Discussion). The form of r(A$; Ado) indicates that it contains significant contributions from high harmonics in A$ (Figure 3a). We find that a good approximation is F(A$; A0,)

x

r0+ rl sin(A$ + a l )+ I'2sin(2A$ + (12)

(4.3)

as illustrated for A& = 15" in Figure 3a. The zeroth harmonic, with amplitude ro, represents the shift in the period of the oscillations and is presently irrelevant. The amplitudes rl and I'2 decrease monotonically with AOo and are zero for A00 > 0, (Fig. 3b). An unexpected result is the presence of large phase parameters, 01 and 02. They are nonzero even at A00 = 0" and increase with increasing values of A&. The nonzero value of the phase parameters appears to originate from the inhibitory feedback within each cluster. Specifically, the long-range excitatory synaptic input to weakly active neurons, that is, those stimulated away from their preferred orientation, increases the activity of these neurons and, in turn, increases the activity of the inhibitory neuron. This indirect activation of the inhibitory neuron contributes an "inhibitory" component to the interaction between the clusters. For sufficiently large values of AOo, all connections between a pair of clusters involve weakly active neurons and the inhibitory contributions dominate r(A$; A&). 4.3 Dynamics of 'Itvo Interacting Clusters. We consider in detail the case of only two interacting clusters, for which one can subtract the equation of $ R ' ( t ) from that for $R(t) (equation 4.2) to obtain an equation for the phase difference A$(t):

A$(t)

=

-F(A$;AOo)

+ V(t)

(4.4)

where f'(A.ri,;A00)= r(A$;A&) - F(-A$;A&) and V ( t ) = &(f). In the absence of noise, the steady-state solution of equation 4.4 is a fixed point and the phase difference between the two clusters will approach is determined from the constraints a constant, A&. The value of f'(A$o;A60) = 0 and Sl='(A$o;A8~)/8(A$)> 0. Note that f'(A$;A&)

Synchronization of Neuronal Assemblies

559

Kl i;m

0.4 W

n 3 k 1

a r a

a4

0

0"

45" RELATIVE ORIENTATION, As, 45"

90"

0"

90"

Figure 3 The interaction in a network of weakly coupled clusters. (a) The form of the effective long-range interaction between pairs of phase variables as a function of the relative difference in their phase (equation 4.2). Each curve corresponds to a different value of A&, the relative orientation of the stimuli. Note that the curves for A80 = 75" and 90" are essentially flat. The open symbols correspond to the form of F(A$J;15') given by its first three harmonics (equation 4.3).(b) The dependence of the amplitude and phase (radians) parameters for the first two harmonic of the interaction (a) on 0, (equation 4.3).

560


is an odd function of A$. Thus A$ = 0 and x are always zeros of f'( A$; A&), although they are not necessarily stable solutions. Further, the fixed points form degenerate pairs, &A$,,, that, except for the cases A,~J, = 0 or x , correspond to two different states. The shape of F(A$; A&) is shown in Figure 4a for several values of A&,. For small values the stable state is A& = 0. As A00 is increased (beyond 6" for our parameters) the fixed point moves to a nonzero, intermediate value of A$,,. As A00 is further increased (beyond 36") the stable fixed point becomes A&, = x and remains so until the force vanishes (beyond BOO). This behavior can be qualitatively understood by approximating F(A$J;A&)in terms of its first two harmonics (equation 4.31, that is, F(A$;AOo) x 2r1COSNI sin(A$) 2r2 cosa2sin(2A$). When the first harmonic dominates the interaction, as occurs when the value of aI is substantially smaller than x/2 and is substantially larger than r2,the phase difference is zero. This situation corresponds to small values of A0, (Fig. 3b and c). Similarly, a value of a1 near x leads to a phase difference of x , as occurs for large values of A&. When the value of a1 is near x / 2 , corresponding to intermediate values of Aso, the contribution from the first harmonic is of the same magnitude of that from the second. This gives rise to the pronounced anharmonic shape of f'( A$; A8) (Fig. 4a) and to an intermediate phase shift A$,, w cos-'( 4'1 cos 4 2 r 2 cos 0 2 ) . In the presence of noise, ij(t) in equation 4.4, the phase difference A$ fluctuates in time rather than approach a fixed value. The average phase coherence between the two clusters can be expressed by the intercluster correlation function

+

(4.5) with bVR(f)= V R ( ~ )(VR(t))fand, as before, denotes an average over time. The correlation function can be calculated from the unperturbed limit cycle of a single cluster and the phase dynamics (Appendix B). Since the clusters are identical, and the interaction between them is symmetric, an extremum will always occur at 7 = 0. The correlation function for several values of A00 are shown in Figure 4b-d. When both stimuli are aligned, that is, A& = 0", the Figure 4 Facing page. Aspects of the dynamics in a network with two clusters. (a) The force that acts on the difference between two clusters as a function of their relative phase difference (equation 4.2). Each curve corresponds to a different value of A&; those for A& = 75" and 90"are essentially flat. (b-d) The intercluster correlation function of the phase differencebetween two clusters as a function of time and different values of A00 (equations4.5 and B.4). The thin line refers to a low level of noise, 1/T = 33, and the thick line refers to an intermediate level, 1/T = 3.3. The network was equilibrated for approximately 150 periods and the correlation functions were averaged over an additional 150 periods.

Synchronizationof Neuronal Assemblies

561

correlation has a prominent peak at T = 0 (Fig. 4b). As the relative angle between the stimuli is increased, C M ~ ( Tdevelops ) a double peak that reflects the fluctuation of the network between two stable intermediate phase shifts. The presence of these phase-shifts also causes a minimum to occur at T = 0. These features are seen at A& = 30", for which

m0

I

I

I

-2a 0.4

I

I

'".

z 0

6 g z

o

a

W

$ I

3

j -0.4 a W

I-

z

a

8

0

21T

lT PHASE DIFFERENCE,A$

1.0

a W

k 3

cn

J

0 a

0

W

I-

I -1

0

1

-1

0

TI ME, T (Units of 27r/w

1

1

562


A$o N f1.3 (Fig. 4c). Note that, in practice, the intermediate phase shift may show up as a single peak at either a positive or a negative value of T if the activity of the network is averaged for only a short time. Lastly, the peak amplitude of CRR,(T)is not a monotonic function of A00 (cf. Fig. 4b-e). Further, while noise suppresses the amplitude of C,,(T) for any value of Ado, the suppression is greatest for intermediate values (cf. thick versus thin line in Fig. 4c). These features reflect the nonmonotonic behavior of I?(A$; A&) with respect to A00 (Fig. 3a). We calculated the equal-time correlation coefficient, CRR~(O), as a function of the relative orientation of two stimuli, A&, and for two levels of noise, ? (Fig. 5). The value of the coefficient rapidly decreases as a function of the relative orientation for either case. Beyond approximately the full-width at half-maximum of the tuning curve, 22" for our parameters (Fig. 2c), the coefficient becomes negative as a consequence of the substantial phase-shifts that occur for large values of A&. However, as the magnitude of the interaction is reduced for these angles, the corresponding magnitude of the coefficientsis also significantlyreduced, particularly at high levels of noise. 5 Discussion

Our main result is that a weak, fixed synaptic coupling between clusters of neurons can generate an effective interaction between the phases of their oscillatory response that depends strongly on the distribution of activity within each cluster. Thus the interaction is sensitive to the dissimilarity of the external inputs that stimulate the clusters. This result implies that stimulus-dependent synchronizing connections, postulated ad hoc in a previous network model of phase oscillators (Sompolinsky ef al. 1990,1991), can originate from neuronal dynamics without the need to invoke mechanisms of fast synaptic modification. This conclusion is consistent with the results of Konig and Schillen (19911, who simulated a network with time delayed connections and with the initial reports of Sporns et al. (1989). Our phase description is strictly valid only in the limit of weak intercluster coupling. In practice, the results of numerical calculations of the full equations for the model (equation 2.1) indicate that the phase model qualitatively describes the dynamics of the clusters even when the synaptic input a neuron receives via intercluster connections is about 5% of its total input ( E = 0.021~;data not shown). The time it takes to synchronize the output of two clusters from an initial, unsynchronized state is relatively short, about three cycles for this strength of interaction (insert to Fig. 5). In contrast to the ad hoc assumption in a previous work (Sompolinsky et al. 1990, 19911, the present analysis shows that dissimilarity in the external stimuli for each of the two clusters not only reduces the amplitude


h

O

v

1.0

..h

7 A

I

U

563

h \

0.5

A

Ct-A--LA

ae, oo

A

0 v

i

0

a

0.5 TIME, t (Units of 2 r / w )

1

a

3

8 -0.5-I

I

I

RELATIVE ORIENTATION, Atlo

Figure 5: The equal-time intercluster correlation coefficient for the phases difference between two clusters as a function of A00 (equations 4.5 and B.4 with T = 0). This coefficient is a measure of the discrimination capability of the network. The thin line is for 1/T = 33, while the thick line is for 1/T = 3.3. The inset shows the amplitude of the coefficient during consecutive periods following the presentation of stimuli. Equations 2.1-2.4 were simulated numerically with the parameters used in the phase model (legend to Fig. 2a), l / T = 3.3 and E = 0.021E. Each datum reflects an average over 64 random initial conditions of the network. of their effective interaction but also induces a tendency to form phaseshifts. When only two clusters are stimulated, the phase-shifts appear in the intercluster correlation function (equation 4.5; Fig. 4c-e). Large differences in orientation between the two stimuli result in a phase shift of 7r (Fig. 4d and e). The phase shifts are less than 7r for intermediate differences in orientation and disappear for small differences. Our results with regard to the occurrence of phase shifts are in apparent contradiction to those of Schuster and Wagner (1990a). These authors studied the phase interaction between weakly coupled clusters of neu-

564


rons and claim that significant phase shifts do not occur. In contrast to the present work, the clusters in the model of Schuster and Wagner (1990a) had uniform external inputs and, further, their analysis was restricted to parameters near a Hopf bifurcation where the nonlinearities in the dynamics are weak. Our results are consistent with the simulations of Schillen and Konig (19911, where phase-shifts in the correlation between the output of two clusters are evident (see their Fig. 4). There is currently little experimental evidence for phase shifts among the oscillatory responses of neurons in visual cortex [but note Fig. Ig in Engel et al. (1991b)l. This is in apparent disagreement with the predictions of our model. One possibility is that the limit of weak, long-range coupling is inappropriate. Yet this limit is suggested from the experimental evidence on stimulus dependent synchronization across visual cortex (Eckhorn et al. 1988; Gray et al. 1989). In brief, stimuli outside the receptive field of a neuron may affect the cross-correlogram between it and other cells but these stimuli do not significantly perturb the magnitude or form of its autocorrelogram. This suggests that the effective interaction between distant neurons affects only their timing and not their rate of firing. A second possibility is that phase-shifts are particular to our choice of local architecture (Fig. 1). The numerical studies of Konig and Schillen (1991) make use of an architecture with solely excitatory connections plus synaptic delays, rather than inhibitory feedback. As mentioned above, the output of different clusters in their model exhibits phase-shifts. Further, Hansel et al. (1992) recently derived the form of the phase interaction between two Hodgkin-Huxley neurons. They show that shifts occur for a range of inputs with neurons coupled either by synapses or electrotonic junctions. Thus a body of evidence suggests that phase-shifts are a generic feature of the interaction between weakly coupled neuronal oscillators. There are a number of experimental issues that relate to the observation of phase shifts. The fully averaged cross-correlogramis symmetric in the presence of shifts. However, the cross-correlogram is likely to appear asymmetric when the averaging is incomplete so that only one of the two possible phases, A,$ = ~ c A(Fig. ~ J 4a), ~ dominates the interaction. Thus asymmetric cross-correlograms, traditionally interpreted as the signature of monosynaptic connections (Perkel etal. 19671, may in some cases reflect phase-shifted correlograms that have been averaged for too short a time. A second issue is that fluctuations in cortical activity may make shifts difficult to detect. The amplitude of the phase-shifted correlograms is expected to be reduced compared with correlograms without phase shifts (cf. Fig. 4a and c-e). This may significantly lower the signal-to-noise ratio of shifted cross-correlograms. However, even in the presence of noise stimulus-dependent phase-shifts should lead to a change in the shape of the cross-correlogram that depends on the form of the stimuli. Indeed, cross-correlogramswhose shape depends on the orientation of the stimulus have been observed (Ts’o et al. 1986). Lastly, both noise and variations


565

in the intrinsic frequency of the oscillation will broaden the phase-shifted peaks in the correlogram. This may cause a shifted correlogram to appear as one with a relatively broad central peak. Such correlograms have been reported in recent studies (Nelson et al. 19921, although it is unclear if they result from the mechanism we propose. We suggest that the existence of phase shifts in the oscillatory part of neuronal responses to dissimilar stimuli deserves further experimental scrutiny. The presence of phase parameters can lead to dramatic changes to the phase dynamics (equation 4.2) when more than two clusters of neurons are stimulated. While the detailed behavior depends on the form of the intercluster interaction, r R R ' ( & - $ J R ~ )qualitative , aspects of the behavior may be accounted for by the simplified model E-'&(f)

=

-

C K ( R - R') J(At90)sin ($R

- $R!

+ a(A00))

R'#R

-k

VR(f)

(5.1)

Here At90 = OO(R) - Oo(R') is the relative orientation of the particular two of stimuli that act on a pair of clusters. The interaction parameter, J( A@,),measures the average overlap of the activities in a pair of clusters. It decreases monotonically with increasing values of At90 and vanishes for A& > &, where 0, is the full width of the tuning curve. Conversely, the phase parameter .(A&) increases monotonically with Ado. As before, K(R - R') specifies the spatial extent of the long-range connections (equation 2.1), and V R ( f ) is a gaussian noise (equation 4.2). The above model explicitly expresses the dependence of the amplitudes and phases of the interaction between the clusters on relative phases of each cluster on the spatial distribution of gradients in the orientation of the stimuli. When the phase parameters a(At90)are zero, as assumed in a previous work (Sompolinsky et al. 1990,1991), the network is unfrustrated. In the absence of noise the stimulated clusters will synchronize with zero phase shifts. In contrast, nonzero values of .(A&) may induce substantial frustration into the network and lead to a stable state with a complicated pattern of phase shifts. Further, the dynamics of the network is not governed by an energy function and thus the phases may not converge to fixed values. In cases where the values of phase parameters are large, such as when the stimulus contains sufficiently large spatial gradients, it is likely that the phases of each cluster, @ R ( t ) , will fluctuate chaotically in time. The phase model proposed here (equation 5.1)is likely to have validity beyond the specific architecture and dynamics of the circuit in the present work (Fig. 1). In fact, the simulation results of other circuits proposed for the 40 Hz oscillations in visual cortex (Sporns et al. 1991; Buhmann and von der Malsburg 1991;Konig and Schillen 1991; Wilson and Bower 1991) can be interpreted by this phase model. Thus, the model may provide a useful framework to probe the nature of spatiotemporal patterns of neuronal responses and their role in sensory processing.

566


Appendix A Here we sketch our derivation of the phase description from the full dynamics of the network. The equations for the dynamics of the full network (equation 2.1) are first reduced to a set of equations that involve only the potentials U R ( ~ )and , U R ( t ) , the input I, and the noise & ( t ) within the clusters. This is accomplished by averaging equation 2.1 over all orientations 8, so that

To close these equations, one must obtain a relationship between the average output V Rand the average potentials, U R ( ~ and ) U R ( f ) . Subtracting equation 2.1 from equation A.l and expanding all terms to first order in E yields

where

with g‘(x) = dg/dx and & ( x ) f g ( x ) - (g(x))e. Substitution of equations A.2 and A.3 into equation A.l gives


567

where

+

Y R R ! ( V RU, p ; f ) = K ( R - R’)G[UR(~)] YRR~(VR U .p ;

(A.5)

t)

and G(x) is defined by equation 3.3. The intercluster interaction term in equation A.4 is nonlocal in time. However, for small values of E the potentials can be approximated by vR(t) = %[wt $ R ( f ) ] and vRl(t) = G , [ w t $RJ(t)],where we have used equation 4.4 and the fact that the phases vary slowly on the time scale of the period for the oscillations, 27r/w-’. Substituting this form into equation A.4 results in equations that are now local in time. These equations represent a system of weakly coupled, two-dimensional limit cycles. By an appropriate average over the fast variables, they are further reduced to a set of equations (equation 4.2) that involves only the slow, phase variables, $ R ( f ) (Fig. 3). For details of this reduction, see Kuramoto (1984).

+

+

Appendix B Here we sketch our calculations of the intercluster correlation function (equation 4.5) in terms of the phase dynamics. The correlation can be expressed as

where (. . .), denotes averaging over time and over the noise in the phase equations (equation 4.2) and GV;;[wt $‘R(f)] = V { [ w t $ R ( f ) ] - ( V R ( f ) ) l where V;;[wt+ $ R ( t ) ] x G { G [ w f+ $ R ( f ) ] } is the solution of the equations for the unperturbed cycle (equations 2.2, 3.3, and 3.4). If we restrict ourselves to values of T that are on the order of w-I, we can make the approximation $R, ( t T ) x $,Q(t). , In this limit C R R( T~) depends only on fluctuations in the phase difference A$(t) = $Rf(t) - @ R ( t ) . For the case when only two clusters are stimulated, the form of equation 4.4 implies that the equilibrium distribution of the phase difference, a stochastic variable, is of a Gibbs’ form, that is,

+

+

+

D(A$)

(8.2)

o< ,-W(A@)/ZT

where the potential W(Ali,) is given in terms of the interaction f’(A$; Ado) (equation 4.4) by

1

A$

W(A$)

=

0

d$r($; Ado)

(8.3)

We thus arrive at

C R R ~ ( T=)

+ +

Jp(d$/27r)D($) Jo2””-’(wdt/27r)GV;;(wt)GV;;,(wt WT qj)

([W3t)12)r

Note that this result for C , ~ ( T ) is valid only for values

T

x

0 (w-’).

(8.4)

568


Acknowledgments

We thank B. Friedman, C. D. Gilbert, D. Hansel, J. A. Hirsch, P. C. Hohenberg, P. Konig, 0. Sporns, and D. Y. Ts'o for useful discussions. D. K. and H. S. thank the Aspen Center for Physics for its hospitality. This work was supported, in part, by the Fund for Basic Research administered by the Israeli Academy of Arts and Sciences and by the U.S.-Israel Binational Science Foundation.

References Aertsen, A. M. H. J., Gerstein, G. L., Habib, M. K., and Palm, G. 1989. 9namics of neuronal firing correlation: Modulation of effective connectivity. I. Neurophys. 61,900-917. Amari, S.-I. 1972. Learning patterns and patterns of sequences by self-organizing nets of threshold elements. IEEE Trans. Comp. 21, 1197-1206. Bouyer, J. J., Montaron, M. F., and Rouged, A. 1981. Fast fronto-parietal rhythms during combined focused attentive behaviour and immobility in cat: Cortical and thalamic localization. Electroenceph. Clim. 51, 244-252. Buhmann, J., and von der Malsburg, Ch. 1991. Sensory segmentation by neural oscillators. In Proceedings of the Znternational Conference on Neural Nefworks, pp. 603-607, Vol. 11. Douglas, R. J., and Martin, K. A. C. 1990. Neocortex. In Synaptic Organization ofthe Brain, 3rd ed., G. M. Shepherd, ed., pp. 356-438. Oxford University Press, New York. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reiboeck, R. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Multiple electrode and correlation analysis in the cat. Biol. Cybern. 60, 121-130. Engel, A. K., Konig, P., Kreiter, A. K., and Singer, W. 1991a. Interhemispheric synchronization of oscillatory neuronal responses in cat visual cortex. Science 252, 1177-1180. Engel, A. K., Kreiter, A. K., Konig, P.,and Singer, W.1991b. Synchronization of oscillatory neuronal responses between striate and extrastriate visual cortical areas of the cat. Proc. Natl. Acad. Sci. U.S.A. 88, 6048-6052. Freeman, W., and van Dijk, B. W. 1987. Spatial patterns of visual cortical fast EEG during conditioned reflex in a rhesus monkey. Brain Res. 422,267-276. Gilbert, C. D., and Wiesel, T. N. 1989. Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex. j . Neurosci. 9, 2432-2442. Gray, C. M., Konig, P.,Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit intercolumnar synchronizationwhich reflects global stimulus properties. Nature (London) 338,334-337. Hansel, D., Mato, G., and Meunier, C. 1992. Phase dynamics for weakly coupled model neurons. CNRS preprint. Hopfield, J. J. 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 8,3088-3092.


569

Ketchum, K. L., and Haberly, L. B. 1991. Fast oscillationsand dispersive propagation in olfactory cortex and other cortical areas: A functional hypothesis. In Olfaction: A Model System for Computational Neuroscience, J. L. Davis and H. Eichenbaum, eds., pp. 69-100. MIT Press, Cambridge, MA. Konig, P., and Schillen, T. B. 1991. Stimulus-dependent assembly formation of oscillatory responses: I. Synchronization. Neural Comp. 3, 155-166. Kuramoto, Y. 1984. Chemical Oscillations, Waves, and Turbulence. Springer-Verlag, New York. Nelson, J. I., Salin, P. A., Munk, M. H., Arzi, M., and Bullier, J. 1992. Spatial and temporal coherence in cortico-cortical connections: A cross-correlation study in areas 17 and 18 in the cat. Visual Neurosci. 9,21-37. Perkel, D. H., Gerstein, G. L., and Moore, G. P. 1967. Neuronal spike trains and stochastic point processes. 11. Simultaneous spike trains. Biophys. J. 7, 419-440. Schillen, T. B., and Konig, P. 1991. Stimulus-dependentassembly formation of oscillatory responses: 11. Desynchronization. Neural Comp. 3, 167-178. Schuster, H. G. 1991. Nonlinear Dynamics and Neuronal Networks: Proceedings of the 63rd W. E . Heraues Seminar, Friedrichsdorf 1990. VCH, New York. Schuster, H. G., and Wagner, P. 1990a. A model for neuronal oscillations in the visual cortex: I. Mean-field theory and the derivation of the phase equations. Biol. Cybern. 64,77432. Schuster, H. G., and Wagner, P. 1990b. A model for neuronal oscillations in the visual cortex: II. Phase description of the feature dependent synchronization. Biol. Cybern. 64, 83-85. Sompolinsky, H., Golomb, D., and Kleinfeld, D. 1990. Global processing of visual stimuli in a network of coupled oscillators. Proc. Natl. Acad. Sci. U.S.A. 87,7200-7204. Sompolinsky,H., Golomb, D., and Kleinfeld, D. 1991. Cooperative dynamics in visual processing. Phys. Rev. A15 43,6990-7011. Sporns, O., Gally, J. A,, Reeke, G. A., and Edelman, G. M. 1989. Reentrant signaling among simulated neuronal groups leads to coherency in their oscillatory activity. Proc. Natl. Acad. Sci. U.S.A. 86, 7265-7269. Sporns, O., Tononi, G., and Edelman, G. M. 1991. Modeling perceptual grouping and figure-ground segregation by means of active reentrant connections. Proc. Natl. Acad. Sci. U.S.A. 88, 129-133. Ts’o, D. Y., Gilbert, C. D., and Wiesel, T. N. 1986. Relationshipsbetween horizontal interactions and functional architecture in cat striate cortex as revealed by cross-correlation analysis. J. Neurosci. 6, 1160-1170. Wilson, W. A., and Bower, J. A. 1991. A computer simulation of oscillatory behavior in primary visual cortex. Neural Comp. 3, 498-509. Wilson, H. R., and Cowan, J. D. 1972. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. ].12,1-24. Winfree, A. T. 1980. The Geometry of Biological Time. Springer-Verlag,New York. Received 4 February 1992; accepted 25 January 1993.

This article has been cited by: 1. B. Pesaran, D. Kleinfeld. 2009. Enter the ratrix. Proceedings of the National Academy of Sciences 106:46, 19209-19210. [CrossRef] 2. Demian Battaglia, Nicolas Brunel, David Hansel. 2007. Temporal Decorrelation of Collective Oscillations in Neural Networks with Local Inhibition and Long-Range Excitation. Physical Review Letters 99:23. . [CrossRef] 3. K. Pyragas, O. V. Popovych, P. A. Tass. 2007. Controlling synchrony in oscillatory networks with a separate stimulation-registration setup. Europhysics Letters (EPL) 80:4, 40002. [CrossRef] 4. Sitabhra Sinha, Jayanta Basak. 2002. Dynamical response of an excitatory-inhibitory neural network to external stimulation: An application to image segmentation. Physical Review E 65:4. . [CrossRef] 5. P. A Tass. 2002. Effective desynchronization with a stimulation technique based on soft phase resetting. Europhysics Letters (EPL) 57:2, 164-170. [CrossRef] 6. M.B.H. Rhouma, H. Frigui. 2001. Self-organization of pulse-coupled oscillators with application to clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence 23:2, 180-195. [CrossRef] 7. Bard Ermentrout. 1998. Reports on Progress in Physics 61:4, 353-430. [CrossRef] 8. U. Ernst, K. Pawelzik, T. Geisel. 1998. Delay-induced multistable synchronization of biological oscillators. Physical Review E 57:2, 2150-2162. [CrossRef] 9. Roger Rodriguez, Henry Tuckwell. 1996. Statistical properties of stochastic nonlinear dynamical models of single spiking neurons and neural networks. Physical Review E 54:5, 5585-5590. [CrossRef] 10. S. Campbell, DeLiang Wang. 1996. Synchronization and desynchronization in a network of locally coupled Wilson-Cowan oscillators. IEEE Transactions on Neural Networks 7:3, 541-554. [CrossRef] 11. H. Sompolinsky , M. Tsodyks . 1994. Segmentation by a Network of Oscillators with Stored MemoriesSegmentation by a Network of Oscillators with Stored Memories. Neural Computation 6:4, 642-657. [Abstract] [PDF] [PDF Plus] 12. D. Hansel, C. Meunier. 1993. Clustering and slow switching in globally coupled phase oscillators. Physical Review E 48:5, 3470-3477. [CrossRef]

Communicated by Laurence Abbott

Dynamics of Populations of Integrate-and-Fire Neurons, Partial Synchronization and Memory Marius Usher Computation and Neural Systems 226-76, Caltech, Pasadena, C A 91125 USA

Heinz Georg Schuster Institut fiir Theoretische Physik, Universitat Kiel, 0-2300 Kiel 2, Germany

Ernst Niebur Computation and Neural Systems 226-76, Caltech, Pasadena, C A 91125 USA

We study the dynamics of completely connected populations of refractory integrate-and-fire neurons in the presence of noise. Solving the master equation based on a mean-field approach, and by computer simulations, we find sustained states of activity that correspond to fixed points and show that for the same value of external input, the system has one or two attractors. The dynamic behavior of the population under the influence of external input and noise manifests hysteresis effects that might have a functional role for memory. The temporal dynamics at higher temporal resolution, finer than the transmission delay times and the refractory period, are characterized by synchronized activity of subpopulations. The global activity of the population shows aperiodic oscillations analogous to experimentally found field potentials. 1 Introduction

Most artificial neural networks are based on binary McCulloch-I'ittG neurons that have no intrinsic temporal characteristics. As opposed to these simplified units, real neurons perform temporal integration over their inputs with some specific decay constant and have refractory periods. Model neurons satisfying these constraints are often called integrate-andfire neurons. Although the dynamic behavior of a single such neuron is straightforward, their population dynamics are highly complex. Recently, several studies have shown that under specific assumptions analytical results characterizing such populations of neurons can be obtained. For example, Amit and Tsodyks (1992) have shown that on a longer time scale the dynamic behavior of integrate-and-fire neurons can be averaged Neural Computation 5, 570-586 (1993) @ 1993 Massachusetts Institute of Technology

Populations of Integrate-and-FireNeurons

571

out and characterized by continuous variables (firing rates or currents). However, this transformation is based on the assumption that there is no synchronicity among inputs and once made, the question of dynamics at higher temporal resolution cannot be addressed. Gerstner and Van Hemmen (1992) proposed a stochastic model for spiking neurons that incorporates refractory periods and transmission delays, but which disregards membrane decay properties. The issue of temporal modulations has been investigated by Mirollo and Strogatz (1990), who showed that a population of integrate-and-fire neurons in a strong external field (receiving a strong external input and acting as pulse oscillators) will phaselock and reach synchronicity if they have excitatory all-to-all connections. Van Vreeswijk and Abbott (1992) analyzed the behavior of completely connected populations of excitatory neurons and found that even in the absence of external input, such populations of neurons can exhibit sustained activity due to self-interaction. In particular, they showed that, in the absence of noise, the population may lock in one of several patterns of cyclic activity. The purpose of this work is to further analyze the dynamic characteristics of such populations in the presence of noise and small external inputs, and to find the specific way that a population responds to different inputs. In the first part we develop a discrete-time mean-field model, using a master equation that takes explicitly into account the stochastic behavior due to noise. Using the master equation and numerical simulations we show that for some values of the external field the population is dominated by two attractors, a self-sustained activity state and a silent state. These states are fixed points of the dynamics, and we discuss their basins of attraction. We also show that the network exhibits hysteresis and thus can function as a memory system that reacts to external inputs in a different way than systems consisting of McCulloch-Pitts neurons. In the last section, we study the temporal dynamics at a finer temporal resolution. Taking explicitly into account axonal and synaptic delays and refractory periods, we obtain synchronized activity of subpopulations and aperiodic oscillations of the whole population's activity.

2 The Model

Consider a fully connected population of N excitatory neurons, each one characterized by a continuous variable t i that represents the cell's potential (1 5 i 5 N). Each neuron integrates over its inputs, and once it reaches a threshold (chosen without loss of generality to be l),the neuron fires, sends its output to the other neurons in the population, and resets its potential to zero. For simplicity, we will use a synchronous updating rule where each iteration step corresponds to a typical time for the spike dynamics (including the refractory period) and transmission delays of

M. Usher, H. G. Schuster, and E. Niebur

572

about 2-3 msec. Thus for neuron i,

where X is a decay constant (0 5 X 5 l ) , I > 0 is the coupling constant, 8 is the unit step function, and the external input is characterized by a mean value h and a gaussian noise term (i (of zero mean and standard deviation (5). The first step function adds the contributions from other cells that are firing at time t, and the second step function enforces refractoriness (a neuron cannot fire at two consecutive time steps). In order to test the validity of our synchronous discretization scheme, we have to test whether its dynamic properties depend upon the length of the iteration steps (in the limit of infinitesimal time steps, our scheme should reduce to a continuous system of differential equations representing currents in the cells). As it stands, equation 2.1 implies that both the delay time and the refractory period are equal to the discretization steps. Thus in order to increase the temporal resolution (without changing the actual values of the biological parameters, i.e., refractoriness and delay time), equation 2.1 should be modified, so that the refractory period and the delay time extend over more than one (now smaller) time bin. We shall show in the last section that the results obtained from equation 2.1 (at low time resolution) are compatible with those obtained from the modified delayed equation. However, at the higher resolution new phenomena are revealed. As shown in Van Vreeswijk and Abbott (1992) for zero external input and noise (h = ( i = O), the population may either remain silent or segregate into a cyclic pattern of activity of length M. At time t a subgroup of n' cells fire and subsequently, at time t 1, a different subgroup of n'+' cells will fire etc., satisfying Ci+'nr = N. For X = 1 the length of the cycle M is limited (Van Vreeswijk and Abbott 1992) by: IN < M < - 2IN (2.2)

+

IN - 1

NI - 1

Although the length of the cycle is limited by this equation, this does ~, denot determine the sequence of activations nr,n'+', . . . ,t ~ ' + which pends on the initial conditions zi(0). As we shall show in the next section, under the influence of noise the temporal behavior is asymptotically characterized by homogeneous sequences [i.e., n ( t ) = n(t 1 ) .. . = n(t M ) ] . Thus the system converges to one of two possible fixed points, a homogeneous sequence of activations or the silent state. This situation is somewhat analogous to a ferromagnetic system.

+

+

3 Master Equation

The stochastic system in equation 2.1 is completely characterized by the . . . ,ZN). Since our system is fully full probability distribution P'(ZI,Z~,

Populations of Integrate-and-FireNeurons

573

connected, a mean-field approach yields a lower dimensional-probability distribution P'(z), 1 P'(z) = -

c J 6(z

- Zi)P'(Zl

N i

*

..

3

where Pf(z) is the probability that neuron i will have the synaptic potential z at time t, and P'(z) is the mean-field average probability of having a postsynaptic potential z in the system at time t. The time dependence of P'(z) is obtained from equation 2.1:

P'+'(z) = m'6(z)+

/

1

-m

dz' < 6[z- (Xz'

+ Jm'+ h + t)]> P'(z')

(3.1)

where m', the normalized fraction of active cells at time t is related to P'(z) via m' = JyP'(z)dz, and denotes the average over the noise probability distribution that is chosen to be gaussian: (3.2)

Inserting this into equation 3.1 we obtain,

P'+'(z) = m'6(z)+ J

1

dz'f(z - XZ' - Jm' - h)P'(z')

(3.3)

-W

from which mtfl can be calculated

m'"

W

=

dzP'+'(z) =

p

-w

dz'JW dzf(z - Xz' 1

- Jm' - h)P'(z')

(3.4)

3.1 Strong Decay. For X = 0, the integral in equation 3.4 factorizes and we obtain a nonlinear one-dimensional map:

m'+' = (1- m') Jw

l-\m'-h

P

d 3) unit vectors (d4) distributed evenly around the unit circle, then C ( d 4 V)d+ = nV 2 4

(3.6)

Position-IndependentDetectors of Rotation and Dilation

605

holds for all vector V. For a proof, write each vector as a complex number, namely, V = @" and d+ = e@, where 8 is the direction angle of V and p = )V1is the radius. Because d6 . V = pcos(8- 4) = &I (ei(@-6)+ e-@-6) is a real number, we have

1

Since is evenly distributed around the circle, C6e2'@ = 0. This proves equation 3.6. Assuming that the preferred directions of the MT-like units at each spatial position arepvenly distributed, we can employ formula 3.6 by identifying V with c2dv(r)so that equation 3.5 can be rewritten as 1 Aw(r) = -nec20v(r) (3.7) 2 where n is the number of the MT-like units at each position. This increment is caused by a single training step with the velocity field v(r). After training with a sequence of velocity fields, the equivalent weight vector field adds up to

w(r) = wo(r) + - n d Ov(r) 2 f I

-

C

(3.8)

where t(= 0,1,2,.. .) stands for all time steps in the training and wo(r) is the initial weight vector. In conclusion, the final equivalent weight vector field is just proportional to the sum of the training velocity fields weighted by the corresponding responses of the MST-like unit. 4 Training with Translation, Rotation, Dilation, and Contraction __

We are now ready to consider the training with translation, rotation, dilation, and contraction velocity fields. To begin with, suppose for a single training step the velocity field is a rotation centered at I, v(r) = w x (r - rc)

(4.1)

and in different steps both the angular velocity w and the center r, vary randomly. Substituting equation 4.1 into 3.8 and ignoring the initial weight vector for its smallness, we obtain the final weight vector field w(r) = qCOv(r) = q t

where v field

=

na?/2 is a constant. This can be identified with the rotational

w(r) = n x (r - ro)

(4.2)

606

Kechen Zhang, Martin I. Sereno, and Margaret E. Sereno

+

Figure 5: The final weight vector field is generally composed of a rotation field (a) and a dilation or contraction field (b). The result (c) is a spiral field. where the weight field "angular velocity" n and the weight field center ro are defined by n := r] Et Ow and n x ro := r] Ct(Ow x rc). The latter equation has a unique solution of ro as long as n # 0. In the special case n = 0, w(r) is a constant vector field (translation). Similarly, training with dilation or contraction v(r) = X(r - rc)

with rate X and center r, varying in time will lead to the final weight vector field

w(r) = A(r - ro)

(4.3)

where A := r] E, OX and Aro := 17 C, OXr,. This is either a dilation (A > 0) or a contraction (A < 0). In the special case A = 0, w(r) is a constant (translation). Note that expressions 4.2 and 4.3 are just what are required for position-independent responses (cf. equations 2.4 and 2.6). It should be realized that the center ro does not affect the curl and divergence of a vector field, and thus does not affect our previous conclusions. For training with a mixture of translations, rotations, dilations, and contractions, it is readily shown by similar argument that the final weight vector field takes the form

w(r) = n x (r - a)

+ A(r - b) + c

It can always be written equivalently as

w(r) = n x (r - ro)

+ A(r - ro)

which is a spiral centered at ro (Fig. 5). An MST-like unit with a spiral

Position-IndependentDetectors of Rotation and Dilation

607

weight vector field has position-independent responses to a particular sense of rotation as well as to either a dilation or contraction. Even if the training velocity fields have a zero average, for example, clockwise and counterclockwise rotations have an equal chance of appearing, the weight vector field is still expected to grow with time. We consider the simple case where all rotations and dilationslcontractions are centered at the same point so that the development of the two corresponding components is strictly independent. Consider the initial stage of development for training with, say, rotation fields alone. Now we need consider only the linear range of the sigmoid function c, and for simplicity we assume 0 = I. According to equations 3.7 and 3.3, at time step t 1

+

where the subscripts refer to time. Thus

It can be expressed as Ot+l = AfWt+l

+ 77(AQ,wt)(Awtw*+1)

(4.4)

where at and wf are the angular speeds for the vector fields wt and vf at time t, respectively, and A is a constant depending on the size and shape of the receptive field as well as the position of the rotation center. Imagine an ensemble of parallel training sessions starting from different initial weights and using different rotation sequences of random angular speeds, which are independent of each other while having identical statistics. We take the ensemble average on both sides of equation 4.4 to get (Ot+l)= A (a,) (u)+qA2 (Q,) (w’) (w), where the subscript for the angular speed w is dropped because the statistics of w does not change over time. If (w)= 0, then (Ot+~) = 0 for all t. However, taking the ensemble average after squaring equation 4.4 and using (q) = A2 (0:) (w2), we can obtain (4.5)

where & := A ( w 2 ) = ( Z r v.v) Q

:=

(w4)/(3>’

are constants. When w is drawn from a gaussian distribution of zero


608

mean, for instance, (r = 3. Applying equation 4.5 iteratively yields (q+l) = e’/r (q), where constant T := I /

ln(1 + 27/f + tr$€*)

+

Because 0 1 = C, w1 . v1 and w1 = wo qOovo= ;rlOovo= r/(C wo . vo)vo, by similar arguments as above we can get (@) = (rr?€’ (0;): Hence (0:)= f$C*e(‘-1)/‘

(03

(4.6)

Consequently, when many MST-like units develop in parallel starting from random initial weights, the responses (either positive or negative) to rotation and to dilation/contraction are expected to grow exponentially in the initial stage of development. The variety of the initial responses leads to a continuous spectrum of selectivity to rotation and dilation/contraction, which is what has actually been found in the neurophysiological experiments (Duffy and Wurtz 1991a; Andersen et al. 1991). 5 Discussion

The model provides a unified, albeit simplified, account for several essential properties of MST neurons and how they might develop. These properties include selectivity to rotation, dilation, and contraction, the position independence of the responses (Saito et al. 1986; Tanaka and Saito 1989; Tanaka et al. 1989; Duffy and Wurtz 1991a,b), the selectivity to spiral velocity fields (Graziano et al. 1990; Andersen et al. 1991), and the continuous spectrum of selectivity (Duffy and Wurtz 1991a; Andersen et al. 1991). The model’s response saturates at higher speeds (as a result of the sigmoid function) as does the response of real neurons (Orban et al. 1992). In addition to rotation and dilation/contraction, shear also naturally arises in the optic flow (Koenderink 1986). Since the linear combination of shear fields is still a shear, according to equation 3.8 the weight vector field itself will also have a shear component. Consistent with the model, neurons selective to shear components were also found in the cortical areas including MST (Lagae et al. 1991). This model differs somewhat from the original model in Sereno and Sereno (1991)and from real MST neurons in that it “linearly decomposes’’ the velocity field-that is, an MST-like unit will respond exclusively to the, say, rotational component of a flow field, regardless of the magnitude of the radial component. Since a cosine tuning curve means that the input unit sees exactly the vector component of the local stimulus movement in the preferred (here rotational) direction, it leads to linear decomposition. With narrower tuning curves, the response of individual MST-like units provides more information about the exact composition of the flow field-for example, the extent to which it approximates a pure

Position-Independent Detectors of Rotation and Dilation

609

rotation; nevertheless, approximate position independence with narrower tuning curves is still explained by a direction-template mechanism like that described above. Roughly speaking, learning with a simple Hebb rule tends to maximize the total response by gradient ascent and thus tune the net to the input patterns that frequently occur. Consider the output 0 = o(Z) = u ( F w ; l ; )

The Hebb rule Awi a Z;O is always of the same sign as the gradient of the function E

aE

=

:02:

ao aw;= Z;Oo’(l)

- = 0-

aw;

because the derivative o‘ is always positive. As a consequence, there should be a general tendency for local direction selectivity to be aligned with the direction of the stimulus velocity. Recently, it was demonstrated that although dilation-sensitive MSTd neurons are basically position invariant in their responses, they often respond best to dilations centered at a particular location in the receptive field (often not the receptive field center) (Duffyand Wurtz 1991~).Similar results were obtained in the simulations in Sereno and Sereno (1991) using MT-like (narrower) input-layer tuning curves. It may be advantageous to retain information about combinations of flow field componentshere, dilation and translation-in single units since these combinations can have particular behavioral relevancefor example, in signaling direction of heading (Perrone 1992). More realistic peaked (instead of linear) speed tuning curves (Maunsell and Van Essen 1983) in the MT-like input layer could potentially sharpen the response to particular flow components since local speeds may be changed from the optimum as flow field components are added. Cross-direction inhibition (known to occur in MT: Snowden et al. 1991) could also be incorporated, effectively deleting portions of the flow field containing conflicting local motion signals. This could improve performance with more complex, real-world motion arrays. The rotation, dilation, and contraction velocity fields required for training are readily produced when an animal is moving around in a rigid environment. Exposure to such velocity fields may be crucial in order for a young animal to develop rotation and dilation cells in its visual system. Human babies, for instance, can distinguish a rotation field from a random velocity field only after several months of visual experience (Spitz et al. 1988). This could be tested by recording from MST in infant monkeys.

610


Feedforward networks using Hebb rules have been shown to be capable of producing detectors selective to a hierarchy of features like those found in the successive stages of visual processing: center-surround units like those in the LGN (Linsker 1986a),orientation-selective units like simple cells in V1 (Linsker 1986b), pattern motion units like some cells in MT (Sereno 1989), and finally position-independent rotation and dilation units like cells in dorsal MST (Sereno and Sereno 1991). The visual system may use simple local learning rules and a richly textured environment to build up complex filters in stages. This strategy could drastically reduce the amount of supervision that is required later on (cf. Geman et al. 1992) as the visual system learns to recognize objects and direct navigation and manipulation.

Note Added in Proof Recently, Gallant et al. (1993) found that neurons in V4 respond selectively, and in a position-invariant way to static patterns containing concentric, radiating, shearing, or spiral contours. The main outlines of our analysis could be extended to explain the selectivity and development of these neurons by substituting an orientation-selective input layer for the direction-selective input layer considered here.

Acknowledgments

M. E. S. was supported by a postdoctoral fellowship, K. Z. by a graduate fellowship, and M. I. S. by a research award from the McDonnell-Pew Center for Cognitive Neuroscience at San Diego. We thank an anonymous reviewer for helpful comments.

References Andersen, R., Graziano, M., and Snowden, R. 1991. Selectivity of area MST neurons for expansion/contraction and rotation motions. Invest. Opkthal. Vis.Sci., Abstr. 32, 823. Duffy, C. J., and Wurtz, R. H. 1991a. Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli. J. Neurophysiol. 65, 1329-1345. Duffy, C. J., and Wurtz, R. H. 1991b. Sensitivity of MST neurons to optic flow stimuli. 11. Mechanisms of response selectivity revealed by small-field stimuli. J. Neurophysiol. 65, 1346-1359. Duffy, C. J., and Wurtz, R. H. 1991~.MSTd neuronal sensitivity to heading of motion in optic flow fields. SOC. Neurosci. Abstr. 17, 441. Felleman, D., and Van Essen, D. C. 1991. Distributed hierarchical processing in primate cerebral cortex. Cerebral Cortex 1, 147.

Position-Independent Detectors of Rotation and Dilation

611

Gallant, J. L., Braun, J., and Van Essen, D. C. 1993. Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex. Science 259, 100-103. Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias/variance dilemma. Neural Comp. 4, 1-58. Graziano, M. S. A,, Andersen, R. A., and Snowden, R. 1990. Stimulus selectivity of neurons in macaque MST. SOC.Neurosci. Abstr. 16, 7. Koenderink, J. J. 1986. Optic flow. Vision Res. 26, 161-180. Koenderink, J. J., and van Doorn, A. J. 1975. Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer. Opt. Acta 22, 773-791. Koenderink, J. J., and van Doorn, A. J. 1976. Local structure of movement parallax of the plane. J . Opt. SOC.Am. 66, 717-723. Lagae, L., Xiao, D., Ralguel, S., Maes, H., and Orban, G. A. 1991. Position invariance of optic flow component selectivity differentiates monkey MST and FST cells from MT cells. Invest. Ophthal. Vis. Sci., Abstr. 32, 823. Linsker, R. 1986a. From basic network principles to neural architecture: emergence of spatial-opponent cells. Proc. Natl. Acud. Sci. U.S.A. 83, 7508-7512. Linsker, R. 1986b. From basic network principles to neural architecture: emergence of orientation-selective cells. Proc. Natl. Acad. Sci. U.S.A.83,8390-8394. Longuet-Higgins, H. C., and Prazdny, K. 1980. The interpretation of a moving retinal image. Proc. R. SOC.London B 208,385-397. Maunsell, J. H. R., and Van Essen, D.C. 1983. Functional properties of neurons in middle temporal visual area (MT) of the macaque monkey: I. Selectivity for stimulus direction, speed and orientation. 1.Neurophysiol. 49, 1127-1147. Movshon, J. A., Adelson, E. H., Gizzi, M. S., and Newsome, W. T. 1985. Analysis of moving visual patterns. In Pattern Recognition Mechanisms, C. Chagas, R. Gattass, and C. Gross, eds., pp. 117-151. Springer-Verlag, New York. Orban, G. A., Lagae, L., Verri, A., Raiguel, S., Xiao, D., Maes, H., and Torre, V. 1992. First-order analysis of optical flow in monkey brain. Proc. Natl. Acad. Sci. U.S.A. 89, 2595-2599. Perrone, J. A. 1992. Model for the computation of self-motion in biological systems. J . Opt. SOC.Am. A 9, 177-194. Poggio, T., Verri, A., and Torre, V. 1990. Does cortical area MST know Green theorems? Instituto per la Ricerca Scientifica e Technologica Tech. Rep. No. 900807,143. Poggio, T., Verri, A., and Torre, V. 1991. Green theorems and qualitative p r o p erties of the optical flow. MIT A.I. Memo, No. 1289, 1-6. Rodman, H. R., and Albright, T. D. 1987. Coding of visual stimulus velocity in area MT of the macaque. Vision Res. 27,2035-2048. Saito, H., Yukie, M., Tanaka, K., Hikosaka, K., Fukada, Y., and Iwai, E. 1986. Integration of direction signals of image motion in the superior temporal sulcus of the macaque monkey. 1.Neurosci. 6, 145-157. Sakata, H., Shibutani, H., Ito, Y., and Tsurugai, K. 1986. Parietal cortical neurons responding to rotary movement of visual stimulus in space. Exp. Brain Res. 61, 658-663. Sereno, M. I. 1989. Learning the solution to the aperture problem for pattern

612


motion with a Hebb rule. In Advances in Neural Information Processing System I, D. S. Touretzky, ed., pp. 468-476. Morgan Kaufmann, San Mateo, CA. Sereno, M. I., and Allman, J. M. 1991. Cortical visual areas in mammals. In The Neural Basis ofvisuul Function, A. G . Leventhal, ed., pp. 160-172. Macmillan, London. Sereno, M. I., and Sereno, M. E. 1990. Learning to discriminate senses of rotation and dilation with a Hebb rule. Invest. Ophthl. Vis. Sci., Abstr. 31, 528. Sereno, M. I., and Sereno, M. E. 1991. Learning to see rotation and dilation with a Hebb rule. In Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. Moody, and D. S. Touretzky, eds., pp. 320-326. Morgan Kaufmann, San Mateo, CA. Snowden, R. J., Treue, S., Erickson, R. G., and Andersen, R. A. 1991. The response of area MT and V1 neurons to transparent motion. J. Neurosci. 11, 2768-2785. Spitz, R. V., Stiles-Davis, J., and Siegel, R. M. 1988. Infant perception of rotation from rigid structure-from-motion displays. SOC.Neurosci. Abstr. 14, 1244. Tanaka, K., and Saito, H.-A. 1989. Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clustered in the dorsal part of the medial superior temporal area of the macaque monkey. 1. Neurophysiol. 62, 626-641. Tanaka, K., Hikosaka, K., Saito, H.-A., Yukie, M., Fukada, Y., and Iwai, E. 1986. Analysis of local and widefield movements in the superior temporal visual areas of the macaque monkey. J.Neurosci. 6, 134-144. Tanaka, K., Fukada, Y., and Saito, H.-A. 1989. Underlying mechanisms of the response specificity of expansion/contraction and rotation cells in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neurophysiol. 62, 642-656. Received 11 October 1991; accepted 11 January 1993.

This article has been cited by: 2. S. Pitzalis, M.I. Sereno, G. Committeri, P. Fattori, G. Galati, F. Patria, C. Galletti. 2010. Human V6: The Medial Motion Area. Cerebral Cortex 20:2, 411-424. [CrossRef] 3. Peter J. Bex, Andrew B. Metha, Walter Makous. 1998. Psychophysical evidence for a functional hierarchy of motion processing mechanisms. Journal of the Optical Society of America A 15:4, 769. [CrossRef] 4. Ruye Wang . 1995. A Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTdA Simple Competitive Account of Some Response Properties of Visual Neurons in Area MSTd. Neural Computation 7:2, 290-306. [Abstract] [PDF] [PDF Plus]

Communicated by Andrew Barto

Improving Generalization for Temporal Difference Learning: The Successor Representation Peter Dayan Computational Neurobiology Laboratory, The Salk Institute, P.0.Box 85800, Sun Diego, CA 92186-5800 USA Estimation of returns over time, the focus of temporal difference (TD) algorithms, imposes particular constraints on good function approximators or representations. Appropriate generalization between states is determined by how similar their successors are, and representations should follow suit. This paper shows how TD machinery can be used to learn such representations, and illustrates, using a navigation task, the appropriately distributed nature of the result. 1 Introduction

The method of temporal differences (TD, Samuel 1959; Sutton 1984,1988) is a way of estimating future outcomes in problems whose temporal structure is paramount. A paradigmatic example is predicting the long-term discounted value of executing a particular policy in a finite Markovian decision task. The information gathered by TD can be used to improve policies in a form of asynchronous dynamic programming (DP; Watkins 1989; Barto et al. 1989; Barto et al. 1991). As briefly reviewed in the next section, TD methods apply to a learning framework, which specifies the goal for learning and precisely how the system fails to attain this goal in particular circumstances. Just like a proposal to minimize mean square error, TD methods lie at the heart of different mechanisms operating over diverse representations. Representation is key-difficult problems can be rendered trivial if looked at in the correct way. It is particularly important for systems to be able to learn appropriate representations, since it is rarely obvious from the outset exactly what they should be. For static tasks, generalization is typically sought by awarding similar representations to states that are nearby in some space. This concept extends to tasks involving prediction over time, except that adjacency is defined in terms of similarity of the future course of the behavior of a dynamic system. Section 3 suggests a way, based on this notion of adjacency, of learning representations that should be particularly appropriate for problems to which TD techniques have been applied. Learning these representations can be viewed as a task itself amenable to TD methods, and so requires Neural Computation 5, 613-624 (1993)


Peter Dayan

614

no extra machinery. Section 4 shows the nature of the resulting representation for a simple navigation task. Part of this work was reported in Dayan (1991a,b). 2

TD Learning

Consider the problem of estimating expected terminal rewards, or returns, in a finite absorbing Markov chain; this was studied in the context of TD methods by Sutton (1988). An agent makes a transition between nonabsorbing states i and j E N according to the ijth element of the Markov matrix Q, or to absorbing state k E 7 with probability Sik, with a stochastic reinforcement or return whose mean is t k and whose variance is finite. In this and the next section, the returns and transition probabilities are assumed to be fixed. The immediate expected return from state i E N, represented as the ith element of a vector h, is the sum of the probabilities of making immediate transitions to absorbing states times the expected returns from those states:

The overall expected returns, taking account of the possibility of making transitions to nonabsorbing states first, are [f]i

+

+

+

=

[h]; [Qh]; [Q2hIi . . .

=

[ ( I - Q)-'h];

(2.1)

where 1 is the identity matrix. The agent estimates the overall expected return from each state (compiled into a vector r) with a vector-valued function i(w), which depends on a set of parameters w whose values are determined during the course of learning. If the agent makes the transition from state if to iltl in one observed sequence, TD(0) specifies that w should be changed to reduce the error: frti =

[i(w)li,+l- ['(w)li,

(2.2)

where, for convenience, [i(w)] is taken to be the delivered return ~ i , + if if+l is absorbing. This enforces a kind of consistency in the estimates of the overall returns from successive states, which is the whole basis of TD learning. More generally, information about the estimates from later states [i(w)] for s > 1 can also be used, and Sutton (1988) defined the TD(A) algorithm, which weighs their contributions exponentially less according to As. With the TD algorithm specifying how the estimates should be manipulated in the light of experience, the remaining task is one of function approximation. How w should change to minimize the error fI+l +% ,;

~

Improving Generalization for Temporal Difference Learning

615

in equation 2.2 depends on exactly how w determines [i(w)],,.Sutton (1988) represented the nonabsorbing states with real-valued vectors {xi}, [r(w)lias the dot product w xi of the state vector with w taken as a vector of weights, and changed w in proportion to

-

-(w . xi,+, - w . Xi,)Xi, using z;,+~instead of w . xi,+l if it+l is absorbing. This is that part of the gradient -Vwct+l that comes from the error at step xi,, ignoring the contribution from xi,+l (Werbos 1990; Dayan 1992). In the "batch-learning" case for which the weights are updated only after absorption, Sutton showed that if the learning rate is sufficiently small and the vectors representing the states are linearly independent, then the expected values of the estimates converge appropriately. Dayan (1992) extended this proof to show the same was true of TD(X) for 0 < x < 1. 3 Time-Based Representations

One of the key problems with TD estimation, and equivalently with TD based control (Barto et al. 19891, is the speed of learning. Choosing a good method of function approximation, which amounts in the linear case to choosing good representations for the states, should make a substantial difference. For prediction problems such as the one above, the estimated expected overall return of one state is a biased sum of the estimated expected overall returns of its potential successors. This implies that for approximation schemes that are linear in the weights w, a good representation for a state would be one that resembles the representations of its successors, being only a small Euclidean distance away from them (with the degrees of resemblance being determined by the biases). In this way, the estimated value of each state can be partially based on the estimated values of those that succeed it, in a way made more formal below. For conventional, static, problems, received wisdom holds that distributed representations perform best, so long as the nature of the distribution somehow conforms with the task-nearby points have nearby solutions. The argument above suggests that the same is true for dynamic tasks, except that neighborliness is defined in terms of temporal succession. If the transition matrix of the chain is initially unknown, this representation will have to be learned directly through experience. Starting at state i E N, imagine trying to predict the expected future occupancy of all other states. For the jth state, j E N, this should be [Xilj

=

+ [Qlij + [Q'lij +

[llij

= [ ( I - Q)-']ij.

.

'

(3.1)

where [MI;, is the ijth element of matrix M and 1 is the identity matrix. Representing state i using Xi is called the successor representation (SR).

Peter Dayan

616

A TD algorithm itself is one way of learning SR. Consider a punctate representation that devotes one dimension to each state and has the Ith element of the vector representing state k, [x& equal to [Ilk/. Starting from i, = i, the prediction of how often [xi,], = 1 for s 2 t is exactly the prediction of how often the agent will visit state j in the future starting from state i, and should correctly be [x,lj. To learn this, the future values of [xi,], for s 2 t can be used in just the same way that the future delivery of reinforcement or return is used in standard TD learning. For a linear function approximator, it turns out that SR makes easy the resulting problem of setting the optimal weights w*, which are defined as those making i = r(w*). If X is the matrix of vectors representing the states in the SR, [Xli, = [x,Ii, then W*is determined as

XTw* = f which implies, from equations 2.1 and 3.1, that W* = h

But h is just the expected immediate return from each state-it is insensitive to all the temporal dependencies that result from transitions to nonabsorbing states. The SR therefore effectively factors out the entire temporal component of the task, leaving a straightforward estimation problem for which TD methods would not be required. This can be seen in the way that the transition matrix Q disappears from the update equation, just as would happen for a nontemporal task without a transition matrix at all. For instance, for the case of an absorbing Markov chain with batch-learning updates, Sutton showed that the TD(0) update equation for the mean value of the weights W,l satisfies Wn+l = W,l

+ nXD(h + QX'W,

- XTW,,)

where X is the representation, a is the learning rate, and, since the updates are made after observing a whole sequence of transitions from start to absorption rather than just a single one, D is the diagonal matrix whose diagonal elements are the average number of times each state is visited on each sequence. Alternatively, directly from the estimates of the values of the states, (XTW,I+l- i) =

[I - aXTXD(Z- Q)](XTW,- i)

Using X instead, the update becomes W,l+l = W,

+ aXD(h - WIl)

or (Wtf+l

- h) = (I - a X D ) ( W , - h)

Improving Generalizationfor Temporal Difference Learning

617

Since X is invertible, Sutton’s proof that XTW, + r, and therefore that W,,+ h as n .+ 00, still holds. I conjecture that the variance of these estimates will be lower than those for other representations X (e.g., X = 0 because of the exclusion of the temporal component. For control problems it is often convenient to weigh future returns exponentially less according to how late they arrive-this effectively employs a discount factor. In this case the occupancy of future states in equation 3.1 should be weighed exponentially less by exactly the same amount. A possible objection to using TD learning for SR is that it turns the original temporal learning problem-that of predicting future reinforcement-into a whole set of temporal learning problems-those of predicting the future occupancy of all the states. This objection is weakened in two cases: 0

0

The learned predictions can be used merely to augment a standard representation such as the punctate one. An approximately appropriate representation can be advantageous even before all the predictions are quite accurate. Unfortunately this case is hard to analyze because of the interaction between the learning of the predictions and the learning of the returns. Such a system is used in the navigation example below. The agent could be allowed to learn the predictions by exploring its environment before it is first rewarded or punished. This can be viewed as a form of latent learning and works because the representation does not depend on the returns.

One could regard these predictions as analogous to the hidden representations in Anderson’s (1986) multilayer backpropagation TD network in that they are fashioned to be appropriate for learning TD predictions but are not directly observable and so have to be learned. Whereas Anderson‘s scheme uses a completely general technique that makes no explicit reference to states’ successors, SR is based precisely on what should comprise a good representation for temporal tasks. 4 Navigation Illustration

Learning the shortest paths to a goal in a maze such as the one in Figure 1 was chosen by Watkins (1989) and Barto et al. (1989) as a good example of how TD control works. For a given policy, that is, mapping from positions in the grid to directions of motion, a TD algorithm is used to estimate the distance of each state from the goal. The agent is provided with a return of -1 for every step that does not take it to the goal and future returns, that is, future steps, are weighed exponentially less using a discount factor. The policy is improved in an asynchronous form of

Peter Dayan

618

Agent Goal

Barrier ’ Figure 1: The grid task. The agent can move one step in any of the four directions except where limited by the barrier or by the walls. dynamic programming’s policy iteration by making more likely those actions whose consequences are better than expected. Issues of representation are made particularly clear in such a simple example. For the punctate case, there can be no generalization between states. Distributed representations can perform better, but there are different methods with different qualities. Watkins (1989), for a similar task, used a representation inspired by Albus’ CMAC (1975). In this case, CMAC squares which cover patches of 3 x 3 grid points are placed regularly over the grid such that each interior grid point is included in 9 squares. The output of the units corresponding to the squares is 0 if the agent is outside their receptive fields, and otherwise, like a radial basis function, is modulated by the distance of the agent from the tenter of the relevant square. Over most of the maze this is an excellent representation-locations that are close in the Manhattan metric on the grid are generally similar distances from the goal, and are also covered by many of the same CMAC squares. Near the barrier, however, the distribution of the CMACs actually hinders learning-locations close in the grid but on opposite sides of the barrier are very different distances from the goal, and yet still share a similar CMAC square representation. By contrast, the successor representation, which was developed in the previous section, produces a CMAC-like representation that adapts correctly to the barrier. If the agent explores the maze with a completely random policy before being forced to find the goal, the learned SR would closely resemble the example shown in Figure 2. Rather like a CMAC square, the representation decays exponentially away from the starting state (5,6)in a spatially ordered fashion-however, note SRs recognition


619

Figure 2: The predictions of future occupancy starting from (5,6) after exploration in the absence of the goal. The z-coordinate shows the (normalized) predictions, and the barrier and the goal are overlaid. The predictions decay away exponentially from the starting location, except across the bamer. that states on the distant side of the bamer are actually very far away in terms of the task (and so the predictions are too small to be visible). Simulations confirm that using the SR in conjunction with a punctate representation leads to faster learning for this simple task (see Fig. 31, even if the agent does not have the chance to explore the maze before being forced to find the goal. This example actually violates the stationarity assumption made in Section 2 that transition probabilities and returns are fixed. As the agent improves its policy, the mean number of steps it takes to go from one state to another changes, and so the SR should change too. Once the agent moves consistently along the optimal path to the goal, locations that are not on it are never visited, and so the prediction of future occupancy of those should be 0. Figure 4 shows the difference between the final and initial sets of predictions of future occupancy starting from the same location ( 5 , 6 ) as before. The exponential decay along the path is caused by the discount factor, and the path taken by the agent is clear. If the task for the agent were changed such that it had to move from anywhere

Peter Dayan

620

.-- Rpunctate

- - --

-

-0

-

b

8 b .

&MAC

RSR, no latent learning RsR,

latent learning

'\'\

- \', *'

'Ib 8 '8

10

100

loo0 Learning iterations

Figure 3 Learning curves comparing punctate representation (Rpunctate),CMACsquares (&MAC) and a punctate representation augmented with the SR (RsR), in the latter case both with and without an initial, unrewarded, latent learning phase. TD control learning as in Barto et al. (1989) is temporarily switched off after the number of trials shown in the x-axis, and the y-axis shows the average number of excess steps the agent makes on the way to the goal starting from every location in the grid. Parameters are in Dayan (1991b). on the grid to a different goal location, this new form of the SR would actually hinder the course of learning, since its distributed character no longer correctly reflects the actual nature of the space. This demise is a function of the linked estimation and control, and would not be true for pure estimation tasks. 5 Discussion

This paper has considered some characteristics of how representation determines the performance of TD learning in simple Markovian environments. It suggests that what amounts to a local kernel for the Markov


621

Figure 4: The degradation of the predictions. Both graphs show the differences between the predictions after 2000 steps and those initially-the top graph as a surface, with the barrier and the goal overlaid, and the bottom graph as a density plot. That the final predictions just give the path to the goal is particularly clear from the white (positive) area of the density plot-the black (negative) area delineates those positions on the grid that are close to the start point (5,6), and therefore featured in the initial predictions, but are not part of this ultimate path.

622

Peter Dayan

chain is an appropriate distributed representation, because it captures all the necessary temporal dependencies. This representation can be constructed during a period of latent learning and is shown to be superior in a simple navigation task, even over others that also share information between similar locations. Designing appropriate representations is a key issue for many of the sophisticated learning control systems that have recently been proposed. However, as Barto et al. (1991) pointed out, a major concern is that the proofs of convergence of TD learning have not been very extensively generalized to different approximation methods. Both Moore (1990) and Chapman and Kaelbling (1991) sought to exorcise the daemon of dimensionality by using better function approximation schemes, which is an equivalent step to using a simple linear scheme with more sophisticated input representations. Moore used kd trees (see Omohundro 1987, for an excellent review), which have the added advantage of preserving the integrity of the actual values they are required to store, and so preserve the proofs of the convergence of Q-learning (Barto et al. 1991; Watkins and Dayan 1992). However just like the CMAC representation described above, the quality of the resulting representation depends on an a priori metric, and so is not malleable to the task. Chapman and Kaelbling also used a tree-like representation for Qlearning, but their trees were based on logical formulas satisfied by their binary-valued input variables. If these variables do not have the appropriate characteristics, the resulting representation can turn out to be unhelpful. It would probably not afford great advantage in the present case. Sutton (1990),Thrun etal. (19911, and others have suggested the utility of learning the complete transition matrix of the Markov chain, or, for the case of control, the mapping from states and actions to next states. Sutton used this information to allow the agent to learn while it is disconnected from the world. Thrun, Moller and Linden used it implicitly to calculate the cost of and then improve a projected sequence of actions. The SR is less powerful in the sense that it provides only an appropriately distributed representation and not a veridical map of the world. A real map has the added advantage that its information is independent of the goals and policies of the agent; however, it is more difficult to learn. Sutton’s scheme could equally well be used to improve a system based on the learned representation. Sutton and Pinette (1985)discussed a method for control in Markovian domains that is closely related to the SR and that uses the complete transition matrix implicitly defined by a policy. In the notation of this paper, they considered a recurrent network effectively implementing the iterative scheme

where xi is the punctate representation of the current state i and Q is the


623

transition matrix. x, converges to Xi from equation 3.1, the SR of state i. Rather than use this for representational purposes, however, Sutton and Pinette augmented Q so that the sum of future returns is directly predicted through this iterative process. This can be seen as an alternative method of eliminating the temporal component of the task, although the use of the recurrence implies that the final predictions are very sensitive to errors in the estimate of Q. The augmented Q matrix is learned using the discrepancies between the predictions at adjacent time steps-however, the iterative scheme complicates the analysis of the convergence of this learning algorithm. A particular advantage of their method is that a small change in the model (e.g., a slight extension to the barrier) can instantaneously lead to dramatic changes in the predictions. Correcting the SR would require relearning all the affected predictions explicitly. Issues of representation and function approximation are just as key for sophisticated as unsophisticated navigation schemes. Having a representation that can learn to conform to the structure of a task has been shown to offer advantages-but any loss of the guarantee of convergence of the approximation and dynamic programming methods is, of course, a significant concern.

Acknowledgments

I am very grateful to Read Montague, Steve Nowlan, Rich Sutton, Terry Sejnowski, Chris Watkins, David Willshaw, the connectionist groups at Edinburgh and Amherst, and the large number of people who read drafts of my thesis for their help and comments. I am especially grateful to Andy Barto for his extensive and detailed criticism and for pointers to relevant literature. Support was from the SERC.

References Albus, J. S. 1975. A new approach to manipulator control: The CerebellarModel Articulation Controller (CMAC). Transact. ASME: J. Dynam. Syst. Measure. Control 97, 220-227. Anderson, C. W. 1986. Learning and problem solving with multilayer connectionist systems. Ph.D. Thesis, University of Massachusetts, Amherst, MA. Barto, A. G., Sutton, R. S., and Watkins, C. J. C. H. 1989. Learning and sequential decision making. Tech. Rep. 89-95, Computer and Information Science, University of Massachusetts, Amherst, MA. Barto, A. G., Bradtke, S. J., and Singh, S. P. 1991. Real-time learning and control using asynchronous dynamic programming. TR 91-57, Department of Computer Science, University of Amherst, MA.

Peter Dayan

624

Chapman, D., and Kaelbling, L. P. 1991. Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. Proceedings of the 1991 International Joint Conference on Artificial Intelligence, 726-731. Dayan, P. 1991a. Navigating through temporal difference. In Advances in Neural Information Processing, Vol. 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 464-470. Morgan Kaufmann, San Mateo, CA. Dayan, P. 1991b. Reinforcing connectionism: Learning the statistical way. Ph.D. Thesis, University of Edinburgh, Scotland. Dayan, P. 1992. The convergence of TD(A) for general A. Machine Learn. 8, 341-362. Moore, A. W. 1990. EfFcient memory-based learning for robot control. Ph.D. Thesis, University of Cambridge Computer Laboratory, Cambridge, England. Omohundro, S. 1987. Efficient algorithms with neural network behaviour. Complex Syst. 1, 273-347. Samuel, A. L. 1959. Some studies in machine learning using the game of checkers. Reprinted in Computers and Thought, E. A. Feigenbaum and J. Feldman, eds. McGraw-Hill, New York, 1963. Sutton, R. S. 1984. Temporal credit assignment in reinforcement learning. Ph.D. Thesis, University of Massachusetts, Amherst, MA. Sutton, R. S. 1988. Learning to predict by the methods of temporal difference. Machine Learn. 3, 9 4 . Sutton, R. S. 1990. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the Seventh lnternational Conference on Machine Learning. Morgan Kaufmann, San Mateo, CA.

Sutton, R. S., and Pinette, 8. 1985. The learning of world models by connectionist networks. In Proceedings of the Seventh Annual Conference of the Cognitive Science Society, pp. 54-64. Lawrence Erlbaum, Irvine, CA. Thrun, S. B., Moller, K., and Linden, A. 1991. Active exploration in dynamic environments. In Advances in Neural lnformation Processing, Vol. 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 450-456. Morgan Kaufmann, San Mateo, CA. Watkins, C. J. C. H. 1989. Learningfrom delayed rewards. Ph.D. Thesis, University of Cambridge, England. Watkins, C. J. C. H., and Dayan, P. 1992. &-learning. Machine Learn. 8,279-292. Werbos, P. J. 1990. Consistency of HDP applied to a simple reinforcement learning problem. Neural Networks 3, 179-189. Received 20 January 1992; accepted 20 October 1992.

This article has been cited by: 2. Sridhar Mahadevan. 2008. Representation Discovery using Harmonic Analysis. Synthesis Lectures on Artificial Intelligence and Machine Learning 2:1, 1-147. [CrossRef] 3. P. Read Montague, Brooks King-Casas, Jonathan D. Cohen. 2006. IMAGING VALUATION MODELS IN HUMAN CHOICE. Annual Review of Neuroscience 29:1, 417-448. [CrossRef] 4. Roland E. Suri , Wolfram Schultz . 2001. Temporal Difference Model Reproduces Anticipatory Neural ActivityTemporal Difference Model Reproduces Anticipatory Neural Activity. Neural Computation 13:4, 841-862. [Abstract] [PDF] [PDF Plus] 5. D.J. Foster, R.G.M. Morris, Peter Dayan. 2000. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10:1, 1-16. [CrossRef] 6. Peter DayanReinforcement Learning . [CrossRef]


Discovering Predictable Classifications Jiirgen Schmidhuber’ Department of Computer Science, University of Colorado, Boulder, CO 80309, USA

Daniel Prelinger Institut fiir Informatik, Technische Universitat Miinchen, Arcisstrasse 21,8000 Miinchen 2, Germany

Prediction problems are among the most common learning problems for neural networks (e.g., in the context of time series prediction, control, etc.). With many such problems, however, perfect prediction is inherently impossible. For such cases we present novel unsupervised systems that learn to classih patterns such that the classifications are predictable while still being as specific as possible. The approach can be related to the IMAX method of Becker and Hinton (1989)and Zemel and Hinton (1991). Experiments include a binary stereo task proposed by Becker and Hinton, which can be solved more readily by our system.

1 Motivation and Basic Approach

Many neural net systems (e.g., for control, time series prediction) rely on adaptive submodules for learning to predict patterns from other patterns. Perfect prediction, however, is often inherently impossible. In this paper we study the problem of finding pattern class@cations such that the classes are predictable, while still being as specific as possible. To grasp the basic idea, let us discuss several examples. Example 1: Hearing the first two words of a sentence “Henrietta eats. .. ” allows you to infer that the third word probably indicates something to eat but you cannot tell what. The class of the third word is predictable from the previous words-the particular instance of the class is not. The class ”food” is not only predictable but also nontrivial and specific in the sense that it does not include aterything-”John,” for instance, is not an instance of “food.” *Current address: Institut fiir Informatik, Technische Universit5t Miinchen, Arcisstrasse 21,8000 Miinchen 2, Germany.


626

Jiirgen Schmidhuber and Daniel Prelinger

The problem is to class+ patterns from a set of training examples such that the classes are both predictable and not too general. A general solution to this problem would be useful for discovering higher level structure in sentences generated by unknown grammars, for instance. Another application would be the unsupervised classification of different pattern instances belonging to the same class, as will be seen in the next example.

Example 2 (stereo task; due to Becker and Hinton 1989): There are two binary images called the "left" image and the "right" image. Each image consists of two "strips"+ach strip being a binary vector. The right image is purely random. The left image is generated from the right image by choosing, at random, a single global shift to be applied to each strip of the right image. An input pattern is generated by concatenating a strip from the right image with the corresponding strip from the left image. "So the input can be interpreted as a fronto-parallel surface at an integer depth. The only local property that is invariant across space is the depth (i.e. shift)'' (Becker and Hinton 1989). With a given pair of different input patterns, the task is to extract a nontrivial classification of whatever is common to both patterns-which happens to be the stereoscopic shift. Example 1 is an instance of the so-called asymmetric case: There we are interested in a predictable nontrivial classification of one pattern (the third word), given some other patterns (the previous words). Example 2 is an instance of the so-called symmetric case: There we are interested in the nontrivial common properties of two patterns from the same class. In its simplest form, our basic approach to unsupervised discovery of predictable classifications is based on two neural networks called TI and T2. Both can be implemented as standard backpropagation networks (Werbos 1974; LeCun 1985; Parker 1985; Rumelhart et al. 1986). With a given pair of input patterns, TI sees the first pattern and T2 sees the second pattern. Let us first focus on the asymmetric case. For instance, with the example 1 above TImay see a representation of the words "Henrietta eats," while T2 may see a representation of the word "vegetables." TZ'S task is to classify its input. Ti's task is not to predict T2's raw environmental input but to predict T2's output instead. Both networks have q output units. Let p E (1,. . . m } index the input patterns. T2 produces as an ou ut the classification yPv2 E (0,.. . 114 in response to an input vector xp? TI'S output in response to its input vector xPJ is the prediction ypJ E [0,. . . 114 of the current classification ypy2 emitted by T2. We have two conflicting goals which in general are not simultaneously satisfiable: (1)All predictions yp~' should match the corresponding classifications yPi2. (2) The ypt2 should be discriminative-different inputs xPv2 should lead to different classifications ypi2. We express the trade-off between (1) and (2) by means of two opposing costs.

Discovering Predictable Classifications

627

(1)is expressed by an error term M (for "Match): m

M=

Ilyp,' - yp*2112

(1.1)

p=l

Here llzlll denotes the euclidean norm. (2) is enforced by an additional error term 0 2 (for "Discrimination") to be minimized by T2 only. 0 2 will be designed to encourage significant euclidean distance between classifications of different input patterns. 02 can be defined in more than one reasonable way. The next section will list four alternative possibilities with mutual advantages and disadvantages. These alternatives include (1) a novel method for constrained variance maximization, (2) autoencoders, and (3) a recent technique called "predictability minimization" (Schmidhuber 1992). The total error to be minimized by T2 is EM+ (1 - E)D2

(1.2)

where 0 < E < 1 determines the relative weighting of the opposing error terms. In the asymmetric case, the total error to be minimized by TI is just €M

(1.3)

The error functions are minimized by gradient descent. This forces the predictions and classifications to be more like each other, while at the same time forcing the classifications not to be too general but to tell something about the current input. The procedure is unsupervised in the sense that no teacher is required to tell T2 how to classify its inputs. With the symmetric case (see example 2 above), both TI and T2 are naturally treated in a symmetric manner. They share the goal of uniquely representing as many of their input patterns as possible-under the constraint of emitting equal (and therefore mutually predictable) classifications in response to a pair of input patterns. Such classifications represent whatever abstract properties are common to both patterns of a typical pair. For handling such symmetric tasks in a natural manner, we only slightly modify TI'S error function for the asymmetric case, by iqroducing an extra "discriminating" error term D1 for TI. Now both T I ,1 = 1,2 minimize EM+ (1- E)D,

(1.4)

where alternative possibilities for defining the DIwill be defined in the next section. Figure 1 shows a system based on (equation 1.4) and a particular implementation of DI(to be explained in Section 2.4). The assumption behind our basic approach is that a prediction that closely (in the euclidean sense) matches the corresponding classification is a nearly accurate prediction. Likewise, two very similar (in the euclidean

JiirgenSchmidhuber and Daniel Prelinger

628

prediction

pndiction

prediction

pediction

prediction

prediction

prediction

pdictim

Figure 1: Two networks try to transform their different inputs to obtain the same representation. Each network is encouraged to tell something about its input by means of the recent technique for “predictability minimization.”This technique requires additional intrurepresentationalpredictors (8 of them shown above) for detecting redundancies among the output units of the networks. Alternatives are provided in the text.

sense) classifications emitted by a particular network are assumed to have very similar ”meaning.” It should be mentioned that in theory, even the slightest differences between classifications of different patterns are sufficient to convey all (maximal) Shannon information about the patterns (assuming noise-free data). But then close matches between predictions and classifications could not necessarily be interpreted as accurate predictions. The alternative designs of DJ (to be described below), however, will have the tendency to emphasize differences between different classifications by increasing the euclidean distance between them (sometimes under certain constraints, see Section 2). There is another reason why this is a reasonable thing to do: In a typical application, a classifier will function as a preprocessor for some higher level network. We usually do not want higher level input representations with different “meaning” to be separated by tiny euclidean distance. Weight sharing. If both TIand T2 are supposed to provide the same outputs in response to the same inputs (this holds for the stereo task but does not hold in the general case) then we need only one set of weights


629

for both classifiers. This reduces the number of free parameters (and may improve generalization performance). Outline. The current section motivated and explained our basic approach. Section 2 presents various instances of the basic approach (based on various possibilities for defining 01). Section 3 mentions previous related work. Section 4 presents illustrative experiments and experimentally demonstrates advantages of our approach. 2 Alternative Definitions of

DI

This section lists four differentapproaches for defining DI, the term which enforces nontrivial discriminative classifications. Section 2.1 presents a novel method that encourages locally represented classes (like with winner-take-all networks). The advantage of this method is that the class representations are orthogonal to each other and easy to understand; its disadvantage is the low representation capacity. In contrast, the remaining methods can generate distributed class representations. Section 2.2 defines DIwith the help of autoencoders. One advantage of this straightforward method is that it is easy to implement. A disadvantage is that predictable information conveyed by some input pattern does not necessarily help to minimize the reconstruction error of an autoencoder (this holds for the stereo task, for instance). Section 2.3 mentions the Infomax approach for defining DIand explains why we do not pursue this approach. Section 2.4 finally defines DIby the recent method for predicfubility minimization (Schmidhuber 1992). An advantage of this method is its potential for creating distributed class representations with statistically independent components. 2.1 Maximizing Constrained Output Variance. We write

(2.1) and minimize DIsubject to the constraint vp:

C$'='

(2.2)

i

Here, as well as throughout the remainder of this paper, subscripts of symbols denoting vectors denote vector components: vi denotes the ith element of some vector D. is a positive constant, and y; denotes the mean of the ith output unit of TI. It is possible to show that the first term on the right-hand side of equation 2.1 is maximized subject to equation 2.2 if each input pattern is locally represented (just like with winnertake-all networks) by exactly one comer of the q-dimensional hypercube spanned by the possible output vectors, if there are sufficient output units

630

Jurgen Schmidhuber and Daniel Prelinger

(Prelinger 19921.' Maximizing the second negative term encourages each local class representation to become active in response to only ith of all possible input patterns. Constraint 2.2 is enforced by setting

where up?' is the activation vector (in response to x p i ) of a q-dimensional layer of hidden units of Ti,which can be considered as its unnormalized output layer. This novel method is easy to implement-it achieves an effect similar to the one of the recent entropy-based method by Bridle and MacKay (1992). 2.2 Autoencoders. With pattern p and classifier TI a reconstructor module A1 (another backpropagation network) receives yp,' as an input. The combination of TIand A1 functions as an autoencoder. The autoencoder is trained to emit the reconstruction bpi' of TI'Sexternal input xp,', thus forcing yp-' to tell something about xpg'. DIis defined as

2.3 Infomax. Following Linsker's Infomax approach (Linsker 1988), we might think of defining -DI explicitly as the mutual information between the inputs and the outputs of TI. We did not use Infomax methods in our experiments for the following reasons: (1)There is no efficient and general method for maximizing mutual information. (2)With our basic approach from Section 1, Infomax makes sense only in situations where it automatically enforces high variance of the outputs of the TI (possibly under certain constraints). This holds for the simplifying gaussian noise models studied by Linsker, but it does not hold for the general case. (3) Even under appropriate gaussian assumptions, with more than one-dimensional representations, Infomax implies maximization of functions of the determinant DET of the covariance matrix of the output activations (Shannon 1948). In a small application, Linsker explicitly calculated DET's derivatives. In general, however, this is clumsy. 'Simply maximizing the variance of the output units without obeying constraint 2.2 will not necessarily maximize the number of different classifications. Example: Consider a set of four different four-dimensional input patterns 1O00, 0100, 0010, Oool. Suppose the classifier maps the first two input patterns to the four-dimensional output pattern 1100 and the other two to 0011. This will yield a variance of 4. A "more discriminative" response would map each pattern to itself, but this will yield a lower variance of 3.


631

2.4 Predictability Minimization. Schmidhuber (1992) shows how DI can be defined with the help of intrarepresentational adaptive predictors that try to predict each output unit of some TI from its remaining output units, while each output unit in turn tries to extract properties of the environment that allow it to escape predictability. This was called the principle of predictability minimization. This principle encourages each output unit of TI to represent environmental properties that are statistically independent from environmental properties represented by the remaining output units. The procedure aims at generating binary “factorial codes” (Barlow et al. 1989). It is our preferred method, because [unlike the methods used by Linsker (19881, Becker and Hinton (1989), and Zemel and Hinton (1991)l it has a potential for removing even nonlinear statistical dependencies2 among the output units of some classifier. Let us define D’ = --1 - $92 (2.4) 2 i

c(sy

where the sr’ are the outputs of Sl, the ith additional so-called intrarepresentational predictor network of TI (one such additional predictor network is required for each output unit of 7’1). The goal of Si is to emit the conditioned expectation of given {$”, k # i } . This goal is achieved by simply training Sl to predict 8’ from {I&’, k # i } (see Fig. 1). To encourage even distributions in output space, we slightly modify DI by introducing a term similar to the one in equation 2.1 and obtain

e’

(2.5) 3 Relation to Previous Work

Becker and Hinton (1989) solve symmetric problems (like the one of example 2, see Section 1) by maximizing the mutual information between the outputs of TI and T2 (IMAX). This corresponds to the notion of finding mutually predictable yet informative input transformations. One variation of the IMAX approach assumes that TI and T2 have single binary probabilistic output units. In another variation, TI and T2 have single real-valued output units. The latter case, however, requires certain (not always realistic) gaussian assumptions about the input and output signals (see also Section 2.3 on Infomax). In the case of vector-valued output representations, Zemel and Hinton (1991) again make simplifymg gaussian assumptions and maximize functions of the determinant D of the 9 x 9-covariance matrices ( D E T M A X ) of the output activations (Shannon 1948) (see Section 2.3). DETMAX *SteveNowlan has described an alternative nonpredictor-based approach for finding nonredundant codes (Nowlan 1988).

632


can remove only linear redundancy among the output units. (It should be mentioned, however, that with Zemel’s and Hinton’s approach the outputs may be nonlinear functions of the inputs.) The nice thing about IMAX is that it expresses the goal of finding mutually predictable yet informative input transformations in a principled way (in terms of a single objective function). In contrast, our approach involves two separate objective functions that have to be combined using a relative weight factor. An interesting feature of our approach is that it conceptually separates two issues: (1) the desire for discriminating mappings from input to representation, and (2) the desire for mutually predictable representations. There are many different approaches (with mutual advantages and disadvantages) for satisfying (1). In the context of a given problem, the most appropriate alternative approach can be “plugged into” our basic architecture. Another difference between IMAX and our approach is that our approach enforces not only mutual predictability but also equality of yp,’ and yPs2. This does not affect the generality of the approach. Note that one could introduce additional “predictor networks”4ne for learning to predict yP,2from yp,’and another one for learning to predict yp~’from ypv2. Then one could design error functions enforcing mutual predictability (instead of using the essentially equivalent error function M used in this paper). However, this would not increase the power of the approach but would only introduce unnecessary additional complexity. In fact, one advantage of our simple approach is that it makes it trivial to decide whether the outputs of both networks essentially represent the same thing. The following section includes an experiment that compares IMAX to our approach. 4 Illustrative Experiments

The following experiments were conducted using an online backpropagation method with constant step size. With each experiment, positive training examples were randomly drawn from the set of legal pairs of input patterns. Details can be found in Schmidhuber and Prelinger (1992). 4.1 Finding Predictable Local Class Representations. This experiment was motivated by example 1 (see Section 1). At a given time, the “next” symbol emitted by a very simple stochastic “language generator” was not precisely predictable from the “previous” symbol but belonged to a certain class defined by the previous symbol. During training, at a given time TI saw the previous symbol while T2 saw the next symbol. T1minimized equation 1.3,T2 minimized equation 1.2 with D2 defined according to equations 2.1 and 2.2. Ten test runs with 15,000 training it-


633

erations were conducted. T2 always learned to emit different localized representations in response to members of predictable classes, while superfluous output units remained switched off. 4.2 Stereo Task. The binary stereo experiment described in Becker and Hinton (1989)(see also example 2 in Section 1) served to compare IMAX to our approach. Becker and Hinton report that their system (based on binary probabilistic units) was able to extract the “shift” between two simple stereoscopic binary images only if IMAX was applied in successive ”layer by layer” bootstrap stages. In addition, they heuristically tuned the learning rate during learning. Finally they introduced a maximal weight change for each weight during gradient ascent. In contrast, the method described herein (based on continuous-valued units) does not rely on successive bootstrap stages or any other heuristic considerations. We minimized equation 1.4 with DIdefined by predictability minimization according to equation 2.5. With a first experiment, we employed a different set of weights for each network. With 10 test runs involving 100,000 training patterns the networks always learned to extract the stereoscopic shift. This performance of our nonbootstrupped system is comparable to the performance of Becker and Hinton’s bootstrapped system. With a second experiment, we used only one set of weights for both networks (this leads to a reduction of free parameters). The result was a significant decrease of learning tim-with 10 test runs the system needed between 20,000 and 50,000 training patterns to learn to extract the shift.

4.3 Finding Predictable Distributed Representations. Two properties of some binary input vector are the truth values of the following expressions:

1. There are more “ones” on the “right” side of the input vector than on the “left” side. 2. The input vector consists of more “ones” than “zeros.”

During one learning cycle, a randomly chosen legal input vector was presented to TI#another input vector randomly chosen among those with the feature combination of the first one was presented to Tz.TI and TZ were constrained to have the same weights. Input vectors with equal numbers of ones and zeros as well as input vectors with equal numbers of ones on both sides were excluded. We minimized equation 1.4 with DIdefined by an autoencoder (equation 2.3). Ten test runs involving 15,000 pattern presentations were conducted. The system always came up with a distributed near-binary representation of the possible feature combinations.

634


With D, defined by modified predictability minimization (equation 2.5), with simultaneous training of both predictors and classifiers, 10 test runs involving 10,000 pattern presentations were conducted. Again, the system always learned to extract the two features. 5 Conclusion In contrast to the principled approach embodied by IMAX, our methods (1)tend to be simpler (e.g., do not require sequential layer by layer "bootstrapping" or learning rate adjustments-the stereo task can be solved more readily by our system), (2) do not require gaussian assumptions about the input or output signals, (3) do not require something like DETMAX, and (4) partly have (unlike DETMAX) a potential for creating classifications with statistically independent components (this holds for DI defined according to Section 2.4). In addition, our approach makes it easier to decide whether the outputs of both networks essentially represent the same thing. The experiments above show that the alternative methods of Section 2 can be useful for implementing the D,terms in equation 1.4 to obtain predictable informative input transformations. More experiments are needed, however, to become clear about their mutual advantages and disadvantages. It also remains to be seen how well the methods of this paper scale to larger problems.

Acknowledgments We thank Mike Mozer for fruitful discussions and Mike Mozer, Sue Becker, Rich Zemel, and an unknown referee for helpful comments on drafts of this paper. This research was supported in part by a DFG fellowship to J. Schmidhuber, as well as by NSF Award IN-9058450, Grant 90-21 from the James S. McDonnell Foundation.

References Barlow, H. B., Kaushal, T. P., and Mitchison, G. J. 1989. Finding minimum entropy codes. Neural Comp., 1(3), 412-423. Becker, S., and Hinton, G. E. 1989. Spatial coherence as an internal teacher for a neural network. Tech. Rep. CRG-TR-89-7, Department of Computer Science, University of Toronto, Ontario. Bridle, J. S., and MacKay, D. J. C. 1992. Unsupervised classifiers, mutual information and 'phantom' targets. In Advances in Neural Information Processing Systems 4, D. S. Lippman, J. E. Moody, and D. S. Touretzky, eds., pp. 10961101. Morgan Kaufmann, San Mateo, CA. LeCun, Y. 1985. Une procedure d'apprentissage pour rbseau A seuil asymbtrique. Proceedings of Cognitiva 85, Paris, 599-604.


635

Linsker, R. 1988. Self-organization in a perceptual network. IEEE Computer 21, 105-117. Nowlan, S. J. 1988. Auto-encoding with entropy constraints. In Proceedings of ZNNS First Annual Meeting, Boston, M A . Also published in special supplement to Neural Networks. Parker, D. 8. 1985. Learning-logic. Tech. Rep. TR-47, Center for Comparative Research in Economics and Management Science, MIT. Prelinger, D. 1992. Diploma thesis. Institut fiir Informatik, Technische Universitat Miinchen. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, Vol. 1, pp. 318-362. MIT Press, Cambridge, MA. Schmidhuber, J. H. 1992. Learning factorial codes by predictability minimization. Neural Comp. 4(6), 863-879. Schmidhuber, J. H., and Prelinger, D. 1992. Discovering predictable classifications. Tech. Rep. CU-CS-626-92, Department of Computer Science, University of Colorado at Boulder. Shannon, C. E. 1948. A mathematical theory of communication (parts I and 11). BelI System Tech. 1. XXVII, 379-423. Werbos, P. J. 1974. Beyond regression: N a o tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University. Zemel, R. S., and Hinton, G. E. 1991. Discovering viewpoint-invariant relationships that characterize objects. In Advances in Neural Information Processing Systems3, D. S. Lippman, J. E. Moody, and D. S. Touretzky, eds., pp. 299-305. Morgan Kaufmann, San Mateo, CA. Received 2 June 1992; accepted 5 January 1993.

This article has been cited by: 2. Sepp Hochreiter , Jürgen Schmidhuber . 1999. Feature Extraction Through LOCOCODEFeature Extraction Through LOCOCODE. Neural Computation 11:3, 679-714. [Abstract] [PDF] [PDF Plus] 3. Virginia R. de Sa , Dana H. Ballard . 1998. Category Learning Through Multimodality SensingCategory Learning Through Multimodality Sensing. Neural Computation 10:5, 1097-1117. [Abstract] [PDF] [PDF Plus] 4. Jim Kay, W. A. Phillips. 1997. Activation Functions, Computational Goals, and Learning Rules for Local Processors with Contextual GuidanceActivation Functions, Computational Goals, and Learning Rules for Local Processors with Contextual Guidance. Neural Computation 9:4, 895-910. [Abstract] [PDF] [PDF Plus] 5. J. Schmidhuber, S. Heil. 1996. Sequential neural text compression. IEEE Transactions on Neural Networks 7:1, 142-146. [CrossRef]

Communicated by Thomas Bart01

A Kinetic Model of Short- and Long-Term Potentiation M. Migliore lnstitute for lnterdisciplinary Applications of Physics, National Research Council, Via Arckirafi 36,l-90123 Palermo, Italy

G . F. Ayala Department of Psyckopkysiology, University of Palermo, Via Pascoli 8,l-90143 Palermo, ltaly

We present a kinetic model that can account for several experimental findings on short- and long-term potentiation (STP and LTP) and their pharmacological modulation. The model, which is consistent with Hebb’s postulate, uses the hypothesis that part of the origin of LTP may be a consequence of an increased release of neurotransmitter due to a retrograde signal. The operation of the model is expressed by a set of irreversible reactions, each of which should be thought of as equivalent to a set of more complex reactions. We show that a retrograde signal alone is not sufficient to maintain LTP unless long-term change of the rate constant of some of the reactions is caused by high-frequency stimulation. Pharmacological manipulation of LTP is interpreted as modifications of the rate constants of one or more of the reactions that express a given mechanism. The model, because of its simplicity, can be useful to test more specific mechanisms by expanding one or more reactions as suggested by new experimental evidence. 1 Introduction STP ( C o h o etal. 1992)and LTP (Bliss and Lomo 1973; Bliss and GardnerMedwin 1973)are the short- or long-lasting increases of synaptic coupling that follow a train of conditioning stimuli. These phenomena are highly reproducible, nevertheless the molecular mechanisms involved in their induction and maintenance are still unclear (Edwards 1991). They are perhaps the most elementary step toward higher brain functions such as memory, learning, associative recalling, and the process of cognition. At this time, only a key role of calcium is well accepted and experimentally demonstrated. Although considerable efforts have been made (Byrne and Berry 1988; Koch and Segev 1989), using a variety of computer models (e.g., Gamble and Koch 1987; Segev and Rall1988; Holmes and Levy 1990), simulations have failed so far to reproduce their central feature, that is the short- or long-lasting enhancement of the excitatory Neural Computation 5, 636-647 (1993) @ 1993 Massachusetts Institute of Technology

A Kinetic Model of Short- and Long-Term Potentiation

637

I

retrograde

presynaptic signal

a

bouton

neurotransmitter level

postsynaptic signal

spine

Figure 1: Schematic representation of the model. Note that the equations derived from this model are a set of simultaneous differentialequations. Each step should be thought of as a representation of a more complex set of events. The two pathways of the presynaptic signal for the production of V are independent in this model. postsynaptic potentials (EPSPs) obtained after high-frequency stimulation (HE) of the afferent pathway. The model presented in this paper mimics this experimental observation. It is based on the growing evidence of a determining role of a retrograde signal (Bredt and Snyder 1992) for the induction of the potentiation, and indicates further that, in order to maintain the potentiation, the rate constant of some reactions should change during the HFS. 2

T h e Model

Our model (Fig. 1)represents a simplified view of the events of the synaptic transmission, where I is an intensive independent variable that represents the input stimulus, and V, C, and K are, respectively, the level of released neurotransmitter molecules, the postsynaptic signal, and the retrograde signal. The operation of the model is expressed by a set of irreversible events, symbolically represented by the following reactions (and corresponding simultaneous differential equations), each of which should be thought of as equivalent to a set of more complex reactions:

I - % V V P C C A K I + K a, V

c’.

(a)

(b) (C)

(d) (e)

The release of neurotransmitter on the synaptic cleft and the production of an output signal are represented, respectively, by reactions (a) and

M. Migliore and G.F. Ayala

638

(b). Reaction (e) represents the degradation of C that could be thought of as the EPSP. Processes represented in these reactions are the classic synaptic events having ample experimental support. Reactions (c) and (d), instead, model (in a way that shall prove productive) the current controversial view (Edwards 1991) that the origin of LTP is due to an increased release of neurotransmitter secondary to a retrograde signal [e.g., nitric oxide (Bredt and Snyder 199211 acting in the feedback loop. In these reactions the variable K is the level of this retrograde signal or plasticity factor (Bliss and Lynch 1988). The system of differential equations derived from this model can be solved analytically. Linear stability analysis (Nicolis and Prigogine 1977) shows that the system is always stable. The only stationary solution that it admits, for 1 > 0, is

Let us now consider the behavior of the system as a consequence of a short pulse of I. While the pulse lasts, the values of V, C, and K will tend to those in equations 1, reaching final values VO, CO,and &, which will depend on the pulse length. From the end of the pulse on, the behavior of the system is described by

v

= Voe-flr

(2.1)

e-Pt)

(2.2)

An interesting consequence of equation 2.2 is that for CO= 0 it reduces to the widely used phenomenological formula describing the time course of EPSPs in terms of empirical times of onset, TO and decay, TD, that is, V E E P = COnSt[TD/(TD-TD)](eWf/m- e d T O ) , The phenomenological formula is thus derived from our model with TO = I/@and TO = l / ( y E ) . Integration of the system of differential equations derived from the model has been carried out with a fourth order Runge-Kutta method (Press et al. 1987)with a fixed time step of 1.O x W4. It should be stressed that the model is too simplified to justify a quantitative comparison with specific biophysical mechanisms that, at the time being, are thought to have a major role in the induction and maintenance of LTP. At this stage, for example, y could represent all those mechanisms that may result in the production of a retrograde signal (e.g., from calcium buffering by calmodulin to NO synthesis by nitric oxide synthase), p the kinetics of activation of ionic currents and E might include leakage, kinetics of

+


639

inactivation of ionic currents and Ca2+pumping, diffusion, and buffering. As we will see, to be qualitatively consistent with experiments, it is not necessary to specify in more details any of these processes, at least within the limits of the scope of this paper. 3 Results and Discussion

3.1 Simulations. Although the values of the rate constants used in our simulations were set arbitrarily, they were chosen to give a relationship among them that is physiologically reasonable and to be convenient from a computational point of view. In fact, E , the rate constant of the “postsynaptic signal degradation,” is the smallest of all the rate constants in order to use most of the postsynaptic signal ”for LTP purposes”; 0 > (y + e), that is, the rate constant of ”channel activation” for the production of C is larger than the sum of the rate constants of the “production of the retrograde signal“ K, and of the “postsynaptic signal degradation,” so that C is always > 0; 6, the rate constant for “the production of neurotransmitter, V, by the retrograde signal,” has a value to yield an amplitude of STP roughly comparable with experimental observations, and y is such that the 1/(7 E ) time constant is not too long, to avoid unnecessarily long simulation times. It turns out that all values are within one order of magnitude, and different values have been tried with no essential differences in the results. The actual model parameters are reported in the legend of Figure 2. The time course of quantities shown in Figure 2 [as simulated according to the equations derived from reactions (a)-(e)] starts from equilibrium with 1 = 0.01, V = VOZ V,, C = CO C, and K = KO K,. If there is enough time between stimuli for all the variables to return to equilibrium, each new stimulus will find the system in unperturbed conditions. This, however, will not be true in the HFS case, that is when the interstimulus interval is shorter than any of the time constants involved in equations 2 in EPSP decaying, that is y and E. In such case, and since > (y E), the average values of V and C will increase. As shown in Figure 2, at the end of HFS, the released neurotransmitter, V, and the postsynaptic signal, C, will return to the equilibrium values but K will be K H s > KO. Thus, the net result of a HFS, is to increase the final value of K to KHB, that will be the new initial, nonequilibrium, condition for the following stimuli. The larger K produces, at each pulse, Vs larger than the ones before HFS, resulting in higher C peaks and initial slopes. This phase can be considered as the equivalent of the STP in the experiments (Colino et al. 19921, and its time scale is essentially dependent on the value of the current in the presynaptic cell [i.e., the 1 independent variable in reaction d ) ] . This prediction of the model might be experimentally tested using different presynaptic holding potentials. It should be noted that still another form of short-lasting potentiation has been

+

+

M. Migliore and G. F. Ayala

640

2

I

1

I

C 1 -

-0 2

K 1 -

I

I

I

0

50

100

150

200

0.0 0

50

100

150

200

0

v 0.5

HFS

Time (a.u.)

Figure 2: Time course of the C, V, and K signals from simulation withlow- and high- (HFS) frequency stimuli. Amplitudes and times are given in arbitrary units. Values for the rate constants are a = 2, p = 5, y = 1, 6 = 10, E = 0.5. Pulses of I, not shown, are of amplitude 1 (arbitrary units) and duration 50 time steps. Interstimulus interval is 4950 and 50 time steps for low and high frequency, respectively. The raising time of K after the HFS as well as after each I pulse is mainly determined by the l / ( y + E ) time constant (see equation 2.3). The duration of the STP phase will also depend from l / ( y + E ) , but it is mainly determined by 6, as consequenceof reaction (d) and by the background current I (in this case I = 0.01).


HFS

time (a.u.)

641

HFS -

time (a.u.)

Figure 3: Left: Time course of C, V, and K from a simulation with reaction (d) changed to K 5 V and KO = 0. Right: Time course of C (solid)and V (dashed) from a simulation with y = 0 and & = 0. Other rate constants, parameters, and protocol of stimulation as in Figure 2. described, the posttetanic potentiation (PTP),and it is usually assumed that it involves an entirely presynaptic mechanism. Our model does not take into account this kind of potentiation because we are assuming that the mechanisms involved in STP and LTP are different from those of I ", namely the role of the retrograde signal. The role of the interaction between I and K in inducing SlT and LTP is also evident from the model. The schematic picture in Figure 1 shows, in its presynaptic portion, two separate pathways for I to produce the neurotransmitter V the first, directly, with a rate constant a, and, the second, interacting with the retrograde signal K, with a rate constant 6. Let us for a moment suppose that the interaction of I with the retrograde signal, K, is impaired, but the first pathway is left intact. In the simulations this is accomplished by suppressing I in reaction ( d ) , changing it to K6'V In such conditions, Figure 3 (left) shows that, using the same protocol of stimulation as before, the synaptic transmission is not impaired, and yet, there is no STP.This example illustrates the role of I in the interaction. The role of K is seen from Figure 3 (right), obtained for y = 0 that is K = 0. The requirement for the simultaneous presence of both I and K is of course consistent with Hebbs postulate (Hebb 1949). This is because the postsynaptic depolarization produces the retrograde signal,

642

M. Migliore and G . F. Ayala

K, capable of enhancing the production of V in response to presynaptic signal. The simulation reported in Figure 4a, where all reaction rate constants are kept fixed, is conveniently compared with the experimental data of Figure 4b. The comparison shows that the peak amplitudes of C (see also Figure 21, in response to stimuli that are qualitatively similar to the standard experimental protocol used to obtain LTP, are not similar to the typical experimental results on LTP but to those on STP. We tested several different simple models, but all of them consistently failed to show LTP. In fact, the system is always stable and it admits only one stable solution. Since the basic idea is to use the retrograde signal, to obtain LTP we can (1) consider alternative and more complex kinetic pathways (e.g., autocatalytic processes) using the retrograde signal or (2) assume that H E triggers the change of some rate constant. Both ways can make explicit predictions and can be used as useful tools to help the interpretation of experiments. Since our major objective in this paper is to keep the model as simple as possible to be mathematically tractable and easily modifiable to include more detailed specific processes, we follow (2). From equations 1 and 2 it follows that the peak value and the initial slope of the output signal C, after HFS, are determined by the equilibrium value of the retrograde signal, K,, and from the rate at which the neurotransmitter produces an output signal, 0.Thus an increase of or any change in some other rate constant that increases K, will give LTP as shown, for example, in Figure 4 c 4 , as obtained by mere alterations of the y and E rate constants [an increase and a decrease, respectively, to keep TD = l / ( y + E ) constant]. These changes appear to be physiologically reasonable and expected if one consider that they correspond to a "more efficient" production of retrograde signal, y, and a change in the processes involved in the postsynaptic signal degradation, E . More important, however, these changes are consistent only with the experimental fact (Gustafsson and Wigstrom 1990) that the time evolution of potentiated EPSPs does not change with LTP, after normalization. It should also be stressed that these changes are to be considered as the net results of changes to kinetic parameters of specific reactions underlying our simple model. The agreement with actual LTP experiments is clearly evident from Figure 4c and d. The transitory increase of C (Fig. 2) during H E has already been simulated and explained in terms of several reactions that involve calcium pumping, diffusion, and buffering in the spine head and neck (Gamble and Koch 1987; Holmes and Levy 1990) as well as the electrotonic characteristics of the membrane. The simple hypothesis at the basis of our model is that the increase of the postsynaptic signal C produces a large retrograde signal that triggers the induction of LTP. Its maintenance is obtained by a change in the rate constant involved in the production of the retrograde signal, and we propose that this change re-


643

0.6

b) 0.4 -

.. .. ..... ......... .:. ...... 0.2 :,:;:': *:., ..$ ;*

.:*,;:t

'

..

0.8

4

*

0.6 ;**:

. . :.. .-.

.:.a

; a : *

0.4 .

....."..'-: "*:. .... ..... .. I

0.2 ;

I time (a.u.)

time (min)

Figure 4: Peak amplitudes of C ( a ) compared with typical experimental findings on STP (b). The arrow on the time axis indicate the time of application of HFS. Parameters and protocol of stimulation as in Figure 1. Peak amplitudes of C (c), from a simulation where at the end of HFS (arrow) the y rate constant is increased to 1.2 and E decreased to 0.3, compared with typical experimental findings on LTP (d). Other rate constants and protocol of stimulation as in Figure 1. (Experimental data taken and redrawn from Colin0 et al. 1992.)


644

quires a short-term increase in the concentration of the retrograde signal itself.

3.2 Comparison with Experimental Data. The model can account for several experimental observations on the mechanisms involved in LTP, as shown by work currently in progress at our laboratory. For example, the transient depression followed by short-term potentiation after N-methylBaspartate (NMDA)application (Kauer et al. 1988) can be reproduced by assuming an initial reduction of & and the transitory increase of y and B. The presence of extracellular hemoglobin or inhibitors of NO production such as nitro-L-arginine methyl ester (L-NAME)has been shown (Haley et al. 1992) to prevent the maintenance of LTP. In terms of our model this observation suggests that the triggering cause required to produce LTP is not, or not only, the high postsynaptic calcium concentration, but the increase of the retrograde signal. Any pharmacological manipulation that interferes with this transient increase during HFS may prevent LTP. However, at this point the model cannot account for those experiments where the application of L-NAME after LTP has been established fails to inhibit the maintenance of the potentiation. The model easily accounts also for the finding that LTP cannot be induced by an HFS without a conjunctive depolarization of the postsynaptic cell, for example, when the postsynaptic cell is voltage clamped (Kelso et al. 1986). In fact, a voltage clamp corresponds, in our model, to force C to a low value during HFS.This has the effect to prevent the increase in the level of the retrograde signal K during HFS, and thus inhibiting the changes of the rate constants, that we assume to be consequent to the high K. It has been shown that calcium chelators ethylene-bis(oxyethy1enenitrite) tetra-acetic acid (EGTA) (Lynch et al. 1983) and Nitr-5 (Malenka et al. 1988) prevent that the Ca2+ influx through NMDA channels contributes to the EPSP potentiation. Since it is known that ionic flux through NMDA channels mediates the late component of an EPSP, we can simulate the effect of the chelators with an increase of the rate constant that represents all the processes involved with the EPSP decay, that is E. The model indicates that because of increase of E, the depolarization during a HFS is not sufficiently large to increase the retrograde signal to a level necessary to obtain LTP. Finally, two different types of potentiation have been reported in r e cent experiments (Gustafsson and Wigstrom 1990). They differ in the changes in the time course of the recorded EPSPs. In one case, only the peak value increases, without changes in their time evolution characteristics. In the other case, the potentiation is expressed as a prolongation of the onset time and a larger peak amplitude. In terms of equation 2.2, both findings can be explained as changes of y and E , such to maintain TD = l / ( y E ) constant or not, respectively.

+


645

4 Conclusions

Each of the (a)-(e) reactions should be considered (as already remarked) as representative of an equivalent set of more complex reactions. The purpose of the present work is to assess a simplified environment, qualitatively consistent with the available experimental data, on which one can expand one or more reactions in order to test a specific mechanism as suggested by experimental evidence. From this point of view, our model does not takes explicitly into account, for example, the kinetics of ionic fluxes through NMDA or other channels, or calcium-dependent protein kinases (Kitajima and Hara 1990). The detailed modeling of these or other processes is, of course, needed to define more precisely the still unknown biophysical mechanisms involved with LTP and to obtain quantitative agreement with experimental data. On the backbone of our set of reactions, any additional process can further modulate the induction or the expression of LTl? We believe that our model can be useful to stimulate discussion and experimentation in the field. The present model uses the hypothesis that part of the origin of LTP may be in the (presynaptic) increased release of neurotransmitter provoked by a retrograde signal produced by a postsynaptic mechanism (depolarization and/ or high calcium concentration). The simulations, using the model presented, further support this hypothesis because the results obtained are (1) in very good qualitative agreement with experimental data on STP and LTP, (2) consistent with Hebb’s postulate, (3) consistent with the expected time course for EPSPs in terms of molecular rate constants rather than empirically, and (4) consistent with and capable of reproducing the effects of several LTP modulators. In this case the model can also predict the possible kinetics of the retrograde signal itself (Fig. 2). Moreover, the model has shown (Fig. 4) that although the presence of a retrograde signal is enough to induce a potentiation very similar to STP, it is not sufficient to induce the maintenance of LTP, and other postsynaptic events at a molecular level (such as the change of the rate constant of its production) are also necessary.

Acknowledgments We thank Prof. M. U. Palma for a critical reading of the manuscript and making valuable suggestions, Prof. S. L. Fornili and Prof. A. Messina for useful discussions, and Mr. S. Pappalardo for technical assistance. This work has been carried out at IAIF-CNR and supported also by CRRN-SM local funds.

646


References

Bliss, T. V. l?, and Gardner-Medwin, A. R. 1973. Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232,357-374. Bliss, T. V. P., and Lomo T. 1973. Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232, 331-356. Bliss, T. V. P.,and Lynch, M. A. 1988. Long-term potentiation of synaptic transmission in the hippocampus: Properties and mechanisms. In Long-Term Potentiation: From Biophysics to Behavior, l? w. Landfield and S. A. Deadwyler, eds., pp. 3-72. Alan R. Liss, New York. Bredt, D. S., and Snyder, S. H. 1992. Nitric Oxide, A novel neuronal messenger. Neuron 8, 3-11. Byrne, J. H., and Berry, W. O., eds. 1988. Neural Models ofPlasticity. Experimental and Theoretical Approaches. Academic Press, New York. Colino, A., Huang, Y.-You, and Malenka, R. C. 1992. Characterization of the integration time for the stabilization of long-term potentiation in area CAI of the hippocampus. 1. Neurosci. 12, 180-187. Edwards, E. 1991. LTP is a long term problem. Nature (London) 350, 271. Gamble, E., and Koch, C. 1987. The dynamics of free calcium in dendritic spines in response to repetitive synaptic input. Science 236, 1311-1315. Gustafsson, B., and Wigstrom, H. 1990. Basic features of long-term potentiation in the hippocampus. Semi. Neurosci. 2, 321-333. Haley, J. E., Wilcox, G. L., and Chapman, P. F. 1992. The role of nitric oxide in hippocampal long-term potentiation. Neuron 8, 211-216. Hebb, D. 0. 1949. The Organization of Behavior. Wiley, New York. Holmes, W. R., and Levy, W. B. 1990. Insights into associative long-term potentiation from computational models of NMDA receptor-mediated calcium influx and intracellular calcium concentration changes. J. Neurophysiol. 63, 1148-11 68. Kauer, J. A., Malenka, R. C., and Nicoll, R. A. 1988. NMDA application potentiates synaptic transmission in the hippocampus. Nature (London) 334, 250-252. Kelso, S. R., Ganong, A. H., and Brown, T. H. 1986. H&bian synapses in hippocampus. Proc. Natl. Acad. Sci. U.S.A. 83, 5326-5330. Kitajima, T., and Hara, K. 1990. A model of the mechanisms of long-term potentiation in the hippocampus. Biol. Cybern. 64, 33-39. Koch, C., and Segev, I., eds. 1989. Methods in Neuronal Modeling: From Synapses to Networks. MIT Press, Cambridge, MA. Lynch, G., Larson, J., Kelso, S., Barrionuevo, G., and Schottler, F. 1983. Intracellular injections of EGTA block induction of hippocampal long-term potentiation. Nature (London) 305, 719-721. Malenka, R. C., Kauer, J. A,, Zucker, R. J., and Nicoll, R. A. 1988. Postsynaptic calcium is sufficient for potentiation of hippocampal synaptic transmission. Science 242, 81-84.


647

Nicolis, G., and Prigogine, I. 1977. Self-Organization in Nonequilibrium Systems. From Dissipative Structures to Order through Fluctuations. Wiley, New York. Press, W. H., Flannery, 8. P., Teukolsky, S. A., and Vetterling, W. T. 1987. Numerical Recipes. The Art of Scientific Computing. Cambridge Univ. Press, Cambridge. Segev, I., and Rall, W. 1988. Computational study of an excitable dendritic spine. I. Neurophysiol. 60, 499. Received 5 June 1992; accepted 24 November 1992.

This article has been cited by: 2. Alain Destexhe, Zachary F. Mainen, Terrence J. Sejnowski. 1994. Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. Journal of Computational Neuroscience 1:3, 195-230. [CrossRef]

Communicated by Anthony Zador and Christof Koch

Artificial Dendritic Trees John G . Elias Department of Electrical Engineering, University of Delaware, N m r k , DE 19716 U S A

The electronic architecture and dynamic signal processing capabilities of an artificial dendritic tree that can be used to process and classify dynamic signals is described. The electrical circuit architecture is modeled after neurons that have spatially extensive dendritic trees. The artificial dendritic tree is a hybrid VLSI circuit and is sensitive to both temporal and spatial signal characteristics. It does not use the conventional neural network concept of weights, and as such it does not use multipliers, adders, look-up-tables, microprocessors, or other complex computational units to process signals. The weights of conventional neural networks, which take the form of numerical, resistive, voltage, or current values, but do not have any spatial or temporal content, are replaced with connections whose spatial location have both a temporal and scaling significance. 1 Introduction

Interest in using artificial neural networks for the identification and control of dynamic systems is growing (e.g., Narendra and Parthasarathy 1990). However, most neural network models do not include spatiotemporal dynamic signal processing capabilities. In these models, the neuron is treated as a point entity that receives and processes inputs at the soma, which makes spatial signal processing difficult or impossible. In our modeling approach, we have looked beyond the soma to the extensive dendritic tree structure of neurons, which not only forms most of the cell's surface area but provides spatiotemporal signal processing capabilities not present in models that assume a point-entity neuron. The artificial dendritic tree described in this paper is a hybrid circuit and is sensitive to both temporal and spatial signal characteristics, but it does not use the conventional neural network concept of weights, and as such it does not require multipliers, adders, look-up-tables, or other complex computational units to process signals. The weights of conventional neural networks, which take the form of numerical, resistive, voltage, or current values, but do not have any spatial or temporal content, are replaced in our system with connections whose spatial location have both a temporal and scaling significance. Neural Computation 5, 648-664 (1993) @ 1993 Massachusetts Institute of Technology

Artificial Dendritic Trees

649

We have only recently begun to experiment with networks of artificial dendritic trees (Elias 1992a). We have fabricated and tested artificial dendritic branches in CMOS analog VLSI (Elias et al. 1992), and we have used a genetic algorithm to train simple networks to follow a maneuvering target that moves in one dimension (Elias 1992b). The research described here attempts to capture neurocomputational principles by applying structure and behavior modeled after synaptic and dendritic levels of biological implementation. We hope to demonstrate that electronic analogs of biological computational devices, that include the properties of spatially extensive dendritic trees and the impulse response of chemical synapses, can form the basis for powerful artificial neurosystems. 2 Artificial Dendrite and Chemical Synapse

In this section, we describe electronic circuits that (1) emulate the electrical behavior of passive dendritic trees and chemical synapses and (2) are simple and robust enough to ensure that networks, which ultimately need to support huge numbers of synapses, can be constructed with standard VLSI processing. Electronic analogs of active dendrite behavior (e.g., Llinas and Sugimori 1980; Shepherd et al. 1985, 1989; Hounsgaard and Midtgaard 1988) will not be treated in this paper. 2.1 Artificial Dendrite. Passive artificial dendrites are formed by a series of standard compartments, where each compartment has a capacitor, Cm,that represents the membrane capacitance, a resistor, Rm, that represents the membrane resistance, and an axial resistor, &, that represents the cytoplasmic resistance (e.g., Rall 1989). Figure la shows a section of artificial dendrite with five standard compartments that is part of a much longer branch like that shown in Figure lb. The transient response of the artificial dendrite is of primary importance. Figure l c shows the impulse response measured at point S due to inward impulse current at four different locations, A, B, C, and D on a passive artificial dendrite as represented in Figure lb. The location S represents the position of the soma. Therefore, the voltages measured at S are those that would affect somatic voltage-sensitive circuits and perhaps cause the generation of an efferent impulse. As with biological passive dendrites, the peak voltage amplitude is largest for transmembrane current nearest the soma and gets rapidly smaller for sites farther away. The time for the voltage to peak shows a similar behavior: time-to-peakvoltage increases with distance from S (e.g., Rall 1989). The behavior shown in Figure 1 illustrates how the concept of weight is an inherent property of the dendritic physical structure. It is clear that position along the artificial dendrite can be used to produce an effective weighting, in both time and amplitude, of afferent signals that are in the form of a transient inward or outward current.

650

John G. Elias

Figure 1: (a) Compartmental model of passive dendrite. Each RC section, Rm, &, and C,, is a standard compartment that simplifies VLSI layout. (b) Standard compartments are abutted on substrate to form silicon dendritic branches. (c) Measured impulse response of single artificial dendritic branch due to transient transmembrane current at indicated locations on branch. 2.2 Artificial Chemical Synapse. The means for enabling inward or outward impulsive current at a specific artificial dendrite location is accomplished by using a single MOS field effect transistor. A p-channel transistor enables inward current, which produces an excitatory type response, and an n-channel transistor enables outward current, which results in an inhibitory type response. The complete artificial dendrite circuit is shown in Figure 2a, where both p-channel (upper) and n-channel (lower) transistors are placed at uniform positions along the branch. The transistors are turned on by an impulse signal applied to their gate terminals. Both transistor types operate in the triode region (e.g., Allen and Holberg 1987). Therefore, the amount of transmembrane current depends on the conductance of the transistor in the on state, the duration of the gate terminal impulse signal, and the potential difference across the transistor, which is dependent on the state of the dendrite at the point of the synapse. All excitatory transistors have identical drawn dimensions (as do inhibitory transistors), and both excitatory and inhibitory artificial synapses are placed at the same locations in the current chip implementation. It is possible that after further system level experimentation, we may find that a different synapse distribution is preferred.

2.3 Electrical Response of Artificial Dendritic Trees and Synapses. In Figure 1, we illustrated the behavior of the impulse response amplitude as a function of the synapse position on the artificial dendritic tree, thus demonstrating the effective weighing of inputs that are mapped onto the tree structure. The impulse response amplitude as a function of the afferent impulse signal width is shown in Figure 3a, which represents measured responses from one of our VLSI circuits for four different


651

S v

Figure 2: (a)A five compartmentsection of artificialdendrite with five excitatory and five inhibitory artificial synapses. V,,,, is the resting voltage, Vtop is the maximum membrane voltage. (b)A multibranched artificial dendritic tree that is constructed by piecing together artificial dendrite sections like that in (a). impulse widths. A similar postsynaptic behavior is found in biological preparations under presynaptic voltage-clamp: presynaptic depolarization produces a nearly linear increase in postsynaptic voltage (e.g., Angstadt and Calabrese 1991). This behavior may be due to a lengthening of the time over which transmitter is released, thereby increasing transmembrane current in the postsynaptic terminal. In any event, the efficacy of existing connections in our system can be changed by altering the impulse width. We are investigating how this may be done on a local basis, perhaps consistent with Hebbs postulate (Hebb 19491, such that both local synaptic strength and the location of the synapse on the branch combine to produce an effective synaptic weight for a given connection. The artificial dendrite's voltage response to closely spaced impulses is shown in Figure 3b. The response due to each synaptic event is added to the resultant branch point voltage from past events until the voltage reaches a maximum value. This behavior is the expected impulse response of an Nth-order system and is solely due to the effective postsynaptic membrane. The same behavior would be observed if the phasing of multiple, transiently conducting, artificial synapses was short compared to the effective membrane constant. An example that utilizes this behavior is discussed in Section 4. Multiple, simultaneously conducting synapses that are electrically close together produce a voltage at the soma that is less than the sum of their individual responses (e.g., Shepherd and Koch 1990). This sublinear effect is due to the shunting load seen by each synaptic site when electrically nearby synapses open their channels. In contrast, multiple, simultaneously conducting synapses that are electrically far apart produce a nearly linear resultant voltage at the soma. Both of these behaviors, as measured at the trunk of a two-branched artificial dendritic tree (e.g., point S in Fig. 2b), are shown in Figure 3c. The smaller voltage transient

John G. Elias

652

................ ......

op.

Figure 3: Experimental results from artificial dendrite-synapse circuit to afferent stimulation. (a) Graded response: amplitude of voltage peak at soma is linearly related to afferent impulse width over wide range. (b) Tetanus response: closely spaced impulses cause voltage response to saturate if impulse rate is faster than membrane decay time. (c) Nonlinear and nearly linear response: curve 1 is the resultant somatic voltage for simultaneous stimulation of two adjacent synapses on same branch (see Fig. 2b). Curve 2 is somatic voltage for simultaneous stimulation of two synapses on different branches. Positions of synapses for curves 1 and 2 were equidistant from soma.

(curve 1) was measured when two adjacent artificial synapses were simultaneously active on the distal end of one of the branches. The larger voltage transient (curve 2) shows the resultant voltage when two artificial synapses on separate branches were simultaneously active. In this case, the resultant is nearly twice that of the previous example. In both cases, the artificial synapses were equal distance from the point of measurement. This type of behavior clearly enriches the signal processing capabilities of systems comprised of spatially extensive dendritic trees (Koch and Poggio 1987). 3 Silicon Dendritic Trees

If artificial dendrites are to be used in real systems then they must be implemented via a process that can make huge numbers of them in a small area inexpensively. The only feasible path for doing this currently is with standard silicon processing methods (e.g., Mead 1989). In this section, we discuss briefly the implementation of a dendritic system in silicon. 3.1 Convergent, Divergent, and Recurrent Connections. Networks that are built with artificial dendrites and synapses process signals that have a spatiotemporal significance by mapping afferent signal pathways to specific locations on the dendritic trees. The connections between synapses and the outputs of sensors and neurons determine the overall system response for a. given dendritic dynamic behavior. The number of


653

different connection patterns is quite large and is a factorial function of the number of synapses and sensor elements. If we limit, for the moment, the number of divergent connections to one, then the total number of different connection patterns is given by

N! ( N - M)! where N is the number of artificial synapses and M is the number of sensor and neuron outputs. Artificial systems may have thousands of afferents and many times more synapses, resulting in an extremely large number of possible connection patterns. In our system, we allow each sensor element or artificial neuron to make unrestricted divergent connections and each synapse to receive multiple convergent connections from both sensor elements and artificial neurons. This tends to make the number of possible connection patterns much larger than that indicated by equation 1.

3.2 Virtual Wires. In the implementation of an electronic system, the number of data pathways in or out of modules is limited by the available technology. Integrated circuit packages rarely exceed 500 pins; our current artificial dendrite chip is in a 40 pin package. This limitation in pin count is of special concern with dynamic artificial neuronal systems because of the analog nature of the computation. Each sensor or neuron output must be able to connect to any one of the artificial synapses in the system, and the spiking outputs from sensors and neurons must arrive at their artificial synapse destinations in a parallel fashion. In order to overcome I/O limitations and to meet connectivity and timing requirements, we make use of a multiplexing scheme that we refer to as virtual wires. In this scheme, the outputs of active neurons and sensors (i.e., those that are currently producing a spike or impulse) cause the synapses that they connect with to become activated after a delay that is programmable for each efferent connection. The process of reading an active output causes that output to return to the inactive (i.e., nonspiking) state. After all sensor and neuron output states have been sampled, the activated synapses throughout the system are turned on transiently by a global impulse stimulus signal. The process of determining active sensors and artificial neurons, delayed activation of synapses that connect to active sensors and neurons, and transiently turning on active artificial synapses continues indefinitely with a period that depends on system dynamics. Virtual wires are formed using four digital circuits: Stimulus Memory, which is closely associated with each synapse, Address Decoding, which serves all on-chip synapses, State Machine, which determines sensor and neuron output states, and Connection List, which specifies the locations of synapses and the axonal delay associated with each connection. Stim-

654

John G. Elias

CUA.'

Figure 4: Circuit diagram of one of the excitatory Stimulus Memory registers shown with its p-channel excitatory synapse transistor. When SET* is asserted the synapse is activated (i.e., excsyn*[nl is set to logic 0) and the artificial synapse will turn on when STIMULATE*is asserted. CLEAR* inactivates all Stimulus Memory locations throughout the system. Both STIMULATE* and CLEAR* are global signals in the system.

ulus Memory and Address Decoding are on-chip circuits; the Connection List and State Machine are off-chip. The circuit diagram of an excitatory Stimulus Memory register connected to its p-channel artificial synapse transistor is shown in Figure 4. In the current implementation, virtual wires add nine transistors to each artificial synapse. A synapse is activated when its excsyn*[n] is set to logic 0 by asserting SET' while CLEAR' is unasserted. The SET' signal is asserted by the on-chip address decoder when the proper combination of external address lines and control signal is asserted. An activated synapse will turn on when the global impulse signal, STIMULATE*, is asserted. The global signal CLEAR* is asserted after every STIMULATE* assertion to inactivate synapses in preparation for the next round of sampling and activation. The Connection List is a multiple-bit-wide memory that holds thg synapse addresses and axonal delay of each efferent connection in its domain. For large systems, we plan to divide the network into domains that will permit a certain level of parallel sampling of neuron and sensor outputs, which should enhance system scalability. The Connection Lists across all domains hold the pattern of connectivity for the system and thus their contents determine system behavior. A connection pattern can be fixed in ROM,or as in our present system, loaded via computer for experimentation. Figure 5 illustrates a simplified single-domain system comprising Connection List, sensor, State Machine, and four neuromorphic chips, each of which contains a number of artificial neurons. The outputs of the artificial neurons on each chip are sampled via a multiplexer, which is


655

Figure 5 Simplified block diagram for single domain system showing basic operation. All sensory and neuronal outputs simultaneously activate the artificial synapses that they connect to through the virtual wires.

selected by the on-chip address decoder. Each neuron is in one of two states, so only a single output pin is needed to read all of them. In operation, the State Machine reads the state of each sensor element and every neuron in its domain. A spiking neuron or sensor output is detected by the State Machine, which then activates all of the synapses that connect to that particular sensor or neuron. After reading all outputs, STIMULATE* is asserted transiently, thus briefly turning on activated synapses. This is followed by asserting CLEAR*, which inactivates all synapses. As with Mahowald’s method of connecting neuron outputs to synapses (Mahowald 1992) addresses of synapses and neurons are used rather than direct connections carrying spikes.

3.3 Standard Dendrite Compartment. Figure 6 illustrates the basic integrated circuit layout of our standard dendrite compartment. Each compartment has a capacitor, C,,, that represents the membrane capacitance, a resistor, R,, that represents the membrane resistance, and an axial resistor, R,, that represents the cytoplasmic resistance. For the results

656

John G. Elias

Figure 6 Basic VLSI layout for standard dendrite compartment. Control lines for resistors permit adjustment of resistance over a limited range. Vmt establishes the resting voltage (typically 1 V). reported here, the size of the artificial dendrite standard compartment was 18 by 180 pm with most of this area being taken up by the capacitor. The capacitor is the largest element in the standard dendrite compartment and is implemented using conventional silicon processing methods (e.g., Allen and Holberg 1987). The capacitor was fabricated with two layers of polysilicon separated by a thin oxide layer. The top plate of the capacitor is polysilicon layer 2 (poly2) and connects to a ground bus that runs perpendicular to the long axis of the capacitor. The bottom plate is polysilicon layer 1 (polyl), which connects directly to the resistors, R, and R,,, and to the synapse transistors in the stimulus memory (see Figs. 4 and 6). The capacitance for the current size standard compartment capacitor is approximately 1 pF, which is based on an oxide thickness of approximately 700 A. There are many techniques to reduce the footprint of the capacitor while keeping the same capacitance: thinner dielectric, use material with a higher dielectric constant (e.g., silicon nitride), employ three-dimensional capacitors (e.g., trench or tower capacitors), but we will not explore these further here. The compartmental resistors may be implemented by a number of standard silicon fabrication techniques: well, pinched, active, and SC (e.g., Allen and Sanchez-Sinencio 1984). The resistor footprint for a particular resistance depends not only on the resistance value but also on the implementation technique. Well resistors are made by n- or p- diffusion and have a footprint advantage over the other techniques because the well resistor can be put under the capacitor. Therefore, a well resistor does not take up any silicon real estate but it has the disadvantage of relatively small resistance (measured as 5 kR per square for our chips). Pinched resistors have a higher resistance but can not be placed under the capacitor. Active and SC resistors are made from MOS transistor circuits that are designed to emulate resistor behavior over a certain range of terminal voltages and resistance values.


657

We have implemented n-well, active, and SC resistors on different chips and will report on the details of their design and relative behavior later (Elias and Meshreki 1993). In our standard dendrite compartment, n-well resistors go under the capacitor and active or SC resistors are placed at the ends of the capacitor as shown in Figure 6. Independent control signals for changing the resistance of the SC or active R, and & pass along both sides of the compartment. For chips with active resistors, the control signals are DC voltages that permit a certain range of adjustment. With SC resistors, the control signals are AC voltages in which the frequency determines the resistance. Presently, the R,n resistors in all of the compartments share the same control signals and the R, resistors share a different set of control signals. Therefore, all compartments have nominally the same R, and & resistances. The standard dendrite compartment was designed to abut with adjacent compartments and was pitch-matched to the on-chip virtual wire circuitry. This method makes the construction of artificial dendritic trees a relatively simple task: to make a branch, standard compartments are placed side by side until the desired branch length is reached. Branches are then connected via metal or poly wires to form trees. The spacing between compartments is the minimum distance between capacitors (2 pm). The compartments are aligned such that the inputs of one compartment connect to the outputs of the previous compartment.

3.4 A Simple Silicon Implementation. The artificial dendritic tree circuit and its on-chip virtual wiring was fabricated using a 2-pm CMOS double-poly n-well process on a 2 by 2 mm Mosis Tiny Chip (e.g., Mead 1989). The artificial somata and output multiplexer were left off the chips to permit experimentation with different soma circuits. Four artificial dendritic branches each having 15 excitatory and 15 inhibitory artificial synapses were implemented on the chip. The number of synapses was kept low in order to leave open silicon areas on the chip for other analog test circuits. Figure 7 shows the complete artificial dendrite chip layout. In the current implementation, the four branches are in-line, with a gap in between each branch, and centered on the die. The ends of each branch are taken out of the chip through package pins to allow experimentation with different tree structures. Multiple chips can be combined as well to produce tree structures with more branches, longer branches, or higher order branching. The remaining circuitry makes up the virtual wires. We are currently working on several new chip designs that we expect will reduce the silicon area needed for the on-chip virtual wires. We are also investigating the addition of shunting inhibition as well as local synaptic weight storage that could be used to modify the effective weights of existing connections.

658

John G. Elias

Figure 7 Chip layout of artificialdendrites fabricated using a MOSIS 2-pm double polysilicon standard CMOS process. The four artificial dendritic branches can be seen in the center of the die. The ends of each branch are connected to pads, which allows experimentation with different branching structures. Each branch has 30 synapses (15 excitatory and 15 inhibitory),which are uniformly spaced along the branch.

4 Simple Test Circuits

Three simple, artificial dendritic tree experiments, two of which do not make use of temporal aspects of dendritic trees, are described next. In each experiment, the output shown represents the measured branch node voltage from one of our circuits for a period of time after the tree received impulsive afferent sensory signals. The sensory signals for each experiment were generated by a computer and were applied to the artificial synapses through a parallel interface. The sensor elements were set to a logic one if the sensor field was above a fixed threshold voltage and a logic zero otherwise. In each experiment, a sequence of sensor data over time was presented to the tree and the resultant waveform was captured with an &bit digitizer. Although binary level, one-dimensional sensor data were used, each test circuit would produce similar results with multilevel, two-dimensional sensor data, albeit with different connection patterns and branching structure. Sensor elements that are a logic one at time, t, cause an impulse signal to be applied to the gate terminals of the artificial synapses that they connect to through the virtual wires. Therefore, active sensor elements (i.e., those that hold a logic 1 at time t ) cause their respective artificial synapses to turn on transiently. Inactive sensor elements (i.e., those that hold a logic zero at time t ) do not cause any artificial synapses to turn on. Figure 8 .illustrates how a two-branched artificial dendritic tree can detect asymmetric patterns in the sensor field. In this experiment, the


659

Figure 8: Input sensor with six elements connected to dual-branch artificial dendritic tree. The connection pattern shown classifiesinput patterns into symmetric and asymmetric classes. The sensor, with its data field, is shown at four different times, the last having a symmetric data pattern. Symmetric data fields result in a null output. Asymmetric sensor data fields produce either a positive or negative voltage trajectory. The output for the four sample times is shown at the right, which represents measured data from one of our circuits. Excitatory artificial synapses are top connections on each branch. Inhibitory artificial synapses are bottom connections.

sensor had 32 elements, but only six elements are shown here for simplicity. The same results would be obtained with virtually any size linear sensor array. The center of the sensor array represents the plane of symmetry, and the connections for a particular branch go only to sensor elements on one side of this plane. In this example, the top branch has one inhibitory and two excitatory connections. The bottom branch has one excitatory and two inhibitory connections. This connection pattern is not the only one that produces acceptable output. In general, there may exist a large number of connection patterns that produce acceptably good results. The sensor contents are shown at four different times, one of which contains a symmetric pattern. Not shown in the figure are the two trivial cases in which all elements are either logic one or logic zero. Both of these are symmetric and produce no output transient voltage. The case in which sensor elements are all ones activates all artificial synapses that are connected to these sensor elements, and the resultant signals sum approximately to zero at the branch point. Each connection from one side of the sensor plane of symmetry is mirrored with an opposite polarity connection on the other side that is equidistant from the branch point. When there is an asymmetric sensory pattern, as for sample numbers 0-2, there exists an imbalance between the activated artificial synapses, which results in a transient voltage at the branch point.

660

John G. Elias

Figure 9: Experimentsshowing the target-direction capabilities of dendrites (after Rall 1964). (a) Target (shaded sensor element)moves from right to left across sensor array. Resultant waveform is lower in peak voltage than in (b) where target moves from left to right. A simple thresholding device classifies target direction.

Figure 9 shows an experiment that follows an analysis of temporal processing in dendrites by Rall (1964). In this example, adjacent excitatory artificial synapses on one branch are connected to adjacent sensor elements. In this experiment, the sensor had 15 elements, but only eight elements are shown for simplicity. This simple connection pattern produces an output that is sensitive to the direction and speed of a moving target. Figure 9a shows the resultant branch point voltage for an eight segment time series in which a target moves across the sensor array from right to left. The target, in this case, is a logic one. The branch point voltage transients occur as the target moves across the sensor field, as can be seen in the plot of voltage vs. time. In Figure 9a, the artificial synapse nearest to the branch point is stimulated first and later followed by the stimulation of the next more distal synapse. The effect is that the resultant voltage transients arrive at the branch point well separated in time and therefore do not overlap significantly. In Figure 9b, the target moves from left to right. In this case, the most distal artificial synapse is stimulated first and followed later by stimulation of the more proximal synapses. The resultant voltage at the branch point is larger than it was for the case when the target moved in the opposite direction because the arrival times of the individual transients are more closely aligned. The classification of target direction is then completed by a simple comparator. Figure 10 shows a simplified circuit diagram that uses a single branch of artificial dendrite to provide a control signal for a maneuvering-target tracking application (Elias 1992~).The sensor, in this example, is a threshold device with seventeen elements, each of which outputs a one or a


661

Figure 10: Input sensor with 17 elements connected to a branch of an artificial dendritic tree that responds to the position of a target in the sensor array. The connection pattern shown produces a dynamic response that depends nonlinearly on how far off the target is from the center. When the target is centered, the output (at S) is a null. As the target moves off center, the resultant voltage increases rapidly with separation distance between target and center. The sensor, with its data field, is shown at nine different times. Each sample time shows the target (in this case a 0) going off center and the resultant output (offset for clarity) from the artificial dendritic branch. The top half of the sensor array connects to only excitatory artificial synapses. The bottom half connects only to inhibitory synapses. The resultant voltage transients at S were captured using an 8-bit digitizer.

zero. In Figure 10, the one-dimensional sensor pattern over time is that of a simple maneuvering target, which, in this example, is a logic zero on a background of logic ones. In general, the target could be of any shape as long as it was distinguishable from the background. When the target is on center, the output of the dendritic branch is approximately zero. This is because all excitatory and inhibitory artificial synapses are simultaneously conducting, thereby cancelling each other. Small variations in target position around the center produce relatively small output voltage transitions which can be used for low gain system control. If the target moves below center, as shown in Figure 10, the resultant voltage transients are positive. If, however, the target moves above center, the transients are negative. As the target moves farther off center, either above or below, the resultant branch output peak voltage rapidly increases. This occurs because more proximal artificial synapses turn on, which, in effect, shifts the system control to higher gain. The relative amplitudes of the branch output voltage transients as a function

662

John G. Elias

of the distance between target and sensor center can be arbitrarily set by moving connections of particular sensor elements to either more distal or more proximal artificial synapses (Elias 1992b).

5 Summary and Discussion

In our research program, we have adopted Mead’s methodology (Mead 1989) for implementing neuromorphic systems: (1) study the relevant biological implementation; (2) extract the important computational principles; (3) make optimum use of the inherent properties of electronic devices; and (4) implement, using standard silicon processing techniques. In the work reported here, we have studied the properties of both active and passive biological dendritic trees as well as the dynamic and static behavior of chemical synapses. We have extracted principles of computation exhibited by passive dendrites with chemical synapses and have translated these principles to a simple and scalable electronic form implemented in standard CMOS technology. Although our electronic models of chemical synapse and passive dendritic tree are, in many respects, extreme simplifications of biological structures, their dynamic electrical behavior appears to satisfactorily follow that of their biological paragons. The artificial dendritic tree structure is based on a current understanding of passive dendritic trees, which results in an extremely simple circuit implementation that is highly scalable. Artificial neurons with extensive dendritic trees have the capability to process signals that have both temporal and spatial significance. In our networks, weights are replaced with connections that, when combined with the sublinear behavior of electrically close synapses and the nearly linear behavior of widely separated synapses, provide a rich computational substrate for signal processing.

Acknowledgments The author wishes to thank Peter Warter for several useful suggestions on chip architecture, Hsu Hua Chu, Samer Meshreki, and Sheela Sastry for assisting with chip layout, design, and testing, and the reviewers for many useful comments.

References Allen, I? E., and Holberg, D. R. 1987. CMOSAnalogCircuit Design. Holt, Rinehart & Winston, New York.


663

Allen, P. E., and Sanchez-Sinencio, E. 1984. Switched Capacitor Circuits. Van Nostrand Reinhold, New York. Angstadt, J. D., and Calabrese, R. L. 1991. Calcium currents and graded synaptic transmission between heart intemeurons of the leech. J. Neurosci. 11(3),746759. Elias, J. G. 1992a. Spatiotemporal properties of artificial dendritic trees. Proc. Int. Joint Conf. Neural Networks, Baltimore 2, 19-26. Elias, J. G. 1992b. Genetic generation of connection patterns for a dynamic artificial neural network. IEEE Computer Society Press ProceedingsofCOGANN92, a workshop on combinations of genetic algorithms and neural networks. Elias, J. G. 1992c. Target tracking using impulsive analog circuits. In Applications of Artificial Neural Networks Ill, S. K. Rogers, ed. Proc. SPIE 1709, 338-350. Elias, J. G., Chu, H. H., and Meshreki, S. 1992. Silicon implementation of an artificial dendritic tree. Proc. Int. Joint Conf. Neural Networks, Baltimore 1, 154159. Elias, J. G., and Meshreki, S. 1993. Wide-range variable dynamics using switchedcapacitor neuromorphs. In preparation. Hebb, D. 0. 1949. The Organization of Behavior. Wiley, New York. Hounsgaard, J., and Midtgaard, J. 1988. Intrinsic determinants of firing pattern in Purkinje cells of the turtle cerebellum in vitro. J. Physiol. 402, 731-749. Koch, C., and Poggio, T. 1987. Biophysics of computation: Neurons, synapses and membranes. In Synaptic Function, G. M. Edelman, W. E. Gall, and W. M. Cowan, eds., Chap. 23. Wiley, New York. Koch, C., Poggio, T., and Torre, V. 1983. Nonlinear interactions in a dendritic tree: Localization, timing and role in information processing. Proc. Natl. Acad. Sci. U.S.A. 80, 2799-2802. Llinas, R., and Sugimori, M. 1980. Electrophysical properties of in vitro purkinje cell dendrites in mammalian cerebellar slices. J. Physiol. 305, 197-213. Mahowald, M. A. 1992. Evolving analog VLSI neurons. In Single Neuron Computation, T. McKenna, J. Davis, and S. F. Zornetzer, eds., Chap. 15. Academic Press, New York. Mahowald, M. A., and Douglas, R. 1991. A silicon neuron. Nature (London) 354, 515-518. Mead, C. 1989. Analog VLSI and Neural Systems. Addison-Wesley, Reading, MA. Narendra, K. S., and Parthasarathy, K. 1990. Identification and control of dynamical systems using neural networks. IEEE Transact. Neural Networks 1, 4-27. Rall, W. 1964. Theoretical significance of dendritic trees for neuronal inputoutput relations. In Neural Theory and Modeling, R. E Reiss, ed., p. 73. Stanford University Press, Stanford, CA. Rall, W. 1989. Cable theory for dendritic neurons. In Methods in Neuronal Modeling: From Synapses to Networks, Chap. 2. C. Koch and I. Segev, eds. MIT Press, Cambridge, MA. Shepherd, G. M., Brayton, R. K., Miller, J. F., Segev, I., Rinzel, J., and Rall, W. 1985. Signal enhancement in distal cortical dendrites by means of interactions between active dendritic spines. Proc. Natl. Acad. Sci. U.S.A. 82, 2192-2195.

664

JohnG. Elias

Shepherd, G. M., and Koch, C. 1990. Dendritic electrotonus and synaptic integration. In The Synaptic Organization of the Bruin, G . M. Shepherd, ed., appendix. Oxford University Press, New York. Shepherd, G. M., Woolf, T. B., and Carnevale, N. T. 1989. Comparisons between active properties of distal dendritic branches and spines: Implications for neuronal computations. 1. Cognit. Neurosci. 1, 273-286. Received 8 May 1992; accepted 11 November 1992.

This article has been cited by: 2. Christoph Rasche. 2007. Neuromorphic Excitable Maps for Visual Processing. IEEE Transactions on Neural Networks 18:2, 520-529. [CrossRef] 3. R. Jacob Vogelstein, Udayan Mallik, Joshua T. Vogelstein, Gert Cauwenberghs. 2007. Dynamically Reconfigurable Silicon Array of Spiking Neurons With Conductance-Based Synapses. IEEE Transactions on Neural Networks 18:1, 253-265. [CrossRef] 4. K.A. Boahen. 2004. A Burst-Mode Word-Serial Address-Event Link—II: Receiver Design. IEEE Transactions on Circuits and Systems I: Regular Papers 51:7, 1281-1291. [CrossRef] 5. K.A. Boahen. 2004. A Burst-Mode Word-Serial Address-Event Link—I: Transmitter Design. IEEE Transactions on Circuits and Systems I: Regular Papers 51:7, 1269-1280. [CrossRef] 6. C.M. Higgins, S.A. Shams. 2002. A biologically inspired modular VLSI system for visual measurement of self-motion. IEEE Sensors Journal 2:6, 508-528. [CrossRef] 7. C. Rasche, R.J. Douglas. 2001. Forward- and backpropagation in a silicon dendrite. IEEE Transactions on Neural Networks 12:2, 386-393. [CrossRef] 8. B.D. Brown, H.C. Card. 2001. Stochastic neural computation. I. Computational elements. IEEE Transactions on Computers 50:9, 891-905. [CrossRef] 9. K.A. Boahen. 2000. Point-to-point connectivity between neuromorphic chips using address events. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 47:5, 416-434. [CrossRef] 10. Timothy K. Horiuchi , Christof Koch . 1999. Analog VLSI-Based Modeling of the Primate Oculomotor SystemAnalog VLSI-Based Modeling of the Primate Oculomotor System. Neural Computation 11:1, 243-265. [Abstract] [PDF] [PDF Plus] 11. John G. Elias , David P. M. Northmore , Wayne Westerman . 1997. An Analog Memory Circuit for Spiking Silicon NeuronsAn Analog Memory Circuit for Spiking Silicon Neurons. Neural Computation 9:2, 419-440. [Abstract] [PDF] [PDF Plus] 12. David P. M. Northmore, John G. Elias. 1996. Spike Train Processing by a Silicon Neuromorph: The Role of Sublinear Summation in DendritesSpike Train Processing by a Silicon Neuromorph: The Role of Sublinear Summation in Dendrites. Neural Computation 8:6, 1245-1265. [Abstract] [PDF] [PDF Plus]

Communicated by A. B. Bonds

Patterns of Local Connectivity in the Neocortex Andrew Nicoll Department of Physiology, School of Medical Sciences, University Walk, Bristol BS8 ZTD,UK

Colin Blakemore University Laboratory of Physiology, Parks Road, Oxford OX2 3PT, UK

Dual intracellular recording of nearby pairs of pyramidal cells in slices of rat visual cortex has shown that there are significant differences in functional connectivity between the superficial and deep layers (Mason et al. 1991; Nicoll and Blakemore 1993). For pairs of cells no farther than 300 pm apart, synaptic connections between layer 2/3 pyramidal neurons were individually weaker (median peak amplitude, A, of single-fiber excitatory postsynaptic potentials, EPSPs, = 0.4 mV) but more frequent (connection probability, p = 0.087) than those between layer 5 pyramidal neurons (mean A = 0.8 mV, p < 0.015). Taken in combination with plausible estimates of the density of pyramidal cells, the total numbers of synapses on them and the number of synapses formed on their intracortical axons, the present analysis of the above data suggests that roughly 70% of the excitatory synapses on any layer 2/3 pyramid, but fewer than 1% of those on a layer 5 pyramidal neuron, are derived from neighboring pyramidal neurons in its near vicinity. Even assuming very extreme values for some parameters, chosen to erode this difference, the calculated proportion of ”local synapses” for layer 5 pyramids was always markedly lower than for layer 2/3 pyramidal neurons. These results imply that local excitatory connections are much more likely to provide significant ”intracortical amplification” of afferent signals in layer 2/3 than in layer 5 of rat visual cortex. 1 Introduction

Pyramidal neurons are the major excitatory cells of the neocortex and almost exclusively constitute the output of the cortex (Peters 198%). Their axons make long-range projections, either to other cortical regions or to subcortical structures, but they also have collaterals forming extensive local arborizations within the same cortical area (e.g., Gilbert et al. 1990; Ojima et al. 1991; Kisvdrday and Eysel1992). A typical pyramidal neuron may possess in the order of 10,000 morphologically identified excitatory synapses (Larkman 1991)and yet, even in visual cortex layer 4, the major Neural Computation 5, 665-680 (1993) @ 1993 Massachusetts Institute of Technology

666

Andrew Nicoll and Colin Blakemore

target of thalamic afferents, only 20-30% of all synapses involve thalamocortical axons (Peters 1987b). Since nonpyramidal, excitatory neurons (spiny stellate cells) are rare, if not absent, in rat primary visual cortex (Peters 1987a,b), it seems likely that the majority of synapses on any particular pyramidal neuron derive from other pyramidal neurons. Morphological studies have indeed demonstrated that the commonest target of pyramidal cell axons are other pyramidal cells (Kisvdrday et al. 1986; Gabbott et al. 1987; Elhanany and White 1990). In addition, cross-correlation techniques and current source density analysis have provided evidence for the existence of “horizontal,” intrinsic connections in neocortex (e.g., Engel et al. 1990; Langdon and Sur 1990, respectively). However, if we are to model the computations performed by cortical circuitry, it is essential to have quantitative information about the origin and effectiveness of the synaptic inputs to individual cells. In previous studies, two methods have been employed to estimate the number of synapses needed to elicit an action potential in a postsynaptic cell (Andersen et al. 1990). One approach is to divide the difference between resting membrane potential and spike threshold (of a certain cell class) by the amplitude of the estimated size of a single, “quantal” excitatory postsynaptic potential (EFSP)(Martin 1984; Sayer et al. 1989; Andersen et al. 1990). However, effective spike threshold is not an invariant parameter because resting membrane potential depends on the prevailing level of tonic afferent activity, on the particular preparation used and on the method of recording. A second approach is to estimate how many presynaptic axons in a surgically isolated %rand of cortex that must be activated to discharge pyramidal cells in the target area when the strand is stimulated (Andersen et al. 1980, 1990; Sayer et al. 1989). This method suffers from the disadvantage that it is not possible to know the number of functional fibers in such a strand of tissue. The most direct way to look at synaptic convergence is to make simultaneous recordings between pairs of identified cortical cells. To improve on the above calculations, in this analysis, data from dual intracellular recording from pyramidal neurons in rat visual cortical slices have been combined with experimentally derived values from other aspects of synaptic distribution to estimate parameters of functional connectivity. We were interested to know, among other things, the proportion of a cell’s connections that is “local” as opposed to “long-range,” that is, what fraction of the total input to a pyramid might be provided by immediately neighboring pyramidal cells, information of special relevance to the visual cortex. 2 Determination of Connection Strength, Probability, and Distance

The strength and probability of occurrence of single-fiber synaptic connections between rat visual cortical pyramidal neurons were obtained

Local Connectivity in the Neocortex

667

from dual intracellular recordings in vitro (Mason et al. 1991; Nicoll and Blakemore 1993). Briefly, a pyramidal neuron was impaled intracellularly and then a second microelectrode, placed within an annular region (usually within a radius of 500 pm) centered on the first, was advanced until a second cell was obtained, usually within the same cortical layer. Single action potentials were elicited in the second cell by injection of depolarizing current pulses, and spike-triggered averaging was used to reveal any EPSP in the first cell. The first neuron was then stimulated to see if there was a connection in the other direction. All cells recorded had electrophysiological characteristics typical of pyramidal cells (McCormick et al. 1985) and whose morphological identity was confirmed for many cases by intracellular staining (see Mason et al. 1991). In layer 2/3, out of a total of 549 cell pairs tested, 48 were synaptically connected, equivalent to a connection probability, p, of 0.087. The median peak amplitude, A, of layer 2/3 single-fiber EPSPs was 0.4 mV (Mason et al. 1991). Within layer 5 however, only four connections (mean A = 0.8 mV) were found out of a possible total of 270, equivalent to p = 0.015 (Nicoll and Blakemore 1993). All EPSPs were monosynaptic (see Mason et al. 1991). Spike-triggered averaging of 50 to 100 sweeps at high gain was initially used to determine whether a cell pair was connected, and several hundred to several thousand sweeps were usually averaged when an EPSP was found. However, it should be noted that EPSPs smaller than about 0.03 mV would not have been detected with our methods. This lower connection probability and greater size of deep layer single-fiber EPSPs have also been observed by Thomson et al. (1992,1993). The separation of the microelectrode tips, d (and, therefore, of the two cell bodies), was calculated by trigonometry using the angles of microelectrode penetration and the depths of the cells within the slices. For the 48 connections between layer 2/3 cells, d ranged from 50 to 350 pm (mean of 150 pm). All connections except one (where d = 350 pm) were recorded between cells less than 300 pm apart. The distance between the cell bodies of the four connected pairs within layer 5 ranged from 70 to 150 pm (mean of 90 pm). The variation of p with d was not rigorously investigated, because that would have required that d values be known for all the unconnected pairs as well as those in which connections were found, data that were not recorded in the original experiments. There was a clear impression, however, that the chance of encountering a coupled pair increased as the microelectrodes were brought together. For layer 2/3 dual impalements, only one connection was found for d = 300500 pm. At least for the smaller sample of layer 5 dual impalements, no connections were found for d = 150-300 pm. The substantial difference in connection probabilities between the two layers therefore seems unlikely to be due to the difference in the distance range sampled. No serious attempt was made to look at connections over the millimeter range, although long-range connections (not necessarily monosynaptic) have been demonstrated in the cat (Ts’o et al. 1986). For the larger samN

N

N

668


ple of layer 2/3 cell pairs, no correlation was found between EPSP onset latency and microelectrode separation (Mason, Nicoll and Stratford, unpublished results). 3 Analysis of Connectivity Patterns

The results of the analysis, incorporating experimentally derived data from dual intracellular impalement and based on the most plausible assumptions, are summarized in Table 1. 3.1 Layer 2/3 Local Connections. In layer 2/3, monosynaptic connections were found between pairs of pyramids no farther than 300 pm apart. Let us imagine one individual pyramid at the center of a “local sphere” that could be connected in all directions to a number of other cells no farther than that distance away (Fig. 1). How many other pyramidal cells are there in a local sphere of radius 300 pm to which one individual cell at its center could be connected? Assuming a uniform and constant density of cells within the local sphere and knowing the density of pyramidal neurons in layer 2/3 of rat visual cortex to be 65,000 cells mm-3 (calculated from Gabbott and Stewart 1987; Peters 1985,1987a1, the number of pyramidal cells contained in a local sphere is 7,345 (Table 1). The center-sphere cell will be connected only to a proportion of those cells though. Therefore, assuming that the layer 2/3 connection probability of 0.087 is uniform within the local sphere, one pyramidal cell at the center of the local sphere could be connected to around 639 other pyramids in this near vicinity. Let us take the case of the pyramidal neuron at the center of the local sphere being synaptically connected to another pyramid within the sphere. How many anatomical synapses (SA) mediate one pyramidpyramid connection? To our knowledge, there is no empirical value for SA in layer 2/3 of the rat visual cortex, but evidence from other species suggests the figure may be small (Kisvdrday et al. 1986; Gabbott et al. 1987). Let us assume, to an order of magnitude, that there are only 10 anatomical synapses in a typical inter-pyramid connection (Martin 19841, an estimate that receives a measure of support from electrophysiological studies (see below). If 639 ”local” pyramidal neurons provide input to one neuron at the center of our sphere of connection, then 6,390 anatomical synapses on that cell would come from those local pyramids (Table 1). However, a typical layer 2/3 pyramidal neuron possesses a total of -9,371 excitatory synapses (Larkman 19911, as estimated from counting dendritic spines. If we assume that the most distal 5%of these synapses (in layer 1) are unlikely to be contacted in “local” interactions then there are a possible maximum of 8,902 excitatory synapses available on an individual pyramidal cell. Therefore, 6,390/ 8,902 synapses


669

Table 1: Patterns of Connectivity in Superficial and Deep Neocortical Layers, Considered in Terms of “Local Spheres” of Pyramidal Cellsu

1. Total number of neurons’ (mmP3) Number of pyramids’ (mm-j) “Connection distance,”* d (pm) Connection probability: p Volume of tissue contained in a “local sphere” of radius d (pm3) [ 4 / 3 ~ ( d ) ~ 1 Number of pyramids, N, within local sphere Number of pyramids, n, within local sphere conneded to one individual pyramid [p x Nl 2. Mean number of ”anatomical synapses,” SA, between two connected pyramids3 Total number of synapses, SL,mediating local pyramidal connections within the sphere [n X SA] Number of excitatory synapses, on one pyramid4 Proportion of synapses on one pyramid mediating local connections [SL/%I

s,

3. EPSP peak amplitude? A (mV) Quanta1 amplitude? a (mV) Number of quanta per trial, q [A/a] Presynaptic release probability: RP Number of ”physiological synapses,” Sp in one connection [q/RP; 1 release site = 1 synapse] 4. Total number of afferent fibers received by an individual pyramid [S.r/SPl [%‘/sAI

Layer 2/3

Layer 5

80 x 103 65 103 300 0.087

45 x 103 36 x 103 150 0.015

1.13 x lo8 7,345

0.14 x lo8 504

639

8

10

10

6,390

80

8,902

12,483

71.8%

0.6%

0.4 0.1 4 0.5

0.8 0.1 8 0.5

8

16

1,113 890

780 1,248

‘See Figure 1. The parameters were determined on the basis of the most plausible estimates available. Calculated on this basis, although the total number of connections on an individual pyramidal neuron may be similar in both layers (step 41, layer 2/3 pyramids have many more local connections than pyramids in layer 5 (step 2). Values of d and p were obtained experimentally (Nicoll and Blakemore 1993;Mason etal. 1991). Other figures were obtained from published literature or calculated as shown. Cited literature: ‘Gabbott and Stewart (1987),Peters (198.51,and Peters (1987a); ’Mason et al. (1991),Nicoll and Blakemore (1993);3Martin (1984);see also Kisvhrday et al. (1986)and Gabbott et al. 1987; 4Larkman (1991);5Sayer et al. (1989),Jack et al. (1990), Kom and Faber (1991),and Larkman et al. (1991).See text for details.

670


Figure 1: The geometry of “local” neocortical connections. Imagine that an individual pyramidal neuron (large triangle) receives synaptic input from the local axon collaterals of some of the other pyramidal cells within a “local sphere” (unhatched region). The radius of the sphere is the separation, d, of connected cell pairs, determined from dual intracellular recording (as in Table l), viz. 300 pm for layer 2/3 and 150 IJm for layer 5. The probability of the center cell being connected to any other in the sphere, determined experimentally, was assumed to be constant within the local sphere.

or 71.8% of all the synapses on a layer 2/3 pyramidal neuron come from other

pyramidal neurons in its near vicinity. The number of synapses involved in a synaptic connection may also be estimated from physiological evidence. The median peak amplitude of a single-fiber or unitary EPSP of a layer 2/3 pyramid was found to be 0.4 mV (Mason et al. 1991). Let us assume a quanta1 amplitude of, say, 0.1 mV (e.g., Sayer et al. 1989; Larkman et al. 1991) and that the


671

presynaptic release probability is, on average, 0.5 (Jack et al. 1990), as in hippocampal synapses. If one synaptic release site is equivalent to one physiological synapse (Korn and Faber 19911, then a 0.4-mV EPSP would be mediated by 8 “functional” synapses, a number similar to the 10 anatomical synapses constituting a typical inter-pyramidal connection suggested by Martin (1984). 3.2 Layer 5 Local Connections. Taking into account the lower pyramidal cell density in layer 5 (36,000 ~ ~ u n Gabbott - ~ : and Stewart 19871, the lower connection probability for layer 5 connections found experimentally (0.015)and using a “local sphere” of radius 150 pm calculated using the same assumptions as in layer 2/3, the number of pyramidal cells in the near vicinity possibly connected to one individual pyramid (at the center of the local sphere) in layer 5 would be only 8 cells (Table 1). For this calculation, we decided to assume again that a typical inter-pyramid connection is made by 10 anatomical synapses, SA,as for layer 2/3. This would make the total number of local inputs on to the center-sphere pyramidal neuron is 80 synapses. The mean synapse-toneuron ratio of both pyramidal cell classes in layer 5 is 13,870 (calculated from Larkman 1991). Because the pyramidal neurons in layer 5 are geometrically very long, let us assume that 90% of those excitatory synapses can mediate “local” connections (12,483; Larkman 1991). That being the case, the proportion of excitatory synapses on a layer 5 pyramid that derives from “local” sources would be only 0.6%, much lower than for layer 2/3 cells. We found mean EPSP peak amplitude to be about 0.8 mV for layer 5 connections, twice that in layer 2/3 (Nicoll and Blakemore 1993). If the same values of quanta1 amplitude and release probability are assumed as for layer 2/3, the number of “physiological synapses,” Sp, involved in a single-fiber EPSP of that amplitude would be 16, calculated in the same way as for layer 2/3. This is double the figure for layer 2/3, but still the same order of magnitude as SA.If a typical pyramidal neuron in layer 2/3 has 8,902 anatomical synapses and each connection it makes is mediated via 8-10 physiological or anatomical synapses, then one might suppose that one layer 2/3 cell receives between 890 and 1,113 afferent fibers (Table 1). A similar calculation for layer 5 reveals that a single pyramid there may receive 780-1,248 afferents. In other words, each cell class receives roughly the same number (1,000) of afferent inputs, although the amount of local input is much greater in layer 2/3. Although both figures have been used in Table 1, we have assumed a rough equivalence between SAand Sp (see Korn and Faber 19911, an assumption also employed below. 4 “Worse Case” Scenario

The analysis illustrated in Table 1 suggested that about 70% of the excitatory synapses on a typical layer 2/3 pyramidal neuron originate from


672

Table 2: Connectivity Parameters Selected so as to Minimize the Differences in Local Connectivitybetween Layers 213 and 5 (”Worse Case Scenario”)“

1. “Connection distance,” d (pm)

Connection probability, p Number of pyramids, N,within “local sphere” Number of pyramids, n, within “local sphere” connected to one individual pyramid [p x N]

Layer 213

Layer 5

300 0.087 7,345

300 0.015 4,068

639

4,068

2

16

1,278

976

8,902

12,483

14.4%

7.8%

2. Mean number of “anatomical”or “physiological” synapses, SA or Sp, between two connected

pyramids Total number of synapses, SL,mediating local pyramidal connections within the sphere [n x SA] Number of excitatory synapses, ST, on one Pyramid Proportion of synapses on one pyramid mediating local connections [SL/%]

’The steps and assumptions from the literature are similar to those in Table 1but the values of d and SA or SP have been altered (see text). Even with this deliberate biasing of the analysis with very extreme parameters, the proportion of “local synapses” is still greater for layer 2/3 and layer 5, and the proportion for layer 5 still does approach that found for layer 2/3 in Table 1.

very nearby cells, while less than 1%of those on a layer 5 pyramid derive from other local pyramids. To test this conclusion, we performed a “worse case scenario” calculation by substituting very extreme values, deliberately chosen to abolish the lamina differences in local connectivity (Table 2). Employing the same connection probabilities, we altered two parameters: number of ”anatomical” or “physiological” synapses in a pyramid-pyramid connection, SA or Sp, and connection distance, d. For the analysis in Table 1, we made SA 10 for both layers 2/3 and 5. However, there is evidence from the superficial layers of cat visual cortex that SA may be even smaller (KisvBrday et al. 1986; Gabbott et al. 1987). In Table 2, we therefore made SA for layer 2/3 equal to 2, thereby substantially reducing the fraction of local synaptic inputs. For layer 5, we wished to make SA or SP as large as possible to upwardly bias the final result. We therefore used the estimate for S p of 16 in layer 5 (Table 1) although this number is somewhat higher than present evidence, at least for SA, suggests is feasible (Martin 19841, assuming SA and SP can be interchanged. In a further attempt to deliberately bias the result, d for layer 5 was increased. We based our estimate of d on empirical evidence from the dual impalement experiments, taking it to be the maximum


673

distance at which cell pairs were actually found to be coupled, within the approximate 500 pm range explored. For layer 2/3, the value of d derived in this manner, 300 pm, corresponds quite well to the overall distribution of spines on the basal dendritic tree plus the probable local density of boutons on axon collaterals (Larkman et al. 1988; Mason et al. 1991). For layer 5, we found interactions only within the surprisingly short distance of 150 pm. Perhaps that is a reflection of different axonal morphology (e.g., Chagnac-Amitai et al. 1990) or, on the other hand, it is possible that the small number of connected pairs recorded in layer 5 by Nicoll and Blakemore (1993) provided an unrepresentative estimate of d. In the “worse case scenario” of Table 2, we therefore substituted d = 300 pm for layer 5, as for layer 2/3, although strictly unjustifiable on the basis of the data to hand (see below). Even with these changes, the proportion of local synapses calculated for a layer 5 cell was still only 7.8% but the proportion of local inputs onto layer 2/3 pyramids was substantially reduced. 5 Assumptions and Parameters

Unfortunately, this analysis is hampered by lack of directly applicable empirical data, so we have had to make reasonable assumptions for some of the parameters. The ”worse case scenario” calculation shows that although the results can be considerably altered by one’s choice of particular values, the overall approach and conclusion seem robust. 5.1 Connection Distance and Probability. Ideally, it would be desirable to derive a quantitative relationship between connection probability and cell separation. Although that information was not available for the present work, it would, in principle, be possible to describe p in terms of d with results from dual impalement experiments, although this would be laborious because d would have to be known for all cell pairs that were unconnected as well as connected. Also, the frequency of monosynaptic connections over larger distances could be low. For layer 2/3, we based the size of the “local sphere” (Fig. 1)on a radius of 300 pm, rather than the mean value for d. This was primarily because the value of p must be an overall estimate that applies to all the connections with cell separations no greater than that distance: the connection probability is unknown for subsamples with greater or smaller d. Similar assumptions were applied to layer 5 in the Table 1analysis, where no connections were found beyond d = 150 pm. Hence, it was implicit in our calculations that p was constant throughout the local spheres used. It was necessary to assume that the connection distances sampled corresponded to the actual cell separations over which monosynaptic connections are found at the observed probabilities. Local excitatory interactions between pyramidal neurons are primarily mediated by their

674


basal dendrites and recurrent axon collaterals, which both extend over similar distances as used for the local spheres here. For pyramidal cells in both layers 2/3 and 5 of rat visual cortex, a minority of the dendritic spines are located at a path length of greater than about 150 pm from the cell body (Larkman 1991). Equivalent data are not available for presynaptic bouton distribution. However, bearing in mind the considerable overlap of basal dendrites and recurrent axon collaterals in pyramidal cells at least in layer 2/3 (Larkman et al. 1988; Mason et al. 19911, let us say, as a very rough approximation, that most presynaptic boutons are also located proximal to that distance as well. Hence, at least for layer 2/3, d = 300 pm tentatively represents the end of a range over which two pyramidal neurons may be connected through their local neurites. However, any dendritic similarities between layer 2/3 and 5 cells do not explain differences in connection probabilities. Perhaps they are a reflection of differing axonal rather than dendritic morphology. Layer 5 pyramids, especially burst-firing ones, may have more horizontal axon collaterals (Chagnac-Amitai et al. 1990), possibly with lower numbers of presynaptic boutons (Qima et al. 1991; Nicoll and Blakemore 1991). Although it may be difficult to apply results derived from a number of cells stained by gross extracellular dye injection to the present problem, Burkhalter (1989) noted, in rat visual cortex, that axons of layer 5 cells display clustered distributions whereas those of layer 2/3 cells do not, although there seems to be stronger evidence for clustered axonal projections in the superficial layers of the cat’s visual cortex (Kisvdrday et al. 1986; Kisvlrday and Eyles 1992). Still, even when d is artificially increased to 300 pm for layer 5 (Table 21, the proportion of local synapses does not approach that found in layer 2/3, mainly as a consequence of the low p found for layer 5. For further discussion of cortical connectivity see Braitenberg and Schiiz (1991). 5.2 Synapses between Pyramidal Cells. The number chosen for SA or S p can have considerable influence on the analysis. We initially assumed, to within an order of magnitude, there were 10 anatomical synapses (SA) between any two connected pyramids (Martin 1984; Step 2, Table 11, which compares well to the estimates of Sp. There is no direct empirical value of SA for our very specific situation in the rat, but the figure of 10 is reasonable on the basis of general morphological and electrophysiological considerations (Martin 1984). Gabbott et al. (1987) studied the synaptic contacts between two ”neighboring” (i.e., sharing a small patch of overlapping axon collaterals some distance from the somas) pyramidal cells in layer 5 of cat visual cortex under the electron microscope. Although probably underestimated, they found the number of anatomical synapses between the two cells to be only 4; Kisvdrday et al. (1986) suggested that this figure could be even lower for layer 3 pyramids in cat striate cortex. In Table 2, we explicitly assumed a direct correspondence between structural and functional synapses so that we


675

could use a very high value in one stage of the calculation. It is unknown whether the values of SA and S p are different in layers 2/3 and 5. 5.3 Synapse-to-Neuron Ratio. Some of the most distal synapses of a pyramidal neuron, especially on the long cells of layer 5, may be unavailable for intralaminal, "local" connections, and a small adjustment was made in the calculation to take account of this. We found, however, that within reasonable limits, the value set for the total number of synapses even on a layer 5 pyramid (s:Table 1) does not overwhelmingly affect the proportion of local synapses (SL)to the total number (SL/%) calculated. In layer 5, when SA = 10, SL comes out as 80. Even if there were only, say, 8,000 excitatory synapses within layer 5 on one layer 5 cell (Larkman 1991), this still represents a small (1%)SL/!?q proportion. For layer 2/3, the number of local pyramids, and hence SL, is much greater than in layer 5. If the number of pyramids in close vicinity to one individual pyramid in layer 2/3 is 639 (Table 11, and the number of anatomical synapses in an inter-pyramid connection, SA, were 2, then SL would be 1,278. If we say there are 5,600 excitatory synapses distributed within layer 2/3 (Larkman 1991), this represents an SL/& proportion of 22.8% on any one layer 2/3 pyramidal cell, still much greater than for layer 5.

5.4 Single-Fiber and Single-Quantum EPSPs. We used estimates of quanta1 amplitudes, a, obtained from studies in the hippocampus for our arguments here, as no published data were available for rat neocortex. The EPSP amplitudes used were based on results from intracellular recording, which may differ from estimates obtained using whole-cell patch pipettes, where cell input resistances may be higher (Staley et al. 1992). With our estimation of physiological synapses (SP),we achieved a reasonable agreement between S A and SP and were able to very tentatively concluded that the number of connections received by a cell were similar for layers 2/3 and 5. We applied a value of a = 0.1 mV for connections in both layers 2/3 and 5 (Table 1) although there is no reason one way or the other to suppose they are the same or different. The reported value of a ranges from about 0.1 to 0.4 mV (see Mason et al. 1991) but an estimate at the lower end of that range was employed because that was considered more reasonable for neocortex, based on current indications (Mason, Nicoll, and Stratford, unpublished observations). For the sake of argument, if a is set at 0.4 mV but the same E S P amplitudes are used as found experimentally with sharp electrodes (0.4 mV, layer 2/3 and 0.8 mV, layer 51, then the number of physiological synapses mediating one single-fiber connection becomes 2 for layer 2/3 (as opposed to 8 for a = 0.1 mV) and 4 for layer 5 (16 for a = 0.1 mV; Table 1). This is in closer agreement to the possible values of anatomical synapses suggested

676


by Kisvirday et al. (1986) and Gabbott et al. (1987) but, of course, a range of answers is possible, depending on the assumptions made. 6 Discussion

It has been suggested that there is a fundamental neuronal circuit underlying cortical function (e.g., Szentdgothai 1978) and that functional differences between cortical areas mainly result from different afferent and efferent connections of the common circuit (Peters and Sethares 1991). The visual cortex contains a precise “map” of the visual field and individual neurons, especially in the central field representation, generally have receptive fields that cover only a tiny fraction of the entire field, even in the rat, which lacks a very pronounced central specialization in its retina (see Sefton and Dreher 1985). This demands that the major suprathreshold excitatory input to each cortical cell must originate ultimately from the restricted region of the retina corresponding to the classical receptive field. This in turn implies that very local intrinsic connections, deriving from cells with overlapping receptive fields, are much more important than any distant “horizontal” connections in constructing the receptive field itself. Douglas and colleagues have proposed a “canonical microcircuit” for the neocortex to account for the intracellular responses of cat striate cortical neurons to stimulation of thalamic afferents (Douglas et al. 1989; Douglas and Martin 1991). No one cell or class of cells receives sufficient synaptic drive directly from the thalamus to make it fire at the high rates observed following presentation of an optimal visual stimulus. They suggest that excitatory drive originating from the thalamus is augmented by successive stages of intracortical recurrent excitation. They term this process “intracorticalamplification” and propose that it is mediated primarily by local excitatory connections between pyramidal cells. The optimal parameters for the canonical model involve pyramidal cells in the superficial and deep layer groups securing half their intracortical excitatory conductance from their own respective populations (Douglas and Martin 1991). Our results support this model for pyramids in layer 2/3 of the rat, which appear to receive around 70% of their synapses from other local pyramids, but not for cells in layer 5. The latter (at least in the rat) are likely to be dominated by other inputs. If only a minority of the excitatory synapses on layer 5 pyramidal neurons derive from either thalamic axons or local pyramids, where do the rest come from? Presumably their origins include (1) excitatory cells in other layers, especially pyramidal cells in layer 2/3, which have a prominent projection to layer 5 in rats (Burkhalter 19891, (2) long-range projections from other pyramids elsewhere in primary visual cortex, (3) possibly cortico-corticalbackprojections from other visual cortical areas (see Sefton and Dreher 1985). As for the cat (Ts’o etal. 1986; Gilbert etaf.1990),


677

the input from distant regions of cortex probably provides information from outside the classical receptive field, which is therefore unlikely to act as an amplifier during localized visual stimulation. Pyramidal neurons in layer 2/3 and nonbursting pyramidal layer 5 neurons project to the contralateral hemisphere through the corpus callosum (Hallman et al. 1988; Hubener and Bolz 19881, but burst-firing cells in layer 5 project to a variety of subcortical targets such as the superior colliculus (Schofield et al. 1987; Hiibener and Bolz 19881, and/or the pons (Hallman et al. 1988). Chagnac-Amitai and Connors (1989a,b) suggested that layer 5 burst-firing neurons form a subnetwork of strongly, yet sparsely, connected neurons, a notion that the results of dual impalement studies would support. However, the exact visual role that local inter-pyramidal synaptic connections play in layer 5 remains unclear, especially as the amount of depolarization they produce is small relative to neuronal spike threshold. Unless the synapses of thalamocortical axons are exceptionally numerous or remarkably effective in depolarizing the cells, the "enabling depolarization," essential for the activation of layer 5 circuitry, must come from elsewhere. The most likely candidate for this enabling signal is the input from layer 2/3, which could then be amplified by both the thalamic and the local pyramidal inputs. 7 Conclusions

The analysis in this paper applies specifically to the visual cortex of the rat. However, to the extent that the neocortex might have common principles of circuitry, conserved across species, the results might be of more general relevance. As more precise values for Table 1 become available, parameters of connectivity can easily be revised and extended so that the numbers of synapses and neurons in whole neocortical columns can eventually be derived. Acknowledgments This work was supported by the Medical Research Council, The Wellcome Trust, and the Human Frontier Science Program. We would like to thank Dr. A. Larkman, Dr. P.Bush, Dr.T. Sejnowski, Dr. B. Connors, and Mr. A. Strassberg for helpful discussion regarding this work. References Andersen, P., Silfvenius, H., Sundberg, S. H., and Sveen, 0. 1980. A comparison of distal and proximal dendritic synapses on CAI pyramids in guinea-pig hippocampal slices in vitro. J. Physiol. (Lond.) 307, 273-299.

678


Andersen, P., Raastad, M., and Storm, J. F. 1990. Excitatory synaptic integration in hippocampal pyramids and dentate granule cells. Cold Spring Harbor Syrnp. Quant. Biol. LV, 81-86. Braitenberg, V., and Schiiz, A. 1991. AnatornyoftheCortex: Statisticsand Geometry. Springer-Verlag, Berlin. Burkhalter, A. 1989. Intrinsic connections of rat primary visual cortex: Lamina organization of axonal projections. J. Comp. Neurol. 279, 171-186. Chagnac-Amitai, Y., and Connors, B. W. 1989a. Horizontal spread of synchronized activity in neocortex and its control by GABA-mediated inhibition. J. Neurophysiol. 61,747-758. Chagnac-Amitai, Y., and Connors, B. W. 1989b. Synchronized excitation and inhibition driven by intrinsically bursting neurons in neocortex. J. Neurophysiol. 62, 1149-1162. Chagnac-Amitai, Y., Luhman, H. J., and Prince, D. A. 1990. Burst generating and regular spiking layer 5 pyramidal neurons of rat neocortex have different morphological features. J. Comp. Neurol. 296, 598-613. Douglas, R. J., and Martin, K. A. C. 1991. A functional microcircuit for cat visual cortex. J. Physiol. (Land.) 440, 735-769. Douglas, R. J., Martin, K. A. C., and Whitteridge, D. 1989. A canonical microcircuit for neocortex. Neural Comp. 1, 480-488. Elhanany, E., and White, E. 1990. Intrinsic circuitry: Synapses involving the local axon collaterals of corticocortical projection neurons in the mouse primary somatosensory cortex. J. Cornp. Neurol. 291,43-54. Engel, A. K., Konig, P., Gray, C. M., and Singer, W. 1990. Stimulus-dependent neuronal oscillations in cat visual cortex: Inter-columnar interaction as determined by cross-correlation analysis. Eur. J. Neurosci. 2, 58f3-606. Gabbott, P. L. A., and Stewart, M. G. 1987. Distribution of neurons and glia in the visual cortex (area 17) of the adult albino rat: A quantitative description. Neuroscience 21, 833-845. Gabbott, P. L. A., Martin, K. A. C., and Whitteridge, D. 1987. Connections between pyramidal neurons in layer 5 of cat visual cortex (area 17). J. Comp. Neurol. 239, 364-381. Gilbert, C. D., Hirsch, J. A., and Wiesel, T. N. 1990. Lateral interactions in visual cortex. Cold Spring Harbor Symp. Quant. Biol. LV,663-677. Hallman, L. E., Schofield, 8. R., and Lin, C.4. 1988. Dendritic morphology and axon collaterals of corticotectal, corticopontine, and callosal neurons in layer V of primary visual cortex of the hooded rat. J. Comp. Neurol. 272, 149-1 60. Hiibener, M., and Bolz, J. 1988. Morphology of identified projection neurons in layer 5 of rat visual cortex. Neurosci. Lett. 94, 76-81. Jack, J. J. B., Kullmann, D. M., Larkman, A. U., Major, G., and Stratford, K. J. 1990. Quanta1analysis of excitatory synaptic mechanisms in the mammalian central nervous system. Cold Spring Harbor Syrnp. Quant. Biol. LV,57-67. Kisvbrday, Z. F., and Eysel, U. T. 1992. Cellular organization of reciprocal patchy networks in layer III of cat visual cortex (area 17). Neuroscience 46,275-286. Kisvbrday, Z. F., Martin, K. A. C., Freund, T. F., Magl6czky Zs., Whitteridge,


679

D., and Somogyi, P. 1986. Synaptic targets of HRP-filled layer I11 pyramidal cells in the cat striate cortex. Exp. Brain Res. 64, 541-552. Korn, H., and Faber, D. S. 1991. Quantal analysis and synaptic efficacy in the CNS. Trends Neurosci. 14, 439-445. Langdon, R. B., and Sur, M. 1990. Components of field potentials evoked by white matter stimulation in isolated slices of primary visual cortex: Spatial distributions and synaptic order. J. Neurophysiol. 64, 1484-1501. Larkman, A. U. 1991. Dendritic morphology of pyramidal neurones of the visual cortex of the rat: 111. Spine distributions. J. Cornp. Neurol. 306, 332-343. Larkman, A. U., Mason, A., and Blakemore, C. 1988. The in vitro slice preparation for combined morphological and electrophysiological studies of rat visual cortex. Neurosci. Res. 6, 1-19. Larkman, A., Stratford, K., and Jack, J. 1991. Quantal analysis of excitatory synaptic action and depression in hippocampal slices. Nature (London) 350, 344-347. Martin, K. A. C. 1984. Neuronal circuits in cat striate cortex. In Cerebral Cortex, Vol. 2, E. G. Jones and A. Peters, eds., pp. 241-284. Plenum Press, New York. Mason, A., Nicoll, A., and Stratford, K. 1991. Synaptic transmission between individual pyramidal neurons of the rat visual cortex in vitro. J. Neurosci. 11, 72-84. McCormick, D. A., Connors, 8. W., Lighthall, J. W., and Prince, D. A. 1985. Comparative electrophysiology of pyramidal and sparsely stellate neurons of neocortex. J. Neurophysiol. 54, 782-806. Nicoll, A., and Blakemore, C. 1991. Differencesin inter-bouton distance between intracortical axons of different classes of pyramidal neurone in rat visual cortex. J. Anat. 179, 209-210. Nicoll, A., and Blakemore, C. 1993. Single-fibre EPSPs in layer 5 of rat visual cortex in vitro. NeuroReport 4, 167-170. Ojima, H., Honda, C. N., and Jones, E. G. 1991. Patterns of axon collateralization of identified supragranular pyramidal neurons in the cat auditory cortex. Cerebral Cortex 1, 80-94. Peters, A. 1985. The visual cortex of the rat. In Cerebral Cortex, Vol. 3, A. Peters and E. G. Jones, eds., pp. 19-80. Plenum Press, New York. Peters, A. 1987a. Number of neurons and synapses in primary visual cortex. In Cerebral Cortex, Vol. 6, E. G. Jones and A. Peters, eds., pp. 267-294. Plenum Press, New York. Peters, A. 198%. Synaptic specificity in the cerebral cortex. In Synaptic Function, G. M. Edelman, W. E. Gall, and W. M. Cowan, eds., pp. 373-397. Wiley, New York. Peters, A., and Sethares, C. 1991. Organization of pyramidal neurons in area 17 of monkey visual cortex. J. Cornp. Neurol. 306, 1-23. Sayer, R. J., Redman, S. J., and Andersen, P. 1989. Amplitude fluctuations in small EPSPs recorded from CAI pyramidal cells in the guinea pig hippocampal slice. 1.Neurosci. 9, 840-850. Schofield, B. R., Hallman, L. E., and Lin, C.-S. 1987. Morphology of corticotectal cells in the primary visual cortex of hooded rats. J. Comp. Neurol. 261,8597.


680

Sefton, A. J., and Dreher, 8. 1985. Visual system. In The Rat Nervous System, Vol. 1, G. Paxinos, ed., pp. 169-221. Academic Press, Sydney. Staley, K. J., Otis, T. S., and Mody, I. 1992. Membrane properties of dentate gyrus granule cells: Comparison with sharp microelectrode and whole-cell recordings. J. Neurophysiol. 67,1346-1358. Szenthgothai, J. 1978. The neuron network of the cerebral cortex: A functional interpretation. Proc. R. Soc, London B 201,219-248. Thomson, A. M., West, D. C., and Deuchars, J. 1992. Local circuit, single axon excitatory postsynaptic potentials (EPSPs) in deep layer neocortical pyramidal neurones. SOC.Neurosci. Abstr. 18, 1340. Thomson, A. M., Deuchars, J., and West, D. C. 1993. Paired intracellular recordings reveal large single axon excitatory connections between deep layer pyramidal neurones in rat neocortical slices. J. Physiol. (London)459,479. Ts’o, D., Gilbert, C. D., and Wiesel, T. N. 1986. Relationship between horizontal interactions and functional architecture in cat striate cortex as revealed by cross-correlation analysis. J.Neurosci. 6, 1160-1170. ~~

Received 18 September 1992; accepted 26 January 1993.

This article has been cited by: 2. Boris Gourévitch, Jos J. Eggermont. 2010. Maximum decoding abilities of temporal patterns and synchronized firings: application to auditory neurons responding to click trains and amplitude modulated white noise. Journal of Computational Neuroscience 29:1-2, 253-277. [CrossRef] 3. Bryan Kolb, Ian Q. Whishaw. 1998. BRAIN PLASTICITY AND BEHAVIOR. Annual Review of Psychology 49:1, 43-64. [CrossRef] 4. Paul Bush, Terrence Sejnowski. 1996. Inhibition synchronizes sparsely connected cortical neurons within and between columns in realistic network models. Journal of Computational Neuroscience 3:2, 91-110. [CrossRef] 5. Birgit Albowitz, Ulrich Kuhnt. 1995. Epileptiform Activity in the Guinea-pig Neocortical Slice Spreads Preferentially along Supragranular Layers-Recordings with Voltage-sensitive Dyes. European Journal of Neuroscience 7:6, 1273-1284. [CrossRef] 6. Tomoki Fukai. 1995. A model cortical circuit for the storage of temporal sequences. Biological Cybernetics 72:4, 321-328. [CrossRef] 7. Ekkehard M. Kasper, Alan U. Larkman, Joachim L�bke, Colin Blakemore. 1994. Pyramidal neurons in layer 5 of the rat visual cortex. I. Correlation among cell morphology, intrinsic electrophysiological properties, and axon targets. The Journal of Comparative Neurology 339:4, 459-474. [CrossRef]

Communicated by Paul Adams

Sensitivity of Synaptic Plasticity to the Ca2+ Permeability of NMDA Channels: A Model of Long-Term Potentiation in Hippocampal Neurons Erik De Schutter James M. Bower Division of Biology 216-76, California Institute of Biology, Pasadena, CA 91125 USA

We have examined a model by Holmes and Levy (1990)of the induction of associative long-term potentiation (LTP) by a rise in the free Ca2+concentration ([Ca2+1)after synaptic activation of dendritic spines. The previously reported amplification of the change in [Ca2+lcaused by coactivation of several synapses was found to be quite sensitive to changes in the permeability of the N-methyl-D-aspartate (NMDA) receptor channels to Ca2+. Varying this parameter indicated that maximum amplification is obtained at values that are close to Ca2+permeabilities reported in the literature. However, amplification failed if permeability is reduced by more than 50%. We also found that the maximum free [Ca2+lreached in an individual spine during synaptic coactivation of several spines depended on the location of that spine on the dendritic tree. Distal spines attained a higher [Ca2+lthan proximal ones, with differences of up to 80%. The implications of this result for the uniformity of induction of associative LTP in spines in different regions of the dendrite are discussed. 1 Introduction

Since Hebb (1949)first proposed that a synaptic modification based on the co-occurrence of pre- and postsynaptic activity might underlie learning, this idea has formed the basis for many models of network associative learning (Byrne and Berry 1989). Over the last decade, neurobiologists have been studying a physiological phenomenon known as long-term potentiation (LTP), which can have many of the associative properties on synaptic strengths that Hebb originally hypothesized (Nicoll et a2. 1988). Recent work in the hippocampus has implicated a particular membrane channel, the N-methyl-D-aspartate (NMDA) receptor channel, in a type of LTP that is clearly associative (Landfield and Deadwyler 1988). In this case, an increase in synaptic strength is induced when synaptic stimulation coincides with depolarization of the postsynaptic membrane. The Neural Computation 5,681-694 (1993) @ 1993 Massachusetts Institute of Technology

682

Erik De Schutter and James M. Bower

dependence on postsynaptic depolarization appears to rely on the release of a voltage dependent block of this channel by M$+ ions (Mayer et al. 1984; Nowak et al. 1984; Ascher and Nowak 1988). When this block is released, binding of glutamate to the NMDA channel causes an influx of Ca2+,a rise in free [Ca2+]in the dendritic spine (Regehr and Tank 1990; Miiller and Connor 19911, and a change in synaptic efficacy by an as yet not understood secondary mechanism. NMDA channels are permeable to Na+ and K+ as well as to Ca2+ (Mayer and Westbrook 1987; Ascher and Nowak 1988). In most experimental studies on LTP total ionic current through the NMDA channel is measured. The Ca2+ influx is only a small fraction of this total current and it is usually not measured separately, despite its crucial role in the induction of LTP. This distinction may be important because it is known that the Ca2+permeability of other glutamate receptor channels can vary depending on the subunit composition of the channel receptor complex (Hollmann et al. 1991). The apparent association between LTP and conditions for associative memory (Landfield and Deadwyler 1988) has made LTP the subject of a growing number of modeling efforts (Gamble and Koch 1987; Holmes and Levy 1990; Zador et al. 1990). Given its putative role in actually triggering the induction of a synaptic change, the Ca2+ influx and the rise in the cytoplasmic Ca2+concentration ([Ca2+l)in the dendritic spine have been a central focus of this work. Holmes and Levy (1990) have used their modeling results to argue that the simple influx of Ca2+alone is not enough to account for associative effects. Instead they stated that associative LTP could be controlled by a steep nonlinearity in the relation between [Ca2+land the number of coactivated synapses. To show this relation, they used an anatomically reconstructed hippocampal dentate granule cell to build a structurally realistic model (HL model) that included NMDA and non-NMDA receptors on dendritic spines, Ca2+diffusion, a Ca2+buffer, and a pump. With this model they demonstrated that while the Ca2+influx increases only moderately if a large number of synapses are coactivated, the resulting internal free [Ca2+lcan increase 20- to 30-fold. To make their case Holmes and Levy explored several of the important parameters of their model. For example, they demonstrated that the amplification result was robust for changes in Ca2+buffer binding characteristics and buffer concentrations. However, they did not examine the dependence of their results on the Ca2+permeability of the NMDA channel. In this paper we have explored the consequencesof changing the Ca2+ permeability, that is, changing the size of the Ca2+ influx for a given NMDA-current. We have reconstructed the original HL model within the GENESIS simulation environment and have replicated the previously published results. In addition, we have shown that the maximum free [Ca2+]after NMDA channel coadivation is actually quite sensitive to

Sensitivity to Ca2+ Permeability of NMDA Channels

683

the Ca2+permeability. We have also demonstrated that there may be a considerable difference in the peak [Ca2+]expected in spines located in different regions of the dendrite. This result reinforces the idea that the induction of LTP does not only depend on the properties of the NMDA channel and of the Ca2+buffers, but also on the electrical structure of the postsynaptic neuron. 2 Implementation of the Holmes-Levy Model in GENESIS

The HL model was ported to the Caltech neuronal simulation system, GENESIS (Wilson et al. 1989). Two major changes were made to the model. The compartmental structure of the model was simplified to reduce the computational overhead and the equations for the conductance of NMDA and non-NMDA channels were changed to a standard type (Perkel et al. 1981). We modeled 98 dendritic spines, each with 7 compartments representing a cylindrical head of 0.55 by 0.55 pm on a neck of 0.10 by 0.73 pm (Fig. 3 of HL). Each compartment in the spine contained a calcium buffer (100 pM, 200 pM at the top of the head), a Ca2+ pump and Ca2+could diffuse between spine compartments and into the neighboring dendrite. At the top of the spine head NMDA and non-NMDA channels were located. In contrast to the HL model, a standard reaction scheme (Perkel et al. 1981) implemented in GENESIS was used for the NMDA and nonNMDA conductance:

A is the transmitter, R the receptor, AR the closed transmitter-receptor complex, and AR' the open transmitter-receptor complex. The original HL model used two variants of the reaction scheme leading to AR'. In the current model the rate factors in equation 2.1 have been optimized to give the same values of AR' as the HL model (cf. their Fig. 4). Because the value of AR' is the only variable directly relevant to the modeling conclusions, these differences in the reaction scheme are not pertinent. Values used for the non-NMDA channel were rg 600 msec, Th 1.25 msec, 7d 0.001 msec, K, 2 pM, g = 5 pS. For the NMDA channel rg 283 sec, 7h 5.88 msec, Td 0.002 msec, K, 50 pM, g = 50 pS. Equations for the voltage-dependent M g + block of NMDA channels and for Ca2+diffusion, buffers, and pumps were identical to those in the HL model. The fraction of NMDA current carried by Ca2+was based on the permeabilities to Ca2+,Na+, and K+ of NMDA channels (Goldman equations of Mayer and Westbrook 1987). Figure 1 shows that within the voltage and [Ca2+lranges used in the model, only voltage changed this fraction. Because Ca2+ inflow was computed as a fraction of total


684

L

a 0.50

Y

a

:

2 0 C

0.40

0.30

,o 0.20 c,

-

!+i

5

0.10

0.00 -100

-80

-60

-40

-20

0

Membrane potential (mV)

Figure 1: Dependence of the fraction of total NMDA current carried by Ca2+ on membrane potential for five different concentrations of internal [CaZ+l(0.02, 0.20,2.00,20,and 200 pM) at an external [Caz+lof 2 mM. Because total NMDA current becomes zero around 0 mV (the reversal potential), the solution becomes asymptotic close to this potential. Note that most of the curves overlap, only at internal Ca2+ concentrations above 100 pM does internal [Ca2+laffect this fraction. NMDA current, there was no need compute the Ca2+ Nemst potential. The reversal potential of the NMDA current itself changed by less than 0.1 mV for a change in internal [CaZ+lfrom 20 nM to 2 mM (which is the external [Ca2+1). The total number of compartments in the model was 1192. The same cable parameters were used as in the HL model. The spines were randomly distributed over two 165-pm-long dendritic segments, which each contained 98 dendritic spines (inset in Fig. 4). The rest of the compartmental model was highly simplified, having only 20 compartments that represented the soma and 5 other dendrites. This simplification was possible because, under conditions where the active spines are the only source of depolarizing currents, the soma and other dendrites act only as a current drain. Accordingly, as long as the passive load corresponding to the soma and these dendrites was correct, a simplified model produces the same results for the dendritic spines as a detailed model. The input resistance at the soma was 74.4 MR, compared to 72.4 MR in the HL model.

Sensitivity to Ca2+Permeability of NMDA Channels

685

To quantify model results, relative Ca2+ permeability was defined as the ratio between the value used in a particular simulation and the Ca2+ permeability reported by Mayer and Westbrook (1987). A relative Ca2+ permeability of one was thus the experimental value, which corres onded to 12.8% of the NMDA current at -70 mV being carried by Ca ions (Fig. 1). We have adopted the same definition of amplification ratio as introduced by HL, that is, the ratio between maximum free [Ca2+]in a particular spine after coactivation of 96 synapses over the maximum free [Ca2+]after activation of the synapse on that single spine. The stimulus paradigm for induction of LIT was 8 pulses at 200 Hz as in HL. The GENESIS implementation of the HL model described in this paper can be obtained by ftp from babel.cns.caltech.edu.

E

3 Results

In general, our implementation of the HL model within the GENESIS simulation software gave qualitatively equivalent results to those reported by Holmes and Levy. The change in membrane potential and NMDA channel conductance in two dendritic spines during coactivation of 96 spines is compared in Figure 2 with the original data from HL (their Fig. 7). The small differences in peak values were probably caused by sampling, because the size of the responses to synaptic activation was different in every spine. Because we implemented the same Ca2+mechanisms as employed in the HL model, our model also reproduced their computations of [Ca2+lexactly (results not shown). The main new modeling results are presented in Figure 3. Figure 3A shows the sharp dependency of the amplification ratio on the relative Ca2+permeability of the NMDA channel. For 96 coactivated spines versus 1 spine, under the standard conditions of the HL model, the amplification curve peaked at a relative permeability of about 1.25. At lower permeabilities the amplification ratio declined steeply and it dropped below 5 at a relative permeability of 0.50. At higher permeabilities it slowly declined to an amplification ratio of about 10. This dependency was similar for all dendritic spines, independent of their location; there was only a difference in amplitude. The relation between maximum free [Ca2+1and relative Ca2+permeability was shallow and nonlinear for low permeabilities, and steeper and linear for higher permeabilities (Fig. 3B). A peak appeared in the amplification ratio to permeability curve because the linear part started at lower permeability values for activation of 96 spines than for 1 spine. We also examined the effect of changing important parameters in the HL model on the amplification ratio to permeability curve. Changes in the buffer concentration in the spine head changed the location and size of the peak, but not the general shape of the curve (Fig. 3C). For lower


686

A

C -10

-10

.a

-20

s.a

i:

s-30

P -40

1

-50

a0

do

.m

.m 0

30 40

1PO 140 100

0

20

40

60

10 100 120 140 160

Tlmo (ma)

Figure 2: Spine head membrane potential and NMDA receptor-mediated synaptic conductancefor two different spines during coactivationof 96 spine synapses at 50 and 200 Hz. A and B are the original figures of Holmes and Levy (1990, courtesy of The American Physiological Society) and C and D are the corresponding figures produced by the implementation of their model described in this report. In C and D the responses in a distal spine (upper lines) and a proximal spine are shown. The responses in the distal spine are always bigger than in the proximal one. (A, C) Membrane potential as a function of time. (B, D) NMDA channel conductance at the synapse on the same spine heads. Proximal spine is spine #1, distal spine is spine #6 (see Fig. 4).

buffer concentrations the peak was smaller and occurred at smaller relative Ca2+ permeabilities. The reverse was true for higher buffer concentrations. Changes of the rate constant of the calcium pump by a factor of 2 did not change the amplification ratio versus permeability curve and changed the maximum free [Ca2+lby less than 1%(results not shown). We found the amplification ratio to permeability curve to be quite sensitive to the amount of transmitter released presynaptically (A in equation 2.1), which would affect both the NMDA- and non-NMDA-mediated components of the postsynaptic response (Fig. 3D). Doubling the amount of transmitter released per stimulus sharpened the peak considerably and shifted it to low relative Ca2+permeability values (peaking at about 0.5).

Sensitivity to Ca2+Permeability of NMDA Channels A

687

B

1

0

Flrlrtlve

2

3

cep.nnubllity

4

1

2

3

4

Reiatlve Ca2*permeability

D

C

1,. 50

0

0

801

r

I

1

2

3

Flrlatlvr c3' p.mbility

4

OJ 0

1

2

3

4

Relative W*pemeabiilty

Figure 3 Amplification ratio after a 200 Hz stimulus as a function of relative Ca2+ permeability under different model conditions. (A) Standard HL model amplification at two different spines, located distally (upper line, spine 6 in Fig. 4) and proximally on the dendrite (spine 1 Fig. in 4), is compared. (B)Maximum [Ca2+]as a function of relative Ca2+permeability after activation of 1 or 96 spines in the proximal spine. (C) Effect of changing buffer concentration [B] in the spine head on the amplification ratio. (D)Effect of changing the amount of transmitter released per stimulus (A) on the amplification ratio. Halving the transmitter release flattened the curve so that no clear peak could be distinguished; it also greatly diminished the amplification ratio at all levels of Caz+ permeability. As our results suggested a big variation in amplification ratios between different spines, we compared the time course of [Ca2+lfor spines located on different parts of the dendritic tree (Fig. 4). Because of the passive electrical properties of dendrites, distal regions of the cell were likely to be more depolarized than proximal regions for the same amount of input (Fig. 2C). This was a consequence of the passive load of the soma and other dendrites on proximal regions. This in turn means that NMDA channels on distal spines were less blocked by MgZ+ than channels in proximal spines (Fig. 2D).As a result, the Ca2+ concentration reached peaks in distal spines that were 20 to 80%higher than in proximal spines. Further, because there was very little difference in maximum free [Ca2+l after activation of a single synapse (2 to 3%, depending on the relative


688

3 2

s

2

v

I

5

3.

I

0'

20

40

60

80

100

120

140

160

180

Time (ms)

Figure 4 Calcium concentration as a function of time in six spines at different dendritic locations during coactivation of 96 spines. The location of the spines is shown on the gray schematic at the upper right; the six spines are shown in black. Ca2+permeability), differences in amplification ratio also varied between 20 and 80% (Fig. 2). This effect was most pronounced for relative Ca2+ permeabilities of 0.5 to 1.5, which straddle reported experimental values. Within spine heads, there was almost no gradient of free Ca2+or free buffer. For example, 18 msec after the last stimulus at 200 Hz, when free [CaZ+]reached its peak value, [Ca2+]was 30.20 pM under the membrane and 29.37 pM at the base of the spine head. There was however a big gradient over the spine neck, as [CaZf]in the underlying dendritic shaft was only 0.09 pM. 4 Discussion

In this paper we have extended the examination of the parameter space in a previously published biophysical model of associative LTP (Holmesand Levy 1990). While this is the most detailed model of [Ca2+]changes during activation of NMDA receptors on dendritic spines published to date, other models of LTP related changes in [Ca2+]have been reported. For ex-

Sensitivity to Ca2+ Permeability of NMDA Channels

689

ample, an older model by Gamble and Koch (1987) predicted changes in internal calcium concentrations during LTP. However, this model did not explicitly make use of NMDA channels. A second, more recent model by Zador etal. (1990) has essentially the same components as the HL model, but simulates only one spine on a compartmental model of a pyramidal cell and uses a fixed Ca2+ permeability that is independent of voltage (compare to Fig. 1). We expect that the changes in relative Ca2+permeability described here would have a similar effect in this model. Other models of LTP (Kitajima and Hara 1990) or NMDA channels Uahr and Stevens 1990)were not constructed to simulate realistic changes in [Ca2+]. The principal results described here are the apparent sensitivity of the Holmes and Levy model to the Ca2+ permeability of the NMDA channel and to the dendritic position of the activated spines. These properties were a direct consequence of the [Ca2+]amplification mechanism on which the HL model is based, that is, buffer saturation in the spine head. Optimal amplification happened when most of the inflowing Ca2+ was bound to buffers after a single spine was activated, while coactivation of many spines saturated buffers completely and caused consequently a large rise in free [Ca2+]. The buffers in the spine head could saturate, because diffusion out of the spine was restricted as shown by the large drop in [Ca2+]over the spine neck. The HL model fits within present theories of the physiological function of dendritic spines, which emphasize the compartmentalization of chemical processes in the spine head (Koch et al. 1992). It has been shown in large cells that diffusion of Ca2+ buffers may have a profound effect on dynamic changes in [Ca2+l(Sala and Hernlndez-Cruz 1990). The HL model does not simulate diffusion of buffers, but based on our results we do not think that such diffusion plays any role in this system. Because of the small volume of the spine head, there was almost no gradient of free or bound Ca2+buffers. There was a big gradient over the spine neck, but presumably buffers would diffuse much slower through this restricted space than Ca2+itself. The interaction between buffer saturation and relative Ca2+ permeability of the NMDA channel produced several interesting results, some of which may be counterintuitive. For example, increasing the amplitude of the synaptic conductance actually decreased the sensitivity of the system to distinguish between activation of a few or a lot of spines. This was evidenced by the small drop in amplification ratio at a relative permeability of 1.0 (Fig. 3D). If synaptic conductance was increased even more, the decrease would have become more pronounced. Decreasing the synaptic conductance was worse, because the amplification ra ti0 dropped below 5. These results are important as both a short-term and long-term potentiation and a depression of synaptic conductance have been described at hippocampal synapses (Larkman et al. 1991; Malenka 1991). Surprisingly, changing the amount of buffer in the spine head had much less effect on the amplification ratio at a relative Ca2+permeability

690


of 1.0 (Fig. 3 0 , as was also pointed out by Holmes and Levy. At higher dative Ca2+ permeabilities, higher buffer concentrations increased the sensitivity of the system. Note that decreasing the buffer concentration could not fully compensate for large decreases in Ca2+permeability. It has not been proven that a nonlinear amplification of [Ca2+lis the critical feature in associative LTP. For example, if the next step in the induction of LTP (e.g., activation of a Ca2+-dependentkinase; Miller and Kennedy 1986) has a sharp, nonlinear dependence on [Ca2+l,then such a mechanism might be robust enough to operate with smaller changes in [Ca2+l(Zador et al. 1990). However, recent imaging experiments do show increases in [Ca2+]from a resting level of 0.05 to 1.30 pM in dendritic spines under conditions that are expected to induce LTP (Miiller and Connor 1991). Holmes and Levy argue that the nonlinearity underlying the induction of associative LTI' should be as steep as possible and they eliminate Ca2+influx itself as a potential inductor because it is amplified by a factor of only 3. Combining this argument with the experimental data, it seems reasonable to assume that a safe amplification factor for the induction of associative LTP should be at least 10. Thus, we have shown that diminishing the Ca2+permeability by 50% makes the amplification ratio too small to function as a reliable inductor of associative LTP. The same is true for a decrease in the synaptic conductance. Increasing the Ca2+permeability changed the amplification ratio also, but it never dropped below 10. Holmes and Levy did not report the effects of changing these critical model parameters on the predictions made by their model. We have also shown that the location of a particular dendritic spine with respect to the electrical structure of the entire cell may have a profound effect on its participation in LTP. Our results suggest that LTP may be a cooperative phenomenon that besides nonlinear interaction between NMDA channels also involves the structure of the entire postsynaptic neuron. The interaction of Ca2+effects with the passive electrical properties of the cell's dendrite could result in changes of the amplification ratio of up to 80% depending on the particular position of a spine. Whether the difference between a peak [Ca2+l of 18.5 versus 29.8 pM (Fig. 4) would also cause quantitative differences in the amount of LTI' induced is unknown at present. This can be determined only after more experimental data have become available, so that biochemical models of the processes involved in LTP induction can be developed. The current simulations have examined the somewhat unlikely occurrence of activation in only two dendritic segments. Similar effects could be produced if network circuitry results in differential activation and/or inhibition of different regions of a particular dendrite. In this regard, our network simulations of the olfactory piriform cortex (Wilson and Bower 1992) and neocortex (Wilson and Bower 1991) make it seem quite likely that the laminar organization of both the cortex and the hippocampus (Brown and Zador 1990) could easily produce such differential effects


691

on pyramidal cells. In this context, Me1 (1992) reported that a modeled cortical pyramidal neuron with NMDA channels responds preferentially to clustered synaptic inputs versus distributed ones. Location-dependent differences in the magnitude of LTP have been reported in the piriform cortex by Kanter and Haberly (1990). They reported, however, an inverse relation to what the model predicts, that is, LTP induced by association fibers on the proximal parts of the dendrite was larger than that induced by the more distally located afferent fibers. This discrepancy can be explained by several factors, among them specific differences in the NMDA receptors themselves (see further) and the effect of somatic action potentials, which would depolarize proximal NMDA channels more than distal ones (and thus remove the effect of the voltage-dependent Mg2+block). There are several possible consequences of such a location dependence. It is conceivable, for example, that variations in amplification effects in dendritic regions could reflect functional differences in projecting fiber systems. It may be that the operation of a particular neuron would depend on excluding synapses in certain positions from participating in LTP even in the presence of NMDA receptors. For example, there are several examples known where NMDA receptors are present but LTP has not been demonstrated (Artola and Singer 1987). In this case the electrical properties of some neurons may not support the amplification effects shown in the HL model. As pointed out above, in other cases spread of somatic action potentials into the proximal parts of the dendritic tree might counteract the location dependence. While it is interesting to speculate on the possible effects of cell structure and changes in presynaptic transmitter release on the induction of associative LTP, there are ways in which the effects we have described could be overcome. For example position-dependent changes in Ca2+conductivity could counteract the effects shown here. This could be achieved by changing the ratio of NMDA versus non-NMDA receptors or by changing the Ca2+permeability of the NMDA channel. In this regard, recent reports of the permeability to cations of reconstituted non-NMDA channels show that permeability can vary with the subunit composition of the channel complex (Hollmann et al. 1991). It has also been shown that the expression of non-NMDA channel subunits that make the channel permeable to Ca2+can be tissue and cell specific (Burnashev et al. 1992). Though subunit specific variability in cation permeability has not been shown for the NMDA channel, it suggests a molecular mechanism for creating localized differences in Ca2+permeability. Bekkers and Stevens (1990) report significantly lower Ca2+permeabilities for NMDA channels in hippocampal neurons, compared to the values determined by Mayer and Westbrook (1987) in mouse spinal cord neurons (respectively, 4.5 or 12.8% of the NMDA current being carried by Ca2+ at 2 mM external [Ca2+l). It is also conceivable that the Ca2+permeability of the NMDA channel might be affected by phosphorylation of the channel proteins. Similar changes in the degree of phosphorylation have been implicated

692


in numerous molecular mechanisms presumed to be involved in synaptic function (Huganir and Greengard 1990) and protein kinase C has been shown to potentiate NMDA current by reducing the voltage-dependent M 8 + block (Ben-Ari et al. 1992). Changing specifically the Ca2+ permeability of NMDA channels, while keeping their density and total conductance unchanged, would have the advantage that the induction of LTP could be controlled without changing the electrical properties of the neuron. Finally, whatever the significance of differences in dendritic location, our modeling results draw attention to the critical question of the actual permeabilities of NMDA channels to Ca2+. Mayer and Westbrook (1987) and Ascher and Nowak (1988) have pointed out that Goldman (1943) equations (used in the HL model) cannot account for the full properties of the NMDA channel. It is interesting to note that the experimental values for Ca2+permeability reported by Mayer and Westbrook (1987) are within 25% of the values that cause a maximum amplification of free [Ca2+1in the spine head in the HL model. Assuming that other parameters of the model are accurate, this suggests that the dendritic spine apparatus and its control over [Ca2+lmay operate close to maximal efficiency for sensing coactivation of synapses. However, this would not be true for the Ca2+ permeabilities reported by Bekkers and Stevens (19901, which at a relative permeability of 0.35 correspond to an amplification of only 4 to 5. In light of the potentially profound effect on LTP of NMDA receptor permeabilities, it appears to be important to make additional measurements of this value in other brain regions showing associative LTI?

Acknowledgments

This work was supported by Fogarty fellowship F05 TWO4368 to EDS and a grant from the Office of Naval Research, Contract N00014-91-51831. We thank the editors for useful comments on a first draft of this paper.

References Artola, A., and Singer, W. 1987. Long term potentiation and NMDA receptors in rat visual cortex. Nature (London) 330,84436. Ascher, P., and Nowak, L. 1988. The role of divalent cations in the N-methylBaspartate responses of mouse central neurones in culture. J. Physiol. 399, 247-266. Bekkers, J. M., and Stevens, C. F. 1990. Computational implications of NMDA receptor channels. C.S.H. Syrnp. Quant. B i d . 55, 131-135.


693

Ben-Ari, Y., Anikstejn, L., and Bregestovski, P. 1992. Protein kinase C modulation of NMDA currents: An important link for LTP induction. Trends Neurosci. 15, 333-339. Brown, T. H., and Zador, A. M. 1990. Hippocampus. In The Synaptic Organization ofthe Brain, G. M. Shepherd, ed.,pp. 346-388. Oxford University Press, New York. Burnashev, N., Khodorova, A., Jonas, P., Helm, P. J., Wisden, W., Monyer, H., Seeburg, P. H., and Sakmann, B. 1992. Calcium-permeable AMPA-kainate receptors in fusiform cerebellar glial cells. Science 256, 1566-1570. Byme, J. H., and Berry, W. O., eds. 1989. Neural Networks of Plasticity: Experimental and Theoretical Approaches. Academic Press, San Diego. Gamble, E., and Koch, C. 1987. The dynamics of free calcium in dendritic spines in response to repetitive synaptic input. Science 236,1311-1315. Goldman, D. E. 1943. Potential, impedance, and rectification in membranes. J. Gen. Physiol. 27,37-60. Hebb, D. 0.1949. The Organization of Behavior: A Neuropsychological Theory. John Wiley, New York. Hollmann, M., Hartley, M., and Heinemann, S. 1991. Ca2+permeability of KAAMPA-gated glutamate receptor channels depends on subunit composition. Science 252, 851-853. Holmes, W. R., and Levy, W. 8. 1990. Insights into associative long-term potentiation from computational models of NMDA receptor-mediated calcium influx and intracellular calcium concentration changes. J. Neurophysiol. 63, 1148-1168. Huganir, R. L., and Greengard, P. 1990. Regulation of neurotransmitter receptor desensitization by protein phosphorylation. Neuron 5, 555-567. Jahr, C. E., and Stevens, C. F. 1990. Voltage dependence of NMDA-activated macroscopic conductances predicted by single-channel kinetics. 1. Neurosci. 10,3178-3182. Kanter, E. D., and Haberly, L. B. 1990. NMDA-dependent induction of longterm potentiation in afferent and association fiber systems of piriform cortex in vitro. Brain Res. 525, 175-179. Kitajima, T., and Hara, K. 1990. A model of the mechanisms of long-term potentiation in the hippocampus. Biol. Cybern. 64,33-39. Koch, C., Zador, A., and Brown, T. H. 1992. Dendritic spines: Convergence of theory and experiment. Science 256,973-974. Landfield, P. W., and Deadwyler, S. A., eds. 1988. Long-term Potentiation: From Biophysics to Behavior. Alan Liss, New York. Larkman, A., Stratford, K., and Jack, J. 1991. Quanta1 analysis of excitatory synaptic action and depression in hippocampal slices. Nature (London) 350, 344-347. Malenka, R. C. 1991. Postsynaptic factors control the duration of synaptic enhancement in area CAI of the hippocampus. Neuron 6,53-60. Mayer, M.L., and Westbrook, G. L. 1987. Permeation and block of N-methyl-D aspartic acid receptor channels by divalent cations in mouse cultured central neurones. J. Physiol. London 394,501-527. Mayer, M. L., Westbrook, G. L., and Guthrie, P. B. 1984. Voltage-dependentblock

694


by M$+ of NMDA-responses in spinal cord neurones. Nature (London) 309, 261-263. Mel, B. 1992. NMDA-based pattern discriminationin a modeled cortical neuron. Neural Comp. 4,502-517. Miller, S. G., and Kennedy, M. 8. 1986. Regulation of brain type I1 Ca2+/calmodulin-dependent protein kinase by autophosphorylation: A Ca2+-triggered molecular switch. Cell 44,861-870. Miiller, W., and Connor, J. A. 1991. Dendritic spines as individual neuronal compartments for synaptic Ca2+ responses. Nature (London) 354,73-76. Nicoll, R. A., Kauer, J. A., and Malenka, R. C. 1988. The current excitement in long-term potentiation. Neuron 1,97-103. Nowak, L., Bregestovski, P., Ascher, P., Herbert, A., and Prochiantz, A. 1984. Magnesium gates glutamate-activated channels in mouse central neurones. Nature (London) 307,462465. Perkel, D.H., Mulloney, B., and Budelli, R. W. 1981. Quantitative methods for predicting neuronal behavior. Neuroscience 4,823-837. Regehr, W. G.,and Tank, D. W. 1990. Postsynaptic NMDA receptor-mediated calcium accumulation in hippocampal CAI pyramidal cell dendrites. Nature (London) 345,807-810. Sala, F., and Herntindez-Cruz, A. 1990. Calcium diffusion modeling in a spherical neuron. Relevance of buffering properties. Biophys. J. 57,313-324. Wilson, M. A., and Bower, J. M. 1991. A computer simulation of oscillatory behavior in primary visual cortex. Neural Comp. 3,498-509. Wilson, M. A., and Bower, J. M. 1992. Cortical oscillations and temporal interactions in a computer simulation of piriform cortex. J. Neurophysiol. 67, 981-995. Wilson, M. A., Bhalla, U. S., LJhley, J. D., and Bower, J. M. 1989. GENESIS A system for simulating neural networks. In Advances in Neural Information Processing Systems, D. Touretzky, ed., pp. 48-92. Morgan Kaufmann, San Mateo, CA. Zador, A., Koch, C., and Brown, T. H. 1990. Biophysical model of a Hebbian synapse. Proc. Natl. Acad. Sci. U.S.A. 87, 67184722. Received 20 March 1992; accepted 25 January 1993.


Communicated by Sidney Lehky

Models of Perceptual Learning in Vernier Hyperacuity Yak Weiss Interdisciplinary Program, Tel Aviv University, Tel Aviv 69978,Israel Shimon Edelman Department of Applied Mathematics and Computer Science, The Weizmann Institute of Science, Rehovot 76100, Israel Manfred Fahle Department of Neuroophthalmology,University Eye Clinic, Schleichstr. 12,07400, Tubingen, Germany Performance of human subjects in a wide variety of early visual processing tasks improves with practice. HyperBF networks (Poggio and Girosi 1990) constitute a mathematically well-founded framework for understanding such improvement in performance, or perceptual learning, in the class of tasks known as visual hyperacuity. The present article concentrates on two issues raised by the recent psychophysical and computational findings reported in Poggio et al. (1992b) and Fahle and Edelman (1992). First, we develop a biologically plausible extension of the HyperBF model that takes into account basic features of the functional architecture of early vision. Second, we explore various learning modes that can coexist within the HyperBF framework and focus on two unsupervised learning rules that may be involved in hyperacuity learning. Finally, we report results of psychophysical experiments that are consistent with the hypothesis that activity-dependent presynaptic amplification may be involved in perceptual learning in hyperacuity. 1 Introduction

The.term “perceptual learning” refers to the significant improvement, precipitated by practice, in the performance of human subjects in various perceptual tasks (Walk 1978). Some of the more intriguing aspects of perceptual learning, such as its specificity for particular stimulus parameters, and the associated lack of performance transfer to new parameter values (Fiorentini and Berardi 1981; Karni and Sagi 1991), remained until not long ago without an adequate computational explanation. In the present report, we show that a recently proposed mathematical framework for learning from examples, known as HyperBF approximation (Poggio and Neural Computation

5,695-7l8 (1993) @ 1993 Massachusetts Institute of Technology

Y. Weiss, S. Edelman, and M. Fahle

696

Girosi 19901, yields a biologically plausible and flexible model of perceptual learning in early vision. Following the work described in Poggio et al. (1992b), we concentrate on the example of learning vernier hyperacuity. 1.1 HyperBF Networks. Within the HyperBF framework, the problems of detection and discrimination of visual stimuli are approached in terms of the computation of multivariate functions defined over the input space. In particular, learning to solve these problems is considered equivalent to approximating the value of an appropriate function at any point in the input space, given its values at other points that belong to a set of examples. In a standard implementation, the task of approximating a function is divided into two stages (Poggio and Girosi 1990): an initial (usually nonlinear) transformation, in which the input is mapped into a set of basis functions, and a linear stage, in which the output function is computed as a linear combination of the basis functions. More precisely, the function f(x) is approximated as f(x) = c h(x) where h(x) is a vector of the values of the (nonlinear) basis functions, and c is a vector of weights. It is possible to divide the initial transformation into two substages: a transduction or dimensionality reduction stage, in which the input is mapped into a real vector space V,and a basis function computation stage, in which the value of each component of h is determined by a function hi : V -+ R. If the basis functions are radial, then hi(x) = hi( IIx - xoll), where ~0 are called the centers of the chosen set of basis functions. In a distributed implementation of this scheme by a multilayer network (Poggioand Girosi 19901, h represents the response of units in an intermediate layer to the stimulus, and c represents the weights of the synapses between the intermediate layer and the output unit.

-

1.2 Modeling 2AFC Experiments with HyperBF Networks. Because the output of the HyperBF module is a continuous function of the input, an additional mechanism is needed to model the decision stage in a two-alternative forced choice (2AFC) experiment. For that purpose, the full model (see Fig. 1) includes a threshold mechanism that outputs +1 or -1, depending on the sign of its input. Such a threshold unit is likely to be affected by noise, whose source can be, for example, the spontaneous activity of nearby units [we call this “decision noise,” as opposed to “early noise” that is already present in the values of the basis functions h(x)l. Thus, the output of the threshold unit can be described by R(x) = sign (c h DN), where DN is a zero-mean normal random variable with a standard deviation ON. Given the distribution of the noise in the system, it is possible to calculate the performance of the model in a 2AFC experiment. For example, if Y is the response of the HyperBF module to a certain right-offset vernier,

+

Models of Perceptual Learning in Vernier Hyperacuity

i

697

threshold HyperBF

module influence of

7

Figure 1: A model of a two-alternativeforced choice (2AFC) experiment. The output of the HyperBF module (left) is thresholded. Because the thresholding unit is also affected by the spontaneous activity of other units, only stimuli that elicit a strong response from the HyperBF network will be detected correctly with a high probability. and if early noise is neglected, the probability of a correct response is

The offset threshold of the model is then defined as the smallest offset, OT, for which the probability of correct responses as defined above exceeds 0.75. If the network's output depends on the input offset in an almost linear fashion (see Fig. 41, then the psychometric curve and the threshold can be predicted analytically. Substituting Y(o) = ao in equation 1.2 gives (1.2)

The percentage of correct responses plotted against the input offset will then be a sigmoid, and the threshold will be inversely proportional to a. Although the above approach makes it possible to model both interpolation and decision performance, we chose to concentrate on the former. This choice was motivated by the assumption that the interesting aspects of learning have to do with changes in the performance of the interpolation module. In the modeling of vernier acuity, this assumption

698


is supported by the psychophysical findings of stimulus specificity of learning, which cannot be accounted for by decision-level changes alone (Poggio et al. 1992b).

1.3 Vernier Hyperacuity. A vernier target consists of two line segments, separated by a small offset in the direction orthogonal to the orientation of the lines. The subject's task in a vernier acuity experiment is to judge whether the misalignment is to the left or to the right. Humans solve this task successfully when the offset is as small as 5" or less, exhibiting a discrimination threshold that is far lower than that for spatial frequency grating discrimination or for two point resolution, and is smaller than the spacing of the cones in the fovea. Moreover, this astonishing precision of vernier discrimination, termed hyperacuity, is maintained even when the target is in motion (Westheimer and McKee 1975). It should be noted that hyperacuity performance does not contradict any physical law, since the optics of the eye at the fovea satisfy the constraints of the sampling theorem, making the spatial information recoverable in principle, by appropriate filtering (Barlow 1979; Crick et al. 1981). O u r choice of vernier hyperacuity as a paradigmatic case of perceptual learning was motivated by two considerations. First, the improvement in the vernier threshold has both fast and slow components (McKee and Westheimer 1978; Fendick and Westheimer 1983; Fahle and Edelman 19921, signifying that a number of distinct learning mechanisms may be at work. Second, as we have already mentioned, perceptual learning in vernier hyperacuity is specific for stimulus orientation (Fahle and Edelman 1992), a possible indication that performance in this task is based, at least in part, on interpolation among input examples acquired during learning. The next two sections describe in detail a computational model of hyperacuity performance based on HyperBF interpolation. 2 Modeling Hyperacuity-Level Performance 2.1 Simulated Experiments. In the simulated psychophysical experiments described below we compared two versions of the HyperBF scheme, one without and the other with a preprocessing or transduction stage. Unlike in Poggio et al. (1992a) where a transduction stage was used, in version A of the present model the basis function vector h represented the activities of three orientation-selectiveunits, with response peaks at -15", 0", and 15" with respect to the vertical. The response of each unit was calculated by convolving the stimulus retinal image with the appropriate receptive field function RF(x,y) and adding noise EN:


699

Figure 2: A simple network for vernier discrimination, obtained by combining responses of orientationally selective units. If the parameters of the oriented units are set according to the data from psychophysical masking experiments (Wilson and Gelb 1984), this network can solve the vernier task at a much smaller threshold than the size of the excitatory region of each unit. This network is suitable for solving the vernier task only when the stimulus is vertically oriented. Networks that combine responses of a range of orientationally selective units can be used for stimuli of different orientations. For the 0" unit RF(x,y) = e-~/"~(e-'/": - Be-"/":+Ce-'/":). Equations for the f15" units were obtained using standard rotation of coordinates. All constants were taken from Wilson and Gelb (1984) as those representing the smallest spatial frequency channel in human subjects. These constants are based on masking experiments, and are consistent with data from single cell recordings in macaque striate cortex (Wilson 1986; Wilson and Gelb 1984). EN was a zero mean gaussian random variable. The responses of these three basis functions to a sequence of vernier stimuli (a "vernier tuning curve") is shown in Figure 3, along with a more conventional orientation tuning curve for the vertically oriented unit. In version B, the orientation-selectiveunits described above were considered as transducers that mapped the activity pattern of the retinal array into R3. The network solved the problem by carrying out radial basis function interpolation in R3. The basis functions comprising h were three gaussians in R3,centered around the transduced representations of three vernier stimuli with offsets of -30,0,30 arcsec, respectively. The widths of the gaussians were set to the average distances (in R3)between their centers. In both versions the weight vector c was obtained by solving the equation:

Hc=Y

(2.1)

Y.Weiss, S. Edelman, and M. Fahle

700

n

8 0.0' (ro

0

-30

-20

- 10

0

10

20

30

Offset (arcsec) Response characteristics of basis units (version A)

Figure 3: "Single-cell recordings" from the basis units in the network, for a range of vernier offset values. The three curves marked by filled symbols are the responses to vernier stimuli of the three orientation-selective units shown in Figure 2. The error bar shows the standard deviation of the noise used to obtain the response curve shown in Figure 4. For comparison, the response curve of the vertically oriented unit to an unbroken line passing through the midpoints of the two lines comprising the vernier target is also shown (the top curve, marked by circles). The network treats such an oriented line as equivalent to a vernier target. Single-cell recordings from area 17 of a cat's visual cortex (Swindale and Cynader 1986) have revealed similar response patterns.

where each row in the matrix H represents the activity of the hidden layer to a vernier stimulus, and Yi is set to -1 for left offsets and to +1 for right offsets. 2.2 Results. We fmt explored the response of the network to vernier stimuli consisting of two lines 8' long. In both versions, the weight vector c was of the form cr (1,0, -l)Tand the network could be described simply as an output unit with an excitatory synapse with the unit tuned to the left-slanting lines and an inhibitory synapse with the unit tuned to the right-slanting lines (see Fig. 2). The zero weight of the vertically oriented unit is due to the fact, pointed out by Wilson, that this unit, despite being the most active one, carries the least amount of information relevant to the solution of the task. The output of this network for offsets ranging from -30" to 30" is shown in Figure 4a. These graphs represent the response of the network


.

a.

.

701

1

b.

Figure 4 (a) Response of the network plotted vs. the vernier offset of a stimulus presented at a fixed location with respect to the receptive fields. The two versions, A and B (topand bottom rows), yielded similar results here. (b)Response of the network vs. the vernier offset of a stimulus presented at random locations within a 20'' horizontal range around the common center of the receptive fields. to a vernier stimulus centered over the orientation detectors. In practice, however, random eye movements of the subject prevent such precise centering of stimuli during psychophysicalexperiments. Figure 4b shows the network's responses to stimuli displaced by random displacements of up to 20" in the vertical and horizontal directions.' It can be seen that the network's output is little affected by stimulus location. Interestingly, while the network is sensitive enough to signal a vernier displacement of only l",it practically ignores a much larger translation of the entire stimulus. Note that in both cases the network's output depends on the input offset in an almost linear fashion, while the function to be approximated, f(x), is a step function. Such poor approximation of the target function is understandable, considering the smoothness of the basis functions. A much better approximation could be reached using discontinuous basis functions, or, alternatively, using a large number of narrow gaussian basis functions. 'The distance between units that were spatial nearest neighbors was used by Wilson and Gelb (1984)as a free parameter to fit spatial frequency discrimination data. For the filters we used in our model, this distance corresponded to 38.3". Thus, a stimulus appearing at a random retinal location would always be within 20" of a set of filters.


702

. ... .

. .. . .

.

a.

.

. .. ...

. .. ....

.

.

. .. ...

...

.

.

l:wth7.raA")

Dmpm&a of lhrnhold mm wr&r

I m W

Figure 5: Dependence of vernier threshold on line length (a) and separation between lines (b). Threshold was estimated by making assumptions regarding the statistical distributions of the noise sources. These distributions were held fixed as stimulus parameters varied. Compare to Figure 6.

......... ................. .....

.....,.. ..............................

10

a.

m

.

..........

'

Length (:;amh)

Dmp.L.n m I lbnbmld *m wr.lmr Im+

.

.............

b.

,

a

*

#

&.Uonl\.ra& Dmpmmhm #I 8 b m s L . M 08 I l u upu.U.m

Figure 6 Dependence of vernier threshold on line length (a) and separation between lines (b). Each plot shows data from two observers, replotted from Westheimer and McKee (1977). 2.3 Discussion. Since both versions of the model (A and B) can serve as the basis for hyperacuity level performance, we adopted the simpler, linear, version that omits the transduction stage, as a minimalist platform for studying the improvement of hyperacuity with practice. The primary motivation for using oriented filters as basis functions for vernier hyperacuity, was the report by Wilson (1986) [see also Klein and Levi (1985) and Watt and Morgan (1985)l that the responses of these filters can explain psychophysical data concerning hyperacuity. In Wilson's model, detection threshold in psychophysical experiments is related to the Euclidean distance between a distributed "filter representation" of the two stimuli. The filter representation is constructed by pooling the responses of filters at all orientations and spatial frequencies concentric with the stimulus as well as over spatial nearest neighbors. Our model,


703

in contrast, replaces the distributed representation with the output of a single neuron. This does not necessarily make our model more biological, but it makes it easier to model hyperacuity learning in terms of synaptic modification. Wilson’s model replicated several psychophysical results concerning the change in hyperacuity thresholds in a variety of hyperacuity tasks when stimulus parameters varied. To see whether these results still hold when the distributed representation is collapsed to a single neuron, we investigated the response of the network to verniers of varying line length and varying gap in the direction of the lines. The results of these simulations appear in Figure 5. In these simulations the statistical distribution of the noise was held fixed (and thus thresholds could be estimated) while the parameters of the stimulus varied. The dependence of the threshold on line length exhibited by the model agrees reasonably well with the data in McKee and Westheimer (1977). Specifically, the threshold decreases steeply with increasing segment length for lengths under 4’, and is essentially unaffected by further increase. The dependence of the threshold on line separation, however, agrees with psychophysical data only qualitatively. The model’s threshold increases steeply for separations greater than 4’, while in human subjects the increase is more gradual and is especially noticeable for separations over 7’. Both the results regarding line length and line separation can be attributed to the fact that the width or of the spatial frequency mechanism used by the model was 3.65’(Wilson and Bergen 1979). As Wilson has pointed out, increasing line length beyond the value of cy does not add any significant information, while increasing line separation beyond uy forces the human subject to use the less sensitive spatial mechanisms. Additional motivation for using orientationally selective units as basis functions in our model comes from the electrophysiological studies of Swindale and Cynader (1986), who studied the response to a vernier break by orientation selective cells in cortical area 17 in the cat. The results of that study showed that orientation-selective cells in area 17 can discriminate between different offsets in a vernier stimulus. Specifically, those cells tended to respond to the vernier stimuli in the same manner as they did to an oriented line passing through the midpoints of the two segments composing the vernier. This effect is basically due to the spatially low-pass action of the orientationally selective units, and has been replicated by our model (Fig. 3). Swindale and Cynader used a method proposed by Parker and Hawken (1985) to estimate the “hyperacuity threshold” of single neurons in area 17. This threshold measures the statistical reliability of a change in a neuron’s response to a vernier break. Because the thresholds of some neurons were as low as the behavioral threshold of the cat in vernier discrimination, the authors suggested that the performance of these neurons was the limiting factor in hyperacuity, obviating the need for a fine-grid reconstruction of the stimulus. In response, Parker and Hawken (1987) argued

704


that the possibility of a fine-grid reconstruction could not be mled out, because the factors limiting the behavioral hyperacuity threshold may be retinal and not cortical, as suggested by the data on the hyperacuity thresholds of cat retinal ganglion cells due to Shapley and Victor (1986). We note that the HyperBF approach is equally capable of modeling retinally or cortically based hyperacuity mechanisms. While our present model used orientationally selective units similar to cortical simple cells, the HyperBF scheme of Poggio et al. (1992a) that relied on responses of circularly symmetric units was equally successful in replicating hyperacuity phenomena. The notion of interpolation, inherent to the HyperBF approach, does provide, however, a useful insight into one issue important for both sides in the retina vs. cortex debate, namely, the way of relating behavioral thresholds to those of single neurons. Consider the vernier tuning curve of the vertically oriented unit in our model (the curve marked by triangles in Fig. 3). Despite the fact that this curve is relatively wide and shallow, the responses of three units of this type can support hyperacuity vernier discrimination. Addressing Westheimer’s (1981) claim that neurons with a wide orientation response characteristic cannot be involved in hyperacuity tasks, Swindale and Cynader argue that a broadly tuned neuron can still support hyperacuity, as long as its response pattern is statistically reliable. This is equivalent to saying that the slope of the tuning curve, and not its width, should be used as a measure of a neuron’s use for hyperacuity. In contrast, our model suggests that neither measure should be considered a sole determinant of the behavioral threshold: the responses of cells slightly rotated with respect to the stimulus actually provide more relevant information for solving the task. Thus, a network of very reliable vernier detectors may perform worse than a network of less reliable units with a large overlap in their receptive fields. The limiting factor in the vernier task seems therefore to be not the performance of a single unit, but rather the ability of the system to pool responses from different units with overlapping receptive fields, and the manner in which these units cover the range of possible stimulus orientations (see Fig. 7).’ To conclude this discussion, we note that Snippe and Koenderink (1992)recently demonstrated analytically using an ideal observer model, that the resolution of a channel-coded system of circularly symmetric receptive fields is determined both by the reliability of each channel and by the degree of overlap between the channels. 3 Modeling Perceptual Learning in Hyperacuity

We now turn to explore the possible ways in which the performance in the vernier task could be made to improve with practice. First, we *Swindaleand Cynader mention pooling the outputs of several neurons, but they suggest that this pooling occurs between neurons with similar responses, thus increasing the reliability of a single channel, and not pooling the responses of different channels.


- - --

....

-I

.

Response of oriented neuron

Response of vertical neuron 120

705

90, 80

,

,

-20

-10

I

I

3

20

30

70 60 50 40

__

2n

_ _- 3 0

-20

-10

0

10

20

30

20 10 0 -10 -30

0

10

Figure 7 To demonstrate the importance of pooling responses of a number of overlapping filters, we conducted two simulations in which the output of the filters was passed through a monotonic nonlinearity before noise was added. In the first simulation, the nonlinearity was compressing at high activity rates, resulting in a shallow vernier tuning curve. In the second simulation, the nonlinearity was accelerating at high activity rates, leading to a steep vernier tuning curve. (a) The average response of two simulated neurons to vernier displacements in a vertically oriented stimulus (lower curve: type I; upper curve: type 11). (b) The average response of two simulated neurons with the same nonlinearity as those in (a) but with oriented receptive fields. A network comprised of three neurons of type I1 (upper curve) performs better in the vernier task than a network comprised of three neurons of type I (lower curve), despite having a shallower vernier tuning curve. show how different types of learning rules can be formulated within the HyperBF framework. We then focus on two likely candidate mechanisms for unsupervised learning in vernier acuity experiments, and describe additional simulations that help distinguish between them. The results of these simulations suggested a psychophysical experiment, reported in the next section. We also discuss a possible biological basis for one of the two unsupervised learning rules.

3.1 Classification of Perceptual Learning Models. Perceptual learning models can be categorized according to two basic criteria: how much prior knowledge is assumed and how dependent the learning is on external supervision. Both parameters can assume a wide range of values, a fact that is sometimes overlooked when models are characterized simply as “supervised” or “unsupervised.” Supervised models, in turn, may differ in the nature of feedback signal they assume. Barto (1989) distinguishes between two types of error signals-those generated by a ”teacher” that can point out which parameter of the model should be modified in response to the error it has committed, and those generated by a ”critic” that is unaware of the work-

706


Figure 8: The structure of the network used throughout the learning simulations. The output neuron receives input from 100 neurons, three of which correspond to the properly oriented and positioned linear filters, and the rest to other, random, inputs. Performance is improved by modifying the connections between these neurons and the output neuron.

ings of the model. “A critic can generate a payoff based on knowledge solely of what it wants accomplished and not of how the learning system can accomplish it” (Barto 1989, p. 89). Unsupervised models can also differ in their dependence on feedback. Some unsupervised models merely replace the external feedback signal with a self-provided feedback signal, an “internal teacher.’’ Alternatively, an unsupervised model can assume complete independence on feedback, either internal or external. A fundamental tradeoff exists between a learning model’s reliance on prior knowledge and on feedback. A model that relies heavily on a teacher can afford to make few prior assumptions, while a model that assumes independence from feedback must rely on prior knowledge to a greater extent. As an example of feedback-independent unsupervised learning, consider the model of hyperacuity, proposed by Barlow and others (Barlow 1979; Crick et al. 1981), that relies on a fine-grid reconstruction of the retinal signal in the cortex. Assume that the module that processes the reconstructed signal has no intrinsic capacity for learning, but operates in such a manner that increasing the accuracy of the he-grid reconstruction causes an improvement in its performance in the hyperacuity task. Any improvement in the fine-grid reconstruction would then cause a decrease in hyperacuity threshold, but this improvement need not be feedback-dependent. Indeed, the reconstruction may improve after training on a completely different task. 3.2 Different Learning Modes in a HyperBF Network. In Poggio et al. (1992a1, a HyperBF network was synthesized from a “tabula rasa” initial state. Only the shape of the basis functions (that were radially symmetric multidimensional gaussians) was assumed to be given. The centers of the basis functions were determined by the training examples in an unsupervised fashion, while the coefficients c were updated using


707

a pseudoinverse technique that assumed an external teacher. Poggio et al. also suggested using self-provided feedback to replace the external teacher. In our model, the structure of the network was assumed to remain fixed throughout training. The network (see Fig. 8) was comprised of 100 units that were connected to an output neuron. Three of the units represented the oriented linear filters described in section 2.1 and the activity of the remaining units was random. The model's performance was improved solely by changing the weight vector c, according to four different update rules: = c(') + vh [Y(x)- O(x)], where Y 1. The Widrow-Hoff rule, and 0 represent the desired and the actual output for the stimulus x. This rule is supervised by a teacher, and is equivalent to solving equation 2.1 by an incremental pseudoinverse technique (Widrow and Steams 1985). 2. The Mel-Koch rule. c('+') = c(') + ahY (x) - P c ( ~ )This . learning rule was suggested by Me1 and Koch (1990) and was designed to maximize the correlation between the output and the activities of the basis function units, while minimizing the total synaptic weight of the linear stage. This model, unlike the previous one, is supervised by a critic who knows only what the correct answer should be. 3. The self-supervised Widrow-Hoff algorithm. This algorithm is similar to the first one, but feedback is provided only for those inputs in which the vernier offset exceeds the baseline threshold (set at 15"). This model is unsupervised, but is still feedback-dependent. It was designed to simulate the conditions in psychophysical experiments in which subjects receive no feedback at all, but nevertheless possess a clear indication of the correctnessof their response for the large values of vernier offset when the stimulus looks trivially easy. Under these conditions, the subjects' thresholds improve with practice, albeit at a slower rate than when explicit feedback is available (Fahle and Edelman 1992). = c!') + ac!'),if 1h:')I > E. This is an unsupervised, usedependent rule, which is independent of feedback. As opposed to the rules listed above, which made no assumptions about the nature of the connections c prior to learning, this rule assumes that the weight vector for the oriented filters is proportional to (+I,o, -11~.

4. Exposure-dependent learning (EDL). c!'")

3.3 Results. The learning curves for the four rules described in the previous section are shown in Figure 9. All rules except one showed a gradual improvement of performance with pra~tice.~ The shapes of the 3The failure of Me1 and Koch's rule to converge was not due to simulation details. An analysis of their rule formulated as a first-order differential equation showed that


708

1 Y 0

Y 0

u

I

u

!i

a:

0.85

2

8

0.8 0.75 0.7 0.65 0.6 0.55

u

B 1

n.5

0

2

4

6

8 10 12 1 4 16 18 20 Block

a.

z

8 Y

B

P

0.6

0.55 0.5 0

2

4

6

8 10 12 1 4 1 6 18 20

Block

0.95

0.95

u u

0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5

z

0.9 0.85

u

0.8

u

0.75

a:

0.65

k

0.7 n _. 6. _

0

2

4

6

0

8 10 12 1 4 16 18 20

Block C.

0.9 0.85 0.8 0.75 0.7 0.65

b.

1 u U

0.95

d.

2

4

6

8 10 12 1 4 16 18 20

Block

Figure 9 Learning curves for different coefficient-learning rules defined in Section 3.2 [(a)Widrow-Hoff; (b) Mel-Koch; (c) self-supervised Widrow-Hoff; (d) EDLI. All rules (except the Mel-Koch rule) show a gradual improvement of performance with practice. The models do not achieve perfect performance due to the noise present in the receptor activities (“early noise”). When this noise is increased, performance still improves, but the final performance level is lower (see the lower curve in condition d). learning curves should be considered as a qualitative characteristic of the models’ performance, since they depend on the scalar parameters 77 and a (we note that a considerable variability in the learning rate is also found in human subjects). The models do not achieve perfect performance due to the noise present in the receptor activities (“early noise”). When this noise is increased, performance still improves, but the final performance level is lower (see the lower curve in Fig. 9d). 3.4 Separating the Noise from the Signal: W o Approaches. Of the four learning rules mentioned in the previous section, the two likely candidates for accounting for the improvement with practice found in psychophysical experiments are the two unsupervised rules, because learning has been found to occur in the absence of feedback. To help eluit was not guaranteed to converge under the conditions of the simulated experiments, in which widely differing inputs are presented in alternation.


709

m 0.95 0.9 0.85

4

0.6

3.5 3

0.5

0.8

0.75 0.7 0.65 0.6 0.55 0.5

0.7

1.5

0.4

a

0.1

1.5 1

0.a

0.1

o a

4

6 8 10 la 14 16 ia

ao

0.)

0

0

0 I

1

0.35

0.9

0.3

0.I

0.15

0.7

0.1

0.6

0.15

0.5

0.1

0.4

0.05

4

6 I 10 11 14 16 18 10

o

1

6

6 I 10 ia 14 16 11 a0

3.S

3 1.5

a 1.5 1 0.5 0

0 I

4

6 I 10 12 14 16 I8 10

o a 4 6 8 10 ia 14 16 18 ao

Figure 10 A comparison of the two unsupervised learning rules under two conditions: one where the activity of noisy neurons was determined randomly before each presentation, and the other where they remained constant during learning. The Widrow-Hoff algorithm causes improved performance under both conditions but the EDL algorithm does not improve in the second condition, since it tends to amplify the noise rather than the signal.

cidate the difference between the two learning rules, we conducted an additional set of learning simulations. These simulations compared the performance of the two learning rules under two conditions which differed only in the firing patterns of the “noise” units. In both conditions, these firing patterns were determined using a zero mean gaussian random variable. In the first condition (conditionA) these firing rates were determined before each presentation of the stimulus (this condition is identical to the one used in the simulations in the previous section) while in the second condition (condition B) these firing rates were determined prior to training and remained constant throughout training. Note that if each presentation is considered by itself, the statistical properties of the “noise neurons” are identical (independent, identically distributed gaussians).

3.4.1 Results. The behavior of the two learning rules under these two conditions is illustrated in Figure 10. The first column shows the percentage of correct responses for the two learning rules. The Widrow-Hoff rule supports learning under both conditions, but the EDL rule does so only in condition A, and actually leads to deterioration of performance with practice in condition B.


710

The second and the third columns show the evolution of signal and noise magnitudes during training. These were defined as

where indices i = 1 , 2 , 3 correspond to the oriented filters. Note that the term "signal" for the value S is somewhat misleading since it also includes the contribution of early noise present in the receptors. The difference between the two learning rules is in the way they distinguish signal from noise. The Widrow-Hoff algorithm converges to a vector c such that c h is closest in the mean square sense to the desired output. Thus,presynaptic activity, which is completely uncorrelated with the desired output, will result in a zero-weight synapse, in effect labeling the corresponding presynaptic unit as noise. Note that according to this definition, the activity of the vertically oriented filter is also labeled as noise, and indeed the Widrow-Hoff algorithm results in a zero-weight synapse between the vertically oriented filter and the output unit. A feedback-independent learning rule such as EDL cannot rely on the correlation between the desired output and the presynaptic activity in distinguishing noise from signal. The heuristic used by this rule is to label as signal those inputs which are consistently active at a significant rate when the stimulus is presented. This is achieved by increasing by a small amount the contribution of a unit whose activity is greater than some threshold E at each presentation. Thus, when the activities of the random units are recalculated prior to each stimulus presentation, the increase in their contribution to the output unit is negligible compared to the increase in the contribution of the oriented filters (see Fig. lo), because only the oriented filters are consistently active. In contrast, in condition B, the number of noise units whose activity is greater than E is the same as in condition A, but the increase in the contribution of the noise units is significant, due to the consistent activity of the same noise units. In the simulations described above, the number of random units with activity greater than E was greater than the number of oriented filters, so that in condition B noise was boosted more than the signal, resulting in decreased performance.

-

3.4.2 A Possible Biological Basis of the EDL Rule. The EDL update rule requires a modulatory signal to mark the time frame within which presynaptic activity should be measured, and a mechanism for updating synapses based on presynaptic activity. A learning rule that satisfies


711

these requirements has been studied extensively both at the behavioral and at the cellular level by Hawkins et al. (1983) in Aplysia. The Aplysia’s gill withdrawal reflex following stimulation of a particular site on the siphon was found to be enhanced by a shock delivered to the tail in parallel with the stimulation of the siphon. This enhancement was specific to the site which was stimulated at the time of the shock delivery to the tail, presumably because it depended on simultaneity of the shock and the siphon stimulation. A correlate of this phenomenon at the cellular level was found in a study of the change of the excitatory postsynaptic potentials (EPSPs) elicited in a common postsynaptic neuron by siphon sensory neurons. During training, two sensory neurons were stimulated intracellularly. Stimulation of one of them immediately preceded the shock to the tail, while stimulation of the other sensory neuron followed the shock by 2.5 min. It was found that the change in the amplitude of the EPSP from the paired neuron was significantly greater than that in the unpaired neuron. Further experiments suggested that “activity-dependent amplification of facilitation is presynaptic in origin and involves a differential increase in spike duration and Ca2+ influx in paired versus unpaired neurons” (Hawkins et al. 1983). The requirements of the model of Hawkins et al. are (1) facilitatory neurons, which are excited by motivationally significant stimuli and which may project very diffusely (in principle, a single such neuron that produced facilitation in all of the sensory neurons would be sufficient to explain the results) and (2) differential activity in the neurons that receive facilitatory input (Hawkins et al. 1983). In our simulations we assumed that all the units received the modulatory signal, that is, the network had no a priori knowledge as to the activities of which units were more likely to be significant. The main difference between our mechanism and the one suggested by Hawkins et al. is that we assume that when synaptic amplification occurs, it is proportional to the previous synaptic strength. Without this assumption, the modification of the synapses distorts whatever structure the network connections had prior to learning. For example, the connections of the vertically oriented unit, which is irrelevant to the task, could increase if this assumption were dropped. We note that this assumption adds a Hebbian element to the learning rule. Consider two units with identical activity, one of which has a strong connection to the output unit and consistently takes part in the activation of the output unit, and the other is weakly connected to the output. The synaptic weight of the strongly connected unit, whose activity is correlated with the output, would increase more significantly than that of the weakly connected unit. We do not assume, however, that there is any causal relationship between the correlation of pre- and postsynaptic activities, and the synaptic modification. Hence, our rule may be classified as a “noninteractive Hebbian rule” (Brown et al. 1990).


712

4 Psychophysical Experiments

In the previous section, we described the difference between the two unsupervised learning rules in terms of their approach to distinguishing between signal and noise. If a group of units is consistently active during stimulus presentation, but their activities are uncorrelated with the desired output, then they would be labeled as noise by the Widrow-Hoff algorithm and as signal by the EDL rule. To elucidate the possible role of an EDL-like rule in the improvement of performance in vernier hyperacuity, we conducted psychophysical experiments using cross-shaped stimuli shown in Figure 11. Two orthogonal verniers appeared simultaneously in each trial, but the subjects were required to judge the sense of the misalignment of only one of the two verniers (the orientation of the relevant vernier was the same throughout each experimental block). In this situation, the units responsive to the irrelevant part of the stimulus are consistently activated during stimulus presentation, but their activity is uncorrelated with the desired output. Hence, i€ the learning mechanism contains a significant use-dependent component (of the kind that can be provided by the EDL rule), such an experiment is expected to demonstrate similar improvement with practice in the two orientations. 4.1 Method. Subjects performed three tasks that involved crossshaped stimuli as in Figure 11. Stimuli were generated and displayed on a Silicon Graphics 4D35/TG workstation. Viewing distance was such that one pixel corresponded to about 8". Stimuli were presented for 100 ms and were separated by a 1 sec interval (during which a frame was displayed to assist fixation). Subjects indicated their response by pressing one of two buttons on the computer mouse. Auditory feedback was given for incorrect responses. The stimuli in the first two tasks, HORIZONTAL CROSS and VERTICAL CROSS, were the same, except that in one (VERTICALCROSS) subjects were required to determine the direction of misalignment of the vertical part of the stimulus (and received appropriate feedback), while in the other (HORIZONTAL CROSS) they were required to judge the misalignment of the horizontal part of the stimulus (again with appropriate feedback). In the third task, DIAGONAL CROSS, the stimuli were oriented diagonally. Each block consisted of a fixed number of presentations of all offsets of the relevant stimulus (in the range from -20 to $20 pixels) in a random order of presentation. The irrelevant part of the stimulus (e.g., the vertical vernier in a HORIZONTAL CROSS block) was presented with a random offset and was thus uncorrelated with the error signal. The experiments consisted of three stages: 0 Measurement of baseline performance in all three tasks. 0

Training in either the vertical or horizontal tasks.

Models of Perceptual Learning in Vernier Hyperacuity 0

713

Testing in the two tasks for which there was no training.

The diagonal task was added only after the first two subjects had completed the experiment, and they were called back for an additional block of testing. Before they were tested on the diagonal cross, we assessed their performance on the horizontal cross to determine whether they re-’ tained their learning despite the elapsed time (10 days for observer YK and 45 days for observer FL). 4.2 Results. Results are shown in Table 1. Because of the extensive coverage of the range of offsets (all offsets smaller than 20 pixels were presented in each block), we were able to plot the observers’ psychometric curves in each block. For some observers (see Fig. 12), these curves assumed the usual sigmoid shape only after training. For this reason, we measured the observers’ performance by the percentage of correct

VERTICAL CROSS

HORIZONTAL CROSS

DIAGONAL CROSS

Figure 11: The stimuli used in the experimentswere similar to those seen above. In each block observers were shown vertical, horizontal, or diagonal crosses with randomly varied offsets.


714

100

80

f

-

+-

.

60

40

-

1

2o0 0

5

10 OFFSETS

15

20

Figure 12: Representative psychometric curves (observer FL) before and after learning in the horizontal task. The response curves start to resemble a sigmoid only after training. responses in a range of offsets kept constant throughout training, rather than by a threshold estimated via probit analysis. As in previous experiments on perceptual learning, individual differences in the learning rates could be observed. When learning did occur for the attended part of the stimulus, it was accompanied by a significant improvement for the part that was present throughout training but was uncorrelated with the feedback. No such concomitant improvement was found after training in the diagonal test stimulus, which was not present in training (but note that only the results of observer RM in the diagonal tasks serve as a true control, as the others were either not tested for baseline or did not learn at all). Observer FL apparently did not retain his performance after a prolonged time break, while observer YK showed a much smaller decrease in performance. Observer YA explained that when tested on the vertical task (following training on the horizontal task) she tried to find a hidden cue in the horizontal verniers displayed simultaneously with the vertical verniers. Since these verniers were uncorrelated with the correct response, this explains the deterioration of her performance on the vertical task.


715

Table 1: The Percentage of Correct Responses in Each Task before and after Training? Subject

Horizontal Before

Vertical

After

Before

After

Diagonal Before

After

-

58 f 1.5 75 f 1.2

Vertical training

FL YK

61 f 1.8 83 f 0.64 (67) 59 f 1.2 75 f0.85 78 f 1.33 91 f 0.5 (88) 67 f 1.7 77 f 1.1

-

Horizontal training YA RM

AS

85 f 1.0 70 f 1.6 81 1.2

*

87 f 0.7 78 f 1.1 83 f 0.9

76 f 1.4 71 f 1.3 75 f 1.5 76 f 1.1 75 f 1.4 84 f 0.8 86 f 0.9 83 f 0.9 73 f 1.5 74 f 1.2 70 f 1.6 60 f 1.5

*An improvement in the horizontal task is accompanied by an improvement in the vertical task, and a lack of improvement in the horizontal task is accompanied by a corresponding lack of learning in the vertical task. The numbers in parentheses are the performance of subjects who were called back for additional testing after a significant time break (45 days for observer FL. and 10 days for observer YK).

4.3 Discussion. These results are consistent with a use-dependent learning rule such as EDL. Note that this rule still predicts that learning will be stimulus-specific and will not transfer to new tasks, but it distinguishes between two notions of novelty: 1. A task is new if an appropriate response function cannot be interpolated from that of familiar examples;

2. A task is new if the units used to compute the response function were not sigruficantly active during familiarization or training. In some cases (Fiorentini and Berardi 1981; Fahle and Edelman 19921, both definitions of novelty apply to the stimuli used to assess transfer of training, and the lack of transfer to these stimuli can be accounted for by models that involve either use-dependent rules or feedback-dependent ones, or both. A further indication that use-dependent synaptic modification may be involved in perceptual learning, has been reported recently by Karni and Sagi (1991). In their experiments, subjects performed letter discrimination followed by texture discrimination in the same complex stimulus. Their resulfs show significant learning in the texture task, even though feedback was given only for the letter discrimination.

716

Y. Weiss, S.Edelman, and M. Fahle

5 Conclusion

The central assumption of the HyperBF approach to the modeling of perceptual function is that the human ability to solve a variety of different perceptual tasks is based on the acquisition of specific input-output examples and on subsequent optimization of the use of the stored examples with practice. Rationale for this twofold assumption has been provided by the results of simulated psychophysical experiments (Poggio et al. 1992a) that demonstrated that a HyperBF model can learn to solve spatial discrimination tasks with hyperacuity precision, starting from a “tabula rasa” state and continuously improving its performance with repeated exposure to the stimuli. In the present paper. we concentrated on two computational details of the HyperBF model of vernier acuity. First, we investigated the possibility that oriented spatial filters known to exist in the primate visual system (namely, units similar to the simple cells of Hubel and Wiesel 1962) can serve as the basis functions in a HyperBF network. Second, we explored the different mechanisms available within the HyperBF framework for incremental learning at the level of the linear combination of basis function activities. Our findings indicate that a simple feedback-independentrule for synaptic modification, that we called EDL, for exposuredependent learning, may be involved in the improvement of the performance of human subjects with practice. Both our simulations and our psychophysical data suggest that a significant component of learning in hyperacuity may be based on stimulusdriven feedback-independentamplification of unit responses, rather than on precise feedback-guided fine tuning within a perceptual module. We remark that the perceptual module whose prior availability is assumed by the EDL rule can either be hard-wired from birth, or synthesized in a task-driven fashion, as suggested in Poggio et al. (1992a). If one accepts the possibility that the visual system is capable of modlfying certain aspects of its functional architecture on the fly, the stimulus-driven learning can be given an alternative account in terms of acquisition of new HyperBF centers (T.Poggio, personal communication). It is not clear to us at present whether or not this possibility can be distinguished psychophysically from our account in terms of synaptic modification using existing centers and the EDL rule. The presence of the initial fast stimulus-specific component in the learning curve in hyperacuity tasks (Poggio et al. 1992b) is consistent with the module synthesis view. The record of the Iast two and a half millenia indicates, however. that the Platonic notion of innate ideas (corresponding,in the present case, to innate perceptual mechanisms tuned by experience) is sufficiently resilient to cope with mere circumstantial evidence to the contrary. It remains to be seen whether a more direct approach, possibly combining physiology with psychophysics and computational modeling, will be more successful in elucidating the nature of perceptual learning.


717

Acknowledgments We thank T. Poggio for stimulating discussions, and two anonymous reviewers for useful and detailed suggestions. Y. W. was supported by the Karen Kupcinet Fund, and by a grant to S. E. from the Basic Research Foundation, administered by the Israel Academy of Sciences and Humanities.

References Barlow, H. B. 1979. Reconstructing the visual image in space and time. Nature (London) 279,189-190. Barto, A. 1989. From chemotaxis to cooperativity: Abstract exercises in neuronal learning strategies. In The Computing Neuron, R. Durbin, C. Miall, and G. Mitchison, pp. 73-98. Addison-Wesley, New York. Brown, T. H., Kairiss, E. W., and Keenan, C. L. 1990. Hebbian synapses: Biophysical mechanisms and algorithms. Annu. Rev. Neurosci. 13,475-511. Crick, F. H. C., Man; D. C., and Poggio, T. 1981. An information-processinga p proach to understanding the visual cortex. In The Organization of the Cerebral Cortex, F. Schmitt,ed., pp. 505-533. MIT Press, Cambridge, MA. Fahle, M. W., and Edelman, S. 1993. Long-term learning in Vernier acuity: Effects of stimulus orientation, range, and of feedback. Vision Res. 33, 397412. Fendick, M., and Westheimer, G. 1983. Effects of practice and the separation of test targets on foveal and perifoveal hyperacuity. Vision Res. 23,145-150. Fiorentini, A.,and Berardi, N. 1981. Perceptual learning specific for orientation and spatial frequency. Nature (London) 287,453-454. Hawkins, R. D., Abrams, T. W., Carew, T. J., and Kandel, E. R. 1983. A cellular mechanism of classical conditioning in Aplysia: Activity-dependent amplification of presynaptic facilitation. Science 219,400-404. Hubel, D. H., and Wiesel, T. N. 1962. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J.Physiol. 160, 106-154. Karni, A., and Sagi, D. 1991. Where practice makes perfect in texture discrimination. Proc. Natl. A d . Sci. U.S.A. 88,4966-4970. Klein, S. A., and Levi, D. M. 1985. Hyperacuity thresholds of 1 sec: theoretical predictions and empirical validation. J. Opt. SOC.Am. A2, 1170-1190. McKee, S. P., and Westheimer, G. 1978. Improvement in vernier acuity with practice. Percept. Psychophys. 24,258-262. Mel, B. W., and Koch, C. 1990. Sigma-Pi learning: On radial basis functions and cortical associative learning. In Neural Znformution Processing Systems, D. Touretzky, ed., Vol. 2, pp. 474-481. Morgan Kaufmann, San Mateo, CA. Parker, A. J., and Hawken, M. J. 1985. Capabilities of monkey cortical cells in spatial resolution tasks. J. Optical SOC.Am. 2, 1101-1114. Parker, A. J.,and Hawken, M. J. 1987. Hyperacuity and the visual cortex. Nature (London) 326,105-106.

718


Poggio, T., Edelman, S., and Fahle, M. 1992a. Learning of visual modules from examples: A framework for understanding adaptive visual performance. Comput. Vision, Graphics, Image Process.: Image Understanding 56, 22-30. Poggio, T., Fahle, M., and Edelman, S.1992b. Fast perceptual learning in visual hyperacuity. Science 256, 1018-1021. Poggio, T.,and Girosi, F. 1990. Regularization algorithms for learning that are equivalent to multilayer networks. Science 247,978-982. Shapley, R.,and Victor, J. 1986. Hyperacuity in cat retinal ganglion cells. Science 231,999-1002. Snippe, H. P., and Koenderink, J. J. 1992. Discrimination thresholds for channelcoded systems. B i d . Cybern. 66, 543-551. Swindale, N. V., and Cynader, M. S. 1986. Vernier acuity of neurones in cat visual cortex. Nature (London) 319,591-593. Walk, R. D. 1978. Perceptual learning. In Handbook of Perception, E. C. Carterette and M. P. Friedman, eds., Vol. IX, pp. 257-298. Academic Press, New York. Watt, R. J., and Morgan, M. J. 1985. A theory of primitive spatial code in human vision. Vision Res. 25, 1661-1674. Westheimer, G. 1981. Visual hyperacuity. Prog. Sensory Physiol. 1, 1-37. Westheimer, G., and McKee, S. P. 1975. Visual acuity in the presence of retinal image motion. 1.Optical Soc. Am. 65,847-850. Westheimer, G., and McKee, S. P. 1977. Spatial configurations for visual hyperacuity. Vision Res. 17,941-947. Widrow, B., and Steams, S. D. 1985. Adaptive Signal Processing. Prentice Hall, Englewood Cliffs, NJ. Wilson, H. R. 1986. Responses of spatial mechanisms can explain hyperacuity. Vision Res. 26,453-469. Wilson, H. R., and Bergen, J. R. 1979. A four mechanism model for threshold spatial vision. Vision Res. 19,19-32. Wilson, H. R., and Gelb, D. J. 1984. Modified line-element theory for spatial frequency and width discrimination. J . Optical SOC.Am. 1,124-131.

Received 24 June 1992; accepted 1 February 1993,

This article has been cited by: 2. Yuka Sasaki, Jose E. Nanez, Takeo Watanabe. 2010. Advances in visual perceptual learning and plasticity. Nature Reviews Neuroscience 11:1, 53-60. [CrossRef] 3. Barbara Anne Dosher, Zhong-Lin Lu. 2009. Hebbian reweighting on stable representations in perceptual learning. Learning & Perception 1:1, 37-58. [CrossRef] 4. Alexander A. Petrov, Barbara Anne Dosher, Zhong-Lin Lu. 2005. The Dynamics of Perceptual Learning: An Incremental Reweighting Model. Psychological Review 112:4, 715-743. [CrossRef] 5. Misha Tsodyks, Charles Gilbert. 2004. Neural networks and perceptual learning. Nature 431:7010, 775-781. [CrossRef] 6. Osamu Hoshino. 2004. Neuronal Bases of Perceptual Learning Revealed by a Synaptic Balance SchemeNeuronal Bases of Perceptual Learning Revealed by a Synaptic Balance Scheme. Neural Computation 16:3, 563-594. [Abstract] [PDF] [PDF Plus] 7. Jason M. Gold, Allison B. Sekuler, Partrick J. Bennett. 2004. Characterizing perceptual learning with external noise. Cognitive Science 28:2, 167-207. [CrossRef] 8. Emre Özgen, Ian R. L. Davies. 2002. Acquisition of categorical color perception: A perceptual learning approach to the linguistic relativity hypothesis. Journal of Experimental Psychology: General 131:4, 477-493. [CrossRef] 9. Anna Schubö, Friederike Schlaghecken, Cristina Meinecke. 2001. Learning to ignore the mask in texture segmentation tasks. Journal of Experimental Psychology: Human Perception and Performance 27:4, 919-931. [CrossRef] 10. Rosario M. Balboa , Norberto M. Grzywacz . 2000. The Minimal Local-Asperity Hypothesis of Early Retinal Lateral InhibitionThe Minimal Local-Asperity Hypothesis of Early Retinal Lateral Inhibition. Neural Computation 12:7, 1485-1517. [Abstract] [PDF] [PDF Plus] 11. G. Mato , H. Sompolinsky . 1996. Neural Network Models of Perceptual Learning of Angle DiscriminationNeural Network Models of Perceptual Learning of Angle Discrimination. Neural Computation 8:2, 270-299. [Abstract] [PDF] [PDF Plus] 12. Shimon Edelman. 1995. Representation, similarity, and the chorus of prototypes. Minds and Machines 5:1, 45-68. [CrossRef] 13. Wolfgang Skrandies, Manfred Fahle. 1994. Neurophysiological correlates of perceptual learning in the human brain. Brain Topography 7:2, 163-168. [CrossRef]

Communicated by James Anderson

Learning to Generalize from Single Examples in the Dynamic Link Architecture Wolfgang Konen Christoph von der Malsburg Institut f i r Neuroinfomatik, Ruhr-Universitat Bochum, Germany A large attraction of neural systems lies in their promise of replacing programming by learning. A problem with many current neural models is that with realistically large input patterns learning time explodes. This is a problem inherent in a notion of learning that is based almost entirely on statistical estimation. We propose here a different learning style where significant relations in the input pattern are recognized and expressed by the unsupervised self-organization of dynamic links. The power of this mechanism is due to the very general a priori principle of conservation of topological structure. We demonstrate that style with a system that learns to classify mirror symmetric pixel patterns from single examples.

1 Introduction Learning is the ability of animals and humans to absorb structure from one scene and apply it to others. The literal storage of whole sensory input fields is of little value since scenes never recur in all detail within our lifetime. Essential for learning is therefore the ability to extract significant patterns from an input field containing mostly patterns with accidental feature constellations, and to apply those significant patterns to the interpretation of later scenes. How can significant patterns be identified? Theories of learning based on layered neural networks [e.g., backpropagation of errors (Rosenblatt 1962,Rumelhart et al. 1986) or the Boltzmann Machine (Ackley et al. 198511 are based on the notion that significant patterns are, above all, recurring patterns. Such systems have an input layer, an output layer, and hidden units. During a learning phase, many examples are presented to input layer and output layer, and the system is enabled by some plasticity mechanism to pick up and represent patterns that recur with statistical sigruficance in the input training set. This method of identifying significant patterns may be the obvious one-going back to the original definition of significance based on recurrencebut with realistic inputs taken from natural environments it is far too costly, in terms of the number of inputs required to discriminate significantpatterns from accidental Neural Computation 5,7l9-735 (1993) @ 1993 Massachusetts Institute of Technology

720

Wolfgang Konen and Christoph von der Malsburg

Figure 1: Symmetrical pixel patterns. Input patterns are arrays of N x N pixels, here N = 8. Pixel II has gray level feature value Fo E { 1, . . . ,Fmx}.In most of our simulations,F,,, = 10. In each input image, pixel values are random, but equal for points symmetrical with respect to one of three axes: (A) horizontal, (8) vertical, (C) diagonal. The system has to solve the task of assigning input patterns to classes according to these symmetries, and to learn this performance from examples. patterns. The reason for this difficulty lies in the combinatorial explosion in the number of subsets that can be selected from large input fields (there are, for instance, ld432possible subsets of size 1000 in a set of 109. Among those subsets there are only relatively few of significant interest (in vision, for example, the criterion of spatial continuity alone singles out relatively very few subsets). There obviously are potent methods, presumably based on a priori knowledge built into the system, to extract significant patterns from a scene. It is generally recognized that methods based purely on scene statistics must be complemented (if not supplanted) by more powerful ones based on a priori structure. One widespread piece of advice is to use input representations that are already adapted to the problem at hand. Down that alley there is, of course, the pitfall of hand-wiring instead of learning the essential structure. The real challenge is to find simple and general architectures that can handle large classes of problems and that can learn with a minimum of scene statistics. The particular problem we are considering here has originally been proposed by Sejnowski et al. (1986). It consists in learning to classify mirror-symmetrical pixel patterns (see Fig. 1). The authors solved the problem with the help of scene statistics. Their system, consisting of a layer of 12 hidden units and 3 output units corresponding to the 3 symmetry classes, learned as a Boltzmann Machine, which is a variant of supervised learning. With input arrays of 10 x 10 pixels the system

Learning to Generalize

721

needed about 40,000 training examples in order to reach a success level of 85%. The system (Sejnowski et al. 1986) demonstrates the strength and the weakness of statistical pattern classification. The strength is full generality with respect to possible patterns. This is best demonstrated with the thought experiment of applying a permutation to the pixels in the input field-the same permutation to all patterns presented. The system would now have the same learning performance, in spite of the complete destruction of topological structure. The weakness is an explosion in the number of examples required when scaling to larger input arrays. This weakness the system shares with a wide class of learning algorithms, which all are based on the statistical detection of classes as clusters in input space and their subsequent representation by single prototypes. Prominent examples are the k-nearest neighbor (k") algorithm (Fix and Hodges 1951; Cover and Hart 1967), the RCE algorithm (Reilly et al. 1982), which is a neural version of k",and adaptive vector quantization (LVQ, LVQ2) (Kohonen et al. 1988). None of those algorithms can easily deal with the symmetry classification problem. The reason is that already with modest problem size there are astronomicalIy many patterns in a symmetry class for the 8 x 8 pixels of 10 features each in Fig. 1) and that these do not form clusters in input space and thus cannot be detected in a small training set. It is this that leads to the exploding thirst in learning time and number of prototypes. Our treatment of the problem is based on the Dynamic Link Architecture (DLA) (von der Malsburg 1981). The strength of the DLA essential in the present context is its ability to detect pattern correspondences. An application of this ability to the problem of invariant pattern recognition has been reported previously (Bienenstock and von der Malsburg 1987; von der Malsburg 1988; Lades et al. 1993). Here we demonstrate that with it symmetry classes can be recorded from single examples for later recognition. Our treatment is based on the a priori restriction that significant relations within the input pattern are those which preserve topological structure. It is in this sense less general than the Boltzmann Machine, not being able to deal with the permutation symmetries mentioned above. On the other hand its extreme speed of adaptation to new symmetries makes it more potent than the Boltzmann Machine. Most of what is achieved in other neural systems with the help of statistical learning is performed here by the self-organization of an explicit representation of the symmetry in the input pattern. 2 Symmetry Detection by Dynamic Link Matching-Qualitative Model

Dynamic link matching is capable of finding and representing topological, feature-preserving mappings between parts of the input plane.

722


Such mappings are systems of pair-wise links that are neighborhoodpreserving and that connect pairs of points with the same local properties in the input pattern. In this section we describe the network and its function qualitatively and establish its relationships to other, previously published models (von der Malsburg 1988; Bienenstock and von der Malsburg 1987; Buhmann et al. 1989; Lades et al. 1993; von der Malsburg and Buhmann 1992), and to the circuitry of cortical structures. In the next section we will describe an explicit quantitative, though somewhat simplified, model. The network resembles primary visual cortex in representing visual images in a columnar fashion: Each resolution unit (“pixel“) of the sensory surface is subserved by a collection (“column”) of neurons, each neuron reacting to a different local feature. (In our concrete model, local features will simply be gray values. In a more realistic version, features would refer to texture, color, and the like.) There are intracolumnar connections, whose function will be explained below, and intercolumnar connections. The latter are what we will refer to as “dynamic links,” are of rather large range in visual space, and are restricted to pairs of neurons with the same feature type. (In our explicit model the connections will run between cells responding to the same gray value in the image.) When a pattern is presented as visual input, those neurons in a column are selected that code for a feature that is present in the corresponding pixel. We refer to the selected cells as “preactivated” neurons. The set of all preactivated neurons represents the input image. During the presentation of an image, the preactivated cells are actually not allowed to fire all at the same time. Rather, activity in the network takes the form of a sequence of “blob activations.” During a blob activation, only those preactivated neurons are permitted to fire that lie in a group of neighboring columns. A blob activation corresponds to the “flash of the searchlight of focal attention” discussed, for instance, by Crick (Crick 1984). In the absence of any other control of attention, blob activations are created spontaneously in random positions in a rapid sequence of “cycles.” When a blob is active, its active cells send out signals that excite preactivated neurons of the same feature type in other locations. Thus, within the total network those preactivated neurons are excited whose type is represented in the active blob. Most of these cells form a diffuse spray over the image domain. If there is a symmetry in the image, however, there will be a location where all the feature types in the active blob are assembled locally again. With appropriate dynamics, those neurons are activated as well, forming a ”shadow blob.” The network thus has discovered the significant relationship between two symmetrical regions in the image, and with the help of rapid synaptic plasticity in the intercolumnar connections (“dynamic links”) it is possible to record it, simply strengthening the synaptic connections between all pairs of neurons lying one in each blob. During a sequence of many blob pairs, a full consis-


723

tent system of point-to-point connections will get established, forming a topological mapping between the symmetric parts of the image. This sequence of events constitutes the dynamic link mapping mechanism. It is very robust. Occasional erroneous blob pairs are of little consequence whereas all correct blob pairs form a cooperative system of mutual reinforcement. Once the covering of the image with blobs is fairly complete the plexus of reinforced connections stabilizes signal correlations between symmetric points and, as our simulations show, false blob pairs do no longer occur. For each new image (or for each new fixation of an image, for that matter), a new mapping of dynamic links has to be built up. A slow, and simpler, version of the dynamic link mapping mechanism was first described in Willshaw and von der Malsburg (1976) to account for the ontogenetic establishment of retinotopic mappings from retina to tectum. A dynamic link mapping system using feature labels has later been proposed as a solution to the problem of invariant object recognition (von der Malsburg 1988; Bienenstock and von der Malsburg 1987; Buhmann et al. 1989; Lades et al. 1993). As a mapping system, the present model goes beyond previous work in needing dramatically fewer activation cycles. The columnar connectivity pattern described here was introduced as part of a proposed solution to the figure ground segmentation problem (Schneider 1986; von der Malsburg and Buhmann 1992). In the explicit model described below some network details are just necessary to realize the qualitative behavior described above. Others, however, we introduced to simplify the dynamics of our system. Prominent among these is the introduction of an “activator cell” (or X-cell) and a ”collector cell” (or Y-cell) for each column (see Fig. 2A). The activator cells spontaneously create the active blob and activate all sensorily preactivated neurons in their column. The collector cells sum up all activity that arrives in the preactivated neurons of their column and that comes from the active blob, and they interact to form the shadow blob. Also, active collector cells gate the preactivated neurons in their columns into full activity. The presence of activator cells and collector cells ensures that all preactivated neurons in a column make their firing decision together. Global inhibition between all activator cells and between all collector cells ensures that there is exactly one active blob and exactly one shadow blob at any one time. An activator cell is kept by a compensating inhibitory connection from exciting the collector cell of its own column via its feature cells. In our explicitly simulated network described below we make the simpllfying assumption that during the presentation of an image exactly one of the feature cells in a column is active (corresponding to one of a number of possible gray values). As a consequence, at most one intercolumnar connection is active between two columns at any one time (exactly when the two columns are preactivated with the same gray value). This justifies our introduction of “compound connections” from the ac-

724


layer Y feature cells layer X pattern

Figure 2: Architecture of the dynamic link network. (A) The complete architecture. The columns in two positions, a and b are shown. Feature cells are preactivated by the pattern presented. Columns are connected with each other by feature-preservinglinks. These links are rapidly modifiable ("dynamic links"). Both the activator cells (layer X)and the collector cells (layer Y) have short-range excitatory and long-range inhibitory connections (not shown) and each have the tendency to form a local blob. Coupling from an X-cell a to a Y-cell b is via the preactivated cells in column a, intercolumnar links, and the preactivated cells in column b. (B)In our case, where only one feature is active per column, a functionally equivalent description uses the effective connections JhTh, where T h encodes the feature similarity between image positions u and b (cf. equation 3.1), and ]h is the rapidly modifiable strength of the dynamic link. tivator cells to the collector cells, treating all columnar quality cells and their connections implicitly (see Fig. 28). 3 Symmetry Detection by Dynamic Link Matching-Explicit Model

After these preparatory heuristic discussions we are ready to introduce the explicit dynamic link mapping network that we have simulated. It has the following parts (cf. Fig. 2B). Our image domain is formed by a grid of 8 x 8 pixels. Positions in the image domain are designated by letters u,b, . . . . An input image is described by a distribution of features F, over the pixel positions a, where Fa E (1,. . . ,F,,,} (see Fig. 1). The image domain is covered by two layers of cells, the X-layer and the Y-layer. The


725

connection from cell a in the X-layer to cell b in the Y-layer is controlled by the dynamic link variable Jh, which is subject to the dynamic described below. The constraint of feature specificity is formulated with the help of the similarity constraint matrix ~ b = a

{

if F, else

1 0

= Fb

and b # a

(3.1)

The total connection from cell a in the X-layer to cell y in the Y-layer is described by the “effective coupling” JhTb. The activities of cells are designated xa or ya. Both layers have homogeneous internal connections of the form &at

= Gaal

-P

(3.2)

Here, G, is a short-range excitatory connection kernel, and P is the strength of a long-range (here: global) inhibitory connection. For both X and Y we assume wrap-around boundary conditions. The dynamic of the X-layer is governed by the differential equations (3.3)

Here, S ( x ) is a sigmoidal nonlinearity that saturates at S ( x ) = 0 for low x and at S ( x ) = 1 for high x, whereas p is a constant excitation. The dynamic of the Y-layer is governed by the differential equations (3.4)

With given effective connections and small noisy initial values (as a model for spontaneous activity) for the xa, the activator and collector cell activities evolve on a fast time scale towards an equilibrium distribution in the form of local blobs of active cells ( S x 11, with the rest of the cells in the layer X or Y inactive ( S M 0). The size of the blob is controlled by the parameters a, P, and o,whereas their position is determined by the noise input in the case of X and by the profile of the activation in the case of Y. Once the activity in X and Y has settled, the dynamic link variables Jba are modified by three consecutive substitutions: (3.5)

The first step encapsulates the general idea of Hebbian plasticity, though regulated here by the constant E for the rapid time scale of a single image presentation. After the second and third steps the new connections conform to divergent and convergent sum rules.

726


When an image is presented, the full sequence of events is the following. First, the connections Ih are initialized with a constant value conforming to the sum rules. Then a number of activity-and-modificationcycles are carried through. For each of these, the X-activities are initialized with noise distribution, the Y-activities are reset to 0, and the dynamics of X and Yare run according to equations 3.3 and 3.4 to reach stationary values. Then the dynamic links are updated according to equation 3.5. After typically 50-80 such cycles the dynamic links relax into a stable configuration displaying the underlying symmetry of the actual input image. For a typical result see Figure 4. The network is now ready for permanently recording the symmetry type if it is new, or for recognizing it according to a previously recorded type. If a link Jh is active, the activity dynamics of equations 3.3 and 3.4 produces correlated activity in the connected cells: In the stationary state towards the end of each cycle, cells a and b are always active or inactive together. In comparison to the dynamic links, activity correlations have the distinction of graceful degradation: Even if a single link is corrupted, the correlation between the corresponding x and y cells is high if there are strong links in the neighborhood (remember that an activity blob always covers a neighborhood along with a given cell).

4 Recording and Recognizing a Symmetry

The main task necessary for solving the symmetry recognition problem is solved for our model by the unsupervised process of dynamic link mapping described in the last section. For a given symmetric pattern it constructs a temporary representation in the form of a set of active links. This set is the same for all input patterns belonging to the same symmetry class. In order to record a symmetry type it is now simply necessary to create hidden units as permanent representatives for some of the active links (or rather the correlations created by them) and to connect them to appropriate output units. Once a symmetry type has been represented by such a network, its second occurrence can be detected and the system is ready to recognize all patterns of this symmetry type as such. O u r recognition network structure is similar to the one used in Sejnowski et al. (1986)and is shown in the upper panel of Figure 3. It consists of three output units Ck,k = 1 , 2 , 3 (sufficient for three symmetry types) and, connected to each output unit, 6 hidden units. Each hidden unit i has a randomly chosen fixed reference cell a ( i ) in X and plastic synapses Wjb from all cells b in Y.' The output hi of a hidden unit is driven by a 'In principle, the number of hidden units per output cell could be one. Recognition is more reliable and faster, however, if the density of reference cells a ( i ) is large enough so that most of the active blobs in X hit at least one of them.

earning to eneralize

72

Figure 3: The complete system. An input pattern (lowest layer) is represented by sets of preactivated neurons in the feature columns (marked here by heavy outline, on the basis of gray levels). Columns are connected by featurepreserving dynamic links (intercolumnararrows). The dynamic link mechanism creates simultaneous blobs in layers X (“active blob) and Y (“shadow blob”) in symmetrically corresponding positions (hatched areas). The symmetry type is then recorded (when it is new) or recognized (when already known) in the classification network (upper part). There are six hidden units per output unit (only four of which are shown). Each hidden unit has one fixed connection to its output unit, one connection from a randomly chosen X-cell, and has plastic onnections wib from all Y-cells. These latter connections record a symmetry type permanently, by being concentrated into the Y-region lying symmetrically o the location of the X-in ut of the hidden unit.


728

coincidence of activity xa(i) of its reference cell in X and activity within its receptive field Wib in Y (4.1) In recording mode, hidden units modify their Y-connections Wib at the end of each activity cycle according to the Hebbian plasticity rule: AWjb=QS(yb)

if hi > 8 and

c k

>0

(4.2)

Synaptic plasticity is supervised in the sense that only those hidden units mod* their connections whose output unit Ck is currently activated by a teacher (the role of the teacher simply being to fixate attention on one group of hidden units during the presentation of one pattern). In this way, a hidden unit whose X connection is hit by a blob learns to associate with it the corresponding blob in the Y plane. The whole process is completed for a symmetry type during one presentation (or in some cases two presentations, see below). In recognition mode, the output units perform a leaky integration of the sum of the activities (equation 4.1) of their group of hidden units. After a number of cycles, the output unit with maximal signal is taken to indicate the class into which the input pattern falls. 5 Simulation Results

Simulations of the model were carried out for input patterns of size 8 x 8. The parameters for the blob formation process in equations 3.3 and 3.4 were adjusted to let the equilibrium blobs cover between 25 and 40% of the layer area; for example, with {o,p, e, Q , C , p , 8) = { .3, .85,1.8,.02, .8, .6, ,125) blobs cover 25% of their layer. As convolution kernel G , ! we used a gaussian of width 4 and strength 2.1, restricted, however, to a window of 5 x 5 pixels. For almost all input patterns, self-organization of the correct mapping J from X to Y was observed. Figure 4 shows a typical example in some stages of the organization process. The degree of organization can be measured quantitatively by the correlation between corresponding cells, which is shown in Figure 5 for a specific input example. During the first 40-50 activation cycles the correlation builds up and reaches almost the theoretical optimum 1. Thus, during all further cycles symmetrically corresponding points in X and Y are marked by strong correlations, whereas pairs of units in far-from-symmetrical positions would have correlation -1. After learning the specific symmetries from either one or two training examples, the network can generalize almost perfectly to new input patterns of the same symmetry class. Figure 6A shows the classification performance on 200 new examples. There is a clear tradeoff between the reliability of recognition and the required time (in terms of activation


729

Figure 4 Dynamic link mapping. The network, with layers X (in front) and Y (in the rear) in different activation states, after 15 (A), 50 (B), and 80 (C)activity cycles, all generated for a fixed input pattern of symmetry class A (cf. Fig. 1). The dynamic link mapping process is based on a sequence of blob activations (white circles). Dynamic links Jh E [0,1]grow between temporally correlated cells. Only links with jtyl 2 0.4 are shown in the figure. cycles). In principle, one example can supply sufficient information for this performance. However, with our parameter settings two examples gave slightly more reliable results (see Fig. 6A). Our network achieves a recognition reliability of 98%. Its level of reliability is only weakly affected by perturbations to the feature similarity matrix T up to t = 40% (Fig. 6B). This is due to the robustness of the dynamic link mapping mechanism (see Fig. 5), which creates near-toperfect signal correlations between symmetric points. Since the hidden units are trained by these correlations, the presence of perturbations in the matrix T even during learning does not affect the performance of the system. We have verified numerically that after training the hidden units with t = 40% the performance is virtually the same as in Figure 6B, for example, 93% reliability if the recognition is forced after 100 cycles. 6 Discussion

We have presented here a network that is able to discover a system of relations in individual input patterns and to immediately generalize to


730

1.o

0.8 0.6

no perturbation (t=O) 0 : 40% perturbation (t=0.4) a:

0.4

0.2 0.0 0

1 20 40 60 80 100 120 number n of activation cycles

Figure 5: Mean correlation between pairs of corresponding cells in layer X and layer Y for a given state of the dynamic links J after n activation cycles (blob pairs). Correlation is computed as

with Ax = d ( ( x - ( x ) ) ~ ) and , s(a) denoting the the cell that lies symmetrically to a. To measure the correlation after n activation cycles, the link state {Ibp}is frozen after n cycles (by setting E = 01, while the blob activation cycles continue. x, and ys(a)are the equilibrium activities of cells a and s(a), respectively, and (.) denotes averaging over many cycles. Possible correlation values range from -1 for perfect anticorrelation to 1 for perfect correlation. What is displayed is the mean of C ( X # , ~ ~with ( ~ ) )respect to all possible positions a, and error bars denote the statistical errors when averaging over 900 cycles. Filled circles: Perfect feature similarity matrix Ttyl E {O,l}. Open circles: All matches Ttyl = 1 are replaced by random values Ttyl E [l - t,l], all nonmatches Ttyl = 0 by a random Tbp E [O,t], to mimic the effects of noisy feature information. The correlations are robust against this perturbation.

further examples of the same type. The network is based on dynamic link mapping. The self-organization of dynamic links in our model is extremely fast, requiring much less than 100 update cycles. This is due to the use of local feature information in conjunction with a topology constraint. For simplicity, we have used gray-values of single pixels as


731

2 examples per class

60

401

I

1 example per dass

20

ot

0

. . . , . . . , . . . , . . . , . . .1 20 40 60 80 100 number n of activation cycles

0

I

nopefhJfmion(bO)

M% perturbation(W.2)

+ 40% perturbation(tro.4)

ot 0

. . . , . . . , . . . , . . . , . . . 1 20 40 60 80 100 number n of activation cycles

(B)

Figure 6 Symmetry recognition performance. A total of 200 input patterns are classified according to one of three possible symmetries (cf. Fig. 1). The symmetry types have been recorded previously. The percentage of correct decisions is displayed as a function of the number n of activation cycles until the decision is forced. (A) Unperturbed features, T h E (0, l}, training with k = 1 or k = 2 examples per class, 120/k learning steps according to equation 4.2 for each example. (8) Influence on performance of perturbations in the feature similarity matrix T during recognition: The network can tolerate perturbations of t = 20% or even t = 40%, where t is defined as in Figure 5.

732


our visual features. In applications to large pixel arrays this would be impractical. The number of dynamic links in the matrix J would have to grow with the fourth power of the linear extent of the input plane. However, if one replaced the gray-level sensitivity of our feature cells by extended receptive fields [e.g., of the Laplace type with a hierarchy of spatial scales, in analogy to the feature jets of (Buhmann et a2. 1991)l one could cover the input plane with a fairly low-density set of sampling points and correspondingly operate with manageably small X and Y planes. The main goal of our paper is to make a point about the learning issue, symmetry detection merely playing the role of an example. It may be interesting, though, to briefly discuss symmetry detection in humans and animals. Infants can detect symmetry at the age of four months (Bornstein et al. 1981). Pigeons learn to discriminate symmetry in very few trials (Delius and Nowak 1982), although one may suspect that they already come equipped with the ability of detecting symmetry and only have to be conditioned for the appropriate response. Our system may shed new light on the old discussion of nature vs. nurture with respect to the symmetry detection issue: Our demonstration that learning time could be extremely short makes it impossible to decide the issue by demonstrating the capability in naive or newborn subjects. At first sight it is tempting to take our system directly as a model for symmetry detection in primary visual cortex, identlfying all of our cell types (Xand Y cells, feature cells and hidden units) with neurons found in cortical hypercolumns in area V1. This view would run, however, into a number of difficulties. One of them is the need, in our model, for long-range connections (intercolumnar links and the wib connections from Y cells to hidden units). With respect to area V1 this requirement creates a conflict, especially in view of the fact that humans are better at detecting symmetry around the vertical midline than around the horizontal (Barlow and Reeves 1979), and callosal connections are absent within V1 except for a narrow seam along the vertical meridian. [This difficulty is mitigated, though, by the fact that symmetry detection in humans relies mainly on a small strip along the symmetry axis of the input pattern, at least in random dot patterns (Julesz 19751.1 The problem can be largely avoided by placing our model in a later area in which larger visual angles are spanned by horizontal fibers. Another hint to this effect is the observation that symmetry detection in humans may be based not on distributions of gray levels directly but rather on object shapes reconstructed from shading (Ramachandran 1988). Another difficulty for a direct biological application of our model [which it shares with the one of Sejnowski et al. (1986)l is its lack of invariance with respect to displacement of the symmetry axis, as for instance caused by eye movements during inspection of a pattern. All of these difficulties point to a slightly more complicated model, which would, however, obscure our main point.


733

O u r system is not limited to mirror symmetry. It could equally record and recognize other pattern relations such as simple duplication with or without rotation (or, in a system of only slightly more general form, expansion). Humans, on the other hand, perform much worse on these (Corballis and Roldan 1974). The reason for such differences may be a rather superficial one. Even if the ontogeny of symmetry detection is of the nature we are putting forward here, the system will after some experience be dominated by the hidden units it has acquired. Once these have sufficient density, the dynamic link mechanism is no longer necessary for the recognition of familiar pattern relations [the same way the correct hidden units in Sejnowski et al. (1986) are activated directly by the known symmetries]. The relative performance on different problem types is then dominated by experience rather than by the nature of the ontogenetic mechanism. This would explain our bias toward vertical mirror symmetry. The heavy reliance of humans on a strip around the symmetry axis mentioned above may point to a mechanism relying on memorized symmetric shapes, such as butterfly patterns and the like, formed on the basis of a general learning mechanism but soon supplanting it by being more rapidly detected. The structure of our model fits very well the general character of cortical columnar organization [as also employed in von der Malsburg and Buhmann (1992)l. Of central importance to our system is the encoding of significant relations with the help of temporal signal correlations. Candidate correlations of an appropriate nature have been observed in visual cortex (Gray et al. 1989; Eckhorn et al. 1988). The model may thus give support to the idea that correlations play an important functional role in cortical information processing. The central point that we would like to make here refers to the general learning issue. The origin of knowledge in our mind has puzzled philosophers for centuries. Extreme empiricism is not tenable. Its most concrete formulation, nonparametric estimation, shows that it requires astronomical learning times. The opposite extreme, assuming all knowledge to be present in the brain at birth, is equally untenable, not doing justice to the flexibility of our mind, and just putting the burden of statistical estimation on evolution. The only possible way out of this dilemma is the existence of general principles simple enough to be discovered by evolution and powerful enough to make learning a very fast process. This can only be imagined in a universe with profound regularities. The one we are exploiting here is the wide-spread existence of similarities between simultaneously visible patterns. This regularity is captured in the rather simple structure of our network, enabling it to generalize from single examples of symmetrical patterns, in striking contrast to the system of Sejnowski et al. (1986), which is based on statistical estimation. With small modifications, dynamic link mapping can be used for the purpose of object recognition invariant with respect to translation, rotation and distortion, making the step from the correspondence of simultaneous

734


patterns to those of consecutive patterns. Again, those transformations could be learned from few examples. The very simple a priori principles incorporated in the learning system that we have presented are feature correspondence, topology, and rapid synaptic plasticity. We feel that it is struchiral principles of this general style that make natural brains so extremely efficient in extracting significant structure from complex scenes. Although statistical estimation certainly plays a role for animal learning, it can evidently not be its only basis-natural scenes are too complex, and it is impossible to keep track of the whole combinatorics of subpatterns. Potent mechanisms are required to identify significant patterns already within single scenes. Ours may be a candidate.

Acknowledgments Supported by a grant from the Bundesministerium fur Forschung und Technologie (413-5839-01 IN 101 B/9), and a research grant from the Human Frontier Science Program.

References Ackley, D. H., Hinton, G. E., and Sejjowski, T. S. 1985. A learning algorithm for Boltzmann machines. Cogn. Sci. 9,147-149. Barlow, H. B., and Reeves, B. C. 1979. The versatility and absolute efficiency of detecting mirror symmetry in random dot displays. Vision Res. 19,783-793. Bienenstock, E., and von der Malsburg, C. 1987. A neural network for invariant pattern recognition. Europhys. Lett. 4,121-126. Bornstein, M. H., Ferdinandsen, K., and Gross, C. G. 1981. Perception of symmetry in infancy. Dev. Psychol. 17, 82-86. BuhmaM, J., Lange, J., and von der Malsburg, C. 1989. Distortion invariant object recognition by matching hierarchically labeled graphs. In IJCNN International ConferenceonNeural Networks, Washington, pp. 155-159. IEEE, New York. Buhmann, J., Lange, J., von der Malsburg, C., Vorbriiggen, J. C., and Wiirtz, R. P. 1991. Object recognition in the dynamic link architecture-parallel implementation on a transputer network. In Neural Netwrks: A Dynamical Systems Approach to Machine Intelligence, B. Kosko, ed., pp. 121-160. Prentice Hall, New York. Coolen, A. C. C., and Kuijk, F. W. 1989. A learning mechanism for invariant pattern recognition in neural networks. Neural Networks 2,495. Corballis, M. C., and Roldan, C. E. 1974. On the perception of symmetrical and repeated patterns. Percept. Psychophys. 16, 136-142. Cover, T.M., and Hart, P. E. 1967. Nearest neighbor pattern classification. IEEE Transact. Inform. Theory IT-13,21-27.


735

Crick, F. 1984. Function of the thalamic reticular complex: The searchlight hypothesis. Proc. Natl. Acad. Sci. U.S.A. 81, 4586-4590. Delius, J. D.,and Nowak, B. 1982. Visual symmetry recognition by pigeons. Psychol. Res. 44, 199-212. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern. 60, 121. Fix, E., and Hodges, J. L. 1951. Discriminatory analysis, non-parametric discrimination. Tech. Rep. USAF School of aviation medicine, Project 21-49-004. Rept. 4. Gray, C. M., Konig, P., Engle, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature (London) 338,334-337. Julesz, 8. 1975. Experiments in the visual perception of texture. Sci. Am. 4. Kohonen, T., Barna, G., and Chrisely, R. 1988. Statistical pattern recognition with neural networks: benchmarking studies. Proceedings of the IEEE ICNN, San Diego. Lades, M., Vorbriiggen, J. C., Buhmann, J., Lange, J., von der Malsburg, C., Wurtz, R. l?, and Konen, W. 1993. Distortion invariant object recognition in the dynamic link architecture. IEEE Transact. Computers 10, 300. Ramachandran, V. 1988. Perceiving shape from shading. Sci. Am. 10,76-83. Reilly, D.L., Cooper, L. N., and Elbaum, C. 1982. A neural model for category learning. Biol. Cybern. 45, 35-41. Rosenblatt, F. 1962. Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Spartan Books, Washington, DC. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning representations by backpropagating errors. Nature (London) 323,533-536. Schneider, W. 1986. Anwendung der Korrelationstheorieder Hirnfunktion auf das akustische Figur-Hintergrund-Problem(Cocktailparty-Effekt).Ph.D. thesis, Universitat Gottingen, 3400 Gottingen, Germany. Sejnowski, T. J., Kienker, P. K., and Hinton, G. E. 1986. Learning symmetry groups with hidden units: Beyond the perceptron. Physica 22D,260-275. von der Malsburg, C. 1981. The correlation theory of brain function. Internal report, 81-2,Max-Planck-Institut fur Biophysikalische Chemie, Postfach 2841,3400 Gottingen, Germany. von der Malsburg, C. 1988. Pattern recognition by labeled graph matching. Neural Networks 1, 141-148. von der Malsburg, C., and Buhmann, J. 1992. Sensory segmentation with coupled neural oscillators. Biol. Cybern. 67, 233-242. Willshaw, D. J., and von der Malsburg, C. 1976. How patterned neural connections can be set up by self-organization. Proc. R. SOC.London B194,431-445.

Received 1 July 1992; accepted 12 February 1993.

This article has been cited by: 2. T. Aonishi, K. Kurata. 2000. Extension of dynamic link matching by introducing local linear maps. IEEE Transactions on Neural Networks 11:3, 817-822. [CrossRef] 3. Toru Aonishi, Koji Kurata. 1998. Deformation Theory of Dynamic Link MatchingDeformation Theory of Dynamic Link Matching. Neural Computation 10:3, 651-669. [Abstract] [PDF] [PDF Plus] 4. Laurenz Wiskott, Terrence Sejnowski. 1998. Constrained Optimization for Neural Map Formation: A Unifying Framework for Weight Growth and NormalizationConstrained Optimization for Neural Map Formation: A Unifying Framework for Weight Growth and Normalization. Neural Computation 10:3, 671-716. [Abstract] [PDF] [PDF Plus] 5. B. Parvin, G. Medioni. 1996. B-rep object description from multiple range views. International Journal of Computer Vision 20:1-2, 81-112. [CrossRef]

Communicated by William W. Lytton

Neural Network Modeling of Memory Deterioration in Alzheimer’s Disease D. Horn School of Physics and Astronomy, Raymond and Beverly Sackler Faculty of Exact Sciences, Tel Aviv University, Tel Aviv 69978, Israel

E. Ruppin Department of Computer Science, Raymond and Beverly Sackler Faculty of Exact Sciences, Tel Aviv University, Tel Aviv 69978, Israel

M.Usher CNS program, Division of Biology 21 6-76, Caltech, Pasadena, CA 91125 USA M.Herrmann Sektion Informatik, UniversitlCt Leipzig, PSF 920,D-0-7010 Leipzig, Germany

The clinical course of Alzheimer’s disease (AD) is generally character ized by progressive gradual deterioration, although large clinical variability exists. Motivated by the recent quantitative reports of synaptic changes in AD, we use a neural network model to investigate how the interplay between synaptic deletion and compensation determines the pattern of memory deterioration, a clinical hallmark of AD. Within the model we show that the deterioration of memory retrieval due to synaptic deletion can be much delayed by multiplying all the remaining synaptic weights by a common factor, which keeps the average input to each neuron at the same level. This parallels the experimental observation that the total synaptic area per unit volume (TSA) is initially preserved when synaptic deletion occurs. By using different dependencies of the compensatory factor on the amount of synaptic deletion one can define various compensation strategies, which can account for the observed variation in the severity and progression rate of AD. 1 Introduction Alzheimer’s disease (AD) is the major degenerative disease of the brain, responsible for a progressive deterioration of the patient’s cognitive and motor function, with a grave prognosis (Adams and Victor 1989). Its clinical course is usually characterized by gradual decay, although both slow and rapidly progressive forms have been reported, exhibitinga large Neural Computation 5,736-749 (1993) @ 1993 Massachusetts Institute of Technology

Neural Network Modeling in AD

737

variation in the rate of AD progression (Drachman et al. 1990). While remarkable progress has been gained in the investigation of neurochemical processes accompanying AD, their role in neural degeneration, the main pathological feature of AD, is yet unclear (Selkoe 1987; Kosik 1991). This work is motivated by recent investigations studying in detail the neurodegenerative changes accompanying AD, on a neuroanatomical level. Following the paradigm that cognitive processes can be accounted for on the neural level, we examine the effect of these neurodegenerative changes within the context of a neural network model. This allows us to obtain a schematic understanding of the clinical course of AD. Neuroanatomical investigations in AD patients demonstrate a considerable decrease in the synapse to neuron ratio, due to synaptic deletion (Davies et al. 1987; Bertoni-Freddari et al. 1990). Synaptic compensation, manifested by an increase of the synaptic size, was found to take place concomitantly, reflecting a functional compensatory increase of synaptic efficacy at the initial stages of the disease (Bertoni-Freddari et al. 1988, 1990; DeKosky and Scheff 1990). The combined outcome of these counteracting synaptic degenerative and compensatory processes can be evaluated by measuring the total synaptic area per unit volume (TSA), which was shown to correlate with the cognitive function of AD patients (DeKosky and Scheff 1990). Our model, presented in Section 2, serves as a framework for examining the interplay of synaptic deletion and compensation. This attractor neural network (ANN) is not supposed to represent any specific neuronal tissue, yet we believe that our results are relevant to a large class of neural systems. Deletion is carried out stochastically by removing the fraction d of all synaptic weights. Compensation is modeled by multiplying all remaining synaptic weights by a common factor c. The TSA value is proportional to c(1 - d ) . Varying c as a function of d specifies a compensation strategy. We assume that the network’s failure rate, measured by the fraction of memories erroneously retrieved, represents the degree of “cognitive deficit” in clinical observations. Reviewing the pertaining pathological and clinical data, we show in Section 3 how our model can account for the variability observed in the clinical course of AD. Our results are further discussed in Section 4. 2

The Model

Concentrating on memory degradation, a clinical hallmark of AD (Adams and Victor 19891, we use as our theoretical framework a neural network model of associative memory. Our model is based on the biologically motivated variant of Hopfield’s model (1982), proposed by Tsodyks and Feigelman (1988). In an ANN,the stored memories are attractors of the network’s dynamics, such that when memory retrieval is modeled then, starting from an initial condition sufficiently similar to one of the memory

D.Horn et al.

738

patterns, the network flows to a stable state identical with that memory. The appeal of attractors, as corresponding to our intuitive notion of the persistence of cognitive concepts along some temporal span, has been fortified by numerous studies testifying to the applicability of ANNs as models of the human memory [for a review see Amit (1989)1, and is also supported by biological findings of delayed, poststimulus, sustained activity (Fuster and Jervey 1982; Miyashita and Chang 1988). All N neurons in the network have a uniform positive threshold T. Each neuron is described by a binary variable S = (1,O) denoting an active (firing) or passive (quiescent) state, respectively. M = aN distributed memory patterns [ p are stored in the network. The elements of each memory pattern are chosen to be 1 (0)with probability p (1 - p ) respectively, with p 1. This changes the dynamics of the system to

where Di denotes a random set of indices corresponding to neurons to which the ith neuron is connected, and IDil/N = 1 - d 5 1. T remains the same value as before. In the network’s “premorbid state, the memories have maximal stability, achieved by choosing the optimal threshold T that maximizes the increase of the overlap (say ml), as shown in the Appendix. When the network is initialized with an input pattern that is a corrupted version


-0.05

739

0

0.05

0.1

h

Figure 1: The distribution of the postsynaptic potential ( p = 0.1,a = 0.05). Solid curve: Initial state: two gaussian distributions peaked at -$(1 - p ) and p(1 - P ) ~ . The optimal threshold T = p(1 - p ) ( i - p ) lies in the middle between the two gaussian mean values. Dashed curve: After deletion (d = 0.25), the new peaks of the postsynaptic potential are no longer equidistant from the threshold T. Dot-dashed curve: The O K strategy restores the initial mean values of the postsynaptic potential (d = 0.75).

of one of the stored memory patterns (e.g., ['I, it will flow dynamically into the attractor given by this memory. To obtain an intuitive notion of the network's behavior when synaptic deletion and compensation are incorporated consider Figure 1. The neurons that stand for firing neurons in the stored memory, and the neurons that stand for quiescent neurons in the stored memory, have distinct postsynaptic potential distributions (the solid curves in Fig. 1). When synaptic deletion takes place, the mean values of the neurons' postsynaptic potential change, and the threshold is no longer optimal (see dashed curves in Fig. 1). Multiplying the weights of the remaining synaptic connections by an optimal performance Compensation factor ( O K ) c = 1/(1- d), restores the original mean values of postsynaptic potential and the optimality of the threshold (dot-dashed curves in Fig. 1). The accom an in increase in the variance of the postsynaptic potential, which is times larger than the original one, leads, however, to performance deterioration. This is further elucidated in the Appendix.

Je

D. Horn et al.

740

1 .o

0.6

0.6

0.4

0.2

0.0

0

0.2

0.6

0.4

0.6

1

d

Figure 2: Performance of a network with fixed k compensation. Starting h m an initial state that is a corrupted version [m'(O) = 0.81 of a stored memory pattern, we define performance as the percentage of cases in which the network converged to the correct memory. The simulation parameters are N = 800 neurons, a = 0.05, and p = 0.1. The curves represent (from left to right) the performance of fixed strategies with increasing k values, for k = 0,0.25,0.375,0.5,0.625,0.75,1. The horizontal dotted lines represent performance levels of 25 and 75%.

We can interpolate between the case of deletion without compensation and the OPC within a class of compensatory strategies, defined by (2.5)

with the parameter 0 5 k 5 1. All the fixed k strategies, examined via simulations measuring the performance of the network at various deletion and compensation levels, display a similar sharp transition from the memory-retrieval phase to a nonretrieval phase, as shown in Figure 2. Varying the compensation magnitude k merely shifts the location of the transition region. This sharp transition is similar to that reported previously in cases of deletion without compensation in other models (Canning and Gardner 1988; Koscielny-Bunde 1990).


741

1.0

0.8

0.6 Y

0.4

0.2

0.0

0

0.2

0.4

0.6

0.8

1

d

Figure 3 The critical transition range in the (k,d) plane. The solid curves represent performance levels of 75 and 25%, derived from Figure 2. The straight lines describe the variations employed in two variable compensations presented in Figure 4.

Figure 3 describes the transition region as a map in the (k,d)plane. The performance levels read off Figure 2 delineate the domain over which deterioration occurs. Staying close to the upper boundary of this domain defines a compensation strategy that enables the system to maintain its performance, with a much smaller amount of synaptic strengthening than that required by the OPC strategy. In the face of restricted compensatory resources, such an optimal resource Compensation strategy (ORC) could be of vital importance. The essence of such ORC strategy is that k is varied as synaptic deletion progresses, in order to retain maximal performance with minimal resource allocation. In Figure 4, we present the performance of two variable k compensation strategies, which we propose to view as expressions of (albeit unsuccessful) attempts of maintaining an ORC. These examples, indicated in Figure 3, include a “gradually decreasing” strategy defined by the variation k = 0.3 O.M, and the “plateau” strategy defined by the variation k = d. The analogs of these strategies can be found in clinical observations, as shown in the next section where we review the biological and clinical evidence relevant to our model.

+

1.0

2.'

'

I

'

.:. . \. .

'

'

'

I

'

"

'

I

"

'

'

I

'

'

'

'

I I

i'

...

Figure 4: Performance of a network with gradually decreasing (dotted curve) and plateau (dashed curve) compensation strategies. 3 Clinical Motivation and Implications

As mentioned in the introduction, while synaptic degeneration occurs, the TSA stays constant in some cortical layers at the initial stages of AD. Qualitatively similar synaptic changes have been observed during normal physiological aging, but with significantly lower deletion (BertoniFreddari et al. 1988, 1990). Hence a plausible general scenario seems to involve some initial period of OPC.As AD progresses, synaptic compensation no longer succeeds in maintaining the TSA (Bertoni-Freddari et al. 1990; DeKosky and Scheff 1990). In advanced AD cases, severe compensatory dysfunction has been observed (Buell and Coleman 1979; Flood and Coleman 1986; DeKosky and Scheff 1990). Young AD patients are likely to have high compensation capacities, and therefore can maintain an OPC strategy (k = 1, in Fig. 2) throughout the course of their disease. This will then lead to a rapid deterioration when the reserve of synaptic connections has been depleted. Indeed, young AD patients have been reported to have a rapid clinical progression (Heston et al. 1981; Heyman et al. 1983), accompanied by severe neuronal and synaptic loss (Hansen et al. 1988). A similar clinical pattern of rapid memory decline, already manifested with less severe neuroanatomical pathology, was found in very old patients (Huff et al. 1987).


743

We propose that in these old patients, the rapid clinical decline results from the lack of compensatory capacities (k = 0, in Fig. 2), possibly of the kind observed by Buell and Coleman (1979) and Flood and Coleman (1986). Rapid cognitive decline characterizes a minority of AD patients. Most patients show a continuous gradual pattern of cognitive decline (Adams and Victor 1989; Katzman 1986; Katzman et al. 1988), taking place along a broad spun of synaptic deletion (DeKosky and Scheff 1990). As shown in Figure 2, this performance decline cannot be accounted for by any network employing fixed k compensation. Variable compensation, such as that defined by the gradually decreasing strategy, is needed to explain the memory decline observed in the majority of AD patients, as shown in Figure 4. The clinical state of some AD patients remains stable at mild to moderate levels for several years before finally rapidly decreasing (Cummings and Benson 1983; Katzman 1985; Botwinick et al. 1986). This can be accounted for by a "plateau" strategy whose performance, shown in Figure 4, stays at an approximately constant level over a large domain of deletion. Synaptic deletion and compensatory mechanisms play a major role also in the pathogenesis of Parkinson disease (Zigmond ef al. 1990; Calne and Zigmond 1991). The significant incidence of AD patients having accompanying extrapyramidal parkinsonian signs (Mayeux ef al. 1985; Stem et al. 1990) naturally raises the possibility that such patients may have a decreased synaptic compensatory potential in general (Horn and Ruppin 1992). The cognitive deterioration of these AD patients is faster than that of AD patients without extrapyramidal signs. This fits well with our proposal that severely deteriorated synaptic compensation capacity leads to an accelerated rate of cognitive decline in AD patients. This issue is still inconclusive because the I'D-AD combination may be a specific syndrome on its own. 4 Discussion

In accordance with the findings that neuronal loss in AD is less than 10% even at advanced stages (Katzman 19861, and that the synapse to neuron ratio is significantly decreased (Davies et al. 1987; Bertoni-Freddari et al. 1990), we have concentrated on studying the role of the synaptic changes. Simulations we have performed incorporating neuronal loss have shown similar results to those presented above. We conclude therefore that the important factors are indeed the number of synapses retained and the compensation strategy employed, whose interplay may lead to various patterns of performance decline. As any current neural model of human cognitive phenomena, our model necessarily involves many simplifications. The TF formal neurons are obviously a very gross simplification of biological neurons. As in

744

D. Horn et al.

most Hopfield-like ANNs, the network has no spatially specified architecture. For clarity of exposition of our main ideas, we have assumed that all compensation strategies are applied uniformly to all retained synapses. Our analysis also holds for nonuniform compensation, that is, when each remaining synaptic weight is multiplied by a random variable with mean value c and variance 02, since the same averages of the postsynaptic potentials are obtained (see Fig. 1 and the Appendix). Obviously, if the variance is too large, then no compensation strategy can be conceived of any more. Motivated by the biological evidence testifying to the sparsity of neural firing (Abeles et al. 19901, we have assumed a relatively small fraction p of firing neurons. Simulations performed with higher p values (e.g., 0.2) indicate that the results remain qualitatively the same. However, it should be noted that as p is increased the approximation of the network's overlap dynamics presented in the Appendix becomes less and less accurate. The variable compensation strategies that we have discussed rely on the fact that there is some span in the (k,d ) plane over which deterioration takes place, as shown in Figure 3. As N is increased, the width of the domain over which deterioration occurs keeps getting narrower, thus limiting the possibilities of maneuvering between deletion and compensation. Hence, one may claim that our conclusions, which are based on simulations of small scale networks, do not hold for the brain. One possible answer to this problem is that there may exist important modules in the brain whose function depends on the correct performance of just some thousands of neurons (Eccles 1981). For large cortical modules, this objection may be resolved by considering the effect of noise present in the brain. To account for the latter, any realistic paradigm of memory recall should entail the recognition of a spectrum of noisy inputs presented to the network. Figure 5 displays the performance of the network in the (k,d ) plane obtained via simulations with two distinct initial overlap values [rn'(O) = 0.8 and m'(0) = 0.951, together with the theoretical results for the infinite N limit. These results show that even in this limit, the corresponding performance curve always retains a finite width as long as the network processes input patterns with a broad range of initial overlaps. Consequently, the realization of variable compensatory strategies may indeed be feasible in the brain. The decline in the network's performance resulting from synaptic deletion is coupled with a decrease in the network's overall activity. This observation gives rise to the possibility that although being defined "globally" as "strategies," synaptic compensation may take place via local feedback mechanisms. Accordingly, the decreased firing rate of a neuron being gradually deprived of its synaptic inputs may enhance the activity of cellular processes strengthening its remaining synapses. This scenario seems to lead to fixed OPC compensation in a rather straightforward manner, but as synaptic deletion may be nonhomogeneous the effects of the resulting spatially nonuniform compensatory changes should be fur-


745

1.o

0.8

0.6

0.4

0.2

0.0

d

Figure 5: The critical transition range in the ( k , d ) plane. The solid curves represent performance levels of 75 and 25%, with initial overlap m(0) = 0.8 (identical to Fig. 2). The dash-dotted curves represent performance levels of 75 and 25%, for initial overlap n ( 0 ) = 0.95. The dotted curves represent the theoretical results that follow from the analysis presented in the Appendix, delineating the estimates of when the correspondingbasins of attraction cease to exist in the infinite N limit. These curves lie close to the 25% lines of the simulations.

ther investigated. The nonvanishing width of the (k,d ) plane transition range shown above is essential for the feasibility of an ORC strategy, so that local mechanisms can “trace” the decreasing performance and “counteract” it before the performance collapses entirely. Finally let us comment on possible examinations and applications of our model. An ideal experiment would involve a series of consecutive measurements of synaptic strength and cognitive abilities. In light of obvious difficulties concerning such tests, we may have to resort to comparing biopsies and autopsies, as in DeKosky and Scheff (1990), preferably on the same patients. Our model demonstrates the importance of maintaining the TSA for the preservation of memory capacity and, therefore, mental ability of AD patients. This may suggest that future therapeutic efforts in AD should include an attempt to mobilize compensatory mechanisms facilitating the regeneration of synaptic connectivity.

D. Horn et al.

746

Appendix Qualitative features of our model can be derived from a simple analysis of the first iteration step. Starting with a state that is close to with overlap m'(0) we wish to find whether the network flows into the correct memory. Using the dynamics defined in the text we find for t = 1

['

=

8 [ ~ (-l d)([! - p)p(l - p)m'(O) + N - T ]

(A.1)

where we have separated the signal from the noise term. The latter has zero average, (N)= 0, and variance of (N2) = c2(1 - d)p2(1 - ~ ) ~ a s ( O ) , where s(0) = P[S,(O) = 11 = l-p-m1(0)+2pm'(0). In view of the gaussian noise term we write the probability in terms of an error-function,

This results in the following expression for the first iteration: m'(1) =

=

1

((1 - p)P([! = l)P[Si(l) = 1"; P(1 -P ) - pP([; = O)P[Sj(l) = lI&!= 01) ~

P[Sj(I) = 1[;

= erf

=

I]

= 11- P[Si(I) = 11~;= 01

(1- p)m'(O)p(l - p)C(l - d) - T

[ Jac2(1 - d)s(O)p2(1 - p ) 2

[

1 1

- erf (-p)m'(O)p(l - p ) d l - d) - T Jac2(1 - d)s(O)p2(1 - p ) 2

(A.3)

In the limit m'(0) -+ 1 one finds the maximal value of equation A.3 to be obtained for the following choice of the optimal threshold:

T' = ~ (- ld)p(l - p)(l - 2p)/2

(A.4)

For c = 1 and d = 0 this coincides with the choice we have made. Moreover, as long as c=-

1

I-d

(A.5)

which was defined as the OPC strategy, T remains optimal. This fact was expressed graphically in Figure 1. The two gaussian distributions in this figure correspond to the two terms in equation A.3.


747

In the simulations we have looked for the cases in which the system converged onto the correct fixed points. This involves iterating the equations of motion, which is in general different from iterating expressions like equation A.3 because of possible correlations between the different time steps. Nonetheless we may think of the iteration of equation A.3 (replacing the labeling 0 and 1 by n - 1 and n ) as a rough estimate for the strongly diluted infinite system (Evans 1989). Starting out with different values for m’(0) we find the dotted curves in Figure 5, which show the borderlines between convergence and nonconvergence to the correct memory. An alternative to this derivation is to use the replica symmetry assumption. We have carried out such an analysis (Herrmann et al. 1992). The results are similar, though not identical, to the ones reported above.

References Abeles, M., Vaadia, E., and Bergman, H. 1990. Firing patterns of single units in the prefrontal cortex and neural network models. Network I,13-25. Adams, R. D., and Victor, M. 1989. Principles of Neurology. McGraw-Hill, New York. Amit, D. J.1989. Modeling Brain Function: The World of Attractor Neural Networks. Cambridge University Press, Cambridge. Bertoni-Freddari,C., Meier-Ruge, W., and Ulrich, J. 1988. Quantitative morphology of synaptic plasticity in the aging brain. Scanning Microsc. 2, 1027-1034. Bertoni-Freddari, C., Fattoretti, P., Casoli, T., Meier-Ruge, W., and Ulrich, J. 1990. Morphological adaptive response of the synaptic junctional zones in the human dentate gyrus during aging and Alzheimer’s disease. Brain Res. 517,69-75. Botwinick, J., Storandt, M., and Berg, L. 1986.A longitudinal behavioral study of senile dementia of the Alzheimer type. Arch. Neurol. 43, 1124-1127. Buell, S.J.,and Coleman, I? D. 1979. Dendritic growth in the aged human brain and failure of growth in senile dementia. Science 206, 854-856. Calne, D. B., and Zigmond, M. J. 1991. Compensatory mechanisms in degenerative neurologic diseases. Arch. Neurol. 48,361-363. Canning, A., and Gardner, E. 1988. Partially connected models of neural networks. 1.Phys. A: Math. Gen. 214,32753284. Cummings, J. L.,and Benson, D. F. 1983. Dementia: A Clinical Approach. Butterworths, London. Davies, C. A., Mann, D. M. A., Sumpter, P. Q., and Yates, P. 0.1987.A quantitative morphometric analysis of the neuronal and synaptic content of frontal and temporal cortex in patient with Alzheimer’s disease. 1.Neurol. Sci. 78, 151-164. DeKosky, S. T., and Scheff, S. W. 1990. Synapse loss in frontal cortex biopsies in Alzheimer’s disease: Correlation with cognitive severity. Ann. Neurol. 27(5), 457-464.

748

D. Horn et al.

Drachman, D. A., O'Donnell, B. F., Lew, R. A,, and Swearer, J. M. 1990. The prognosis in Alzheimer's disease. Arch. Neurol. 47, 851-856. Eccles, J. C. 1981. The modular operation of the cerebral neocortex considered as the material basis of mental events. Neuroscience 6, 1839-1855. Evans, M. R. 1989. Random dilution in a neural network for biased pattern. J. Phys. A: Math. Gen. 22,2103-2118. Flood, D. G., and Coleman, P. D. 1986. Failed compensatory dendritic growth as a pathophysiological process in Alzheimer's disease. Can. J. Neurol. Sci. 13,475-479. Fuster, J. M.,and Jervey, J. P. 1982. Neuronal firing in the inferotemporal cortex of the monkey in a visual memory task. J. Neurosci. 2(3), 361-375. Hansen, L. A., DeTeresa, R., Davies, P., and Terry, R. D. 1988. Neocortical morphometry, lesion counts, and choline acetyltransferase levels in the age spectrum of Alzheimer's disease. Neurology 38,48-54. Hemnann, M., Horn, D., Ruppin, E., and Usher, M. 1992. Variability in the pathogenesis of Alzheimer's disease: analytical results. To appear in the Proc. ICA""92, September, Brighton, UK (in press). Heston, L. L., Mastri, A. R., Anderson, V. E., and White, J. 1981. Dementia of the Alzheimer type: clinical genetics, natural history, and associated conditions. Arch. Gen. Psychiat. 38, 108!%1090. Heyman, A., Wilkinson, W. E., Hurwitz, P. J., Schmechel, D., Sigmon, A. H., Weinberg, T., Helms, M. J., and Swift, M. 1983. Alzheimer's disease: Genetic aspects and associated clinical disorders. Ann. Neurol. 14(5), 507-515. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Horn, D., and Ruppin, E. 1992. Extra-pyramidal symptoms in Alzheimer's disease: a hypothesis. Med.Hypotheses 39(4), 316-318. Huff, F. J., Growden, J. H., Gorkin, S., and Rosen, T. J. 1987. Age of onset and rate of progression of Alzheimer's disease. J. Am. Geriatr. SOC. 35,27-30. Jansen, K. L.R., Faull, R. L. M., Dragunow, M., and Synek, 8. L. 1990. Alzheimer's disease: Changes in hippocampal N-methyl-Daspartate, quisqualate, neurotensin, adenosine, benzodiazepine, serotonin and opoid receptors-an autoradiographic study. Neuroscience 39(3), 613-627. Katzman, R. 1985. Clinical presentation of the course of Alzheimer's disease: The atypical patient. Interdiscipl. Topics. Gerontol. 20, 12-18. Katzman, R. 1986. Alzheimer's disease. N. Engl. J. Med. 314(15), 964-973. Katzman, R., et al. 1988. Comparison of rate of annual change of mental status score in four independent studies of patients with Alzheimer's disease. Ann. Neurol. 24(3), 384-389. Koscielny-Bunde, E. 1990. Effect of damage in neural networks. 1.Statist. Phys. 58,1257-1266. Kosik, K. S. 1991. Alzheimer's plaques and tangles: Advances in both fronts. TINS 14,218-219. Mayeux, R., Stem, Y., and Spanton, S. 1985. Heterogeneity in dementia of the Alzheimer type: Evidence of subgroups. Neurology 35,453-461. Miyashita, Y.,and Chang, H. S. 1988. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature (London) 331,68.


749

Selkoe, D. J. 1987. Deciphering Alzheimer’s disease: The pace quickens. TINS 10, 181-184. Stern, Y., Hesdorffer, D., Sano, M., and Mayeux, R. 1990. Measurement and prediction of functional capacity in Alzheimer’s disease.’ Neurology 40,8-14. Tsodyks, M. V., and Feigelman, M. V. 1988. The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 6, 101-105. Zigmond, M. J., Abercrombie, E. D., Berger, T. W., Grace, A. A., and Stricker, E. M. 1990. Compensations after lesions of central dopaminergic neurons: Some clinical and basic implications. TINS 13, 290. Received 1 July 1992;accepted 14 January 1993.

This article has been cited by: 2. Andre Fischer, Farahnaz Sananbenesi, Xinyu Wang, Matthew Dobbin, Li-Huei Tsai. 2007. Recovery of learning and memory is associated with chromatin remodelling. Nature 447:7141, 178-182. [CrossRef] 3. Greg J. Siegle, Michael E. Hasselmo. 2002. Using connectionist models to guide assessment of psycological disorder. Psychological Assessment 14:3, 263-278. [CrossRef] 4. Leif H. Finkel. 2000. NEUROENGINEERING MODELS OF BRAIN DISEASE. Annual Review of Biomedical Engineering 2:1, 577-606. [CrossRef] 5. Jean-Marc Fellous, Christiane Linster. 1998. Computational Models of NeuromodulationComputational Models of Neuromodulation. Neural Computation 10:4, 771-805. [Abstract] [PDF] [PDF Plus] 6. Asnat Greenstein-Messica , Eytan Ruppin . 1998. Synaptic Runaway in Associative Networks and the Pathogenesis of SchizophreniaSynaptic Runaway in Associative Networks and the Pathogenesis of Schizophrenia. Neural Computation 10:2, 451-465. [Abstract] [PDF] [PDF Plus] 7. Joseph T. Devlin , Laura M. Gonnerman , Elaine S. Andersen , Mark S. Seidenberg . 1998. Category-Specific Semantic Deficits in Focal and Widespread Brain Damage: A Computational AccountCategory-Specific Semantic Deficits in Focal and Widespread Brain Damage: A Computational Account. Journal of Cognitive Neuroscience 10:1, 77-94. [Abstract] [PDF] [PDF Plus] 8. Eytan Ruppin , James A. Reggia . 1995. Patterns of Functional Damage in Neural Network Models of Associative MemoryPatterns of Functional Damage in Neural Network Models of Associative Memory. Neural Computation 7:5, 1105-1127. [Abstract] [PDF] [PDF Plus] 9. Sergio A. Cannas . 1995. Arithmetic PerceptronsArithmetic Perceptrons. Neural Computation 7:1, 173-181. [Abstract] [PDF] [PDF Plus] 10. D. Horn , E. Ruppin . 1995. Compensatory Mechanisms in an Attractor Neural Network Model of SchizophreniaCompensatory Mechanisms in an Attractor Neural Network Model of Schizophrenia. Neural Computation 7:1, 182-205. [Abstract] [PDF] [PDF Plus] 11. M. R. W. Dawson, A. Dobbs, H. R. Hooper, A. J. B. McEwan, J. Triscott, J. Cooney. 1994. Artificial neural networks that use single-photon emission tomography to identify patients with probable Alzheimer's disease. European Journal of Nuclear Medicine 21:12, 1303-1311. [CrossRef]


Supervised Factorial Learning A. Norman Redlich The Rockefeller University, 1230 York Avenue, New York, NY 10021 USA

Factorial learning, finding a statistically independent representation of a sensory "image"-a factorial code-is applied here to solve multilayer supemised learning problems that have traditionally required backpropagation. This lends support to Barlow's argument for factorial sensory processing, by demonstrating how it can solve actual pattern recognition problems. W o techniques for supervised factorial learning are explored, one of which gives a novel distributed solution requiring only positive examples. Also, a new nonlinear technique for factorial learning is introduced that uses neural networks based on almost reversible cellular automata. Due to the special functional connectivity of these networks-which resemble some biological microcircuitslearning requires only simple local algorithms. Also, supervised factorial learning is shown to be a viable alternative to backpropagation. One significant advantage is the existence of a measure for the performance of intermediate learning stages. 1 Introduction Inhibition of neurons by each other, both laterally and temporally, is ubiquitous in the brain, and shows up most obviously in sensory processing as sensitivity to both spatial and temporal contrast. For example, in the retina the center-surround mechanism and a similar temporal differentiating mechanism produce outputs that are most prominent at locations and times where luminance is varying. Another way to characterizethese outputs is that they are signaling only the least predictable portion of the sensory message. In other words, they have stripped the sensory signals of their redundancy. Barlow has proposed such redundancy reduction as one of the main purposes of inhibition in both the retina and neocortex (Barlow 1961, 1989, 1992). Theories based on this idea have explained both single cell and psychophysical data, with the analyses especially successful for the retina (Srinivisan et al. 1982; Field 1987, 1989; Linsker 1988, 1989; Atick and Redlich 1990, 1992; Atick et al. 1992, 1993). Barlow's thesis is that by identdymg redundancy between different afferent signals, the brain discovers their statistical relationships, and this statistical knowledge is critical for object recognition and associative learning. More specifically Neurd Computation 5, 750-766 (1993)


Supervised Factorial Learning

751

he proposes that a representation where the individual signals, for example, neural outputs, are statistically independent+ factorial code-is the most efficient one for storing statistical knowledge. Although learning a factorial code can be very difficult, it can be accomplished through a step by step reduction in redundancy (Barlow and Foldiak 1989; Atick and Redlich 1991; Redlich 1992). At successive stages the code becomes more factorial and in this way statistical knowledge is acquired. Here I shall refer to this as factorial learning. It can be an enormously effective nonparametric strategy for measuring the statistics of an ensemble (Redlich 1992). There remains in my opinion one major conceptual gap in the arguments in favor of factorial coding and learning in the brain: How can this type of processing actually help solve difficult cognitive problems such as pattern recognition? In this paper, I explore one possible answer to this question, by demonstrating that factorial learning can incorporate supervision. In principle this amounts to using factorial learning to find the probabilities P(s, u ) of both an unsupervised signal u, such as the retinal image, plus an explicit supervising signal s. However, in practice the way supervision is implemented makes a big difference in the learning efficiency. Here, two very different implementations are explored, each with different advantages. The first is most similar to traditional supervised learning. The second produces a novel distributed representation of P ( s , u ) by using the redundancy in u to suppress the network output most when u corresponds to the desired concept s. Also, this network learns using only positive examples of s. In this paper I also introduce a new approach to factorial learning using almost reversible cellular automata (ARCA). This makes possible completely nonlinear coding as needed for example to solve binary problems. It has been the lack of such nonlinear techniques that has been one of the greatest obstacles to applying factorial coding to a theory of the neocortex, as was done for the retina in Atick and Redlich (1990,1992). One of the nicest features of ARCA networks is their similarity to biological lateral inhibition circuits, as seen, for example, in the retina, olfactory bulb, and visual cortex (Shepherd 1988, 1990). They thus provide one explanation for the purpose of this inhibition, since in ARCA networks some neurons attempt to predict and then shut down other neurons in order to remove redundancy. Moreover, factorial learning using ARCA networks is a concrete example where restricting the functional connectivity of the network allows the use of very simple neural learning algorithms. Beyond the potential biological applications, supervised factorial learning is a genuine alternative (or complement) to backpropagation for practical applications. The primary advantage is that it comes with a measure for the performance of intermediate stages. No reference is made to the final stage output and there is no need to be stuck with a fixed network connectivity. In this sense the approach here is most similar to

752

A. Norman Redlich

Cascade-Correlation (Fahlman and Lebiere 19901, as will be discussed in Section 4. A further advantage is that supervised factorial coding is a broad theoretical framework that encompasses both of the ARCA implementations described here as well as many others including the linear algorithms in Atick and Redlich (1991) and the nonlinear bound state methods in Redlich (1992). The paper is organized as follows: in Section 2 the general theory of factorial learning-both supervised and unsupervised-is outlined, while in Section 3 ARCA networks are described and neural learning algorithms derived. Section 4 then introduces supervision; it is broken into two subsections to discuss the two different implementations mentioned above. To be as concrete as possible, and also to test whether supervised factorial learning is a viable alternative to backpropagation, I revisit in both parts of Section 4, some classical backpropagationproblems (Rumelhart et al. 1986) and solve them. Some issues of generalization are also discussed.

2 Factorial Learning

The goal of factorial learning is to find successivelybetter approximations to the joint probabilities P(u1,u 2 , . . .)-to be denoted P(u)-of a set of signals ui. The difficulty is that the number of joint probabilities can be enormous: if there are n signals Ui, each with N gray levels the number is N".Instead of directly attempting to measure these N" probabilities, one would like to determine P ( u ) from a much smaller set of measurements, for example by measuring only the n probabilities Pi(ui). This can be achieved by approximating the joint probabilities P ( u ) by the product of individual probabilities P i ( U i )

that is, by assuming statistical independence. Of course, for the original input u this factorial assumption is likely to be wildly wrong. The goal of factorial learning however is to map the signals u to a new representation u', which is statistically independent. This is accomplished in stages, as shown in Figure 1, so that ultimately one finds an encoding for which equation 2.1 is a good approximation. But what is meant by a better approximation to a factorial code? To quantify the improvements at each stage, it is necessary to define a measure of statistical independence for each representation of the signal u. This learning meusure should also be a function only of the local probabilities Pi(ui). A measure that does this is the sum of the individual


Pl(U"1)

Plh(U"2)

753

P113(u1'3)

P"4(u"4)

Figure 1: A feedforward example of a multilayer network for factorial learning. Each layer L, L', L" . . . of neurons gives a complete representation, with the outputs ui,u;, uy . .. of successive stages becoming more statistically independent. entropies'

where the second sum is over all possible gray scale values of ui. Given one additional constraint, this measure is minimal only when the code is factorial, so reducing E in stages improves the approximation 2.1. Of course one cannot get something for nothing: one cannot learn the global P(u) using a function E of only the local Pi(ui). This is why it is also necessary to impose the global constraint, H[P(u)]= H'[P'(u')],of no total information loss from u to u', where H[P(u)]is the total entropy, With this constraint, minimizing E produces a factorial code due to the well known theorem CiHi 2 H, with equality only when the code is factorial. But doesn't this require knowledge of P(u),just what we do not know? Absolutely not, since this constraint can be replaced by imposing rmersibility on the map u -, u': ui

= u':'(u')

Exists

(2.3)

'Note that minimizing E in equation 2.2 is referred to as redundancy reduction even though it does not strictly speaking reduce the total Shannon redundancy (Shannon and Weaver 1949). It does reduce all of the redundancy between individual signal elements, which is important here.

A. Norman Redlich

754 ~

~~

u'o

u'1

d3

u'2

u'4

u's

..

:" uo

u1

u3

u2

u4

us

Figure 2 An example of a one-dimensional ARCA network where inputs ui can be inhibited only by inputs to their left, in order to ensure reversibility: equations 3.1 with p = 3. The X neurons implement the modular subtraction u{ = ui = fi(u),while the f i interneurons attempt to predict and shut down ui in order to reduce redundancy. Reversibility is sufficient-though not always necessary-to ensure no information loss. I now introduce a class of maps for which reversibility is ensured by restricting the functional connectivity of a "neural" network. 3 Reversible Coding: ARCA

It is simplest to describe an "almost" reversible cellular automata (ARCA) network using a one-dimensional example. [These automata were called almost reversible in Eriksson et al. (1987)to distinguish them from the usual reversible automata (Margolus 1984); however, in this paper they produce fully reversible maps, and I apologize for any confusion this terminology may cause.] Suppose there are n signals U i , with i = 0, . . . n 1,which are to be mapped reversibly to a new set of n signals u:, as shown in Figure 2. Also, assume that the signals have N gray levels. Then one possible ARCA rule/network, as shown in the figure for p = 3, is

- fi(u+, ui -fi(uo, ~

..

u; = ui

U ~ - ~ + I ,uj-1) .

U; =

1 , . Ui-1)

..

mod N, mod N,

for i 2 p for i < p

(3.1)

where 0 < p < n. For continuous signals this rule is replaced by ordinary subtraction. In Figure 2, the "X neurons'' perform the modular arithmetic,


755

which in the binary case is XOR, hence the X. They are shown as distinct units in the figure, but could be combined into a single unit with the f i "interneurons." It is not difficult to convince oneself that such a rule is completely reversible2 for arbitrary functions f i , assuming the f i are quantized to have N gray levels. This follows because modular subtraction is reversible, and to compute any given ui from the u: one need only start from i = 0 and work deductively up to i. Also, it is not difficult to generalize this rule, for example, by working from i = n downward. Furthermore, ARCA networks need not be asymmetrical. For example, one can take ui = ui -fi(u,) for i even and j odd, and u: = ui for i odd. All that is needed for reversibility is that the fi(u) for any given i depend only on those uj that can be calculated first without knowing ui. ARCA networks can also be cascaded to compute larger classes of reversible functions. For linear coding of continuous signals the rule in equation 3.1 becomes simply u: = Cj Wjjuj with W ,a matrix of the form Wij = 1 and Wi,.=0 for j > i. Such a triangular matrix automatically has det(W) = 1, so its inverse is guaranteed to exist. Also, cascading different triangular matrices can produce arbitrary linear transformations with determinant one. 3.1 Similarities to Local Biological Circuits. Comparing the ARCA network in Figure 2 to microcircuits in the brain one finds some similarities. First, in most brain circuits there are neurons like the X neurons in the ARCA network, which provide straight-through transmission of afferent signals (Shepherd 1988,1990). These are, for example, the bipolar cells in the retina, the mitral cells in the olfactory bulb, the relay cells in the LGN,and the pyramidal neurons in olfactory, hippocampal, and neocortex. In the ARCA network the purpose of the X cells is to preserve information flow, and it seems likely that this is also partly the role of "straight-through" neurons in the brain, as is perhaps clearest for the bipolar cells. Second, the f interneurons in the ARCA network provide another major type of interaction seen in basic brain circuits: inhibitory horizontal interactions. In Figure 2 these are of the feedforward type like the horizontal cells in the retina and periglomerular cells in the olfactory bulb; such inhibition is also seen in the cortex and and there is evidence for it in the LGN (Shepherd 1990). The nonbiological aspect of the network in Figure 2 is its asymmetry, but as mentioned above, there are also symmetrical ARCA networks, though they require skipping some inputs. In any case, I do not argue that the brain's microcircuits are precisely ARCA networks, but that functionally there are significant similarities. *The original ARCA rule in Eriksson et al. (1987) includes only the first line in equation 3.1 so in an infinite string there are always p unknown signals making the code almost reversible. However, in the infinite system there is no information loss because the missing p signals carry negligible information.

A. Norman Redlich

756

3.2 Factorial Learning Algorithms for ARCA Networks. In applying an ARCA rule/network such as equation 3.1 (Fig. 2) to factorial learning, the functions fi(u) are learned by minimizing E in equation 2.2. In principle this involves measuring the probabilities Pi(ui)-but not P(u)after each change in fi(u). However, in practice it is usually sufficient to replace E by a function such as C i ( U i -fi)2, which requires no probability measurements. For the binary case and for the case of continuous gaussian signals this is guaranteed to lower the Hi [see below for the binary case and Atick and Redlich (1990) for the gaussian]. For other types of signals it should also lower E, but I have not yet attempted a proof. I should emphasize that such simplification may not lead to lowering E optimally at each stage, but with many stages this is not necessary. To see why minimizing Ci(Uj-fi)2 can be sufficient, consider the very simple problem of factorizing a pair of signals uo and u1 as depicted in Figure 2, ignoring the neurons with i > 1. Here ub = uo and u{ = u1 - f1 (ug), and furthermore I assume the two signals are binary. In this case, ub = uo so we can take E’ = -P log(P‘)- (1 - P‘) log(1 - P’) where P is the probability that u; = 1. Also, because the signals are binary P’ = ( u ; ) = ((u:)~)= ((u1 - f ~ ( u o ) ) ’ )so reducing the square difference between u1 and its prediction fi(u0) reduces P’ and thus E’. This is true even if P’ is initially greater than 1/2 since then, by minimizing P’, a function of the complement 1 - u1 is found that has P’ < 1/2. Going back to the many neuron problem, it is straightforward to find an explicit learning algorithm for the binary case (as well as other cases). Take the functionsfi(uj) to be sigmoids ~ ( uof) the linear s u m of inputs u.: fi(Uj) = U ( w i j *U j - ti). Then using gradient descent to minimize C ( U i - f i f i gives the update

(3.2) dwij = O(ui - f i ) ( l -fi)fiuj for the synapse wv and similarly for the threshold ti. Here r) is the usual

gradient descent parameter; I have found learning to be insensitive to its value. Also, because this is basically a one-layer algorithm it tends to converge very rapidly to both a lower sum of squares and a lower E , equation 2.2. I have never found it to increase E . Note also that in the linear case where ui = Wiju,, minimizing the sum of squares Ci(ui)2through gradient descent gives

xi

d Wij = -QU{U~

(3.3)

which is a simple anti-Hebb rule for the output ui. This rule is simpler than the linear rule in Atick and Redlich (1991) because there the reversibility constraint was added explicitly in E. 4 Supervised Learning

In supervised learning there are two signals, the unsupervised “image” ui plus a set of supervisor signals sj. The sj may be coincident with


757

S"'

S"

S'

U

S

Figure 3 Schematic of ARCA networks for supervised learning of the more "traditional" type. Here only the redundancy between u and s is reduced with the fi interneurons functions of u alone, and the X neurons performing s[ = si - fi(u). At successive stages only that portion of the s signal, which could not be predicted, is passed forward, simplifying the learning at the next stage, until the problem is solved. or occur before or after the signals ui: the index i may include both space and time. The goal is to factorially learn P(s, u ) from which can be calculated P(slu) = P ( s , u ) / P ( u ) ,which is the probability that u belongs to the concept s. For this it may be useful for the early stages of sensory processing to have already learned the background statistics P(u). In applying factorial learning to the supervised case, however, there is a major issue of implementation to be resolved. The problem is that during supervised learning the input consists of pairs u,s, but after learning only the signal u is given since the aim is to predict s. So if supervised factorial learning were implemented naively one would need to cycle through all possible s serially to find the one with largest P(s1u). Though the brain does perform serial searches, it also seems capable of solving many problems in parallel. I therefore give below two approaches to supervised factorial that avoid the need for serial processing. 4.1 "Traditional" Oracle-Type Approach. One way to avoid serial processing is to exploit the fact that an ARCA network factorizes by using one set of signals to predict others, with one set in effect acting as the "oracle" for the others. To apply this to the supervised case, one uses u as the predicting and s as the predicted signals. This is shown schematically in Figure 3 where the functions fi(u) attempt to predict the

758

A. Norman Redlich

signals s;. The outputs s: of the XOR neurons are the unpredicted error signals, which at the next learning stage the fi' attempt to predict. Since the error signals at successive stages contain less and less information, they become easier to predict. If at first, for example, the problem is not half plane separable, it becomes more and more so, until at the final stage it can be learned by a perceptron. Cycling through possible s is not required in Figure 3 since given a u, the network has a direct prediction of the signals Si as the modular sum of the f i from all stages. This is because when learning is complete, the final stage outputs s r remain off for all pairs u , s so Therefore, if instead of equation 4.1, the unknown inputs si are artificially set to zero, then the outputs sy will equal (minus) the predicted si. 4.1 .1 Some Classical Backpropagation Problems Revisited. For illustration, I now revisit some classical backpropagation problems (Rumelhart et al. 1986). For every one of these, supervised factorial leaning finds a solution without false minima difficulties, although as in all gradient descent problems q must not be too large. Also, the solutions here use fewer or the same number of learning interneurons f ; as the backpropagation solutions. The XOwParity problem is the usual paradigm for multilayer learning since even XOR is not perceptron learnable. Here, this is perhaps not the best example since ARCA networks already include some XOR neurons. However, the learning elements in the network are the f i interneurons for which learning XOR/Parity remains nontrivial. More importantly, XOR is the simplest example that illustrates how factorial leaming in distinct stages differs from backpropagation: An ARCA network for XOR is shown in Figure 4a. In the first learning step the f ( u 0 , u l ) neuron attempts to predict s, but cannot do so completely because f(u) is a perceptron. Instead, it shuts down s as much as possible, leaving for example s' = 1 for u = (1,O) only, rather than also for (0,l). The problem at the next stage is then much simpler since s' can be predicted by a perceptron. Although there are separate XOR/modular neurons in Figure 4a for each stage, they can be combined into a single "parity" neuron as in Figure 4b. This figure shows a network that solves both the Parity problem and the Symmetry problem (s = 1iff UO,ul, u2 = us,u4,ug) by learning the functions f , f ' , f" . . . in stages as was done for XOR in Figure 4a. These solutions use the same number of learning f neurons as used by backpropagation in Rumelhart et al. (1986). Another problem (see Fig. 4c), that was solved using the same number of neurons as backpropagation was the Negation problem (SO,s1, s2 equals the complements of UO,UI,u2 when u3 = 1 and simply equal UO,ul,u2 otherwise). Here all the f; were learned in one step.

759


S'

4a

t

4b

.. U

uo

S

4c

4d

f7 s'o 4

s'1

s'2

4

4

so

s1

s2

U

U

U

so

U

U

U

u = uo u1 u2 u3

u = uo u1 u2 u3

Figure 4: ARCA networks used to solve some classical backpropagation problems: XOR in a, Parity and Symmetry in b, Negation in c, and Addition in d. See text for details.

A more interesting result was discovered for the Addition problem since redundancy between the si could be exploited to find a more compact solution than in Rumelhart et al. (1986). In this problem the two bit binary number UO,u1 is added to U P ,243 giving the output SO, SI,SP. However, since the sums do not occur with equal frequency, the si are not statistically independent. Therefore, for example, knowing SO can im-

760

A. Norman Redlich

prove the prediction of sl. One can exploit this by first learning fo(u) (which predicts s1) and then using it with u to learn fi(u,fo), and then using them both to learnfz(u,fo,fI). This is illustrated in Figure 4d, which turns out to be a three neuron solution-in contrast to the five neuron solution in Rumelhart et al. (1986). I have also solved the classical T,C problem using the approach of this section, although it and the Symmetry problem above are far more elegantly solved in Section 4.2. The cost of using the the “traditional” approach here is the need for many learning stages. For example the T,C problem on a 10 x 10 grid required 37 stages, although this represents only 37 neurons, fewer than needed for the backpropagation solution. Also the learning here was accomplished without assuming translation invariance as in Rumelhart et a1 (1986). 4.1.2 Comparison with Cascade-Correlation. As mentioned in the introduction, the cascade-correlation approach of Fahlman and Lebiere (1990) is similar to the approach just described. The main points of similarity are that they both (1) build the network architecture as needed using relatively simple one layer learning algorithms at each step, and (2) use the output error signals at each step. However, the way in which neurons are added to the network differs, since in cascade-correlation, both the “f” and “ X neurons have adaptable links, which are learned in a two-stepprocedure. This is different from the simpler one-step procedure here using only modifiable ‘7” synapses. The more significant difference between the two approaches is that the algorithm described above is but one of many possible implementations of supervised factorial learning. This is evident from the implementation in the following section, which bears no resemblance to cascadecorrelation. Moreover, factorial coding can be learned using algorithms such as the ”bound state” technique in Redlich (19921, which are very different from the ARCA ones used here. Also, the end product of supervised (and unsupervised) factorial learning offers the additional benefit of actually learning an approximation to the probablities P ( u ) and P(u, s). In terms of performance, only the Parity problem has been tried using both cascade-correlation and the above algorithm and I found that the performance of the latter is of the same order as the 325 epochs for &bit Parity, with both approaches showing an improvement over backpropagation. However, it is in the implementation below that I find supervised factorial learning shows the greatest improvement over backpropagation (e.g., a reduction from 75,000 to 50 learning steps for the Symmetry problem). 4.2 Positive Example Learning Approach. Just as the Addition problem illustrates the usefulness of exploiting redundancy within s, there are many problems where redundancy in u can be exploited. Such problems


761

include the Symmetry and T,C problems mentioned above, as well as the Shifter problem in Hinton and Sejnowski (1986), and any one- or twodimensional translation problem-to save space, only the Symmetry and T,C solutions are given. These all use a representation u that is highly redundant. They are also closest to the type of problems the brain needs to solve. One way to fully exploit the redundancy in u, while avoiding the need for serial processing, is to build separate ARCA networks N* for each possible s*, with each network in parallel calculating its P(uls*). After this fully parallel processing, the outputs of the different N' can be compared to see which has the largest P(s*]u)for a given u. In practice, see below, the comparison can be as simple as seeing which of the N' responds least to the input u. The learning process can be further simplified by calculating P(u Is*) = P(u,s*)/P(s*) in place of P(u,s*).The P(s*) can be calculated separately; in the examples here it can be completely ignored since for them P(s*) is flat. The beauty of having the N' calculate P(uls*)is that then the learning set consists only of positive examples of each concept S* (as in "maximum likelihood" learning, in contrast to the discriminative learning of Section 4.1). Each N* therefore learns the statistical structure of members of its s*, and in this way learns what makes S* unique. 4.2.1 The Symmetry Problem. To illustrate this type of learning, I return to the Symmetry problem defined in Section 4.1.1. Since P(s* = llu) = 1-P(s* = Olu) only a single network that calculates P(uls* = 1) is needed, as shown in Figure 5. Here the learning set consists of only the z3 = 8 positive examples of symmetric inputs; the 64 - 8 negative examples are ignored. As a result the f i in Figure 5 learn to factorize P(uls*) in fewer than 50 total steps as opposed to the 75,000 used in Rumelhart et al. (1986). Following this learning, the ARCA solution is ub = ui = uh = 0 for all symmetric patterns, while u: = ui for i > 2. This is because knowing that a pattern is symmetric allows the network to predict and thus shut down UO,ul1u2 based on 243, ~4,145.On the other hand, since the network always predicts symmetry, it makes "false" predictions when shown nonsymmetric patterns. Then at least one of the ubl ui,uh will be turned on, with more on as the pattern deviates more from symmetry. So if any of the ui = 1 for i 5 2 we know the pattern must be nonsymmetric. 4.2.2 Generalization and Multiple Networks. In the Symmetry problem the network in Figure 5 correctly identifies as nonsymmetric all 56 patterns it has never seen. This is because the Symmetry problem is somewhat special: for it the learning set contains all possible values of the subset of inputs u3, u4,1.45, used to predict uo,ul, u2. Therefore, even for nonsymmetric patterns the network makes unambiguous predictions for

A. Norman Redlich

762

u'o

uo

U'l

u'2

u1

u2

u'3

u3

u'4

U'J

u4

us

s=l

Figure 5 ARCA network for positive example learning of the Symmetry problem. Knowing that patterns are symmetric allows the network to use the values of ~ 3 ~ ~ to4 predict , ~ s and completely shut off U O , U ~ , for U ~ symmetric patterns. For nonsymmetric patterns, which the network has never seen, it correctly generalizes by turning on at least one bit of UO,u1 ,u2. uo,u l , u2. In this sense the learning set is complete with respect to this

particular ARCA network. Problems that are complete in this sense are not difficult to find: they include the Shifter problem of Hinton and Sejnowski (1986). On the other hand, when the learning set-even if it includes all members of s*-is not complete for a specific ARCA network then some nonmembers of the concept s* may be falsely identified as members of s*: false-positive errors. However, such 'false-positive errors are rare. For example, in the symmetry problem to falsely identify a nonsymmetric input as symmetric the network would need to "correctly" predict and shut down all three UO,u1, u2. One way to reduce falsepositive errors, which works extremely well, is to use more than one ARCA network for each concept and then average the P(uls*). The different ARCA networks, using the same learning set, rarely make the same generalization errors because each seeks out different sources of redundancy in u. 4.2.3 The T,CProblem. A relatively simple example where falsepositive errors can occur is the classical T,C problem in Rumelhart et al. (1986) (T and C can occur at any location and at one of four rotations). This will


763

@)

Figure 6 Examples of some two-dimensional ARCA networks. The crosshatched inputs are used to predict the black input in a way that guarantees reversibility.

be discussed at the end of this section after I first explain how factorial learning is applied in this case. For two-dimensional problems there exist a greater variety of ARCA networks, some of which are shown in Figure 6. In Figure 6a the two-dimensional array at each learning stage is divided into ”even” and “odd subarrays: a checkerboard. At each learning stage either the odd inputs are predicted by their even neighbors or vice versa. Another option is Figure 6b where all inputs in the lower left quadrant are used for the prediction; any other quadrant can also be used. Also all inputs to one side of a half plane drawn through the predicted input may be used as in Figure 6c. In every case these arrangements ensure reversibility. For the T,C problem only a P(ulT) network is needed since what is not a T must be a C. The learning set then consists only of positive examples of the letter T. Initially the letter T has five bits on as shown in Figure 7a. However, some of these bits are redundant, since knowing that the letter is T, no more than three bits are needed to indicate position and orientation. For example, by learning in stages (it took 5 to 6 stages) using checkerboard networks (Fig. 6a), the number of bits on was reduced to three as shown for the T in Figure 7e, with corresponding results for the other three rotations. [Fewer bits on indicates higher P(ulT) because P(ulT) is given by the product of individual bit probabilities, and P(bit on) . A sample is a set of examples. The examples are assumed to be generated randomly according to some unknown probability distribution D that can be any member of the family 2) of all product distributions. Distribution D belongs to V if and only if the setting of each input variable xi is chosen independently of the settings of the other variables. The uniform distribution, where each xi is set independently to f l with probability 1/2, is a member of 2). We denote by P(A) the probability of event A and by @ ( A )its empirical estimate based on a given finite sample. All probabilities are taken with respect to the product distribution D on I". We denote by E(r) and Var(x) the expectation and variance of the random variable x. If a, b E {-l,+l}, we denote by P(g = blxi = a ) the conditional probability that g = b given the fact that xi = a. The injuence of a variable xi, denoted Inf(xi), is defined as Inf(xi) = P ( g = +I(xi = +I) - P(g = +Itxi = -1) - P(g = -1 IXi = +1) P(g = -1 [ X i = -1)

+

(2.2)

Intuitively, the influence of a variable is positive (negative) if its weight is positive (negative). 3 PAC Learning Single Binary Percepbons

3.1 The Learning Model. In this section, we adopt the PAC learning model (Valiant 1984; Blumer et al. 1989). Here the methodology is to draw, according to D, a sample of a certain size labeled according to an unknown target perceptron g and then to find a "good" approximation g' of g . The error of the hypothesis perceptron g', with respect to the target g , is defined to be P(g' # g) = P[g'(x) # g ( x ) ] ,where x is drawn according to the same distribution D used to generate the training sample. An algorithm PAC learns from examples the class G of binary perceptrons, under a family 2, of distributions on I", if for every g E G, any D E '0, and any 0 < ~ , < 6 1, the algorithm runs in time polynomial in (n,l/q 1/6) and outputs, with probability at least 1 - 6, an hypothesis g' E G that makes an error at most 6 with g . 3.2 The Learning Algorithm. We assume that the examples are generated according to a (unknown) product distribution D on { -1, +1}" and labeled according to a target binary perceptron g given by equation 2.1. The learning algorithm proceeds in three steps: 1. Estimating, for each input variable xi, the probability that it is set to +l. If this probability is too high (too low), the variable is set to +1 (-1). Note that setting a variable to a given value is equivalent to neglecting this variable because any constant can be absorbed in the threshold.

Mostefa Golea and Mario Marchand

770

2. Estimating the weight values (signs). This is done by estimating the influence of each variable. 3. Estimating the threshold value.

To simplify the analysis, we introduce the following notation. Let y be the vector whose components yi are defined as (3.1)

yi = Wi X Xi

Then equation 2.1 can be written as

In addition, we define Inf(yi) by Inf(yi) = P(g = +llyi = +1) - P(g = +llyi = -1) - P(g = -qyi = +1) + P ( g = -11yi = -1)

(3.3)

Note that if D ( x ) is a product distribution on {-1, +1}", then so is D ( y ) .

Lemma 1. Let g be a binary perceptron. Let Xi be a variable in g . Let a E (-1, +l}. Let g' be a perceptron obtained from g by setting xi to a. Then, if P(xi = -a) 5 c/2n,

Proof. Follows directly from the fact that P(g # g') 5 P(Xi = -a).

0

Lemma 1 implies that we can neglect any variable xi for which P(xi = f l ) is too high (too low). In what follows, we consider only variables that have not been neglected. As we said earlier, intuition suggests that the influence of a variable is positive (negative) if its weight is positive (negative). The following lemma strengthens this intuition by showing that there is a measurable gap between the two cases. This gap will be used to estimate the weight values (signs).

Lemma 2. Let g be a perceptron such that P ( g = +l),P(g = -1) > p, where 0 < p < 1. Then for any product distribution D, Inf(xi)

{

>
p. The weight values obtained in the previous step define the weight vector of our hypothesis perceptron g’. The next step is to estimate an appropriate threshold for g’, using these weight values. For that, we appeal to the following lemma. Lemma 3. Let g bea perceptron with a threshold 8. Let gbea perceptron obtained from g by substituting r for 6 . Then, if r 6 8,

P ( g # g’) I 1 - P(g = +1y = +1) Proof.

P(g # 8’) 5 P(g = -1Ig’ = +1)+ P(g = +llg’ = -1) = 1 - P(g = +1(g’ = +1) + P(g = +l(g’ = -1) = 1 - P(g = +llg’ = +1) The last equality follows from the fact that P(g = +11g’ = -1) = 0 for T

5 e.

So, if we estimate P(g = +Ilg‘ = +1) for r = -n - 1,-n, -n + 1,.. . and then choose as a threshold for g‘ the least r for which P(g = +lid = +1) 2 (1- c), we are guaranteed to have P(g # g‘) I c. Obviously, such an t exists and is always 5 0 because P(g = +llg‘ = +1) = 1 for r = 8. A sketch of the algorithm for learning single binary perceptrons is given in Figure 1.

Theorem 1. The class of binary perceptrons is PAC learnable under the family of product distributions.

Learning Perceptrons with Binary Weights

773

Algorithm LEARN-BINARY-PERCEPTRON(n,e,6) Parameters: n is the number of input variables, 6 is the accuracy parameter and 6 is the confidence parameter. Output: a binary perceptron g' defined by a weight vector (w1,... ,w,) and a threshold Y. Description: 1. Call m =- 1 In examples. This sample will be used to estimate the different probabilities. Initialize g' to the constant perceptron -1.

2. (Are most examples positive?) If P(g = +1) L (1- f) then set g' = 1 and return g'. 3. (Are most examples negative?) If P(g = +1) 5 f then set g' = -1 and return g'. 4. Set p =

4.

5. (1s P(Xi = +1) too low or too high ?) For each input variable xi: (a) Estimate P(xi = +l). (b)

If + ( X i = +1) 5 f or 1-P(xi = +1) 5 f , neglect this variable.

6. (Determine the weight values) For each input variable xi:

(a) If ~ ; l f ( x i>) f&, (b)

set wi = I.

Else if I;lf(xi) < -;A, set wi = -1.

(c) Else set wi = 0 (xi is not an influential variable). 7. (Estimatingthe threshold) Initialize Y (the threshold of g') to -(n+l).

(a) Estimate P(g = +llg' = +l).

If P(g = +llg' = +1) > 1 - &, go to step 8. (c) Y = Y + 1. Go to step 7a.

(b)

8. Return g' (that is ( w l , . . ,w,; Y)).

Figure 1: An algorithm for learning single binary perceptrons on product distributions.

Proof. Using Chernoff bounds (Hagerup and Rub 1989), one can show that a sample of size rn = [160n(n l)]*/e4 ln32n/6 is sufficient to ensure that

+

0

IP(g = a ) - P(g = u)1 5 4 4 with confidence at least 1 - 6/2.


774 lP(Xi

0

0

= a ) - P(Xi = a)l

5 c/4n with confidence at least 1 - 6/4n.

IInf(xi)- Inf(xi)l 5 4 4 ( n + 1) with confidence at least 1 - 6/8n.

IP(g = +l(g‘ = +1) - P(g = +llg‘ = +I)( 5 c/4 with confidence at least 1 - 6/16n.

Combining all these factors, it is easy to show that the hypothesis g‘ returned by the algorithm will make an error at most E with the target g, with confidence at least 1 - 6. Since it takes m units of time to estimate a conditional probability using a sample of size m, the running time of the algorithm will be of O(m x n).

4 Reduction to the Clipped Hebb Rule

The perceptron with binary weights and zero-threshold has been extensively studied by many authors (Krauth and Mezard 1989; Kohler et al. 1990; Opper et al. 1990; Venkatesh 1991). All these studies assume a uniform distribution of examples. So, we come to ask how the algorithm of Figure 1 relates to the learning rules proposed previously To answer this, let us first rewrite the influence of a variable as

and observe that under the uniform distribution, P(Xi = +1) = P(Xi = -1). Next, we notice that in the algorithm of Figure 1, each weight wi is basically assigned to the sign of Inf(xi). Hence apart from E and 6, the algorithm can be summarized by the following rule: wj

=

sgn [myxi)]

=

sgn Cg(x”)xr

r

lu

1

I

(4.1)

where sgn(x) = +1 when x > 0 and -1 otherwise and xiy denotes the ith component of the vth training example. Equation 4.1 is simply the well known clipped Hebb rule (Opper et al. 19901, also called the majority rule in Venkatesh (1991). Since this rule


775

is just the restriction of the learning algorithm of Figure 1 for uniform distributions, Theorem 1 has the following corollary:

Corollary 1. The clipped Hebb rule PAC learns the class of binary perceptrons with zero thresholds under the uniform distribution. 5 Average Case Behavior in the Limit of Infinite n The bound on the number of examples needed by the algorithm of Figure 1 to achieve a given accuracy with a given confidence is overly pessimistic. In our approach, this overestimate can be traced to the inequalities present in the proofs of Lemmas 2 and 3 and to the use of the Chernoff bounds (Hagerup and Rub 1989). To obtain the typical behavior of the algorithm we will calculate analytically, for any target perceptron, the average generalization rate (i.e., the learning curve). By generalization rate we mean the curve of the generalization ability as a function of the size of the training set m. The central limit theorem will tell us that the average behavior becomes the typical behavior in the limit of infinite n and infinite m with Q = m / n kept constant. As it is generally the case Wallet 1989; @per et al. 1990; Opper and Haussler 19911, we limit ourselves, for the sake of mathematical simplicity, to the case of uniform distribution and zero threshold. Therefore, we will calculate the average generalization rate of the clipped Hebb rule (hereafter CHR) (equation 4.1)for both noise-free and noisy examples. 5.1 Zero Noise. Let w t= (wi,w i , . . . ,w;) be the target weight vector and let w = (w,,wz, . . .,w,) be the hypothesis weight vector constructed by the CHR with m training examples. The generalization rate G is defined to be the probability that the hypothesis agrees with the target on a random example x chosen according to the uniform distribution. Let us start by defining the following sums of random variables:

x

n

=

cwixi

(5.1)

i=l

n

Y =

CWfXi

(5.2)

i=l

The generalization rate is given by

G = P [sgn(X)= sgn(Y)] = P[XY>O]

(5.3) (5.4)

where we have assumed w.1.o.g. that n is an odd number. Since x is


776

distributed uniformly, we easily find that

E(X) = E ( Y ) = O Var(X) = Var(Y) = n

(5.5) (5.6)

n

nxp

=

E(XY) = C w i w f

(5.7)

i=l

where -1 5 p 5 +1 is defined to be the normalized overlap between the target and the hypothesis weight vector. According to the central limit theorem, in the limit n + 00, X and Y will be distributed according to a bivariate normal distribution with moments given by equations 5.5, 5.6, and 5.7. Hence, for fixed w tand w,the generalization rate G is given by

where the joint probability distribution p(x,y) is given by

This integral easily evaluates to give 1 G(p) = 1 - -arccosp lr

(5.8)

So, as n -+ 00, the generalization rate depends only on the angle between the target and the hypothesis weight vectors. Now, to average this result over the all training samples of size m, we argue that for large n, the distribution of the random variable p becomes sharply peaked at its mean. Denoting the average over the training samples by >, this amounts to approximating > by G(>) as n -+ 00. Using equation 5.7, we can write (for a fixed w9:

1 "

= -CWf>

n i=l

(5.9) (5.10)

where pi is the probability that wlwi = +l. We introduce the independent random variables = wfxr, and use equation 4.1 to write: (5.11)


777

Let us define the new random variables $'

(5.12) With that, pi can be written as

(5.13) Let q be the probability that q/'= +l. From equation 5.12, we can write 9 as

n-l-k

1

1

(5.14)

asnjoo

5+=

=

where, by using Stirling's formula, we have kept only the leading term in l / f i as n 00. Hence, in this limit, each T$' has unit variance and a mean of 2 / 6 . Since 7; and $"" are statistically independent, the central limit theorem tells us that, when m 00, the variable --$

-+

m

Z=C# V=l

becomes a gaussian variable with mean pZ = m x and variance m. Hence, as m 00 and a = m/n is kept constant, equation 5.13 becomes -+

(5.15) - 1 +-erf(&) 1

- 5

2 Hence, using equations 5.8, 5.10, and 5.16, we have finally:

= erf(m)

=

1

(5.16)

(5.17) (5.18)

This result is independent of the target w'. The average generalization rate and normalized overlap are plotted in Figure 2 and compared with numerical simulations. We see that the


778

1.o

I

03 0.8 0.7 0.6

0.5 I 0

5

10

15

20

a 1.o

g

1

0.8

0.8 0.4

f

0.2

' t

0.0

0

5

10

15

20

a

Figure 2: (a) The average generalization rate > and (b) the average normalized overlap > as a function of normalized number of examples (1 = m/n. Numerical results are shown for n = 50,100,500. Each point denotes an average over 50 different training samples and the error bars denote the standard deviations. agreement with the theory is excellent, even for moderate values of n. Notice that the agreement is slightly better for > than it is for >. This illustrates the difference between > and G(
). To compare this average-case analytic result to the bounds given by PAC learning, we use the fact that we can bound erf(z) by an exponential (Abramowitz and Stegun 1972) and thus bound the error rate 1 - > = by (5.19)


779

That is, the error rate decreases exponentially with the number of examples and, on average, a training set of size of O[nln(l/e)] is sufficient to produce an hypothesis with error rate 6. This is an important improvement over the bound of O [ ( r ~ /ln(n/b)] e ) ~ given by our PAC learning analysis. Thus, the CHR is a striking example of a very simple “inconsistent”algorithm that does not always produce hypotheses that agrees with all the training examples, but nonetheless produce hypotheses with outstanding generalization ability. Moreover, the exponential convergence outlines the computational advantage of learning binary perceptrons using binary perceptrons. In fact, if one allows real weights, no algorithm can outperform the Bayes optimal algorithm (Opper and Haussler 1991). The latter’s error rate improves only algebraically, approximately as 0.44/(r. On the other hand, for consistent learning rules that produce perceptrons with binary weights, a phase transition to perfect generalization is known to take place at a critical value of a (Sompolinsky et al. 1990; Gyorgyi 1990). Thus, these rules have a slightly better sample complexity than the CHR. Unfortunately, they are much more computationally expensive (with a running time that generally increases exponentialIy with the number of inputs n). Since it is an “inconsistent” learning rule, the CHR does not exhibit a phase transition to perfect generalization. We think that the exponential convergence is the reminiscence of the “lost” phase transition. An interesting question is how the CHR behaves when learning binary perceptrons on product distributions. To answer this, we first note that the CHR works by exploiting the correlation between the state of each input variable xi and the classification label (equation 4.1). Under the uniform distribution, this correlation is positive if wf= +1 and negative if wf = -1. This is no more true for product distributions: one can easily craft some malicious product distributions where, for example, this correlation is negative although wi = +l. The CHR will be fooled by such distributions because it does not take into account the fact that the settings of the input variables do not occur with the same probability. The algorithm of Figure 1 fixes this problem by taking this fact into consideration, through the conditional probabilities. Finally, it is important to mention that binary perceptrons trained with the CHR on examples generated uniformly will perform well even when tested on examples generated by nonuniform distributions, as long as these distributions are reasonable [for a precise definition of reasonable distributions, see Bartlett and Williamson (1991)l. 5.2 Classification Noise. In this section we are interested in the generalization rate when learning from noisy examples. We assume that the classification label of each training example is flipped independently with some pmbability 0 . Since the object of the learning algorithm is to construct an hypothesis w that agrees the most with the underlying tar-


780

get wfl the generalization rate G is defined to be the probability that the hypothesis agrees with the noise-free target on a new random example x. The generalization rate for a fixed w and W*is still given by equation 5.8. To calculate the effect of noise on >, let us define q' as the probability that $' = +1 in the presence of noise whereas q denotes this probability in the noise-free regime (i.e., equation 5.14). These two probabilities are related by:

+

q' = q(1- a) (1 -q)o 1 I-2a = +asn-,oo

2Jz?nz

(5.20) (5.21)

where we have used equation 5.14 for the last equality. This leads to the following expressions for the normalized overlap and the generalization rate, in the presence of noise:

>

= erf

>

= 1-

(5.22) lr

(5.23)

One can see that the algorithm is very robust with respect to classification noise: the average generalization rate still converges exponentially to 1 as long as a < 1/2. The only difference with the noise-free regime is the presence of the prefactor (1 - 20). The average generalization rate for different noise levels a is plotted in Figure 3. We see that the numerical simulations are in excellent agreement with the theoretical curves. 6 Summary

We have proposed a very simple algorithm that PAC learns the class of perceptrons with binary weights and arbitrary threshold under the family of product distributions. The sample complexity of this algorithm is of O[(n/~)~ln(n/S)] and its running time increases only linearly with the sample size. We have shown that this algorithm reduces to the clipped Hebb rule when learning binary perceptrons with zero threshold under the uniform distribution. We have calculated exactly its learning curve in the limit n oo where the average behavior becomes the typical behavior. We have found that the error rate converges exponentially to zero and have thus improved the sample complexity to O[nl n ( l / ~ ) ]The . analytic expression of the learning curve is in excellent agreement with the numerical simulations. The algorithm is very robust with respect to random classification noise. --$


781

--

o noise 0.0 noise 0.1 A nobe- 0 2 nolse- 0.4 Theory: nobe-0.0 Theory: nolse-0. I Theory: nobe-0.2 Theory: nohe-0.4

-

-.-.-

---*-a-

a

5

10

15

20

25

30 a

Figure 3: The average generalization rate > for different noise levels u. Numerical results are shown for n = 100. Each point denotes the average over 50 different simulations (i.e., 50 different noisy training sets). The error bars (indicated only for 0 = 0.4 for clarity) denote the standard deviations.

Acknowledgments

This work was supported by NSERC grant OGP0122405. References Abramowitz, M., and Stegun, I. A. 1972. Handbook of Mathematical Functions. Dover, New York. (eq. 7.1.13). Barkai, E., and Kanter, I. 1991. Storage capacity of a multilayer neural network with binary weights. Europhys. Lett. 14,107-112. Bartlett, P. L., and Williamson, R. C. 1991. Investigating the distribution assumptions in the PAC learning model. In Proceedings of the 4th Workshop on Computational Learning Theory, pp. 24-32. Morgan Kaufmann, San Mateo, CA. Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, K. 1989. Learnability and the Vapnik-Chervonenkis dimension. 1.ACM, 36,929-965. Gyorgyi, G. 1990. First-order transition to perfect generalization in a neural network with binary synapses. Phys. Rev.A 41, 7097-7100. Hagerup, T., and Rub, C. 1989. A guided tour to Chernoff bounds. Info. Proc. Lett. 33, 305-308. Ibragimov, I. A. 1956. On the composition of unimodal distributions. Theory Prob. Appl. 1,255-260.

782


Kohler, H., Diederich, S., Kinzel, W., and Opper, M. 1990. Learning algorithm for a neural network with binary synapses. Z. Phys. B 78,333-342. Krauth, W.,and Mezard, M. 1989. Storage capacity of memory networks with binary couplings. I. Phys. France 50,3057-3066. MacDonald, D.R. 1979. On local limit theorems for integer-valued random variables. The0y Prob. Statistics Acad. Nauk. 3, 607-614. Meir, R.,and Fontanari,J. F. 1992. Calculation of learning curves for inconsistent algorithms. Phys. Rev. A 92,8874-8884. Opper, M., and Haussler, H. 1991. Generalizationperformance of Bayes optimal classification algorithm for learning a perceptron. Phys. Rev. Lett. 66, 26772680. Opper, M., Kinzel, W., Kleinz, J., and Nehl, R. 1990. On the ability of the optimal perceptron to generalize. I. Phys. A: Math. Gen. 23,L.5814586. Pitt, L., and Valiant, L. G. 1988. Computational limitations on learning from examples. I. ACM 35,965-984. Sompolinsky,H.,Tishby, N., and Seung, H. S. 1990. Learning from examples in large neural networks. Phys. Rev. Lett. 65, 1683-1686. Vallet, F. 1989. The Hebb rule for learning linearly separable Boolean functions: Learning and generalization. Europhys. Lett. 8, 747-751. Valiant, L. G. 1984. A theory of the learnable. Cornm. ACM 27,1134-1142. Venkatesh, S. 1991. On learning binary weights for majority functions. In Proceedings of the 4th Workshop on Computational Learning Theoy, pp. 257-266. Morgan Kaufmann, San Mateo, CA. Received 29 July 1992; accepted 26 January 1993.

This article has been cited by: 2. Shao C. Fang, Santosh S. Venkatesh. 1999. Learning finite binary sequences from half-space data. Random Structures and Algorithms 14:4, 345-381. [CrossRef]


Construction of Minimal n-2-n Encoders for Any n D. S.Phatak H.Choi I. Koren Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA 01003 USA

The encoding problem (Rumelhart and McClelland 1986) is an important canonical problem. It has been widely used as a benchmark. Here, we have analytically derived minimal-sized nets necessary and sufficient to solve encoding problems of arbitrary size. The proofs are constructive: we construct n-2-n encoders and show that two hidden units are also necessary for n > 2. Moreover, the geometric approach employed is general and has much wider applications. For example, this method has also helped us derive lower bounds on the redundancy necessary for achieving complete fault tolerance (Phatak and Koren 1992a,b). 1 Introduction

The encoding problem is an important canonical problem for neural networks (Rumelhart and McClelland 1986). In this problem, a set of orthogonal input patterns is mapped onto a set of orthogonal output patterns through a (small) set of hidden units. Typically, the inputs and outputs are assumed to be binary. There are n input units, n output units, and rn hidden units where m = log,n. The hidden units are generally arranged in a single layer resulting in three layers of units. There are n input/output patterns. The hidden units are expected to form some sort of compact code for each of the patterns. Henceforth, we refer to an encoding problem of size n by the acronym n x n problem and a net for a problem of this size that has m hidden units as an n-m-n encoding net. The inputs and outputs of the units are continuous valued. That raises the question: are log, n hidden units necessary to solve an n x n problem? If fewer units can do the job, what is the minimum number of units needed for an n x n encoding problem? We have analytically derived this minimum number of hidden units and established the capabilities of n-m-n encoding nets. The next section describes the topology and states the assumptions. Section 3 presents and proves the results on the bounds and related parameters. The following sections present discussion and conclusion. Neural Computation 5,783-794 (1993) @ 1993 Massachusetts Institute of Technology

D. S. Phatak, H. Choi, and I. Koren

704

Output layer, n

units

Hidden layer, rn units

Inputlayer, n units

Figure 1: An n-m-n encoding net.

2 Topology

The network is arranged into three layers as shown in Figure 1. Every unit in a layer feeds all other units in the next layer. There are no layerskipping connections. Besides the incoming weights, each unit (in the hidden and output layers) has one more independently adjustable parameter, that is, threshold or bias. The units are assumed to be sigmoidal and the output of the ith unit is given by Outputi = S(resultuntinputi)

-

and 1 +e+ resultuntinputi = netinputi - biasi S(u) =

netinputi

=

where

1

Cwpj

and (2.1)

j=1

Here, r is the number of units that feed unit i and Wij is the weight of the link from unit j (sender) to i (receiver). The output is considered to be on or at logical level “1” if it is greater than or equal to 0.50, and off or at level “0” if it is less than 0.50. The input patterns are the rows of n x n identity matrix. The target outputs are identical to the inputs, that is, the hidden units are expected to simply replicate the input pattern onto the output layer. The hidden layer encodes each of the n patterns with rn < n units and the output layer decodes the compact codes developed by the hidden units back to the original patterns.

Construction of Minimal n-2-n Encoders

785

(4)

Figure 2: A 2-1-2 encoding net with weights and biases zoi and bi > 0 for all i. Unit indices are shown in parentheses. 3 Results

With the above topology and assumptions we now proceed to state the following results. Theorem 1. An encoding net with one single hidden unit (i.e., m = 1 ) can learn at most 2 x 2 encoding problem.

Proof. That it can learn 1 x 1 and 2 x 2 problems can be demonstrated by giving an example. In Figure 2, a 2-1-2 net is illustrated along with all the weights and biases. Unit numbers are shown in parentheses and the bias values are indicated inside the circles representing the units. Units 4 and 5 constitute the input layer and 1 and 2 belong to the output layer. It can be verified that w3 = w4 = bl = b2 = 5.0; w 1 = w2 = 10.0 along with the signs indicated in the figure lead to correct reproduction of the two input patterns (viz. (1,O) and (0,l)) at the output layer. This is one of the infinitely many sets of weight and bias values that lead to correct outputs.


786

We now prove that it is impossible to reproduce 3 x 3 patterns using only one hidden unit. Here, the hidden unit must have three distinct outputs, one corresponding to each of the three input patterns, otherwise the output units cannot distinguish between those patterns that map onto the same output value of the hidden unit. Denote the three distinct outputs of the hidden unit as 01, 02, and 03, respectively, where without loss of generality, 01 > 02 > 03. Let the weights from the hidden unit to the output units be wl,W Z , and w3 and biases of the output units be fll, 02, and fls, respectively. Then, the resultant input to the ith output unit (denoted by yi) is given by yi = WiX - fli

where

and

i = 1,2,3

x = 01,02,03

(3.2)

Here, x denotes the output of the hidden unit. Note that the functions fi(x) =

x =

S[Yi(x)]=

1 1 + c-(wix - fli)

where i = 1 , 2 , 3 and (3.3)

o1,02,03

are monotonic. Without loss of generality, the input patterns are assumed to be {l,O,O}, {O,l,O}, and {O,O,l}. These same patterns should be reproduced at the output, which implies "1";

fl(0,)

=

fi(o1)

> 0.5;

fz(o1) = "0"; fz(oi) < 0.5;

fi(o2)

= "0";

fi(02)

< 0.5; "1"; > 0.5;

fz(o2) =

fz(0z)

f3(01)

= "0";

fs(o2) =

f3(01)

< 0.5;

f3(oz)

"0";

< 0.5;

fl(o3) fi(03)

= "O",

f2(03) = fZ(o3)

"O",

(3.4)

that is,

< 0.5 "l", > 0.5

f3(03) = f3(o3)

that is,

< 0.5

(3.5)

that is, (3.6)

From equation 3.3 it is seen that constraints 3.4 and 3.6 can be satisfied since they obey monotonicity. Constraints 3.5, however, cannot be satisfied since the function on the left-hand side is monotonic while the required outputs on the right-hand side are not monotonic. It can be verified that for any permutation of input patterns and output values, the constraints on one of the three units are impossible to satisfy since the inputs to that unit are monotonic but the target outputs are not monotonic. Thus the 3 x 3 problem cannot be solved by just one hidden unit. The proof for the n x n sized problem with n > 3 is identical to the above proof for the 3 x 3 case. 0 There is a geometric interpretation of the above result that is illustrated in Figure 3. This interpretation is critical for the proof of the next theorem, which establishes a bound for the general n x n problem. For a 2-1-2 net, the output of the hidden unit corresponding to each of the (input) patterns can be represented by a point along one dimension or a line. Without loss of generality, choose that line to be the x axis. Then,


787

Figure 3: A geometric interpretation of the 2-1-2 encoding problem.

the output of the hidden unit corresponding to each of the two input patterns is a point between [0,1] on the x axis, as illustrated by points PI and P2 in Figure 3. Because of the one-to-one mapping from the input patterns to the points representing the outputs of the hidden unit, the symbols PI and P2 will also be used to refer to the patterns. The resultant input to the ith unit is given by equation 3.2, where i = 1 , 2 and wi and Bi are the weight and bias associated with the ith unit. Note that these equations represent straight lines (hyperplanes in general) in the x-y plane, as illustrated by lines I1 and 12 in Figure 3. Henceforth, we just use the labels 1 and 2 to refer to the output units as well as the corresponding lines (hyperplanes) implemented by the units. A point xo is considered to be on the positive side of the line y = wx - 0 if wxo- 0 > 0, and on the negative side of the line if wxo- 0 < 0. For example, in Figure 3, all points (on the x axis) to the right of point Q are on the positive side of line lI and on the negative side of line 12. The vertical distance PIA between point PI and the line l~ represents the resultantinput to output unit 1 for pattern PI. Similarly, distance PIB represents the resultantinput to unit 2 for pattern PI. It is useful to think of directed distances from the points PI,P2 to lines 1 1 , l ~ .If the direction is upwards (along +y axis), then the corresponding resultantinput is positive (i.e., the output of the unit is "l"), while a downward distance (along -y axis) implies a negative resultantinput ("0" output). For the patterns (points) on the positive side of the line, the resultant input to the corresponding unit is positive and the unit output is on or "1." Conversely, a unit is on only if the pattern lies on the positive side of the line it implements. Similarly, a unit is of if and only if the pattern lies on the negative side of the line corresponding to the unit.


788

Learning implies finding weights and biases that satisfy the constraints y1(01) > 0;

y1(02) < 0;

yz(01) < 0;

Yz(O2)

>0

(3.7)

The first two inequalities say that points PI and P2 must be on positive and negative sides of line 11, because unit 1 should be on for pattern 1 and off for pattern 2. The interpretation of the last two inequalities is similar. Together, the constraints imply that both lines II and 12 intersect the x axis between P1and P2 and that one of them has a positive slope and the other has a negative slope. Figure 3 illustrates a case where the points P I ,P2 and lines II, I2 satisfy the above constraints. In this figure, both I1 and 12 intersect the x axis at the same point Q. In general, this may not be the case, as long as the constraints are satisfied. In general, learning implies constraints similar to equation 3.7. The constraints are such that 1. An output unit is on for only one pattern. This means that the weight(s1 and bias associated with that unit define a hyperplane that has only one of the points Pi is on its positive side, all others are on its negative side. 2. Each point Pi is such that for the corresponding input pattern, only one output unit is on and this unit stays off for all other input patterns. This means that each of the points Pi it is on the positive side of exactly one hyperplane and on the negative side of all others.

In Figure 3, PI is on the positive side of only one line viz. lI and P2 is on the positive side of only one line viz. 12. Similarly line lI has only one point on its positive side viz. PI and line l2 has only one point on its positive side viz. P2. For the n x n encoding problem, it may be expected that the minimum number of hidden units required is a function of n. Contrary to this expectation, however, it turns out that only two hidden units are sufficient to solve any n x n problem for arbitrarily large n. Theorem 2. Only two hidden units are suflcient to encode and decode n x n puttems for any positive integer n.

Proof. We prove this by a geometric construction similar to the one illustrated above for the 2-1-2 case. Here the network is n-2-n, that is, there are n input units, 2 hidden units, and n output units. For each input pattern, the hidden units develop outputs that can be represented by a distinct point in the x-y plane, where the x coordinate denotes the output of the first hidden unit and the y coordinate denotes the output of the second hidden unit. These points are denoted by Pi, i = 1,2. . .n. The hidden units feed all the output units. Let the weight associated with the link between hidden unit 1 and output unit i be denoted by w,!. The weight from hidden unit 2 to output unit i is denoted by 4. Let the


789

bias of the output unit i be denoted by Bi. Then, the resultant input to the ith output unit (denoted by zi) is given by ti

=

w:x+z$y-Bi

(x,y) = (o;,o;),...,(oM

where

i = 1,...,n

and (3.8)

Here, x and y correspond to the axes or dimensions representing the outputs of the hidden units, and z represents the dimension that corresponds to the resuZtantinput to the output units. These equations represent (hyper) planes in the three-dimensionalspace that will henceforth be denoted by lli where i = 1,. . . ,n. These planes are the decision surfaces implemented by the corresponding units. We say that a point (x0,yo) is on the positive side of plane I& if

z0 = w:x0 +$yo

-

ei > o

(3.9)

and on the negative side if

z0 = w;x0 + z$y0 - ei < o

(3.10)

In order to map the input patterns onto the output patterns, the points pk and the planes IIi have to satisfy constraints similar to those listed above in the exposition on geometric interpretation. Once again we observe that each plane lli defines the output of one of the units in the output layer, and each of the points Pk corresponds to a pattern. An output unit is on only for one of the n patterns and of for the others. Similarly, each pattern has exactly one output unit on and all others of. These constraints can be geometrically interpreted as follows: 1. Each plane lli has only one point on its positive side, all other points are on its negative side.

2. Each point Pk is on the positive side of only one plane and on the negative side of all other planes. If there exist points P k and planes lli, i, k = 1,2, . . . , n that satisfy the above constraints, then they constitute a valid solution for the n x n problem using only two hidden units. Figure 4 shows the geometric construction that proves the existence of such solution(s). It shows a 6-24 case for the purpose of illustration, but the procedure can be applied to any n-2-n problem. As a first step toward the solution of the n-2-n problem, a regular polygon of n sides is constructed in the x-y plane. This is illustrated by the hexagon with vertices (a,b,c,d,e,f) drawn in solid linestyle in Figure 4. Next, every edge is extended beyond the vertex up to a point where it meets the extension of some other edge of the polygon, so that (isoceles) triangles are obtained on the exterior of the original polygon, with the edges of the polygon as the bases of these triangles. This is illustrated by the shaded triangles in Figure 4. Now consider the original polygon

790


Figure 4 The geometric construction to obtain the weights and biases for a 6 - 2 4 (or n-2-n in general) encoding net.

as the base of a pyramid or a cross section of the pyramid along the x-y plane. The faces of the pyramid intersect at a point directly (vertically) below (along the -2 direction) the center of the circumcircle of the polygon. In Figure 4, for example, the center of the circumcircle is labeled as V. The vertex of the hexagonal pyramid lies directly (vertically)below the point V (i-e., on a line in the -2 direction, directed into the page from point V). The n faces of the pyramid define the n planes l l i . The points Pk have to be located within the isoceles triangles on the exterior of the polygon in the x-y plane, in order to satisfy the two constraints mentioned above. One point is placed inside each triangle, as illustrated by points P I , . . . ,P6 inside the shaded triangles in Figure 4. With this construction, each plane rIi is such that only one point is on


791

its positive side and all other points are on its negative side. For example, in Figure 4, the plane I l l passing through the vertex of the pyramid and edge ab is such that only one point, viz., P1 is on its positive side while all others are on its negative side. Similarly, each point is on the positive side of exactly one plane and the negative side of all the others. In Figure 4, for example, point P2 is on the positive side of plane l l 2 only, and is on the negative side of all the other planes. Thus the points and planes satisfy all the above constraints and represent a valid solution. The outputs of all the units have to be in [0,11. This means that the entire diagram should be within the unit square in the x-y plane, which is bounded by vertices (O,O), (O,l), (l,O), (1,l). This is always possible to do since the polygon can be shrunk to any desired size so that the entire diagram can fit inside the unit square. This proves that a solution (in fact infinitely many of them) always exists for the n-2-n 0 problem and can be obtained by the above construction. 4 Discussion

The above results hold for the complementaryencoding problem (Usand 1’s are interchanged) as well. For a complementary encoding problem, the vertex of the pyramid in the above construction lies directly (vertically) above the circumcenter V, which is in the x-y plane. Also note that the I/O patterns for the complementary encoding problem are not mutually orthogonal. In the above construction, the points corresponding to the outputs of the hidden units must lie within the triangles formed on the edges of the polygon. Hence the area of the tiangles is, in a crude sense, related to the probability of finding a valid solution. The larger the area, the higher is the probability that the gradient descent will latch on to a valid solution. Note that the outputs of the hidden units are confined to be between two circles, viz., an inner circle which touches (is tangent to) each edge of the polygon and an outer circle that passes through the tips of all the tiangles on the exterior of the polygon. Both these circles are drawn in dotted linestyle in Figure 4. For a given n, the triangles have the largest area when the outer circle is as large as possible, that is, it touches the edges of unit square in the x-y plane. Hence the net is more likely to hit on this solution. This is consistent with the observation that neural nets tend to stabilize at vertices or comers of the solution space. As n 00, the circles approach each other and in the limit they coincide. This means that the volume (area in this case) of the solution space approach 0 and, therefore, the probability that the search algorithm converges to a valid solution also approaches 0, as expected. The distance (along the z direction) between the point P, and the corresponding plane n, represents the resultant input to a unit. In the limit as n + 00, the points Pi approach planes ni and the vertical distance --$

792


between the planes and the points approaches 0 as well. This means that the resultant inputs to the output units approach 0. Hence the outputs of units that are on approach 0.5 from above, that is, output values indicating a logical level “1” -+ 0.5+ and the outputs of the units that are off approach the limit 0.5 from the other side, that is, logical “0” + 0.5-. If the output tolerances are specified (for example a “1” cannot be below 0.75 and a “0” cannot be above 0.25) then, in the above construction, it is possible to find out the maximum value of n that will deliver the outputs within the desired tolerances, for a given m. Conversely, given an n, the number of hidden units, m, required to deliver the outputs within the specified tolerance can be also calculated from the above construction. If n 5 4, the “allowable” regions for the points Pi are no longer triangles since the edges of a regular polygon with n 5 4 sides when extended beyond the vertices do not intersect the extensions of any of the other edges. It should also be noted that in the above construction, the polygon need not be regular. If the polygon is not regular, however, some of the “allowable” areas shrink and the others expand. Also, the planes IIi need not intersect at the same point or need not form a pyramid, as long as the relative placement of the planes and the points satisfy the two constraints mentioned above. The unbounded allowable areas for the points Pi that arise due to n 5 4 or due to irregularity of the underlying polygon, as well as the asymmetry in allowable areas that arises when the polygon is irregular, is illustrated in Figure 5. Note that the construction remains the same in all these cases. The points Pi still have to be in the regions exterior to the polygon, and between the lines obtained by extending the edges of the polygon beyond the vertices. This is illustrated by the shaded regions in Figure 5. If the quadrilateral shown in Figure 5 was regular (i.e., it was a square), then all the “allowable” regions for the points Pi would be identical in shape and unbounded on one side. Because the quadrilateral is irregular, some allowable regions have shrunk and the others have grown. For example, the shaded region to the left of plane ll2 has shrunk from a rectangular strip unbounded on the left side, to a bounded and triangular region shown in the figure. Similarly the shaded region to the right of ll, has expanded from a rectangular strip to an unbounded quadrilateral. It seems that the symmetric solution is more fault tolerant. The reasoning is as follows. The edges and planes of the polygon can be jiggled without changing the classificationor logical output of the network. This corresponds to changing the weights and biases of the units represented by the planes. How much change is allowed in the weight and bias Values depends on n and other factors. For the symmetric solution, it is evident that whatever tolerance applies to a point or a plane also applies to all the other points or planes. In contrast, if the polygon is not regular or if the planes do not form a pyramid, then some points and planes must


793

Figure 5 The construction for the case when n 5 4 and the polygon is not regular.

be confined to smaller tolerances (smaller than the corresponding one in the symmetric case) while the others can have larger tolerance. The total amount of deviation allowed can be measured by the volume enclosed between the original positions of the planes and the extreme positions after large deviations in parameters (or faults), at which the solution (relative placement of planes and points) still satisfies the above constraints. It is conjectured that the total of such "fault-tolerance volumes" is maximum for the symmetric case, or, in other words, a symmetric solution is more fault tolerant. 5 Conclusion

Bounds have been established for the solution of the encoding problem using a feedforward network with one layer of hidden units. Existence of solution(s) is demonstrated by constructive proofs, leading to the actual solutions. The discussion reveals interesting connections to limiting cases, fault tolerance, probability of finding a valid solution and other

794


issues. The geometric interpretation is general and applicable to other problems as well. For instance, this approach was employed in Phatak and Koren (1992a,b) to derive lower bounds on the redundancy necessary to achieve complete fault tolerance for all single faults. The encoding problem directly reflects on the ability of the net to develop distributed representations among the hidden units and map them back onto localized representations on the output units. These results will possibly help to define a meaningful measure of the distributedness of representations. References Rumelhart, D. E., and McClelland, J. L. 1986. Parallel Distributed Processing, Vol. 1: Foundations. MIT Press, Cambridge, MA. Phatak, D. S., and Koren, I. 1992. Fault tolerance of feedforward neural nets for classification tasks. In Proceedings of International Ioint Conference on Neural Nets (IJCNN),Vol. 11, pp. 11386-11-391. Baltimore, MD. Phatak, D. S., and Koren, I. 1992. Complete and partial fault tolerance of feedfornard neural nets. Tech. Rep. TR-92-CSE-26, Electrical and Computer Engineering Department, University of Massachusetts, Amherst. Received 21 September 1992; accepted 12 January 1993.

This article has been cited by: 2. Elko B. Tchernev , Rory G. Mulvaney , Dhananjay S. Phatak . 2005. Perfect Fault Tolerance of the n-k-n NetworkPerfect Fault Tolerance of the n-k-n Network. Neural Computation 17:9, 1911-1920. [Abstract] [PDF] [PDF Plus] 3. Nathalie Japkowicz , Stephen José Hanson , Mark A. Gluck . 2000. Nonlinear Autoassociation Is Not Equivalent to PCANonlinear Autoassociation Is Not Equivalent to PCA. Neural Computation 12:3, 531-545. [Abstract] [PDF] [PDF Plus]

Communicated by Steven Nowlan

Recurrent and Feedforward Polynomial Modeling of Coupled Time Series Vicente Ldpez Ram6n Huerta JosC R. Dorronsoro Instituto de lngenieria del Conocimiento, Universidad Aut6noma de Madrid, 28049 Madrid, Spain

We present two methods for the prediction of coupled time series. The first one is based on modeling the series by a dynamic system with a polynomial format. This method can be formulated in terms of learning in a recurrent network, for which we give a computationally effective algorithm. The second method is a purely feedforward D-T network procedure whose architecture derives from the recurrence relations for the derivatives of the trajectories of a Ricatti format dynamic system. It can also be used for the modeling of discrete series in terms of nonlinear mappings. Both methods have been tested successfully against chaotic series. 1 Introduction

In this paper we will consider the problem of predicting the future evolution of a certain D-dimensional vector f = (fi, . . . , f ~knowing ) its past behavior. More precisely, it is assumed that f depends continuously on a given one-dimensional variable t, often taken as time, and that a certain number K of past samples f-i = f(t-i), i = 1,. . . ,K, are known up to time to, with usually equally spaced times, i.e., t-i = to - ir. It is then desired to forecast the future values of f at times t, = to jr, j = 1,. . . ,L. As stated, this problem has been widely studied and several a p proaches have been proposed for its solution (see for instance Box and Jenkins 1970; Gabor et al. 1960; Farmer and Sidorowich 1987). Of particular interest here are the so-called dynamic systems methods, which in general consider the trajectory vector f as the evolution of a dynamic system, that is, as the solution of an ordinary differential equation (ODE)

+

X

= F(x,W)

(1.1)

where F = (FI,. . . ,FD) denotes a D-dimensional function of D variables; obviously, a coupling relationship is assumed among the D individual components of x. The parameters w determine a specific realization Neural Computation 5, 795-811 (1993) @ 1993 Massachusetts Institute of Technology

Vicente Mpez, Rambn Huerta, and JoseR. Dorronsoro

796

within the functional approximation model given by F and they are to be adjusted, usually by least-squares minimization of the distance between the known past trajectory of f and a particular solution x of equation 1.1. This point of view has recently received great attention, either as stated (Eisenhammeretal. 1991) or recast as the problem of obtaining a mapping giving certain sections of the dynamical system evolution (Crutchfield and McNamara 1987). The time series prediction problem has also aroused considerable interest in the neural network research community. In particular, the training procedures for the well-known continuous Hopfield models (Hopfield 1982) require the target driven parameter fitting of ODE systems written in the Hopfield format. Although in principle this model building was done to obtain adequate mappings between static inputs and outputs, the training target values were essentially taken as constant trajectories that the underlying dynamic system tried to match. This point of view readily suggests that the same training methods can be used to obtain ODE-based recurrent networks capable of learning certain space state trajectories (Pineda 1987; Pearlmutter 1989; Williams and Zipser 1989; Toomariam and Barhen 1991). In these methods the training set contains a number of coupled trajectories, whose step i targets are clocked back to the network as step i + 1 inputs. These networks are then indeed capable of learning certain state space trajectories; on the other hand, their modeling scope seems to be limited by the somewhat restricted nature of the ODE format employed. It is clear that the predictive power of a model such as equation 1.1 will strongly depend on the ability of the chosen functional model to approximate adequately general D-dimensional functions. Here (see also L6pez and Dorronsoro 1991) we will use as a functional model a polynomial function of fixed degree. More precisely, we will replace each of the D components of the general multivalued function F in equation 1.1 by a polynomial function P on D variables with degree K, that is, Xi

= wi

+ C W ; X ~+ C wj,j2xjlxj2

...

i

il92

Other choices are possible (see, e.g., Cremers and Hubler 1987) but this format, sometimes called the universal polynomial format, is a natural one under various points of view. First, it can be viewed as a Kth order Taylor approximation to the general function F in equation 1.1, giving for reasonable F and high enough K a good numerical approximation. It also encompasses a rich family of possible differential equations (Kerner 1981). On the other hand, for D-dimensional trajectories, essentially DK+' free coefficients have to be adjusted, which for large values of D sharply limits the allowable choices of K. As a compromise we will concentrate

Polynomial Modeling of Coupled Time Series

797

here on the case K = 2, that is, the so-called Ricatti format, i j

= ui

+ c bijxj + c c@jxk i

(1.2)

E k

for which the exact number of free coefficients is then 02(0+ 1)/2+ 02 + D = D3/2 O(@). Although restricted, this format also turns out to be quite rich; for instance a number of systems with chaotic dynamics fall into its scope, such as the Lorenz, Rossler, or Henon-Heiles equations (Lichtenberger and Lieberman 1983). Also we have found that the addition to the original trajectories of a few of its own successive powers enables a Ricatti model to yield a good approximation to a general analytical F. We will study in Section 2 the target driven Ricatti modeling of a given set of coupled time series, which, as mentioned before, can be interpreted in terms of learning in a recurrent neural network. In Section 3 we will propose a purely feedforward alternative approach in terms of a network. Its particular architecture will be derived after the consideration of partial Taylor series summation for the integration of the Ricatti ODES. In Section 4 we will illustrate numerically both approaches for the modeling of ODE generated time series derived from the so called Morse equations of molecular dynamics and from the very well known Lorenz equations. We will also consider the applicability of these techniques to series generated from discrete mappings using as an example the well known Henon map. Finally, in Section 5 we will summarize the main results of the paper.

+

2 Ricatti Recurrent Modeling

We assume that the known coupled trajectories of f are given as a certain function of time f(t) for values of t on an interval [t-K, to], and that predictions are desired for the values of f on the interval [to, t ~ ]We . want to determine a set of parameters wosuch that the trajectory xo = Q(t, WO), obtained by integrating equation 1.1 for these values of w and initial conditions x ( f - K ) = f(t-K), minimizes a certain error measuring function E(f, x), which in our examples will be

The most effective numerical procedures to find the minimizing wo require gradient information. Although for the E above several methods can be devised, there is an efficient gradient computation procedure that uses the constraints on x imposed by equation 1.1. It was first proposed by Sat0 (1990a) as a teaching algorithm for recurrent Hopfield networks and then used in Sat0 (1990b) for the learning of spatiotemporal patterns by the same networks. In any case the method can also be applied to the

798

Vicente Mpez, Ram6n Huerta, and Jose R. Dorronsoro

present situation. The main idea is to add an extra, 0-valued term to the definition of E above, namely,

where the z = ( 2 1 , .. . ,ZD) is an auxiliary set of trajectories, which is to be chosen to ease the gradient computation. For a Ricatti format system equation 1.2 these differential equations for z are found to be

with end point conditions Zi(t0) = 0. The gradient is then computed by the formulas

The computational benefits of this approach are now clear: we just integrate two systems, equations 1.2 and 2.2, and then perform the @ / 2 integrals equation 2.3. The cost of obtaining the gradient in this fashion is then essentially of the order of magnitude of 03. I We conclude this section with some considerations on the numerical implementation of the above algorithm. First it must be noted that the integration of the systems equations 1.2 and 2.2 is usually numerically unstable. This is particularly so in the early phases of the minimization process, which give rise to large gradients and, therefore, to abruptly changing equations. The breakup of moderate to large intervals [ t - K , to] into smaller ones is thus needed to avoid overflow problems. There is another, deeper reason for this breakup. In our numerical simulations we always started with small, uniformly distributed initial values for w. This often gives as first approximations to F highly dissipative systems. Their solutions evolve rapidly to constant numbers, usually the mean values of F on the training interval and, therefore, independent of w. Hence, even for relatively small intervals, the procedures easily tend to fall in local minima far away from an appropriate F. Thus, to avoid this constant, 0-valued behavior of &/aW over most of the training interval, it is again necessary to use small subintervals where the dissipation does not set in completely. Finally, notice that the integration of equation 2.2 has to be performed backward in time, which requires the memory storage of the xi. Therefore, the minimization of this storage adds a third reason to break up the training trajectory. In our simulations the breakup was done as follows. We will assume that the past behavior of f is only known as a finite number K of discrete, 'In contrast, the more straightforward gradient computing procedure of taking partials with respect to w in equation 2.1 and integrating the ODES for &(f aU derived from equations 1.2 requires on the order of 06 operations.


799

equally spaced samples f-k = f(t-k), k = 1,. . . ,K. We will thus divide the training interval [t-K, to] in K subintervals [to - k T , to - (k - 1 ) 4 k = 1, . . . ,K. We will also replace the continuous mean sum of squares error (equation 2.1) by its discrete counterpart i

X-1

D

Moreover, the numerical ODE integration will also proceed in a discrete fashion, stepping forward from time t-k to time t-k+l to obtain xi(t0 (k - 1)~)from f i ( t 0 - k T ) and backward to obtain zi(to - k T ) from q ( t 0 (k - 1 ) ~ )= 0. In the following section, the analysis of this concrete algorithmic implementation will lead us to an alternative approach, based in feedforward u-T networks, to the above recurrent modeling. 3 Feedforward Modeling

Partial Taylor series summation provides a natural integrating device for Ricatti ODES (Fairen et al. 1988) because of the simple recurrence for computing the successive derivatives of x. Each xi(to - ( k - 1 ) ~ )can be approximated to order Q as (3.1)

starting with x y ) ( t o - k T ) = f i ( f o - k T ) ; here x p ) stands for the qth derivative of X i , which can be recurrently evaluated by means of the formulas XP’(t)

= ai6Iq

+

cb i j % y ’ ( t )

with blq being 1 when q = 1 and 0 otherwise. Now, observe that if we conceptually place for each q the x p ) in a single layer, the computation in equation 3.2 corresponds for each i to the weighted sum of linear outputs coming from all the units in the previous 9 - lth layer, and quadratic outputs from pairs j , p of units in layers 1, m such that l m = q - 1 (for q = 1 there is an extra term that can be seen as coming from an outside unit with constant 1 activation). Moreover, the outcome of equation 3.1 is for each i the sum of the activations of the i units in the just defined Q + 1 layers, each one constantly weighted by ~ q / q ! In . other words, the outcome of a Qth order Taylor series integrator as given by equations 3.1

+

800

Vicente L6pez, Ram6n Huerta, and JoseR. Dorronsoro

and 3.2 can be formally seen as the output of a CT-T feedforward network with 1. one input layer, corresponding to 0 order derivatives;

2. Q intermediate layers, corresponding to order q derivatives, 1 I q 5 Q, with linear connections with all the units in the preceding q - lth layer and quadratic connections from pairs of units in layers I, m such that I + m = 9 - 1 and connecting weights constrained as in equation 3.2; 3. one output layer whose ith unit is connected to the ith units of all previous qth layers with fixed weights 7 4 / q ! . Figure 1 contains the connections diagram of a network performing firstorder Taylor integration of a two-trajectory system. Once these outputs xi(to - k7 + 7)have been computed, the interval errors E k can be approximated as before by

where we have used the notation xi(t0 - k 7 ) = xi(k),fi(to- k7) = fi(k). This form of the error readily identifies as the usual backpropagation total sum of squares error between the targets f(k) and outputs x(k) corresponding to inputs f(k - 1). As it is now clear, this concrete numerical implementation lends itself to an alternative modeling device: we simply consider all the values connecting the first Q 1 layers in the above network as free parameters (of course, unit activations cannot be taken any longer as derivatives) and view the network output as given by a particularly structured feedforward network. Learning is then performed by means of the usual, well-known backpropagation algorithm, either on line or in batch mode. This approach offers clearly greater speed in the gradient computations, although with a drawback, due to the fact that the number of weights to be adjusted depends now not only on the dimension D of the coupled trajectories but on the order Q of the Taylor series to be used: if great accuracy is needed, Q will have to be increased and so the dimension of the weights search space. In principle, this could make these networks vulnerable to the well-known phenomenon of overfitting: the perfect memorization of the training patterns but the inability to deal with other inputs not seen before. Nevertheless, the value of the time series networks presented here depends essentially on their prediction abilities: if they are capable of providing adequate medium range forecasts, we can conclude with a certain amount of confidence that they are not hampered by an excessive number of weights. We will illustrate this issue on our numerical examples: relatively high values of Q were needed for them

+


801

Figure 1: Connections diagram for first-order Taylor integration of a twotrajectory system (an extra unit with constant activation 1 and connected to the intermediate layer has not been depicted). Circles with a + represent additive units; dotted ones multiplicative units. Solid circles denote weight multiplication (a few of the corresponding coefficient labels are shown); four of these weights have fixed values 1and 7. This network is used in the Henon trajectory simulation of Section 4. to successfully model the training trajectories, yet the resulting networks still gave good predictions. A second advantage of these feedforward networks is that, unlike their recurrent counterparts, they can successfully be also used to study discrete time series even when a continuous time evolution cannot be assumed. In this situation an ODE based approach is bound to fail but these series can sometimes still be studied in terms of mappings, that is, the k + 1 term xk+l being generated as @(xk), with @ the mapping function we want to model. Used with a time step of 1, the feedforward networks just mentioned above provide precisely a particular mapping 9 that from input xk tries to adjust the step k+ 1 target xk+l. This target is then clocked back to the network as input and will be mapped by 9 into an approximation to the next target, and so on. In particular, an order 1 feedforward network defines a (nonhomogeneous) quadratic mapping in

Vicente Lbpez, Ram611Huerta, and JoseR. Dorronsom

802

terms of the components of the xk. We will also illustrate this approach in the next section. Finally, and although we will not pursue it here any further, there is a third, potentially very important advantage in this feedforward alternative. In contrast with the conjugate gradient or quasi-Newton methods for numerical optimization, backpropagation networks lend themselves naturally to hardware implementations; obviously this is more true for the above u--T architecture, since it just involves additions and multiplications. 4 Numerical Results

We will now apply the above techniques to three numerical examples of coupled time series derived from the so-called Morse equations, and from the well-known Lorenz equations and Henon mappings.

4.1 Morse Trajectories. The first example involves the Morse equations of molecular physics. Although we will not discuss them in detail, we point out that they arise as equations of motion of model Hamiltonians used in many studies of molecular oscillations. Here we will consider the two-trajectory system

9 = P

i

= -2(1 - e-4)e-q

where 9 would denote bond length; notice that it is very far from being of Ricatti type. The trajectory behavior depends on its total energy H. In our example, if the total energy H is 0, unless P = NP. [Similar results for other problems arising in the context of analyzing Hopfield nets have been obtained in Florkn and Orponen (1989) and Godbeer et al. (1988). For general introductions to computational complexity issues in neural networks, see Orponen (19921, Parberry (1990), and Wiedermann (19901.1 We start by examining the easier case: the synchronous network. As will be seen, the boundary between tractability and intractability is here located between computing direct (one-step) and two-step attraction radii. We first observe that the former can be computed in polynomial time. Theorem 1. The problem “Givena synchronous Hopfield network, a stable vector u, and a distance k, is the direct attraction radius of u equal to k?” is polynomially solvable.

Proof. The following polynomial time procedure determines the direct attraction radius of u: 1. Radius := n;

2. For each node i

a. compute the values wipj for j = 1, . . ., n and order them as a1 2 2 a,; b. sum := cjaj - ti % this is the total input to node i; c. if ui = 1 then i. k := 1; ii. repeat s u m := sum-2ak; k := k+ 1 until sum < 0 or a k I 0 or k = n 1; iii. if sum < 0 then radius := min{radius, k - 2 ) d. if ui = -1 then i. k := n; *

*

a

+

Patrik Floreen and Pekka Orponen

814

ii. repeat sum := sum-2ak; k := k- 1 until sum 2 0 or ctk 2 0 or k = 0; iii. if sum 2 0 then radius := mintradius, n - k - 1) 3. Return (radius).

Intuitively, we check for each node how many of its inputs must be altered to change its state in the update. The minimum of these numbers is the distance to the nearest vector that results in something else than u. If this distance is k, the radius of direct attraction is k - 1. 0 Next we consider the problem of computing the asymptotic attraction radius. Note that this problem is in NP, if the weights in the network are polynomially bounded in n. A nondeterministic algorithm for the problem works as follows: Given a vector u and a distance k, guess a vector that is within distance k from u and does not converge to u, witnessing that the attraction radius of u is less than k. When the weights are polynomially bounded, any vector converges to either a stable vector or a cycle of length two in polynomial time (Goles et al. 1985).

Theorem 2. The problem "Given a synchronous Hopfield network,a stable vector u, and a distance k; is the attraction radius of u less than k?" is NP-hard. Proof. We prove that the problem is NP-hard by a reduction from the NPcomplete satisfiability problem SAT: "Given a conjunctive normal form (CNl? formula F with k variables and m clauses, is there a satisfying truth assignment for F?"' In fact, we need a special version of SAT: we require that no clause contains both a variable and its negation, and we require that the number of clauses is greater than the number of variables. It is easy to see that these requirements do not affect the NP-completeness, since clauses with both a variable and its negation can be excluded, and if the number of clauses is too small, we can simply repeat one of the clauses the required number of times. Let i = (XI, . . .,i k ) be some truth assignment not satisfying F. Such an assignment is easy to find: Take, for instance, the first clause and choose values for the variables against their appearance in the clause, that is, if variable xi is in the clause, choose false for it, if xi appears negated in the clause, choose true for it; otherwise the value of the variable does not affect the value of the clause, so choose for instance value false for it. Transform formula F to an equivalent formula F' by adding k times one of the false clauses. This ensures that at least k + 1 of the clauses evaluate to false under i . In the following, rn refers to the number of clauses in F'. 'A CNF formula is a conjunction of clauses c1, c2, . . .,cm, where a clause is a disjunction of boolean variables XI, X Z , . ..,xk and their negations, for example, xl V 7 x 3 V 4 ; and a satisfymg truth assignment is a choice of values (true or f alae) for the variables so that the formula gets value true.

Attraction Radii in Binary Hopfield Nets

815

Now we construct a Hopfield network in such a way that there is a stable vector i corresponding to the truth assignment 2, and unless there is a satisfying truth assignment, all input vectors differing from ii in at most k elements converge to ii. On the other hand, if there is a satisfying truth assignment f, the vector corresponding to f differs from the stable vector ii in at most k elements and does not converge to ii; hence the attraction radius of ii is less than k. In the construction, truth value true is represented by node state +1, and truth value false is represented by node state -1. In the following, we make no distinction between the truth values and the corresponding node states. The network has nodes corresponding to the variables and the clauses, and 2k 2 additional nodes, in total 3k + m 2 nodes. We denote a state vector of the network as ( x ,c, r,s), where subvector x = (XI, . . ., x k ) corresponds to the variables, subvector c = (cl, . . . ,)c, corresponds to the clauses, and subvectors r and s, each of length P = k + 1, correspond to the additional nodes. Let c, be the vector of truth values for the clauses resulting from assignment x . Especially, denote S = ci. The stable vector in our construction will be ii = (2, S, -7, -7). Each r-node represents the conjunction of the clauses represented by the c-nodes; the r-nodes are replicated to guarantee that not all of them can be +1 for vectors within Hamming distance k from the stable vector ii, which has T = -7. The s-nodes work in such a way that as soon as all of them get state +1, their states cannot change any more. Let a = P2 + 1 = k2 + 2k + 2. The Hopfield network is constructed in the following way (see Fig. 1):

+

0 0

+

The threshold value of each node xi is -(am

+ 1)Xj;

The threshold value of each node c, is -a(kj - l), where k, is the number of literals (i.e., variables and negations of variables) in the clause cj;

0

The threshold value of each node ri is P(m - 1);

0

The threshold value of each node Sj is 0;

0

0 0

0

0

There is an edge between each xi and each cj with weight a if literal xi is in clause c, and with weight -a if literal i x i is in clause cj; There is an edge between each c, and each ri with weight P; There is an edge between each ri and the corresponding weight k;

Si

with

The s-nodes constitute a fully connected subnetwork there is an edge between each si and each s, (where j # i) with weight 1; and All other edges have weight 0, that is, they are excluded.

Patrik Florkn and Pekka Orponen

816

x1

XZ

7

xk

thresholds:

Figure 1: The structure of the Hopfield network in Theorem 2. It is easy to check that ii = ( i , Z , -7, -7) is a stable vector in this network. This construction is polynomial in the size of the input formula, and we can now proceed to proving that this is the desired reduction. We prove first that if there is no satisfying truth assignment, all vectors at distance at most k from ii converge to ii, and after that we prove that if there is a satisfying truth assignment i,then vector ( i ,Z, -i,-T) does not converge to ii, and hence the attraction radius of ii is strictly less than the Hamming distance between i and i,which is at most k. 1. Assume there is no satisfying truth assignment. In this case, take an arbitrary input vector (x, c, r, s) with Hamming distance at most k from ii. At the first update step, the states of the x-nodes become X, since the input to node xi is between -am and am, and hence xi gets state sgn[fam (am 1)4] = i i . We use here an abbreviation of type sgn(a fb) for sgn(x), where a - b 5 x I u b. As the threshold values in this way force the x-nodes to get states 5, the states of the x-nodes do not change any more during the computation. The states of the c-nodes get values c,. As there is no satisfymg truth assignment, at least one of the cx-valuesis -1. Recall that at least one cj has initial state -1 even if k c-nodes have their states differing from E. Hence the input to each r-node from the c-nodes is at most P(m - 2), and as there is only one connection from an s-node to each r-node, the s-nodes contribute at most k to the input. Thus the r-nodes get state -1, and the only situation

+

+

+


817

in which the r-nodes can get states 1 is when all the c-nodes have state 1. At the second update step the states of the c-nodes become S. This can be seen as follows. As S represents the truth values resulting from 2, the input from the x-nodes is -akj if Sj = -1, and at least -a(kj - 2) if Sj = 1. The input from the r-nodes is between -p2 and p2, SO if Sj = -1, then cj gets state sgn(-akj fp2 + a(kj - l)]= -1, and if Zj = 1, then Cj gets state sgn[-a(kj - 2) f p2 a(kj - I)] = 1. As the x-nodes do not change any more, also the c-nodes do not change any more during the computation. As there is some c-node with state -1, the states of the r-nodes do not change: They are all still -1. As the Hamming distance between ( x , c, r, s) and ii is at most k, at least one r-node and at least one s-node have initial states -1. Thus at the first update step, the absolute value of the input from other s-nodes to each m o d e is at most k - 2, whereas the input from the corresponding m o d e is the state of the r-node times k. This results in the s-nodes having the same states after the first update step as the r-nodes had before the first update step. Consequently, there is at least one s-node with state -1 after the first update step. The first update step results in all r-nodes having state -1. Consequently, all s-nodes get states -1 in the second update step. To sum up: Starting with ( x , c, Y, s), the first update step results in (i,cx, -7, r), and the second update step results in ii.

+

2. Assume that there is a satisfymg assignment f. We show that the input vector (f,S, -7, -T) does not converge to ii, which implies that the attraction radius must be less than k. In the first update step, the x-nodes become 2, each cj gets state sgn[-a(kj - 2) f p2 + a(kj - l)] = 1, and r and s stay -1. In the second update step, the cj-nodes become I., but each ri gets state sgn[pmfk-p(rn-l)] = 1, whiles still stays -7. In the third update step, the r-nodes become -T but each Si gets state sgn(k - k) = 1. Now s stays as it is, since from now on the total input to each si is -k k = 0. The computation has converged to ( i S, , -T, 1)# ii.

+

The proof is now completed.

0

From the construction in the proof, we see that just determining the two-step attraction radius is NP-hard. Computing the direct attraction radius is easy while computing the two-step attraction radius is hard, because for the direct radius it is enough to check the change of one element at a time while for the two-step radius we have to check the changes of the changes. Also approximating the attraction radius is hard. We say that an approximation algorithm to a minimization problem approximates the

Patrik Florken and Pekka Orponen

818

problem within a factor K, if for all sufficiently large problem instances I, the result of the algorithm is at most Kmin(I), where min(1) is the optimal result for instance 1. If a CNF formula is satisfiable, it can in general be satisfied by many different truth assignments. We use the name MIN ONESfor the problem of finding the minimum number of true variables in a satisfying truth assignment. [The analogous maximization problem MAXONEShas been considered in Panconesi and Ranjan (1990).1 We see from the construction of the network in Theorem 2 that the attraction radius is one less than the minimum Hamming distance between vector i and a satisfying vector i. Now, construct from a given instance of SAT a formula in the way described in Theorem 2. For each i i = 1, change all literals xi to xi and all literals xi to xi. Now setting all variables to false yields a nonsatisfying truth assignment for the formula, and (-T,c-T, -1, -7) is the stable vector we consider. Thus, the problem of computing the attraction radius is equivalent to the problem MIN ONESof finding the minimum number of true variables in a satisfying truth assignment to a CNF formula. It is easy to show that there is no polynomial time algorithm approximating MIN ONESwithin a factor K for any fixed K > 1, unless P = NP. Given a CNF formula F with k variables, denote n = LKkJ and add n + 1 new variables z, zl, zz, . . . ,zn.Construct the formula n

G = (FVZ)A

A[(zV T Z ~ )A ( 7 2V Z ~ ) ] i=l

Note that G can be made into a CNF formula by distributing the z in the first conjunct over the clauses of F. Now the number of true variables needed to make G true is either at most k (if F is satisfiable) or n + 1 > Kk (setting zl, ZZ, . . ., zn, and z to true). Consequently, an algorithm approximating MIN ONES within a factor K would in fact decide the satisfiability of formula F. We shall introduce here also a stronger construction, which was suggested to us by Viggo Kann (Kann 1992). For this construction, we need SET minimization problem, the MINIMUMINDEPENDENT DOMINATING which asks for the size of a minimum independent dominating set in an undirected graph. Let (V, E) be a graph, where V is the set of nodes and E C V x V is the set of edges. Then a subset S of V is an independent dominating set if there is no edge between any two nodes in S and every node in the graph is either in S or is a neighbor of a node in S. Magnds Halld6rsson has shown that MINIMUM INDEPENDENT DOMINATING SET cannot be approximated in polynomial time within a factor nl-' for any fixed 6 > 0, unless P = NP (Halld6rsson 1993). Here n is the number of nodes in the graph.

Lemma 1. There is no polynomial time algorithm approximating MIN ONES within a factor nl-€for any fixed c, where 0 < 6 5 1, unless P = NP. Here n is the number of variables in the CNF formula.


819

Proof. We prove that a polynomial time algorithm approximating MIN ONES within a factor K would give a polynomial time algorithm approximating MINIMUMINDEPENDENT DOMINATING SET within a factor K. Consequently, MIN ONESis at least as hard to approximate as MINIMUM INDEPENDENT DOMINATING SET, and the claim follows from Halldbrsson’s result. Let ( V ,E ) be a graph with n nodes. Create one variable si for each node Si E V . Note that for simplicity we use the same notation for both the node and the variable. Now we transform the property that a node is in an independent dominating set to the property that the corresponding variable is true. Denote the set of neighbors of node si by Ei = { s, I {si,sj} E E } . Construct the CNF formula

G=

A (si V V sj) A A s,EV

s,EE,

( T S ~v T S ~ )

{s,,q}~E

But every satisfying truth assignment to this formula corresponds to an independent dominating set in the graph and the size of this set is equal to the number of true variables. This completes the proof. 0

Corollary 1. There is no polynomial timealgorithm approximating the attraction radii in a synchronous Hopfield network (with n nodes) within a factor n*-€for any fixed E , where 0 < E 5 1, unless P = NP. Proof. We have already seen that the problem of computing the attraction radius is equivalent to the problem MIN ONESof finding the minimum number of true variables. The claim follows immediately from Lemma 1. 0 The results above for synchronous Hopfield memories can be extended to asynchronous Hopfield memories. In the asynchronous case, the results are valid for the asymptotic attraction radius only; the k-step attraction radius is not interesting. We sketch below how the proof of Theorem 2 must be modified in order to apply for asynchronous Hopfield memories. The nonapproximability result then follows in the same manner as in Corollary 1.

Theorem 3. The problem “Given an asynchronous Hopfield network, a stable vector u, and a distance k; is the attraction radius of u less than k?” is NP-hard. Proof (sketch). The problem in applying the proof of Theorem 2 to the asynchronous case lies in the free update order. To avoid this problem, we add for each clause c, a subnetwork checking that cj has the correct value for the current variables x, that is, the subnetwork computes cj 2 (xi, V xi2 V . - .V xiki)? The results of the subnetworks are used by the 2Note that some variables may appear negated in the disjunction. For simplicity of notation, we assume that cj is of the expressed form.

820

Patrik Florken and Pekka Orponen

r-nodes: Node rj gets value 1 if and only if Cj = 1 and, additionally, the equivalence is satisfied. In order to avoid cheating the equivalence test by choosing suitable initial values, the subnetworks are replicated so that there are k = P - 1 such subnetworks for each cj. Node r, gets value 1 if and only if all the subnetworks connected to it yield 1. Additionally, to avoid cheating by manipulating a small set of the x variables, the result of the equivalence test must be false for the stable state ii. Hence, we extend the equivalence test to [Cj

(Xi,

V Xiz V

*

a

*

V X i , ) ] A (X f

i)

The subnetworks are put between the layer of c-nodes and the layer of r-nodes: Each subnetwork has connections from each x-node and from the corresponding c-node and the result of the subnetwork is used by the corresponding r-node. Thus node rj is connected to node c, and to all the - 1 subnetworks connected to cj; each connection has weight P (see Fig. 2). Each subnetwork has 3k +5 nodes and depth 4; the weights must again be chosen so that nodes to the right cannot influence the result (nodes to the left determine solely the outcome of the update). This

thresholds:

-(dm

+ I)&

-a’(kj - 1)

P’m - P

Figure 2: The network part with direct connections to node c1 in the modified network in Theorem 3 (cf. Fig. 1). Note that there in fact are four connections with different weights between each x-node and each subnetwork (denoted by a square).


821

means also that the weight cr must be increased to a’ = kmx(P2 + 2P 3) + P + 1, where k,, is the maximum number of literals in a clause in the formula. Now we can proceed roughly as in Theorem 2. Acknowledgments P. 0. would like to thank Max Garzon for inspiring discussions on the topics of this work. The work of P.F. was supported by the Academy of Finland. References Florkn, P., and Orponen, P. 1989. On the computational complexity of analyzing Hopfield nets. Complex Syst. 3,577-587. Garey, M. R., and Johnson, D. S. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, New York. Godbeer, G. H., Lipscomb, J., and Luby, M. 1988. On the computational complexity of finding stable state vectors in connectionist models (Hopfield nets). Tech. Rep. 208/88, Dept. of Computer Science, Univ. of Toronto. Goles Ch., E., Fogelman-Soulie, F., and Pellegrin, D. 1985. Decreasing energy functions as a tool for studying threshold networks. Discrete Applied Math. 12, 261-277. Halldbrsson, M. M. 1993. Approximating the minimum maximal independence number. JAIST Research Report ISRR-93-0001F, School of Information Science, Japan Advanced Institute of Science and Technology, Japan. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Kann, V. 1992. Personal communication. Orponen, P. 1992. Neural networks and complexity theory. In Proceedings of the 17th International Symposium on Mathematical Foundations of Computer Science, pp. 50-61. Lecture Notes in Computer Science 629. Springer-Verlag,Berlin. Panconesi, A., and Ranjan, D. 1990. Quantifiers and approximation. In Proceedings of the 22nd Annual ACM Symposium on Theory of Computing, pp. 446-456. ACM, New York. Parberry, I. 1990. A primer on the complexity theory of neural networks. In Formal Techniquesin Art@cial Intelligence: A Sourcebook, R. B. Banerji, ed., pp. 217268. Elsevier, Amsterdam. Wiedermann, J. 1990. Complexity issues in discrete neurocomputing. In Proceedings on Aspects and Prospects of Theoretical Computer Science, pp. 480-491. Ledure Notes in Computer Science 464. Springer-Verlag,Berlin.

Received 4 May 1992; accepted 2 February 1993.

This article has been cited by: 2. Jiří Šíma , Pekka Orponen . 2003. General-Purpose Computation with Neural Networks: A Survey of Complexity Theoretic ResultsGeneral-Purpose Computation with Neural Networks: A Survey of Complexity Theoretic Results. Neural Computation 15:12, 2727-2778. [Abstract] [PDF] [PDF Plus] 3. Yu G. Smetanin. 1998. Neural networks as systems for recognizing patterns. Journal of Mathematical Sciences 89:4, 1406-1457. [CrossRef]

ARTICLE

Communicated by John Rime1

Analysis of Neuron Models with Dynamically Regulated Conductances L. F. Abbott* Gwendal LeMassont Departments of Physics' and Biologyt, and Center for Complex Systems, Brandeis University, Waltham, M A 02254 USA We analyze neuron models in which the maximal conductances of membrane currents are slowly varying dynamic variables regulated by the intracellular calcium concentration. These models allow us to study possible activity-dependent effects arising from processes that maintain and modify membrane channels in real neurons. Regulated model neurons maintain a constant average level of activity over a wide range of conditions by appropriately adjusting their conductances. The intracellular calcium concentration acts as a feedback element linking maximal conductances to electrical activity. The resulting plasticity of intrinsic characteristics has important implications for network behavior. We first study a simple two-conductance model, then introduce techniques that allow us to analyze dynamic regulation with an arbitrary number of conductances, and finally illustrate this method by studying a seven-conductance model. We conclude with an analysis of spontaneous differentiation of identical model neurons in a two-cell network. 1 Introduction Mathematical models based on the Hodgkin-Huxley approach (Hodgkin and Huxley 1952)describe active neuronal conductances quite accurately over time scales ranging from milliseconds to several seconds. Model neurons constructed from these descriptions (see for example Koch and Segev 1989) exhibit a wide variety of behaviors similar to those found in real neurons including tonic spiking, plateau potentials and periodic bursting. However, neuronal conductances can change over longer time scales through additional processes not modeled by the Hodgkin-Huxley equations. These include modification of channel structure and/or density through biochemical pathways involving protein phosphorylation (Kaczmarek 1987; Chad and Eckert 1986; Kaczmarek and Levitan 1987) and gene expression (Morgan and Curran 1991; Sheng and Greenberg 1990;Smeyne et al. 1992). These processes can be activity dependent. For Nard Computation 5,823-842 (1993) @ 1993 Massachusetts Institute of Technology

L. F. Abbott and Gwendal LeMasson

824

example, when rat myenteric neurons are chronically depolarized they show decreased calcium currents (Franklin et al. 1992). Electrical activity can induce the expression of immediateearly genes like fos over a period of about 15 min (Morgan and Curran 1991; Sheng and Greenberg 1990; Smeyne et al. 1992) and expression of the immediate-early gene ras has been associated with an increased potassium conductance (Hemmick et al. 1992). From these studies it is clear that the biochemical processes that affect membrane conductances act on many different time scales. Relatively fast effects, such as the voltage and calcium dependence of channel conductances, are included in the usual Hodgkin-Huxley descriptions. However, activity-dependent modifications of membrane currents due to slower and less direct processes are not. Unfortunately, there is not enough information available at the present time to build a detailed model of the biochemical processes producing slow modification or, as we will call it, regulation of membrane conductances. However, we feel that it is not too early to try to assess what impact such a process might have on the behavior of neurons and neural networks. To do this we have constructed a simple phenomenological model with slowly varying, dynamically regulated conductances and studied its behavior using computer simulation (LeMasson et al. 1992). The model reveals several interesting features: 0

0

0

0

Starting from a wide variety of initial conductances, the model neurons can automatically develop the currents needed to produce a particular pattern of electrical activity. Slow regulatory processes can significantly enhance the stability of the model neuron to environmental perturbations such as changes in the extracellular ion concentrations. The intrinsic properties of model neurons are modified by sustained external currents or synaptic inputs. In simple networks, model neurons can spontaneously differentiate developing different intrinsic properties and playing different roles in the network.

These features have obvious implications for the development and plasticity of neuronal circuits. Our previous work (LeMasson et al. 1992) relied solely on computer simulation involving a fairly complex neuronal model. In this paper we devise a general procedure for analyzing the process of dynamic regulation. We will examine the properties listed above in detail both for a simple neuron model and for the more complex model considered previously.

Analysis of Neuron Models

825

2 A Model of Dynamic Regulation

We consider a single compartment, conductance-based neuron model with the membrane potential V determined by the basic equation (2.1)

C is the membrane capacitance and Ii are the membrane currents, which are written in the form (Hodgkin and Huxley 1952; see Koch and Segev 1989)

I., - g,mfh?(V - Ei)

(2.2)

where Ei is the equilibrium potential corresponding to the particular ion producing the ith current, pi and qi are integers, and gi is the maximal conductance for the current i. The dynamic variables mi and hi are determined by first-order, differential equations linear in mi and h; but with nonlinear voltage-dependent coefficients,

and

These equations describe the voltage-dependent characteristics of the conductance. Calcium dependent properties can be included by allowing a and p to depend on the intracellular calcium concentration as well as on the voltage. In conventional Hodgkin-Huxley type models, the maximal conductances gi are fixed constants. However, these are likely candidates for the slow modulation that we refer to as dynamic regulation. This is because the maximal conductance of a given current is the product of the conductance of an individual membrane channel times the density of channels in the membrane. Any slow process that alters the conductance properties of the channel or adds or removes channels from the membrane will affect gi.These slow, regulatory processes can be included in the model by making the maximal conductances dynamic variables instead of fixed parameters (LeMasson et al. 1992). Regulatory mechanisms could also, in principle, mod* the kinetics of channel activation and inactivation, but we will not consider this possibility here. To construct a model with dynamic regulation, we need to describe a mechanism by which the activity of a neuron can affect the maximal conductances of its membrane currents. Numerous possibilities exist including modified rates of channel gene expression, structural modifications of the channels either before or after insertion into the membrane,

826


and changes in the rates of insertion or degradation of channels. These (and many other) processes often depend on the intracellular calcium concentration (Kennedy 1989; Rasmussen and Barrett 1984; Sheng and Greenberg 1990; Murphy et al. 1991). For example, activity-dependent expression of immediate early genes has been linked to an elevation in calcium levels due to influx through voltage-dependent calcium channels (Murphy et al. 1991) and calcium is implicated in many other examples of slow, activity-dependent modulation (Kennedy 1989; Rasmussen and Barrett 1984; Sheng and Greenberg 1990). In addition, the intracellular calcium concentration is highly correlated with the electrical activity of the neuron (Ross 1989; LeMasson et al. 1992). For these reasons, we use the intracellular calcium concentration as the feedback element linking activity to maximal conductance strengths (LeMasson et al. 1992). Since the maximal conductances gi depend on both the number and properties of the membrane channels, their values will be affected by the processes outlined above. If these processes are regulated by calcium the values of the maximal conductances will also depend on the intracellular calcium concentration. We will assume that the kinetics is first-order and that both the equilibrium values of the maximal conductances and the rate at which they approach the equilibrium value may be calcium dependent. As a result, the behavior of the maximal conductances g, is described by the equations (2.5)

where [Ca]is the intracellular calcium concentration. At fixed intracellular calcium concentration [Ca],the maximal conductance g, will approach the asymptotic value Fi[Ca] over a time of order q[Ca]. If the calcium concentration changes the maximal conductances will also change their values. This regulation is a slow process occurring over a time ranging from several minutes to hours. This time scale distinguishes the calcium regulation of equation 2.5 from the more familiar and rapid calcium dependence of currents like the calcium-dependent potassium current. The full neuron model with dynamic regulation of conductances is described by equations 2.1-2.5 and an equation for the intracellular calcium concentration [Ca]. For the model to work, it is crucial that one of the membrane currents Z, = Zca be a voltage-dependent calcium current because this is what links the intracellular calcium concentration to activity. We will assume that entry through voltage-dependent calcium channels is the only source of intracellular calcium and will not consider release from intracellular stores. Calcium is removed by processes that result in an exponential decay of [Ca]. Thus, [Ca] is described by the equation


827

The constant A depends on the ratio of surface area to volume for the cell. We typically use a value between 1/(100 msec) and l/sec for the constant k controlling the rate of calcium buffering. To complete the model we must specify the functions ~i([Ca])and Fi( [Ca])appearing in equation 2.5. As in our previous work (LeMasson et al. 1992) we are guided in the choice of these functions by considerations of simplicity and stability. We are primarily interested in the equilibrium behavior of the regulated model. Because of this, we can simpllfy equation 2.5 by setting all the time constants equal and making them calcium-independent, Ti( [ca]) = 7

(2.7)

where T is a constant independent of [Ca]. This simplification has no effect on the equilibrium behavior of the model. In our simulations, we have taken the time constant T to vary from 1 to 50 sec. We expect real regulatory processes to be considerably slower than this. However, the only condition on the model is that T be much longer than the time scales associated with the membrane currents so we have accelerated the regulatory process to speed up our simulations. The functions Fi determine how the asymptotic values of the maximal conductances depend on the calcium concentration. We assume that the regulation mechanism can vary the maximal conductances gi over a range 0 < gi < Gi where Gi is the largest value that gi can possibly take. In addition, a given maximal conductance can either increase or decrease as a function of the intracellular calcium concentration. These considerations lead us to consider just two possible forms (up to an overall constant) for the Fi, either a rising or a falling sigmoidal function,

where Gi, CT,and A are constants and u is the standard sigmoidal function 1 u(x) = (2.9) 1 exp(- x )

+

In equation 2.8, the parameter Gi sets the scale for the particular maximal conductance gi. CT determines the concentration at which the asymptotic value of gi is Gi/2 and A sets the slope of the sigmoid. The choice of the plus or minus in equation 2.8 determines whether gi will fall or rise as a function of [Ca]. The slow regulatory processes we are modeling must not destabilize the activity of the neuron. To assure stability of the neuron, the choice of the plus or minus sign in equation 2.8 must be made correctly. Suppose that a specific set of maximal conductances has been established producing a certain level of electrical activity. If the neuron becomes more active

828


than this level, calcium entering through voltage-activated channels will raise the intracellular calcium concentration. Under these conditions, outward currents should increase in strength and inward currents decrease so that the activity of the neuron will be reduced back to the original level. Conversely, if the activity level drops, the calcium concentration will also fall. In this case, the inward currents should increase in strength and the outward currents should decrease. In other words, the feedback from activity to maximal conductances should be negative. To assure this we use the plus sign in equation 2.8 for inward currents and the minus sign for outward currents. With this sign convention increased calcium results in an increase of the outward and a decrease of the inward currents while decreased calcium has the opposite effect. With the choices we have made, the evolution of the maximal conductances is given by (2.10)

where the variable sign is plus for inward currents and minus for outward currents. Because the intracellular calcium concentration depends on the maximal conductances, these are highly nonlinear equations. The parameter CT in equation 2.10 plays the role of a target calcium concentration. If [Ca] is well below CT,activity will increase due to the enhancement of inward and depression of outward currents. This will bring [Cal up closer to the target value CT. If [Ca] is well above CT, there will be an opposite effect on the currents and [Ca]will drop toward CT.Since the electrical activity of the neuron is highly correlated with the intracellular calcium concentration, stabilization of the intracellular calcium concentration results in a stabilization of the electrical activity of the neuron. As we will see, there is a direct connection between the target calcium concentration CTand the activity level maintained by the model neuron. Even without the dynamic regulation we have added, conductancebased neuronal models tend to be quite complex. However, the model specified above can be analyzed in considerable detail because of the large difference between the rates of the slow regulatory processes described by equations 2.10 and the faster processes of equations 2.1-2.4 and 2.6. 3 A %o-Conductance Model

The simplest model we will use to study dynamic regulation of conductances is the Morris-Lecar model (Morris and Lecar 19811, which has one inward and one outward active current. The inward current is a calcium current given (using the parameters we have chosen) by


829

and the outward current is a potassium current,

IK =g,n(V - EK)

(3.2)

with n given by

v-10

3

(3.3)

In addition, there is a passive leakage current

IL = 0.5(V + 50)

(3.4)

and we will sometimes add an external current as well. In these equations, V is measured in millivolts and time in milliseconds. Under control conditions, we take EG = 100 mV and EK = -70 mV although we will vary these parameters to simulate changes in the extracellular ion concentrations. We have added a persistent component (the 0.1 in equation 3.1) to the calcium current, which is not present in the original model (Morris and Lecar 1981). This is useful in the regulated model because calcium provides the feedback signal for the regulation process. Without a persistent component, loss of the calcium current would mean a loss of this signal. We take C = 1 pF/cm2, GQ = 3 mS/cm2, and GK = 6 mS/cm2. The behavior of this model neuron for the control values of the parameters is shown in Figure 1A. In the two-conductance model, the maximal conductances gc, and gK are regulated by equations like 2.10, specifically (3.5)

and (3.6)

We wish to analyze the dynamics of these two maximal conductances. Dividing the first equation by Gca and the second by GK we find that the quantities gca/GQ and g K / G obey ~ very similar equations. By adding the resulting two equations and using the identity u(x)+ cr(-x) = 1 we find that the quantity y defined by -

-

y = h + & Gca GK

(3.7)

obeys the trivial equation

7 dY -=l-y at

(3.8)


830

A

so

time (mSec)

B

3

I

I

I

2

-

gcn

0

Figure 1: (A) Membrane potential versus time at the quasi-steady-state point for the two-conductance model. (B)Maximal conductance "phase-plane" for the two-conductance model. Straight lines are nullclines of the slow, regulatory dynamics. Region marked Osc. is where oscillations of the regulatory system occur. The quasi-steady-state is where the two nullclines cross. Dashed paths marked 1 4 show routes to the steady-state point from four different starting conditions. For convenience (and without loss of generality) we have chosen the units of [Ca]so that the coefficient A in equation 2.6 is one. In these units CT = 20 and A = 5. In addition, we take k = 1/(100 msec). These parameters are used for Figures 2-4 as well (except that CT is vaned in Fig. 3). Likewise taking the difference of these two equations and defining -

,=Bca-& GCa

-

GK

(3.9)

we find that r -dz =tanh( dt

CT 2h - [ca]) - z

(3.10)


831

Using equations 3.8 and 3.10, we can completely analyze the behavior of the model in the "phase-plane" of maximal conductances 9c, and gK. First, there is a nullcline y = 1 or equivalently (3.11)

from equation 3.8 and this is approached exponentially with time constant 7. The behavior of the z variable is more complex. Under some conditions, z will approach a quasi-equilibrium state. An equilibrium solution of equation 3.10 would occur when z = tanh[(CT - [Ca))/2A]. However, if this value of z results in oscillatory behavior of the model neuron the calcium concentration [Ca] will oscillate as well. Thus, this value of z will not truly be fixed. We can circumvent this complication because we are assuming that the time scale T governing the motion of z is much greater than the time scale of the membrane potential oscillations. Although z will oscillate around the quasi-equilibrium value, if 7 is large these oscillations will be very small. The quasi-equilibrium value of z is just the average value of the hyperbolic tangent z = (tanh

(

cT

;PI))

(3.12)

where the brackets denote a time average over many membrane potential oscillation cycles. Equation 3.12 defines an approximate nullcline for the dynamics of the z variable for the maximal conductances. In Figure lB, the solid lines indicate the nullclines 3.11 and 3.12 for the regulatory dynamics. The diagonal line with negative slope is the y nullcline g&/G& + gK/GK = 1 while the more horizontal line is the z nullcline. In the center of the figure, where the two nullclines cross, is the quasi-steady-state point of the full system which results in the behavior seen in Figure 1A. This point is stable and its domain of attraction is the entire plane. There is a region of the plane (at the lower left of Fig. 1B) where z does not approach quasi-steady-state behavior at fixed y but instead goes into oscillations with a period of order 7. In this area there is, of course, no z nullcline. Instead, we have drawn the upper and lower bounds of the region over which the oscillations in z take place. Regions like this provide an interesting mechanism for generating rhythms with very long periods such as circadian rhythms. These slow oscillations arise from the regulatory process interacting dynamically with the more conventional mechanisms producing the much faster membrane potential oscillations. The dynamically regulated model can spontaneously construct its conductances starting from any initial values of gc, and &. The dashed curves in Figure 1B show the approach to steady-state behavior from four different sets of initial conductances. There are no obstructions to the recovery of the quasi-steady-state values from any initial position in the plane.


a32

0.07

0.06

1

'i?0.05

-

0.04

-

0.03

-

.5!

x 0

C

e

e

LL

0.02 -

10

15

20 25 Target Calcium (CJ

30

35

Figure 2: The range of steady-state oscillation frequencies that can be obtained using different values of the target calcium concentration CT.Units of calcium concentration are as in Figure 1. In the usual, unregulated, conductance-based models, the values of the maximal conductance parameters determine the behavior of the model neuron. In the regulated model, the maximal conductances are dynamic variables and, instead, the behavior of the model is governed by the parameters CT and A that control the quasi-steady-state values of the maximal conductances. Of these, CT is by far the more important parameter. By adjusting the value of this target calcium concentration, we can determine what sort of behavior the neuron will exhibit. In contrast to conventional models, once this value is chosen the desired behavior will be exhibited over a variety of external conditions. In Figure 2, we see that a wide range of oscillation frequencies can be obtained in the regulated, two-conductance model by choosing different values for the target calcium concentration CT without changing any other parameters of the model. The stabilizing effects of dynamic regulation are illustrated in Figure 3. When dynamic regulation is not included in the model, the firing frequency is extremely sensitive to the values of Eca and EK and firing only occurs over a limited range of these parameters. With dynamic regulation, stable firing at roughly the same frequency can be maintained over a wide range of Eca and EK. Since these parameters are affected by the extracellular ionic concentrations, this reflects the ability of a dy-


a33

0""'"""""""""" -100 -90 - 8 0 -70

-60

-50

-40

El;(mV)

B

Figure 3: The dependence of oscillation frequency on the equilibrium potentials for (A) potassium and (B) calcium in the regulated and unregulated two-conductance models. Dynamic regulation stabilizes the frequency against changes in EK and Eta. For the unregulated case, we fix the maximal conductances at the control values for the unregulated model.


834

-0.3 -0.35 c -0.4

-0.45 Z

-0.5 -0.55

t

-0.6 -0.65 -6

-4

-2 0 2 Current (nA)

4

6

Figure 4: The quasi-steady-state value of z as a function of the amplitude of an injected current. Both DC and pulsed injection cause shifts in the value of z that modify the balance between inward and outward currents and change the intrinsic properties of the model neuron. Pulses last for 250 msec and are repeated every 500 msec. namically regulated neuron to adjust to varying external conditions. The model maintains its firing frequency by shifting its maximal conductances in response to changes of these parameters. This is done through shifts in the value of z, which change the balance of inward and outward currents. Dynamically regulated neurons also exhibit activity-dependent shifts in their intrinsic characteristics. As we have seen, the regulatory mechanism tends to stabilize the activity of the neuron by shifting the values of the maximal conductances to maintain the level of activity that results in an average intracellular calcium concentration near the target value CT. The introduction of external or synaptic inputs will likewise cause slow shifts in the values of the maximal conductances as the regulatory mechanism tries to maintain the same level of calcium and activity that existed in the absence of inputs. As a result, prolonged inputs cause changes in the intrinsic characteristics of the neuron. This is shown in Figure 4 where we investigate the effect of external current on a regulated model neuron. The external current causes a shift in the value of z which changes the intrinsic electrical properties of the neuron by modifying the


835

balance between inward and outward currents according to equation 3.9. The quasi-steady-state value of z depends not only on the amplitude of the applied current but also on its time course. As shown in Figure 4, DC current injection has a different effect than pulses of current and we have found that the shift in z is also sensitive to the frequency and duty cycle of the pulses, in particular, the relation of the pulse frequency to the natural frequency of the model. These shifts occur over a slow time scale. Thus, the regulated model neuron will respond normally to brief pulses of current. However, prolonged current injection or synaptic input will change intrinsic properties. 4 General Analysis

The type of analysis we performed for the two-conductance model in the last section can be extended to models with arbitrarily large numbers of conductances. The key observation is that when equation 2.10 is divided by Gi, all of the ratios gi/Gi satisfy the same equation except for the plus and minus sign difference for inward and outward currents. This implies that the difference gJGi - gj/Gj between any two outward or any two inward currents will go exponentially to zero with the time constant r. Furthermore, the identity D ( X ) D ( - X ) = 1 we used before implies that the sum gi/Gi + gj/Gj, where i is an outward current and 1 is an inward current, goes exponentially to one with the same time constant. As a result, we can write an explicit solution for all of the maximal conductances satisfying equation 2.10 expressed in terms of just one dynamic variable z,

+

where the plus/minus sign is for inward/outward currents and ci are constants that determine the initial values of the maximal conductances gi(0). The remaining dynamic variable z obeys the same equation as before, r -dz= t a n h ( dt

CT 2a - [cal) - z

(4.2)

Thus we have reduced the analysis of dynamic regulation in a model with any number of currents to the study of this single equation interacting with the rest of the model through the z dependence of [Ca]. As in the two-conductance case, there are two general types of behavior. First, the system can settle down to a quasi-steady-state as far as the slow dynamics is concerned. Again, although the membrane potential and calcium concentration may fluctuate (due to action potentials for example), there are no fluctuations over the time scale associated with dynamic regulation. These faster fluctuations have little effect on the slowly


836

varying maximal conductances. Alternately, the slow system may never settle down and oscillations or even chaotic behavior characterized by the slow time scale typical of regulatory processes may appear. Again, these can provide a model of circadian or other slow rhythms. 5 A Seven-Conductance Model

We have studied dynamic regulation in a more complex and realistic model, a variant of the model of Buccholtz et al. (1992) describing the LP neuron in the stomatogastric ganglion of the crab. This model has seven active conductances corresponding to Hodgkin-Huxley sodium and potassium currents, a slow and a fast A current, a calcium-dependent potassium current, a calcium current, and a mixed-ion current ZH. In addition, there is a passive leakage current. We allow all seven maximal conductances for the active currents to be modified by the calciumdependent regulation scheme as described by equations 2.10. Depending on the value of the target calcium concentration CT,the regulated LP model can exhibit silent, tonic firing, bursting, or lockedup (permanently depolarized) behavior. Although the model has seven dynamic maximal conductance variables, we can analyze the regulatory dynamics quite simply by using the z variable defined in the last section. After the exponential terms in equation 4.1 get small, the maximal conductances will take values Gi gi = -(1 f2 )

(5.1) 2 with z determined by equation 4.2. To study the behavior of z in this model, we plot dz/dt given by the right side of equation 4.2 as a function of z in Figure 5. We also note the type of activity displayed by the model neuron for different values of z. For this figure, we have chosen the target calcium concentration CT so that the neuron exhibits bursting behavior once the z parameter has relaxed to the point where d z / d t = 0. The quasi-steady-state is given by the zero crossing in the center of the figure and it exhibits bursting behavior. In the bursting range, Figure 5 shows a double line because we have plotted both the maximum and minimum values of dzldt. At a given z value (the quasi-steady-state value for example) dz/dt will oscillate rapidly between the two lines shown due to the bursting behavior. These oscillations are not the same as those shown in Figure 1. The oscillations in Figure 1 are slow and are caused by the regulatory mechanism itself, while the oscillations here are just the result of the normal bursting activity of the neuron. In our previous work on this model (LeMasson et al. 1992) we observed an interesting phenomenon when two regulated neurons were electrically coupled. The techniques we have developed here allow us to explore this phenomenon more completely. The two-neuron circuit is shown in Figure 6. We start with two dynamically regulated model


a37

1

0.5

3

0

E_ c

.d

Ej -0.5

v

$

-1

-1.5

t -2 -0.2 -0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Figure 5: Plot of dz/dt versus z for the seven conductance model. Both minimum and maximum values of dz/dt at a given z value have been plotted. Two lines appear in the bursting region due to fluctuationsin the calcium level during bursting activity. Distinct behaviors obtained for different values of z are indicated by the inserts. Locked-up refers to a permanently depolarized state. The quasi-equilibriumvalue of z produces bursting behavior as indicated by the zero crossing of dz/dt. Model parameters used for Figures 5 and 6 are as in LeMasson et al. (1992). neurons described by identical sets of equations with the same parameter values. The identical activity of the two model neurons when they are uncoupled is shown in Figure 6A. The two neurons are then coupled through an electrical synapse (synaptic current proportional to the voltage difference between the two neurons) that is likewise completely symmetrical. Figure 6B shows the steadystate activity of the coupled network. The two neurons burst in unison. To examine the intrinsic properties of the two neurons individually, we uncouple them once again and show in Figure 6C their activity immediately after they are decoupled. Despite the fact that two model neurons are governed by identical sets of equations, the coupling between them has causes one neuron to display intrinsic bursting activity while the other fires tonically in isolation. The symmetric, two-cell network has spontaneously differentiated into a circuit involving a pacemaker and a follower neuron. If the two neurons are left uncoupled, the regulation process will eventually return them to their initial identical states as seen in Figure 6D.


838

To study this system, we monitor the maximal conductances of the two neurons and see how the coupling between them affects their behavior by performing the following numerical experiment. We hold 21, the z value for one of the two neurons, fixed but allow z2 to evolve according to equation 4.2 until it reaches its quasi-equilibrium value. This value will

E 0.24

0.23

3- 0.22 Y

F

+ 4

0.21

2 0.2 0.19

0.19

0.2

0.21

0.22

0.23

0.24

[Ca++I,(vM)


839

depend on the fixed value ZI we have chosen for the first neuron because the two neurons are coupled and this coupling effects the behavior of zz through effects like those shown in Figure 4. We then record the timeaveraged intracellular calcium concentrations of the two neurons, [Ca] and [Ca],. By repeating this process for many different holding values 21 we obtain the curves shown in Figure 6E. Actually, only one of these curves corresponds to the procedure just outlined while the other is its reflection obtained by interchanging the roles of neuron 1 and neuron 2. One curve thus shows the quasi-equilibrium calcium concentrations of neuron 2 when neuron 1 is held fixed and the other the quasi-equilibrium concentrations of neuron 1 when neuron 2 is held fixed. The values of z1 and zz determine the maximal conductances of the two neurons through the relation 5.1 and this in turn will control their intracellular calcium concentrations. Because z and [Ca] are related, we can use either the value of z or the value of the intracellular calcium concentration to characterize the balance of inward and outward maximal conductances. Up to now, we have used z because it is directly related to the maximal conductances through equation 5.1. However, to illustrate the two-neuron network we use the timeaverage of the calcium concentration in the two neurons rather than their z values because the fluctuations caused by the bursting activity of the two neurons are smaller for the timeaverage calcium concentration making the plot clearer. 0thenvise, the two approaches are completely equivalent. The quasi-steady-state configurations of the fully regulated, interacting, two-neuron circuit are given by the points where the two curves in Figure 6E cross. The interesting feature of this particular network is that the lines cross in three places. The middle of these three crossings is the symmetric equilibrium point where the calcium concentrations, the z values, and the maximal conductances of the two neurons are identical. However, as is typical in cases of spontaneous symmetry breaking, this point is unstable for this particular network. The other two crossings are stable equilibrium points and they have the novel feature that the intrinsic conductances of the two neurons are different. One neuron exhibits a higher calcium concentration than the other so, according to equation 4.2, its z value will be lower than that of the other neuron. As a Figure 6: Facing page. (A) The behavior of two identical model neurons before they are coupled. (B) Electrical coupling between the neurons results in a bursting two-cell network. (C)Decoupling the two neurons reveals their intrinsic properties and indicates that one is acting as a pacemaker and the other as a tonically firing follower. (D) Long after the two neurons are decoupled, the regulation mechanism has returned them to their original identical states. (E) A plot of the time-averaged calcium concentration of one neuron when the other neuron's regulation dynamics is held fixed. The three crossing points are equilibrium points. The central, symmetry crossing is unstable while the two outer crossing are stable quasi-steady-stateswith nonsymmetric properties.

840


result, one of the neurons will have smaller inward and larger outward conductances than the other neuron as given by equation 5.1. This is what causes the spontaneous differentiation of intrinsic properties seen in Figure 6C. The symmetry-breakingphenomenon that we have discussed requires electrical coupling between the two neurons that lies in a specific range. The coupling must be strong enough so that the two neurons have an impact on each other, but not so strong that their activity is forced to be identical. 6 Discussion

We have used a single second messenger, the intracellular calcium concentration, to act as the negative feedback element linking the maximal conductances of a model neuron to its electrical activity. If similar mechanisms exist in real neurons they may be controlled by multiple second messengers. In addition, we have taken a particularly simple form of the regulatory equations by choosing a single sigmoidal curve (and its flipped version) for all of the conductances. What is surprising about these simplifications is that they nevertheless allow the full range of behaviors of the model neuron to be explored as seen in Figure 5. The parameterization of equation 5.1 may thus be useful even in cases where dynamic regulation is not being studied. Any scheme based on a single second messenger will similarly probe a single line in the multidimensional space of maximal conductance values characterizing a particular model. The simple form of the functions Fi we used means this line is given by the simple equation 5.1, more general forms of the Fi would result in more complex curves. Nevertheless, it should be possible to find a variable like z, even with nonidentical forms for the Fi, that parameterizes path length along this general curve. As a result, we expect that the behavior of the model in the more general case will be qualitatively similar to the simple case we have analyzed. This argument also applies to models in which some of the maximal conductances are not regulated at all. We have thus far studied dynamic regulation as a global phenomenon in single compartment models. A local form of dynamic regulation could have important consequences in a multicompartment model of a neuron. In such a model, the density of channels in various parts of the neuron would be correlated with the time-average calcium concentration in that region. This provides a mechanism for controlling the distribution of conductances over the surface of a neuron (for a different approach to this problem see Bell 1991) and for correlating the local channel density with structural and geometrical characteristics affecting calcium buffering and diffusion (preliminary work done in collaboration with M.Siegel). The dynamic regulation scheme was motivated by a need to build more robust neuronal models, and Figure 3 clearly shows that this goal


841

has been achieved. The fact that the dynamically regulated model also exhibits shifts in intrinsic characteristics due to interactions with other neurons is an interesting and unavoidable consequence of this robustness. If maximal conductances depend on activity, neurons in networks will be affected by each other and will adapt accordingly. O u r two-neuron model resulted in a n oscillating circuit with a pacemaker and a follower neuron. This differentiation was caused solely by the interaction of the two neurons. Either neuron could have developed into the pacemaker with the other becoming the follower. As in this simple example, it should be possible for identical dynamically regulated model neurons to self-assemble into more complex networks in which they play welldefined but different functional roles.

Acknowledgments We wish to thank Eve Marder for her collaboration during the development of these ideas and John Rinzel for helpful comments about the mathematical reduction of slow/fast systems. Research supported by National Institute of Mental Health Grant MH-46742 and National Science Foundation Grant DMS-9208206.

References Bell, A. 1992. Self-Organization in real neurons: anti-Hebb in 'channel space'? In Neural lnfonnation Processing Systems 4, J. E. Moody and S. J. Hanson, eds., pp. 59-66. Morgan Kaufmann, San Mateo, CA. Buchholtz, F., Golowasch, J., Epstein, I., and Marder, E. 1992. Mathematical model of an identified stomatogastric neuron. I. Neurophysiol. 67,332-340. Chad, J. E., and Eckert, R. 1986. An enzymatic mechanism for calcium current inactivation in dialysed Helix neurones. 1.Physiol. (London) 378, 31-51. Franklin, J. L., Fickbohm, D. J., and Willard, A. L. 1992. Long-term regulation of neuronal calcium currents by prolonged changes of membrane potential. 1.Neurosci. 12, 1726-1735. Hemmick, L. M., Perney, T. M., Flamm, R. E., Kaczmarek, L. K., and Birnberg, N. C. 1992. Expression of the h-ras oncogene induces potassium conductance and neuron-specific potassium channel mRNAs in the AtT20 cell line. 1.Neurosci. 12, 2007-2014. Hodgkin, A. L., and Huxley, A. F. 1952. A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Phys. 117,500-544. Kaczmarek, L. K. 1987. The role of protein kinase C in the regulation of ion channels and neurotransmitter release. TINS 10,30-34. Kaczmarek, L. K., and Levitan, I. B., eds. 1987. Neuromodulation. The Biochemical Control of Neuronal Excitability. Oxford Univ. Press, New York, NY. Kennedy, M. B. ed. 1989. TINS 12,417-479.

842


Koch, C., and Segev, I., eds. 1989. Methods in Neuronal Modeling. MIT Press, Cambridge, MA. LeMasson, G., Marder, E., and Abbott, L. F. 1992. Activity-dependent regulation of conductances in model neurons. Science 259,1915-1917. Morgan, J. I., and Curran T. 1991. Stimulus-transcription coupling in the nervous system: Involvement of the inducible proto-oncogenes fos and jun. Annu. Rev.Neurosci. 14,421-451. Morris, C., and Lecar, H. 1981. Voltage oscillations in the barnacle giant muscle fiber. Biophys. 1. 35, 193-213. Murphy, T. H.,Worley, P. F., and Baraban, J. M. 1991. L-type voltage-sensitive calcium channels mediate synaptic activation of immediate early genes. Neuron 7,625-635. Rasmussen, H., and Barrett, P. Q. 1984. Calcium messenger system: An integrated view. Physiol. Rev. 64, 938-984. Ross, W.M. 1989. Changes in intracellular calcium duting neuron activity. Annu. Rev.Physiol. 51,491-506. Sheng, M., and Greenberg, M. E. 1990. The regulation and function of c-fos and other immediate early genes in the nervous system. Neuron 4,477485. Smeyne, R.J.,Schilling, K., Robertson, L., Luk, D., Oberdick, J., Curran, T.,and Morgan, J. 1992. Fos-lacZ transgenic mice: Mapping sites of gene induction in the central nervous system. Neuron 8, 13-23. Received 10 December 1992; accepted 26 February 1993.

This article has been cited by: 1. Niranjan Chakravarthy, Kostas Tsakalis, Shivkumar Sabesan, Leon Iasemidis. 2009. Homeostasis of Brain Dynamics in Epilepsy: A Feedback Control Systems Perspective of Seizures. Annals of Biomedical Engineering 37:3, 565-585. [CrossRef] 2. Mario F. Simoni, Stephen P. DeWeerth. 2007. Sensory Feedback in a Half-Center Oscillator Model. IEEE Transactions on Biomedical Engineering 54:2, 193-204. [CrossRef] 3. M.F. Simoni, S.P. Deweerth. 2006. Two-Dimensional Variation of Bursting Properties in a Silicon-Neuron Half-Center Oscillator. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14:3, 281-289. [CrossRef] 4. T. Kohno, K. Aihara. 2005. A MOSFET-Based Model of a Class 2 Nerve Membrane. IEEE Transactions on Neural Networks 16:3, 754-773. [CrossRef] 5. Astrid A Prinz, Dirk Bucher, Eve Marder. 2004. Similar network activity from disparate circuit parameters. Nature Neuroscience 7:12, 1345-1352. [CrossRef] 6. M.F. Simoni, G.S. Cymbalyuk, M.E. Sorensen, R.L. Calabrese, S.P. DeWeerth. 2004. A Multiconductance Silicon Neuron With Biologically Matched Dynamics. IEEE Transactions on Biomedical Engineering 51:2, 342-354. [CrossRef] 7. Jonghan Shin , Christof Koch , Rodney Douglas . 1999. Adaptive Neural Coding Dependent on the Time-Varying Statistics of the Somatic Input CurrentAdaptive Neural Coding Dependent on the Time-Varying Statistics of the Somatic Input Current. Neural Computation 11:8, 1893-1913. [Abstract] [PDF] [PDF Plus] 8. Jorge Golowasch , Michael Casey , L. F. Abbott , Eve Marder . 1999. Network Stability from Activity-Dependent Regulation of Neuronal ConductancesNetwork Stability from Activity-Dependent Regulation of Neuronal Conductances. Neural Computation 11:5, 1079-1096. [Abstract] [PDF] [PDF Plus] 9. M.F. Simoni, S.P. DeWeerth. 1999. Adaptation in a VLSI model of a neuron. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 46:7, 967-970. [CrossRef] 10. Eve Marder. 1998. FROM BIOPHYSICS TO MODELS OF NETWORK FUNCTION. Annual Review of Neuroscience 21:1, 25-45. [CrossRef] 11. Arthur Sherman. 1994. Anti-phase, asymmetric and aperiodic oscillations in excitable cells—I. Coupled bursters. Bulletin of Mathematical Biology 56:5, 811-835. [CrossRef] 12. Bard Ermentrout , Nancy Kopell . 1994. Learning of Phase Lags in Coupled Neural OscillatorsLearning of Phase Lags in Coupled Neural Oscillators. Neural Computation 6:2, 225-241. [Abstract] [PDF] [PDF Plus]

ARTICLE

Communicated by Idan Segev

Limitations of the Hodgkin-Huxley Formalism: Effects of Single Channel Kinetics on Transmembrane Voltage Dynamics Adam F. Strassberg Computation and Neural Systems Program, California Institute of Technology, Pasadena, C A 91125 U S A

Louis J. DeFelice Department of Anatomy and Cell Biology, Emory University School of Medicine, Atlanta, G A 30322 U S A

A standard membrane model, based on the continuous deterministic Hodgkin-Huxley equations, is compared to an alternative membrane model, based on discrete stochastic ion channel populations represented through Marlcov processes. Simulations explore the relationship between these two levels of description: the behavior predicted by the macroscopic membrane currents versus the behavior predicted by their microscopic ion channels. Discussion considers the extent to which these random events underlying neural signals mediate random events in neural computation.

1 Introduction Action potentials within the neuron arise from the time-variant and voltage-dependent changes in the conductance of the neural membrane to specific ions. Hodgkin and Huxley based their famous model of active membrane on an assumption that the ion permeation processes existing within the membrane can be approximated as both continuous and deterministic (Hodgkin and Huxley 1952). However, the permeation processes existing within active membrane are known to be neither continuous nor deterministic. Active membrane is studded with discrete ion channels undergoing random fluctuations between open and closed stable states (Hille 1992). There have been few studies of the relationship between these two levels of description, the discrete stochastic behavior of the microscopic ion channels versus the continuous deterministic behavior of their macroscopic membrane currents (Clay and DeFelice 1983). This paper investigates these two regimes of activity through a comparison of the standard membrane model, based on the continuous Hodgkin-Huxley equations, to an alternative membrane model, based Neural Computation 5,843-655 (1993) @ 1993 Massachusetts Institute of Technology

844

Adam F. Strassberg and Louis J. DeFelice

on discrete ion channel populations represented through Markov processes. When both models are used to simulate the active membrane of the squid Loligo giant axon, the convergence of the alternative model to the standard model can be examined. Under certain conditions, the b e havior predicted by the alternative model will diverge from the behavior predicted by the standard model. Under these conditions, simulations suggest that random microscopic behavior, such as single channel fluctuations, becomes capable of generating random macroscopic behavior, such as entire action potentials. 2 Methods

The neural membrane of a space-clamped squid giant axon is modeled with an equivalent electric circuit. The space-clamp technique removes the spatial dependence of the membrane voltage and so the axon becomes effectively equivalent to an isopotential patch of membrane. A simple lumped circuit model thus can interpret the electrical characteristicsof the membrane. The macroscopic membrane conductances are represented by the conductive elements gNa, gK, and gL and the transmembrane voltage V, behaves according to the equation:

Behavior predicted by the standard Hodgkin-Huxley equations for the time-variant and voltage-dependent membrane conductances gNa and gK is compared to behavior predicted by alternative descriptions for these conductances based on their underlying ion channel populations. Ion channel activity is modeled well by Markov processes. Each channel is assumed to randomly fluctuate between only a finite number of discrete stable states. Transition probabilities between these stable states are assumed to depend on the present stable state and the present membrane voltage and to be independent of the duration for which this present stable state has been occupied. Such Markov assumptions are used to interpret the data from patch-clamp experiments on single ion channels. These data often are too limited to allow for the isolation of a single Markov kinetic scheme from the several alternative schemes (Strassberg and DeFelice 1992; Kienker 1989; Clay and DeFelice 1983; Conti et al. 1975; Hille 1992). For this simulation of the ion channel populations underlying the membrane conductances gNa and gK, the simplest noncooperative and serial schemes have been chosen from the set of schemes capable of generating the desired macroscopic behavior. Llano et al. (1988) have patch-clamped voltage-gated potassium channels in the active membrane of the squid giant axon. These channels show a single open state with a potassium ion conductance of 20 pS

-

Limitations of the Hodgkin-Huxley Formalism

845

(Llano et al. 1988). The following Markov kinetic scheme will reproduce the observed microscopic potassium channel behavior:

[no] '2' [nl] '2 4n

2411

[n2]

'2 34"

[n3]

2 44n

[n4]

(2.2)

where [nil refers to the number of channels within the population currently in stable state ni, n4 labels the single open state, and a, and /?n are the voltage-dependent rate constants from the Hodgkin-Huxley formalism (Armstrong 1969; Llano et al. 1988; Fitzhugh 1965; Hille 1992). Vandenberg and Bezanilla (1988) have patch-clamped voltage-gated sodium channels in the active membrane of the squid giant axon. These channels show a single open state with a sodium ion conductance of 20 pS (Bezanilla 1987; Vandenberg and Bezanilla 1988). The following Markov kinetic scheme will reproduce the observed microscopic sodium channel behavior: N

(2.3)

where [rnih,]refers to the nurr.,er of channels with..\ the population currently in stable state rnihj, m3hl labels the single open state, and a,, P,, ahr and /?h are the voltage-dependent rate constants from the HodgkinHuxley formalism (Bezanilla 1987; Vandenberg and Bezanilla 1988; Hille 1992; Fitzhugh 1965). Simulation parameters are chosen to be identical to those values for squid axonal membrane used by Hodgkin and Huxley in their seminal paper (Hodgkin and Hwley 1952):

C,

T EL gL EK

gK DK 7~

EN^ gNa

DN

ma

1 pF/cm2 Membrane capacitance 6.3 "C Temperature 10.613 mV Leakage Nernst potential 0.3 mS/cm2 Leakage conductance -12.0 mV Potassium Nernst potential 36 mS/cm2 Maximal potassium conductance Potassium ion channel density 18 channels/pm2 Potassium channel open state conductance 20 ps 115.0 mV Sodium Nernst potential 120 mS/cm2 Maximal sodium conductance 60 channels/pm2 ~Sodium ion channel density Sodium channel open state conductance 20 p s

For a membrane model using discrete stochastic channel populations

Adam F. Strassbergand Louis J. DeFelice

846

with the given Markov kinetics, 2.2 and 2.3, the potassium and sodium membrane conductances will satisfy

gK(v,t ) = yK[n4]

g N a ( v , t ) = ?Na[m3h]

For a membrane model using continuous deterministic Hodgkin-Huxley equations, the potassium and sodium membrane conductances will satisfy

gK(v,t ) = g K n 4

gNa(v,

t ) =gNam3h

All simulations of membrane behavior are performed using GENESIS,' an object-oriented general purpose neural simulator for the UNIX/Xwindows environment. Two new GENESIS objects are designed to model squid potassium and sodium ion channel populations undergoing Markov kinetics as given by 2.2 and 2.3 respectively2 GENESIS scripts are produced for an isopotential patch of squid giant axon membrane under both the voltage-clamp and free-running experimental paradigm with membrane conductances represented through either Hodgkin-Huxley equations or channel populations with Markov kinetics. 3 Results

Figure 1 shows the voltage-clamp step response of the membrane conductances 8K and &. Both the continuous Hodgkin-Huxley equations and the discrete channel population Markov models are used alternatively to represent the membrane conductances. Note that as the size of each channel population is increased, the response from the discrete channel model converges to the behavior predicted by the continuous Hodgkin-Huxley currents. Figure 2 shows the response of a free-running membrane patch to a constant current injection and Figure 3 shows the resting response with no current injection. Figure 4 compares the mean firing frequencies of these responses to the membrane surface area of their underlying patches. For fixed channel densities, as the membrane surface area is increased, the response from the simulation of a constant density of ion channels converges to the response from the standard 'GENESIS 01989 designed by Matt Wilson and Jim Bower at California Institute of Technology. Inquiries may be directed to [email protected] [email protected]. %Vera1 new objects and scripts have been incorporated into the source code of GENESIS v1.4 as of July,1992. Inquiries may be directed to [email protected].


0

5

10

1

Tme(rnsec)

step

847

15

20

Figure 1: Voltage-clamp step response of membrane conductance. Membrane voltage is clamped to V,t and stepped to V,t 50.0 mV at t = 5.0 msec. The response of each active membrane conductance is simulated with varying populations of discrete channels undergoing appropriate voltage-dependent Markov kinetics. These responses are compared to the behaviors predicted by the continuous Hodgkin-Huxley equations. Note that all outputs are normalized and displaced appropriately for display. As the size of each channel population is increased, the response from the discrete channel model converges to the behavior predicted by the continuous Hodgkin-Huxley equations, for both (top) the potassium conductance & ( t ) and (bottom) the sodium conductance gNa(t).

+


848

1

0

1

1

1

1

1

1

1

1

1

1

40

1

60

1

1

1

I

80

I

1

1

100

TinPlnsec)

Figure 2 Membrane response with injection current. The membrane model is simulated with standard biophysical parameters for squid axonal membrane (Cm, ENa, EK,EL,gL) and with constant current injection (1inject= 100 pA/pm2). The continuous Hodgkin-Huxley equations and the discrete channel populations are used alternatively to represent the membrane conductances gNa and gK. As the membrane surface area is increased, the response from the channel model converges to the response from the standard Hodgkin-Huxley model. Both models predict that a regular train of action potentials will occur when this constant current is injected. Note that, as the membrane surface area is decreased, the regularity of the spike train generated by the channel model diverges from the behavior predicted by the Hodgkin-Huxley model.

Hodgkin-Huxley model. Both models predict that, for large membrane surface areas, a train of action potentials will occur when constant current is injected and that no activity will occur when no current is injected. However, note that, as the membrane suface area is decreased, the be-


l

I

I

I

I

l

I

I

I

l

l

I

849

I

l

I

l

I

I

J

Continuous

0

20

60

40

80

100

Timelmsecl

Figure 3 Membrane response without injection current. The membrane model is simulated with standard biophysical parameters for squid axonal membrane (CmrENalEK,EL,gL) and with no current injection (Iinp. = 0 pA/pm2). The continuous Hodgkin-Huxley equations and the discrete channel populations are used alternatively to represent the membrane conductancesgNa and gK. As the membrane surface area is increased, the response from the channel model converges to the response from the standard Hodgkin-Huxley model. Both models predict that no activity occurs when no current is injected. However, as the membrane surface area is decreased, the active behavior predicted by the channel model diverges dramatically from the lack of activity predicted by the Hodgkin-Huxley model.

havior predicted by the channel model diverges dramatically from the behavior predicted by the Hodgkin-Huxley model. These simulations suggest that, for an isopotential membrane patch with constant densities


850

t

Channel model response t o :100 PA/&

Hcdkqin-Huxley lode1 response

to

:

0 PA/&

Figure 4 Mean firing frequency versus membrane area. For a given membrane area and a given constant current injection, the number of attendant action potentials is averaged over a l sec duration to derive a mean firing frequency. As membrane area increases, the firing frequencies from the channel model converge to the firing frequencies from the Hodgkin-Huxley model. However, as membrane area decreases, these responses diverge dramatically. These simulations suggest that, as the area of an isopotential membrane patch is decreased, the voltage noise from single channel fluctuations will become capable of eliciting entire action potentials. (Over the smaller membrane surface areas, the graph shows the mean firing rates to first increase and then decrease. For such small regions, the opening of a single channel will depolarize the membrane to EN^ and so the definition of "action potential" becomes somewhat obfuscated.)

of sodium and potassium channels, as the membrane area is decreased, the fluctuations of single channels will become capable of eliciting entire action potentials.


851

4 Discussion

The standard membrane model, based on the Hodgkin-Huxley equations, has been compared to an alternative membrane model, based on ion channel populations represented through Markov processes. When both models are used to simulate the active membrane of the squid Loligo giant axon, the explicit convergence of the alternative model to the standard model can be observed. However, under certain conditions, the behavior predicted by the alternative model diverges dramatically from the behavior predicted by the standard model. 4.1 Membrane Voltage Perturbations Due to Single Ion Channel Fluctuations. The divergent behavior can be explained through an analysis of the voltage perturbations across the membrane due to single ion channel fluctuations. Whenever a single ion channel moves from a closed state into an open state, the transmembrane voltage V , behaves according to the first-order transient:

Vm(t)= AV,(l

-e-;)

+ vest

The magnitude AV,, of the resultant voltage perturbation is mediated by a voltage divider between the conductance of the opened ion channel and the conductance of the surrounding membrane, which includes both the leakage conductance and the summed conductances of all other currently opened ion channels. The rise-time r of this resultant voltage perturbation is equal to the membrane capacitance divided by the total membrane conductance. Note that there will be a correction term to the usual area-independent T because the total membrane conductance is now the sum of both the conductance of the membrane surrounding the opened channel, which does scale with area, and the conductance of the individual opened channel, which does not scale with area. For a given ion channel, the magnitude of the open state conductance and the voltage dependence of the stable state kinetics are independent of the surface area of the surrounding membrane. However, when this ion channel enters the open state, both the magnitude AV,,, and the risetime 7 of the resultant voltage perturbation across the membrane are dependent on the surface area .of the surrounding membrane. For the specific biophysical parameters of squid axonal membrane, the voltage perturbation due to the random opening of a single sodium channel simplifies to

where A is the total membrane surface area (Strassberg and DeFelice 1992). As this surface area A is decreased, the magnitude AV,, increases

854


of averaged data trials, which show much less variability. However, the fluctuations of single channels are probabilistic events with a high degree of variability from one trial to the next. With the current absence of any strong consensus on how the nervous system encodes information (beyond the level of sensory transduction), one is unable to distinguish strongly the “signal” from the “noise.” The filtering and averaging of the data to remove the “noise” thus may be removing important components of the data. (Bower and Koch 1992). Although the full prevalence and abundance of spontaneous action potentials are presently unknown, many potential roles for such spontaneous activations do exist in neural computation. While the effect of noise in a sensory system may be generally detrimental, the effect of noise in a planning, coordination, or motor system would not necessarily be as severe. For example, spontaneous action potentials could stop the repetition of unrewarded preprogrammed behaviors or perhaps even allow for the generation of entirely new responses to novel stimuli. During neurodevelopment, random activity could play a role in the coordination, correlation, and robust tuning of receptive field structures. From neuroethology, we know that organisms generate a host of spontaneous behavior patterns on the macroscopic level, thus it is reasonable to hypothesize that such spontaneous macroscopic behaviors arise from spontaneous microscopic behaviors. This paper has used simulation and analysis to show that theoretical mechanisms exist for both the attenuation and the amplification of single channel noise. Experimental convention typically has ignored the underlying stochastic nature of the neuron in favor of the averaged neural response properties. However, as more physiological data on spontaneous activations do become available, the degree to which the random microscopic events underlying neural signals mediate random macroscopic events in neural computation will become more apparent. Acknowledgments This material is based on work supported under a National Science Foundation Graduate Fellowship and an NIH HL-27385. We would like to express our deep appreciation to Dr. Christof Koch for his comments and suggestions throughout the preparation of this manuscript. We also would like to thank Hsiaolan Hsu and Dr. Henry Lester for helpful insights. References Armstrong, C. M. 1969. Inactivation of the potassium conductance and related phenomena caused by quaternary ammonium ion injected in squid axons. 1.Gen. Physiol. 54, 553-575.

Limitations of the Hodgkin-Hwdey Formalism

855

Bezanilla, F. 1987. Single sodium channels from the squid giant axon. Biophys. J. 52, 1087-1090. Bower, J., and Koch, C. 1992. Experimentalists and modelers: can we all just get along? Tr. Neurosci. 15,458461. Clay, J., and DeFelice, L. 1983. Relationship between membrane excitability and single channel open-close kinetics. Biophys. J. 42, 151-157. Conti, F., DeFelice, L. J., and Wanke, E. 1975. Potassium and sodium ion current noise in the membrane of the squid giant axon. J. Physiol. (London) 248,4582. Fitzhugh, R. 1965. A kinetic model of the conductance changes in nerve membrane. J. Cell. Comp. Physiol. 66,Suppl., 111-117. Franciolini, F. 1987. Spontaneous firing and myelination of very small axons. J. Theor. Bid. 128, 127-134. Hille, B. 1992. Ionic Channels of Excitable Membrane, 2nd ed. Sinauer Associates, Sunderland, MA. Hodgkin, A. L., and Huxley, A. F. 1952. A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. (London) 117,500-544. Kienker, P. 1989. Equivalence of aggregated Markov models of ion-channel gating. Proc. R. SOC.London B 236,269-309. Koch, C., Zador, A., and Brown, T. H. 1992. Dendritic spines: Convergence of theory and experiment. Science 256,973-974. Llano, I., Webb, C. K., and Bezanilla, F. 1988. Potassium conductance of the squid giant axon. J.Gen. Physiol. 92, 179-196. Segev, I., and Rall, W. 1988. Computational study of an excitable dendritic spine. J. Neurophys. 60,499-523. Strassberg, A. F., and DeFelice, L. J. 1992. Limitations of the Hodgkin-Huxley formalism. Computation and Neural Systems Program-Memo 24, California Institute of Technology. Vandenberg, C. A., and Bezanilla, F. 1988. Single-channel, macroscopic and gating currents from Na channels in squid giant axon. Biophys. J. 53,226a.

Received 16 March 1992; accepted 26 February 1993.

This article has been cited by: 2. B. Sengupta, S. B. Laughlin, J. E. Niven. 2010. Comparison of Langevin and Markov channel noise models for neuronal signal generation. Physical Review E 81:1. . [CrossRef] 3. Shawn R Lockery, Miriam B Goodman. 2009. The quest for action potentials in C. elegans neurons hits a plateau. Nature Neuroscience 12:4, 377-378. [CrossRef] 4. Ian C. Bruce. 2009. Evaluation of Stochastic Differential Equation Approximation of Ion Channel Gating Models. Annals of Biomedical Engineering 37:4, 824-838. [CrossRef] 5. YuBing Gong, YanHang Xie, Bo Xu, XiaoGuang Ma. 2009. Effect of gating currents of ion channels on the collective spiking activity of coupled Hodgkin-Huxley neurons. Science in China Series B: Chemistry 52:1, 20-25. [CrossRef] 6. Yubing Gong, Yanhang Xie, Yinghang Hao. 2009. Coherence resonance induced by the deviation of non-Gaussian noise in coupled Hodgkin–Huxley neurons. The Journal of Chemical Physics 130:16, 165106. [CrossRef] 7. C. M. Gómez. 2008. Numerical exploration of the influence of neural noise on the psychometric function at low stimulation intensity levels. Journal of Biosciences 33:5, 743-753. [CrossRef] 8. Marifi Güler. 2008. Detailed numerical investigation of the dissipative stochastic mechanics based neuron model. Journal of Computational Neuroscience 25:2, 211-227. [CrossRef] 9. A. Aldo Faisal, Luc P. J. Selen, Daniel M. Wolpert. 2008. Noise in the nervous system. Nature Reviews Neuroscience 9:4, 292-303. [CrossRef] 10. YuBing Gong, Bo Xu, XiaoGuang Ma, JiQu Han. 2008. Effect of channel block on the collective spiking activity of coupled stochastic Hodgkin-Huxley neurons. Science in China Series B: Chemistry 51:4, 341-346. [CrossRef] 11. Marifi Güler. 2007. Dissipative stochastic mechanics for capturing neuronal dynamics under the influence of ion channel noise: Formalism using a special membrane. Physical Review E 76:4. . [CrossRef] 12. Ian C. Bruce. 2007. Implementation Issues in Approximate Methods for Stochastic Hodgkin–Huxley Models. Annals of Biomedical Engineering 35:2, 315-318. [CrossRef] 13. G Schmid, I Goychuk, P Hänggi. 2006. Capacitance fluctuations causing channel noise reduction in stochastic Hodgkin–Huxley systems. Physical Biology 3:4, 248-254. [CrossRef] 14. Yubing Gong, Maosheng Wang, Zhonghuai Hou, Houwen Xin. 2005. Optimal Spike Coherence and Synchronization on Complex Hodgkin-Huxley Neuron Networks. ChemPhysChem 6:6, 1042-1047. [CrossRef] 15. Loredana Mereuta, T. Luchian. 2005. How could a chirp be more effective than a louder clock-resonant transfer of energy between subthreshold excitation pulses

and excitable tissues. Journal of Cellular and Molecular Medicine 9:2, 446-456. [CrossRef] 16. G Schmid, I Goychuk, P Hänggi. 2004. Effect of channel block on the spiking activity of excitable membranes in a stochastic Hodgkin–Huxley model. Physical Biology 1:2, 61-66. [CrossRef] 17. J. Casado, J. Baltanás. 2003. Phase switching in a system of two noisy Hodgkin-Huxley neurons coupled by a diffusive interaction. Physical Review E 68:6. . [CrossRef] 18. Amit Manwani , Peter N. Steinmetz , Christof Koch . 2002. The Impact of Spike Timing Variability on the Signal-Encoding Performance of Neural Spiking ModelsThe Impact of Spike Timing Variability on the Signal-Encoding Performance of Neural Spiking Models. Neural Computation 14:2, 347-367. [Abstract] [PDF] [PDF Plus] 19. J. Shuai, P. Jung. 2002. Optimal Intracellular Calcium Signaling. Physical Review Letters 88:6. . [CrossRef] 20. G Schmid, I Goychuk, P Hänggi. 2001. Stochastic resonance as a collective property of ion channel assemblies. Europhysics Letters (EPL) 56:1, 22-28. [CrossRef] 21. P Jung, J. W Shuai. 2001. Optimal sizes of ion channel clusters. Europhysics Letters (EPL) 56:1, 29-35. [CrossRef] 22. Amit Manwani , Christof Koch . 1999. Detecting and Estimating Signals in Noisy Cable Structures, I: Neuronal Noise SourcesDetecting and Estimating Signals in Noisy Cable Structures, I: Neuronal Noise Sources. Neural Computation 11:8, 1797-1829. [Abstract] [PDF] [PDF Plus] 23. Elad Schneidman , Barry Freedman , Idan Segev . 1998. Ion Channel Stochasticity May Be Critical in Determining the Reliability and Precision of Spike TimingIon Channel Stochasticity May Be Critical in Determining the Reliability and Precision of Spike Timing. Neural Computation 10:7, 1679-1703. [Abstract] [PDF] [PDF Plus] 24. G Renversez, O Parodi. 1996. Potential distribution on a neuronal somatic membrane during an action potential. Europhysics Letters (EPL) 36:4, 313-318. [CrossRef] 25. Alain Destexhe, Zachary F. Mainen, Terrence J. Sejnowski. 1994. Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. Journal of Computational Neuroscience 1:3, 195-230. [CrossRef] 26. M. Giugliano, M. ArsieroBiological Neuronal Networks, Modeling of . [CrossRef]

Communicated by Christof Koch

Two-Dimensional Motion Perception in Flies A. Borst

M. Egelhaaf Max-Planck-lnstitut fiir biologische Kybernetik, Spemannstrasse 38,7400 Tiibingen, Germany

H. S . Seung* Racah lnstitute of Physics and Center for Neural Computation, Hebrew University, Jerusalem 91904, Israel

We study two-dimensional motion perception in flies using a semicircular visual stimulus. Measurements of both the H1-neuron and the optomotor response are consistent with a simple model supposing spatial integration of the outputs of correlation-type motion detectors. In both experiment and model, there is substantial H1 and horizontal (yaw) optomotor response to purely vertical motion of the stimulus. We conclude that the fly's optomotor response to a two-dimensional pattern, depending on its structure, may deviate considerably from the direction of pattern motion. 1 Introduction

The projection of the velocity vectors of objects moving in three-dimensional space on the image plane of an eye or a camera can be described as a vector field. This two-dimensional velocity field is time-dependent and assigns the direction and magnitude of a velocity vector to each point in the image plane. However, the velocity field is a purely geometric concept and does not directly represent the input of a visual system. Instead, the only information available to a visual system is given by the time-dependent brightness values as sensed by photoreceptors in the image plane. The problem of motion perception has often been posed as that of recovering the velocity field from these brightness values. For the case of simple translation of a Lambertian surface under uniform illumination, this computation can be done (Verri and Poggio 1989). Such a physical motion leads to the translation of a brightness pattern across the image plane. Several known local algorithms (Reichardt et al. 1988; Srinivasan 1990; Uras et al. 1988) recover the correct velocity field, which is constant in space and time. Algorithms utilizing a smoothness ~~~

'Present address: AT&T Bell Laboratories, Murray Hill, NJ 07974 USA.

Neural Computation 5, 8!%-868 (1993) @ 1993 Massachusetts Institute of Technology

Motion Perception in Flies

857

regularizer (Horn and Schunck 1981; Hildreth and Koch 1987) also perform well in extracting the true velocity. All these algorithms fail to yield a unique answer only for the special case of one-dimensional patterns. This is because a moving one-dimensional pattern is consistent with an infinite number of velocity fields (Ullman 1983; Horn and Schunk 1981; Hildreth and Koch 1987; Reichardt et al. 1988). In contrast, the superposition of two differently oriented one-dimensional sine gratings, a plaid pattern, has a uniquely determined velocity vector. The direction of motion of such a two-dimensional (2d) pattern, which is different from the orientations of its one-dimensional (Id) component gratings, is perceived by human observers under certain conditions (Adelsen and Movshon 1982; Ferrera and Wilson 1990; Stone et al. 1990; Stoner et al. 1990). On the basis of physiological experiments with plaid stimuli, Movshon and co-workers (1986) have argued that motion processing in the primate visual system takes place in two stages. The first stage is composed of local movement detectors in areas V1 and MT sensitive to the orientation of the components. The second stage of processing is composed of neurons in MT that respond to the direction of pattern motion, presumably computing it from the output of the first stage. In this work, we examine whether the fly visual system computes the direction of motion of a 2d pattern. In the past, the fly has proven to be an excellent model system for analyzing motion detection. Most notably, much is known about the structure and physiology of its visual ganglia and the motion-dependent behaviors controlled by them (fw review, see Egelhaaf et al. 1988; Hansen and Egelhaaf 1989; Egelhaaf and Borst 1993). One such motion-dependent behavior is the optomotor response, in which the fly tends to move so as to stabilize a moving visual surround (Fermi and Reichardt 1963; Gotz 1972). A simple model of the fly’s optomotor pathway has been quite successful in accounting for both neurophysiological and behavioral data. According to this integrated correlation model, there are local movement detectors of the correlation type (Fig. 1A) (Hassenstein and Reichardt 1956; Reichardt 1961; Borst and Egelhaaf 1989) that are organized in two-dimensional retinotopic arrays covering the entire visual field. A set of identified, directionally selective, large-field interneurons in the third visual ganglion spatially integrates over the responses of horizontally oriented detectors in this array (Hansen and Egelhaaf 1989). The yaw optomotor response is a low-pass filtered version of the output of this horizontal system (Egelhaaf 1987). There is also a vertical system in the third visual ganglion, believed to mediate the pitch optomotor response according to an analogous model (Hengstenberg 1982). The visual pattern used in our experiments was a dark circular disk moving on a bright background. The predictions of the integrated correlation model were compared with the responses of the H1-neuron (a cell

A. Borst, M. Egelhaaf, and H. S. S u n g

858

A

8

- input lines - temporal filters

- nonlinear interactions

4-

-

subtraction

Figure 1: Outline of the motion detector model used to derive the predictions shown in Figures 2 and 3. (A) Single correlation-typemotion detector consisting of two mirror-symmetrical subunits with opposite preferred directions. Each subunit has two input lines. The signal in one line is temporally filtered and then multiplied with the unfiltered signal of the other line. The outputs of the subunits are subtracted from each other. (B)Responses of a two-dimensional array of orthogonally oriented pairs of motion detectors to a disk moving in the y direction. Shown is the vector field of equation 2.3, which was calculated using the continuum approximation of Reichardt (1987). The components of each vector are the responses of the x- and y-detectors at that point. The response is only nonzero on the boundary of the disk.


859

integrating horizontally oriented motion detectors) and the optomotor response about the vertical axis (yaw torque). 2 Responses of Correlation-'Ilpe Motion Detectors

Our model consists of a twodimensional square lattice of correlation-

type motion detectors.' At each point of the lattice is a pair of detectors that is oriented along the x and y axes. The luminance of the stimulus is denoted by I(r,t), where r is a vector in the xy plane and t is time. We treat the responses of the detector pair at r as the components of a vector,

+

Z(r, t)Z(r ex,t ') = I(r, t)Z(r ey,t

[

+

+ At) - Z(r + ex,t)Z(rlt + At) + At) - I(r + ey,t)Z(r,t + At)

1

(2.1)

Here ex and ey are vectors in the x and y directions, corresponding to the spacing between adjacent lattice points. The two terms of each component of d correspond to the opponent subunits shown in Figure lA, each a spatiotemporal cross-correlation of luminances. In equation 2.1 the temporal filtering is written as a simple time delay At, but in our computer simulations was more realistically modeled as a low-pass filter. The response of the H1-neuron is modeled as the x component of the integrated response vector, D(t) = /drd(r, t)

(2.2)

The yaw torque of the fly is modeled as a low-pass filtered version of the neural response. We chose a time constant of 3 sec, which is consistent with Egelhaaf (1987) and yields a good fit to the experimental data. Figure 1B shows that the local response d(r, t) to a upwardly moving circular disk is very different from its velocity field. The velocity field (not shown) is everywhere constant and points in the y direction. The local response, on the other hand, varies greatly: it is zero inside and outside the disk, and takes the form

d(6) oc isin6

(2.3)

on the boundary. Here 6 denotes the angle from the x axis, and i is the unit vector in the radial direction. This formula follows from equation 4.4 of the appendix, where the local response is calculated analytically using a continuum approximation (Reichardtand Guo 1986; Reichardt 1987; Reichardt and Schlogl1988). It is evident from Figure 1B that the x-detectors bias the direction away from the true velocity. At the upper right and lower left edges they signal positive (rightward) motion whereas at the opposite sides they signal negative (leftward) motion. Thus the circle in vertical motion mimics horizontal motion at its obliquely oriented edges. 'A more realistic triangular lattice model of the fly photoreceptor array (Buchner 1976; Buchner ef al. 1978) yields similar predictions.

860

A. Borst, M. Egelhaaf, and H. S.Seung

This is not surprising, since for this stimulus the brightness in both input channels of a horizontal motion detector is either increased or decreased during vertical motion in the same temporal sequence as during horizontal motion. The pattern dependence of the Zocal response is manifest, and has been studied previously (Reichardt 1987; Borst and Egelhaaf 1989). However, the flight control system of the fly is thought to depend on the integrated response of such an array of motion detectors (Egelhaaf et al. 1989). Although there is significant local x-response to vertical motion of a circle (Fig. lB), the integrated x-response is exactly zero because the contributions from the left and right halves of the circle cancel each other out. Hence, for a full circular stimulus the direction of the integrated response of an array of correlation-type motion detectors is the same as that of a true velocity sensor. To create a stimulus without such cancellation effects, the circIe was moved behind a square aperture in such a way that maximally only a semicircle was visible. Figures 2 and 3A exhibit the analytic results and numerical simulations for this semicircular stimulus. Tho features of the model predictions are noteworthy: (1) In contrast to a true velocity sensor, the integrated x-output responds not only to horizontal motion but also to vertical motion, with a time course that depends on the stimulus pattern. (2) The response to horizontal motion remains the same sign throughout the duration of the stimulus, the sign depending on the direction of motion. In contrast, the response to vertical motion changes sign when half of the semicircle is visible in the aperture, thereby erroneously mimicking an inversion of the direction of motion. 3 Responses of the Fly

These predictions were first tested by recording the spike activity of the H1-neuron in female blowflies (Calliphora erythrocephla) following standard procedures (Hausen 1982; Borst and Egelhaaf 1990). The resulting spike-frequency histograms of response to a moving semicircular stimulus are shown in Figure 3B. The preferred direction of the H1-neuron, when tested with Id grating patterns, is horizontal motion from back to front in the entire visual field of one eye (McCann and Dill 1969; Hansen 1976). Similarly, we found that the neuron was excited by horizontal motion of the semicircular stimulus in the preferred direction, and slightly inhibited by motion in the null direction. The magnitude of the null response is smaller than that of the preferred response, probably due to the low spontaneous activity of the cell and the resulting rectification nonlinearity. The response of the H1-neuron to vertical motion of a Id grating pattern is negligible (Hansen 1976). However, during vertical motion of the semicircular stimulus, the neuron’s response shows pronounced modulations and even a sign reversal relative to the resting level (hori-


861

3 2.5 2 1.5 1

Wma

0.5 0

-0.5 -1

b

Y

1

t

Figure 2 Spatially integrated response of correlation-type motion detectors to a moving circle seen through an aperture, calculated in equations A.8 and A.9 using the continuum approximation of Reichardt (1987). The x-response to motion in the y-direction (transverse response), takes the form Mxy(7)= cr [l - (1- 2 ~ ) ~and ] , the x-response to motion in the x-direction (longitudinal response) takes the form M,,(y) = crcos-'(l - 27). The prefactor cr has a quadratic dependence on the contrast of the stimulus, and 7 is the fraction of the semicircle that is visible in the aperture. These formulas are valid for the first half of the stimulus period, when 7 is increasing from 0 to 1. The formulas for the second half of the period are similar. The stimulus trace indicates the visible part of the circle at various instances of time. zontal line in Figure 3B). The responses to upward and downward motion are modulated in the same way but have opposite signs. Except for the fact that the neuron's responses to vertical motion are almost as strong as its responses to horizontal motion, all response features are in good agreement with the predictions of the correlation-detector model (compare Fig. 3B with Figs. 3A and 2). Hence, the response of the H1-neuron

A. Borst, M. Egelhaaf, and H. S. Seung

862

100

i

m

521

ID

P 0

2

oucluoucluo

Figure 3: (A) Spatially integrated responses of a square lattice of correlationtype motion detectors to a black circle moving in various directions on a bright background behind a square aperture. A 20 x 20 array of motion detectors of the correlation type (Fig. 1) was simulated on an IBM PS/2 using the MYST language (Keithley Instruments). The motion detectors had a sampling base of one lattice constant, a first-order low-pass filter as delay line, and were horizontally oriented with preferred direction from left to right. Note that the responses to horizontal and vertical motion are scaled differently. (B) Responses of an identified, directionally selective, motion-sensitive,large-field neuron (H1-cell) of the blowfly Culliphoru eythrocephlu to the same stimulus. Stimuli were generated on a CRT (Tektronix 608) controlled by an IBM AT through an image synthesizer (Picasso, Innisfree Inc.) with 200 Hz frame rate. The luminance of the circle was 4 cd/m2, and that of the background was 24 cd/m2. The contrast (I,,, - I-)/(lmx I m h ) amounted to 71%. The square aperture had the same extent as the diameter of the circle. The stimulus was presented to only the left eye of the fly at a distance of 7 cm. The circle had a diameter of 70" as seen by the fly. The center of the aperture was at 35" horizontal position and 0" vertical position with respect to the fly. Shown are the mean spike frequency histograms (40ms binwidth) f the SEM of the recordings of the H1 responses of 10 flies. Each fly was tested between 50 and 100 times (71 times on average) for each stimulus condition. The cell had rightward motion as its preferred direction. The horizontal line marks the resting firing level. The stimulus trace indicates the visible part of the circle at various instants in time.

+

is not simply the horizontal component of pattern motion. Measuring from neurons in the vertical system would presumably produce analogous results. We can conclude that the large-field cells in the third visual


863

ganglion of the fly do not represent the x and y coordinates of the pattern motion vector.2 This finding, however, does not rule out the possibility that the x and y components of pattern motion are computed at some later processing stage in the motion pathway of the fly. Therefore, we recorded the fly's behavioral turning responses about its vertical axis.3 These measurements were done on female flies of the species Muscu domesticu suspended from a torque-meter (Gotz 1964)following standard procedures (Egelhaaf 1987).The signals of the computer simulations shown in Figure 3A were passed through a first-order low-pass filter with a time-constant of 3 sec to account for the experimentally established low-pass filter between the thud visual ganglion and the final motor output (Egelhaaf 1987). This leads to smoothing and phase shifting of the original signal (compare Fig. 4A with 3A). As was found for the H1-neuron, the behavioral responses induced by the semicircle moving either horizontally or vertically are almost perfectly mimicked by the computer simulations (Fig. 4). Again, pronounced responses are induced not only during horizontal motion but also during vertical motion. The latter responses show a quasisinusoidal modulation and, hence, the same sign reversal observed in the H1-response and in the simulations. 4 Conclusions

In principle it is possible to compute pattern velocity from the output of an array of correlation-type motion detectors (Reichardt et al. 1988; Reichardt and Schlogl1988), provided that the second spatial derivatives of the pattern are also available and nonzero. Nevertheless, we find no evidence of such a computation in the fly; the output of its local motion detectors appears to undergo no more than a simple spatial integration and temporal filtering. Consequently, depending on the structure of the stimulus pattern, the direction of the optomotor response is not generally the same as the direction of pattern velocity. Since the function of the optomotor response is believed to be course stabilization, it might seem a deficiency for the response to be in the "wrong" direction. How can an organism such as the fly that is able to perform fast and virtuosic visually guided flight maneuvers afford to confound different directions in such a dramatic way? *The response of the H1-neuron to the vertical motion of a full circle was also measured. Contrary to the predictions of our simple model, there was some small nonzero response. Refinements of the model can be introduced to account for such incomplete cancellation of response, such as unbalanced subtraction of the two subunits (Egelhaaf et al. 1989) and/or spatially inhomogeneous sensitivity (Hansen 1982). 3Unlike the H1 experiments, two copies of the stimulus were presented simultaneously, one to each eye of the fly. Since the optomotor response integrates signals from both eyes, the flicker response is thereby cancelled, leaving only the motion-selective response, which is of interest here. Duplication of the stimulus would have been irrelevant in the H1 experiments, since H1 receives almost exclusively monocular input.

A. Borst, M.Egelhaaf, and H. S. S u n g

864

Figure 4 (A)The integrated responses shown in Figure 3A of a two-dimensional array of correlation-typemotion detectors, but fed through a first-order low-pass filter with a 3 sec time constant. Note that responses to horizontal and vertical motion are differently scaled. (8) Averaged optomotor turning responses (* SEMI obtained from 10 flies of the species Musca dornestica each tested 20 times for each stimulus condition. Clockwiseturning tendencies are shown as positive signals, and counterclockwise as negative signals. The stimulus trace indicates the visible part of the circle at various instants in time. Stimulus conditions were the same as for the electrophysiological recordings (Figure 3B) except for the following: (1)Stimuli were presented on either side of the fly. (2) The square aperture had an extent of 60" as seen by the fly. (3) The aperture was centered at 45" horizontal position and 0" vertical position with respect to the fly.

A plausible answer is that under natural conditions, the fly does not confound directions as dramatically as it does with our artificial stimulus. For the great majority of ecologically relevant stimuli, it may be that the spatially integrated response is very close to the direction of motion. Recall that the symmetry of the full circle led to exact cancellation of the simulated transverse response. For a natural pattern, such exact cancellation is no doubt rare, but there may be an approximate cancellation due to statistical averaging over the complex shapes in the pattern. Appendix: Continuum Approximation Consider an image that consists of a circle of radius ro with luminance 1 surrounded by a background of luminance 0. This can be written in polar coordinates as

I(r, 0) = Q ( T ~- T)

(A.1)


865

where 0 ( x ) is the Heaviside step function. Because of the spatial lowpass properties of the fly eye, the effective input to the detector array is a smoothed form of (A.l), which we can write as W l O )

=f(d

(A.2)

The precise form off is not important in what follows. What is important is that the radius YO is much larger than the scale of the smoothing, so that f’(r) is negligible except for T x yo. This visual stimulus, moving at velocity v, is input to an array of orthogonally oriented detector pairs. The response of a pair is given by equation 2.1. Each detector has sampling base Ax and delay time At. In the continuum approximation to equation 2.1, the local response d(x,y) is related to the velocity vector v by an expression of the form (Reichardt 1987; Reichardt and Schlogl 1988)

For a circular stimulus, the response matrix m is

m(r,d) =

e

sin 8 cos 0 ) a(r) sin2 - sin’ e sin 6 cos e b(r) (sinecose - cos2e COS~

(sin 0 cos 6 +

)

(A.4)

where a(r) = f ’ ( r ) ’ - f ( r ) f ” ( r ) , and b(r) = f ( r ) f ’ ( r ) / r . The off-diagonal element mxy,the transverse response, is of special interest. It is the response of the xdetector to motion in the y-direction. The diagonal element mxx is the longitudinal response, i.e., the response of the x-detector to motion in the x-direction. Assuming that the detector array is a square lattice of spacing Ax, the integrated output is

where M is the integrated response over the portion of the circle that is visible. If the full circle is visible, the off-diagonal elements of the integrated response vanish, so that 1 0 M = f f ( o 1)

(A.6)

where

7ri drra(r) m

ff R5

(A.7)

866

A. Borst, M. Egelhaaf, and H. S. Seung

The integral of the b(r) term in A.4 has been neglected, since it is much smaller than the a ( r ) term. Because M is proportional to the identity matrix, the integrated response vector D is in the same direction as the stimulus velocity v. For the stimulus used in these experiments, a semicircle moving behind a square aperture (shown in Fig. 2), the integrated response matrix is

where y o( t is the fraction of the semicircle that is visible in the aperture. These formulas are valid for the first half of the stimulus period, when y is increasing from 0 to 1. The formulas for the second half of the period are derived similarly. The full response curves are shown in Figure 2. Acknowledgments We are grateful to K. G. Gotz, W. Reichardt, and J. M. Zanker for carefully reading the manuscript. We also thank the people from the summer 1990 Woods Hole course "Neural Systems and Behavior," where this work was started, for the supportive and stimulating atmosphere. We especially thank B. Mensh and C. Gilbert for assistance in the early stages of this investigation. References Adelson, E. H., and Movshon, J. A. 1982. Phenomenal coherence of moving visual patterns. Nuture (London) 300, 523-525. Borst, A,, and Egelhaaf, M. 1989. Principles of visual motion detection. Trends Neurosci. 12,297-306. Borst, A., and Egelhaaf, M. 1990. Direction selectivity of fly motion-sensitive neurons is computed in a two-stage process. Proc. Natl. Acud. Sci. U.S.A. 87, 9363-9367. Buchner, E. 1976. Elementary movement detectors in an insect visual system. Biol. Cybern. 24, 85-101. Buchner, E., Gotz, K.G., and Straub, C. 1978. Elementary detectors for vertical movement in the visual system of Drosophila. B i d . Cybern. 31,235-242. Egelhaaf, M.1987. Dynamic properties of two control systems underlying visually guided turning in house-flies. J. Comp. Physiol. A161, 777-783. Egelhaaf, M., and Borst, A. 1993. Motion computation and visual orientation in flies. Comp. Physiol. Biochem. (in press). Egelhaaf, M., Hausen, K., Reichardt, W., and Wehrhahn, C. 1988. Visual course control in flies relies on neuronal computation of object and background motion. Trends Neurosci. 11,351-358.


867

Egelhaaf, M., Borst, A., and Reichardt, W. 1989. Computational structure of a biological motion-detection system as revealed by local detector analysis in the fly’s nervous system. I. Opt. SOC. Am. A6,1070-1087. Fermi, G., and Reichardt, W. 1963. Optomotorische Reaktionen der Fliege Musca domestica. Abhiingigkeit der Reaktion von der Wellenlange, der Geschwindigkeit, dem Kontrast und der mittleren Leuchtdichtebewegter periodischer Muster. Kybernetik 2, 15-28. Ferrera, V. P., and Wilson, H. R. 1990. Perceived direction of moving twodimensional patterns. Vision Res. 30, 273-287. Gotz, K. G. 1964. Optomotorischeuntersuchungen des visuellen systems einiger augenmutanten der fruchtfliege Drosphila. Kybernetik 2, 7i-92. Gotz, K. G. 1972. Principles of optomotor reactions in insects. Bibl. Opthal. 82, 251-259. Hassenstein, B., and Reichardt, W. 1956. SystemtheoretischeAnalyse der Zeit-, Reihenfolgen-und Vorzeichenauswertungbei der Bewegungsperzeptiondes RiisselkiifersChlorophanus. Z. Natut.forsch. llb,513-524. Hausen, K. 1976. Functional characterization and anatomical identification of motion sensitive neurons in the lobula plate of the blowfly Calliphora erythrocephala. Z . Naturforsch 31c, 629-633. Hausen, K. 1982. Motion sensitive interneurons in the optomotor system of the fly. I. the horizontal cells: Structure and signals. Biol. Cybern. 45, 143-156. Hausen, K., and Egelhaaf, M. 1989. Neural mechanisms of visual course control in insects. In Facets ofvision, D. G. Stavenga and R. C. Hardie, eds., Chap. 18, pp. 391424. Springer-Verlag,Berlin. Hengstenberg, R. 1982. Common visual response properties of giant vertical cells in the lobula plate of the blowfly Calliphora. 1. Comp. Physiol. A149, 179-193. Hildreth, E. C., and Koch, C. 1987. The analysis of visual motion: From computational theorem to neuronal mechanisms. Annu. Rev. Neurosci. 10,477-533. Horn, B. K. P., and Schunk, B. G. 1981. Determining optical flow. Artif. Intell. 17,185-203. McCann, G. D., and Dill,J. C. 1969. Fundamental properties of intensity, form, and motion perception in the visual nervous system of Calliphora phaenicia and Musca domestica. 1.Gen. Physiol. 53, 385-413. Movshon, J. A., Adelson, E. H., Gizzi, M. S., and Newsome, W. T. 1986. The analysis of moving visual patterns. Exp. Brain Res. 11, 117-152. Reichardt, W. E. 1987. Evaluation of optical motion information by movement detectors. 1.Comp. Physiol. A161, 533-547. Reichardt, W., Egelhaaf, M., and Schlogl, R. W. 1988. Movement detectors provide sufficient information for local computation of 2-d velocity field. Natumissenschajten 75, 313-315. Reichardt, W. 1961. Autocorrelation, a principle for the evaluation of sensory information by the central nervous system. In Sensory Communication, W. A. Rosenblith, ed., pp. 303-317. MIT Press and J. Wiley, New York. Reichardt, W., and Guo, A.-K. 1986. Elementary pattern discrimination (behavioural experiments with the fly Musca domestica). Biol. Cybern. 53, 285306.

868

A. Borst, M. Egelhaaf, and H. S. S u n g

Reichardt, W. E., and Schlogl, R. W. 1988. A two-dimensional field theory for motion computation. Bid. Cybern. 60, 23-35. Srinivasan, M. V. 1990. Generalized gradient schemes for the measurement of two-dimensional image motion. Bid. Cybern. 63,421-431. Stoner, G. R., Albright, T. D., and Ramachandran, V. S. 1990. Transparency and coherence in human motion perception. Nature (London) 344, 153-155. Stone, L. S., Watson, A. B., and Mulligan, J. B. 1990. Effect of contrast on the perceived direction of a moving plaid. Vision Res. 30, 1049-1067. Ullman, S. 1983. The measurement of visual motion. Trends Neurosci. 6,177-179. Uras, S., Girosi, F., Verri, A., and Torre, V. 1988. A computational approach to motion perception. Bid. Cybern. 60, 79-87. Verri, A., and Poggio, T. 1989. Motion field and optical flow: Qualitative properties. IEEE Trans. PAMI 11, 490-498. Received 28 August 1992; accepted 10 March 1993.


Communicated by Bruce McNaughton

Neural Representation of Space Using Sinusoidal Arrays David S. Touretzky A. David Redish Hank S. Wan School of Computer Science, CarnegieMellon University, Pittsburgh, PA 15213 USA O'Keefe (1991)has proposed that spatial information in rats might be represented as phasors: phase and amplitude of a sine wave encoding angle and distance to a landmark. We describe computer simulations showing that operations on phasors can be efficiently realized by arrays of spiking neurons that recode the temporal dimension of the sine wave spatially. Some cells in motor and parietal cortex exhibit response properties compatible with this proposal. 1 Introduction

Any vector in polar coordinates v = (Y, 4) can be represented as a sine wave fit) = r cos(wt + 4), where r is amplitude, 4 is phase, and w is (constant) frequency. This is commonly known as a phasor. The advantage of phasor representation is that translation and rotation of a vector are both trivial operations. Translation is achieved by addition of sine waves, and rotation can be obtained by phase shifting or temporal delay. O'Keefe (1991) suggested that rats might use phasors to encode angle and distance to landmarks. In his proposal, hippocampal theta provides the reference signal for determining phase. This temporal approach to encoding a sine wave has some drawbacks. The 7-12 Hz theta rhythm may be too slow to support real-time spatial reasoning tasks requiring rapid manipulation of phasors. Furthermore, maintaining even a modest angular resolution of 10" relative to a roughly 10 Hz reference signal requires a temporal resolution of 3 msec. Although some specialized sensory systems are known to make much finer discriminations (e.g., acoustic imaging in bats and dolphins, or auditory localization in barn owls), we are reluctant to require this degree of temporal precision at the higher cognitive level associated with spatial reasoning. Instead, we suggest that phasor operations are more plausibly realized by recoding the temporal dimension of the sine wave spatially, using populations of spiking neurons. We propose an architecture called the sinusoidal array for manipulating vectors in phasor form, and report the results of computer simulations. Neural Computation 5,869-884 (1993) @ 1993 Massachusetts Institute of Technology

870

David S.Touretzky, A. David Redish, and Hank S. Wan

There is some experimental evidence that sinusoidal array representations may exist in rat parietal cortex and in rhesus motor or parietal cortex. We propose an experiment to test this hypothesis in rats. 2 Sinusoidal Arrays

To encode a phasor as a sinusoidal array, we replace the continuous temporal signal f i t ) by a distributed pattern of activity over an array of N elements, as in Figure 1. The value encoded by the ith array element is the amplitude of the sine wave sampled at point 27rilN. That is, the activity level of the ith array element encoding the vector (Y, 4) is given by f ( ~ 4, ,i) = Y cos(4 + 27ri/N), for 0 5 i < N. Note that for the special case of N = 4, the sinusoidal array encoding is exactly the Cartesian encoding ( x ,y, -x, -y), where x = Y cos 4 and y = Y sin 4. Each sinusoidal array element is a collection of neurons. Its activity level is encoded by the neurons' average firing rate, or equivalently, the average percentage of neurons firing at any instant. If the neuronal p o p ulation is sufficiently large, this representation can encode values with high precision even when individual neurons are noisy and have a limited number of discriminable firing rates. In order to be able to represent the negative half of the sine wave, neurons in a sinusoidal array fire at a rate F(Y,4, i) = k.f(r, 4, i) + b, where k is a gain parameter and b the baseline firing rate. In our simulations, the baseline firing rate is 40 spikes/sec. This gives the neuron a dynamic range of 0-80 Hz,which is compatible with cells in parietal cortex. A significant advantage of the sinusoidal array representation is that it allows coordinate transforms to be done nearly instantaneously. If the signal fit) were represented temporally, the simplest way to determine its

Figure 1: The phasor (r, 4) and its sinusoidal array representation.

Neural Representation of Space

871

phase would be to wait for the peak. But one might have to wait up to one full period, which would be 140 msec for a 7 Hz signal. Alternatively, the phase could be determined by estimating the slope f ' ( t ) and taking the arc cosine, but this solution seems less neurally plausible than the spatial encoding investigated here. A drawback of the sinusoidal array representation is that angular resolution is limited to 27rlN. But even modest values of N appear to give sufficient resolution for navigation tasks. In our simulations we chose N = 24, giving an angular resolution of f7.5". 3 Vector Operations with Sinusoidal Arrays

In order to successfully complete a landmark-based navigation task, an animal must perform some coordinate transformations to derive a goal location from observed landmark positions. These transformations include at least translation, probably negation, and perhaps also rotation. In a phasor-based coordinate system, translation of a vector is accomplished by adding the corresponding sine waves, e.g., f(t) = f ~ ( t )+ f Z ( t ) . In the sinusoidal array representation, translation is accomplished by element-wise linear addition of firing rates: F (i) = FI(i) F2(i)- b, for 0 5 i < N. The subtraction of one baseline value b normalizes the result; it can be accomplished by giving the summation neuron an inhibitory bias term equal to the baseline firing rate. Negation of a vector can be accomplished in a variety of ways. Given a maximum activity level M for an element, we can compute F ( i ) = M Fl(i) for 0 5 i < N by using inhibitory connections to units whose baseline firing rate is M. However, since negation of a vector is usually required only as part of a vector subtraction operation, the easiest solution may be to use the addition mechanism described in the previous paragraph, but with one of the vectors rotated by 180". This gives F ( i ) = Fl(i) Fz(i N/2 mod N) - b. If translation and negation were the only required operations, there would be no advantage to using phasors. All computations could be done in Cartesian coordinates, using any neural encoding that correctly maintained the independent values x and y. However, when rotation is introduced, x and y are no longer independent. And since rotation in a Cartesian system is a nonmonotonic function, it is not easily computed with neuron-like units. (Of course, rotation is linear in a polar coordinate system, but then translation becomes nonmonotonic.) A point f ~ ( t )in phasor form can be rotated by cr radians about the origin by simply computing f(t) = f,(t + a). We implement the equivalent operation in sinusoidal arrays by rotating the array, using shifter circuitry similar to that proposed in Anderson and Van Essen (1987) and Olshausen et al. (1992). The shifter takes two vectors as input: one is the signal to be shifted, while the other specifies the angle of rotation. (The

+

+

+

David S. Touretzky, A. David Redish, and Hank S. Wan

872

prrmulation

channels

Figure 2: Schematic diagram of the shifter circuit. The signal entering at right goes through a contrast enhancement stage and winner-take-allphase detector, which determines the amount by which the input sine wave (top left) should be shifted. Light-colored lines indicate lateral inhibition connections. Only a subset of the permutation channel connections is shown. amplitude of the latter sine wave is ignored.) The shifter itself has two components: a I-of-N phase detector and a set of N gated permutation channels, as shown in Figure 2. The I-of-N phase detector contains one neuron for each of the N sinusoidal array elements. These neurons integrate their inputs over a brief time interval; the one receiving the largest input reaches threshold and fires first. We think of these phase detector neurons as similar to fastspike inhibitory interneurons. They have small refractory times and two sorts of postsynaptic effect: a short timescale inhibition of other phase detector cells (whose recovery from inhibition initiates a new winnertakeall round), and a long timescale inhibition that acts as a gating signal for channel-inhibitory neurons in the second half of the shifter. The shifter’s N gated permutation channels each copy the activity of the N-element input array to the N-element output array, permuting the elements along the way. When the jth channel is active, it copies the activation of input element i to output element i-j mod N, for 0 5 i < N. The channels have associated with them tonically active channel-inhibitory neurons that keep them silent most of the time. These are the same type of inhibitory units as the phase detector neurons, except that their only inputs are inhibitory. When a neuron in the phase detector fires, it


a73

inhibits the corresponding channel-inhibitory neuron, thereby disinhibiting the channel and allowing the shifted sine wave to appear across the output array. Anderson and Van Essen (1987) describe a shifter using log, N levels where each level has two permutation channels, giving O(N) connections. In a refinement of this model, Olshausen et al. (1992) use four levels with varying numbers of nodes, and fan-ins of approximately 1000, mirroring the connectivity of cortical areas V1, V2, V4, and IT. Because our own N is so small (N = 24 in the simulations), we can use a single level with N channels and O ( V ) connections. Aside from the obvious advantage of simplicity of connection structure, this allows us to use a simple 1of-N representation for the amount by which to shift, rather than the more complex binary encoding required by Anderson and Van Essen, or the distributed encoding of Olshausen et al. O u r model is also simpler because it requires only shunting inhibition, whereas theirs requires multiplicative synapses. The shifter circuit is not central to our theory. As discussed in the next section, many rodent navigation tasks can be performed using just translation. However, in situations where the reference frame must be determined anew on each trial based on the orientation of a cue array, there does appear to be a need for mental rotation of some sort. The shifter offers a solution to the general problem of rotation of vectors. But for some navigation tasks, the animal could instead slew its internal compass.

4 Rodent Navigation

In a remarkable series of experiments, Collett, Cartwright, and Smith investigated landmark learning behavior in gerbils (Collett et al. 1986). We will describe two of their simpler experiments here. Figure 3 shows the result of training a gerbil to find a food reward at a constant distance (50 cm)and compass bearing from a cylindrical landmark. The landmark was moved to a different location at each trial. Once trained (requiring on the order of 150 trials), the gerbil proceeded directly to the goal location from any starting location, and spent most of its time searching in the goal location. To model this behavior, we assume that the gerbil has learned that a constant memory vector M describes the remembered angle and distance of the landmark from the goal. On each trial, the gerbil's perceptual apparatus produces a vector P that describes the location of the landmark relative to the animal's current position. Thus, the position of the goal relative to the animal can be computed by vector subtraction: G = P - M. Collett et al. (1986)show that the animal must be computing this location, rather than simply moving to make its view of the landmark match a


874

0

0

S

F

S

S S

s Training

s Testing

Figure 3: Learning to find food at a constant distance and bearing from a landmark, after Collett et al. (1986) and Leonard and McNaughton (1990). S marks sample starting locations; F is food location; solid circle is the landmark; concentric circles show distribution of search time when trying to locate food reward. The majority of search time is spent at the goal location. stored memory of the goal, by turning off the lights after it had begun moving toward the goal. The animal still proceeded directly to the goal. The calculation of the goal location relies on a critical assumption: that the memory vector M, the perceptual vector P, and the goal vector G share the same reference direction, which we call global north. In Collett et al. (1986) this commonality was attributed to "unspecified directional cues." Recently it has been shown that rodents possess a highly accurate internal compass, which allows them to judge head direction even in the absence of visual cues (Chen et al. 1990; Chen 1991; Taube et al. 1990a,b). The compass is not related to magnetic north; it is maintained in part by integrating angular accelerations over time (Mittelstaedt and Mittlestadt 1980; Etienne 1987; Markus et al. 1990). McNaughton (personal communication) has observed that rats taken from their home cages will maintain their compass as they are carried into another room for experiments, so they have a consistent reference frame available even if the experimental environment is poor in directional cues.' This is significant because an internal compass that provides a stable north across all learning trials allows many simple navigation tasks to be performed without resorting to mental rotation. In our simulation of the Collett et al. task, we assume that the per'However, the compass can be confused if the box used to transport the animal is spun through several revolutions.


875

.

S

S

L1

F OL 2

S

S S

~~

Training

Testing

Figure 4 Learning to find food at a constant distance and bearing relative to a rotating cue array, after Collett et al. (1986).

F' Figure 5: Bearing a to food reward is measured with respect to the line joining landmarks L1 and L2, not global north. ceptual and memory systems orient their respective vectors, P and M, using the same internal compass. The sinusoidal array then computes G by vector subtraction. Figure 4 shows a more demanding experiment in which the cue array is rotated as well as translated on each trial. The bearing of the food reward is constant with respect to the line joining landmarks L1 and L2,as shown in Figure 5, but not with respect to the more salient cue provided by the internal compass. Tasks of this sort, in which bearings must be measured with respect to the cue array, should be more difficult to learn (Collett et d ' s observations support this), and would seem to require mental rotation.

876


.

Figure 6 Computations involved in solving the task shown in Figure 4. PI,Pz: coordinates of landmarks in the perceptual reference frame; M1,Mz: remembered coordinates of landmarks as seen from the food location; R: rotational alignment factor; G: computed goal location in the perceptual reference frame. Here is one way to solve the task in Figure 4. The line joining the two landmarks defines a "local north consistent across trials for the reference frame in which the memory vectors MI and MZare expressed. The perceptual vectors PI and PZ are expressed relative to the animal's internal compass, which is not aligned with this reference frame. If local north did coincide with the internal compass on some particular trial, then MI -MZ would equal PI -Pz, and the goal vector G would be simply P2 - Mz. (It would also be equal to P1 - MI, but the closer landmark is likely to give a more accurate solution.) In general, though, we will have to bring the two reference frames into alignment before locating the goal. The vector from the second to the first landmark, MI - MZin the memory frame, should correspond to the vector P1 - PZ in the perceptual frame. The required rotational alignment factor is therefore Phase(M1 - Mz) - Phase(P1- P z ) . Let rot(v, w) denote the antirotation of vector o by the phase of w. In other words, let rot(v, w) have the same magnitude as v, but phase equal to Phase(v) - Phase(w). Then the rotational alignment factor we require is equal to the phase of R = rot(M1 - Mz,PI - Pz), and the goal location is given by G = Pz - rot(M2,R). Each of the two rotation operations can be computed by the shifter described earlier. Our computer simulation of this task involves three vector subtractions and two rotations, as shown in Figure 6. We are not suggesting that rodent brains are wired to perform this specific computation; it seems more likely that some general spatial reasoning mechanism is involved. But the mechanism's primitive operations are likely to include translation and rotation. Our simulation shows that a combination of five of these operations is sufficient for performing the task in Figure 4. This number


877

can be reduced to four if the vector MI - M2 is remembered rather than computed on the fly. Further simplifications are possible. Instead of aligning memory with perception to compute P2 - rot(M2,R), the animal could slew its internal compass by the phase of R, realigning its perception to match memory. Then it need only compute Pi -M2, where P2 is the new perceptual vector measured with the slewed compass. McNaughton (personal communication) reports that rats do in fact realign the preferred directions of their head direction cells when the visual world is rotated at a perceptible rate, provided that the environment is familiar to them. Slewing the compass keeps landmarks at their learned headings. In unfamiliar environments the animal does not respond to rotation this way; it maintains its compass using inertial cues, as it does in the dark.

5 Details of the Computer Simulations

Our computer simulations are based on abstract neuron models that are

simpler than compartmental models, but retain many important properties of real neurons, such as spiking behavior. The simulation uses two types of neurons: pyramidal cells and inhibitory interneurons. Sinusoidal arrays contain 24 elements with 100 pyramidal cells each. Our abstract pyramidal cell has a resting potential of 0, a threshold 0 = 1, and a typical fan-in of 20 (but as high as 240 in the shifter) with synaptic weights of 0.1. It sums its inputs over time, and when it reaches threshold, it fires a spike. Spiking is treated as an instantaneous event, i.e., it lasts for one clock tick, after which the cell zeros its net activation and enters a refractory state. For the experiments reported here, a clock tick, At, is 0.1 msec. The cell's refractory period is 1/80 sec, limiting the peak firing rate to 80 Hz.It is important that the clock rate be significantly faster than peak firing rate, so that inputs are not lost when a cell zeros its net activation. Only impulses arriving at the exact moment a cell spikes will be lost; during the refractory period the cell continues to integrate its inputs. Pyramidal cells make up the summation module used for addition and subtraction of phasors. Cells in the summation module receive excitatory inputs from two sinusoidal arrays, following the equation F ( i ) = F1(i) + F 2 ( i ) - b. A neuron in the ith array element will receive inputs from 10 randomly chosen neurons from the ith element of each input array. The bias term b = 40 Hz is implemented by decrementing the net activation by -b . 0 . At every clock tick, but the total activation of the cell is not permitted to go below 0. Pyramidal cells also make up the output array of the shifter module. These cells have a fan-in of 240 since they receive 10 inputs from each of N permutation channels. They do not require a bias term.

878


The second type of model neuron is a fast-spike inhibitory interneuron used in the shifter. Both the phase detector neurons and the permutation channel inhibitory neurons are of this type. It has a resting level of 0 and a threshold of I, like the pyramidal cell, but the refractory period is only 5 msec. The firing of a phase detector neuron has two distinct effects. First, it inhibits all the other phase detector neurons, essentially setting their net activation to zero. Second, it inhibits the corresponding channelinhibitory neuron, allowing the permutation channel to open. Lateral inhibition of phase detector cells should have a short time course, so that when a neuron loses the race to fire first it can reenter the competition in short order. But channel-inhibitory neurons should be inhibited for a relatively long time, because we do not want the channel to close again between successive firings of its controlling phase detector. In cortex, GABAAinhibition has a short time course, while GABAe inhibition has a long time course. It therefore does not seem unreasonable to posit different inhibitory effects arising from the same interneuron, The channel-inhibitory neurons, when not themselves inhibited, shut down the permutation channel. This could be accomplished in real neural systems in several ways. If we assume that the ith channel's bundle of connections from input cells to a cell in the shifter's output array are distributed throughout the output cell's dendritic tree, then shutting down the channel would require inhibitory axoaxonic synapses at many select sites. But if connections comprising the ith channel were localized to a specific region of the output cell's dendritic tree, the channel-inhibitory interneuron would require only a single synapse onto the base of this subtree. Because our simulation is not at the compartmental level, we do not distinguish between these possibilities in our model. We add noise to the model by perturbing each cell's activation level by a small random value at each clock tick. For 5% noise, we use perturbations in the range f0.025M- 0 * At, where M is the cell's maximum firing rate. Small amounts of noise actually improve the shifter's performance by preventing the output cells within an array element from synchronizing with each other due to inhibition from contrast enhancement, described below? Noise also prevents a phase detector cell from consistently winning the race to inhibit its neighbors just because the cells that synapse onto it happened to start out with a slightly higher initial activation level. A technical problem with the shifter suggests that we may want to add basket cells to our model. We found that for the shifter to work correctly, the phase detector must be producing a stable output, i.e., reporting a consistent phase. However, when the sine wave input to the phase detector is of small amplitude, the peak can be difficult to determine precisely, so the phase detector's output wanders among several *Synchronizationwould cause anomalous behavior in any phase detector that used this signal as input, unless the phase detector cells integrated their inputs over a much longer time period.


879

nearby values. This results in the opening of different permutation channels at different times, degrading the shifter’s output representation. To prevent this, we introduced a contrast enhancement layer with a form of ”center-surround” feedback inhibition to preprocess the phase detector’s input and make the peak easier to find. In real neural systems, this type of inhibitory feedback is thought to be provided by basket cells (McNaughton and Nadel 1990). The details of our model’s contrast enhancement mechanism are a bit ad hoc at present and are in need of refinement, but preliminary results show that it does result in correct and stable phase detector output. If the inhibitory feedback is set at a high level, the contrast enhancement process yields an array representation with only one active element, thereby anticipating the winner-takeall function of the phase detector. However, for a range of lower values, instead of winner-take-allbehavior contrast enhancement produces cells with triangular response functions. The firing rates of these cells peak at a certain preferred direction, fall off roughly linearly within 30-60 degrees of that direction, and are elsewhere flat and close to zero. As discussed in the next section, cells with this behavior have been found in postsubiculum by Taube et al. We have also run simulations varying the number of neurons in a sinusoidal array. There was no appreciable advantage to doubling the number to 200 neurons per element. There was a slight penalty for using only 50 neurons: it took longer for the shifter to settle down and produce a consistent output signal, because contrast enhancement had to be done more slowly to avoid errors. With 20 neurons per element the system was unstable. 6 Experimental Evidence for Sinusoidal Arrays

A necessary condition for sinusoidal arrays to exist in cortex is the presence of cells whose response pattern obeys the function F(r, 4) = b k r cos 4, where distance r and angle 4 are measured either egocentrically or allocentrically. Georgopoulos et al. have formulated a similar equation, d ( M ) = b k cos(8cM),to describe the behavior of neurons in rhesus parietal cortex (Kalaska et al. 1983) and motor cortex. These neurons have firing rates proportional to the cosine of the angle between a “preferred direction vector” C and an intended reaching vector M. Different cells have different directional preferences3and hence different firing rates for a given movement. Their collective activity forms a “neural population vector” that can express any angle of intended motion. Another important piece of evidence in support of the sinusoidal array hypothesis is the finding in rats of cells that encode head direction with

+

s

+

3The preferred direction C plays the role of the array position i in our formula for F(r, hi).


880

0

90

180 b a d Dlroctlon (dog)

270

360

Figure 7 Tuning curves for a cell in parietal area Oc2M when the animal is motionless or making a left or right turn. Modified from Chen (1991, p. 118).

respect to either a visual landmark or an inertial reference frame. These cells appear to be part of the animal's internal compass referred to earlier. Taube et al. (1990a)report head-direction sensitive cells in postsubiculum with sharp directional preferences that are independent of the animal's location in the environment. When a prominent landmark is shifted along the wall of a cylindrical chamber, the cells' directional tuning curves are shifted by a comparable amount, indicating that the animal is using visual cues to maintain the compass (Taube et al. 1990b). The cells Taube et al. describe have triangular tuning curves, not sinusoidal ones. But Chen et al. (19901, recording from parietal/retrosplenial association cortex, also found head-direction sensitive cells, and in some cases the response pattern was a cosine function. Figure 7 shows the tuning curve of one such cell described in Chen (1991). The crucial question for both the rat and primate data is whether cells with a sinusoidal response function are also sensitive to distance. Schwartz and Georgopoulos (1987) have found this to be the case in rhesus motor cortex. They first varied the angle of a constant-distance target in a reaching task, to determine the preferred direction for each cell. Subsequently, they varied the distance between the animal and the target when the target was located at the cell's preferred direction. They report a substantial number of direction-sensitive cells with weak but statistically significant linear response as a function of target distance. In the case of the rat parietal recordings, to measure sensitivity to distance the animal would have to be attending to some known location. One way to accomplish this would be to train the rat to perform a landmark-based navigation task as in Figure 3, and then look


881

for direction-sensitive parietal cells whose response varied linearly with distance to either the landmark or the goal. 7 Discussion

Hippocampal theta may play some role as a reference signal for navigation, but it is probably not related to compass direction. OKeefe and Recce (1992) report that the phase at which place cells fire relative to the theta rhythm varies through 360" as the animal enters, proceeds through, and exits the cell's place field. This has led Burgess, OKeefe, and Recce to propose a navigation model in which phase information is used to distinguish entering vs. exiting. In conjunction with head direction information and a separate layer of goal cells, the net firing field of subicular place cells at phase 0" is peaked ahead of the rat, allowing the animal to navigate by homing to a goal location (Burgess et al. 1993). The Burgess et al. model has a number of interesting properties, but it cannot deal with complex navigation tasks of the sort Collett et al. have studied, with cue arrays that change position and orientation from trial to trial. While the hippocampus is known to play an important role in spatial behavior, researchers such as Nadel (1991) claim that its role is spatial memory, not planning and navigation. Parietal cortex appears to be involved in these latter tasks (Stein 1991). McNaughton et a2. (1991)propose a model of directional sense based on both vestibular sensations and visual cues. In darkness or unfamiliar environments, the animal maintains its compass by inertial means, using an associative memory "table lookup" scheme to compute its new heading from the old heading plus angular acceleration. But in familiar environments, "local view" cells (possibly hippocampal place cells) adjust the compass to agree with the learned heading associated with that view direction. We mentioned previously that compass slewing might replace the second rotation when performing Collett et al.'s rotating cue array task. McNaughton (personal communication)suggested that if local view cells can determine compass direction by direct matching of visual landmarks, the first subtraction and rotation steps might also be eliminated, leaving just one vector subtraction. We agree with the notion that distant landmarks should control the animal's compass in familiar environments. But it seems less plausible that viewing a configuration of nearby landmarks would provide sufficiently accurate heading information to solve the rotating cue array task by table lookup, because the view could change significantly with relatively small translations. Hence we believe at least one mental rotation step is required. Elsewhere in their paper McNaughton et al. speculate that trajectory computations (vector subtractions) might be done by the same table lookup mechanism as they propose for updating the inertial compass.

882


The drawback of this proposal is the large table that would be required to represent all possible pairs of vectors, and the cost of filling in the entries. The sinusoidal array appears to offer a simpler solution for vector arithmetic. The neural architecture we have described is compatible with properties of parietal cortex. It manipulates phasors as sinusoidal arrays, but it does not explain how such representations arise in the first place. We simply assume that perceptual and memory vectors are available in the required form. We defend this assumption by noting that sinusoidal encodings of angles have already been found in rats and monkeys. Indications of a linear sensitivity to distance in rhesus sinusoidal cells reported by Schwartz and Georgopoulos offer additional support. At this point, the most important test of our model is whether rat parietal cells can be found with cosine response functions that are also linearly sensitive to distance. Two other properties of our model are worth noting. As presently formulated, all cells in a sinusoidal array element have the same preferred direction (as do cells in a single orientation column in visual cortex), so there are only N directions represented. If the preferred directions of real parietal cells are found to cluster into a small number of discrete, evenly spaced values, this would be strong evidence in favor of the sinusoidal array hypothesis. However, we expect our model would also function correctly using input units with preferred directions smoothly distributed around the circle, so that neurons in bin i had a preferred direction somewhere within 2.rr(if0.5)/N.We have not yet verified this experimentally, however. Due to the many-to-one connectivity of pyramidal cells, units in the output sinusoidal array should still show preferred direction values close to the centers of their respective bins. The model also assigns the same scale factor k to all neurons in an array. But experimenters report a wide range of peak firing rates for direction-sensitive cells in both postsubiculum and parietal cortex. We again expect the model to function correctly under this condition, assuming only that the mean scale factor is the same across elements.

Acknowledgments This work was supported by a grant from Fujitsu Corporation. Hank Wan and David Redish were supported by NSF Graduate Fellowships. We thank Bruce McNaughton and an anonymous referee for helpful comments on an earlier draft of this paper, and Longtang Chen for permission to reproduce one of his figures.


883

References Anderson, C. H., and Van Essen, D. C. 1987. Shifter circuits: A computational strategy for dynamic aspects of visual processing. Proc. Natl. Acud. Sci. U.S.A. 84,1148-1167. Burgess, N., OKeefe, J., and Recce, M. 1993. Using hippocampal ‘place cells’ for navigation, exploiting phase coding. In Advances in Neural Information Processing Systems 5, S . Hanson, J. Cowan, and L. Giles, eds., pp. 929-936. Morgan Kaufmann, San Mateo, CA. Chen, L. L. 1991. Head-directional information processing in the rat posterior cortical areas. Doctoral dissertation, University of Colorado. Chen, L. L., McNaughton, B. L., Barnes, C. A., and Ortiz, E. R. 1990. Headdirectional and behavioral correlates of posterior cingulate and medial prestriate cortex nuerons in freely-moving rats. SOC.Neurosci. Abstr. 16,441. Collett, T. S., Cartwright, B. A., and Smith, 8. A. 1986. Landmark learning and visuospatial memories in gerbils. J. Comp. Physiol. A 158, 835-851. Etienne, A. S. 1987. The control of short-distance homing in the golden hamster. In Cognitive Processes and Spatial Orientation in Animals and Man, I? Ellen & C. Thinus-Blanc, eds.,pp. 233-251. Martinus Nijhoff, Dordrecht. Georgopoulos, A. P., Schwartz, A. B., and Kettner, R. E. 1986. Neuronal population coding of movement direction. Science 233,1416-1419. Kalaska, J. F., Caminiti, R., and Georgopoulos, A. P.1983. Cortical mechanisms related to the direction of two-dimensional arm movements: Relations in parietal area 5 and comparison with motor cortex. Exp. Brain Res. 51, 247260. Leonard, B., and McNaughton, B. L. 1990. Spatial representation in the rat: Conceptual, behavioral, and neurophysiological perspectives. In Neurobiology of Comparative Cognition, R. P. Kesner and D. S. Olton, eds., pp. 363-422. Erlbaum, Hillsdale, NJ. Markus, E. J., McNaughton, B. L., Barnes, C. A., Green, J. C., and Meltzer, J. 1990. Head direction cells in the dorsal presubiculum integrate both visual and angular velocity information. SOC.Neurosci. Abstr. 16,441. McNaughton, B. L., and Nadel, L. 1990. Hebb-Marr networks and the neurobiological representation of action in space. In Neuroscience and Connectionist Theory, M. A. Gluck and D. E. Rumelhart, eds., pp. 1-63. Erlbaum, Hillsdale,

NJ.

McNaughton, B. L., Chen, L. L., and Markus, E. J. 1991. “Dead reckoning,” landmark learning, and the sense of direction: A neurophysiological and computational hypothesis. J. Cog. Neurosci. 3(2), 190-202. Mitttelstaedt, M.-L., and Mittelstaedt, H. 1980. Homing by path integration in a mammal. Natunvissenschaften 67, 566-567. Nadel, L. 1991. The Hippocampus and space revisited. Hippocumpus 1(3), 221229. OKeefe, J. 1991. An allocentric spatial model for the hippocampal cognitive map. Hippocampus 1(3), 230-235. OKeefe, J., and Recce, M. 1993. Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocumpus 3, (in press).

884


Olshausen, B., Anderson, C., and Van Essen, D. 1992. A neural model of visual attention and pattern recognition. CNS Memo 18, Computation and Neural Systems Program, California Institute of Technology. Schwartz, A. B.,and Georgopoulos, A. I? 1987. Relations between the amplitude of 2-dimensional arm movements and single cell discharge in primate motor cortex. SOC.Neurosci. Abstr. 13, 244. Stein, J. 1991. Space and the parietal association areas. In Brain and Space, J. Paillard, ed., pp. 185-222. Oxford University Press, New York. Taube, J. S., Muller, R. I., and Ranck, J. 8.J. 1990a. Head direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis. 1. Neurosci. 10, 420-435. Taube, J. S., Muller, R. I., and Ranck, J. B. J. 199Ob. Head direction cells recorded from the postsubiculum in freely moving rats. 11. Effects of environmental manipulations. 1.Neumsci. 10,436-447. Received 20 July 1992; accepted 4 March 1993.

This article has been cited by: 2. John L. Kubie, André A. Fenton. 2009. Heading-vector navigation based on head-direction cells and path integration. Hippocampus 19:5, 456-479. [CrossRef] 3. Pierre Baraduc , Emmanuel Guigon . 2002. Population Computation of Vectorial TransformationsPopulation Computation of Vectorial Transformations. Neural Computation 14:4, 845-871. [Abstract] [PDF] [PDF Plus] 4. Darlene M. Skinner, Gerard M. Martin, Christa-Jo Scanlon, Christina M. Thorpe, Jeremy Barry, John H. Evans, Carolyn W. Harley. 2001. A two-platform task reveals a deficit in the ability of rats to return to the start location in the water maze. Behavioral Neuroscience 115:1, 220-228. [CrossRef] 5. Alexandre Pouget, Terrence J. Sejnowski. 2001. Simulating a lesion in a basis function model of spatial representations: Comparison with hemineglect. Psychological Review 108:3, 653-673. [CrossRef] 6. M. Matsuoka, S. Hosogi, Y. Maeda. 1998. Hippocampal neural network model performing navigation by homing vector field adhesion to sensor map. Artificial Life and Robotics 2:3, 129-133. [CrossRef] 7. Sabine Gillner , Hanspeter A. Mallot . 1998. Navigation and Acquisition of Spatial Knowledge in a Virtual MazeNavigation and Acquisition of Spatial Knowledge in a Virtual Maze. Journal of Cognitive Neuroscience 10:4, 445-463. [Abstract] [PDF] [PDF Plus] 8. Asohan Amarasingham , William B. Levy . 1998. Predicting the Distribution of Synaptic Strengths and Cell Firing Correlations in a Self-Organizing, Sequence Prediction ModelPredicting the Distribution of Synaptic Strengths and Cell Firing Correlations in a Self-Organizing, Sequence Prediction Model. Neural Computation 10:1, 25-57. [Abstract] [PDF] [PDF Plus] 9. A. David Redish, David S. Touretzky. 1997. Cognitive maps beyond the hippocampus. Hippocampus 7:1, 15-35. [CrossRef] 10. Sofyan H. Alyan. 1996. Evidence for Resetting the Directional Component of Path Integration in the House Mouse (Mus musculus). Ethology 102:4, 629-638. [CrossRef] 11. David S. Touretzky, A. David Redish. 1996. Theory of rodent navigation based on interacting representations of space. Hippocampus 6:3, 247-270. [CrossRef] 12. Georg Hartmann, Rüdiger Wehner. 1995. The ant's path integration system: a neural architecture. Biological Cybernetics 73:6, 483-497. [CrossRef] 13. Thomas Wittmann, Helmut Schwegler. 1995. Path integration — a network model. Biological Cybernetics 73:6, 569-575. [CrossRef] 14. A. David Redish, David S. Touretzky. 1994. The reaching task: evidence for vector arithmetic in the motor system?. Biological Cybernetics 71:4, 307-317. [CrossRef] 15. L. F. Abbott. 1994. Decoding neuronal firing and modelling neural networks. Quarterly Reviews of Biophysics 27:03, 291. [CrossRef]

16. Alexandre Pouget, Lawrence H SnyderModeling Coordinate Transformations . [CrossRef]

Communicated by Lawrence Jackel

Fast Recognition of Noisy Digits Jeffrey N. Kidder Daniel Seligson Intel Architecture Labs, Intel Corporation, Mailstop RN6-25, 2200 Mission College Blvd., Santa Clara, C A 95052 USA

We describe a hardware solution to a high-speed optical character recognition (OCR) problem. Noisy 15 x 10 binary images of machine written digits were processed and applied as input to Intel's Electrically Trainable Analog Neural Network (ETA"). In software simulation, we trained an 80 x 54 x 10 feedforward network using a modified version of backprop. We then downloaded the synaptic weights of the trained network to ETANN and tweaked them to account for differences between the simulation and the chip itself. The best recognition error rate was 0.9% in hardware with a 3.7% rejection rate on a 1000character test set. 1 Introduction We have solved a difficult optical character recognition (OCR)problem using a feedforward neural network configured as a "1 of 10" classifier. To meet the challenging throughput requirements of this application, we have deployed the solution on Intel's Electrically Trainable Analog Neural Network (ETA") chip (Holler et al. 1989). In the OCR problem, we receive a 15 x 10 map from the output of a binary image acquisition system. The characters are the digits 0,1,. ..9, and the digit presentation rate is 12,500 characters per second, requiring a classification time of less than 80 psec. Each digit is machine written with a laser on plastic, but the combined process of writing, processing of the plastic, and image acquisition is very noisy (see Fig. 1). Furthermore, the digits vary somewhat in size from 8 x 5 to 10 x 9 and they drift around in the larger 15 x 10 field.' ETANN is an analog neural network with flexible architecture. For our application, we configured it as an M input (M 5 128), N output, 64 - N hidden node feedforward network with binary inputs and out'The details of the application, e.g., why it was developed and where it is being deployed, are proprietary information. The problems we encountered and the solutions we found while developing it are, we believe, generic and should be disseminated.

Neural Computation

5,885-892 (1993) @ 1993 Massachusetts Institute of Technology

886

JeffreyN. Kidder and Daniel Seligson

Figure 1: Examples of digits before any preprocessing was performed.

puts, but analog hidden units. In this mode, its feedforward execution time is about 6 psec, thereby outperforming the 80 psec classification requirement. As a "1 of 1 0 binary classifier (i.e., N = 10) with M = 80, its forward computation rate is 0.8 x lo9 connections per second, exceeding the capabilities of conventional microprocessors or digital signal processors? Because the binary image as received has 150 pixels and ETA" has only 128 inputs, some compressive preprocessing was required. The scheme we developed required approximately 200 integer additions and 14 comparisonsper pattern. This amounts to a few million operations per second given the pattern presentation rate requirements and is easily achievable with commercial microprocessors. In the sys'ETANN's peak computation rate, achieved with a 128x 64 network, having a 3 psec execution time is 2.7 x lo9.

Fast Recognition of Noisy Digits

887

tem prototype deployed thus far, the actual throughput achieved is 22 characters per second, as compared to the 167,000 character per second theoretical throughput of ETA“. The difference is due to bandwidth limitations of the development system. Initially we simulated E T A ” in software and used a modified version of backprop (Rumelhart et al. 1986) to train it. The weights were then downloaded to an E T A ” development system. A few iterations of chip-in-loop learning modified the weights to adjust for approximations in the simulation. We describe a preprocessing scheme, network architecture, and training algorithm. From a set of 5000 images, we selected 4000 at random for training, leaving the remaining 1000 images for test. The best performance we achieved was a test set error rate of 0.6% in software and 0.9% in hardware at rejection rates of 3.6 and 3.7%, respectively.

2 Preprocessing

The primary task of the preprocessing stage was to reduce the number of inputs from 150 to a maximum of 128. Since the digits were small and their centers moved around in the 15 x 10 image field, the recognition system needed some sort of built-in translation independence. We tested three preprocessing schemes: blocking, balancing, and linear compression. Blocking and balancing attempt to effect translation invariance in the preprocessing itself. With linear compression, the translation invariance must be trained into or built into the network. These are summarized in Figure 2. The blocking algorithm computes the number of ”on” pixels in each of the character-sized (10 x 8) windows in the larger (15x 10)image field. The contents of the window containing the most “on” pixels were applied as input to the network. The “empty” rows and columns were ignored. However, noise confounds the algorithm and some of the character itself was stripped away occasionally. Expanded blocking takes the subwindow from the blocking algorithm and includes the adjoining pixels on its periphery. In another approach, called balancing, we find the centroid of the image field and use it as the center of a new window. By itself balancing does not reduce the total number of inputs to the network. We used two different methods to accomplish this. In the first we stripped the peripheral rows and columns from the balanced character, producing a 13 x 8 character. In the second we applied a linear compression. To visualize a linear compression, superimpose an rn x n grid on the original 15 x 10 character. Then, for each cell in the new grid assign the percentage of “black space” filling it.

888

Jeffrey N. Kidder and Daniel Seligson

Figure 2 Preprocessing of a 3. 3 Neural Network Architecture and Training

Using the SN2TMconnectionist simulator (Bottou and Le Cun 19881, we wrote routines to model the ETA" in the desired mode. ETANN deviates from the standard neural network model in two ways. First, its neuron output sigmoid is not the usual hyperbolic tangent (Holler et al. 1989). Second, the synaptic multiply (UiWij) is only a true multiply for small values of ui and wi,, saturating outside this domain. Our model includes a cubic spline interpolation of the chip's sigmoid, but does not address the saturating multiply. The network is designed as a "1 of 10" classifier, meaning that it has 10 output units and that it is intended that only one should fire.During training and testing, the outputs are real-valued on the interval -1 to 1. In deployment, the outputs would be constrained to the binary limits, -1 and 1. In training and testing, the network is said to have classified


889

the input if the maximum output unit is within some tolerance, E, of +I .O and if all other output units are within E of -1.0. For all other cases (i.e., more than one unit not within E of -1.0, or no unit within E of +1.0), the network is said to have rejected the input. When deployed with binary outputs, the tolerance is effectively E = 1.0. The training procedure was a modification (Seligson et al. 1992) of the backprop algorithm employing both pattern and epoch learning. In one epoch, each training pattern is fed forward through the net and the output is compared with the desired output. If the character is incorrectly classified or the output is rejected, then the error vector is propagated back through the net and the weights are updated; momentum was used and set equal to 0.5. If the character is correctly classified, no change is made. This is different from “plain vanilla” backprop in which an error vector is backpropagated for every pattern. At the end of the epoch, if the percentage of correctly classified patterns is sufficiently high (98%in most of our work) and the error rate sufficiently low (0.1-1.0%) then the network is said to have converged at the current tolerance. If it has not converged, then another epoch is initiated. If it has converged, we lower the tolerance and the learning rate, and then begin another epoch. This procedure is iterated until some other convergence criterion is met. These modifications have two principal advantages. First, by ignoring training vectors which pass the tolerance criterion, the network is encouraged to find weights such that all training vectors are equally bad (or good). Second, by linking the learning rate and the tolerance criterion, we can initialize the training procedure with large values of each (0.1 and 1.0 for this problem) and still achieve good convergence. In practice we have seen that this procedure results in faster and tighter convergence than would be achieved by vanilla backprop using any one fixed value of the learning rate. Gaussian input noise was added to each pixel of the input pattern before a forward pass. We varied the standard deviation of the noise between 0.01 and 0.2. We found the final error rate to be more sensitive to the choice of preprocessing scheme than to small variations in initialization parameters, momentum, or the number of hidden nodes. 4 Results

The best results were obtained by blocking to 10 x 8. We achieved a 0.6% error rate on the 1000 digit test set with a rejection rate of 3.6%. Figure 1 illustrates that this performance is close to the limit of human perception. Table 1 summarizes the performance of six other networks, including a perceptron (i.e., a template matcher trained with backprop). Having selected the 10 x 8 input format, we were ready to map the 80 input, 54 hidden node, 10 output unit network onto the chip. Figure 3 shows, schematically, the 128 x 64 array of weights in the ETA” chip.


890

Figure 3 Mapping the feedforward network into ETANN. Table 1: Recognition Results.

Input 10 x 8 blocked 10 x 8 blocked 10 x 8 blocked 10 x 8 blocked 12 x 10 blocked with periphery 13 x 8 balanced and trimmed 9 x 7 balanced and compressed

Hidden

Emr %

Reject %

Comments

54

0.9 0.6 3.6 4.0 1.1 1.9 2.7

3.7 3.6 7.6 12.4 2.6 8.9 9.9

On ETA"

54 10 0 54

54 32

Perceptmn


891

The first layer of weights are loaded as an 80 x 54 block. The outputs (summing down the columns) of this layer are fed into a 54 x 10 block of weights? The output of the second layer is the output of the chip. After computing the weights with SN2TM,we transferred them to iBrainMakerTM2.0 and Intel's Neural Network Training Systemm for implementation and testing in hardware. Downloading weights to E T A " is a process of applying high voltage pulses to program analog floating gate memory devices. We then trained the chip itself for a few epochs, to account for chipto-chip variations, imperfections in the downloading process, and limitations in our software model. The three most significant limitations were (1) that the device synapses have a dynamic range and precision of about 1 in 100, whereas the simulator used floating point arithmetic with effectively unlimited dynamic range and precision, (2) that we used a perfect multiply rather than the chip's saturating multiply, and (3) that we did not account for fabrication related synapse-tosynapse or neuron-to-neuron variations present in E T A " (Holler et al. 1989). On E T A " itself, we achieved an error rate of 0.9% on the test set with a tolerance of 1.0 and a 3.7% rejection rate, substantially the same as in software. Thus we can conclude that ETANN's finite dynamic range and precision did not restrict its ability to perform this classificationtask, and that the learning algorithm as applied to the chip is able to overcome the simplifications of the model.

5 Discussion

We mention briefly some obstacles which hindered us from achieving lower error rates, and some options which were not explored. Firstly, the character font was ill-suited to the task at hand, primarily because the character set was not sufficiently orthogonalized, especially among the 0, 2, 3, 5, 6, 8, and 9. Noise produces ambiguities and misclassifications, both for our network and for human subjects viewing the same data (see Fig. 1). Second, there was not enough training data available. A model of the image generation and acquisition system (in particular the noise) would have allowed us to synthesize a large data set for training, testing, and evaluation. Finally, there were altogether different architectures that we considered but did not investigate. In our chosen method, we preprocessed the 15 x 10 data by scanning a smaller window over the input pattern and then we presented the best window as input to the net. Since we had 80 psec for the task, and the net executes in 6 psec, another alternative would have been to scan the net over the input pattern, postprocessing the outputs to determine the most probable input digit. 3Thresholdsare &plied, not shown.

892


Acknowledgments We would like to thank Steven Anderson, Finn Martin, and Simon Tam for assistance with ETA", and Maria Douglas for assistance with the manuscript. SN2 is a trademark of Neurestique, iBrainMaker is a trademark of California Scientific Software, and i " T S is a trademark of Intel Corporation.

References Bottou, L.-Y., and Le Cun, Y. 1988. SN: A simulator for connectionist models. In NeuroNirnes 88, Nimes, France. Holler, M., Tam, S., Castro, H., and Benson, R. 1989. An electrically trainable artificial neural network (ETA") with 10240 floating gate synapses. Proc. IJCNN-89-WASH-DC., pp. 191-196. Summer 1989. Rumelhart, D. E., McClelland,J. L., (eds.). 1986. In Parallel DisfributedProcessing: Explorations in theMicrostructure of Cognition. Vol. 1: Foundations. MIT Press, Cambridge, MA. Seligson, D., Hansel, D., Griniasty, M., and Shoresh, N. 1992. Computing with a difference neuron. Network 3,187-204. Received 18 February 1992; accepted 18 March 1993.



Local Algorithms for Pattern Recognition and Dependencies Estimation V. Vapnik L. Bottou AT&T Bell Laboratories, Holmdel, NJ07733 USA

In previous publications (Bottou and Vapnik 1992; Vapnik 1992) we described local learning algorithms, which result in performance improvements for real problems. We present here the theoretical framework on which these algorithms are based. First, we present a new statement of certain learning problems, namely the local risk minimization. We review the basic results of the uniform convergence theory of learning, and extend these results to local risk minimization. We also extend the structural risk minimization principle for both pattern recognition problems and regression problems. This extended induction principle is the basis for a new class of algorithms. 1 Introduction The concept of learning is wide enough to encompass several mathematical statements. The notions of risk minimization and of loss function (cf. Vapnik 19821, for instance, have unified several problems, such as pattern recognition, regression, and density estimation. The classical analysis of learning deals with the modelization of a hypothetical truth, given a set of independent examples. In the classical statement of the pattern recognition problem, for instance, we select a classification function given a training set. This selection aims at keeping small the number of misclassifications observed when the selected function is applied to new patterns extracted from the same distribution as the training patterns. In this statement, the underlying distribution of examples plays the role of a hypothetical truth, and the selected function models a large part of this truth, i.e., the dependence of the class on the input data. We introduce in this paper a different statement of the learning problem. In this local statement, the selection of the classification function aims at reducing the probability of misclassification for a given test pattern. This process, of course, can be repeated for any particular test pattern. This is quite a different task instead of estimating a function, we estimate the value of a function on a given point (or in the vicinity of a given point). Neural Computation 5,893-909 (1993) @ 1993 Massachusetts Institute of Technology

V. Vapnik and L. Bottou

894

The difference between these two statements can be illustrated by a practical example. A multilayer network illustrates the classical approach The training procedure builds a model using a training set. This model then is used for all testing patterns. On the other hand, the nearestneighbor method is the simplest local algorithm: Given a testing point, we estimate its class by searching the closest pattern in the training set. This process must be repeated for each particular test pattern. The statement of the problem defines the goal of the learning procedure. This goal is evaluated a posteriori by the performance of the system on some test data. The training data, however, do not even provide enough information to define this goal unambiguously. We must then rely on an induction principle, i.e., a heuristic method for "guessing" a general truth on the basis of a limited number of examples. Any learning algorithm assumes explicitly or implicitly some induction principle, which determines the elementary properties of the algorithm. The simplest induction principle, namely the principle of empirical risk minimization (ERM), is also the most commonly used. According to this principle, we should choose the function that minimizes the number of errors on the training set. The theory of empirical risk minimization was developed (Vapnik 1982) in the 1970s. In the case of pattern recognition, this theory provides a bound on the probability of errors, p, when the classification function is chosen among a set of functions of finite VC dimension. In the simplest case, with probability 1- q the following inequality is true (Vapnik 1982, p. 156, Theorem 6.7).

plv+D

(1.1)

where v is the frequency of error on the training set, and

is a confidence interval, which depends on the number of training examples I, on the VC-dimension h of the set of functions, and on the confidence q. When the VC-dimension of the set of functions increases, the frequency of error on the training set decreases, but the width D of the confidence interval increases. This behavior leads to a new induction principle, namely the principle of structural risk minimiza tion (SRM) (Vapnik 1982). Consider a collection of subsets imbedded in the set of functions,

s, c s2 c ... c S" where s k is a subset of functions with VC-dimension hk, and hk < h k + l . For each subset sk, a function fk minimizes the frequency of error on the training set, and thus fulfills the inequality 1.1. Successive functions

Local Algorithms for Pattern Recognition

895

yield a decreasing number of errors v on the training set, but have increasingly wider confidence interval D. The principle of structural risk minimization consists in selecting the subset Sk' and the function fk', which minimizes the right-hand side of inequality 1.1, named the guaranteed risk. The SRM principle requires the choice of a nested structure on the set of functions. An adequate structure can significantly improve the generalization performance; a poor structure may have a limited negative impact. In the local statement of the learning problem, we aim at selecting a valid function in the vicinity of a given test point xo. On the basis of the training set, we will select a "width" for this vicinity, as well as a function for classification vectors in this vicinity. To solve this problem, an extended SRM principle will be considered. We will minimize the guaranteed risk not only by selecting a subset Sk' and a function fk* E Sk', but also by selecting the width /3 of the vicinity of the point XO. Using /3 as an additional parameter allows us to find a deeper minimum of the guaranteed risk as demonstrated on a practical application in Bottou and Vapnik (1992). The paper is organized as follows. First, we state and discuss the problem of risk minimization and the problem of local risk minimization. In Section 3, we derive a few useful bounds for uniform convergence of averages to their expectations. In Sections 4 and 5 we derive bounds of the "local risk for the problem for pattern recognition and the problem of regression. In Section 6, we extend the structural risk minimization principle to local algorithms.

fk

2 Global

and Local Risk Minimization

Like many illustrous scientists, we will assume, in this paper, that a metaphysical truth rules both OUT training examples and OUT testing cases. Like many illustrous statisticians, we will also assume that this truth can be represented, for our purposes, by an unknown probability distribution F ( x , y ) , defined on a space of input-output pairs ( x , y ) E R" x R'. In the classical statement of global risk minimization, a parameter a E A defines a model x f ( x , a) of the output y . A loss function, Q[y,f(x, a)], measurable with respect to F ( x , y ) , quantifies the quality of the estimate f ( x , a) for the outputs y . We wish then to minimize the global risk functional

-

over all functions c f ( x , a),a E A}, when the distribution F(x,y ) is un-


896

known, but when a random independent sample of size l XI1 y1;.

. . ; X I , yl

(2.2)

is given. Let us introduce the statement of local risk minimization, in the vicinity of a given point XO. In this statement, we aim at modeling the truth in a small neighborhood around X O . A nonnegative function K ( x , X O , P) embodies the notion of vicinity. This function depends on the point XO, on a "locality" parameter P E (0,oo),and satisfies: i. 0 5 K(xlXO, P) I 1,

ii. K(xo,xolP) = 1. For example, both the "hard threshold locality function (2.3) and the "normal" locality function (2.4) meet these conditions (2.3) (2.4)

Let us define the norm of locality function as IIK(xolP)II = S K ( x > x 0, P ) W l Y )

Let us consider again a parametric functionf ( x , a),and a measurable loss function, Qh,f(x1a)].We want to minimize the local risk functional (2.5)

over the parameters a and P, when the distribution function F(xly) is unknown, but when a random independent sample XI, y~, . .. ,XI, y~is given. In most cases, the knowledge of the distribution F(x,y) would make this problem trivial. For example, if the locality function is either the hard threshold locality function (2.31, or the normal locality function (2.41, we would select /3 = 0, and adjust a to get the "right" value for f ( x 0 , a). The true distribution, F(x,y), however, is unknown. Selecting a nontrivial value for the locality parameter might reduce the generalization error induced by the unavoidable inaccuracy of parameter a. A new induction principle has been developed to take advantage of this fact. It is described in Section 6. Let us apply the statement of local risk minimization to the problems of pattern recognition and regression.


897

In the case of pattern recognition, the outputs y take only two values 0 or 1 and c f ( x , a ) ,cr E A} is a set of indicator functions. The simplest loss function

merely indicates the presence or absence of classification error. The risk functional (2.1) then measures the probability of classification error for the function f ( x , a). The global pattern recognition problem consists in selecting, on the basis of the training set, a function f(x,a*)that guarantees a small probability of classification error. Now, let us consider the local risk functional (2.51,using the hard threshold locality function (2.3). This functional measures the conditional probability of classification error knowing IIx - XOI( 5 ,@/2. The local pattern recognition problem consists in selecting, on the basis of the training set, a value for the locality parameter P' and a function f ( x , a*)which guarantee a small probability of classification error in Ilx -xoll IP/2. In the case of the regression problem, the outputs y are real values, and c f ( x , a),a E A} is a set of real functions. We will consider a quadratic loss function,

Qb,f(x,a11 = b - f(x, a)]'

(2.6)

The minimum of the global risk functional (2.1) is achieved by the closest function of the class c f ( x , a),a E A} to the regression function Y(X)

= EtY I x ) = J

Y W Y I x)

(2.7)

using a quadratic metric

The minimum of the local risk functional (2.5), using the locality function (2.3),is achieved by the closest function of the class c f ( x , a),a E A} to the regression function, ~ ( x using ) a metric

(2.8) where

The local regression problem consists, on the basis of the training set, in selecting a value for the locality parameter p' and a function f ( x , a*), which guarantees a small conditional quadratic error.


898 3 Theory of Uniform Convergence

For simplicity, we will refer to the pairs (x,y) as a vector z, and we will denote the loss function Q[y,f(x, a)]as Q(z,a). The notation F(z)denotes a probability distribution on the pairs ( x , y). First, we will review uniform convergence results for the global risk functional

These results are then extended to the local risk functional, using a transformation of the probability distribution F(z). The global risk can be made local by "oversampling" the probability distribution around the point xo. We already have stressed the fact that optimizing (3.1) is generally impossible, unless we know F(z) exactly. If our knowledge of F(z) is limited to a random independent sample (3.2)

Zl,*'*,ZI

we must rely on an induction principle, like the empirical risk minimization (ERM) or the structural risk minimization (SRM). A good induction principle should provide a way to select a value that guarantees a small value for the risk R(a;). More precisely, two questions should be answered: 1. When is the method of empirical risk minimization consistent? In other words, does the generalization risk R(af) converge to the minimum of the risk functional R(a)when the size l of the training set increases?

2. How fast is this convergence? In general, the number of training examples is limited, and the answer to this question is of crucial practical importance.

Many induction principles, including SRM and ERM, rely on the empirical risk functional 1

1

(3.3)

which estimates the risk R(a) using the training set (3.2). In these cases, the answers to the two questions stated above depend on the quality of the estimation (3.3). More precisely, 1. Does the empirical risk functional El(a) converge to the risk functional R ( a ) when the size of the training set increases, uniformly


899

over the set of functions {Q(z,a),a E A}? The uniform convergence takes place if sup IR(a)- El(a)l > 6 UEA

2. What is the rate of this convergence? The theory of uniform convergence of empirical risk to actual risk developed in the 1970s and 1980s (cf. Vapnik 19821, contains a necessary and sufficient condition for uniform convergence, and provides bounds on the rate of uniform convergence. These bounds do not depend on the distribution function F(z); they are based on a ,measure of the capacity (VC-dimension) of the set of functions {Q(z,a), a E A}.

Definition 1. The VC-dimension of the set of indicator functions {Q(z,a),a E A} is the maximum number h of vectors 21, . . . ,Zh that functions of the set {Q(z,a), a E A} can separate into two classes in all 2h possible ways. Definition 2. The VC-dimension of the set of real functions {Q(z,a), a E A} is defined as the VC-dimension of the following set of indicator functions Q&a) = e [ Q ( z , ~-) c],

0

E A, c E i$Q(z,a),supQ(z,a)

w

1

where c is fixed,

e(u) =

1 ifu>O 0 otherwise.

Two theorems are valid for a set of indicator loss functions. We assume here that the loss functions Q(z, a),a E A are set indicator functions

defined in z-space. Theorem 1. Let the set of indicator functions {Q(z,a),a E A} have VCdimension h. Then the following inequality is true: (3.4)

This theorem is proven in Vapnik (1982, p. 170, Theorem A.2). The quantity (21)h/h! has been bounded by the more convenient quantity (2l~?/h)~, using Stirling's formula. Bound (3.4),however, is limited by the behavior of the absolute difference between the risk and the empirical risk when the risk R(a) is close to 1/2. Theorem 2 provides a bound on the "relative" difference between the risk and the empirical risk.


900

Theorem 2. Let the set of indicator functions {Q(z, a ) , a E A} have VCdimension h. Then the following inequality is true:

This theorem is proven in Vapnik (1982, p. 176, Theorem A.3). Again, the quantity (21)h/h!has been bounded by the more convenient quantity (21e/h)h,using Stirling's formula. Both Theorem 1 and 2 can be generalized to the case of uniformly bounded loss functions. We assume now that the loss functions Q(z, a) are nonnegative, and satisfy the condition 0 5 Q ( z , a )IB ,

aEA

(3.6)

Theorem 3. Let the uniformly bounded set of real functions (3.6) have VCdimension h. Then the following bound is true: (3.7) This theorem is proved in Appendix 1.

Theorem 4. Let a uniformly bounded set offunctions (3.6) have VC-dimension h. Then thefollowing bound is valid:

This theorem is proved in Appendix 2. Finally, we need a bound on the rate of uniform convergence for a set of unbounded real functions {Q(z,a),a E A}. Such a bound requires some restriction on the large deviations of the set of loss functions. This is also true for the classical bounds. Although the law of large numbers says that the average of random values converges to their mathematical expectation, the rate of convergence could be slow. The next example shows that even in the case when the set of functions contains only one function it is impossible to bound the rate of convergence without additional information. Consider a random variable 6 that takes two values: 0 with probability 1 - E and l/$ with probability E . The expectation of this random variable is €

1

E ( C ) = (1- 6)O + - = ; €2


901

The empirical average is null if all 1 observations are 0. The probability of this event is

P(0) = (1 - E ) I For a small E, the expectation E ( { ) is large, but the empirical average is null with a high probability. In Theorems 1 to 4, we have assumed a uniform bound (1 or B) on the losses Q(z,a). This bound forbids large deviations. We consider now the case of nonnegative, unbounded losses, which satisfy the following mild restriction: (3.9)

This condition reflects a restriction on the “tails” of the distribution of the losses Q(z, a). Generally, it means that the probability that random value supaE,, Q(z,a) exceeds some value A “decreases fast” when A increases. Value T determine how fast it decreases. For instance, let Q ( z , Q )be a quadratic loss - f ( x , If the random variable

.2I).

ca

=y-f(x,a)

is distributed accordin to normal law, the ratio of moments in condition (3.9) is equal to 3 (independent of the values of parameters). If the random variable is distributed according to Laplace law, this ratio is (also independent of the value of the parameters). equal to The following result has been proved in Vapnik (1982, p. 202).

/

Theorem 5. Let {Q(z,a ) ,a E A} be a set of nonnegative real functions with VC-dimensionh. Then the following bound is true:

where a ( € )=

J 1 - -1nc

In this formulation again, (21)h/h!has been bounded by (21e/h)h.We obtain a uniform bound for the relative difference between the risk and the empirical risk by applying condition (3.9) to this result.


902

Let us extend this inequality to the case of local algorithms. First, for any fixed value of a and P, note that the local risk functional

is equal to the expectation of the loss function Q(z,a) with respect to a new distribution function F(z,P) defined by

We will consider the set of functions Q(z,a),a E A and the set of probability distribution functions F(z,P), P E ( 0 , ~ that ) satisfies the following inequality:

(3.12) Let us define the unnormalized local risk, R ( a ,P, X O ) , and the unnormalized empirical local risk, &l(a,P, X O ) , as follows:

R(w P, X O ) =

J Q ( z ,a ) K ( x ,

X O , P)dF(z)

1

1

We will show that under condition (3.12)the following inequality is true

w a ,P, xo) - Eda,P, xo) > CXEA

Tea(€)

\/m (3.13)

where h' is VC-dimension of the set of functions

{Qk, a ) W ,xo, PI, a E 4 P E [O,4) To prove this inequality, we note that Theorem 5 implies the following inequality:

(3.14)


903

Moreover, since 0 5 K(x,xo,P) 5 1, we have

-/,

\IJa2(1,u)K2odF(T)5

,/J Q Z ( ~ , a )I I K ( ~P)IIW~, ~, P)

5

and according to (3.12), the following inequality is true for any P

E

(3.15)

LO,4. (3.16)

Inequality (3.13) is derived from inequalities (3.141, (3.15), and (3.16). 4 Bounds for the Local Risk in Pattern Recognition

In this section, we apply the previous results to the problem of pattern recognition. Consider the set of integrands of the unnormalized local risk functional, R(a,P, XO):

{ Q(z,a)K(x,xo,P), a

E A,

P E (0,m)

1

(4.1)

where Q(z,a) is an indicator function and K(x, xo, P ) is a nonnegative real function. Let hl be the VC-dimension of the set of indicator loss functions { Q ( z , a ) ,a E A}. Let hZ be the VC-dimension of the set of nonnegative real functions {K(x, XO, p),P E ( 0 , ~ ) ) . Since Q(z, a) takes only the values 0 or 1, the following equality is true for any nonnegative real function T ( Z , P). e{Q(z, ~ ) T ( zP), - C }

=

Q(z, a)d{r(z,P ) - c}, a

E

A, P

E (O,m),c E (0,W)

Moreover, it is known that the VC-dimension of the product of two sets of indicator functions does not exceed the sum of the VC-dimension of each set of indicator functions. Therefore, the definition of the VCdimension of a set of real function implies that the VC-dimension of the set of functions

{ Q(z,a)K(x,xo,P), a E A, P does not exceed hl

E (0,m)

1

+ hz. Let us apply Theorem 4 to this set of functions.


904

Let 7l/2 denote the right-hand side of this inequality. By solving the equation

and replacing the result into our inequality, we obtain an equivalent formulation: With probability 1- 7/2, the following inequality is true for all functions in { Q ( z ,a), (Y E A, p E (0,~)).

W a ,P, 2 0 ) I El(@,P, 2 0 ) + .9

(4.2)

where

By dividing both sides of inequality (4.2) by

IlK(x0,

P)II, we obtain

The value of I IK(x0, p)I I in the right-hand side of inequality (4.4) depends on the distribution function F(z). A lower bound for the value IlK(x0, p)II is obtained by using the empirical functional:

where zi = (xi,yi) are the elements of the training set (3.2). Applying Theorem 3 to the set of uniformly bounded functions { K ( x , x ~ , p ) , PE (0,oo)) results in

In other words, the following inequality is simultaneously true for all /3 E [0,w[, with probability 1 - 7,1/2:


905

where (u)+ = Max{O, u } . Let us define K ( x 0 , P ) as the right-hand side of inequality (4.5).

By combining inequalities (4.4)and (4.51,we obtain the following theorem, which provides a bound for the local risk functional in the case of pattern recognition.

Theorem 6. Let the VC-dimension of the set of indicator functions { Q ( z ,a ) , a E A} behl. Let the VC-dimensionof theset of realfunctions { K ( x , x o , P ) , P E (0, m)} be h2. The following equality is simultaneousely fulfilled for all a E A and /3 E (0,oo),with probability 1 - 7:

where

As expected, the VC-dimension hl and h2 affect the quantity E, which controls the second term of the sum. The VC-dimension h2 of the set of locality functions {K(x,xo,P),P E (O,m)}, however, also affects the first term of the sum, which is the empirical estimate of the local risk functional. Therefore, it seems extremely advisable to use monotonic radial basis functions for defining the vicinity of a point xo. In fact, the VCdimension of the set of radial basis functions

where the Kp(r) are the monotonically decreasing functions of r, is equal to 1. 5 Bounds of the Local Risk in Regression Estimation

In this section we apply the results presented in Section 3 to the problem of local regression estimation. The loss functions Q(z,a) are now real functions. In the case of pattern recognition, the loss functions were indicator functions. In this case, we have proved that the VC-dimension of the set { Q ( z , a ) K ( x , x ~ , Pa) ,E A, /3 E ( 0 , ~ ) )does not exceed the sum of the VC-dimensions of the sets of functions { Q ( z , a ) , a E A} and { K ( x ,xo, PI, P E (0,..)I.


906

This is no longer true in the case of real loss functions. For example, let { Q ( z ,a),a E A} be the set of monotonically increasing functions, and { K ( x , XO, P), P E (0,oo)) be the set of monotonically decreasing functions. Although the VC-dimension of both sets is 1, the VC-dimension of the product of these sets is infinite. To apply the uniform convergence results, we will assume that the VCdimension h' of the set of function {Q(z,a ) K ( x ,X O , P), a E A, 0 E (0,~)) is finite. We also assume that the functions Q(z,a) are nonnegative, and satisfy condition (3.12). From inequality (3.13) we derive the following inequality, which is simultaneously valid for all a E A, P E (0,oo),with probability 1 - 77/2.

where

+

h[ln(2l/h) 1) - ln(p/24)

(5.2)

W ( x o , P)II

In section 4, we proved that inequality (4.6) is true. Using (4.6) and (5.1), we obtain the following result: Theorem 7. Let the VC-dimension of the set of nonnegative real functions

{Qk, a ) K ( x ,X O , PI, E A, P E ( O , C Q ) ) Q

be h. Let the VC-dimension of the set of locality functions

be hp. The following inequality is simultaneously valid for all a with probability 1- 77,

E

A, P E (0,a),

(5.3)

where

+

h[ln(2l/h) 11- 1n(7/24) UXO,

P)

and IC(x0, P) is defined in (4.6). This result provides a bound on the local risk functional for the case of regression estimation.


907

6 Local Structural Risk Minimization

We can now formulate the principle of local structural risk minimization, using the bounds provided by Theorems 6 and 7. In this section, the local structural risk minimization (LSRM) principle is formulated for pattern recognition. The regression case is essentially similar. Let us consider a nested structure on the set of indicator functions { Q ( z , a ) , cl E 1111

S1 C Sz C,. . . , C S, = {Q(z, C Y ) ,

CY

E A}

(6.1)

Let the VCdimension of each subset S, be hl(p), with

hl(1) < hI(2) < ... < h l ( # ) We have proved, in Section 4, that the VC-dimension of the set of functions

{Q(z,a)K(x,xo,PI,

QI

E

P E (0,

is smaller than hl(p) + hZ, where h2 denotes the VC-dimension of the set of real functions { K ( x ,XO, P), P E (0,~)). According to Theorem 6, the following inequality is simultaneously valid for all element S, of the structure, with probability 1 - 1117:

Principle. The local structural risk minimization principle consists in choosing the element of structure S, and the parameters a E Ap and P E (0,m)that minimize the guaranteed risk as defined by the right-hand side of inequality (6.2). The various constants in bound (6.2)are the result of technical properties of the bounding derivations. The “proven” value is irrelevant to practical problems. Therefore, it is advisable to design experiments to measure these constants, and to use these measured values instead of using the “proven” values.

908


Appendix 1: Proof of Theorem 3 Using Lebesgue's sums, we can write:

where v{Q(z,a) > Bn/N} denotes the frequency of the event

obtained on the basis of the sample 2 1 , .. . ,ZI.Then

Using Theorem 1 and this inequality, we obtain

where h is the VC-dimension of the set of indicator functions

According to Definition 2, this quantity is the VC-dimension of the set of real loss functions {Q(z,a), a E A}. Theorem 3 is thus proven.


909

Appendix 2: Proof of Theorem 4 Again, consider a set real functions { Q ( z ,a),a E A} of VC-dimension h, and assume 0 < Q ( z , a ) < B. The following result is proven in Vapnik (1982, p. 197, Lemma).

Using the Cauchy inequality, we can write

=

d ~ ~ ( a )

We replace this result in inequality (6.1); we bound (21)h/h!by the more convenient expression (2lell1)~; and obtain

Theorem 4 is thus proven. Acknowedgments We thank the members of the Neural Network research group at Bell Labs, Holmdel, for useful discussions. S. Solla and C. Cortes provided help to render this article more clear. References Bottou, L., and Vapnik, V. 1992. Local learning algorithm. Neural Comp. 4(6), 888-901. Vapnik, V. 1982. Estimation of Dependencies Based on Empirical Data. SpringerVerlag, New York. Vapnik, V. 1992. Principles of risk minimization for learning theory. In Neural Information Proceeding System, David S. Touretszky, ed., Vol. 4, pp. 831-839. Morgan Kaufmann, San Mateo, CA. Received 14 July 1992; accepted 4 March 1993.

This article has been cited by: 2. Nicola Segata, Enrico Blanzieri. 2010. Operators for transforming kernels into quasi-local kernels that improve SVM accuracy. Journal of Intelligent Information Systems . [CrossRef] 3. Lee K. Jones. 2009. Local Minimax Learning of Functions With Best Finite Sample Estimation Error Bounds: Applications to Ridge and Lasso Regression, Boosting, Tree Learning, Kernel Machines, and Inverse Problems. IEEE Transactions on Information Theory 55:12, 5700-5727. [CrossRef] 4. Erik McDermott, Shigeru Katagiri. 2006. Discriminative training via minimization of risk estimates based on Parzen smoothing. Applied Intelligence 25:1, 37-57. [CrossRef] 5. M. Muselli. 1997. On convergence properties of pocket algorithm. IEEE Transactions on Neural Networks 8:3, 623-629. [CrossRef] 6. David H. Wolpert. 1996. The Lack of A Priori Distinctions Between Learning AlgorithmsThe Lack of A Priori Distinctions Between Learning Algorithms. Neural Computation 8:7, 1341-1390. [Abstract] [PDF] [PDF Plus]

Communicated by Shun-ichi Amari and Halbert White

On the Geometry of Feedforward Neural Network Error Surfaces An Mei Chen Haw-minn Lu University of California, Sun Diego, C A USA

Robert Hecht-Nielsen HNC, Inc. and University of California, San Diego, C A USA

Many feedforward neural network architectureshave the property that their overall input-output function is unchanged by certain weight permutations and sign flips. In this paper, the geometric structure of these equioutput weight space transformations is explored for the case of multilayer perceptron networks with tunh activation functions (similar results hold for many other types of neural networks). It is shown that these transformations form an algebraic group isomorphic to a direct product of Weyl groups. Results concerning the root spaces of the Lie algebras associated with these Weyl groups are then used to derive sets of simple equations for minimal sufficient search sets in weight space. These sets, which take the geometric forms of a wedge and a cone, occupy only a minute fraction of the volume of weight space. A separate analysis shows that large numbers of copies of a network performance function optimum weight vector are created by the action of the equioutput trandformation group and that these copies all lie on the same sphere. Some implications of these results for learning are discussed. 1 Introduction

For the sake of concreteness, we will concentrate in this paper on the “multilayer perceptron” or ‘%ackpropagation” feedforward neural network architecture (Rumelhart et al. 1986; Hecht-Nielsen 1992). However, many of the results we present can be reformulated to apply to other neural network architectures as well [e.g., the radial basis function networks of Reilly et al. (1982), Broomhead and Lowe (1988), Moody and Darken (1989), and Poggio and Girosi (1990); the ART networks of Carpenter and Grossberg (19911; counterpropagation networks (Hecht-Nielsen Neural Computation 5,910-927 (1993) @ 1993 Massachusetts Institute of Technology

Feedforward Neural Network Error Surfaces

911

1991);and the mutual information preserving networks of Linsker (1988) and Becker and Hinton (1992)J. The layers of the neural networks we consider in this paper are assumed to have units with transfer functions of the form zli = s(4i) MI

Ili

=

C wlijz(l-l)i j=O

for 1 > 1, where tanh(u) for layers 2 through K - 1 u for layer K

number of layers in the network (including input and output layers) layer number (1 through K) number of units on layer 1, assumed to be > 1 xi

= ith component of the

external input vector x

15iSn

yi = jth component of the network output vector y’

15j5rn 1.0 weight of unit i of layer 1 associated with input z(l-l)jfrom layer I - 1 Each layer in this architecture receives inputs from all of the units of the previous layer, but none from any other layer. (Note: all of our results either remain the same or easily generalize when connections skip layers, but the mathematical notation becomes messy. Thus, only the simplest case is presented here.) The network weight vector of a multilayer perceptron neural network is the q-dimensional real Euclidean vector w whose components consist of all of the weights of the network in some fixed order. We shall refer to the space of all such weight vectors (namely, R4) as weight space and denote it by W. Clearly, the network weight vector determines the input-output transfer function of the network.

An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen

912

For the purposes of this paper, we shall assume that each multilayer perceptron neural network under consideration is being used to approximate a fixed, square integrable (i.e., Lz)function

f : A c R "- R m where A is a compact subset of R". Further, we shall assume that the performance of the network is being measured by some performancefunction F(w) that depends only on the network's input-output transfer function (which depends only on the selection of the network weight vector w) and on the manner in which the x vectors in A are chosen (which is assumed to be in accordance with a fixed scheme: namely, random selection with respect to a fixed smooth [C" probability density function p(x) such that the elements of a random vector with this density are linearly independent with probability one]. It suffices (but is not necessary) that the covariance matrix associated with p exist and be finite and nonsingular. Note that this method of choosing the input vectors ensures that they will not have a fixed linear relationship with one anotherwhich could introduce undesired symmetries into the weights of the first hidden layer. Given network performance function F(w), we can view F as a surface hovering over the weight space W with its altitude at each point w determined by the performance of the network with that selection of weight values. Given a multilayer perceptron network, a function f : A c R" -, Rm to approximate, a probability density function p(x), and a particular performance function F(w), we shall refer to such a surface as the performance surface of the network. These assumptions about the network performance function are very mild. For example, functions as diverse as mean squared error, median of squared errors, and the supremum of errors, namely

Fm(w) = N+" lim median[If(xl)- y'(xI,w)(', . . . , If(x~)- y ' ( x ~,w)I2] FdW) = XEA

SUP and p(xp.0

!m-

YYX, w)l

are accommodated within the definition, where y'(x,w) is the output of the multilayer perceptron network, which is an approximation of the desired output of the network y = f(x). The key element of the performance function definition is its dependence only on the input-output transfer function of the network. This allows the network performance to be evaluated not only in terms of


913

just the errors it makes, but also, if desired, in terms of other factorssuch as the curvature of its “approximating surface” (as determined by functions of derivatives of y’(x, w)with respect to the components of the input vector x), as in the networks of Bishop (1991,1990)and Poggio and Girosi (1990). However, explicit dependence of the performance function on factors that are not determined by the input-output behavior of the network-such as direct dependence on the number of hidden units-is not allowed by this definition. The main focus of this paper is the study of geometric transformations of weight space that have the property that they leave the input-output transformation of the neural network unchanged. Obviously, such transformations will also leave all network performance functions unchanged. We begin in Section 2 by showing that all such equioutput transformations are compositions of two simple classes of isometries. Following this, we show that the set of all equioutput transformations forms an algebraic group of surprisingly large order. The fact that there exists a large group of equioutput transformations in weight space implies that performance surfaces are highly redundant, since each network weight vector is equivalent (in terms of determining the same network input-output transfer function and performance) to a multitude of other weight vectors. Thus, if we are searching for an optimum of a performance function, it would seem to be possible, at least in principle, to make our search more efficient by confining it to a small subset of weight space in which all redundancy has been eliminated. In Section 3 we proceed to further analyze our equioutput transformation group by showing that it is isomorphic to a direct product of Weyl groups. In Section 4 we then exploit known facts about these Weyl groups and the root spaces of their associated Lie algebras to derive a set of simple inequalities that defines nonredundant search sets having the geometric forms of a wedge and a cone. These minimal suficient search sets occupy only a minute fraction of the volume of weight space and contain no two equivalent weight vectors, while containing weight vectors equivalent to every weight vector in the space. In Section 5 we consider yet another implication of our transformation group: that each weight vector that optimizes the network performance function is equivalent to many other such optima, and that all of these lie on the same sphere. Finally, in Section 6 we consider the implications of the results of Sections 2,3,4, and 5 for neural network learning. 2 Equioutput Transformations

We begin by studying the properties of weight space transformations that leave the network input-output transfer function unchanged. These transformations are now defined.


914

Definition 1. An equioutput transformation is an analytic (i.e., continuous and expandable in a power series around any point) mapping g : W -+ W from weight space to weight space which leaves the output of the neural network unchanged. In other words,

y’(x,g(w))= y’(x,w) for all x E R” and all w E W . First, consider two types of equioutput transformations: hidden unit weight interchanges and hidden unit weight sign flips. For simplicity, we will refer to these transformations as interchanges and sign pips. An interchange consists of a network weight vector component permutation in which the weight vectors of two hidden units on the same hidden layer are simply interchanged without changing the orderings of the weights within the units. (Note: the term unit weight vector refers to the vector with components equal to the weights within a single unit.) A compensatory interchange of the weights of the next layer units that receive the inputs from the two interchanged units then removes the effect on the network output of the exchange of weights in the previous layer (see Fig. 1). The other type of equioutput transformation is where the weight vector of a hidden layer unit is multiplied by -1 (resulting in a sign flip of the output of the unit-since tanh is an odd function). A compensatory sign flip is then carried out on all of the weights of units of the next layer associated with the input from the sign flipped unit output (see Fig. 2). We now show that: Theorem 1. All equioutput transformations of W to W a r e compositions of interchange and sign pip transformations. Proof. By induction. Let g be any equioutput transformation. We first note that, since y’ is the output of the network for input x, we have MK-I

y:(x,w)

WKij z(K-l)j(x,w)

= j=O

MK-I g(W)Kij z(K-l)j(x,g(w))

= j=O

=

Ykdw))

(2.1)

where y:(x,w) is the output of the original network, y:(x,g(w)) is the output of the network with weight vector g(w) (the transformed network), , and z(~-~)j(x, g(w))are, respectively, the outputs and where q ~ - l ) j ( xW) of the jth units of the last hidden layers of the original and transformed networks. The first step is to take the second partial derivatives of both sums of i W j K , , ~ .Note equations 2.1 with respect to output layer weights ~ ~ and


...

915

...

Figure 1: Interchange transformations involve interchanging the weight vectors of two units on a hidden layer (which has the same effect as interchanging the units themselves). The weights within the units of the next layer that act upon the inputs from the interchanged units are themselves interchanged. The net result is no change in the outputs of these next-layer units. that the first partial of the first sum with respect to WKij is equal to w ) and the second partial of this sum is zero because the output of the last hidden layer of the original network has no dependence on the output layer weights. Thus, we get

Z(K-~)~(X,

(2.2)

for the second partial derivative of the second sum. If we write out the mathematical forms of the four sums of equation 2.2 we see that each nonzero term in each is a transcendental form


916

*

*

*

Figure 2: In a signflip transformation, the signs of all of the weights of a single unit on a hidden layer are changed. The signs of the weights acting on this unit’s output in all of the units on the next layer are also flipped. Again, as with interchange transformations, the outputs of the next-layer units are unchanged that will be, by inspection, in general (i.e., for a11 but a set of weights of measure zero), linearly independent with respect to all of the other terms as the network input x is varied (except for pairs of terms in the middle two sums that have matching values of 1). Thus, except for these pairs, each term in these sums must itself be identically zero. The bad sets of weights need not concern us because we can analytically continue into them. Consider the fourth sum first. In this sum the g(w)Kilare not zero in general so we must have ~ z ( K - l ) l ( x i g ( w=) O ) hKuv h K i j

for all u , v,i,j, and 2. Therefore, (2.3) where w is the vector w with all of the output layer components WKjj removed. In the first sum of equation 2.2 the z(K-1)I(x7g(W))sare clearly


917

nonzero in general, so we must have

for all u , ZI, i, j , and I (we will see below why this is true). Finally, we note that not all of the dg(w)Kil/&.uKuvcan be zero, for if they were, then the output weight g(w)Kil would not depend on any of the output weights of the original network. This cannot be, since then there would be no way to set all of the outputs of the transformed network to zero for all x inputs (which we can do by simply setting all of the output weights of the original network to zero). Thus, we conclude that dz(~-~y(x, g(w))/bKij must be zero. Thus, the weights of the hidden layers of the network do not depend upon the output layer weights [i.e., all of the blv(x,w) of equation 2.3 must be zero]. We now explore the relationship between the output layer weights W K ~ , and g(w)Kij of the original and transformed networks. To do this we expand both z ( K - ~ ) ~W) ( x , and z(K-l),(x,g(w))as power series in x and substitute these into equations 2.1. These expansions are given by

z(K-l)j(x,W)

= Uij

+

and

z(K-l)j(x,g(w))= U2j

b1j + -XTCljX I

XT

2

+ ..

I T + XT bzj + ~x Czj + X

* * *

When we substitute these quantities into the sums of equations 2.1 we note that, since these sums are equal for all values of x, that all of the coefficients in these power series expansions must be equal as well. Thus, we get an infinite set of linear equations MK-I

c

j=O

c g(w)Kij

MK-I

WKij alj

=

j=O

and so on. Since the coefficients in the multidimensional Taylor’s series for z(~-l)j(x, W) and z(K-l)j(x,g(w))are set by controlling the nonoutput layer weights of the network, and since the functional forms achievable by setting these nonoutput layer weights are a rich set of functions [see Sussmann (1992) for a discussion of this property], these equations are, in

918


general, not linearly dependent [note that as the coefficients are changed the W K and ~ g(W)KijS remain fixed]. Thus, the W K ~and S g(w)fijs must be linearly dependent. So, we can write each g(w)Kijas a linear combination of the WKijS, and vice versa, with coefficients that themselves cannot be functions of the output layer weights. Thus, we can write

Substituting equation 2.4 into equation 2.1 and taking the partial derivative with respect to WKij then gives

MK-I

Note that if we set all of the output weights but W K to ~ zero (without changing any of the other weights in the network), equations 2.4 and 2.5 imply that exactly one of the d(w)iuj must equal either +1 or -1 for each fixed i and j. The others must be zero. This follows because neither the d ( W ) i u j S nor the z(K-l)u[x,g(w)]s can be functions of the output weights. Finally, if we substitute equations 2.4 and 2.5 into equation 2.1 we get

j=O

u=O

From this we see that d(w)iuj = d(w)~-,,jfor all i and k. Thus, the d values for each output unit are the same. Thus, the only possible equioutput transformations are those that have the effect of sign flips and interchanges on the last hidden layer. That this is true for all of the other hidden layers (if any) is clear, since we can simply "clip off the output layer of the network and invert the tanh functions of the last hidden layer


919

outputs to create another multilayer perceptron. Applying the above argument to this network then shows that all equioutput transformations act as compositions of interchanges and sign flips on the second to last hidden layer, and so on. 0 We believe, but cannot prove, that the above theorem would hold even if we only demanded continuity (and not analyticity) of our equioutput transformations. In addition to our analytic equioutput transformations, there exist discontinuous conditional equioutput transformations as well. For example, if all of the weights acting on the output of a hidden unit are zero, then the weights of that unit can take on any values without altering the output of the network. So, this unit’s transformed weights might, for example, be set to constant values under this condition, yielding a discontinuous equioutput transformation. Sussmann (1992) has studied and cataloged some of these situations (he has also established a result closely related to Theorem 1). These discontinuous transformations may be worthy of further investigation,as they establish the existence of affine subspace “generators” at at least some points on performance surfaces. If it could be shown that all performance surfaces are generated (like the cylinder (x2/a2)+(y2/a2) = 1 or the hyperboloid (x2/a2)+(y2/a2)-(z2/b2) = 1 in three-dimensional space can be generated by moving a line along a trajectory and correctly controlling the attitude of the line as it moves), this might provide a new line of attack for understanding the geometric structure of such surfaces. Whether performance surfaces will, in general, turn out to be generated is unclear. 3 The Structure of Group G

In this section we show that the set of all equioutput transformations forms a group. Then we analyze the structure of this group. Theorem 2. The set of all equioutput transformations on W forms a nonAbeliun group G of order #G,with

n (M/!)(2M’)

K-1

#G =

1 s

Proof. We first note that the set of interchange transformations involving the interchange of unit weight vectors on hidden layer 1 is in one-to-one correspondence with the set of all permutations of MI elements. Thus, there are (MI!) different interchange transformations for layer 1. The number of si n flips is the same as the number of binary numbers with MI bits, or 2%’. It is easy to show that the interchange and sign flip transformations of one layer commute with those of any other layer. Thus, they are independent and the numbers of transformations on different layers are multiplied to obtain the order of the group. Finally, the set of all such transformations forms a group because, first, it is

920


a subset of the finite symmetry group of a coordinate-axis-aligned cube centered at the origin in weight space, and second, because it is closed under composition (i.e., the product of any two transformations in the 0 set is another transformation in the set). Thus, the set of all weight space transformations that leave the network input-output function unchanged forms an algebraic group G. We now analyze the structure of this group, beginning with a definition. Definition 2. The group Ok is the set of transformations on the vector space Rkgenerated by changing the sign of any one coordinate component and by interchanging any two coordinate components. The 0 notation is used because Ok is isomorphic to the symmetry group for a cube in k-dimensional space. An important fact about Ok is that it is the Weyl group of the n-dimensional classical Lie algebra Bk (Humphreys 1972;Helgason 1962;Weyl 1946). For the results presented here the most important property of the group uk is the fact that it can be represented as the set of reflections generated by the roots of the specific Lie algebra Bk with which it is associated. From Weyl group theory (Humphreys 19721, the order of the group Ok is k! 2k-which is exactly the size of the set of interchangeand sign flip equioutput transformations for a hidden layer with k units. Thus, G might be constructible from Weyl groups. We now show that this suspicion is correct. Theorem 3. The group G is isomorphic to 0~~x OM, x . x OM^-, . Proof. Write the weight space W as B2 x

U2 x B3

x U3 x

x

BK x UK,

where BIis the subspace of bias weights of layer I, and UIis the subspace of nonbias weights of layer 1. The group action can then be expressed for each hidden layer as the direct operation of the cube symmetry group OM, on each subspace BJ and as the indirect, but isomorphic, operation of nonbias weight interchanges and sign flips on the subspaces Ul and U(I+,). Only the hidden layers have symmetry groups associated with them, since the input and output layer units cannot be permuted. Thus, each hidden layer contributes exactly one cube symmetry group to the overall group action. The group is isomorphic to the direct product of these groups because the actions of the individual groups operating on different layers commute. 0 4 Search Sets

In this section we consider minimal sufficient search sets for multilayer perceptron neural networks. First, we provide some definitions that will be needed.


92 1

Definition 3. Two network weight vectors u and v in Ware equivalent iff there exists g E G such that g(u) = v. Definition 4. A minimal suficient search set is a subset S of W such that each w in W is equivalent to exactly one element in S. Definition 5. An open minimal sufficient search set is the interior of a minimal sufficient search set whose closure is equal to the closure of its interior. Previous work by Hecht-Nielsen (1990)and Hecht-Nielsen and Evans (1991)demonstrated that there exist reasonably small, but nonminimal, sufficient search sets in the form of a cone bounded by hyperplanes. We now improve on this earlier work by showing that there exist open minimal sufficient search sets in the geometric forms of a wedge and a cone bounded by hyperplanes. Further, we supply formulas for these sets in terms of simple linear and quadratic inequalities, respectively. Theorem 4. The wedge interior described by the inequalities

Proof. We construct the wedge by piecing together the Weyl chambers in the subspaces BI of W. The cone is then constructed from the wedge. First, to simpllfy the notation, we define U Uzx U3 x . . . x UKso we can rewrite our decomposition of weight space as W = BZ x B3 x . . x BK x U.

=

-

To begin our proof we observe that, since the Weyl group OM, acts directly on BI, BI can be identified with the root space of the classical Lie algebra BM, (Varadarajan 1984). This identification is unique because this particular Weyl group acts directly only on the root space of this one Lie algebra. An open minimal search set for the action of on the root space of B M ~is an open convex subset of the root space known as a Weyl chamber (Varadarajan 1984; Humphreys 1972; Helgason 1962). We will use DI to denote the corresponding subset of BI. To proceed, we shall need the following technical Results concerning compositionsof a group GIdirectly acting on space V1 with open minimal search set SI,and a group GZ directly acting on space VZ with open minimal search set Sz. In particular: 1. Let GIx GZact on VI x Vz coordinatewise. Then S1 x S2 is an open minimal search set of V1 x V z under GIx G1.


922

2. If GI = G2 = G and g E G acts on (v1,vz)by g(v1,vZ) = (gvl,gv2), then S1 x V 2is an open minimal search set for V1 x V2 under GI.

The proofs, which are elementary, are omitted. By applying Result 1 successively to B2 x B3, then to ( B 2 x B3) x B l , and so on, we see that 02 x 0 3 x . . x D(x-1) is an open minimal sufficient search set for B2 x B3 x . x B ( K - I ) . Applying Result 2 to (B2 x B3 x . x B ( K - I ) ) x ( B K x U) then shows that 02 x D3 x . x D ( K - ~x) BK x U is a minimal sufficient search set for W. Having characterized an open minimal sufficient search set for W,we now use the fact from Lie algebra theory that a Weyl chamber in the root space of Bk is determined by the inequalities a . w > 0, where Q is the Riesz representation vector for a positive root of the algebra (Varadarajan 1984; Humphreys 1972; Helgason 1962; Bachman and Narici 1966). For Lie algebra Bk there are k2 positive roots of the form &?; and &(& f i j ) , where the & are basis vectors in root space. We choose the roots i i and 2i f i j , with i < j, to be positive. Identifying these basis vectors with the hidden layer bias weight space positive coordinate axes gives us the following three sets of equations

-

Wlio>O

--

for l < i < M l , 2 < I < ( K - l )

wlio-w~j0>O

for l < i < j < M ~ ,2 < I < ( K - 1 )

wlio+wl,o>O

for 1 l i < j < M I , 2 5 I < ( K - 1 )

However, the last set of inequalities is redundant, given the first set. Also, some of the inequalities in the second set are redundant. For example, if W I N > W I Nand W I N > W I W , then there is no need for the condition that WIN > WIN.Thus, we are left with inequalities 4.1 describing an open minimal sufficient search set in the form of a wedge. Note that this is a wedge because any positive real multiple of a member of the set is also a member of the set (as opposed to a cone, for which any real multiple must also be in the set). The cone interior described by inequalities 4.2 is constructed by simply breaking the wedge across the hyperplane through the origin perpendicular to the bias weight axis of the first output layer unit, throwing away the portion of the wedge that intersects this hyperplane, and then rotating the bottom (negative) half-space half of the broken wedge by 180". We can do this since the bias weights of the output units are unaffected by the group. 0 Note that the wedge equations can be summarized by the simple statement that the weight vectors within (in the interior of) the fundamental wedge have all of their hidden layer bias weights positive and that the bias weight of each hidden unit is larger than the bias weight of the unit directly to its right. A similar statement holds for the cone. Also note that to turn these open minimal sufficient search sets into minimal sufficient search sets, we would have to add certain (but not all)


923

points on their boundaries. For example, points with Wljo = WI,O for all units of each hidden layer would have to be added. Thus, for practical applications, we might simply want to use 2 inequalities in 4.1 and 4.2. Note that the images of a minimal sufficient search set S under different transformations in G are, except for certain points on their boundaries, disjoint. The entire weight space thus consists of the union of these sets

w = ug[SI gEG

As a result of this fact, and of the fact that the equioutput transformations themselves (namely, the elements of G) preserve not only the output of the network for all inputs but, thereby, the value of the network performance function as well, the network performance function is uniquely determined everywhere by its behavior on a minimal sufficient search set. Unfortunately, the manner in which the behavior of a performance function in a wedge copy is determined from the behavior of that function within the fundamental wedge is not simple, since the hyperplanes that bound the wedge are not planes of symmetry for the transformations of G. That they are not is easy to see, since if w and w' are weight vectors that have all of their components equal except for one hidden unit bias weight differing in sign, or the bias weights of two adjacent hidden units interchanged (i.e., points placed symmetrically with respect to one of the bounding hyperplanes of the fundamental wedge), then, in general, there will be no transformation g E G such that g(w) will equal w' (since the other weights involved in a sign flip or interchange transformation are not properly modified by this hyperplane reflection). Thus, in general, Fk(w)] will not equal F ( w ' ) . Understanding the relationship between the symmetries of G and the geometry of the fundamental wedge (or other minima\ sufficient search sets) would seem to be an issue worthy of further investigation. In this section we have examined one ramification of the geometric structure of weight space. Namely, the fact that searches for optimum weight vector values can be confined to an extremely small portion of the weight space. In the next section we consider another ramification of the group G. 5 Spherical Throngs of Optima

Another fact about the transformations in the group G is that they are isometries. Thus, for any w E W and any g E G,

Ig(w)I = IWI That this is so, is easy to see because the only effect of applying any combination of sign flips and interchanges is to change the signs and

924


permute the components of the weight vector, neither of which affect the Euclidean length of the vector. Given that the elements of G are isometries, if W*is a finite weight vector which optimizes the network performance function, then the points obtained by applying the elements of G to w*all lie on the same sphere around the origin in weight space. In general, these points are all different and there are #G of them. We call such a set of points a spherical throng of optima. Note that the copies of an optimal weight vector w*in its spherical throng will, on average, have half of their weights modified from their corresponding values in w*by a transformation in G. This is easy to see, since half of the permutation and sign flip compositions change more than half of the weights and half fewer than half. Thus, the copies of w* in its throng will, in general, be scattered all over the sphere. Of course, in rare cases (such as where all of the weights have the same value), these copies are not scattered widely; but in general they are. It is easy to see that, given any member w of the throng, the nearest neighbor member on the sphere will typically be a vector that is the same as w except for a single sign flip or interchange transformation. Thus, nearest neighbors in the throng will tend to have most of their components the same. The areal density of a spherical throng of optima depends upon the magnitude of w*,since the number of members of the throng is, in general, #G, which depends only on the architecture of the network. If this magnitude (Iw*I)is small, then the density will be high. Extensive experience with multilayer perceptrons at HNC, Inc. has shown that the lengths of apparently near-optimal weight vectors rarely, if ever, have lengths greater than fl,where again q is the number of weights in the network. In other words, the rms weight values in near-optimum weight vectors are typically considerably less than 1.0 in magnitude (exactly why this is so is unknown). Thus, it might be that these throngs are not only large, but often dense as well. 6 Implications

In this section we consider some implications of the results of Sections 2, 3, 4, and 5. The existence of simple formulas for minimal sufficient search sets raises the question of whether such formulas will be of use in learning. They probably will not. In the case of gradient descent learning they would not be of use, since if we are going downhill, and happen to cross the boundary of the minimal sufficient search set, we should just continue the descent. Even if we wanted to move to the corresponding point of the performance surface within the fundamental wedge we could not do so, since (as pointed out in Section 4) we do not yet have a formula for finding this point.


925

For learning methods that employ stochastic jumping, rule-based weight modification, or another nongradient descent method, it might seem to be of use to constrain the hidden layer bias weights so as to force the network weight vector to remain within the fundamental wedge (or some other minimal sufficient search set). However, this is not really true, as the following example shows. Imagine a simple performance surface with one and only one finite minimum (located near, but not at, the origin) within the fundamental wedge. The goal is to find a weight vector within 6 distance of this minimum. Suppose that a simple unconstrained discrete-time gaussian random weight space search were being used to find this minimum. Then there would appear to be a search speed-up of #G to be gained by constraining the search to an equivalent search process within a minimal sufficient search set. However, this is an illusion, because the unconstrained search process is not trying to find a single minimum (as the constrained process is). It need only find one of #G equivalent copies of the minimum. Therefore, both searches will have the same expected number of steps. Thus, we conclude that knowing the geometry of a minimal sufficient search set has no obvious benefit for learning. With respect to spherical throngs of optima, we speculate that gradient descent learning may be aided by the fact that most learning procedures follow the tradition of starting the weight vector components at random values chosen over a small interval centered at zero. Starting near the origin has more than one benefit. First, starting near the origin causes the “approximating surface” of the network to start out nearly “flat”-with its initial output value near zero everywhere. As the training process proceeds this initially flat approximating surface begins to “crinkle up” as it tries to fit the training data. Thus, starting the weight values near zero provides a parsimonious surface of initially nearly zero curvature. Another reason why this tradition is so apt is that the usual activation functions tanh and (1 e-”)-’ have all of their significant behavior near zero. Thus, one would naturally expect that, if the inputs to the network tend to be small in magnitude, large weight values would be needed only rarely. As mentioned above, anecdotal evidence suggests that this is what occurs in many practical situations. Geometrically, when gradient descent training begins with an initial weight vector near the origin, we conjecture that the learning process consists of an initial, generally outward, movement from the origin to a radius at which a spherical throng of optima is encountered, followed by a “homing in” process that guides the weight vector toward an optimum. If this conjecture is correct, and if, as we suspect, many practical problems have a performance surface with a spherical throng of optima located at a relatively small radius, then this dense shell of optima may be a relatively “easy” target to hit. In other words, in contrast to the typical optimization situation (e.g., in linear programming, combinatorial

+


926

optimization, or unconstrained optimization), where we are searching a high-dimensional space for a single optimum (or one of a small number of optima), here we are searching a high-dimensional space for any one of a vast multitude of optima. This may partially explain why the training of multilayer perceptron networks with thousands or even millions of adaptive weights is often practically feasible (a fact that we now take for granted, but which, a priori, is rather surprising). Acknowledgments We thank the referees for several helpful comments, including the observation that discontinuous equioutput transformations can exist (this fact was also pointed out to us by Arcady Nemirovskii). Thanks also to Luis Almeida, Charles Fefferman, Lee Giles, David Hestenes, Margarita Kuzmina, Irina Surina, and Robert Williamson for valuable discussions. References Bachman, G., and Narici, L. 1966. Functional Analysis. Academic Press, New York. Becker, S., and Hinton, G. E. 1992. Self-organizingneural network that discovers surfaces in random-dot stereograms. Nature (London) 355, 161-163. Bishop, C. M. 1991. Improving the generalization properties of radial basis function neural networks. Neural Comp. 3,579-588. Bishop, C. M. 1990. Curvature-driven smoothing in backpropagation neural networks. Proc. of the lnternational Neural Network Conf., Paris. 2, 749-752. Kluwer, Dordrecht. Broomhead, D. S., and Lowe, D. 1988. Multivariable function interpolation and adaptive networks. Complex Syst. 2,321-355. Carpenter, G . A., Grossberg, S., and Reynolds, J. H. 1991. ARTMAP: Supervised real-time learning and classification of nonstationary data by a selforganizing neural network. Neural Networks 4, 565-588. Chen, A. M., and Hecht-Nielsen, R. 1991. On the geometry of feedforward neural network weight spaces. Proc. Second 1EE International Conference on Neural Networks, 1-4. IEE Press,London. Hartmann, E. J., Keeler, J. D., and Kowalski, J. M. 1990. Layered neural networks with gaussian hidden units as universal approximations. Neural Comp. 2, 210-215.

Hecht-Nielsen, R. 1992. Theory of the backpropagation neural network. In Neural Networks for Human and Machine Perception, Volume 2, H. Wechsler, ed., pp. 65-93. Academic Press, Boston, MA. Hecht-Nielsen, R. 1991. Neurocomputing. Addison-Wesley, Reading, MA. Hecht-Nielsen, R. 1990. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, R. Eckmiller, ed. ElsevierNorth Holland, Amsterdam.


927

Hecht-Nielsen, R., and Evans, K. M. 1991. A method for error surface analysis. In Theoretical Aspects of Neurocomputing, M. NovAk and E. Pelikh, eds., pp. 13-18. World Scientific, Singapore. Helgason, S. 1962. Differential Geometry and Symmetric Spaces. Academic Press, New York. Humphreys, J. E. 1972. Introduction to Lie Algebras and Representation Theory. Springer-Verlag,New York. Linsker, R. 1988. Self-organization in a perceptual network. IEEE Computer Mag. 21, 105-117. Moody, J,, and Darken, C. J. 1989. Fast learning in networks of locally-tuned processing units. Neural Comp. 1, 281-294. Poggio, T., and Girosi, F. 1990. Regularization algorithms for learning that are equivalent to multilayer networks. Science 247,978-982. Reilly, D. L., Cooper, L. N., and Elbaum, C. 1982. A neural model for category learning. Biol. Cyber. 45, 35-41. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, eds. MIT Press, Cambridge, MA. Sussmann, H. J. 1992. Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks 5, 589-593. Varadarajan, V. S. 1984. Lie Groups, Lie Algebras, and Their Representations. Springer-Verlag, New York. Weyl, H. 1946. The Classical Groups. Princeton University Press, Princeton.

Received 27 March 1992; accepted 15 March 1993.

This article has been cited by: 2. Christopher DiMattina, Kechen Zhang. 2010. How to Modify a Neural Network Gradually Without Changing Its Input-Output FunctionalityHow to Modify a Neural Network Gradually Without Changing Its Input-Output Functionality. Neural Computation 22:1, 1-47. [Abstract] [Full Text] [PDF] [PDF Plus] [Supplementary material] 3. Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, Shun-ichi Amari. 2008. Dynamics of Learning Near Singularities in Layered NetworksDynamics of Learning Near Singularities in Layered Networks. Neural Computation 20:3, 813-843. [Abstract] [PDF] [PDF Plus] 4. Manfred M Fischer. 2006. Neural Networks: A General Framework for Non-Linear Function Approximation. Transactions in GIS 10:4, 521-533. [CrossRef] 5. Shun-ichi Amari , Hyeyoung Park , Tomoko Ozeki . 2006. Singularities Affect Dynamics of Learning in NeuromanifoldsSingularities Affect Dynamics of Learning in Neuromanifolds. Neural Computation 18:5, 1007-1065. [Abstract] [PDF] [PDF Plus] 6. V. Rossi, J.-P. Vila. 2006. Bayesian Multioutput Feedforward Neural Networks Comparison: A Conjugate Prior Approach. IEEE Transactions on Neural Networks 17:1, 35-47. [CrossRef] 7. Masahiro Kimura . 2002. On Unique Representations of Certain Dynamical Systems Produced by Continuous-Time Recurrent Neural NetworksOn Unique Representations of Certain Dynamical Systems Produced by Continuous-Time Recurrent Neural Networks. Neural Computation 14:12, 2981-2996. [Abstract] [PDF] [PDF Plus] 8. H. White, J. Racine. 2001. Statistical inference, the bootstrap, and neural-network modeling with application to foreign exchange rates. IEEE Transactions on Neural Networks 12:4, 657-673. [CrossRef] 9. J.-P. Vila, V. Wagner, P. Neveu. 2000. Bayesian nonlinear model selection and neural networks: a conjugate prior approach. IEEE Transactions on Neural Networks 11:2, 265-278. [CrossRef] 10. M. Gori, Ah Chung Tsoi. 1998. Comments on local minima free conditions in multilayer perceptrons. IEEE Transactions on Neural Networks 9:5, 1051-1053. [CrossRef] 11. M. Gori, M. Maggini. 1996. Optimal convergence of on-line backpropagation. IEEE Transactions on Neural Networks 7:1, 251-254. [CrossRef] 12. Frans M. Coetzee , Virginia L. Stonick . 1995. Topology and Geometry of Single Hidden Layer Network, Least Squares Weight SolutionsTopology and Geometry of Single Hidden Layer Network, Least Squares Weight Solutions. Neural Computation 7:4, 672-705. [Abstract] [PDF] [PDF Plus]

13. Věra Kůrková , Paul C. Kainen . 1994. Functionally Equivalent Feedforward Neural NetworksFunctionally Equivalent Feedforward Neural Networks. Neural Computation 6:3, 543-558. [Abstract] [PDF] [PDF Plus] 14. D Saad. 1994. Journal of Physics A: Mathematical and General 27:8, 2719-2734. [CrossRef]

Communicated by Stephen J. Nowlan

Rational Function Neural Network Henry Leung Surface Radar Section, Radar Division, Defence Research Establishment Ottawa, Ottawa, Ontario, Canada KIA OK2

Simon Haykin Communications Research Laboratoy,McMaster University, Hamilton, Ontario, Canada LBS 4Kl

In this paper we observe that a particular class of rational function (RF) approximations may be viewed as feedforward networks. Like the radial basis function (RBF) network, the training of the RF network may be performed using a linear adaptive filtering algorithm. We illustrate the application of the RF network by considering two nonlinear signal processing problems. The first problem concerns the one-step prediction of a time series consisting of a pair of complex sinusoid in the presence of colored non-gaussian noise. Simulated data were used for this problem. In the second problem, we use the RF network to build a nonlinear dynamic model of sea clutter (radar backscattering from a sea surface); here, real-life data were used for the study.

Neural networks are nonlinear parametric models that can approximate any continuous input-output relation. The problem of finding a suitable set of parameters that approximates an unknown relation is usually solved using a learning algorithm. The problem of learning a mapping between an input and an output space is equivalent to the problem of synthesizing an associative memory that retrieves the appropriate output pattern when presented with the associated input pattern, and generalizes when presented with new inputs. A classical framework for this problem is approximation theory. Neural networks functioning as approximators of general maps are currently under intense investigation (Poggio and Girosi 1990; Moody and Darken 1988). A very important class of applications is nonlinear signal processing, particularly the prediction of a chaotic time series. In this application, a neural network predictor may tell the difference between purely random and deterministic processes, and in the latter case allow longer time predictions. The practical engineering problem that we attempt to solve using neural networks in this paper is one such Neural Computation 5,928-938 (1993) @ 1993 Massachusetts Institute of Technology

Rational Fundion Neural Network

929

example. In Leung and Haykin (19901, we demonstrated that sea clutter (electromagnetic backscattering from a sea surface) contains a strange attractor. In other words, sea clutter may permit a chaotic description. To detect a small target in an ocean environment, a common technique is to build a model for sea clutter, and then suppress the clutter (interference) by means of that model. Since a neural network is powerful in predicting a chaotic sequence (Lapedes and Farber 1987) (in other words, it can provide a good model for a chaotic sequence), we would like to use a neural network to build a model for sea clutter to perform adaptive radar detection. Approximation theory deals with the problem of approximating a function f(x) of an input vector x by another function F(w,x) having a fixed number of parameters denoted by the vector w. The parameters of F are chosen so as to achieve the best possible approximation of the function f. As pointed out in Poggio and Girosi (1990), there are three main problems of designing a neural network from the point of view of approximation theory: 1. Which particular approximation F to use.

2. For a given choice of F, which algorithm to use for finding the optimal values of the parameters that characterize F. 3. For a selected algorithm, which efficient implementation to use.

Problem 1 relates to the computational power of a neural network. A basic requirement is that the network should be a universal approximator; that is, it should approximate any continuous function. The conventional multilayer perceptron (MLP) is one such example (Funahashi 1989), although it is not usually designed for this purpose. The radial basis function (RBD neural network (Poggioand Girosi 1990; Moody and Darken 1988; Broomhead and Lowe 1988) is another example. The RBF network is particularly powerful from the point of view of approximation theory. Not only is it a universal approximator, but it also has many nice function approximation properties such as the existence of best approximation (Girosi and Poggio 1990) and the ability of regularizing an ill-posed problem (Poggioand Girosi 1990); the latter feature cannot be found in a conventional MLP. These properties make it possible for the RBF network to exhibit a robust performance even in noisy data. Problem 2 relates to the efficiency of a neural network. It is pointed out in Moody and Darken (1988) that a major handicap of an MLP is the inefficiency of the backpropagation (BP) algorithm commonly used to train it. Even when the MLP is implemented using a faster optimization procedure such as the conjugate gradient algorithm, it is still painfully slow. The convergence is so slow that training usually requires repeatedly feeding the training data into the network. It is suggested in Moody and Darken (1988) that while this approach makes sense for

930

Henry Leung and Simon Haykin

“off-line” problems, it is probably too inefficient for solving many realtime problems found in such areas as adaptive signal processing and biological information processing. In this paper we propose a network architecture that uses a rational function (RF)(Braess 1986) to construct a mapping neural network. Our motivation for using a rational function is 5-fold: 1. The class of rational functions has been proven to universally approximate real-valued functions having certain smoothness properties (Braess 1986). It is a global approximator as a polynomial function (Barron and Barron 1988). A rational function usually requires fewer parameters than a polynomial (lower complexity) and it is a better extrapolator than a polynomial (better generalization ability) (Lee and Lee 1988). However, the complexity problem is still considered as a major drawback of the rational function approach. One suggestion to handle this problem is through the self-organization method described in Farlow (1984). 2. Rational functions constitute a well-developed class of approxima-

tion techniques. Many useful properties such as the existence of best approximation are already known; knowledge of these properties can help us better understand the ability and the performance of a neural network based on a rational function. 3. The parameters of a rational function can be computed by a linear adaptive filtering algorithm such as the recursive least square (RLS) algorithm. Consequently, an RF neural network can reach a global optimum without using a nonconvex optimization technique.

4. An RF network can be implemented efficiently on a systolic array by using a linear adaptive filtering algorithm such as the recursive least squares-QR decomposition (RLSQRD) algorithm (Haykin 1991 or its fast versions (Haykin 1991; Cioffi 1990);it is therefore well-suited for solving real-time problems. 5. Rational functions can model many real-life problems [e.g., optical

transformations (Pitas and Venetsanopoulos 19901, interpolation of

TV image sequences (Pitas and Venetsanopoulos 1990), input resistance of cascaded resistance networks (Daiuto etal. 1989), and image propagation for two inward-facing parabolic mirrors (Daiuto et al. 1989)l. Points 1 to 2 relate to Problem 1 of designing a neural network from the viewpoint of approximation theory mentioned previously. Points 3 and 4 provide possible solutions to Problems 2 and 3, respectively. Point 5 emphasizes the wide applicability of rational functions for solving reallife problems.

Rational Function Neural Network A rational function R” shown by

+

931

R is the quotient of two polynomials, as

where X I , x2, . . . ,xm are the scalar inputs applied to the network. The set ( X I , x2, . . . ,x,) forms a vector x in R”, and y is the value of the mapping of that vector in the range R. The representation of equation 1 is unique, up to constant factors in the numerator and the denominator polynomials. The rational function must clearly have a finite order for it to be useful in solving a real-life problem. Let the order of the numerator polynomial be a and that of the denominator polynomial be ,f3. Then, we say that the rational function has order (a,p), and so denote it by Rap. Assume that we have an (&,@rational function and the desired response is d. To get the best approximation, we seek a rational function belonging to Rap that solves the following minimization problem: t

m i n x Id(i) - Rap(i)12

(2)

i=l

where t is the total number of examples available for learning. To obtain the optimum estimation of the coefficients of a rational function neural network, we may first transform equation 2 to a leastsquares problem of solving a set of nonlinear equations, that is,

. = ...... . - ...... . - ......

Note that the number of equations should be larger than the number of variables (i.e., we have an overdetermined system of equations), which is a general setting for least-squares estimation. Solving these nonlinear equations directly is a difficult job, which usually requires convex optimization. To avoid this, we use cross-multiplication to move the denominator on the right side of equation 3 to the left side. Also, without loss of generality, bo can be assumed to be unity. After rearrangement of terms, we obtain a new set of linear equations (linear in the parameters).

932

Henry bung and Simon Haykin

Expressing these equations in matrix notation, we have

We now have a linear learning problem. Moreover, the minimum norm solution of the least-squares estimation problem can be obtained by any linear adaptive filtering algorithm such as the least mean squares (LMS) algorithm or the recursive least squares (RLS) algorithm. In Table I, we present a summary of the adaptive algorithm for a rational function using the RLS algorithm. Note that the nonlinearity of the RF neural network as described herein manifests itself by virtue of the nonlinear way in which the vector u(n) is defined in terms of the input data x l ( n ) , . . . ,x,,,(n) and the desired response d(n). The weight vector w(n) of the network is defined in terms of the numerator and denominator coefficients of the rational function. Next we try to show how a neural network based on rational functions can be used to represent a multidimensional mapping from R"' to R". The network has m input units and n output units. There are hidden layers that form all the polynomial combinations needed to construct the rational function of interest. Each output unit combines all the outputs of the hidden units to form a rational function representation as shown in equation 1. The input layer of a rational function neural network consists of a set of m nodes, into which we feed the components of the m-dimensional vector ( X , , X ~ , . .. ,xm). The first hidden layer is designed to form all the second-order components that are common to the numerator and d e nominator polynomials of the rational function. The des'ired response is also fed into this hidden layer to form second-order components. The second hidden layer is then assigned to the formation of third-order components, and so on for all the other hidden layers. Consider, for example, a rational function with a highest order of seven in either the numerator or denominator polynomial. We will then have six hidden layers to get all the polynomial combinations. Basically, the hidden layer units multiply their inputs and all incoming hidden weights are equal to l, and

Rational Function Neural Network

933

Table 1: RLSBased Rational Function Neural Network Algorithm. Initialize the algorithm by setting

P(o) = 6-'I w(0) = 0

6 = small positive constant

For each instant of time, n = 1,2,.. ., compute

k(n) =

X-'P(n

1

- l)u(n) - l)u(n)

+ X-'uT(n)P(n

r(n) = d(n) - wT(n- l)u(n) w(n) = w(n - 1) + k(n)r(n) P(n) = X-'P(n - 1) - X-'k(n)uT(n)P(n

- I)

where

and where x l ( n ) ,. . . ,x,(n) are the input data, d ( n ) is the desired response, and w(n) is the adjustable parameter vector to be computed.

nonadaptive while the output layer units calculate a weighted sum of their inputs, with the weights adaptive and corresponding to the rational function coefficients. For the purpose of illustration, an RF neural network of general character is depicted in Figure la. We have also included Figure l b to illustrate that there are direct connections from all hidden units of the network to the output layer. Note that an RF neural network does not feed back the training error but rather the desired response. Another noteworthy point is that those terms that contain a d(i), the desired response, will not have any d(i) after the learning period. In actual fact, such terms constitute the denominator polynomial of the rational function of interest. We next illustrate the application of RF neural networks by considering two different nonlinear signal processing problems. We first apply the RF network to a nonlinear prediction problem, for which the signal used is described by the following equation: x ( t ) = p i ( 5 ) t + 2=i(-5)f + n ( t )

(5)

The additive colored noise n ( t ) is generated by passing a white uniformly distributed process through a finite-duration-impulseresponse(FIR)fil-

Henry b u n g and Simon Haykin

934

I

a

I

dl

dN

'1

'2 :

OUTPUT LAYER

HIDDEN LAYERS

INPUT LAYER

XN

DENOMINATOR NODE

0 : NUMERATOR NODE (2 : NODES EXISTING IN TRAINING ONLY

Figure 1: (a) A rational function neural network. (b) A portion of the RF network illustrating connections within the network. ter with impulse response 15

h ( t ) = Cajs(t- i)

(6)

i=O

where the filter coefficients are chosen to be the same as those used in Papadopoulos and Nikias (1990). They are {0.5,0.6,0.7,0.8,0.7,0.6,0.5, 0.0, 0.0,0.5,0.6,0.7,0.8,0.7,0.6,0.5}. The signal-to-noise ratio is set to be 0 dB. We use an Rll predictor to perform the one-step-ahead prediction, and compare the result with that obtained using a linear predictor. The input vector for both predictors has a dimension of 4, and 50 samples are used for training. The training is carried out using the RLS algorithm. After training, we present new data (not in the training sets), which are


935

generated by the same model, to test both predictors. The mean and standard deviation of the prediction error for the RF predictor are 0.37 and 0.24, respectively. For the linear predictor, the mean and the standard deviation of the prediction error are 0.52 and 0.37,respectively. The mean prediction error for the RF predictor is about 3 dB less than that for the linear predictor. For the second experiment, we use the RF network to study the sea clutter modeling problem using real-life radar data. In particular, the sea clutter data used were obtained using the IPIX radar (Krasnor etal. 1989) located at a site on Cape Bonavista, Newfoundland, Canada. The radar was used to probe the Ocean surface along a radial line. The radar pulse repetition rate was 2 kHz. The sea state was about 1.57 m. The modeling begins by using the neural network as a predictor. The way to do it is very simple. The number of input neurons depends on the embedding dimension of the sea clutter process, which has been shown experimentally to be an integer greater than 6.5 (Leung and Haykin 1990), and the output layer consists of a single neuron that provides the predicted value. After the learning phase is completed, the network is frozen, that is, the connection weights are not allowed to change any more because the dynamic process is assumed to be time-invariant. The rational function neural network used here had a ( 2 , l ) structure, which may be justified as follows. Obviously, order (1,O) cannot be used since it is just a linear model. Also, order (1,l) is not suitable by a recent discovery (Daiuto et al. 1989) that this structure cannot produce chaotic behavior since it is not sensitive to initial conditions. Thus, the simplest rational function which can generate a chaotic behavior is the ( 2 , l ) structure. Of course, a higher order structure also has the potential to produce a chaotic behavior, but the complexity would be greatly increased, especially when the dimension of the input data is high. Introducing too many parameters is not recommended by the informational Occam’s razor. For this particular data set, we chose an embedding dimension of 7. The second layer then contains 49 elements, which form the second-order components of the polynomials. Thus, there are a total of 65 parameters for estimation in this structure. The resulting training error is shown in Figure 2. We observe that the training speed in terms of number of training samples of the RF network is comparable to that of a RBF network. However, the computational time required for each iteration for the RF network is much less than that for the RBF network. To confirm the validity of the model, we cannot simply look at the training error. A small training error tells us only that the network fits the training data, a task that in principle can be accomplished by any model with sufficient parameters. More specifically, after the learning is completed, we have to study the ability of the network to generalize. To do so, we present the network with data not seen before, and observe the performance of the network. If the prediction error is reasonably small,

Henry Leung and Simon Haykin

936

Training error of sea clutter prediction using RBF and RF network

-

rational function

--radial basis function

'0

10

20

30

40 50 60 Number of training sample

70

80

90

100

Figure 2 Learning curve of RF and RBF network for sea clutter prediction. The y-axis is the absolute value of the normalized training error (i.e., the magnitudes of training error are scaled into the range [0,1]).The x-axis represents the number of samples fed into the network for training.

we can then say that the model is an appropriate one. In this paper, we use the onestepahead prediction to demonstrate the generalization ability of the RF network. The normalized prediction error used here is a dynamic range independent measure (Casdagli 1989), which is defined as the absolute prediction error divided by the standard deviation of the data sequence. The prediction errors are computed as an average of 50 trials and each trial consists of 50 points for prediction. The mean and the standard deviation of the normalized prediction error for the RF network are 0.327 and 0.254,respectively. The same procedure was also applied to the RBF neural network (Moody and Darken 1988) for comparison. (We do not choose a conventional MLP for comparison, because this network requires repeated training that is not suitable for OUT real-time signal processing problem.) The mean and the standard deviation of the


937

normalized prediction error for the RBF network are 0.329 and 0.223, respectively. We observe that the prediction error performance of the RF network is about the same as the RBF network. However, the complexity of the RF network is lower than that of the RBF network. Not only does the RBF network need to compute the complicated Euclidean distance of high-dimensional vectors and use a time-consuming k-means algorithm, but it also needs 200 to 300 hidden units to obtain a similar performance. Based on the mathematical treatment presented and the nonlinear signal processing applications described herein, we suggest that a neural network based on a rational function approximation is a reasonably good mapping network, especially for real-time applications.

References Barron, A. R., and Barron, R. L. 1988. Statistical learning networks A unifymg view. In Symposium on the Interface: Statistics and Computing Science, E . Wegman, ed., pp. 192-203. American StatisticalAssociation, Washington,

Dc. Braess, D. 1986. Nonlinear Approximation Theory. Springer-Verlag,Berlin. Broomhead, D. S., and Lowe, D. 1988. Multivariable functional interpolation and adaptive networks. Complex Systems 2,321-355. Casdagli, M. 1989. Nonlinear prediction of chaotic time series. Physica D 35, 335-356. Cioff?, J. M. 1990. The fast adaptive ROTORS RLS algorithm. IEEE Trans. Acoustics, Speech, Signal Process. 38(4), 631-653. Daiuto, B. J., Hartley, T. T., and Chicatelli, S. P. 1989. The Hyperbolic Map and Applications to the Linear Quadratic Regulator. Ledure Notes in Control and Information Sciences, Vol. 110, M. Thoma and A. Wyner, eds. SpringerVerlag, Berlin. Farlow, S., ed., Self-organizing Methods in Modeling. Marcel Dekker, New York. Funahashi, K. 1989. On the approximate realization of continuous mapping by neural network. Neural Networks 2, 183-192. Gimsi, E, and Poggio, T. 1990. Networks and the best approximation property. Biological Cybernetics 63, 169-176. Haykin, S. 1991. AdaptiveFilter Theory, 2nd ed. h n t i c e Hall, Englewood Cliffs, NJ. Krasnor, C., Haykin, S., Cume, B., and Nohara, T.1989. A dual-polarized radar system. Presented at the International Conference on Radar, Paris, France. Lapedes, A., and Farber, R. 1987. Nonlinear signal processing using neural networks : Prediction and system modelling. Los Alamos National Laboratory, LA-UR-87-2662. Lee, K., and Lee, Y. C. 1988. System modeling with rational function. Los Alamos National Laboratory Report. hung, H., and Haykin, S. 1990. Is there a radar clutter attractor? Appl. Phys. Lett. 56(6), 593-595.

938

Henry h u n g and Simon Haykin

Moody, J., and Darken, C. 1988. Learning with localized receptive fields. In Proceedings of the 1988 Connectionist Models Summer School, D. Touretzky, G . Hinton, and T. Sejnowski, eds., pp. 133-143. Morgan Kaufmann, San Mateo, CA. Papadopoulos, C. K., and Nikias, C. L. 1990. Parameter estimation of exponentially damped sinusoid using higher-order statistics. IEEE Trans. Acoustic, Speech, Signal Process. 38(8), 1424-1436. Pitas, I., and Venetsanopoulos, A. N. 1990. Nonlinear Digital Filters-Principles and Applications. Kluwer Academic Publishers, Dordrecht. Poggio, T., and Girosi, F. 1990. Networks for approximation and learning. Proc. IEEE 78(9), 1481-1497. Received 5 March 1991; accepted 26 February 1993.

This article has been cited by: 2. Gecynalda S. da S. Gomes, Teresa B. Ludermir, Leyla M. M. R. Lima. 2010. Comparison of new activation functions in neural network for forecasting financial time series. Neural Computing and Applications . [CrossRef] 3. John W. Bandler, Mostafa A. Ismail, Jos� E. Rayas-S�nchez. 2001. Broadband physics-based modeling of microwave passive devices through frequency mapping. International Journal of RF and Microwave Computer-Aided Engineering 11:3, 156-170. [CrossRef] 4. A. Gati, M. F. Wong, G. Alquie, V. Fouad Hanna. 2000. Neural networks modeling and parameterization applied to coplanar waveguide components. International Journal of RF and Microwave Computer-Aided Engineering 10:5, 296-307. [CrossRef] 5. E. Gelenbe, Zhi-Hong Mao, Yan-Da Li. 1999. Function approximation with spiked random networks. IEEE Transactions on Neural Networks 10:1, 3-9. [CrossRef] 6. N. Sbirrazzuoli, D. Brunel, L. Elégant. 1997. Neural networks for kinetic parameters determination, signal filtering and deconvolution in thermal analysis. Journal of thermal analysis 49:3, 1553-1564. [CrossRef] 7. Fang Wang, Qi-Jun Zhang. 1997. Knowledge-based neural models for microwave design. IEEE Transactions on Microwave Theory and Techniques 45:12, 2333-2343. [CrossRef] 8. A. Materka, S. Mizushina. 1996. Parametric signal restoration using artificial neural networks. IEEE Transactions on Biomedical Engineering 43:4, 357-372. [CrossRef]

Communicated by john Platt and JohnLazzam

On an Unsupervised Learning Rule for Scalar Quantization following the Maximum Entropy Principle Marc M. Van Hulle’ Dominique Martinez+ Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, M A 02139 USA

A novel unsupervised learning rule, called Boundary Adaptation Rule (BAR),is introduced for scalar quantization. It is shown that the rule maximizes information-theoreticentropy and thus yields equiprobable quantizations of univariate probability density functions. It is shown by simulations that BAR outperforms other unsupervised competitive learning rules in generating equiprobable quantizations. It is also shown that our rule can do better or worse than the Lloyd I algorithm in minimizing average mean square error, depending on the input distribution. Finally, an application to adaptive non-uniform analog to digital ( A D )conversion is considered.

1 Introduction The main objective of scalar and vector quantization is to build discrete approximations to continuous functions such as input probability density functions (p.d.f.s)(Kohonen 1990). Two criteria have been widely used for designing quantizers: the minimization of the average distortion due to quantization and the maximization of information-theoreticentropy, i.e., ensure that each of the quantization intervals is used equally frequently in encoding the input signal (Ahalt et al. 1990). In general, these two criteria are not equivalent and a particular quantizer is only optimal with respect to a given design criterion. In case the input p.d.f. is not known a priori, the quantizer is constructed by a training process. We will restrict ourselves to this case only. In case of data representation and compression, the most often used techniques are batch algorithms based on the Lloyd I algorithm (see, e.g., Linde et al. 1980; Gersho and Gray 1991) and are aimed at minimizing average distortion. The major drawback of batch algorithms is that the ‘Present address: Laboratorium vmr Neuro- en Psychofysiologie, K. U. Leuven, Campus Gasthuisberg, Herestraat, B-3OOO Leuven, Belgium. +Presentaddress: Laboratoire d’Automatique et d’Analyse des SystPmes-CNRS, 7 Av. du Col. Roche, 31077 Todouse, France. Neural Computation 5,939-953 (1993) @ 1993 Massachusetts Institute of Technology

940

Marc M. Van Hulle and Dominique Martinez

design of the quantizer only begins after the entire training set is available. By consequence, these algorithms are not able to accommodate ”on-line” changes in the input p.d.f. A number of researchers have developed unsupervised competitive learning (UCL) algorithms for training artificial neural networks (ANNs) for the purpose of scalar and vector quantization. The quantizer is built “on the fly,” after the presentation of each input sample. In standard UCL, the weights of the network are updated so as to minimize average distortion. The quantization intervals are defined by nearestneighbor classification. However, as enunciated by Grossberg (1976a,b) and Rumelhart and Zipser (1985) among others, one problem with standard UCL is that some neurons may never win the competition and, therefore, never learn (dead units). In practical applications for data representation and compression, it is essential to add mechanisms to avoid dead units and to ensure an equitable distribution of weights in the input signal space. This has been done in Kohonen learning (Kohonen 1989)by adding a neigborhood to each neuron (Nasrabadi and Feng 1988; Naylor and Li 1988). In another approach, Grossberg (1976a,b) added a ”conscience” to frequently winning neurons to feel “guilty” and reduce their winning rate. Several researchers have introduced various methods inspired by the “conscience” mechanism (for references, see Hertz et al. 1991), usually with the purpose of achieving an equiprobable quantization, and thus maximizing entropy. In this article, an unsupervised learning rule is introduced for scalar quantization. The design criterion adopted is entropy maximization. The rule is completely different since it adapts the boundary points that separate the quantization intervals. Hence it is called Boundary Adaptation Rule (BAR). For this rule, it can be shown mathematically that it maximizes entropy (Van Hulle and Martinez, 1993). Due to this property, BAR can be used in a number of applications such as entropy estimation (Mokkadem 1989), and the formation of nonparametric models of input p.d.f.s (Silverman 1986). Here we show how the rule can be used for building a nonuniform A/D converter that is able to adapt itself to long-term drifts in sensor characteristics and changing environmental conditions. 2 Nonuniform Scalar Quantization

Scalar quantization transforms continuous-valued signals into a discrete number of quantization levels. In uniform quantization, the analog signal range R is partitioned into k equally sized regions, called quantization intervals or partition cells, separated by k - 1 equally spaced boundary points, Such a quantization is optimal only for stationary, uniform input distributions. In the general case, however, quantization should follow the distribution as closely as possible in order to quantize the distribution

Unsupervised Learning Rule for Scalar Quantization

941

efficiently given limited quantization resources. This way, the dynamic range that can be accommodated is significantly increased and a better nonparametric model of the input p.d.f. is obtained. A standard quantizer comprises an encoder and a decoder. The encoder is uniquely determined by the set of quantization intervals and the decoder by an associated set of output values. To formalize this, let x be a scalar input value and p ( x ) its p.d.f. Suppose that we have k nonoverlapping quantization intervals Di for partitioning the analog signal range R: k

R= u D i

with D i n D j = O ,

Vifj

(2.1)

i=l

Quantization intervals are encoded into digital codes. For this, let ActD, be a binary variable indicating code membership so that Act~, (X ) =

1 if x E Di 0 if x 6 Di

(2.2)

The corresponding code is then represented by the k-dimensional vector (Ac~D, ,Actq, . . . ,ActD,). Since the x are drawn from a p.d.f. p ( x ) , the probability of x falling in interval Di satisfies

p(Di) =

JD, ~ ( x dx) = S, Act~,(x)~ ( xdx) = E[Acb,]

(2.3)

with p ( R) = Ebl p( Di) = 1. The efficiency of quantization is proportional to how well the density of the k - 1 boundary points approximates the p.d.f. p ( x ) . Regardless of the type of p.d.f., we want each code to be active with an equal probability E[Act~,l= l / k . This way, the informationtheoretic entropy or channel capacity of the quantizer

is maximized and equal to log, k. In general, this implies a nonuniform quantization of the signal range R. Finally, the digital codes are decoded into analog output values. Define the set C of k output levels

c {yl,yZ,...,yk I yl E DlrYZ E DZi..*,ykE Dk)

(2.5)

as the quantizer’s codebook. 3 Boundary Adaptation Rule

In case of UCL for scalar (vector) quantization, the system compares the present input x with k weights Wi. If wi is “closest” to x then the ith neuron wins the competition and its output represents the code membership function for code i. Different rules exist for modifying the weights.(Hertz

Marc M. Van Hulle and Dominique Martinez

942

al. 1991) but in practice they amount to finding the wis representing the centroids of the quantization intervals; the boundary points are then marked by nearest-neighbor classification. In contrast, our approach does not rely on nearest-neighbor classification. Rather than finding the centroids of the quantization intervals, our BAR directly computes the boundary points. Let 6i-l and 6i be two boundary points that demarcate the interval Di so that Di = [6j-l,6j) for 1 < i < k; in case of D1 and Dk we have that D1 = (-co,&)and Dk I [6k-l, +co).The interval [61,6k-1] is called the dynamic range of the quantizer. Assume that for input x, Ac~D, = 1. We then modify Di by increasing 6;-l and decreasing 6;. In its simplest form, the rule reduces to et

BAR

A6; = q (ACtD,,, - A&,),

1Ii 0, and we wish to go downhill. The real gradient of S ( x ) would not give us any information, since it is zero at XO. If instead we look at the gradient of the function f(x), which is positive at xo and increases as xo -, 0, it tells us that the downhill direction is to decrease XO, which is also the case in S(x). In addition, the magnitude of the gradient tells us how close we are to a step down in S(x). Therefore, we can use that gradient as a heuristic hint as to which direction (and how close) a step down would be. This heuristic hint is what we used as the pseudo-gradient in our gradient update calculation.

sf-'/&!.

Zheng Zeng, Rodney Goodman, and Padhraic Smyth

986

..

c

oOAO .\,.;o,,*9, 0.20

t 0.00t, 0.00

,

,

,

,

0.20

0.40

0.60

0.80

-

.fl

0.00

0.00

30

0.40

0.60

080

1 1.00h0

h3

oao 0.60 1.00

0.40

0.60

-

0.20-

0.00

c

J.bo

*

O0.20 AI

oio o i o

0.k

oh

1.WhO

Figure 4 Discretized network learning Tomita grammar #4. (a) ho-h3 during 1st epoch of training. (b)ho-h3 during 15th epoch of training. (c) h&3 after 27 epochs when training succeeds, weights are fixed. (d) SO-s3, the discretized copy of h o d 3 in (c).

5 Experimental Results Shown in Figure 4a-c are the ho-h3 activation-space records of the learning process of a discretized network (h values are the undiscretized values

Learning Finite State Machines

987

from the sigmoids). The underlying grammar is again the Tomita Grammar #4. The parameters of the network and the training set are the same as in the previous case. Again, any of the other 2-Dplots from any run in learning any of the grammar in the experiments could have been used here. Figure 4c is the final result after learning, where the weights are fixed. Notice that there are only a finite number of points in the final plot in the analog activation h-space due to the discretization. Figure 4d shows the discretized value plot in SO-S~,where only three points can be seen. Each point in the discretized activation S-space is automatically defined as a distinct state, no point is shared by any of the states. The transition rules are calculated as before, and an internal state machine in the network is thus constructed. In this manner, the network performs self-clustering. For this example, six points are found in S-space, so a six-state-machine is constructed as shown in Figure 5a. Not surprisingly this machine reduces by Moore’s algorithm to a minimum machine with four states, which is exactly the Tomita Grammar #4 (Fig. 5b). Similar results were observed for all the other grammars in the experiments. There are several advantages in introducing discretization into the network: 1. Once the network has successfully learned the state machine from the training set, its internal states are stable. The network will always class@ input strings correctly, independent of the’lengths of these strings.

2. No clustering is needed to extract out the state machine, since instead of using vague clusters as its states, the network has formed distinct, isolated points as states. Each point in activation space is a distinct state. The network behaves exactly like a state machine. 3. Experimental results show that the size of the state machines extracted out in this approach, which need not be decided manually (no need to choose k for k-means) as in the previous undiscretized case, is much smaller than found previously by the clustering method.

It should be noted that convergence has a different meaning in the case of training discrete networks as opposed to the case of training analog networks. In the analog networks’ case, learning is considered to have converged when the error for each sample is below a certain error tolerance level. In the case of discrete networks, however, learning is stopped and considered to have converged only when zero error is obtained on all samples in the training set. In the experiments reported in this paper the analog tolerance level was set to 0.2. The discretized networks took on average 30% longer to train in terms of learning epochs compared to the analog networks for this specific error tolerance level.

988


Figure 5 Extracted state machine from the discretized network after learning Tomita grammar #4 (doublecircle means “accept”state, single circle means “reject’’ state). (a) Six-state machine extracted directly from the discrete activation space. (b)Equivalent minimal machine of (a). 6 Conclusion

In this paper we explored the formation of clusters in hidden unit activation space as an internal state representation for second-order recurrent networks that learn regular grammars. These states formed by such a network during learning are not a stable representation, i.e., when long strings are seen by the network the states merge into each other and eventually become indistinguishable. We suggested introducing hard-limiting threshold discretization into the network and presented a pseudo-gradient learning method to train such a network. The method is heuristically plausible and experimental results show that the network has similar capabilities in learning finite

Learning Finite State Machines

989

state machines as the original second-order network, but is stable regardless of string length since the internal representation of states in this network consists of isolated points in activation space. The proposed pseudogradient learning method suggests a general approach for training networks with threshold activation functions.

Acknowledgments The research described in this paper was supported in part by ONR and ARPA under Grants AFOSR-90-0199 and N00014-92-J-1860. In addition this work was carried out in part by the Jet Propulsion Laboratories, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.

References Angluin, D. 1972. Inference of reversible languages. 1.Assoc. Comput. Machin. 29(3), 741-765. Angluin, D. 1978. On the complexity of minimum inference of regular sets. Inform. Control 39, 337-350. Angluin, D., and Smith, C. H. 1983. Inductive inference: theory and methods. ACM Computing Survey, 15(3), 237. Carroll, J., and Long, D. 1989. Theory of Finite Automata. Prentice Hall, Englewood Cliffs, NJ. Cleeremans, A., Servan-Schreiber, D., and McClelland, J. L. 1989. Finite state automata and simple recurrent networks. Neural Comp. 1,372-381. Elman, J. L. 1990. Finding structure in time. Cog. Sci. 14, 179-211. Elman, J. L. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learn. 7(2/3), 195-225. Fahlman, S.E.1990. The recurrent cascade-correlationarchitecture. In Advances in Neural Information Processing Systems, pp. 190-196. Fu, K. S. 1982. Syntactic Pattern Recognition and Applications. Prentice Hall, Englewood Cliffs, NJ. Giles, C. L., Sun, G. Z., Chen, H. H., Lee, Y. C., and Chen, D. 1990. Higher order recurrent networks and grammatical inference. In Advances in Neural lnformation Processing Systems, pp. 380-387. Giles, C. L., Miller, C. B., Chen, D., Chen, H. H., Sun, G. Z., and Lee, Y. C. 1992. Second-order recurrent neural networks. Neural Comp. 4(3), 393-405. Gold, E.M.1972. System identification via state characterization. Automatica 8, 621-636. Gold, E. M. 1978. Complexity of automaton identification from given data. lnformation and Control 37, 302-320. Hopcroft, J.E., and Ullman, J. D. 1979.Introduction to Automata Theory, Languages and Computation. Addison-Wesley, Reading MA.

990


Jordan, M. I. 1986. Serial order: A parallel distributed processing approach. Tech. Rep. No.8604, San Diego: University of California, Institute for Cognitive Science. Kudo, M., and Shimbo, M. 1988. Efficient regular grammatical inference techniques by the use of partial similarities and their logical relationships. Pattern Recog. 21(4), 401409. Muggleton, S. 1990. Grammatical Induction Theory. Addison-Wesley, Turing Institute Press, Reading, MA. Pollack, J. 8. 1991. The induction of dynamical recognizers. Machine h r n . 7(2/3), 227-252. Rumelhart, D. E., McClelland, J. L., and the PDP Research Group. 1986. In Parallel Distributed Prowsing, pp. 354-361.The MIT Press. Servan-Schreiber,D., Cleeremans, A., and McClelland, J. L. 1991. Graded state machines: The representation of temporal contingenciesin simple recurrent networks. Machine Learn. 7(2/3),161-193. Tomita, M. 1982. Dynamic construction of finite-state automata from examples using hill climbing. In Proceedings of the Fourth Annual Cognitive Science Conference, p. 105. Williams, R. J., and Zipser, D. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Comp. 1(2), 270-280. Received 15 June 1992;accepted 8 March 1993.

This article has been cited by: 2. Takahumi Oohori, Hidenori Naganuma, Kazuhisa Watanabe. 2007. A New Backpropagation Learning Algorithm for Layered Neural Networks with Nondifferentiable UnitsA New Backpropagation Learning Algorithm for Layered Neural Networks with Nondifferentiable Units. Neural Computation 19:5, 1422-1435. [Abstract] [PDF] [PDF Plus] 3. Hidenori Naganuma, Takahumi Oohori, Kazuhisa Watanabe. 2007. A new error backpropagation learning algorithm for a layered neural network with nondifferentiable units. Electronics and Communications in Japan (Part III: Fundamental Electronic Science) 90:5, 40-49. [CrossRef] 4. Henrik Jacobsson. 2006. The Crystallizing Substochastic Sequential Machine Extractor: CrySSMExThe Crystallizing Substochastic Sequential Machine Extractor: CrySSMEx. Neural Computation 18:9, 2211-2255. [Abstract] [PDF] [PDF Plus] 5. V. P. Plagianakos, G. D. Magoulas, M. N. Vrahatis. 2006. Evolutionary training of hardware realizable multilayer perceptrons. Neural Computing and Applications 15:1, 33-40. [CrossRef] 6. Henrik Jacobsson . 2005. Rule Extraction from Recurrent Neural Networks: ATaxonomy and ReviewRule Extraction from Recurrent Neural Networks: ATaxonomy and Review. Neural Computation 17:6, 1223-1263. [Abstract] [PDF] [PDF Plus] 7. A. Vahed, C. W. Omlin. 2004. A Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural NetworksA Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural Networks. Neural Computation 16:1, 59-71. [Abstract] [PDF] [PDF Plus] 8. Peter Tiňo , Bill G. Horne , C. Lee Giles . 2001. Attractive Periodic Sets in Discrete-Time Recurrent Networks (with Emphasis on Fixed-Point Stability and Bifurcations in Two-Neuron Networks)Attractive Periodic Sets in Discrete-Time Recurrent Networks (with Emphasis on Fixed-Point Stability and Bifurcations in Two-Neuron Networks). Neural Computation 13:6, 1379-1414. [Abstract] [PDF] [PDF Plus] 9. Rafael C. Carrasco , Mikel L. Forcada , M. Ángeles Valdés-Muñoz , Ramón P. Ñeco . 2000. Stable Encoding of Finite-State Machines in Discrete-Time Recurrent Neural Nets with Sigmoid UnitsStable Encoding of Finite-State Machines in Discrete-Time Recurrent Neural Nets with Sigmoid Units. Neural Computation 12:9, 2129-2174. [Abstract] [PDF] [PDF Plus] 10. A. Blanco, M. Delgado, M. C. Pegalajar. 2000. Extracting rules from a (fuzzy/crisp) recurrent neural network using a self-organizing map. International Journal of Intelligent Systems 15:7, 595-621. [CrossRef]

11. S. Lawrence, C.L. Giles, S. Fong. 2000. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering 12:1, 126-140. [CrossRef] 12. C.L. Giles, C.W. Omlin, K.K. Thornber. 1999. Equivalence in knowledge representation: automata, recurrent neural networks, and dynamical fuzzy systems. Proceedings of the IEEE 87:9, 1623-1640. [CrossRef] 13. C.W. Omlin, K.K. Thornber, C.L. Giles. 1998. Fuzzy finite-state automata can be deterministically encoded into recurrent neural networks. IEEE Transactions on Fuzzy Systems 6:1, 76-89. [CrossRef] 14. Alan D. Blair, Jordan B. Pollack. 1997. Analysis of Dynamical RecognizersAnalysis of Dynamical Recognizers. Neural Computation 9:5, 1127-1142. [Abstract] [PDF] [PDF Plus] 15. S.C. Kremer. 1996. Comments on "Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution". IEEE Transactions on Neural Networks 7:4, 1047-1051. [CrossRef] 16. Christian W. Omlin, C. Lee Giles. 1996. Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid DiscriminantsStable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants. Neural Computation 8:4, 675-696. [Abstract] [PDF] [PDF Plus] 17. Paolo Frasconi, Marco Gori, Marco Maggini, Giovanni Soda. 1996. Representation of finite state automata in Recurrent Radial Basis Function networks. Machine Learning 23:1, 5-32. [CrossRef] 18. Kam-Chuen Jim, C.L. Giles, B.G. Horne. 1996. An analysis of noise in recurrent neural networks: convergence and generalization. IEEE Transactions on Neural Networks 7:6, 1424-1438. [CrossRef] 19. R. Alquézar , A. Sanfeliu . 1995. An Algebraic Framework to Represent Finite State Machines in Single-Layer Recurrent Neural NetworksAn Algebraic Framework to Represent Finite State Machines in Single-Layer Recurrent Neural Networks. Neural Computation 7:5, 931-949. [Abstract] [PDF] [PDF Plus] 20. Peter Tiňo , Jozef Šajda . 1995. Learning and Extracting Initial Mealy Automata with a Modular Neural Network ModelLearning and Extracting Initial Mealy Automata with a Modular Neural Network Model. Neural Computation 7:4, 822-844. [Abstract] [PDF] [PDF Plus]

991

Index Volume 5 By Author Abbott, L. E and LeMasson, G. Analysis of Neuron Models with Dynamically Regulated Conductances (Article)

5(6):823-842

Abu-Mostafa, Y.S. Hints and the VC Dimension (Letter)

5(2):278-288

Amari, S. and Murata, N. Statistical Theory of Learning Curves under Entropic Loss Criterion (Letter)

5(1):140-153

Amit, D. J.

- See Griniasty, M.

Atick, J. J. and Redlich, A. N. Convergent Algorithm for Sensory Receptive Field Development (Letter)

5(1h45-60

Ayala, G. F. - Migliore, M. Atlan, H. - Rosenberg, C. Baddeley, R. J. - See Cairns, D. E. Baldi, P. and Chauvin, Y. Neural Networks for Fingerprint Recognition (Letter)

5(3):402418

Back, A. D. and Tsoi, A. C. A Simplified Gradient Algorithm for IIR Synapse Multilayer Perceptrons (Letter)

5(3):45&462

Bartlett, I? L. Vapnik-Chervonenkis Dimension Bounds for Two- and Three-Layer Networks (Note)

5(3):371-373

Becker, S. and Hinton, G. E. Learning Mixture Models of Spatial Coherence (Letter)

5(2):267-277

Bialek, W. - See Kruglyak, L. Blakemore, C. - See Nicoll, A. Blazis, D. E. J., Fischer, T. M. and Carew, T. J. A Neural Network Model of Inhibitory Information Processing in Aplysia (Letter)

5(2):213-227

992

Borst, A., Engelhaaf, M., and Seung, H. S. Two-Dimensional Motion Perception in Flies (Letter)

Index

5(6):856-868

Bottou, L. - Vapnik, V. Bower, J. M. - See De Schutter, E. Bowtell, G. - See Ferrar, C. H. Bromley, J. and Denker, J. S. Improving Rejection Performance on Handwritten Digits by Training with "Rubbish (Note) Buhmann, J. and Kuhnel, H. Complexity Optimized Data Clustering by Competitive Neural Networks (Letter) Cairns, D. E., Baddeley, R. J., and Smith, L. S. Constraints on Synchronizing Oscillator Networks (Letter)

5(3):367-370

5(3):75-88

5(2):260-266

Carew, T. J. - See Blazis, D. E. J. Changeux, J. P. - Kerzberg, M. Chauvin, Y. - See Baldi, l? Chiel, H. J. - See Srinivasan, R. Chen, A. M., Lu, H., and Hecht-Nielsen, R. On the Geometry of Feedforward Neural Network Error Surfaces (Letter)

5(6):91&927

Cho, S. and Reggia, J. A. Learning Competition and Cooperation (Letter)

5(2)~242-259

Choi, H. - See Phatak, D. S. Conwell, P. R. - See Cotter, N. E. Cotter, N. E. and Conwell, P. R. Universal Approximation by Phase Series and Fixed-Weight Networks (Note)

5(3)~359-362

Dayan, P. Arbitrary Elastic Topologies and Ocular Dominance (Letter)

5(3):392-401

Dayan, P. Improving Generalization for Temporal Difference Learning: The Successor Representation (Letter)

5(4):613-624

Index

993

Dayan, P. and Sejnowski, T. J. The Variance of Covariance Rules for Associative Matrix Memories and Reinforcement Learning (Note)

5(2):205-209

Deco, G. and Ebmeyer, J. Coarse Coding Resource-Allocating Network (Letter)

5(1):105-114

DeFelice, L. J. - See Strassberg, A. F. Denby, B. The Use of Neural Networks in High-Energy Physics (Review)

5(4):505-549

Denker, J. S. - See Bromley, J. De Schutter, E. and Bower, J. M. Sensitivity of Synaptic Plasticity to the CA2+ Permeability of NMDA Channels: A Model of Long-Term Potentiation in Hippocampal Neurons (Letter)

5(5):681-694

Dorronsoro, J. R. - Lopez, V. Dreyfus, G. - Linster, C.

Dreyfus, G. - See Nerrand, 0. Ebmeyer, J. - See Deco, G. Edelman, S. - See Weiss, Y. Elias, J. G. Artificial Dendritic Trees (Letter)

5(4):64&664

Engelhaaf, M. - See Borst, A. Erel, J. - See Rosenberg, C. Fahle, M. - See Weiss, Y. Ferrar, C. H., Williams, T. L., and Bowtell, G. The Effects of Cell Duplication and Noise in a Pattern Generating Network (Letter)

5(4):587-596

Fischer, T. M. - See Blazis, D. E. J. Floreen, P. and Orponen, P. Attraction Radii in Binary Hopfield Nets are Hard to Compute (Letter)

5(5):812-821

Index

994

Gelenbe, E. Learning in the Recurrent Random Neural Network (Letter)

5(1)A54164

Gold, J. I. - See Intrator, N. Golea, M. and Marchand, M. On Learning Perceptrons with Binary Weights (Letter)

5(5):767-782

Goodman, R. M. - See Zeng, Z. Govil, S. - See Mukhopadhyay, S. Grannan, E. R., Kleinfeld, D. and Sompolinsky, H. Stimulus-Dependent Synchronization of Neuronal Assemblies (Article)

5(4):55&569

Griniasty, M., Tsodyks, M. V., and Amit, D. J. Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors (Article)

5(1)A-17

Hall, T. J. - Kendall, G. D. Hasselmo, M. E. Acetylcholine and Learning in a Cortical Associative Memory (Letter)

5(1):32-44

Haykin, S. - See Leung, H. Hecht-Nielsen, R.

- See Chen, A. M.

Herrmann, M. - See Horn, D. Hinton, G. E. - See Becker, S. Horn, D., Ruppin, E., Usher, M. and Herrmann, M. Neural Network Modeling of Memory Deterioration (Letter)

5(5):736749

Huerta, R. - See Lopez, V. Intrator, N. Combining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural Networks (Letter) Intrator, N. and Gold J. I. Three-Dimensional Object Recognition Using an Unsupervised BCM Network The Usefulness of Distinguishing Features (Letter)

5(3):443-457

5(1):61-74

Index

995

Kadirkamanathan, V. and Niranjan, M. A Function Estimation Approach to Sequential Learning with Neural Networks (Letter)

5(6):954-975

Kendall, G. D. and Hall, T. J. Optimal Network Construction by Minimum Description Length (Note)

5(2):210-212

Kerlirzin, P. and Vallet, F. Robustness in Multilayer Perceptrons (Letter)

5(3):473482

Kerszberg, M. and Changeux, J. P. A Model for Motor Endplate Morphogenesis: Diffusible Morphogens, Transmembrane Signaling, and Compartmentalized Gene Expression (Article)

5(3):341-358

Kersberg, M. - See Linster, C. Kidder, J. N. and Seligson, D. Fast Recognition of Noisy Digits (Letter)

5(6):885-892

Kim, L. S. - See Mukhopadhyay, S. Kleinfeld, D. - See Grannan, E. R. Konen, W.and Von der Malsberg, C. Learning to Generalize from Single Examples in the Dynamic Link Architecture (Letter)

5(5):719-735

Koren, I. - See Phatak, D. S. Kottas, J. A. Training Periodic Sequences Using Fourier Series Error Criterion (Letter)

5(1):115-131

Kruglyak, L. and Bialek, W. Statistical Mechanics for a Network of Spiking Neurons (Letter) 5(1):21-31 Kuhnel, H. - See Buhmann, J. Lappe, M. and Rauschecker, J. P. A Neural Network for the Processing of Optic Flow from Ego-Motion in Man and Higher Mammals 5(3):374391 (Letter) LeMasson, G. - See Abbott, L. F. Leung, H. and Haykin, S. Rational Function Neural Network (Letter)

5(6):928-938

996

Lin, J. and Unbehauen, R. On the Realization of a Kolmogorov Network (Note)

Index

5(1):18-20

Linster, C., Masson, C. Kerszberg, M., Personnaz, L., and Dreyfus, G. Computational Diversity in a Formal Model of the Insect Olfactory Macroglomerulus (Letter) Lopez, V.,Huerta, R., and Dorronsoro, J. R. Recurrent and Feedforward Polynomial Modeling of Coupled Time Series (Letter) Lu, H. - See Chen, A. M. Marchand, M. - Golea, M. Marcos, S. - See Nerrand, 0. Martin, G. L. Centered-Object Integrated Segmentation and Recognition of Overlapping Handprinted Characters (Letter) Martinez, D. - See Van Hulle, M. M. Migliore, M. and Ayala, G. F. A Kinetic Model of Short- and Long-Term Potentiation (Letter)

5(4):636-647

Mukhopadhyay, S., Roy, A,, Kim, L. S., and Govil, S. A Polynomial Time Algorithm for Generating Neural Networks for Pattern Classification: Its Stability Properties and Some Test Results (Letter)

5(2):317-330

Murata, N. - See Amari, S. Nerrand, O., Roussel-Ragot, P., Personnaz, L., Dreyfus, G., and Marcos, S. Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms 5(2):165-199 (Review) Nicoll, A. and Blakemore, C. Patterns of Local Connectivity in the Neocortex (Letter) Niebur, E. - See Usher, M. Niranjan, M. - See Kadirkamanathan, V.

5(5):665-680

Index Ohlsson, M., Peterson, C., and Soderberg, B. Neural Networks for Optimization Problems with Inequality Constraints: The Knapsack Problem (Letter)

997

5(2):331-339

Orponen, l? - See Floreen, l? Park, J. and Sandberg, I. W. Approximation and Radial-Basis-Function Networks (Letter)

5(3):305-316

Pentland, A. I? Surface Interpolation Networks (Letter)

5(3):430-442

Personnaz, L. - See Linster, C. Peterson, C. - See Ohlsson, M. Personnaz, L. - See Nerrand, 0. Phatak, D. S., Choi, H., and Koren, I. Construction of Minimal n-2-n Encoders for Any n (Letter)

5(5):783-794

Prelinger, D. - See Schmidhuber, J. Rauschecker, J. P. - See Lappe, M. Redish, A. D. - See Touretzky, D. S. Redlich, A. N. Redundancy Reduction as a Strategy for Unsupervised Learning (Letter)

5(2):289-304

Redlich, A. N. Supervised Factorial Learning (Letter)

5(5):750-766

Redlich, A. N. - See also Atick, J. J. Reggia, J. A. - See Cho, S. Rognvaldsson, T. Pattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior (Letter)

5(3):483-491

Rosenberg, C., Erel, J., and Atlan, H. A Neural Network That Learns to Interpret Myocardial Planar Thallium Scintigrams (Letter)

5(3):492-502

Roussel-Ragot, I? - See Nerrand, 0. Roy, A. - See Mukhopadhyay, S.

998

Index

Ruppin, E. - See Horn, D. Sandberg, I. W. - See Park, J. Schmidhuber, J. and Prelinger, D. Discovering Predictable Classifications (Letter)

5(4):625435

Schuster, H. G. - See Usher, M. Sejnowski, T. J. - See Dayan, P. Seligson, D. - Kidder, J. Sereno, M. E. - See Zhang, K. Sereno, M. I. - See Zhang, K. Seung, H. S. - See Botst, A. Smith, L. E. - See Cairns, D. E, Smyth, P. - See Zeng, Z. Soderberg, 8. - See Ohlsson, M. Sompolinsky, H. - Grannan, E. R. Srinivasan, R. and Chiel, H. J. Fast Calculation of Synaptic Conductances (Note)

5(2):20&204

Strassberg, A. F. and DeFelice, L. J. Limitations of the Hodgkin-Huxley Formalism: Effects of Single Channel Kinetics on Transmembrane Voltage Dynamics (Letter)

5(6):843-855

Takahashi, Y. Generalization and Approximation Capabilities of Multilayer Networks (Letter)

5(1):132-139

Tanaka, T. and Yamada M. The Characteristics of the Convergence Time of Associative Neural Networks (Letter)

5(3):463-472

Touretzky, D. S., Redish, A. D., and Wan, H. S. Neural Representation of Space Using Sinusoidal Arrays (Letter)

5(6):869-884

Tsodyks, M. V. - See Griniasty, M. Tsoi, A. C. - See Back, A. D. Unbehauen, R. - Lin, J.

Index

Usher, M., Schuster, H. G., Niebur E. Dynamics of Populations of Integrate-and-Fire Neurons, Partial Synchronization and Memory (Letter)

999

5(4):570-586

Usher, M. - See Horn, D. Vallet, F. - See Kerlirzin, I? Van Hulle, M. M. and Martinez, D. On an Unsupervised Learning Rule for Scalar Quantization Following the Maximum Entropy Principle (Letter)

5(6)~939-953

Vapnik, V. and Bottou, L. Local Algorithms for Pattern Recognition and Dependencies Estimation

5(6):893-909

Von der Malsberg, C. - See Konen, W. Wan, H. S . - See Touretzky, D. S. Weiss, Y., Edelman, S., and Fahle, M. Models of Perceptual Learning in Vernier Hyperacuity (Letter)

5(5):695-718

Williams, T. L. - See Ferrar, C. H. Wong, Y. Clustering Data by Melting (Letter)

5(1):89-104

Yamada, M. - See Tanaka, T. Yang, L. and Yu, W. Backpropagation with Homotopy (Note)

5(3):363-366

Yu, W. - See Yang, L. Zeng, Z., Goodman, R., and Smyth, P. Learning Finite State Machines With SelfClustering Recurrent Networks (Letter)

5(6):976-990

Zhang, K., Sereno, M. I., and Sereno, M. E. Emergence of Position-Independent Detectors of Sense of Rotation and Dilation with Hebbian Learning: An Analysis (Letter)

5(4):597-612

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

Recommend Documents